Driving instruction support system

Abstract

The aim is to provide a driving instruction support device that can facilitate a shared understanding between trainee drivers and instructors regarding the driving behavior of trainee drivers, and that can be easily and inexpensively constructed. [Solution] The driving instruction support device comprises a first smartphone 1 that captures driving scene video, a second smartphone 2 that acquires information regarding the driver's gaze, and a tablet terminal 5 that displays a dashboard screen 60. The video display unit 61 of the dashboard screen 60 displays a superimposed image in which the gaze points 71A to 71C of the driver being instructed are superimposed on the driving scene video.

Description

Technical Field

[0001] The present invention relates to a driving training support device for supporting driving training in a driving school or the like.

Background Art

[0002] Conventionally, various technologies have been proposed to appropriately evaluate and improve the driving skills of vehicle drivers. For example, as a technology for improving the quality of driving skill training in a driving school, Patent Document 1 (Patent No. 6086515) discloses an invention in which, in a driving skill evaluation device that evaluates the driving skills of a subject, a driving operation (vehicle behavior) extracted by CAN or an external server is compared with reference data to perform skill evaluation. Further, Patent Document 2 (PCT / JP2018 / 33295) discloses an invention in which the result of driving skill evaluation is displayed on a screen in a driving skill evaluation system. Further, Patent Document 3 (Patent No. 7135913) discloses an invention in which, in a driving evaluation screen showing the driving situation of a vehicle driver, a reproduced image of the driver, a graph of detection data indicating the driver situation and the vehicle running state, and a map showing the running route of the vehicle are displayed on one screen.

[0003] Furthermore, as a technology for evaluating driving skills by focusing on the driver's gaze, for example, Patent Document 4 (Patent No. 7235438) discloses an invention for a driving evaluation device that performs driving evaluation based on the driver's gaze detection data at the evaluation point and the deviation of the gaze calculated from the point data. Also, Patent Document 5 (Japanese Unexamined Patent Publication No. 2004-157880) discloses an invention for a confirmation behavior evaluation device that evaluates the confirmation behavior of an occupant based on the occupant's behavior and gaze behavior. Also, Patent Document 6 (Japanese Unexamined Patent Publication No. 2023-104122) discloses an invention for an information processing device that performs driving evaluation based on the degree of agreement between the driver's gaze of a moving object (vehicle) and the object being confirmed (traffic object), and generates insurance information regarding the driver's insurance coverage (insurance premium). Also, Patent Document 7 (Patent No. 6758788) discloses an invention for a display control device that changes the displayed content according to the driver's gaze detected by a smartphone installed on the instrument panel.

[0004] Furthermore, as a technology that develops upon the driver's gaze-based driving evaluation, Patent Document 8 (Japanese Patent Application No. 2023-051384) by the present applicant proposes an invention that estimates the driver's level of interest (an indicator showing what the driver is interested in) based on the relationship between the driver's gaze and the saliency at various points in the vehicle's forward field of view. In addition, Patent Document 9 (Japanese Patent Application No. 2023-051385) by the present applicant discloses an invention that builds upon the invention of Patent Document 8, by dividing the forward field of view into multiple divided regions and calculating the driver's level of interest based on the representative saliency value in the divided region toward which the driver's gaze is directed, thereby simplifying the configuration and reducing costs. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Patent No. 6086515 [Patent Document 2] PCT / JP2018 / 33295 [Patent Document 3] Patent No. 7135913 [Patent Document 4] Patent No. 7235438 [Patent Document 5] Japanese Patent Publication No. 2004-157880 [Patent Document 6] Japanese Patent Publication No. 2023-104122 [Patent Document 7] Patent No. 6758788 [Patent Document 8] Patent application No. 2023-051384 [Patent Document 9] Patent application No. 2023-051385 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] As described above, various technologies have been proposed to evaluate and improve the driving skills of vehicle drivers. However, conventional technologies have not specifically focused on improving the relationship between students and instructors in driving schools. In other words, when students improve their driving skills under the guidance of an instructor, such as in driving lessons, communication based on a shared understanding between the student and the instructor is especially important. For example, even if an instructor gives appropriate feedback to a student during driving lessons, if the student does not have a correct understanding of their own driving, and there is a gap in understanding between the instructor and the student (for example, if the student believes they are performing the driving actions the instructor wants them to, even though they are not actually doing so), the student will not be able to readily accept the instructor's guidance, and appropriate guidance will not be possible. Therefore, in driving lessons, there is a particular need for technologies that enable students to correctly understand their own driving.

[0007] On the other hand, considering actual use in driving schools, equipping each training vehicle with expensive devices such as ADAS (Advanced Driver-Assistance Systems) or systems utilizing artificial intelligence or autonomous driving to accurately understand (detect) the driving status of trainees would be prohibitively expensive and impractical. Therefore, even if a system is introduced to promote correct awareness among trainees, it needs to be low-cost and simple to be usable for consistent driving instruction in real driving schools.

[0008] This invention was made in consideration of the circumstances described above, and aims to provide a driving instruction support device that can promote a shared understanding between the driver being trained and the instructor (trainer) regarding the driver's driving, and that can be easily constructed at low cost. [Means for solving the problem]

[0009] To achieve the above objective, the present invention adopts the following solution. That is, as described in claim 1, the driving instruction support device comprises: a driving scene video acquisition terminal that acquires driving scene video, which is a video of the front of the vehicle in a driving scene; a gaze information acquisition terminal that acquires information about the gaze of the driver of the vehicle while it is driving in the driving scene; and a driving instruction screen display terminal that can display a driving instruction screen including a superimposed display in which the driver's gaze point is superimposed on the driving scene video, which is generated based on the driving scene video acquired by the driving scene video acquisition terminal and the information about the driver's gaze acquired by the gaze information acquisition terminal.

[0010] According to the above solution, the driver's gaze points are superimposed on the driving scene video on the training screen (for example, dashboard screens 60, 80, 90, 100, and 110), making it possible to visualize the driver's gaze (position and trajectory of gaze points) while driving and objectively recognize problems in how the trainee driver distributes their gaze. Therefore, it is possible to conduct driving evaluations based on objective evidence that are easy for trainees to accept, thus reducing the gap in perception between trainees and instructors in driving instruction. It also reduces individual differences in judgment among instructors. Furthermore, it is possible to accurately grasp the habits and shortcomings of each trainee, making it possible to provide advice tailored to each trainee. Thus, it is possible to conduct effective driving instruction that is easy for trainees to accept.

[0011] A preferred embodiment based on the above solution method is as described in claim 2 and subsequent claims of the patent. That is, the training screen display terminal can superimpose the gaze points of the model driver who drove the driving scene onto the driving scene video on the training screen (corresponding to claim 2). In this case, the difference in gaze points between the trainee driver and the model driver can be confirmed at a glance on the driving scene video, so that problems with the trainee driver's line of sight can be accurately grasped.

[0012] The system includes a terminal for acquiring passenger gaze information that acquires information regarding the gaze of the passenger in the passenger seat of the vehicle while it is in motion, and the training screen display terminal can superimpose the gaze points of the passenger calculated based on the information regarding the passenger's gaze onto the driving scene video on the training screen (corresponding to claim 3). In this case, the instructor sitting in the passenger seat can confirm the areas that the training driver is paying attention to while the training driver is driving, based on the gaze points of the passenger. By comparing the gaze points of the driver and the passenger, problems with the training driver's gaze can be accurately identified.

[0013] The system includes a salience calculation means for calculating the salience in the driving scene video, and the training screen display terminal can add the salience in the driving scene video calculated by the salience calculation means to the driving scene video and display it (corresponding to claim 4). In this case, the driver's eye movements can be accurately analyzed based on the salience in the driving scene video.

[0014] The system includes an interest level calculation means that calculates an index of the driver's level of interest based on information regarding the driver's gaze and the saliency in the driving scene video (corresponding to claim 5). In this case, since the index of interest level (e.g., interest level index) for each scene of the driving scene video can be checked, it is possible to accurately evaluate whether the driver was driving with an appropriate level of interest.

[0015] The aforementioned training screen display terminal is configured to allow writing within the driving scene video on the training screen (corresponding to claim 6). In this case, by writing within the driving scene video, it is possible to specifically indicate areas that require attention in the driving scene video, thereby enabling concrete and easy-to-understand driving instruction for trainees.

[0016] The aforementioned terminal for acquiring driving scene video and the aforementioned terminal for acquiring gaze information are smartphones (corresponding to claim 7). In this case, the terminal for acquiring driving scene video and the terminal for acquiring gaze information can be easily and inexpensively configured using existing general-purpose smartphones (for example, the first smartphone 1 and the second smartphone 2).

[0017] The aforementioned training screen display terminal is a tablet terminal (corresponding to claim 87). In this case, the training screen display terminal can be easily and inexpensively configured using an existing general-purpose tablet terminal (for example, tablet terminal 5). [Effects of the Invention]

[0018] According to the present invention, since the driver's line of sight is superimposed and displayed on the driving scene video, problems in the driver's line of sight can be objectively recognized, and effective driving training that is easy for trainees to accept can be performed. In addition, the driving training support device can be configured simply and at low cost using existing general-purpose products (for example, smartphones and tablet terminals).

Brief Description of the Drawings

[0019] [Figure 1] A block configuration diagram showing an example of the control system of the driving training support device of the present invention. [Figure 2] A diagram showing an example of installation of the driving training support device of the present invention on a vehicle. [Figure 3] A diagram showing an example of installation of the driving support device on a vehicle as well. [Figure 4] A diagram showing another example of installation of the driving support device on a vehicle as well. [Figure 5] A diagram showing another example of installation of the driving support device on a vehicle as well. [Figure 6] A perspective view showing another example of a fixing member for fixing the first and second smartphones. [Figure 7] A diagram for explaining a method of superimposing the driver's line of sight and the driving scene video. [Figure 8] A diagram showing an example of a training screen (dashboard screen) in the present invention. [Figure 9] A diagram showing an example of display in a video display section within the training screen. [Figure 10] A diagram showing an example of display in a graph display section within the training screen. [Figure 11] A diagram showing another example of the training screen in the present invention. [Figure 12] A diagram showing still another example of the training screen in the present invention. [Figure 13] A diagram showing still another example of the training screen in the present invention. [Figure 14] A diagram showing still another example of the training screen in the present invention.

Embodiments for Carrying Out the Invention

[0020] Embodiments of the present invention will be described below based on the attached drawings. Figure 1 shows an example of the control system of the driver training support device of the present invention. Figures 2 and 3 show examples of the installation of the driver training support device of the present invention in a vehicle.

[0021] The embodiments described below are primarily intended for use within a driving school, and their use is limited to use within the driving school (not on public roads) (however, this does not apply if it complies with legal standards). In order to use the driving school support device of the present invention outside of a driving school (for example, on public roads), it is necessary to appropriately modify and adjust the specific form to fully comply with actual legal standards (for example, legal standards regarding driving on public roads).

[0022] As shown in the diagram, the driver training support device includes a first smartphone 1, a second smartphone 2, a third smartphone 2A, a CAN measuring device 3, a GPS positioning device 4, and a tablet terminal 5.

[0023] The first smartphone 1 is a terminal for acquiring driving scene video, which is a video (hereinafter referred to as "driving scene video") that is continuously captured in chronological order of the forward view of the vehicle, that is, the forward view of the vehicle driver. In this embodiment, the first smartphone 1 acquires (records) the driving scene video using the camera function (video recording function) of the smartphone.

[0024] On the other hand, the second smartphone 2 is a terminal for acquiring gaze information to obtain information about the vehicle driver's gaze (time-series data on which direction the driver's gaze is directed in the forward field of view). In this embodiment, the second smartphone 2 captures the driver's facial expression using the camera function provided in the smartphone, and calculates the driver's gaze direction using a gaze measurement application built into the smartphone (software that calculates the gaze direction from the orientation of the driver's face and the direction of their eyes captured in the image).

[0025] Furthermore, the third smartphone 2A is a terminal for acquiring passenger gaze information to obtain information regarding the gaze of the passenger in the front seat (for example, an instructor in a driving lesson) (time-series data on which direction the passenger's gaze is directed in the forward field of view). Similar to the second smartphone 2, the third smartphone 2A also uses the smartphone's camera function to capture the driver's facial expression and calculates the driver's gaze direction using a built-in gaze measurement app.

[0026] The first smartphone is attached to the vehicle by a mounting device 6 and positioned above the dashboard 11 inside the passenger compartment 10. More specifically, the mounting device 6 comprises a suction part 6a at the base end, a holding part 6b at the tip end, and an intermediate arm part 6c. The suction part 6a is attached near the lower left end of the front windshield 12, and the tip part 6b holds the smartphone 1. As a result, the first smartphone 1 is positioned above the left end of the dashboard 11. Furthermore, the first smartphone 1 is positioned horizontally, allowing it to capture forward-facing images over a wide range in the left-right direction.

[0027] Meanwhile, the second smartphone 2 is attached to the back of the rearview mirror 13 by a fixing device (not shown). Smartphone 2A is attached to the back of the passenger-side rearview mirror 13A, which is positioned adjacent to the rearview mirror 13, by a fixing device (not shown). In this case, the cameras of smartphones 2 and 2A are positioned so that their lenses protrude to the side of the rearview mirrors 13 and 13A, so that they can capture the facial expressions (face and eye direction) of the driver and passenger.

[0028] Thus, this embodiment includes a single smartphone 1 shared by the driver and the front passenger for recording driving scene video (forward video), and two smartphones for eye-tracking, one for the driver and one for the front passenger, namely a smartphone 2 for driver eye-tracking and a smartphone 2A for front passenger eye-tracking. However, the present invention is not limited to this form.

[0029] For example, a configuration could be adopted in which separate smartphones are provided on the driver's side and the passenger's side for recording driving scene footage (forward-facing footage) (i.e., a separate smartphone for recording forward-facing footage on the driver's side and a separate smartphone for recording forward-facing footage on the passenger's side). In this case, the forward-facing smartphone on the driver's side (for the driver) can record the forward-facing footage as seen by the driver (the driver's view), while the forward-facing smartphone on the passenger's side (for the passenger) can record the forward-facing footage as seen by the passenger (the passenger's view).

[0030] Furthermore, a configuration can be adopted in which only a smartphone for measuring the driver's gaze is provided, and no smartphone for measuring the passenger's gaze is provided. Figures 4 and 5 show an example of such a configuration. It should be noted that the embodiments shown in Figures 4 and 5 are basically limited to use within a driving school (not on public roads), and in order to use them outside of a driving school (for example on public roads), the configuration will need to be modified and adjusted to fully comply with legal standards.

[0031] In the examples shown in Figures 4 and 5, the second smartphone 2 is attached to the vehicle by a fixing device 7 and positioned above the dashboard 11 inside the passenger compartment 10. The fixing device 7 comprises a suction part 7a at the base end, a holding part 7b at the tip end, and an arm part 7c in the middle. The suction part 7a is attached near the lower right end of the front windshield 12, and the tip part 7b holds the smartphone 2. As a result, the second smartphone 2 is positioned above the right end of the dashboard 11. Furthermore, the second smartphone 2 is positioned vertically to facilitate capturing the driver's face (facial expression).

[0032] Furthermore, various installation methods and fasteners different from the examples described above can be used to install smartphones 1 and 2. For example, a fastener like the one shown in Figure 6 can be used. The fastener 50 shown in Figure 6 comprises a first support part 51 for supporting the first smartphone 1, a second support part 52 for supporting the second smartphone 2, and a base part 53 to which the support parts 51 and 52 are connected. With the fastener 50, the first and second smartphones 1 and 2 can be compactly installed on the dashboard 11 by fixing the base part 53 to a suitable position on the dashboard 11 (for example, behind (in front of) the car navigation system display 18).

[0033] Information regarding the driver's gaze (time-series data of the driver's gaze direction) acquired by the second smartphone 2 is superimposed with time-series data of the driving scene video acquired by the first smartphone 1, generating a superimposed image in which the driver's point of fixation (the point the driver is looking at) is added to the driving scene video. Similarly, information regarding the passenger's gaze (time-series data of the passenger's gaze direction) acquired by smartphone 2A is also superimposed with time-series data of the driving scene video acquired by the first smartphone 1, generating a superimposed image in which the passenger's point of fixation (the point the passenger is looking at) is added to the driving scene video.

[0034] In this embodiment, information regarding the driver's and passenger's gaze is transmitted from the second smartphone 2 and the third smartphone 2A to the first smartphone 1, for example, by wireless communication (Wi-Fi, Bluetooth, etc.), and an overlaid image is generated on the first smartphone 1. However, the generation of the overlaid image does not necessarily have to be done on the first smartphone 1; for example, it may be done on the second and third smartphones 2 and 2A or a tablet terminal 5.

[0035] Here, we will briefly explain the method of superimposing the driver's viewpoint onto the driving scene video, following Figure 7. The method for superimposing the passenger's viewpoint onto the driving scene video is similar, so we will explain it using the driver's viewpoint as a representative example below. In superimposing the driver's viewpoint, first, the first smartphone 1 that shoots the driving scene video and the second smartphone 2 that acquires the driver's viewpoint are calibrated. That is, in this embodiment, the driver's face 41 is captured by the second smartphone 2, which is different from the first smartphone 1 that shoots the driving scene video. Therefore, when smartphones 1 and 2 are set up, their coordinates do not match, and it is necessary to correct (align) the positional relationship of the two smartphones 1 and 2. For this reason, the position of the driver's face 41 is determined by performing calibration with the first smartphone 1 as the reference.

[0036] In this way, once the position of the driver's face 41 is identified, the driving scene video is projected onto a plane 42 5m away, and the point where the driver's line of sight 43, calculated based on the orientation of the driver's face 41 and the direction of their eyes acquired by the second smartphone, intersects with the plane 42 is defined as the driver's point of gaze 44 in the driving scene video (projected image).

[0037] Returning to Figures 1-3, the CAN measuring instrument 4 is a device that acquires information regarding the vehicle's operating state (measured data from various detection means 21 installed in the vehicle) via CAN (Controller Area Network). The information regarding the operating state acquired by the CAN measuring instrument 4 can include any information regarding the vehicle's operating state (vehicle behavior and driving operations by the driver). For example, time-series data of various measured values ​​indicating the vehicle's driving state, such as accelerator opening, steering angle, and G (gravitational acceleration) acting on the driver, as well as data regarding driving operations, such as whether the turn signal was used in a given driving scene, are acquired as information regarding the operating state (operating state data). The operating state data is acquired as time-series data synchronized with the driving scene video.

[0038] The GPS positioning device 5 is a device that uses GPS (Global Positioning System) to determine the vehicle's current position. While the vehicle is in motion, the GPS positioning device 5 continuously measures the vehicle's position, and time-series data of the vehicle's position is acquired in synchronization with the driving scene video and driving status data.

[0039] The CAN measuring instrument 4 and GPS positioning device 5 can be installed at any location within the vehicle, but in this embodiment, the small CAN measuring instrument 4 and GPS positioning device 5 (not shown in Figures 2 and 3) are positioned below the front seats (driver's seat 14 or passenger seat 15). The antenna (not shown) of the GPS positioning device 5 extends towards the rear seats.

[0040] Time-series data acquired by smartphones 1, 2, and 2A, CAN measuring instrument 4, and GPS positioning device 5 is transmitted to tablet terminal 5 via data transmission means 22. The data transmission means 22 may be a wired communication means (cable) or a wireless communication means (Wi-Fi, Bluetooth, etc.), or a storage medium such as an SD card may be used as the data transmission means 22 for data transfer.

[0041] The tablet terminal 5 is a training screen display terminal that integrates information acquired from smartphones 1, 2, and 2A, CAN measuring instrument 4, and GPS positioning instrument 5 to display a training screen (a screen that displays a list of various information used in training). It comprises a display unit 31, a saliency calculation means 32, and an interest level calculation means 33. The display unit 31 is, for example, the LCD screen on the front of the tablet terminal. The saliency calculation means 32 and the interest level calculation means 33 are software installed on the tablet terminal 5.

[0042] The saliency calculation means 32 analyzes each scene (visual image at each time point) of the driving scene video acquired by the first smartphone 1 and calculates the distribution of saliency (saliency map) at each point in the visual image. As for saliency, for example, saliency that evokes bottom-up attention (passive attention instinctively triggered by the nature of the visual stimulus itself) in the visual image (bottom-up saliency) is calculated based on neuroscientifically appealing features (color, brightness, movement, etc.) contained in the visual image itself that attract visual attention. This generates a bottom-up saliency map showing the distribution of bottom-up saliency. As described later, the calculated saliency can be added to and displayed on the driving scene video according to the user's selection.

[0043] The interest level calculation means 33 is a means for calculating the interest level index of the driver or passenger based on the saliency map calculated by the saliency calculation means 32 and the gaze position of the driver or passenger (point of gaze in the driving scene video). The interest level index is a parameter for estimating the degree of interest of the driver or passenger, and specifically, the interest level index disclosed in the above-mentioned Patent Document 8 (Japanese Patent Application No. 2023-051384) and Patent Document 9 (Japanese Patent Application No. 2023-051385) by the applicant is also adopted in the present invention.

[0044] To explain in more detail, in calculating the interest index, the distribution of saliency values ​​in the bottom-up saliency map is first transformed into a three-dimensional probability density function. That is, by reconstructing the bottom-up saliency map into a probability density function that represents the probability of seeing each point, the saliency value S(x, y) at each point in the bottom-up saliency map is taken as the probability density (likelihood) L(x, y) in the probability density function. As a result, points with high saliency values ​​in the bottom-up saliency map are points with high likelihood in the probability density function, i.e., points with a high probability of being seen by the driver from the perspective of bottom-up attention. Here, x and y are the coordinates of each point in the visual image.

[0045] Next, the log-likelihood ln(x, y) is calculated by multiplying the natural logarithm of the likelihood L(x, y) by -1. That is, the log-likelihood ln(x, y) is calculated using the following equation (1). ln(x, y)=-log(L(x, y)) …(1) The interest index I(Sn) is calculated as the time-averaged value obtained by averaging the log-likelihood ln(X, Y) at gaze positions X and Y over the duration of the driving scene Sn. Here, X and Y are the coordinates of the gaze position, and Sn is a parameter that characterizes each driving scene.

[0046] Furthermore, the interest index can also be calculated by dividing the visual image into divided regions and basing the calculation on the representative saliency value in the region where the driver's gaze point (eye position) is located. In this case, a representative value lnR (a representative value SR of the saliency value S) of the log-likelihood ln is set for each divided region of the visual image, and the interest index I(Sn) is calculated as the time-averaged value of the representative value lnR(X,Y) of the log-likelihood in the divided region containing eye positions X and Y, averaged over the duration of the driving scene Sn. This allows for accurate estimation even when the precise eye position within the divided region cannot be detected, and enables low-cost estimation of interest levels.

[0047] The calculated interest index visualizes the driver's or passenger's interest in the external environment, and is therefore used to determine whether the driver is driving with appropriate interest and attention in the given driving scenario.

[0048] In this embodiment, an example is shown in which the calculation of saliency and interest index is performed on the tablet terminal 5. However, it is not necessary to perform all of these calculations on the tablet terminal 5. If the load of the various calculation processes performed on the tablet terminal 5 is large, it is possible to perform the calculations on a separate computer 23 (e.g., a personal computer) as needed, and receive and display the calculation results on the tablet terminal 5. For example, the calculation of saliency by the saliency calculation means 33 (adding saliency to the driving scene video) has a large computational load, so the calculation may be performed on the computer 23.

[0049] The tablet device 3 is attached to the vehicle via a fixing device 8. More specifically, the fixing device 8 comprises a cylindrical base end 8A and an arm portion 8B extending from the base end 8A. The base end 8A fits into the cup holder 17 (a hole for accommodating a beverage container) of the center console 16, thereby attaching it to the center console 16, while the tip end (not shown) of the arm portion 8B holds the tablet device 3.

[0050] The tablet device 5 is detachable from the mounting bracket 8 and can be freely removed and used as needed. For example, during practical driving lessons, when a student is driving the vehicle, the instructor in the passenger seat may use the tablet device 5, which has been removed from the mounting bracket 8. Also, during the review of driving in the classroom after the practical driving lesson, the tablet device 5 will be removed from the mounting bracket 8 and taken to the classroom for use.

[0051] Next, the details of the screen display on the tablet terminal 5 will be explained with reference to Figures 8 to 14. Figure 6 shows an example of the dashboard screen 60, which is a training screen displayed on the display unit 31 of the tablet terminal 3. The dashboard screen (training screen) integrates and displays various information used in training (videos, graphs, etc.) on a single screen. Users of the training support device (for example, students and instructors at a driving school) can receive feedback on their driving results by viewing the dashboard screen 60. .

[0052] As shown in Figure 6, the dashboard screen 60 includes a video display unit 61, a map display unit 62, a graph display unit 64, an interest level display unit 64, a data selection unit 65, and a graph selection unit 66. Each of these units is assigned to a divided area of ​​the display unit 31 of the tablet terminal 5.

[0053] The video display unit 61 is a display unit that displays driving scene video (a video that reproduces the passage of time of the forward view while driving a vehicle) acquired (shot) by the first smartphone 1. Depending on the user's selection (touch operation of the data selection unit 65), the point of focus (the position where the driver's gaze is directed) of the driver being trained (the driver who is the target of driving instruction using this driving instruction support device), the point of focus of the model driver, and saliency (severity) in the driving scene video may be superimposed on the driving scene video. The display of saliency in the driving scene video is done, for example, by displaying areas with high saliency in color with varying shades according to the magnitude of the saliency (for example, red).

[0054] Figure 7 shows an example of the display on the video display unit 61. As shown in the figure, the gaze points 71A, 71B, and 71C of the training driver are superimposed on the driving scene video on the video display unit 61. In addition, the gaze points 72A, 72B, and 72C of the model driver are also superimposed on the driving scene video on the video display unit 61.

[0055] Here, the model driver's gaze points 72A, 72B, and 72C are, for example, those acquired in advance. Specifically, the model driver (for example, a driving school instructor) drives the same route as the driver being trained (the target route) in advance, and the model driver's gaze points 72A, 72B, and 72C are generated based on the model driver's gaze information acquired by the driving training support device at that time.

[0056] As eye-tracking information for the model driver, it is also possible to use eye-tracking information from the passenger in the front seat (for example, an instructor sitting in the front seat while the trainee driver is driving) acquired by a third smartphone. This is because the instructor in the front seat is likely to be paying attention to points that should be noted when they are driving themselves while giving instructions, and is therefore likely to be focusing on those points.

[0057] Furthermore, when the model driver travels along the designated route, measurements are simultaneously taken using the CAN measuring device 4 and the GPS positioning device 5. Various detection data regarding the model driver's driving state and vehicle position data are also acquired and made available for use in overlaying onto the driving state data and interest index display described later.

[0058] The driving scene video displayed on the video display unit 61 is shown as a video that reproduces the passage of time of the forward view while driving the vehicle, and therefore changes successively as the driving progresses. Accordingly, the gaze points 71A-71C and 72A-72C are basically displayed as points that move in accordance with the changes in the driving scene video (line of sight position at each time). Figure 7 shows how the gaze point of the trainee driver moves from 71A to 72B to 73C, and the gaze point of the model driver moves from 72A to 72B to 72C. The gaze points of the trainee driver 71A-71C and the gaze points of the model driver 72A-72C are displayed in different colors, for example, so that they can be easily distinguished at a glance.

[0059] As a method for displaying the point of focus, it is also possible to display the movement trajectory of the point of focus within the forward field of view during the target driving scene (point of focus trajectory). For example, in the example in Figure 7, the point of focus trajectory of the training driver is displayed as a trajectory from point of focus 71A to point of focus 71C, and the point of focus trajectory of the model driver is displayed as a trajectory from point of focus 72A to point of focus 72C. In addition to such point of focus trajectories, it is also possible to display the point of focus moving along the point of focus trajectory at each time point.

[0060] This display of gaze trajectories allows for a quick comparison of the overall eye movements of the trainee driver and the model driver in the target driving scene, enabling accurate evaluation of any problems with the trainee driver's overall eye movements. For example, if the model driver's gaze trajectory covers a wide area in the target driving scene, while the trainee driver's gaze trajectory is concentrated in a narrow area, it can be evaluated that the model driver is scanning a wide area to cover necessary locations, while the trainee driver is not scanning the necessary locations.

[0061] The display mode on the video display unit 61 can be changed by operating the data selection unit 65 (for example, by touch operation). For example, the display of the gaze points of the training driver and the model driver can be freely selected to be shown or hidden for one or both. It is also possible to switch between displaying the gaze point as a moving point and displaying it as a gaze point trajectory. Furthermore, the driving scene video displayed on the video display unit 61 can be selected from any time (driving location) by operating the data selection unit 65 or the slider 73 in the video display unit 61.

[0062] Furthermore, the video display unit 61 allows users to write on it (for example, by using a stylus or touching it with their finger). This allows, for example, in driving lessons where students are watching a video of a driving scene, to write lines or other characters on the video of the driving scene, thereby specifically indicating areas that require attention in the video, and enabling concrete and easy-to-understand instruction that is easily accepted by the students.

[0063] The map display unit 62 is a display unit that displays the vehicle's movement trajectory and travel points (travel points displayed on the video display unit 61) on a map based on time-series data of the vehicle's position acquired by the GPS positioning device 5. The display mode of the map display unit 62 can be selected by the user through operation (touch operation) of the data selection unit. For example, the display mode can show either the travel points or the movement trajectory, or it can be displayed in a way that the display representing the travel points moves along the movement trajectory. The map displayed on the map display unit 62 is generated based on map information stored inside the tablet terminal 5 or map information acquired from an external source.

[0064] The graph display unit 63 is a display unit that graphs the time changes of various detected values ​​indicating the driving state (for example, driving state data such as vehicle speed, accelerator opening, steering angle, and gravitational acceleration) based on time-series data of information on the driving state acquired by the CAN measuring instrument 4. The driving state data to be displayed on the graph display unit 63 can be selected by operating (touching) the graph selection unit.

[0065] The graph display unit 63 can overlay the driving state data of a model driver onto the driving state data of the driver being trained. Figure 8 shows an example of a vehicle speed graph in which the time change of the driving state data of the driver being trained (solid line in the figure) and the time change of the driving state data of a model driver (dashed line in the figure) are displayed in sync and overlaid. In actual graph displays, the data of the driver being trained and the model driver are displayed in different colors to ensure high visibility. By displaying the driving state data overlaid in this way, the differences between the driving of the driver being trained and the model driver can be clearly visualized.

[0066] The interest level display unit 64 is a display unit that displays the interest level index of the driver being trained, which is calculated by the interest level calculation means 32. Similar to the video display unit 61 and the graph display unit 63, the interest level display unit 64 can also display the interest level index of the model driver overlaid on top of the display unit 64.

[0067] In this embodiment, the driver's level of interest is calculated using representative saliency values ​​in the divided regions of the driving scene video. Therefore, the video display unit 61 in Figure 7 shows the driving scene video divided into six sections. The level of interest index is calculated based on the representative saliency values ​​in these six divided regions.

[0068] Users of the driving instruction support system can check the level of interest index for each scene in the driving video and, by comparing the level of interest index of the trainee driver with that of the model driver, accurately evaluate whether the trainee driver was driving with an appropriate level of interest in that scene (for example, whether they were paying appropriate attention in accordance with the instructor's instructions).

[0069] On the dashboard screen 60, in addition to the video display unit 61, it is possible to write on each of the other display units (map display unit 62, graph display unit 63, interest level display unit 64) (or on the screen itself, including each of the display units) using, for example, a stylus or finger touch operation. This allows, for example, to specifically indicate problems with the graphs in the graph display unit 63 through writing, thereby improving the effectiveness of driving instruction.

[0070] The displays in each section of the dashboard screen 60 are synchronized with each other. For example, when the video display section 61 starts displaying video, the driving location display in the map display section 62, the graph display of driving status data in the graph display section 63, and the interest level index display in the interest level display section 64 all progress in synchronization with the video display. As a result, the user of the driving instruction support device can view and accurately recognize the points of focus, driving status, and interest level index at the driving locations shown in the video display section 61 and the map display section 62, all on a single screen. Therefore, the driving of the driver being instructed can be comprehensively analyzed from various angles, enabling effective driving instruction.

[0071] Figure 11 shows a dashboard screen 80 as another example of a training screen. This dashboard screen 80 is designed to display information particularly suited to a configuration in which a smartphone 1 is provided for forward-facing video recording, which is common to both the driver and the passenger, as shown in the embodiments in Figures 2 and 3, while smartphones 2 and 2A are provided for eye-tracking, respectively, for the driver and the passenger.

[0072] As shown in the figure, the dashboard screen 80 includes a video display unit 81, a map display unit 82, graph display units 83 and 84, an interest level display unit 85, and a display selection unit 86. The video display unit 81 is a display unit that displays driving scene video (a video that reproduces the passage of time of the forward view while driving a vehicle) acquired (shot) by the first smartphone 1. Below the video display unit 81, there is a driving state display unit 81A that shows the driving state (for example, steering operation, etc.) in the displayed driving scene.

[0073] The driving scene video displayed on the video display unit 81 superimposed the gaze trajectory 81B of the driver (for example, the driver being trained) and the gaze trajectory 81C of the passenger (for example, the instructor in the driving lesson). Here, the driver's gaze trajectory 81B is calculated from the driver's eye gaze information acquired by the smartphone 2, and the passenger's gaze trajectory 81C is calculated from the passenger's eye gaze information acquired by the smartphone 2A.

[0074] In Figure 11, the driver's gaze trajectory 81B is shown as a solid line, and the passenger's gaze trajectory 81C is shown as a dashed line. However, in the actual display on the video display unit 81, it is preferable to display the gaze trajectory 81B and gaze trajectory 81C in different colors, for example, so that they can be easily distinguished at a glance.

[0075] Furthermore, the gaze point trajectories 81B and 81C may be displayed, for example, in such a way that the gaze points of the driver and passenger move as the driving scene video progresses (changes) on the video display unit 81, and a trajectory is drawn behind these gaze points. This allows the gaze point in each driving scene and the movement of the gaze point due to changes in the driving scene to be recognized at a glance.

[0076] In this way, the video display unit 81 displays the driver's gaze trajectory 81B and the passenger's gaze trajectory 81C superimposed on a single driving scene video. Therefore, when the dashboard screen 80 is used for driving instruction, the gaze trajectory 81B of the student driver and the gaze trajectory 81C of the instructor passenger can be directly compared on a single video, and the differences in gaze points (eye positions) and eye movements between the student and the instructor can be easily confirmed visually. Thus, it becomes possible to conduct effective driving instruction while checking the video display unit 81.

[0077] The map display unit 82 is a display unit that, in synchronization with the display on the video display unit 81, displays the vehicle's travel location 82A (the travel location displayed on the video display unit 81) and the travel trajectory 82B leading up to travel location 82A on a map. The graph display units 83 and 84 are display units that graph the time changes of various detected values ​​indicating the driving state (for example, driving state data such as vehicle speed, accelerator opening, steering angle, and gravitational acceleration).

[0078] Furthermore, the interest level display unit 64 is a display unit that displays the likelihood graph of the driver (corresponding to the level of interest) calculated by the interest level calculation means 32. The display selection unit 86 is an operation unit for selecting the display method in the video display unit 81, the interest level display unit 64, etc., by touch operation.

[0079] The video display unit 81, map display unit 82, graph display units 83 and 84, and interest level display unit 85 on the dashboard screen 80 are basically the same as the video display unit 61, map display unit 62, graph display unit 63, and interest level display unit 64 on the dashboard screen 60, so further detailed explanations will be omitted.

[0080] Figure 12 shows another example of a training screen, the dashboard screen 90. This dashboard screen 90 is specifically designed to display images in a configuration where separate smartphones for the driver and the passenger are provided for recording forward-facing video.

[0081] More specifically, the dashboard screen 90 includes a driver's side video display unit 91 and a passenger's side video display unit 92. The driver's side video display unit 91 and the passenger's side video display unit 92 are arranged vertically on the left side of the screen. The driver's side video display unit 91 displays a superimposed image of the front of the vehicle (the image seen from the driver's perspective) captured by a smartphone for forward-facing video recording on the driver's side, along with the driver's gaze trajectory 91A. On the other hand, the passenger's side video display unit 91 displays a superimposed image of the front of the vehicle (the image seen from the passenger's perspective) captured by a smartphone for forward-facing video recording on the passenger's side, along with the passenger's gaze trajectory 92A.

[0082] In this way, on the dashboard screen 90, the driver's side video display unit 91 and the passenger's side video display unit 92 are displayed side by side vertically, so that the user of the driving instruction support device can easily compare the gaze trajectory 91A of the driver (e.g., the student) and the gaze trajectory 92A of the passenger (e.g., the instructor) by visual inspection. Furthermore, the driver's side video display unit 91 displays the driver's gaze trajectory 91A superimposed on the forward view from the driver's perspective, and the passenger's side video display unit 91 displays the passenger's gaze trajectory 91B superimposed on the forward view from the passenger's perspective, so that the driver's side video display unit 91 and the passenger's side video display unit 92 display appropriately reflect the difference in the driver's and passenger's field of view.

[0083] Figure 13 shows another example of a training screen, the dashboard screen 100. The dashboard screen 100 is designed to display information that is particularly suited to a configuration where only the driver's gaze is measured and the passenger's gaze is not measured, as in the embodiments shown in Figures 4 and 5.

[0084] On the dashboard screen 100, since there is no need for a separate video display for the passenger seat, a single video display unit 101 is displayed enlarged to cover almost the entire screen, and on this video display unit, the driver's gaze trajectory 101A is superimposed on the driving scene video. In addition, the map display unit 102 and the interest level display unit 105 are located on the left side of the screen, the graph display units 103 and 104 are located on the right side of the screen, and the display selection unit 106 is located at the bottom of the screen.

[0085] Figure 14 shows another example of a training screen, the dashboard screen 110. This dashboard screen 110 is suitable for displaying the eye-tracking information of the training driver and the model driver side by side for comparison.

[0086] As shown in the figure, the dashboard screen 110 includes a first video display unit 111 and a second video display unit 112 arranged vertically, with the first video display unit 111 displaying the gaze trajectory 111A of the first driver and the second video display unit 112 displaying the gaze trajectory 112A of the second driver.

[0087] For example, the video display unit 111 displays information about the driver being trained (i.e., video footage of the driving scene acquired by the first smartphone 1 when the driver being trained drove a predetermined route, and the gaze trajectory of the target driver acquired by the second smartphone 2 111A). On the other hand, the video display unit 112 displays information about the model driver (i.e., the model driver (for example, the instructor) drove the same route as the driver being trained in advance (for example, on a different day), and the gaze trajectory of the model driver 112A).

[0088] In this case, the video display unit 111 and the video display unit 112 will display images of the same driving location simultaneously, with synchronized timing. To enable such display, the model driver's driving should be appropriately adjusted so that they travel through each driving location at a similar timing to that of the driver being trained.

[0089] As shown above, the dashboard screen 110 displays the eye gaze information of the trainee driver (gaze trajectory 111A) and the eye gaze information of the model driver (gaze trajectory 112A) side by side in appropriate synchronization. This allows the trainee driver's gaze to be compared with the model driver's gaze when actually driving the same route, and enables accurate visual confirmation of the differences in the trainee driver's and model driver's gazes while driving.

[0090] As described above, with the driving instruction support device of this embodiment, information regarding the driver's gaze (point of gaze in the driving scene video) is superimposed on the driving scene video, making it possible to visualize how the driver moved their gaze during the target driving scene. Furthermore, on the dashboard screen, the vehicle position, driving state data, and interest index are displayed in a list in synchronization with the driving scene video, so that the user of the driving instruction support device can objectively recognize the details of the training driver's driving from various perspectives. Moreover, the display in the driving instruction support device can also superimpose data from the model driver (point of gaze, driving state data, interest index), so the difference between the training driver's driving and the model driver's driving can be recognized at a glance.

[0091] Therefore, when reviewing a trainee driver's driving during driving lessons, the trainee and instructor can conduct the lessons while sharing an objective understanding. For example, the instructor can accurately identify the trainee driver's habits and shortcomings based on objective evidence, enabling them to provide appropriate guidance tailored to each individual trainee. Furthermore, the trainee can become aware of the objective facts regarding their driving (for example, realizing that their previous understanding differed from the objective facts), and accept the lesson content with understanding. Thus, the gap (discrepancy in understanding) in the interaction between the trainee and instructor is reduced, making it possible to conduct effective driving lessons.

[0092] Furthermore, the driving instruction support device of this embodiment can be simply constructed from two commercially available (existing general-purpose) smartphones (the first smartphone 1 and the second smartphone 2), one commercially available tablet terminal 5, a small CAN measuring instrument 4, and a small GPS positioning device 5. Therefore, it can be easily and inexpensively installed on existing training vehicles at driving schools.

[0093] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and appropriate modifications can be made within the scope described in the claims. For example, in this embodiment, the terminal for acquiring driving scene video, the terminal for acquiring gaze information, the terminal for acquiring passenger gaze information, and the terminal for displaying the training screen were designated as the first smartphone 1, the second smartphone 2, the third smartphone 2A, and the tablet terminal 5, respectively, but the present invention is not limited to this form. As the terminal for acquiring driving scene video, any terminal (for example, a small video camera) that can acquire (shoot) video of the area in front of the vehicle can be used. Similarly, as the terminal for acquiring gaze information, any terminal that can acquire the driver's gaze information can be used. Furthermore, as the terminal for displaying the training screen, any terminal (for example, a personal computer) that can generate and display the training screen according to the present invention can be used. [Industrial applicability]

[0094] This invention can be used for driving instruction at driving schools. [Explanation of Symbols]

[0095] 1. First smartphone (device for acquiring driving scene video) 2. Second smartphone (a device for acquiring driver's eye-tracking information) 2A Third Smartphone (Terminal for acquiring eye-tracking information from the passenger in the front seat) 3 CAN measuring instruments 4 GPS positioning devices 5. Tablet device (device for displaying the training screen) 6. First smartphone holder 7. Second smartphone holder 8. Fixing device for tablet-type devices 10 Cabin 11 Dashboard 12 Front windshield 13 Rearview mirror 13A Passenger-side rearview mirror 14. Driver's seat 15 Passenger seat 16 Center Console 17 Cup Holders 18 displays 21 Detection device 22 Data transmission means 23 Computers 31 Display unit of a tablet device 32. Salience Calculation Method 33. Means for calculating the level of interest 41 Driver's Face 42 A flat surface onto which driving scene video is projected. 43. Driver's line of sight 50 Fixtures 51 First support 52 Second support 53 Base 60 Dashboard screen (Driving lesson screen) 61 Video display unit 62 Map display section 63 Graph display section 64. Interest level display section 65 Data Selection Section 66 Graph Selection Section 71A~71C Points of focus for drivers being trained 72A~72C Key points of the exemplary driver 80 Dashboard screen 81 Video display unit 81A Driver's gaze point trajectory 81B Passenger seat occupant's gaze trajectory 82 Map display section 82A Travel point 82B Route 83 Graph display section 84. Interest Level Display Section 85 Display Selection Section 90 Dashboard screen 91 Driver's side video display unit 92 Passenger-side video display unit 100 Dashboard screen 101 Video display unit 102 Map display section 103 Graph display section 104 Graph display section 105 Interest level display section 106 Display Selection Section 110 Dashboard screen 111 First video display unit 112 Second video display unit

Claims

1. A device for acquiring driving scene video, which captures forward-facing footage of a vehicle while it is in motion, A terminal for acquiring gaze information that acquires information regarding the driver's gaze while the vehicle is in motion during the aforementioned driving scene, A training screen display terminal capable of displaying a training screen that includes a superimposed display of the driver's gaze point superimposed on the driving scene video generated based on the driving scene video acquired by the driving scene video acquisition terminal and the driver's gaze information acquired by the gaze information acquisition terminal, and A driving instruction support device equipped with this system.

2. In the driver training support device according to claim 1, The aforementioned training screen display terminal is a driving instruction support device capable of displaying, on the training screen, the point of focus of the model driver who drove the driving scene superimposed on the driving scene video.

3. In the driver training support device according to claim 1, The vehicle is equipped with a terminal for acquiring passenger eye-line information, which acquires information regarding the gaze of the passenger in the passenger seat of the vehicle while the vehicle is in motion as described above. The aforementioned training screen display terminal is a driving training support device capable of displaying, on the training screen, the point of focus of the passenger seat occupant, calculated based on information regarding the passenger seat occupant's gaze, superimposed on the driving scene video.

4. In the driver training support device according to claim 1, The system includes a saliency calculation means for calculating the saliency in the aforementioned driving scene video, The aforementioned training screen display terminal is a driving training support device capable of displaying the saliency in the driving scene video calculated by the saliency calculation means by adding the information to the driving scene video.

5. In the driver training support device according to claim 1, A driving instruction support device comprising interest level calculation means for calculating an index of the driver's level of interest based on information related to the driver's gaze and the saliency in the driving scene video.

6. In the driver training support device according to claim 1, The aforementioned training screen display terminal is a driving instruction support device that allows writing to be done within the driving scene video on the training screen.

7. In the driver training support device according to claim 1, The aforementioned terminal for acquiring driving scene video and the aforementioned terminal for acquiring gaze information are smartphones, which are driving instruction support devices.

8. In the driver training support device according to claim 1, The aforementioned terminal for displaying the training screen is a tablet-type device used as a driving instruction support system.

Images