Visual line behavior evaluation device

Abstract

To provide a visual line behavior evaluation device capable of evaluating a behavior of a visual line of a motorman of a train.SOLUTION: A visual line behavior evaluation device 1 which evaluates a behavior of a visual line of a motorman 2 of a train comprises: an image data acquisition section 10 for successively acquiring image data from a scene moving image MS corresponding to a scene around the train in a view from a motorman seat of the train; a visual line position information acquisition section 12 for acquiring as visual line position information a position of the visual line of the motorman 2 detected by an eye tracker ET mounted on a head 2A of the motorman 2; a coordinate calculation section 13 for detecting an object that the motorman 2 should confirm during travel of the train, from the image data and calculating a coordinate of the object in the image data; and a visual line estimation section 14 for estimating the visual line of the motorman 2 based on the calculated coordinate of the object and the visual line position information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a gaze behavior evaluation device that evaluates the gaze behavior of a train driver. [Background technology]

[0002] Train drivers are required to have the skill to drive while checking a lot of information, such as not only the train ahead and signs, but also the instruments and timetables installed inside the train. For example, such skills are improved by detecting the eye movements of a driver who performs a simulated driving session using a driving simulator and comparing them with the eye movements of an experienced driver. For example, a technology for evaluating the driver's eye movements is described in Patent Document 1, the source of which is shown below.

[0003] Patent Document 1 describes a train driving simulator. This train driving simulator includes a storage means, a driving screen display control means, and a margin timing display control means. The storage means stores textbook data including data that defines the model gaze position of the driver for each running position when the train runs through a predetermined running section where a railway signal indicating the driving condition for the driver is installed at a predetermined position, and data that defines the margin timing that allows the driver to temporarily relax attention from the front of the train in association with the train position. The driving screen display control means sets a gaze range including the model gaze position corresponding to the current simulation running position during the execution of a driving simulation of the running section based on the user's operation input, and controls the display of the driving screen in the driving simulation in which the gaze range is distinguishably displayed. The margin timing display control means performs a predetermined display control when the current simulation running position is a running position corresponding to the margin timing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-049258 A Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, the train operation simulator described in Patent Document 1 needs to store textbook data including data that defines the model eye gaze position of the driver in the storage means. For this reason, it is necessary to store data that defines the model eye gaze position for each line section on which the train runs, which is time-consuming.

[0006] Therefore, there is a demand for an eye gaze behavior evaluation device that can easily evaluate the eye gaze behavior of train drivers. [Means for solving the problem]

[0007] The characteristic configuration of the gaze behavior evaluation device of the present invention is that it is a gaze behavior evaluation device that evaluates the gaze behavior of a train driver, and is equipped with an image data acquisition unit that sequentially acquires image data from a scene video corresponding to the scene around the train as seen from the driver's seat of the train, a gaze position information acquisition unit that acquires the position of the driver's gaze detected by an eye tracker worn on the driver's head as gaze position information, a coordinate calculation unit that detects an object that the driver should check from the image data while the train is running and calculates the coordinates of the object in the image data, and a gaze estimation unit that estimates the driver's gaze based on the calculated coordinates of the object and the gaze position information.

[0008] With such a characteristic configuration, the evaluation of the driver's gaze behavior can be completed in the gaze behavior evaluation device based on the coordinates of the object calculated by the coordinate calculation unit and the acquired gaze position information. Therefore, there is no need to transmit information indicating the coordinates of the object or gaze position information to a device other than the gaze behavior evaluation device, and the behavior can be evaluated automatically in a short time. In addition, with this gaze behavior evaluation device, for example, the frequency with which the driver looks at an object that should be checked while the train is running and the time for which the driver looks at the object can be easily calculated. Therefore, it is possible to easily evaluate the gaze behavior of the train driver.

[0009] It is also preferable to further include an analysis unit that analyzes the driver's visibility of the object and the information presenters arranged around the driver's seat based on the estimation result of the gaze estimation unit.

[0010] With this configuration, the driver's visibility state for the object and the information display can be quantitatively evaluated based on the analysis results. Such quantitative evaluation results make it possible to appropriately evaluate whether the driver's line of sight behavior is appropriate.

[0011] It is also preferable that the vehicle navigation system further comprises a display control unit that displays, on a display device, a comparison result obtained by comparing the analysis result of the visibility state with a pre-stored analysis result of an expert train driver.

[0012] With this configuration, the driver's analysis results are compared with the analysis results of an expert train driver and displayed on the display device, making it easy to see the difference between the driver's eye gaze behavior and that of an expert train driver, and to visually grasp whether the driver's eye gaze behavior is appropriate or not.

[0013] In addition, it is preferable that the scene video is a video that has been captured in advance or a video created by a computer, and the gaze position information acquisition unit acquires the gaze position information of the driver when watching the scene video and performing a simulated train operation.

[0014] With this configuration, it is not necessary to run the train each time to evaluate the driver's gaze behavior. Therefore, since it is not necessary to run the train to evaluate the gaze behavior, the cost required for evaluating the gaze behavior can be reduced. In addition, it is possible to evaluate the driver's gaze behavior by simply installing the gaze behavior evaluation device in a driver training facility or education facility.

[0015] It is also preferable that the coordinate calculation unit performs machine learning to detect from the image data traffic lights and signs installed along the tracks, people on the platform, and information displays positioned around the driver's seat.

[0016] With this configuration, traffic lights, signs, people, and information displays can be easily detected from the image data, and it is therefore possible to appropriately evaluate whether the driver is looking at the traffic lights, signs, people, and information displays.

[0017] It is also preferable that the line of sight estimation unit sets a larger determination region for the object, which is used when estimating the line of sight, for the object that is relatively small in size in the image data.

[0018] In image data, a relatively small object may be recognized by moving one's gaze around the object without looking directly at it. In such a case, if the driver does not move his / her gaze within a small judgment area corresponding to the size of the object, it will be judged as not being seen, resulting in an erroneous judgment. Therefore, by configuring as described above, it is possible to appropriately estimate whether the driver is looking at a relatively small object in image data. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a gaze behavior evaluation device. [Diagram 2] 1 is an example of image data. [Diagram 3] FIG. 1 is a diagram showing a state in which a traffic light is being viewed; [Figure 4] FIG. 13 is a diagram showing a state in which a speedometer is visually confirmed. [Diagram 5] FIG. 13 is a diagram showing the analysis results by an analysis unit. [Figure 6] 13 is a display example of a comparison result with an expert. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] The gaze behavior evaluation device according to the present invention is capable of evaluating the gaze behavior of a train driver. A train is a vehicle that runs along a railroad track, and corresponds to an electric train that runs using electricity as its power source, or a diesel railcar that runs using the output of an internal combustion engine or a steam engine as its power source. The gaze behavior of a driver refers to the state of the driver's gaze, and specifically means the driver's visual state. Therefore, the gaze behavior evaluation device evaluates the visual state of a driver who drives a vehicle running along a railroad track. The gaze behavior evaluation device 1 of this embodiment will be described below. However, the gaze behavior evaluation device 1 is not limited to the following embodiment, and various modifications are possible within the scope of the gist of the device.

[0021] Fig. 1 is a schematic diagram showing the configuration of the gaze behavior evaluation device 1. As shown in Fig. 1, the gaze behavior evaluation device 1 is configured to include an image data acquisition unit 10, a storage unit 11, a gaze position information acquisition unit 12, a coordinate calculation unit 13, a gaze estimation unit 14, an analysis unit 15, and a display control unit 16, and each functional unit is constructed with hardware or software or both with a CPU as a core member in order to perform processing related to the evaluation of the gaze behavior of a train driver 2.

[0022] The image data acquisition unit 10 sequentially acquires image data from a scene video MS corresponding to the scene around the train as seen from the driver's seat of the train. Usually, the driver's seat of the train is provided in the car at the front of the train in the traveling direction. In addition, the train driver 2 basically drives the train while looking at the front of the train from the driver's seat. Therefore, the scene around the train as seen from the driver's seat of the train corresponds to the scene around the train including the scene ahead of the train. Therefore, the scene video MS corresponding to the scene around the train as seen from the driver's seat of the train corresponds to a video that is actually captured so as to include the scene ahead of the train. Here, the scene around the train as seen from the driver's seat of the train can also be created by computer graphics, for example. Therefore, the scene video MS corresponding to the scene around the train as seen from the driver's seat of the train includes not only the above-mentioned video that is actually captured, but also a video (i.e., a video created by a computer) that is created by computer graphics, for example, of the scene around the train as seen from the driver's seat of the train. The image data acquisition unit 10 extracts still images from such a scene moving image MS at predetermined time intervals and continuously acquires image data of the extracted still images. Note that the time interval for extracting still images can be set arbitrarily, and may be set according to, for example, the capacity of the computation process or the evaluation accuracy of the gaze behavior.

[0023] In this embodiment, the scene video MS is a video captured in advance. In this case, as shown in FIG. 2, the scene video MS may be displayed on the screen SC so that the subject (driver 2) who is to be evaluated for the gaze behavior can see it. This allows the driver 2's gaze behavior to be evaluated using a video obtained in the past, so there is no need to actually run the train every time the driver 2's gaze behavior is evaluated. In this case, the image data acquisition unit 10 acquires image data from the scene video MS displayed on the screen SC. Note that, when using such a scene video MS, although the train is not actually driven, it is preferable to provide an information presenter such as an instrument I, a display device P, a clock T, and a timetable J that is provided in the driver's seat of the train in front of the seat where the subject sits. In addition, although not shown, a master controller and a brake valve handle that the subject can hold in a simulated manner may be provided. Note that the timetable J and the like may be provided on the side of the seat where the subject sits, and the installation position of the information presenter is not particularly limited.

[0024] 1, the storage unit 11 stores image data acquired by the image data acquisition unit 10. As described above, the image data is continuously acquired by the image data acquisition unit 10. Therefore, the storage unit 11 stores the image data sent from the image data acquisition unit 10 in chronological order.

[0025] The gaze position information acquisition unit 12 acquires the gaze position of the driver 2 detected by the eye tracker ET attached to the head 2A of the driver 2 as gaze position information. The eye tracker ET is a dedicated glass attached to a person's head, and is used to track the gaze movement of the person wearing it. In this embodiment, the eye tracker ET is attached to the head 2A of the train driver 2 and tracks the gaze movement of the driver 2. The eye tracker ET acquires the tracked gaze position of the driver 2 as gaze position information. For example, the gaze position information is preferably indicated by defining the imaging range of a camera provided in the eye tracker ET by, for example, X coordinates and Y coordinates, and by associating the gaze position with the X coordinates and Y coordinates. In this case, the gaze position information is indicated based on the coordinates that define the imaging range of the camera provided in the eye tracker ET. It is preferable that the gaze position information acquisition unit 12 corrects data loss due to the blinking of the driver 2 by front and rear complementation (average value).

[0026] As described above, in this embodiment, the scene video MS is a video captured in advance. In this case, the gaze position information acquisition unit 12 acquires gaze position information of the driver 2 when the driver 2 performs a simulated train operation while watching the scene video MS, which is a video captured in advance.

[0027] The coordinate calculation unit 13 detects an object that the driver 2 should check from the image data while the train is running, and calculates the coordinates of the object in the image data. The object that the driver 2 should check while the train is running corresponds to, for example, a traffic light or a sign installed at the side of the track or on the platform, an indicator on the platform indicating a target stop position, a person on the platform, etc. The coordinate calculation unit 13 detects, for example, a traffic light or a sign installed at the side of the track or on the platform, an indicator on the platform indicating a target stop position, and a person on the platform from the image data stored in the storage unit 11 by image recognition, and calculates the coordinates in the image data. In calculating such coordinates, it is preferable to define the image range in the image data by X coordinates and Y coordinates.

[0028] In addition, while the objects to be checked while the train is running are objects outside the train, the driver 2 also needs to check instruments such as a speedometer and other information display devices such as a clock and a timetable arranged around the driver's seat while the train is running. Therefore, the coordinate calculation unit 13 may not only calculate the coordinates of the objects in the image data, but may also store the coordinates of the instruments I, the display device P, the clock T, and the timetable J arranged around the driver's seat in advance in association with the coordinates in the scene video MS displayed on the screen SC. Alternatively, it is also possible to configure the coordinate calculation unit 13 to take images of the instruments I, the display device P, the clock T, and the timetable J using a separate imaging device, and calculate the coordinates using the images thus taken.

[0029] It is preferable that the coordinate calculation unit 13 performs machine learning to detect, from the image data, traffic lights and signs provided along the tracks, indicators provided on the platform indicating the target stopping position, people present on the platform, and information display devices provided around the driver's seat. In this case, it is possible to accurately detect, from the image data, traffic lights and signs provided along the tracks, indicators provided on the platform indicating the target stopping position, people present on the platform, and information display devices provided around the driver's seat. Note that the machine learning uses a known object detection method that adds annotations to image data to recognize objects.

[0030] The gaze estimation unit 14 estimates the gaze of the driver 2 based on the calculated coordinates of the object and the information presenter and the gaze position information. As described above, the coordinates of the object and the information presenter are calculated based on the coordinates of the image data. Meanwhile, the gaze position information is calculated based on the coordinates of the imaging range of the camera provided in the eye tracker ET. The gaze estimation unit 14 estimates the gaze of the driver 2 based on the coordinates of the object and the information presenter and the coordinates of the imaging range of the camera provided in the eye tracker ET. Specifically, a reference point (a reference point for calculating the behavior of the gaze) set in the imaging range of the camera provided in the eye tracker ET is superimposed on the image data, the coordinates in the eye tracker ET and the coordinates in the image data are associated with each other, and the coordinates in the image data corresponding to the gaze position of the driver 2 detected by the eye tracker ET are calculated, and the position in the image data where the gaze of the driver 2 was located is estimated.

[0031] At this time, the gaze estimation unit 14 may set a larger judgment area of ​​the object to be used when estimating the gaze for objects that are relatively small in size in the image data. For example, when visually inspecting an object, some objects can be visually recognized by simply directing the gaze to the outer edge or vicinity of the object without gazing at the center of the object. This tendency is particularly noticeable for objects that are small in size. Therefore, by configuring as described above, it becomes possible to appropriately determine whether the gaze of the driver 2 was on the object even for objects that are relatively small in size. This judgment area may be set by machine learning, with a weight multiplied by the amount of deviation between the gaze position and the object.

[0032] Fig. 3 is an example of a case where the driver 2 looks at a traffic light in the scene video MS. In this case, as shown in Fig. 3, the line of sight of the driver 2 is within the judgment area A of the traffic light calculated by the coordinate calculation unit 13, so the line of sight estimation unit 14 estimates that the line of sight of the driver 2 was on the traffic light. Although not shown, the line of sight estimation unit 14 refers to the next image data, and if the state in which the driver 2's line of sight is within the judgment area A of the traffic light continues in the subsequent image data, it is possible to calculate the time that the line of sight of the driver 2 was on the traffic light based on the time that the image data was acquired.

[0033] Similarly, Fig. 4 is an example of a case where the driver 2 looks at an instrument I in the scene video MS. In this case, as shown in Fig. 4, the line of sight of the driver 2 is within the judgment area B of the instrument I calculated by the coordinate calculation unit 13, so the line of sight estimation unit 14 estimates that the line of sight of the driver 2 was on the instrument I.

[0034] Returning to Fig. 1, the analysis unit 15 analyzes the visibility state of the driver 2 with respect to objects and information presenters arranged around the driver's seat, based on the estimation result of the line of sight estimation unit 14. The visibility state of the driver 2 is information indicating the state in which the driver 2 looked at objects and information presenters within a given period of time, in terms of the number of times and duration that the driver 2 looked at the objects and information presenters.

[0035] Fig. 5 shows the analysis result by the analysis unit 15. The example of Fig. 5 shows the result of analyzing the visibility state of a driver 2 with employee number "1234567" using a scene video MS between "Station A and Station B" on "March 10, 2022".

[0036] Specifically, the percentage of times the driver checked ahead is shown as a "forward checking percentage" in a pie chart and a time chart, and is divided into "operating devices," "forward," "timetable," "instruments," and "moving." Furthermore, the "number of times the driver looked and the duration of the look" are shown for each of the "instruments," "operating devices," and "timetable." Furthermore, the number of times the driver looked, the average duration of the look, and the total duration of the look for other objects and information displays are shown as the "number of times the driver looked at each object to be checked," the "average duration of the duration of the look for each object to be checked," and the "total duration of the duration of the look for each object to be checked." Furthermore, the behavior of the driver's gaze when passing through a railroad crossing is also shown as "gaze movement when passing through a railroad crossing."

[0037] Such analysis results are displayed on the display device 20 by the display control unit 16. This makes it possible to visually grasp the analysis results of the gaze behavior. The display device 20 may be provided in the train operation simulator, or may be provided in a terminal installed in another location.

[0038] The display control unit 16 can also be configured to display on the display device 20 the results of comparing the analysis results of the visual state with the previously stored analysis results of an expert train driver. An example of the comparison results is shown in FIG. 6. In the example of FIG. 6, the results of comparing the number of times the driver 2, who is the subject of evaluation, looked at the instruments and the maximum gaze time, and the results of comparing the number of times the driver 2, who is the subject of evaluation, looked at the timetable and the maximum gaze time are shown. By displaying such comparison results on the display device 20, it becomes possible to visually grasp whether the visual state of the driver 2, who is the subject of evaluation, is comparable to that of an expert.

[0039] When the gaze behavior of the driver 2 is evaluated without using the gaze behavior evaluation device 1 of this embodiment, the evaluation is performed visually by an operator who evaluates the gaze behavior, and the evaluation takes several days (for example, about one month). In addition, such evaluation by an operator requires costs (for example, labor costs, etc.). However, when the gaze behavior of the driver 2 is evaluated using this gaze behavior evaluation device 1, the evaluation can be performed in a short time and an increase in costs such as labor costs can be suppressed.

[0040] Other embodiments In the above embodiment, the gaze behavior evaluation device 1 has been described as including the analysis unit 15 that analyzes the visual recognition state of the driver 2 with respect to objects and information presenters arranged around the driver's seat based on the estimation result of the gaze estimation unit 14. However, the gaze behavior evaluation device 1 can also be configured without including the analysis unit 15. In this case, the gaze behavior evaluation device 1 may be configured to display the estimation result of the gaze estimation unit 14 on the display device 20, for example.

[0041] In the above embodiment, it has been described that the display control unit 16 displays the comparison result, which is obtained by comparing the analysis result of the visibility state with the pre-stored analysis result of an expert train driver, on the display device 30. However, it is also possible to configure the display control unit 16 to display only the evaluation result of the driver 2, who is the subject of evaluation, on the display device 20, without displaying the above comparison result.

[0042] In addition, a video showing the gaze behavior of the driver 2 of the person being evaluated is recorded, and the gaze behavior of the driver 2 is displayed on the display device 30 using a specified color, while simultaneously displaying the gaze behavior of the expert driver using a color different from the gaze behavior of the driver 2, making it easier for the driver 2 to understand the differences between him and the expert driver.

[0043] In the above embodiment, an example was given of the scene video MS being a video captured in advance. However, the scene video MS may also be a video captured while the train is actually being driven. In this case, the scene video MS may be captured by an imaging device installed in the driver's seat to capture the surroundings of the train as seen from the driver's seat, and the coordinate calculation unit 13 may calculate the coordinates of the object from the scene video MS. In addition, the gaze position information acquisition unit 12 may acquire gaze position information of the driver 2 who is driving the train in real time.

[0044] The scene video MS may be a video created by a computer. In this case, the gaze position information acquisition unit 12 may acquire gaze position information of the driver 2 when the driver 2 performs a simulated train operation while watching a video created by a computer (a video created by CG) as the scene video MS.

[0045] In the above embodiment, it has been described that the coordinate calculation unit 13 performs machine learning to detect traffic lights and signs installed along the tracks, people present on the platform, and information display devices installed around the driver's seat from image data. However, the coordinate calculation unit 13 can be configured to perform detection by image recognition processing as described above without performing machine learning.

[0046] In the above embodiment, the gaze estimation unit 14 has been described as setting a large object determination region to be used when estimating the gaze for an object that is relatively small in size in the image data. However, the gaze estimation unit 14 can also be configured to set a determination region according to the size included in the image data, without setting a large object determination region to be used when estimating the gaze, even if the object is relatively small in size in the image data. [Industrial Applicability]

[0047] The present invention can be used in a gaze behavior evaluation device that evaluates the gaze behavior of a train driver. [Explanation of symbols]

[0048] 1: Eye gaze behavior evaluation device 2: Driver 2A: Head 10: Image data acquisition section 12: Gaze position information acquisition unit 13: Coordinate calculation section 14: Gaze estimation part 15:Analysis Department 16: Display control unit 20:Display device ET: Eye tracker MS: Scenery video

Claims

1. A gaze behavior evaluation device for evaluating the gaze behavior of a train driver, an image data acquisition unit that sequentially acquires image data from a scene video corresponding to a scene around the train as seen from a driver's seat of the train; A gaze position information acquisition unit that acquires the gaze position of the driver detected by an eye tracker attached to the head of the driver as gaze position information; a coordinate calculation unit that detects an object that the driver should check from the image data while the train is running and calculates coordinates of the object in the image data; A gaze estimation unit that estimates the gaze of the driver based on the calculated coordinates of the object and the gaze position information; A gaze behavior evaluation device comprising:

2. The gaze behavior evaluation device according to claim 1 , further comprising an analysis unit that analyzes the driver's visual state with respect to the object and information presenters arranged around the driver's seat based on an estimation result from the gaze estimation unit.

3. The gaze behavior evaluation device according to claim 2 , further comprising a display control unit that displays, on a display device, a comparison result obtained by comparing the analysis result of the visual recognition state with a pre-stored analysis result of an expert train driver.

4. The scene video is a video captured in advance or a video created by a computer, The gaze behavior evaluation device according to claim 1 , wherein the gaze position information acquisition unit acquires the gaze position information of the driver when viewing the scene video and performing a simulated train operation.

5. The gaze behavior evaluation device of any one of claims 1 to 4, wherein the coordinate calculation unit performs machine learning to detect from the image data traffic lights and signs installed along the tracks, people on the platform, and information displays arranged around the driver's seat.

6. The gaze behavior evaluation device according to claim 1 , wherein the gaze estimation unit sets a larger judgment region for the object used when estimating the gaze for the object having a relatively small size in the image data.

Images