Method and system for determining objects that can be seen.

Abstract

This invention provides a method for determining what a driver should be looking at while driving. [Solution] In the method for determining the object to be seen, the driver's line of sight is detected (S11), the gaze area is detected based on the detected line of sight of the driver (S12), an image obtained by photographing the area around the vehicle is acquired (S13), a candidate object to be seen is detected by recognizing the object contained in the image (S14), a traffic scene is determined (S15), it is determined whether or not the candidate object to be seen corresponding to the determined traffic scene is included in the gaze area (S16), if the candidate object to be seen corresponding to the determined traffic scene is included in the gaze area, it is determined that the driver performed appropriate visual actions in the determined traffic scene (S17), and if the candidate object to be seen corresponding to the determined traffic scene is not included in the gaze area, it is determined that the driver did not perform appropriate visual actions in the determined traffic scene (S18).

Description

Technical Field

[0001] The present disclosure relates to a visual target determination method and a visual target determination system for determining a visual target of a vehicle driver.

Background Art

[0002] Patent Document 1 discloses a technique for generating a saliency map based on a recognition result of an object around a vehicle and a prediction result of vehicle behavior using vehicle information, and determining a visual target of a driver based on the saliency map and a gaze area of the vehicle driver.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a desire to determine whether a driver is looking at what he or she should look at during driving, but the technique disclosed in Patent Document 1 does not describe such determination.

[0005] *]]Therefore, the present disclosure provides a visual target determination method and the like that can determine whether a driver is looking at what he or she should look at during driving.

Means for Solving the Problems

[0006] The method for determining a target of observation according to this disclosure includes: a gaze detection step for detecting the gaze of a vehicle driver; a gaze area detection step for detecting a gaze area, which is an area that the driver is fixated on, based on the detected gaze of the driver; an image acquisition step for acquiring an image obtained by photographing the area around the vehicle; a target of observation candidate detection step for detecting a target of observation candidate, which is a candidate for an object that the driver's gaze is directed at, by recognizing an object included in the image; a scene determination step for determining a traffic scene, which is the situation when the vehicle is being driven; and a viewing behavior determination step for determining whether a target of observation candidate corresponding to the determined traffic scene is included in the gaze area, and if a target of observation candidate corresponding to the determined traffic scene is included in the gaze area, determining that the driver performed appropriate viewing behavior in the determined traffic scene, and if a target of observation candidate corresponding to the determined traffic scene is not included in the gaze area, determining that the driver did not perform appropriate viewing behavior in the determined traffic scene.

[0007] The viewing target determination system according to this disclosure includes: a gaze detection unit that detects the gaze of a vehicle driver; a gaze area detection unit that detects a gaze area which is an area that the driver is fixated on based on the detected gaze of the driver; an image acquisition unit that acquires an image obtained by photographing the area around the vehicle; a viewing target candidate detection unit that detects a candidate viewing target which is a candidate for an object that the driver's gaze is directed at by recognizing an object included in the image; a scene determination unit that determines the traffic scene of the vehicle; and a viewing behavior determination unit that determines whether a viewing target candidate corresponding to the determined traffic scene is included in the gaze area, and if a viewing target candidate corresponding to the determined traffic scene is included in the gaze area, determines that the driver performed appropriate viewing behavior in the determined traffic scene, and if a viewing target candidate corresponding to the determined traffic scene is not included in the gaze area, determines that the driver did not perform appropriate viewing behavior in the determined traffic scene.

[0008] These comprehensive or specific embodiments may be implemented as a system, method, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of a system, method, integrated circuit, computer program, and recording medium. [Effects of the Invention]

[0009] According to a method for determining what a driver should be looking at while driving, as described in one aspect of this disclosure, it is possible to determine whether or not the driver was looking at something that they should be looking at while driving. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the input source and output destination of information handled by the visual target determination system according to the embodiment. [Figure 2] This block diagram shows an example of a visual target determination system according to an embodiment. [Figure 3] This flowchart shows an example of a method for determining a visually observable object according to an embodiment. [Figure 4] This diagram illustrates a method for detecting the area of ​​focus. [Figure 5] This is a diagram illustrating the method for detecting potential targets for visual inspection. [Figure 6] This diagram illustrates the weights assigned to the gaze region and the candidate regions for viewing. [Figure 7A] This is a diagram illustrating the table for determining the object to be seen. [Figure 7B] This is a diagram illustrating the table for determining the object to be seen. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the drawings.

[0012] The embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection configurations of components, steps, and the order of steps shown in the following embodiments are examples only and are not intended to limit this disclosure.

[0013] (Embodiment) The following describes the method and system for determining objects that can be visually identified according to the embodiment.

[0014] Figure 1 shows an example of the input source and output destination of information handled by the visual target determination system 1 according to the embodiment.

[0015] The Visibility Object Determination System 1 is a system applied to a vehicle, which acquires information from an external camera 100, an internal camera 200, and a GPS sensor 300 installed on the vehicle, and outputs the result of the determination made by the Visibility Object Determination System 1 to a monitor 400 and a speaker 500. The external camera 100 is a camera that photographs the area around the vehicle, for example, a camera installed on a drive recorder. The internal camera 200 is a camera that photographs the inside of the vehicle, for example, a camera installed on a driver monitor. The GPS sensor 300 is a sensor that detects the position of the vehicle. The monitor 400 may be a display installed on the vehicle (for example, a car navigation display), or it may be a display on a smartphone or PC (Personal Computer). The PC may be installed outside the vehicle. The speaker 500 may be a speaker installed on the vehicle, or it may be a speaker installed outside the vehicle. The object detection system 1 may also include an external camera 100, an internal camera 200, a GPS sensor 300, and a monitor 400 or a speaker 500.

[0016] Figure 2 is a block diagram showing an example of a visual target determination system 1 according to an embodiment.

[0017] The visual recognition target determination system 1 is a system that determines the visual recognition target of a vehicle driver. Specifically, it is a system that determines whether the vehicle driver is looking at what he or she should see during driving.

[0018] The visual recognition target determination system 1 includes a line-of-sight detection unit 11, a fixation area detection unit 12, an image acquisition unit 13, a visual recognition target candidate detection unit 14, a scene determination unit 15, a scene creation unit 16, a table creation unit 17, and a visual recognition behavior determination unit 18. Further, the visual recognition target determination system 1 stores a visual recognition target determination table 21, a scene classification database 22, and rule data 23. The visual recognition target determination system 1 is a computer including a processor (microprocessor) and a memory, etc. The memory is such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and can store a program executed by the processor. The line-of-sight detection unit 11, the fixation area detection unit 12, the image acquisition unit 13, the visual recognition target candidate detection unit 14, the scene determination unit 15, the scene creation unit 16, the table creation unit 17, and the visual recognition behavior determination unit 18 are realized by a processor or the like that executes a program stored in the memory.

[0019] For example, the visual recognition target determination system 1 may be a computer (device) in one housing, or may be a system composed of a plurality of computers. For example, the visual recognition target determination system 1 may be a computer arranged in a vehicle, or may be realized as one function of a drive recorder or a driver monitor. Also, for example, the visual recognition target determination system 1 may be a server. Note that the components included in the visual recognition target determination system 1 and the data stored (such as the visual recognition target determination table 21, the scene classification database 22, and the rule data 23) may be arranged in one server, or may be distributed and arranged in a plurality of servers, or may be distributed and arranged between a vehicle and a server.

[0020] Here, the details of the components included in the visual recognition target determination system 1 will be described using FIG. 3.

[0021] Figure 3 is a flowchart illustrating an example of a method for determining a target of view according to an embodiment. Since the method for determining a target of view is performed by the target of view determination system 1, Figure 3 is also a flowchart showing an example of the operation of the target of view determination system 1. The gaze detection step is performed by the gaze detection unit 11, the gaze area detection step is performed by the gaze area detection unit 12, the image acquisition step is performed by the image acquisition unit 13, the target candidate detection step is performed by the target candidate detection unit 14, the scene determination step is performed by the scene determination unit 15, and the viewing behavior determination step is performed by the viewing behavior determination unit 18.

[0022] In the gaze detection step, the gaze detection unit 11 detects the driver's gaze (step S11), and in the gaze area detection step, the gaze area detection unit 12 detects the gaze area, which is the area the driver is fixated on, based on the detected driver's gaze (step S12). The method for detecting the gaze area will now be explained using Figure 4.

[0023] Figure 4 is a diagram illustrating the method for detecting the gaze area. Figure 4 shows the interior of a vehicle and the driver.

[0024] For example, the gaze detection unit 11 detects the driver's face and eyes from the image of the driver obtained by the inner camera 200, and detects the driver's gaze (specifically, the direction of the gaze) from the detected face orientation and eye movement. For example, as shown in Figure 4, the direction of the driver's gaze, and thus the point of fixation (the star-marked point in Figure 4), can be detected from the driver's face orientation and eye movement. The fixation area detection unit 12 then detects an area of ​​a predetermined size centered on the detected point of fixation of the driver as the fixation area. The predetermined size is not particularly limited, but can be set, for example, from the range visible from a typical person's point of fixation.

[0025] In the gaze area detection step, the gaze area detection unit 12 may adjust the size of the gaze area based on the driver's attributes, the vehicle's condition, or the environment surrounding the vehicle. For example, if the driver is elderly or has a low level of driving skill, their field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's field of view according to their attributes. Also, for example, if the vehicle is traveling at a high speed, the driver's effective field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's effective field of view according to the vehicle's condition. Furthermore, for example, if the vehicle is traveling in bad weather or at night, the driver's field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's field of view according to the environment surrounding the vehicle.

[0026] Returning to the explanation in Figure 3, in the image acquisition step, the image acquisition unit 13 acquires an image obtained by photographing the area around the vehicle (step S13). In the candidate target detection step, the candidate target detection unit 14 detects candidates for targets that the driver's gaze may be directed towards by recognizing objects included in the image (step S14). Here, the method for detecting candidates for targets will be explained using Figure 5.

[0027] Figure 5 is a diagram illustrating the method for detecting potential targets. Figure 5 shows objects located inside and outside a vehicle.

[0028] For example, the image acquisition unit 13 acquires an image of the area around the vehicle obtained by the external camera 100, and the visibility target candidate detection unit 14 uses object recognition AI (Artificial Intelligence) to recognize objects in the image and detects visibility target candidates that the driver can direct their gaze towards while driving. For example, as shown in Figure 5, the visibility target candidate detection unit 14 detects traffic lights and pedestrians as visibility target candidates. Although not shown, animals such as dogs and cats, automobiles and bicycles can also be visibility target candidates.

[0029] In the visibility target candidate detection step, the visibility target candidate detection unit 14 may further detect objects installed on the vehicle that have been pre-set as visibility target candidates. For example, as shown in Figure 5, objects installed on the vehicle such as the rearview mirror, left mirror, right mirror, and car navigation system can be detected as visibility target candidates by setting a relative angle with respect to the assumed position of the driver. Although not shown, the left window and right window may also be detected as visibility target candidates. Furthermore, since the objects that the driver should look at vary depending on the type of vehicle, the visibility target candidates detected will also vary depending on the type of vehicle. For example, a box-type truck may not have a rearview mirror and may have a backup monitor instead, so the backup monitor may be a visibility target candidate. Also, for example, a large truck may have more mirrors than a normal vehicle or windows installed at the footwell, so these may also be visibility target candidates.

[0030] Returning to the explanation in Figure 3, in the scene determination step, the scene determination unit 15 determines the traffic scene, which is the situation when the vehicle is being driven (step S15). For example, a traffic scene is the situation when the vehicle is being driven, such as where the vehicle is and what kind of control it is performing. For example, a traffic scene could be a situation where the vehicle is in an intersection when turning left, or a situation where the vehicle is just before an intersection when turning left. In this case, the traffic scene may be divided into cases where the vehicle stops at a red light and then turns left, and cases where the vehicle approaches the intersection on a green light and turns left immediately. Another example of a traffic scene could be a situation where the vehicle is overtaking on a highway. Furthermore, traffic scenes may be further divided based on the number of lanes, etc. There may also be traffic scenes corresponding to specific locations, such as a situation where the vehicle is in an accident-prone area, or a situation where the vehicle is at an intersection with poor visibility due to obstacles such as trees. In addition, even if the vehicle is traveling on similar roads, the situation when it is traveling on a road with heavy traffic and the situation when it is traveling on a road with light traffic may be different traffic scenes.

[0031] Many such traffic scenes are included in the scene classification database 22. Furthermore, since there may be cases where additional traffic scenes are desired, for example, the administrator of the visibility target determination system 1, the operations manager, or the vehicle user may input the traffic scenes they wish to add, and the scene creation unit 16 may create new traffic scenes and add them to the scene classification database 22.

[0032] For example, in the scene determination step, the scene determination unit 15 may determine the vehicle's traffic scene based on GPS data, images obtained by photographing the area around the vehicle, or map information. For example, if the direction of travel of the vehicle indicated by the GPS data changes by 90 degrees, it can be determined that the vehicle is turning right or left, and if the direction of travel changes by 180 degrees, it can be determined that the vehicle is backing up. These can also be determined from changes in the scenery captured in images of the area around the vehicle. Furthermore, if the map information includes information such as road signs, traffic lights, or stop signs, the map information can be used to determine what kind of road the vehicle is traveling on.

[0033] Furthermore, in the scene determination step, for example, the scene determination unit 15 may determine the vehicle's traffic scene based on vehicle information obtained from the vehicle's ECU (Electronic Control Unit) or the like. For example, it can determine whether the vehicle is turning right or left based on information about the vehicle's steering angle, and it can determine the speed at which the vehicle is traveling based on information about the vehicle's engine speed.

[0034] In the visual behavior determination step, the visual behavior determination unit 18 determines whether a candidate for a visual target corresponding to the determined traffic scene is included in the gaze area (step S16). If the candidate for a visual target corresponding to the determined traffic scene is included in the gaze area (Yes in step S16), it determines that the driver performed appropriate visual behavior in the determined traffic scene (step S17). If the candidate for a visual target corresponding to the determined traffic scene is not included in the gaze area (No in step S16), it determines that the driver did not perform appropriate visual behavior in the determined traffic scene (step S18). Visual targets that the driver should look at are predetermined for each traffic scene. For example, in a traffic scene where a vehicle is in an intersection when turning left, if a traffic light is a visual target that the driver should look at, the visual behavior determination unit 18 determines whether the traffic light is included in the gaze area if the determined traffic scene is a traffic scene where a vehicle is in an intersection when turning left. For example, if the traffic light is not included in the gaze area, it can be determined that the driver did not perform appropriate visual behavior.

[0035] For example, the visual behavior determination unit 18 outputs a determination result indicating whether the driver performed an appropriate visual behavior in the determined traffic scene (determination result output step). For example, the visual behavior determination unit 18 may display an image indicating the determination result on the monitor 400 or output an audio message indicating the determination result from the speaker 500. As described above, the monitor 400 and speaker 500 may be installed inside the vehicle or outside the vehicle. In other words, the determination result may be notified to the vehicle driver or to a fleet manager, etc.

[0036] Furthermore, the determination may be made after weighting the gaze area and the areas of potential targets for viewing. This will be explained using Figure 6.

[0037] Figure 6 is a diagram illustrating the weights assigned to the gaze region and the candidate region for viewing.

[0038] For example, in the gaze area detection step, the gaze area detection unit 12 may assign a weight to the gaze area that is closer to the center of the gaze area, with a larger weight for each area. Similarly, in the view target candidate detection step, the view target candidate detection unit 14 may assign a weight to the view target candidate area, which is the area of ​​the detected view target candidate, with a larger weight for each area that is closer to the center of the view target candidate area. For example, as shown in Figure 6, areas close to the center of the gaze area and the view target candidate area are assigned a weight of 100, while areas far from the center of the gaze area and the view target candidate area are assigned a weight of 50. Then, in the viewing behavior determination step, the viewing behavior determination unit 18 may determine whether or not a view target candidate constituting the view target candidate area is included in the gaze area, based on the sum of the values ​​of the gaze area and the view target candidate area in the area where the gaze area and the view target candidate area overlap.

[0039] Areas outside the center of the driver's gaze area, which is the driver's point of focus, are difficult for the driver to see, even if they are within the gaze area. Therefore, there is a possibility that the driver may not be correctly seeing objects in those areas. Also, if the driver looks at an area outside the center of the candidate target area, there is a possibility that the driver may not be correctly seeing the candidate target. Therefore, by assigning weights to the gaze area and the candidate target area, and calculating the sum of the values ​​in the overlapping area between the gaze area and the candidate target area, it is possible to comprehensively determine whether the driver was looking at the candidate target that should have been seen.

[0040] Furthermore, for example, in the visibility target candidate detection step, if the detected visibility target candidate is a predetermined object, the visibility target candidate area, which is the area of ​​the detected visibility target candidate, may be expanded. For example, a traffic light is a small object from the driver's perspective, but the driver can see the traffic light even without intently staring at it, as long as it is within their field of vision to some extent. This is because man-made objects related to traffic are designed to be easily visible to drivers. If such a small object is detected at its original size, it may be determined that such an object is not included in the gaze area, and the system may incorrectly determine that the driver has not seen the object even though they have. Therefore, by expanding the visibility target candidate area for such predetermined objects, it is possible to suppress the incorrect determination that the driver has not seen the object even though they have seen it.

[0041] Furthermore, for example, in the visual behavior determination step, the visual behavior determination unit 18 may determine that a candidate for a visual target corresponding to the determined traffic scene is included in the gaze area if it has been included in the gaze area for a predetermined time or longer. In other words, the visual behavior determination unit 18 may determine that a candidate for a visual target corresponding to the determined traffic scene is not included in the gaze area if it has not been included in the gaze area for a predetermined time or longer. This prevents the system from determining that the driver was looking at a candidate for a visual target that they should have been looking at, even if the driver only glanced at it for a moment.

[0042] For example, in the visual behavior determination step, the visual behavior determination unit 18 may determine whether the candidate for a visual target included in the visual target determination table 21 corresponding to the determined traffic scene is included in the gaze area, from among the multiple visual target determination tables 21 created for each traffic scene. By creating a visual target determination table 21 containing candidate visual targets that the driver should visually inspect in advance for each traffic scene, it is possible to easily determine whether the candidate for a visual target corresponding to the determined traffic scene is included in the gaze area. Here, a specific example of the visual target determination table 21 will be explained using Figures 7A and 7B.

[0043] Figures 7A and 7B are diagrams illustrating the visibility target determination table 21. Figures 7A and 7B show the visibility target determination table 21 corresponding to a traffic scene in which a vehicle is in an intersection when turning left, and the visibility target determination table 21 corresponding to a traffic scene in which a vehicle is before an intersection when turning left. For example, the visibility target determination table 21 includes the visibility targets that the driver should look for in the traffic scene and their priority. Note that Figure 7B shows an example of the visibility target determination table 21 when there is an obstacle in the intersection, and the traffic scene may differ depending on whether there is an obstacle in the intersection or not, and the corresponding visibility target determination table 21 may also differ. For example, as shown in Figures 7A and 7B, the visibility target determination table 21 corresponding to a traffic scene in which a vehicle is before an intersection when turning left and there is no obstacle in the intersection is different from the visibility target determination table 21 corresponding to a traffic scene in which a vehicle is before an intersection when turning left and there is an obstacle in the intersection.

[0044] For example, in the visual behavior determination step, the visual behavior determination unit 18 may determine whether a candidate for a visual target with a priority of a predetermined priority or higher, included in the visual target determination table 21 corresponding to the determined traffic scene, is included in the gaze area. For example, if the predetermined priority is 90, as shown in Figure 7A, in a traffic scene where a vehicle is about to make a left turn and is approaching an intersection, if the gaze area includes a traffic light and the left mirror but not the rearview mirror, the visual behavior determination unit 18 determines that the driver did not perform appropriate visual behavior in that traffic scene. In this way, a candidate for a visual target with a high priority included in the visual target determination table 21 can be designated as a candidate for a visual target that the driver should view. Furthermore, by changing the predetermined priority, it is possible to adjust the candidate for a visual target that the driver should view. For example, by decreasing the predetermined priority, the number of candidate visual targets that the driver should view can be increased, and by increasing the predetermined priority, the number of candidate visual targets that the driver should view can be decreased.

[0045] For example, in the visibility behavior determination step, the visibility behavior determination unit 18 may change the priority of a candidate for a visible target included in the visibility target determination table 21 corresponding to the determined traffic scene, depending on the distance between the vehicle and the candidate for the visible target. This makes it possible to optimize the priority of the candidates for the visible target included in the visibility target determination table 21 according to the distance between the vehicle and the candidate for the visible target. For example, if the distance between the vehicle and the candidate for the visible target is large, the need for the driver to see the candidate for the visible target decreases, so the priority of the candidate for the visible target can be lowered and it can be excluded from the candidates for the driver to see.

[0046] For example, in the visual behavior determination step, if a candidate for a visual target included in the visual target determination table 21 corresponding to the first traffic scene that has been determined is included in the gaze area, the visual behavior determination unit 18 may lower the priority of that candidate for a visual target included in the visual target determination table 21 corresponding to the second traffic scene that follows the first traffic scene that has been determined. For example, a candidate for a visual target that the driver has seen in the first traffic scene in a series of first and second traffic scenes may not need to be seen again in the second traffic scene that follows the first traffic scene. Therefore, by lowering the priority of a candidate for a visual target that the driver has seen in the first traffic scene in the second traffic scene, that candidate can be excluded from the list of candidates for visual targets that the driver should see in the second traffic scene.

[0047] For example, multiple visibility target determination tables 21 may be created for each traffic scene based on the Road Traffic Act, traffic manuals, or the visibility behavior history of any driver. For example, the table creation unit 17 may create the visibility target determination tables 21 based on the Road Traffic Act and traffic manuals included in the rule data 23. By using the Road Traffic Act or manuals, it is possible to set candidate visibility targets that drivers should check for each traffic scene. Furthermore, for traffic scenes where a vehicle is traveling on premises where the Road Traffic Act does not apply or on uncommon roads, candidate visibility targets that drivers should check can be set by referring to the visibility behavior history of excellent drivers, etc. Also, for example, the contents of the visibility target determination table 21 may be determined by the operations manager.

[0048] As explained above, for each traffic scenario, it is possible to determine whether the potential objects that the driver should have seen in that traffic scenario were included in the driver's field of gaze, or in other words, whether the driver was looking at the potential objects that the driver should have seen in that traffic scenario. Therefore, it is possible to determine whether the driver was looking at what they should have been looking at while driving.

[0049] (Other embodiments) As described above, embodiments have been explained as examples of the technology relating to this disclosure. However, the technology relating to this disclosure is not limited thereto and can be applied to embodiments that have been modified, replaced, added to, or omitted as appropriate. For example, the following modified examples are also included in one embodiment of this disclosure.

[0050] For example, in the above embodiment, an example was described in which the visibility target determination system 1 includes a scene creation unit 16, but the visibility target determination system 1 does not necessarily have to include a scene creation unit 16. In other words, the visibility target determination system 1 does not necessarily have to have a function to add new traffic scenes.

[0051] For example, in the above embodiment, an example was described in which the visual target determination system 1 includes a table creation unit 17 and rule data 23, but the visual target determination system 1 does not necessarily have to include a table creation unit 17 and rule data 23. In other words, the visual target determination system 1 does not necessarily have to have a function to add a new visual target determination table 21.

[0052] For example, this disclosure can be implemented as a program that causes a computer (processor) to execute the steps included in the method for determining a visually observable object. Furthermore, this disclosure can be implemented as a non-temporary computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

[0053] For example, if this disclosure is implemented in a program (software), each step is executed by the program using hardware resources such as the computer's CPU, memory, and input / output circuits. In other words, each step is executed by the CPU obtaining data from memory or input / output circuits, performing calculations, and outputting the calculation results to memory or input / output circuits.

[0054] In the above embodiment, each component included in the visual target determination system 1 may be implemented by dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0055] Some or all of the functions of the visual target determination system 1 according to the above embodiment are typically implemented as an integrated circuit (LSI). These may be individually integrated onto a single chip, or some or all of them may be integrated onto a single chip. Furthermore, the implementation is not limited to an LSI; it may also be implemented using a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that allows for the reconfiguration of the connections and settings of the circuit cells inside the LSI, may also be used.

[0056] Furthermore, if advances in semiconductor technology or other derived technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, those technologies may be used to integrate each component included in the visual target determination system 1 into an integrated circuit.

[0057] Furthermore, this disclosure also includes forms obtained by applying various modifications to the embodiments that a person skilled in the art could conceive, and forms realized by arbitrarily combining the components and functions of each embodiment without departing from the spirit of this disclosure.

[0058] (Note) Based on the above description of embodiments, the following technologies are disclosed.

[0059] (Technical 1) A method for determining a target of a vehicle, comprising: a gaze detection step for detecting the gaze of a vehicle driver; a gaze area detection step for detecting a gaze area, which is an area that the driver is fixated on, based on the detected gaze of the driver; an image acquisition step for acquiring an image obtained by photographing the area around the vehicle; a target of a vehicle candidate detection step for detecting a target of a vehicle, which is a candidate for an object that the driver's gaze is directed at, by recognizing an object included in the image; a scene determination step for determining a traffic scene, which is the situation when the vehicle is being driven; and a viewing behavior determination step for determining whether a target of a vehicle candidate corresponding to the determined traffic scene is included in the gaze area, and if a target of a vehicle candidate corresponding to the determined traffic scene is included in the gaze area, determining that the driver performed an appropriate viewing behavior in the determined traffic scene, and if a target of a vehicle candidate corresponding to the determined traffic scene is not included in the gaze area, determining that the driver did not perform an appropriate viewing behavior in the determined traffic scene.

[0060] According to this, for each traffic scene, it is possible to determine whether the potential objects that the driver should have seen in that traffic scene were included in the driver's field of gaze, or in other words, whether the driver was looking at the potential objects that the driver should have seen in that traffic scene. Therefore, it is possible to determine whether the driver was looking at what they should have been looking at while driving.

[0061] (Technology 2) The method for determining a target to be seen according to Technology 1, wherein the step of determining a target to be seen includes determining whether a candidate target to be seen, which is included in the target to be seen table corresponding to the determined traffic scene, is included in the gaze area, from among a plurality of target to be seen tables created for each traffic scene.

[0062] According to this method, by creating a visibility target determination table in advance for each traffic scene, which contains candidate visibility targets that the driver should visually inspect, it becomes easy to determine whether or not the candidate visibility targets corresponding to the determined traffic scene are included in the gaze area.

[0063] (Technology 3) The method for determining a target to be seen according to Technology 2, wherein the step of determining the sighting behavior determines whether or not a candidate target to be seen, which is included in the sighting target determination table corresponding to the determined traffic scene and has a priority of a predetermined priority or higher, is included in the gaze area.

[0064] According to this, the candidates for visible targets with high priority included in the visible target determination table can be designated as candidates for visible targets that the driver should look for. Furthermore, by changing the predetermined priority, it becomes possible to adjust the candidates for visible targets that the driver should look for. For example, by decreasing the predetermined priority, the number of candidates for visible targets that the driver should look for can be increased, and by increasing the predetermined priority, the number of candidates for visible targets that the driver should look for can be decreased.

[0065] (Technical 4) The method for determining a target to be seen according to Technical 3, wherein in the step of determining the visual behavior, the priority of the candidate target to be seen is changed according to the distance between the vehicle and the candidate target to be seen included in the target to be seen determination table corresponding to the determined traffic scene.

[0066] According to this, it becomes possible to optimize the priority of potential targets included in the target determination table according to the distance between the vehicle and the potential target. For example, if the vehicle is far from a potential target, the need for the driver to see that target decreases, so the priority of that target can be lowered and it can be excluded from the list of potential targets that the driver should see.

[0067] (Technical 5) The method for determining a target to be seen according to Technical 3 or 4, wherein in the visual behavior determination step, if a candidate target to be seen included in the visual target determination table corresponding to the first traffic scene that has been determined is included in the gaze area, the priority of the candidate target to be seen included in the visual target determination table corresponding to the second traffic scene following the first traffic scene that has been determined is lowered.

[0068] For example, in a series of consecutive traffic scenes, a driver may not need to re-examine a candidate object in the first traffic scene that was observed in the first traffic scene. Therefore, by lowering the priority of a candidate object observed in the first traffic scene in the second traffic scene, that candidate object can be excluded from the list of candidates that the driver should observe in the second traffic scene.

[0069] (Technology 6) The plurality of sight target determination tables are created for each traffic scene based on the Road Traffic Act, traffic manuals, or the sight behavior history of any driver, using the sight target determination method described in any of Technologies 2 to 5.

[0070] In this way, by using the Road Traffic Act or a driver's manual, it is possible to set potential targets that drivers should visually check for each traffic scenario. Furthermore, for traffic scenarios where vehicles are traveling on private property or on unconventional roads to which the Road Traffic Act does not apply, potential targets that drivers should visually check can be set by referring to the visual behavior history of exemplary drivers.

[0071] (Technical 7) The method for determining a target of visibility according to any one of Technical 1 to 6, wherein the step of detecting a candidate target of visibility further involves detecting an object installed on the vehicle that has been set in advance as a candidate target of visibility.

[0072] According to this method, by pre-setting objects such as mirrors and windows installed on a vehicle, they can be detected as potential targets for viewing.

[0073] (Technical 8) A method for determining a target of a vehicle to be seen according to any one of Technical 1 to 7, wherein the scene determination step determines the traffic scene of the vehicle based on GPS data, the image, or map information.

[0074] In this way, the vehicle's traffic situation can be determined by using GPS data, images of the vehicle's surroundings, or map information. For example, if the vehicle's direction of travel, as indicated by GPS data, changes by 90 degrees, it can be determined that the vehicle is turning right or left, and if the direction of travel changes by 180 degrees, it can be determined that the vehicle is backing up. These can also be determined from changes in the scenery captured in images of the vehicle's surroundings. Furthermore, if the map information includes information such as road signs, traffic lights, or stop signs, the map information can be used to determine what kind of road the vehicle is traveling on.

[0075] (Technical 9) A method for determining a target to be viewed according to any one of Technical 1 to 8, wherein in the gaze area detection step, the size of the gaze area is adjusted based on the attributes of the driver, the state of the vehicle, or the environment surrounding the vehicle.

[0076] According to this, the size of the gaze area can be adjusted according to the driver's attributes, the vehicle's condition, or the environment surrounding the vehicle. For example, if the driver is elderly or has a low level of driving skill, their field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's field of view according to their attributes. Also, for example, if the vehicle is traveling at high speed, the driver's effective field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's effective field of view according to the vehicle's condition. Furthermore, for example, if the vehicle is traveling in bad weather or at night, the driver's field of view may be narrowed, so the size of the gaze area can be reduced to match the driver's field of view according to the environment surrounding the vehicle.

[0077] (Technical 10) A method for determining a target to be seen according to any one of Technical 1 to 9, wherein in the gaze area detection step, the gaze area is weighted with a larger value the closer it is to the center of the gaze area; in the view target candidate detection step, the view target candidate area, which is the area of ​​the detected view target candidate, is weighted with a larger value the closer it is to the center of the view target candidate area; and in the view behavior determination step, it is determined whether or not a view target candidate constituting the view target candidate area is included in the gaze area based on the sum of the values ​​of the gaze area and the view target candidate area in the area where the gaze area and the view target candidate area overlap.

[0078] Areas outside the center of the driver's gaze area, which is the driver's point of focus, are difficult for the driver to see, even if they are within the gaze area. Therefore, there is a possibility that the driver may not be correctly seeing objects in those areas. Also, if the driver looks at an area outside the center of the candidate target area, there is a possibility that the driver may not be correctly seeing the candidate target. Therefore, by assigning weights to the gaze area and the candidate target area, and calculating the sum of the values ​​in the overlapping area between the gaze area and the candidate target area, it is possible to comprehensively determine whether the driver was looking at the candidate target that should have been seen.

[0079] (Technical 11) A method for determining a visible target according to any one of Technical 1 to 10, wherein in the step of detecting a candidate visible target, if the detected candidate visible target is a predetermined object, the visible target candidate area, which is the area of ​​the detected candidate visible target, is expanded.

[0080] For example, traffic lights are small objects from a driver's perspective, but drivers can see them even without intently staring at the lights themselves, as long as they are within their field of vision to some extent. This is because man-made structures related to traffic are designed to be easily visible to drivers. If such small objects are detected at their actual size, the system may determine that they are not included in the gaze area, potentially leading to a false determination that the driver has not seen them even though they have. Therefore, by expanding the candidate area for visible objects for such predetermined objects, it is possible to suppress false determinations that the driver has not seen them even though they have.

[0081] (Technical 12) A method for determining a target to be seen according to any one of Technical 1 to 11, wherein in the visual behavior determination step, if a candidate target to be seen corresponding to the determined traffic scene is included in the gaze area for a predetermined time or longer, it is determined that the candidate target to be seen is included in the gaze area.

[0082] According to this, it is possible to prevent the system from determining that the driver was looking at a potential target that they should have been looking at, simply because the driver glanced at it for a moment.

[0083] (Technical 13) A viewing target determination system comprising: a gaze detection unit for detecting the gaze of a vehicle driver; a gaze area detection unit for detecting a gaze area, which is the area the driver is fixated on, based on the detected gaze of the driver; an image acquisition unit for acquiring an image obtained by photographing the area around the vehicle; a viewing target candidate detection unit for detecting a viewing target candidate, which is a candidate for the target the driver's gaze is directed at, by recognizing an object included in the image; a scene determination unit for determining the traffic scene of the vehicle; and a viewing behavior determination unit for determining whether a viewing target candidate corresponding to the determined traffic scene is included in the gaze area, and if a viewing target candidate corresponding to the determined traffic scene is included in the gaze area, it determines that the driver performed an appropriate viewing action in the determined traffic scene, and if a viewing target candidate corresponding to the determined traffic scene is not included in the gaze area, it determines that the driver did not perform an appropriate viewing action in the determined traffic scene.

[0084] This allows for the provision of a visual target determination system that can determine whether or not a driver was looking at what they should have been looking at while driving. [Industrial applicability]

[0085] This disclosure can be applied to systems that determine whether or not a driver was looking at what they should have been looking at while driving. [Explanation of Symbols]

[0086] 1. Visual Target Determination System 11 Eye-line detection unit 12. Gaze Area Detection Unit 13 Image acquisition unit 14. Detection unit for targeting visually observable objects 15 Scene determination unit 16 Scene Creation Department 17 Table Creation Section 18. Visual Recognition and Behavior Determination Unit 21. Table for determining the object of visual inspection 22 Scene Classification Database 23 Rule Data 100 External Cameras 200 In-Person Cameras 300 GPS sensors 400 monitors 500 speakers

Claims

1. A gaze detection step that detects the driver's gaze, A gaze area detection step, which detects a gaze area that is the area the driver is fixated on, based on the detected gaze of the driver, An image acquisition step involves obtaining an image obtained by photographing the area around the aforementioned vehicle, A candidate for a target to be seen by the driver by recognizing an object contained in the aforementioned image, A scene determination step to determine the traffic scene, which is the situation when the vehicle is being driven, The visual behavior determination step includes determining whether a candidate for a visual target corresponding to the determined traffic scene is included in the gaze area, determining that the driver performed appropriate visual behavior in the determined traffic scene if the candidate for a visual target corresponding to the determined traffic scene is included in the gaze area, and determining that the driver did not perform appropriate visual behavior in the determined traffic scene if the candidate for a visual target corresponding to the determined traffic scene is not included in the gaze area. Method for determining what is visible.

2. In the aforementioned viewing behavior determination step, it is determined whether or not a candidate for a viewing target included in the viewing target determination table corresponding to the determined traffic scene is included in the gaze area, from among the multiple viewing target determination tables created for each traffic scene. The method for determining a visually observable object according to claim 1.

3. In the aforementioned viewing behavior determination step, it is determined whether or not a candidate for a target to be viewed, which is included in the viewing target determination table corresponding to the determined traffic scene and has a priority of a predetermined priority or higher, is included in the gaze area. The method for determining a visually observable object according to claim 2.

4. In the aforementioned visibility behavior determination step, the priority of the visibility target candidate is changed according to the distance between the vehicle and the visibility target candidate included in the visibility target determination table corresponding to the determined traffic scene. The method for determining a visually observable object according to claim 3.

5. In the aforementioned viewing behavior determination step, if a candidate for a viewing target included in the viewing target determination table corresponding to the determined first traffic scene is included in the gaze area, the priority of that candidate for a viewing target included in the viewing target determination table corresponding to the second traffic scene following the determined first traffic scene is lowered. The method for determining a visually observable object according to claim 3.

6. The aforementioned multiple visibility target determination tables are created for each traffic scenario based on the Road Traffic Act, traffic manuals, or the visibility behavior history of an arbitrary driver. The method for determining a visually observable object according to claim 2.

7. In the aforementioned target detection step, objects installed on the vehicle that have been pre-configured are further detected as target candidates. A method for determining a visually observable object according to any one of claims 1 to 6.

8. In the scene determination step, the traffic scene of the vehicle is determined based on GPS data, the image, or map information. A method for determining a visually observable object according to any one of claims 1 to 6.

9. In the gaze area detection step, the size of the gaze area is adjusted based on the driver's attributes, the vehicle's condition, or the environment surrounding the vehicle. A method for determining a visually observable object according to any one of claims 1 to 6.

10. In the gaze region detection step, the gaze region is weighted with a larger value the closer it is to the center of the gaze region. In the aforementioned target candidate detection step, the target candidate region, which is the area of ​​the detected target candidate, is weighted with a larger value the closer it is to the center of the target candidate region. In the aforementioned viewing behavior determination step, it is determined whether or not a candidate for a target to be viewed that constitutes the candidate for a target to be viewed is included in the gaze area, based on the sum of the gaze area value and the candidate for a target to be viewed area value in the area where the gaze area and the candidate for a target to be viewed overlap. A method for determining a visually observable object according to any one of claims 1 to 6.

11. In the aforementioned target detection step, if the detected target is a predetermined object, the target area, which is the region of the detected target, is expanded. A method for determining a visually observable object according to any one of claims 1 to 6.

12. In the aforementioned visual behavior determination step, if a candidate for a visual target corresponding to the determined traffic scene is included in the gaze area for a predetermined period of time or longer, it is determined that the candidate for the visual target is included in the gaze area. A method for determining a visually observable object according to any one of claims 1 to 6.

13. A gaze detection unit that detects the driver's line of sight, A gaze area detection unit detects a gaze area, which is the area the driver is fixated on, based on the detected gaze of the driver, An image acquisition unit that acquires images obtained by photographing the area around the aforementioned vehicle, A visual target candidate detection unit detects a candidate for a visual target that is a candidate for an object to which the driver's gaze is directed by recognizing an object contained in the aforementioned image, A scene determination unit that determines the traffic scene of the aforementioned vehicle, The system includes a visual behavior determination unit that determines whether a candidate for a visual target corresponding to a determined traffic scene is included in the gaze area, and if a candidate for a visual target corresponding to a determined traffic scene is included in the gaze area, it determines that the driver performed appropriate visual behavior in the determined traffic scene, and if a candidate for a visual target corresponding to a determined traffic scene is not included in the gaze area, it determines that the driver did not perform appropriate visual behavior in the determined traffic scene. A system for determining objects that can be visually identified.

Images