Visual identification and determination device and control method for visual identification and determination device

By adjusting the cone area of ​​the line of sight according to the driver's face orientation and vehicle state in the visual recognition determination device, the problem of misjudgment caused by the reduction of the line of sight vector estimation accuracy is solved, and the driver's visual recognition accuracy of the object is improved.

CN118942074APending Publication Date: 2024-11-12TOYOTA JIDOSHA KK +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410551259.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-05-07
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the prior art, the driver's face orientation affects the line of sight vector estimation decreases, resulting in an increased risk of misjudging that the driver does not visually identify the object.

Method used

The estimation section of the estimation line cone area estimates the estimation line cone area according to the driver's face orientation, and combines the vehicle's speed, turning state and moving speed of the object, adjusts the size and shape of the estimation line cone area to improve the driver's estimation of the object The accuracy of visual recognition judgment.

Benefits of technology

It improves the accuracy of the driver's visual recognition and judgment of the object, reduces misjudgment, and enhances the driver's monitoring ability of the surrounding environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942074A_ABST
    Figure CN118942074A_ABST
Patent Text Reader

Abstract

The invention relates to a visual identification and determination apparatus and a control method of the visual identification and determination apparatus. A visual recognition determination device determines whether or not a driver is visually recognizing an object detected by an external sensor of a vehicle, estimates a sight line vector of the driver by a driver monitoring camera of the vehicle, and determines whether or not the driver visually recognizes the object detected by the external sensor of the vehicle on the basis of the sight line vector of the driver. A line-of-sight conical region that expands in a conical shape further away from the viewpoint of the driver is estimated, and whether or not the driver is visually recognizing the object is determined on the basis of the overlap between the line-of-sight conical region and the object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a visual recognition determination device and a control method of the visual recognition determination device. Background Art

[0002] In the past, as a technical document related to a visual recognition determination device, Japanese Patent Publication No. 2014-120113 is known. The following device is shown in the publication: the driver's sight line position, sight circle, and visual field area are detected from an image captured by a driver monitoring camera, and a feature object located in front of the vehicle in the direction of travel is detected from an image captured by a front camera, and the driver's visual recognition method for the feature object is determined based on each detection result.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2014-120113

[0006] Problems to be solved by the invention

[0007] However, the estimation accuracy of the sight line vector based on the image captured by the driver monitoring camera may decrease depending on the driver's face orientation. If the estimation accuracy of the sight line vector decreases, it may be erroneously determined that the driver does not visually recognize the object that the driver is visually recognizing. Summary of the invention

[0008] One embodiment of the present invention is a visual recognition determination device for determining whether a driver is visually identifying an object detected by an external sensor of a vehicle, wherein the visual recognition determination device comprises: a sight cone area estimating unit, which estimates a sight line vector of a driver through a driver monitoring camera of the vehicle and, based on the driver's sight line vector, estimates a sight cone area that expands in a conical shape as it moves away from the driver's viewpoint; and a visual recognition determination unit, which determines whether the driver is visually identifying an object based on an overlap between the sight cone area and the object, and the sight cone area estimating unit estimates the sight cone area based on the driver's facial orientation.

[0009] According to a visual recognition judgment device of one embodiment of the present invention, since the estimation accuracy of the line of sight vector based on the image captured by the driver monitoring camera will be reduced according to the driver's facial orientation, the accuracy of the driver's visual recognition judgment of the object can be improved by estimating the line of sight cone area according to the facial orientation.

[0010] In a visual recognition determination device of one embodiment of the present invention, it may be that, when the driver's face is directed away from the front direction of the vehicle, the sight cone area estimation unit increases the magnification rate of the sight cone area based on the distance from the viewpoint, compared to a case where the driver's face is directed toward the front direction.

[0011] In a visual recognition judgment device of one embodiment of the present invention, it may be that when the driver's face is facing toward the front direction of the vehicle, the sight cone area estimating unit estimates that the cross-section of the sight cone area orthogonal to the sight vector is a longitudinally long elliptical sight cone area, and when the driver's face is deviated laterally from the front direction of the vehicle, the sight cone area estimating unit estimates that the cross-section of the sight cone area is a transversely long elliptical sight cone area.

[0012] In a visual recognition judgment device of one embodiment of the present invention, it may be that when the vehicle speed is greater than a vehicle speed judgment threshold, the sight cone area estimation unit reduces the magnification rate of the sight cone area based on the distance from the viewpoint compared to a case where the vehicle speed is less than the vehicle speed judgment threshold.

[0013] In a visual recognition determination device of one embodiment of the present invention, when the vehicle is turning, the sight cone area estimation unit may enlarge the cross section of the sight cone area orthogonal to the sight vector in the turning direction of the vehicle compared to a case where the vehicle is not turning.

[0014] According to the visual recognition judgment device, when the vehicle is turning, it can be considered that the driver is consciously observing the turning destination. Therefore, by expanding the cross-section of the line of sight cone area orthogonal to the line of sight vector in the turning direction of the vehicle, the accuracy of the driver's visual recognition judgment of the object can be improved.

[0015] In a visual recognition judgment device of one embodiment of the present invention, it may be that, when the moving speed of an object observed by the driver is greater than a moving speed threshold, the line of sight cone area estimation unit estimates the line of sight cone area so that a cross-section orthogonal to the line of sight vector expands in the moving direction of the object observed by the driver.

[0016] In a visual recognition determination device of one embodiment of the present invention, the line of sight cone area estimation unit may increase the magnification rate of the line of sight cone area during the period after it is determined that the driver is visually recognizing an object until it is determined that the driver is not visually recognizing the object, compared to before the driver visually recognizes the object.

[0017] Another embodiment of the present invention may also be a control method for a visual recognition determination device, which determines whether a driver is visually identifying an object detected by an external sensor of a vehicle, wherein the control method for the visual recognition determination device includes: estimating a line of sight vector of the driver through a driver monitoring camera of the vehicle, and based on the driver's line of sight vector, estimating a line of sight cone area that expands in a conical shape as it moves away from the driver's viewpoint; estimating the line of sight cone area based on the driver's facial orientation; and determining whether the driver is visually identifying the object based on the overlap between the line of sight cone area and the object.

[0018] According to the control method of the visual recognition judgment device of other aspects of the present invention, since the estimation accuracy of the line of sight vector based on the image captured by the driver monitoring camera will be reduced according to the driver's facial orientation, the accuracy of the driver's visual recognition judgment of the object can be improved by estimating the size of the line of sight cone area according to the facial orientation.

[0019] Effects of the Invention

[0020] According to each aspect of the present invention, it is possible to improve the accuracy of the visual recognition judgment of the driver on the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a block diagram showing a visual recognition determination device according to one embodiment.

[0022] Figure 2 It is a plan view for explaining the sight line vector of the driver.

[0023] Figure 3 (a) is a diagram for explaining a situation where the driver is visually recognizing an object, and (b) is a diagram for explaining a situation where the driver is not visually recognizing an object.

[0024] Figure 4 (a) is a diagram showing an example of a cross section of a viewing cone region, and (b) is a diagram showing an example of a cross section of a viewing cone region when the magnification is increased.

[0025] Fig. 5 (a) is a diagram showing an example of a longitudinal cross section of a sight cone region, and (b) is a diagram showing an example of a transverse cross section of a sight cone region.

[0026] Figure 6 This is a flowchart showing an example of visual recognition determination processing.

[0027] Figure 7 (a) is a flowchart showing an example of the process of estimating the sight cone area according to the vehicle speed. (b) is a flowchart showing an example of the process of estimating the sight cone area according to the vehicle turning.

[0028] Figure 8 (a) is a flowchart showing an example of the process of estimating the sight cone area according to the moving speed of the object. (b) is a flowchart showing an example of the process of estimating the sight cone area according to the visual recognition status of the driver.

[0029] Description of Reference Numerals

[0030] 1 driver monitoring camera, 2 external sensor, 10 visual recognition and determination ECU, 11 object recognition unit, 12 sight cone area estimation unit, 13 visual recognition and determination unit, 50 pedestrian (object), 100 visual recognition and determination device, EB sight vector, EC sight cone area, EP viewpoint, M vehicle. DETAILED DESCRIPTION

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] Figure 1 1 is a block diagram showing a visual recognition determination device 100 according to an embodiment. Figure 1 The visual recognition determination device 100 shown is a device mounted on a vehicle such as a passenger car or a truck, and determines whether the driver of the vehicle is visually recognizing an object outside the vehicle. Even if the vehicle is equipped with a high-level driving assistance function, the driver is obliged to monitor the surroundings. In addition, it is required to determine whether the driver is visually recognizing the object appropriately.

[0033] [Structure of visual recognition determination device]

[0034] The structure of the visual recognition determination device 100 of this embodiment is described below. Figure 1 As shown, the visual recognition determination device 100 includes a visual recognition determination ECU [Electronic Control Unit] 10 that manages the device in a unified manner. The visual recognition determination ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit such as a ROM [Read Only Memory] or a RAM [Random Access Memory]. In the visual recognition determination ECU 10, various functions are realized by, for example, the CPU executing a program stored in the storage unit. The visual recognition determination ECU 10 may also be composed of a plurality of electronic units.

[0035] The visual recognition determination ECU 10 is connected to a driver monitoring camera 1 , an external sensor 2 , an internal sensor 3 , and an HMI (Human Machine Interface) 4 .

[0036] The driver monitoring camera 1 is a camera for photographing the driver of the vehicle. The driver monitoring camera 1 can be a digital camera having an imaging element such as a CCD [Charge Coupled Device] or a CIS [CMOS Image Sensor]. The driver monitoring camera 1 is disposed on the cover of the steering column in front of the driver's seat. The driver monitoring camera 1 photographs the driver's head at a prescribed frame rate. The driver monitoring camera 1 can be embedded in the dashboard in front of the driver's seat, or can be disposed on the upper part of the dashboard. The driver monitoring camera 1 can also be composed of a plurality of cameras. The driver monitoring camera 1 sends the image captured by the driver to the visual recognition determination ECU 10.

[0037] The external sensor 2 is a detection device for detecting the surrounding conditions of the vehicle. The external sensor 2 includes at least one of a camera and a radar sensor. The camera is a camera device for photographing the external conditions of the vehicle. The camera is, for example, arranged on the back side of the front windshield of the vehicle to photograph the front of the vehicle. The camera may also include multiple cameras for photographing in multiple directions around the vehicle. The camera sends the photographic information related to the external conditions of the vehicle to the visual recognition determination ECU 10.

[0038] A radar sensor is a detection device that uses radio waves (e.g., millimeter waves) or light to detect objects around the vehicle. Radar sensors include, for example, millimeter wave radars or laser radars [LIDAR: Light Detection and Ranging]. Radar sensors may also include multiple detection devices for detecting objects in multiple directions around the vehicle. The radar sensor sends information about the detected object to the visual recognition determination ECU 10.

[0039] The internal sensor 3 is a detection device for detecting the driving state of the vehicle. The internal sensor 3 includes a vehicle speed sensor and a yaw rate sensor. The vehicle speed sensor is a detector for detecting the speed of the vehicle. The vehicle speed sensor sends the detected vehicle speed information (wheel speed information) to the visual recognition determination ECU 10.

[0040] The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) of the center of gravity of the vehicle about a vertical axis. As the yaw rate sensor, for example, a gyro sensor can be used. The yaw rate sensor sends the detected yaw rate information of the vehicle to the visual recognition determination ECU 10 .

[0041] HMI4 is an interface for inputting and outputting information between the visual recognition determination ECU10 and the driver. HMI4, for example, includes a display and a speaker arranged in the vehicle cabin. HMI4 outputs images from the display and sounds from the speaker according to a control signal from the visual recognition determination ECU10. The display may be a MID [Multi Information Display], a display of a navigation system, or a HUD [Head Up Display]. HMI4 may also include various indicators.

[0042] Next, the functional structure of the visual recognition determination ECU 10 is described. The visual recognition determination ECU 10 includes an object recognition unit 11, a sight cone area estimation unit 12, and a visual recognition determination unit 13. It should be noted that part of the functions of the visual recognition determination ECU 10 described below may also be executed in a server that can communicate with the vehicle.

[0043] The object recognition unit 11 recognizes an object outside the vehicle based on the detection result of the external sensor 2. The object refers to an object that is visually recognized by the driver of the vehicle. The object includes, for example, other vehicles, bicycles, personal mobility tools, pedestrians and other mobile objects. The object may also include stationary objects such as traffic lights, road signs, road guide boards, road surface displays, etc. The object may also be limited to an object within a distance that the driver can visually recognize. The object may also be limited to an object located in front of the vehicle.

[0044] The object recognition unit 11 recognizes the relative position of the object with respect to the vehicle and the relative speed of the object with respect to the vehicle. The object recognition unit 11 may also recognize the object by performing sensor fusion by integrating the image captured by the camera in the external sensor 2 and the detection result of the radar sensor. For the recognition of the object, a known technology can be used, and therefore, a detailed description is omitted.

[0045] The object recognition unit 11 may recognize the position of the object as viewed from the driver based on the detection result of the external sensor 2. The position of the object as viewed from the driver refers to the position of the object as viewed from the driver's viewpoint.

[0046] The driver's viewpoint refers to the position where the eyes of the driver sitting in the driver's seat are assumed to be located in the vehicle. The driver's viewpoint can also be expressed as coordinates in a three-dimensional space. The driver's viewpoint is set, for example, based on the vehicle body. The driver's viewpoint can be a fixed position or an adjustable position.

[0047] The driver's viewpoint can also be adjusted based on the position of the driver's eyes recognized from the driver's photographic image captured by the driver monitoring camera 1. The driver's viewpoint is sometimes adjusted based on the position of the driver's seat, such as the front-back position of the seat cushion. The driver's viewpoint can also be adjusted based on the inclination of the seat back of the driver's seat. The driver's viewpoint can also be adjusted based on personal information such as the driver's height and seat height shared by the visual recognition determination device 100. The driver's viewpoint can be adjusted using other well-known techniques.

[0048] The object recognition unit 11 can obtain the position of the object as viewed by the driver based on the object recognition result obtained by sensor fusion. The object recognition unit 11 can also obtain the position of the object as viewed by the driver by performing viewpoint conversion processing so that the image captured by the camera of the external sensor 2 becomes an image observed from the driver's viewpoint. The object recognition unit 11 calculates the moving speed of the object as viewed by the driver based on the time change of the position of the object as viewed by the driver. In addition, the object recognition unit 11 recognizes the size of the object as viewed by the driver.

[0049] The object recognition unit 11 may also recognize the type of the object. The types of objects include four-wheeled vehicles, two-wheeled vehicles, pedestrians, traffic lights, road signs, etc. The object recognition unit 11 recognizes the type of the object, for example, by pattern matching. The object recognition unit 11 may also recognize the type of the object from the image captured by the camera of the external sensor 2 by matching an image pattern prepared in advance for each type.

[0050] It should be noted that a machine learning model may also be used to perform various recognitions of objects. The machine learning model is, for example, a neural network such as a convolutional neural network (CNN). The neural network may have a structure including a plurality of convolutional layers and a pooling layer. As the neural network, a deep learning network based on deep learning may also be used.

[0051] The sight cone area estimation unit 12 estimates the driver's sight vector based on the driver's image captured by the vehicle's driver monitoring camera 1, and estimates a sight cone area based on the driver's sight vector. In addition, the sight cone area estimation unit 12 also recognizes the driver's facial orientation.

[0052] The driver's sight line vector is a vector in three-dimensional space corresponding to the driver's sight line extending in a straight line starting from the driver's viewpoint. Methods for estimating the sight line vector include methods using facial orientation and the center position of the pupil measured by corneal reflection method. Machine learning models can also be used in the estimation of the sight line vector. Since the estimation of the sight line vector is a well-known technology, a detailed description is omitted. It should be noted that the sight line vector does not need to include a length and can be used in the sense of indicating the direction of the sight line.

[0053] Here, Figure 2 It is a plan view for explaining the sight line vector of the driver. Figure 2 , which represents a vehicle M, a road R on which the vehicle M is traveling, a pedestrian 50 (object), and a sight line vector EB. The pedestrian 50 is a pedestrian who is about to cross the crosswalk in front of the vehicle M. The direction of travel of the pedestrian 50 is indicated by an arrow P. Figure 2 As shown, the sight line vector EB corresponds to the sight line of the driver of the vehicle M toward the pedestrian 50 as the object.

[0054] Figure 3 (a) is a diagram showing an example of a viewing cone area. Figure 3 (a) shows the sight cone area EC extending from the sight vector EB with the driver's viewpoint EP as the starting point. Figure 3 (a) shows a bounding box 60 used when determining the visual recognition of the pedestrian 50 as the object by the driver. The bounding box 60 is set as a three-dimensional space, for example, as a rectangular box that surrounds the pedestrian 50 with the smallest volume. It should be noted that the setting of the bounding box 60 is not essential. It should be noted that Figure 3 (b) is a diagram for explaining a situation where the driver fails to visually recognize an object. Figure 3 The situation (b) will be described later.

[0055] like Figure 3 As shown in (a), the sight cone area estimation unit 12 estimates the sight cone area EC based on the sight vector EB extending from the driver's viewpoint EP. The sight cone area EC is an area that makes the driver's sight vector EB have an extended cone shape. It should be noted that the sight cone area EC does not need to be a strict cone shape. The cross-sectional shape of the sight cone area EC is not limited to a circle or an ellipse, as long as it is a shape that is generally close to a circle or an ellipse.

[0056] The sight cone area estimation unit 12 may estimate sight cone areas of different sizes for each object. When there are multiple objects at the same time, the sight cone area estimation unit 12 can estimate multiple sight cone areas for visual recognition determination of each object based on the driver's sight vector.

[0057] The sight cone area EC can also be estimated by considering the effective viewing angle of the driver. The effective field of view is an area located in the center of the human field of view and capable of recognizing objects with high resolution. There are individual differences in the effective viewing angle (the viewing angle of the effective field of view), and can be set to 20°, for example. There is a large range of peripheral field of view outside the effective field of view. The peripheral field of view is an area capable of recognizing objects with low resolution. The sight cone area estimation unit 12 can also use the effective viewing angle as the vertex angle of the sight cone area EC. It should be noted that the sight cone area estimation unit 12 can also estimate a value unrelated to the effective viewing angle as the vertex angle of the sight cone area EC.

[0058] The sight cone area estimation unit 12 can estimate the sight cone area EC in consideration of the estimation accuracy of the sight vector EB. The driver's face orientation has a great influence on the estimation accuracy of the sight vector EB.

[0059] The sight cone area estimation unit 12 estimates the sight cone area EC according to the driver's facial orientation. The sight cone area estimation unit 12 recognizes the driver's facial orientation based on the driver's photographed image of the driver monitoring camera 1. The recognition of the driver's facial orientation can be performed using a known technique, and therefore, a detailed description thereof is omitted.

[0060] Specifically, when the driver's face is oriented away from the front of the vehicle M, the sight cone area estimating unit 12 can increase the magnification of the sight cone area EC based on the distance from the viewpoint EP, compared to when the driver's face is oriented toward the front of the vehicle M.

[0061] The situation where the driver's face is oriented toward the front direction of the vehicle M refers to a situation where, for example, the driver's face orientation is within a set angle range relative to an axis parallel to the front and rear axes of the vehicle M passing through the viewpoint EP. The set angle range corresponds, for example, to a range in which the driver monitoring camera 1 can estimate the line of sight vector with high accuracy. The set angle range may be changed according to the specifications of the driver monitoring camera 1, and may also be changed according to the positional relationship between the driver monitoring camera 1 and the viewpoint EP. The situation where the driver's face orientation deviates from the front direction of the vehicle M refers to a situation where the driver's face orientation is outside the set angle range. It should be noted that the direction in which the driver's face orientation deviates is not limited to the lateral direction, but may also include the up and down direction.

[0062] Here, Figure 4 (a) is a diagram showing an example of a cross section of the visual cone area EC. Figure 4 (b) is a diagram showing an example of a cross section of the visual cone area EC when the magnification factor is increased. Figure 4 The cross section of the sight cone area EC of (b) becomes Figure 4(a) is a circle with a large cross section. When the driver's face is facing the front direction of the vehicle M, the sight cone area estimation unit 12 estimates Figure 4 (a) shows the sight cone area EC. When the driver's face is facing away from the front direction of the vehicle M, the sight cone area estimation unit 12 Figure 4 As shown in (b), the estimated size ratio Figure 4 (a) Large viewing cone area EC.

[0063] Since the estimation accuracy of the line of sight vector using the image captured by the driver monitoring camera 1 is likely to decrease as the driver's face direction deviates further, when the driver's face direction deviates from the front direction of the vehicle M, the line of sight cone area estimation unit 12 can improve the accuracy of the driver's visual recognition judgment of the object by increasing the magnification of the line of sight cone area EC.

[0064] In addition, the sight cone region estimating unit 12 may estimate the sight cone region EC whose cross section orthogonal to the sight vector EB is a longitudinally long ellipse when the driver's face is oriented toward the front direction of the vehicle M, and may estimate the sight cone region EC whose cross section is a transversely long ellipse when the driver's face is oriented transversely away from the front direction of the vehicle M. The longitudinal length means that the length in the longitudinal direction (up and down direction) is longer than the transverse length of the cross section orthogonal to the sight vector EB. Similarly, the transverse length means that the length in the transverse direction is longer than the longitudinal length of the cross section orthogonal to the sight vector EB.

[0065] Here, FIG. 5(a) is a diagram showing an example of a longitudinal cross section of the sight cone area EC. FIG. 5(b) is a diagram showing an example of a transverse cross section of the sight cone area EC. When the driver's face is oriented toward the front direction of the vehicle M, the sight cone area estimation unit 12 estimates the sight cone area EC with a longitudinally long elliptical cross section as shown in FIG. 5(a). When the driver's face is oriented laterally away from the front direction of the vehicle M, the sight cone area estimation unit 12 estimates the sight cone area with a transversely long elliptical cross section as shown in FIG. 5(b).

[0066] Since the estimation accuracy of the sight line vector in the pitch direction of the vehicle M is likely to decrease when the driver's face is oriented toward the front direction of the vehicle M, the sight line cone area estimation unit 12 can avoid erroneous judgment that the driver does not visually recognize the object by setting the cross section of the sight line cone area EC to a vertically long ellipse. Similarly, when the driver's face is oriented in the lateral direction of the vehicle M, the estimation accuracy of the sight line vector in the yaw direction of the vehicle M is likely to decrease, and therefore, the sight line cone area estimation unit 12 can improve the accuracy of the driver's visual recognition judgment of the object by setting the cross section of the sight line cone area EC to a horizontally long ellipse.

[0067] The estimation accuracy of the sight line vector EB varies according to the driver's face orientation, the light conditions that illuminate the driver's face, individual differences in glasses, makeup, facial expressions, etc., in addition to the above-mentioned driver's face orientation. The estimation accuracy of the sight line vector EB decreases when, for example, strong light from the outside illuminates the driver's face and white highlights appear around the driver's eyes. The estimation accuracy of the sight line vector EB also decreases when shadows appear only around the eyes of the driver's face due to the illumination method of light from the outside.

[0068] When the driver wears glasses, the estimation accuracy of the sight line vector EB is sometimes lower than when the driver does not wear glasses. The estimation accuracy of the sight line vector EB also varies depending on the edge shape of the glasses, etc. When makeup such as eyeliner or false eyelashes is applied around the eyes, the estimation accuracy of the sight line vector EB may be reduced due to the influence of the makeup. The estimation accuracy of the sight line vector EB may be reduced when the eyelids tend to droop due to individual differences among drivers such as advanced age. In the case where there is a major cause of reduced estimation accuracy of the sight line vector EB as described above, the sight line cone area estimation unit 12 may also increase the magnification of the sight line cone area EC compared to the case where there is no major cause of reduced estimation accuracy.

[0069] In addition, the sight cone area estimation unit 12 may also estimate the sight cone area EC according to the speed of the vehicle M. It is known that the effective viewing angle of the driver changes according to the vehicle speed. Specifically, when the speed of the vehicle M is above the speed determination threshold, the sight cone area estimation unit 12 may also reduce the magnification of the sight cone area EC based on the distance from the viewpoint EP compared to the case where the speed is less than the speed determination threshold. The speed determination threshold is a threshold of a preset value. The speed determination threshold may be 40 km / h, 50 km / h, or 60 km / h. The value of the speed determination threshold is not particularly limited.

[0070] Since the effective visual angle of the driver is narrowed when the vehicle speed is high, the sight cone area estimation unit 12 reduces the magnification of the sight cone area EC when the vehicle speed is equal to or higher than the vehicle speed determination threshold, thereby improving the accuracy of the driver's visual recognition determination of the object.

[0071] It should be noted that the sight cone area estimation unit 12 can set the effective viewing angle to a fixed value shared by all drivers, or can set a value for each driver individually when performing personal authentication of the driver. The value of the driver's individual effective viewing angle can be set, for example, according to the driver's age, height, gender, etc. The sight cone area estimation unit 12 can also perform a driver check related to the effective viewing angle by projecting a virtual object on the front windshield during driver registration and moving the line of sight within the range of visual recognition that the driver is considered to be. The sight cone area estimation unit 12 can also set the value of the driver's individual effective viewing angle based on the result of the driver check related to the effective viewing angle.

[0072] Furthermore, the sight cone area estimating unit 12 may estimate the sight cone area EC according to the driving state of the vehicle M. Specifically, the sight cone area estimating unit 12 may estimate the sight cone area such that a cross section orthogonal to the sight vector EB is enlarged in the turning direction (right direction or left direction) of the vehicle M when the vehicle M is turning, compared to a case where the vehicle M is not turning.

[0073] The sight cone area estimation unit 12 determines whether the vehicle M is turning based on the yaw rate information detected by the yaw rate sensor of the internal sensor 3, for example. The sight cone area estimation unit 12 determines that the vehicle M is turning when the yaw rate of the vehicle M is greater than the turning threshold. The sight cone area estimation unit 12 may also determine whether the vehicle M is turning based on the steering angle of the vehicle M. The sight cone area estimation unit 12 determines that the vehicle M is turning when the steering angle of the vehicle M is greater than the turning steering angle threshold. The turning threshold and the turning steering angle threshold are thresholds of preset values.

[0074] When the vehicle M is turning, the sight cone area estimation unit 12 estimates the sight cone area EC so that the cross section is expanded in the turning direction of the vehicle M. Referring to FIG. 5( b ), for example, when the turning direction is consistent with the arrow P shown in FIG. 5( b ), the sight cone area estimation unit 12 estimates the sight cone area EC whose cross section is expanded in the turning direction as shown in FIG. 5( b ). At this time, the longitudinal width of the sight cone area EC may be the same as when the vehicle M is not turning, or may be shorter than when the vehicle M is not turning.

[0075] The sight cone area estimation unit 12 expands the cross section of the sight cone area orthogonal to the sight vector in the turning direction of the vehicle. This is because when the vehicle is turning, it can be considered that the driver is aware of the turning destination and performs visual recognition. The sight cone area estimation unit 12 can improve the accuracy of the driver's visual recognition judgment of the object by expanding the sight cone area in the turning direction.

[0076] In addition, the sight cone area estimation unit 12 may also estimate the sight cone area EC so that the cross section expands in the moving direction of the object observed by the driver when the moving speed of the object observed by the driver is greater than the moving speed threshold. The moving speed of the object observed by the driver is recognized by the object recognition unit 11. The moving speed threshold is a threshold of a preset value.

[0077] Referring to FIG. 5( b ), the sight cone region estimation unit 12 estimates a sight cone region EC whose cross section is enlarged in the direction of travel P of the pedestrian 50, for example, when the moving speed of the pedestrian 50 is above the moving speed threshold, as shown in FIG. 5( b ). In this case, it can be considered that the driver wants to perform visual recognition by considering the movement of the object with a fast moving speed. Therefore, the sight cone region is estimated so that the cross section is enlarged in the moving direction of the object. As a result, the accuracy of the driver's visual recognition judgment of the object can be improved.

[0078] In addition, after determining that the driver is visually recognizing the object until determining that the driver is not visually recognizing the object, the sight cone area estimation unit 12 may increase the magnification of the sight cone area EC compared to before the driver visually recognizes the object. The visual recognition determination unit 13 to be described later performs the visual recognition determination of the object.

[0079] Reference Figure 4 (a) and Figure 4 (b) Explain, for example, Figure 4 (a) shows that the driver visually recognizes the pedestrian 50, and the period from when the driver visually recognizes the pedestrian 50 to when the driver is determined not to have visually recognized the pedestrian 50 is the same as the period before the driver visually recognizes the object. Figure 4 (a) Compared with the above, the sight cone area estimation unit 12 Figure 4 (b) shows that the magnification of the sight cone EC is increased. Since it is considered that even if the driver slightly deviates his sight from an object that he has visually recognized once, there is a high possibility that he is visually recognizing the object. Therefore, the sight cone estimating unit 12 can improve the accuracy of the driver's visual recognition judgment of the object by increasing the magnification of the sight cone.

[0080] Furthermore, the sight cone area estimation unit 12 can estimate the sight cone area EC in consideration of the recognition accuracy of the object. The recognition accuracy of the object includes the recognition accuracy of the position of the object and the size of the object. The recognition accuracy of the object varies depending on the detection error of the external sensor 2, the relative speed between the vehicle M and the object, and the like. As for the method of calculating the detection error of the external sensor 2, a known technique can be used, and therefore, a detailed description is omitted.

[0081] When the detection error of the external sensor 2 is greater than the error threshold, the recognition accuracy of the object may be lower than when the detection error is less than the error threshold. Therefore, the sight cone area estimation unit 12 may increase the magnification of the sight cone area EC.

[0082] When the relative speed between the vehicle M and the object is greater than the relative speed threshold, the recognition accuracy of the object may be reduced compared to the case where the relative speed between the vehicle M and the object is less than the relative speed threshold. Therefore, the sight cone area estimation unit 12 may increase the magnification of the sight cone area EC. The error threshold and the relative speed threshold are thresholds of preset values.

[0083] In the case of size misrecognition in which the size of the object is mistakenly recognized as larger, the sight cone area estimation unit 12 may reduce the magnification of the sight cone area EC compared to the case where the size misrecognition does not occur. The occurrence of size misrecognition can be determined based on the detection error of the external sensor 2 and the information of the size estimation error included in the fusion information. The occurrence of size misrecognition can be determined by a well-known technique.

[0084] In addition, the sight cone area estimation unit 12 may also estimate the sight cone area EC based on the position of the object relative to the vehicle M. Specifically, the sight cone area estimation unit 12 may also estimate the sight cone area EC as follows: the more the position of the object deviates laterally from the front direction of the vehicle M, the longer the cross section is in the lateral direction. In addition, the sight cone area estimation unit 12 may also estimate the sight cone area EC based on the category of the object. Specifically, when the category of the object is a two-wheeled vehicle, it is easier to miss than when the category of the object is a four-wheeled vehicle. Therefore, the sight cone area estimation unit 12 may also reduce the magnification of the sight cone area EC. In addition to two-wheeled vehicles, the reduction of the magnification of the sight cone area EC can also be applied to objects that are easy to miss, such as pedestrians.

[0085] The visual recognition determination unit 13 determines whether the driver is visually recognizing the object based on the object recognized by the object recognition unit 11 and the driver's sight cone area EC estimated by the sight cone area estimation unit 12. The visual recognition determination unit 13 determines whether the driver is visually recognizing the object based on, for example, the overlap of the sight cone area EC and the object in a three-dimensional coordinate system based on the vehicle M or the driver's viewpoint EP.

[0086] For example, when the overlap ratio between the sight cone area EC and the object is equal to or greater than the visual recognition determination threshold, the visual recognition determination unit 13 determines that the driver is visually recognizing the object (see Figure 3(a)). When the overlap ratio between the sight cone area EC and the object is less than the visual recognition determination threshold, the visual recognition determination unit 13 determines that the driver has not visually recognized the object (see Figure 3 (b)). The visual recognition determination threshold is a threshold value of a preset value. The visual recognition determination threshold may be 30%, 40%, or 50%. The value of the visual recognition determination threshold is not particularly limited.

[0087] The overlap ratio is the ratio of the object to the view cone area EC. The overlap ratio can be calculated using the object itself or using Figure 3 as well as Figure 4 A bounding box 60 as shown.

[0088] The visual recognition determination unit 13 may also determine whether the driver is visually recognizing the object based on the overlap between the sight cone area EC and the object in the plane coordinate system. The visual recognition determination unit 13 may also project the object and the sight cone area EC on a captured image that has been transformed in a manner based on the driver's viewpoint EP, and calculate the proportion of the object relative to the range of the sight cone area EC on the captured image as the overlap ratio. The visual recognition determination unit 13 may also use other well-known techniques to calculate the overlap ratio between the object and the sight cone area EC. The visual recognition determination unit 13 determines that the driver is visually recognizing the object when the overlap ratio between the sight cone area EC and the object is greater than the visual recognition determination threshold.

[0089] The visual recognition determination unit 13 may use a machine learning model to determine whether the driver is visually recognizing the object based on the overlap between the sight cone area EC and the object in the three-dimensional space coordinate system or the plane coordinate system.

[0090] The visual recognition determination unit 13 may be configured as follows: when the overlap ratio between the sight cone area EC and the object is greater than the visual recognition determination threshold value for a certain period of time, the visual recognition determination unit 13 may determine that the driver is visually recognizing the object. The visual recognition determination unit 13 may change the certain period of time to a longer period of time when it is nighttime than when it is daytime. The certain period of time may also be set to a different time depending on the type of the object.

[0091] The visual recognition determination unit 13 may also remind the driver of the need for attention or provide information based on the visual recognition determination result. The visual recognition determination unit 13 executes the reminder of the need for attention or provides information to the driver by sending a control signal to the HMI4. The visual recognition determination unit 13 reminds the driver of the need for attention of an object that is not visually recognized by the driver, for example, by means of the sound output of the speaker of the HMI4 or by means of an image display on a display. The visual recognition determination unit 13 may also provide information related to the object that the driver is visually recognizing by means of the sound output of the speaker or by means of an image display on a display. In addition, the visual recognition determination unit 13 may also send the driver's visual recognition determination result of the object to the driving assistance system or the automatic driving system of the vehicle M.

[0092] [Control method of visual recognition judgment device]

[0093] Next, a control method of the visual recognition assessment device 100 according to the present embodiment will be described with reference to the drawings. Figure 6 2 is a flowchart showing an example of the visual recognition determination process. The visual recognition determination process is executed when the ignition switch of the vehicle M is turned on, for example.

[0094] like Figure 6 As shown, as S10, the visual recognition determination ECU 10 of the visual recognition determination device 100 determines whether an object outside the vehicle M is recognized by the object recognition unit 11. The object recognition unit 11 recognizes the object by, for example, sensor fusion. When the visual recognition determination ECU 10 determines that the object is recognized (S10: Yes), it transfers to S11. When the visual recognition determination ECU 10 does not determine that the object is recognized (S10: No), it ends the visual recognition determination processing. The visual recognition determination ECU 10 repeats the processing from S10 after a certain period of time.

[0095] In S11, the visual recognition determination ECU 10 recognizes the driver's facial orientation through the sight cone area estimation unit 12. The sight cone area estimation unit 12 recognizes the driver's facial orientation based on the driver photographed image of the driver monitoring camera 1. Thereafter, the visual recognition determination ECU 10 shifts to S12.

[0096] In S12, the visual recognition determination ECU 10 estimates the driver's sight line vector EB through the sight line cone area estimation unit 12. The sight line cone area estimation unit 12 estimates the driver's sight line vector EB extending from the driver's viewpoint EP based on the driver's photographed image of the driver monitoring camera 1. Thereafter, the visual recognition determination ECU 10 transfers to S13. It should be noted that the recognition of the driver's facial orientation and the estimation of the driver's sight line vector EB may also be performed simultaneously.

[0097] In S13, the visual recognition determination ECU 10 determines whether the driver's face direction is deviated from the front direction of the vehicle M through the sight cone area estimation unit 12. When the visual recognition determination ECU 10 determines that the driver's face direction is deviated from the front direction of the vehicle M (S13: Yes), the visual recognition determination ECU 10 shifts to S14. When the visual recognition determination ECU 10 does not determine that the driver's face direction is deviated from the front direction of the vehicle M (S13: No), the visual recognition determination ECU 10 shifts to S15.

[0098] In S14, the visual recognition determination ECU 10 estimates, through the sight cone area estimation unit 12, an enlarged sight cone area EC compared to the case of S15 described later. The enlarged sight cone area EC refers to an increased magnification of the sight cone area EC based on the distance from the viewpoint EP. The sight cone area estimation unit 12 sets the sight cone area EC that expands in a conical shape based on the sight vector EB. It should be noted that the sight cone area estimation unit 12 can also estimate the sight cone area EC with a horizontally long cross-section as shown in Figure 5(b). After that, the visual recognition determination ECU 10 transfers to S16.

[0099] In S15, the visual recognition determination ECU 10 estimates the visual conic area EC through the visual conic area estimation unit 12. Here, the visual conic area EC is not expanded according to the driver's face orientation. It should be noted that the visual conic area estimation unit 12 can also estimate the visual conic area EC with a vertically long cross section as shown in FIG. 5(a). After that, the visual recognition determination ECU 10 transfers to S16.

[0100] In S16, the visual recognition determination ECU 10 determines whether the overlap ratio between the sight cone area EC and the object is greater than the visual recognition determination threshold value through the visual recognition determination unit 13. When the visual recognition determination ECU 10 determines that the overlap ratio between the sight cone area EC and the object is greater than the visual recognition determination threshold value (S16: Yes), the process moves to S17. When the visual recognition determination ECU 10 does not determine that the overlap ratio between the sight cone area EC and the object is greater than the visual recognition determination threshold value (S16: No), the process moves to S18.

[0101] In S17, the visual recognition determination ECU 10 determines that the driver is visually recognizing the object through the visual recognition determination unit 13. The visual recognition determination unit 13 sends the driver's visual recognition determination result to the driving assistance system or the automatic driving system of the vehicle M. After that, the visual recognition determination ECU 10 ends the visual recognition determination processing and repeats the processing from S10 after a certain period of time.

[0102] In S18, the visual recognition determination ECU 10 determines that the driver has not visually recognized the object through the visual recognition determination unit 13. The visual recognition determination unit 13 sends the driver's visual recognition determination result to the driving assistance system or the automatic driving system of the vehicle M. After that, the visual recognition determination ECU 10 ends the visual recognition determination processing and repeats the processing from S10 after a certain time has passed.

[0103] The vision recognition determination ECU 10 may estimate the sight line cone area EC based on various conditions other than the driver's facial orientation. Figure 7 (a) is a flowchart showing an example of a sight cone area estimation process according to vehicle speed. The sight cone area estimation process can be performed, for example, Figure 6 The sight cone area estimation process may be executed between S12 and S13 in the flowchart of FIG. 14 or between S15 and S16. The same applies to various sight cone area estimation processes described later.

[0104] like Figure 7 As shown in (a), as S20, the visual recognition determination ECU 10 determines whether the vehicle speed of the vehicle M is greater than or equal to the vehicle speed determination threshold value through the sight cone area estimation unit 12. When the visual recognition determination ECU 10 determines that the vehicle speed is greater than or equal to the vehicle speed determination threshold value (S20: Yes), it transfers to S21. When the visual recognition determination ECU 10 does not determine that the vehicle speed is greater than or equal to the vehicle speed determination threshold value (S20: No), it transfers to S22.

[0105] In S21, the vision recognition determination ECU 10 estimates the vision cone area EC by the vision cone area estimation unit 12. In this case, the vision cone area EC is not changed according to the vehicle speed. Thereafter, the vision recognition determination ECU 10 ends the vision cone area estimation process.

[0106] In S22, the visual recognition determination ECU 10 estimates a visual cone area EC that is reduced compared to the case of S21 through the visual cone area estimation unit 12. The reduced visual cone area EC means that the magnification of the visual cone area EC based on the distance from the viewpoint EP is reduced. Thereafter, the visual recognition determination ECU 10 ends the visual cone area estimation process.

[0107] Figure 7 (b) is a flowchart showing an example of the processing of estimating the sight cone area corresponding to the turning of the vehicle. Figure 7As shown in (b), as S30, the visual recognition determination ECU 10 determines whether the vehicle M is turning through the sight cone area estimation unit 12. The sight cone area estimation unit 12 determines whether the vehicle M is turning based on the yaw rate of the vehicle M or the steering angle of the vehicle M. When the visual recognition determination ECU 10 determines that the vehicle M is turning (S30: Yes), it transfers to S31. When the visual recognition determination ECU 10 does not determine that the vehicle M is turning (S30: No), it transfers to S32.

[0108] In S31, the visual recognition determination ECU 10 estimates, through the visual cone area estimation unit 12, a visual cone area EC that is enlarged in the turning direction of the vehicle M compared to the case of S32 described later. The visual cone area EC that is enlarged in the turning direction of the vehicle M refers to a visual cone area EC in which a cross section orthogonal to the visual line vector EB is enlarged in the turning direction of the vehicle M. Thereafter, the visual recognition determination ECU 10 ends the visual cone area estimation process.

[0109] In S32 , the vision recognition determination ECU 10 estimates the vision cone area EC by the vision cone area estimation unit 12 . In this case, the vision cone area EC is not changed according to the turning of the vehicle M. Thereafter, the vision recognition determination ECU 10 ends the vision cone area estimation process.

[0110] Figure 8 (a) is a flowchart showing an example of the processing of estimating the sight cone area according to the moving speed of the object. Figure 8 As shown in (a), as S40, the visual recognition determination ECU 10 determines whether the moving speed of the object observed from the driver is greater than the moving speed threshold value through the sight cone area estimation unit 12. When the visual recognition determination ECU 10 determines that the moving speed of the object observed from the driver is greater than the moving speed threshold value (S40: Yes), it transfers to S41. When the visual recognition determination ECU 10 does not determine that the moving speed of the object observed from the driver is greater than the moving speed threshold value (S40: No), it transfers to S42.

[0111] In S41, the visual recognition determination ECU 10 estimates the visual cone area EC expanded in the moving direction of the object compared with the case of S42 described later through the visual cone area estimation unit 12. The visual cone area EC expanded in the moving direction of the object refers to the visual cone area EC whose cross section orthogonal to the visual vector EB is expanded in the moving direction of the object. After that, the visual recognition determination ECU 10 ends the visual cone area estimation process.

[0112] In S42, the vision recognition determination ECU 10 estimates the vision cone area EC by the vision cone area estimation unit 12. In this case, the vision cone area EC is not changed according to the moving speed of the object. Thereafter, the vision recognition determination ECU 10 ends the vision cone area estimation process.

[0113] Figure 8 (b) is a flowchart showing an example of the processing of estimating the sight cone area according to the visual recognition status of the driver. Figure 8 As shown in (b), as S50, the visual recognition determination ECU 10 determines whether the object of the visual recognition determination processing is the object determined to be visually recognized by the driver through the sight cone area estimation unit 12. When the visual recognition determination ECU 10 determines that it is the object determined to be visually recognized by the driver (S50: Yes), it transfers to S51. When the visual recognition determination ECU 10 does not determine that it is the object determined to be visually recognized by the driver (S50: No), it transfers to S52.

[0114] In S51 , the vision recognition determination ECU 10 estimates the vision cone area EC expanded compared with the case of S52 described later by the vision cone area estimation unit 12. Thereafter, the vision recognition determination ECU 10 ends the vision cone area estimation process.

[0115] In S52, the visual recognition determination ECU 10 estimates the visual cone area EC by the visual cone area estimation unit 12. In this case, the visual cone area EC is not changed according to whether or not the object is visually recognized. Thereafter, the visual recognition determination ECU 10 ends the visual cone area estimation process.

[0116] According to the visual recognition determination device 100 (and the control method of the visual recognition determination device 100) of the present embodiment described above, the estimation accuracy of the sight line vector EB based on the image captured by the driver monitoring camera 1 decreases depending on the driver's face orientation. Therefore, by estimating the sight line cone area EC based on the face orientation, the accuracy of the driver's visual recognition determination of the object can be improved.

[0117] In addition, according to the visual recognition determination device 100, when the driver's face is oriented toward the front direction of the vehicle M, the estimation accuracy of the sight line vector in the pitch direction of the vehicle M is low. Therefore, the estimated cross section becomes a vertically long elliptical sight line cone area EC. When the driver's face is oriented laterally from the front direction of the vehicle M, the estimation accuracy of the sight line vector in the yaw direction of the vehicle M becomes low. As a result, the estimated cross section becomes a horizontally long elliptical sight line cone area EC. As a result, the accuracy of the driver's visual recognition determination of the object can be improved.

[0118] In addition, according to the visual recognition judgment device 100, since the driver's effective viewing angle becomes narrower when the vehicle speed is high, when the vehicle speed is above the vehicle speed judgment threshold, the driver's visual recognition judgment accuracy of the object can be improved by reducing the magnification of the visual cone area.

[0119] Moreover, according to the visual recognition judgment device 100, when the vehicle M is turning, it can be considered that the driver is consciously observing the turning destination. Therefore, by expanding the cross-section orthogonal to the line of sight vector of the line of sight cone area EC in the turning direction of the vehicle M, the accuracy of the driver's visual recognition judgment of the object can be improved.

[0120] In addition, according to the visual recognition judgment device 100, when the moving speed of an object observed by the driver is fast, the driver wants to consider the movement of the object for visual recognition. Therefore, by estimating the line of sight cone area EC in a manner that expands a cross section orthogonal to the line of sight vector in the moving direction of the object, the accuracy of the driver's visual recognition judgment of the object can be improved.

[0121] In addition, according to the visual recognition judgment device 100, since it can be considered that even if the driver slightly deviates his line of sight from an object that has been visually recognized once, there is a high possibility that the object is being visually recognized. Therefore, by increasing the magnification of the visual cone area EC, the accuracy of the driver's visual recognition judgment of the object can be improved.

[0122] Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. The present invention can be implemented in various forms represented by the above embodiments and variously estimated and improved based on the knowledge of those skilled in the art.

Claims

1. A visual recognition determination device for determining whether a driver is visually recognizing an object detected by an external sensor of a vehicle, wherein: The visual recognition determination device comprises: a sight cone region estimating unit, the sight cone region estimating unit estimating a sight vector of the driver through a driver monitoring camera of the vehicle, and estimating a sight cone region that expands in a conical shape as it moves away from a viewpoint of the driver, based on the sight vector of the driver; as well as a visual recognition determination unit configured to determine whether the driver is visually recognizing the object based on an overlap between the sight cone area and the object, The sight cone area estimating unit deforms the sight cone area according to the driver's face orientation.

2. The visual recognition determination device according to claim 1, wherein: When the driver's face is oriented away from the front direction of the vehicle, the sight cone area estimating unit increases the magnification of the sight cone area based on the distance from the viewpoint, compared to when the driver's face is oriented toward the front direction.

3. The visual recognition determination device according to claim 1 or 2, wherein: When the driver's face is oriented toward the front direction of the vehicle, the sight cone area estimating unit estimates that the sight cone area has a cross section orthogonal to the sight vector and is in the shape of a longitudinally elongated ellipse. When the driver's face is oriented laterally away from the front direction of the vehicle, the sight cone area estimating unit estimates that the sight cone area has a cross section and is in the shape of a transversely elongated ellipse.

4. The visual recognition determination device according to claim 1 or 2, wherein: When the vehicle speed is equal to or greater than a vehicle speed determination threshold, the sight cone area estimation unit reduces the magnification rate of the sight cone area based on the distance from the viewpoint compared to when the vehicle speed is less than the vehicle speed determination threshold.

5. The visual recognition determination device according to claim 1 or 2, wherein: When the vehicle is turning, the sight cone area estimating unit estimates the sight cone area such that a cross section orthogonal to the sight vector is enlarged in the turning direction of the vehicle, compared to a case where the vehicle is not turning.

6. The visual recognition determination device according to claim 1 or 2, wherein: When the moving speed of the object viewed from the driver is equal to or greater than a moving speed threshold, the sight line cone area estimating unit estimates the sight line cone area such that a cross section orthogonal to the sight line vector expands in a moving direction of the object viewed from the driver.

7. The visual recognition determination device according to any one of claims 1 to 3, wherein: After determining that the driver is visually recognizing the object, until determining that the driver is not visually recognizing the object, the sight cone area estimating unit increases the magnification rate of the sight cone area compared to before determining that the driver is visually recognizing the object.

8. A control method for a visual recognition determination device for determining whether a driver is visually recognizing an object detected by an external sensor of a vehicle, wherein: The control method of the visual recognition determination device includes: estimating a sight line vector of the driver by a driver monitoring camera of the vehicle, and estimating a sight line cone area that expands in a conical shape as it moves away from the driver's viewpoint based on the sight line vector of the driver; estimating the sight cone area according to the driver's facial orientation; and Whether the driver is visually recognizing the object is determined based on the overlap between the sight cone area and the object.