Visual recognition determination device

Abstract

To provide a visual recognition determination device for suppressing deterioration of accuracy in visual recognition determination of a driver with respect to an object by correction of a visual recognition range of the driver in an inappropriate state.SOLUTION: A visual recognition determination device compares a visual line of a driver of a vehicle detected by a driver monitor camera of the vehicle with object information in front of the vehicle acquired by an external sensor of the vehicle to correct a visual recognition range of the driver. When a vehicle speed of the vehicle is equal to or more than a correction prohibition threshold or when a travel posture of the vehicle is not a direct advance posture, the correction of the visual recognition range of the driver is not performed.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a visual recognition determination device.

Background Art

[0002] Conventionally, as a technical document related to a visual recognition determination device, Japanese Patent Application Laid-Open No. 2020-071773 is known. This publication shows a device that calibrates the driver's line of sight detected by a line-of-sight detection unit by comparing the position of obstacle information in the driver's field of view with the line of sight detected by the line-of-sight detection unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the above-described conventional device, the influence of the vehicle running state on calibration is not considered. For this reason, there is a risk of reducing the accuracy of visual recognition determination by performing calibration under inappropriate situations such as when the vehicle is running at high speed or turning.

Means for Solving the Problems

[0005] One aspect of the present invention is It has a fixation area map which is a coordinate map preset with respect to a vehicle, Based on the line of sight of the driver of the vehicle detected by the driver monitor camera of the vehicle The visual range of the driver of the vehicle estimated on the fixation area map And the object in front of the vehicle acquired by the external sensor of the vehicle The position on the fixation area map By comparing the two, The fixation area map used for the driver's visual recognition determination A visual recognition determination device that corrects, when the vehicle speed of the vehicle is equal to or higher than a correction prohibition threshold value, or when the running posture of the vehicle is not a straight-ahead posture, The fixation area map Does not perform the correction.

Advantages of the Invention

[0006] According to one aspect of the present invention, it is possible to suppress a decrease in the accuracy of a driver's visual recognition determination for an object by correcting the driver's visual recognition range in an inappropriate situation.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Modes for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0009] The visual recognition determination device 100 shown in FIG. 1 is mounted on a vehicle such as a passenger car, and is a device that determines whether a driver of the vehicle is visually recognizing an object outside the vehicle. When the driver has never visually recognized an object to be visually recognized such as a pedestrian, the visual recognition determination device 100 alerts the driver by outputting an alarm. Further, the visual recognition determination device 100 may be used to confirm the monitoring state of the driver during driving support by transmitting the driver's visual recognition determination result to the driving support device.

[0010] The visual recognition determination device 100 performs visual recognition determination by estimating the driver's line of sight and detecting an object such as another vehicle in front of the vehicle. Further, the visual recognition determination device 100 corrects the visual recognition range by comparing the estimated result of the driver's visual recognition range with the detection result of the object. Since it may not be possible to appropriately estimate the visual recognition range due to individual differences of the driver or the like in visual recognition determination, the decrease in the accuracy of visual recognition determination is suppressed by performing correction of the visual recognition range under predetermined conditions.

[0011] [Configuration of Visual Recognition Determination Device] As shown in FIG. 1, the visual recognition determination device 100 includes a visual recognition determination ECU 10 [Electronic Control Unit]. The visual recognition determination ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit such as a ROM [Read Only Memory] or a RAM [Random Access Memory]. The visual recognition determination ECU 10 may be a part of the driving support ECU or the automatic driving ECU.

[0012] The visual recognition determination ECU 10 is connected to the driver monitor camera 1, the external sensor 2, the internal sensor 3, the map database 4, and the HMI 5 [Human Machine Interface].

[0013] The driver monitor camera 1 is a camera for imaging the driver. The driver monitor camera 1 can be a digital camera having an imaging element such as a CCD [Charge Coupled Device] or a CIS [CMOS Image Sensor]. The driver monitor camera 1 is provided, for example, on the cover of the steering column in front of the driver, and images the head including the driver's face at a predetermined frame rate. The driver monitor camera 1 transmits the driver imaging image to the visual recognition determination ECU 10.

[0014] The external sensor 2 is a detection device for detecting the situation around the vehicle. The external sensor 2 includes at least one of a camera and a radar sensor.

[0015] The camera is an imaging device that captures the external situation of the vehicle. The camera is provided, for example, on the back side of the front windshield of the vehicle and captures the front of the vehicle. The camera transmits imaging information regarding the external situation of the vehicle to the visual recognition determination ECU 10. The radar sensor is a detection device that detects objects around the vehicle using radio waves (e.g., millimeter waves) or light. The radar sensor includes, for example, a millimeter wave radar or a lidar [LIDAR: Light Detection and Ranging]. The radar sensor transmits information on the detected objects to the visual recognition determination ECU 10.

[0016] The internal sensor 3 is a detection device that detects the driving state of the vehicle. The internal sensor 3 includes a vehicle speed sensor and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle. The vehicle speed sensor transmits the detected vehicle speed information (wheel speed information) to the visual recognition determination ECU 10.

[0017] The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the vehicle. As the yaw rate sensor, for example, a gyro sensor can be used. The yaw rate sensor transmits the detected yaw rate information of the vehicle to the visual recognition determination ECU 10.

[0018] The map database 4 is a database that stores map information. The map information includes road position information, road shape information (e.g., types of curves, straight sections, curvature of curves, etc.), intersection and branch point position information, and structure position information, etc.

[0019] The HMI 5 is an interface for inputting and outputting information between the visual recognition determination device 100 and the driver. The HMI 5 is equipped with a display and a speaker. The HMI 5 may be configured as a device including a meter ECU, and the display is, for example, an MID [Multi Information Display]. The HMI 5 performs image output of the display and voice output from the speaker according to a control signal from the visual recognition determination ECU 10.

[0020] Next, the functional configuration of the visual recognition determination ECU 10 will be described. As shown in FIG. 1, the visual recognition determination ECU 10 includes a visual range estimation unit 11, an object recognition unit 12, a visual recognition determination unit 13, and a correction execution unit 14.

[0021] The visual range estimation unit 11 estimates the driver's visual range based on the driver imaging image captured by the driver monitor camera 1. The visual range is a range estimated to be visually recognized by the driver on a preset fixation area map. The fixation area map is used for associating an object outside the vehicle with the driver's visual range in a coordinate system (vehicle coordinate system) based on the vehicle. The fixation area map may be formed as a three-dimensional space coordinate map considering the influence of the depth direction, or may be formed as a two-dimensional plane coordinate map. For the fixation area map, a well-known map (coordinate system) can be adopted.

[0022] A plurality of fixation area maps may be prepared. When driver personal authentication is performed, an appropriate map may be selected according to the driver's personal data (physical build, seat height, eye point height in the reference posture, intensity of perspective, etc.). Also, the fixation area map may have an appropriate map selected at regular intervals according to the driver's current posture, current seat position, current eye point position, etc.

[0023] Here, FIG. 2(a) is a diagram for explaining an example of a situation where the visual range is corrected. In FIG. 2(a), a fixation area map C, the driver's line of sight EL, the driver's visual range EA, a first other vehicle N, a detection box DN of the first other vehicle N, a second other vehicle M, and a detection box DM of the second other vehicle M are shown. Also, the movement vector BE of the driver's visual range EA and the movement vector BN of the detection box DN of the first other vehicle N are shown. Here, for ease of understanding, the fixation area map C will be described as a plane coordinate map. The plane coordinate map is set as a plane orthogonal to the vehicle depth direction as seen from the driver, for example.

[0024] In the situation shown in Fig. 2(a), the visual range estimation unit 11 estimates the driver's line of sight EL based on the driver imaging image of the driver monitor camera 1. The visual range estimation unit 11 estimates the area corresponding to the line of sight EL on the preset fixation area map C as the visual range EA. Note that the visual range estimation unit 11 does not necessarily have to estimate the line of sight EL, and the visual range EA may be directly estimated from the driver imaging image by a machine learning model or the like.

[0025] Based on the estimation result of the driver's visual range EA, the visual range estimation unit 11 recognizes the movement vector BE of the visual range EA. The movement vector BE is obtained from the movement direction and movement speed of the visual range EA on the fixation area map C.

[0026] Based on the detection result of the external sensor 2, the object recognition unit 12 recognizes the position of the object in front of the vehicle, the movement direction of the object, and the movement speed of the object. The objects include moving bodies such as other vehicles, bicycles, pedestrians, and animals. The objects may include stationary objects such as pylons, guardrails, road signs, road markings, utility poles, curbs, billboards, and buildings such as stores and houses. Even if the object is a stationary object, it can be used for correcting the visual range by moving relatively due to the running of the vehicle. The object recognition unit 12 may recognize stationary objects using the vehicle's position information and the map information of the map database 4.

[0027] Also, the object recognition unit 12 recognizes the position of the object on the fixation area map C, the movement direction of the object, and the movement speed of the object by a well-known method (for example, coordinate transformation). The movement direction of the object on the fixation area map C is the relative movement direction seen from the driver of the vehicle. The movement speed of the object on the fixation area map C is the relative movement speed seen from the driver of the vehicle. The object recognition unit 12 also recognizes the size (area) of the object on the fixation area map C.

[0028] In the situation shown in Fig. 2(a), the object recognition unit 12 recognizes a first other vehicle N moving in a direction crossing the front of the vehicle. The object recognition unit 12 recognizes and tracks the first other vehicle N, for example, in the form of a detection box DN of a cube. The object recognition unit 12 recognizes the position of the first other vehicle N on the fixation area map C, the moving direction of the first other vehicle N, and the moving speed of the first other vehicle N by coordinate conversion. The object recognition unit 12 obtains a movement vector BN from the moving direction and moving speed of the first other vehicle N. Similarly, the object recognition unit 12 recognizes a second other vehicle M, which is an oncoming vehicle of the vehicle, and a detection box DM.

[0029] Based on the visual recognition range EA recognized by the visual recognition range estimation unit 11 and the object in front of the vehicle recognized by the object recognition unit 12, the visual recognition determination unit 13 performs a visual recognition determination of the object for the driver. The visual recognition determination unit 13 determines whether the driver can visually recognize the object from the overlap between the driver's visual recognition range EA and the object on the fixation area map C.

[0030] For example, when the overlapping ratio of the driver's visual recognition range EA for an object such as the detection box DN of the first other vehicle N on the fixation area map C is equal to or greater than a certain threshold value, the visual recognition determination unit 13 determines that the driver can visually recognize the object. The certain threshold value may be 70% or 80%. The value of the certain threshold value is not particularly limited.

[0031] In the situation shown in Fig. 2(a), since the driver's visual recognition range EA and the detection box DN of the first other vehicle N do not overlap, the visual recognition determination unit 13 determines that the driver cannot visually recognize the first other vehicle N. Note that the visual recognition determination unit 13 may perform a visual recognition determination by other well-known methods based on the relationship between the visual recognition range EA and the object on the fixation area map C.

[0032] When a preset correction condition is satisfied, the correction execution unit 14 corrects the visual recognition determination of the driver based on the relationship between the driver's visual recognition range EA and the object on the fixation area map C. Specifically, the correction execution unit 14 corrects the fixation area map C used for the driver's visual recognition determination.

[0033] The correction execution unit 14 determines, as a correction condition, whether or not the movement vector BE of the visual range EA is equal to or greater than a certain length on the fixation area map C. The length as the determination threshold value is a preset value. If the correction execution unit 14 determines that the movement vector BE of the visual range EA is equal to or greater than a certain length, it is considered that one of the correction conditions is satisfied.

[0034] In addition, the correction execution unit 14 determines, as a correction condition, whether or not there is an object within a predetermined range from the driver's visual range EA on the fixation area map C. The zero distance is not included in the predetermined range (that is, an object overlapping with the visual range EA is not included). If the correction execution unit 14 determines that there is an object within the predetermined range from the visual range EA, it is considered that one of the correction conditions is satisfied. The distance determination may be made using the distance between the starting point of the movement vector BE of the driver's visual range EA and the starting point of the movement vector of the object. Hereinafter, an object within the predetermined range from the visual range EA is referred to as a target object.

[0035] The correction execution unit 14 determines, as a correction condition, whether or not the movement vector of the target object and the movement vector BE of the driver's visual range EA are similar on the fixation area map C. An example of the case where two movement vectors are similar is when the angular difference formed by the two movement vectors on the fixation area map C is less than the angular threshold value, and the magnitude of the difference in the lengths of the two movement vectors (absolute value) is less than the length threshold value. The angular threshold value and the length threshold value are preset values. When the fixation area map C is a spatial coordinate map, the determination is made using the angular difference formed by the two movement vectors on a plane orthogonal to the vehicle depth direction as seen from the driver.

[0036] In addition, the similarity determination of the two movement vectors is not limited to the method described above. The correction execution unit 14 may determine that the two movement vectors are similar when they draw the same normal line.

[0037] When the correction execution unit 14 determines that, for example, in the situation shown in Fig. 2(a), the movement vector BN of the first other vehicle N is similar to the movement vector BE of the driver's visual range EA. When the correction execution unit 14 determines that the movement vector of the object (for example, the movement vector BN of the first other vehicle N) on the fixation area map C is similar to the movement vector BE of the driver's visual range EA, it is considered that one of the correction conditions is satisfied.

[0038] Furthermore, the correction execution unit 14 determines whether or not the vehicle speed of the vehicle is less than the correction prohibition threshold as a correction condition. The correction prohibition threshold may be 40 km / h, may be 50 km / h, or may be 60 km / h. The value of the correction prohibition threshold is not particularly limited. The vehicle speed of the vehicle can be obtained from the vehicle speed sensor of the internal sensor 3. When the correction execution unit 14 determines that the vehicle speed of the vehicle is less than the correction prohibition threshold, it is considered that one of the correction conditions is satisfied.

[0039] Also, the correction execution unit 14 determines whether or not the traveling posture of the vehicle is a straight-ahead posture as a correction condition. The straight-ahead posture is a posture in which the vehicle is not turning. The correction execution unit 14 determines the straight-ahead posture, for example, based on the yaw rate of the vehicle. The correction execution unit 14 may determine that the traveling posture of the vehicle is a straight-ahead posture when the state where the yaw rate of the vehicle is less than a predetermined value continues for a certain period of time. The correction execution unit 14 may determine the straight-ahead posture from the steering angle of the vehicle. When the correction execution unit 14 determines that the traveling posture of the vehicle is a straight-ahead posture, it is considered that one of the correction conditions is satisfied.

[0040] When all the correction conditions are satisfied, the correction execution unit 14 corrects the fixation area map C. The correction execution unit 14 determines that all the correction conditions are satisfied in the situation shown in Fig. 2(b). When even one of the correction conditions is not satisfied, the correction execution unit 14 does not correct the fixation area map C.

[0041] The correction execution unit 14 corrects the attention area map C by enlarging, reducing, or shifting the area corresponding to the object in the attention area map C up, down, left, or right. FIG. 2(b) is a diagram for explaining an example of the correction of the attention area map.

[0042] As shown in FIG. 2(b), the correction execution unit 14 corrects the attention area map C based on the relationship between the first other vehicle N and the driver's visual range EA on the attention area map C. The correction execution unit 14 corrects the attention area map C so that it is easier to determine that the driver is visually recognizing the first other vehicle N. Specifically, the correction execution unit 14 performs an enlargement correction in which the area corresponding to the first other vehicle N in the attention area map C is enlarged up, down, left, and right to combine the surrounding areas. As a result, the driver's visual range EA coordinate-converted on the attention area map C slides toward the first other vehicle N. The correction execution unit 14 may set a larger slide correction amount as the distance between the visual range EA and the first other vehicle N on the attention area map C is greater, for example.

[0043] Similarly, the correction execution unit 14 corrects the attention area map C based on the relationship between the second other vehicle M and the driver's visual range EA on the attention area map C so that it is easier to determine that the driver is visually recognizing the second other vehicle M. For example, assume that the driver's visual range EA has moved to a position away from the second other vehicle M to the right. In this case, the correction execution unit 14 performs a right shift correction in which the area corresponding to the second other vehicle M in the attention area map C is shifted to the right, for example. As a result, the driver's visual range EA coordinate-converted on the attention area map C slides toward the second other vehicle M, making it easier to determine that the driver is visually recognizing the second other vehicle M.

[0044] Note that the correction execution unit 14 may make it easier to determine that the driver is visually recognizing the first other vehicle N by sliding not the area corresponding to the first other vehicle N in the attention area map C but the surrounding area of the area or the entire map excluding the area.

[0045] The correction execution unit 14 may limit the number of corrections for each region corresponding to the object. For the region where a predetermined number of corrections have been performed, the correction execution unit 14 may make the correction conditions stricter so that corrections are less likely to occur, or may reduce the correction amount. The predetermined number may be one, two, or three or more. The correction conditions can be made stricter, for example, by changing the values of various thresholds to values that are difficult to judge.

[0046] [Processing of the Visual Recognition Determination Device] Next, the processing of the visual recognition determination device 100 according to the present embodiment will be described with reference to FIG. 3. FIG. 3 is a flowchart showing an example of the visual range correction process. The visual range correction process is repeatedly executed when the ignition of the vehicle is in the ON state.

[0047] As shown in FIG. 3, the visual recognition determination ECU 10 of the visual recognition determination device 100, as S10, estimates the driver's visual range EA by the visual range estimation unit 11. Based on the driver imaging image captured by the driver monitor camera 1, the driver's visual range EA on the fixation area map C is estimated.

[0048] In S11, the visual recognition determination ECU 10 recognizes the objects in front of the vehicle by the object recognition unit 12. The object recognition unit 12 recognizes the position of the object, the moving direction of the object, and the moving speed of the object on the fixation area map C based on the detection result of the external sensor 2. The object recognition unit 12 also recognizes the movement vector of the object on the fixation area map C.

[0049] In S12, the visual recognition determination ECU 10 determines whether the length of the movement vector BE of the visual range EA is equal to or greater than a certain value by the correction execution unit 14. When the visual recognition determination ECU 10 determines that the length of the movement vector BE of the visual range EA is equal to or greater than a certain value (S12: YES), it proceeds to S13. When the visual recognition determination ECU 10 does not determine that the length of the movement vector BE of the visual range EA is equal to or greater than a certain value (S12: NO), the visual range correction process ends.

[0050] In S13, the visual recognition determination ECU 10 determines whether there is an object within a predetermined range from the driver's visual range EA on the fixation area map C by the correction execution unit 14. When the visual recognition determination ECU 10 determines that there is an object within a predetermined range from the driver's visual range EA (S13: YES), it proceeds to S14. When the visual recognition determination ECU 10 determines that there is no object within a predetermined range from the driver's visual range EA (S13: NO), it ends the visual range correction process.

[0051] In S14, the visual recognition determination ECU 10 determines whether the movement vector of the target object and the movement vector BE of the driver's visual range EA are similar on the fixation area map C by the correction execution unit 14. When the visual recognition determination ECU 10 determines that the movement vector of the target object and the movement vector BE of the driver's visual range EA are similar (S14: YES), it proceeds to S15. When the visual recognition determination ECU 10 determines that the movement vector of the target object and the movement vector BE of the driver's visual range EA are not similar (S14: NO), it ends the visual range correction process.

[0052] In S15, the visual recognition determination ECU 10 determines whether the vehicle speed of the vehicle is less than the correction prohibition threshold by the correction execution unit 14. When the visual recognition determination ECU 10 determines that the vehicle speed of the vehicle is less than the correction prohibition threshold (S15: YES), it proceeds to S16. When the visual recognition determination ECU 10 determines that the vehicle speed of the vehicle is not less than the correction prohibition threshold (S15: NO), it ends the visual range correction process.

[0053] In S16, the visual recognition determination ECU 10 determines whether the running posture of the vehicle is a straight-ahead posture by the correction execution unit 14. When the visual recognition determination ECU 10 determines that the running posture of the vehicle is a straight-ahead posture (S16: YES), it proceeds to S17. When the visual recognition determination ECU 10 determines that the running posture of the vehicle is not a straight-ahead posture (S16: NO), it ends the visual range correction process.

[0054] In S17, the visual recognition determination ECU 10 corrects the fixation area map C (correction of the visual recognition range) by the correction execution unit 14. The correction execution unit 14 corrects the fixation area map C based on, for example, the relationship between the first other vehicle N and the driver's visual recognition range EA on the fixation area map C shown in FIG. 2(a). The correction execution unit 14 corrects the fixation area map C by sliding the area so that it is easier to determine that the driver is visually recognizing the first other vehicle N as shown in FIG. 2(b).

[0055] According to the visual recognition determination device 100 according to the present embodiment described above, when the correction condition is satisfied, by correcting the fixation area map C so that it is easier to determine that the driver's visual recognition range EA is visually recognizing the object, the accuracy of the driver's visual recognition determination for the object in front of the vehicle can be improved.

[0056] Furthermore, according to the visual recognition determination device 100, when the vehicle speed of the vehicle is not less than the correction prohibition threshold value (when the vehicle speed is greater than or equal to the correction prohibition threshold value) or when the traveling posture of the vehicle is not a straight-ahead posture, the correction of the fixation area map C is not performed, so that it is possible to suppress a decrease in the accuracy of the driver's visual recognition determination due to correction in inappropriate situations.

[0057] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. The present invention can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art, including the above-described embodiments.

[0058] For example, it is not necessary to satisfy all of the above-described correction conditions. The correction conditions may not include the determination of the length of the movement vector BE of the visual recognition range EA (S12 in FIG. 3). The correction conditions include at least the determination of the vehicle speed of the vehicle (S15 in FIG. 3) and the determination of the posture of the vehicle (S16 in FIG. 3).

Explanation of Reference Numerals

[0059] 1…Driver monitoring camera, 10…Visual recognition determination ECU, 11…Visual range estimation unit, 12…Object recognition unit, 13…Visual recognition determination unit, 14…Correction execution unit, 100…Visual recognition determination device, EA…Visual range, EL…Line of sight.

Claims

A visual recognition determination device having a fixation area map which is a coordinate map preset with respect to a vehicle, and correcting the fixation area map used for the driver's visual recognition determination by comparing a visual recognition range of the driver of the vehicle estimated on the fixation area map based on the line of sight of the driver of the vehicle detected by a driver monitoring camera of the vehicle with a position on the fixation area map of an object in front of the vehicle acquired by an external sensor of the vehicle, The visual recognition determination device that does not correct the fixation area map when the vehicle speed of the vehicle is equal to or higher than a correction prohibition threshold value, or when the traveling posture of the vehicle is not a straight-ahead posture. The visual recognition determination device according to claim 1, wherein when the vehicle speed of the vehicle is equal to or higher than the correction prohibition threshold value, and when the traveling posture of the vehicle is not the straight-ahead posture, the fixation area map is not corrected.

Images