Sight line calibration system

Abstract

The present application provides a line-of-sight calibration system that improves the accuracy of line-of-sight calibration in a wide area including a region deviating from the optical axis of a camera. The present application is provided with: a first display portion provided at the central portion in the vehicle width direction in front of the driver's seat, a second display portion provided on the front face of the driver's seat, a camera portion provided on the outer edge upper portion of the first display portion, a display image control portion that controls the display image displayed by the first display portion and the second display portion, a calibration data generation portion that detects the face orientation and line-of-sight direction when the driver D visually recognizes the first display portion and the face orientation and line-of-sight direction when the driver D visually recognizes the second display portion based on the image captured by the camera portion and generates calibration data, and a calibration portion that calibrates the line-of-sight direction corresponding to the face orientation that becomes the reference in the case of visually recognizing the first display portion and the line-of-sight direction corresponding to the face orientation that becomes the reference in the case of visually recognizing the second display portion based on the calibration data generated.

Description

Technical Field

[0001] This invention relates to a line-of-sight calibration system. Background Technology

[0002] In recent years, driver monitoring systems (DMS) have been continuously developed to monitor the driver's state in order to assist in safe driving of vehicles. Such DMS systems, for example, detect where the driver is looking and alert the driver if the driver is not paying attention to the current situation.

[0003] Therefore, in order to achieve such functionality, it is important to detect the driver's line of sight with high precision, and high-precision calibration processing for line of sight direction detection is required.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2009-183473

[0007] Patent Document 2: Japanese Patent Application Publication No. 2019-46240 Summary of the Invention

[0008] Technical issues

[0009] However, in the conventional line-of-sight calibration process described in Patent Document 1, since the line-of-sight calibration process is performed based on line-of-sight calibration data obtained in a target, there is a problem that the line-of-sight calibration is not accurate enough in areas that deviate from the optical axis of the camera.

[0010] Furthermore, even when multiple targets are set as described in Patent Document 2, the accuracy of wide-area line-of-sight calibration, including areas deviating from the camera's optical axis, is not improved because these multiple targets are close together.

[0011] Therefore, the present invention was made in view of the above-mentioned problems, and its object is to provide a line-of-sight calibration system that improves the accuracy of line-of-sight calibration in a wide area including regions deviating from the optical axis of the camera.

[0012] Technical solution

[0013] Method 1: One or more embodiments of the present invention provide a gaze calibration system, characterized by comprising: a first display unit disposed in the center of the driver's seat in the vehicle width direction; a second display unit disposed in front of the driver's seat; a camera unit disposed on the upper outer edge of the first display unit; a display image control unit that controls the display images displayed by the first display unit and the second display unit; a calibration data generation unit that detects the driver's facial orientation and gaze direction when visually recognizing the first display unit and the driver's facial orientation and gaze direction when visually recognizing the second display unit based on the image captured by the camera unit, and generates calibration data; and a calibration unit that calibrates the gaze direction corresponding to the reference facial orientation when visually recognizing the first display unit and the gaze direction corresponding to the reference facial orientation when visually recognizing the second display unit based on the calibration data generated in the calibration data generation unit.

[0014] Method 2: One or more embodiments of the present invention provide a line-of-sight calibration system, characterized in that the first display unit, the shooting unit, the display image control unit, the calibration data generation unit, and the calibration unit are disposed in a driving state monitoring device for monitoring the driver.

[0015] Method 3: One or more embodiments of the present invention provide a gaze calibration system, characterized in that, after the driver's facial image for personal registration is acquired using the driving state monitoring device, the display image control unit displays the image on the second display unit.

[0016] Method 4: One or more embodiments of the present invention provide a gaze calibration system, characterized in that the display image control unit causes the second display unit to display a dynamic image.

[0017] Method 5: One or more embodiments of the present invention provide a gaze calibration system, characterized in that the display image control unit displays a continuous dynamic image of a marker that is sequentially shifted for the driver's visual recognition on the entire area of ​​the second display unit.

[0018] Method 6: One or more embodiments of the present invention provide a line-of-sight calibration system, characterized in that the line-of-sight calibration system includes a data storage unit, which stores calibration data generated in the calibration data generation unit, and the calibration unit performs calibration when a predetermined amount of calibration data is stored in the data storage unit.

[0019] Method 7: One or more embodiments of the present invention provide a gaze calibration system, characterized in that the second display unit is composed of a head-up display.

[0020] Invention Effects

[0021] According to one or more embodiments of the present invention, the accuracy of line-of-sight calibration can be improved over a wide area, including regions deviating from the optical axis of the camera. Attached Figure Description

[0022] Figure 1 A diagram illustrating the configuration of the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0023] Figure 2 This diagram illustrates the configuration of the first display unit 20 and the second display unit 200 in the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0024] Figure 3 This is a graph illustrating the relationship between the actual driver's facial orientation in the first display unit 20 of the gaze calibration system 1 according to an embodiment of the present invention and the driver's facial orientation and gaze direction detected from the captured image.

[0025] Figure 4 This is a graph illustrating the relationship between the actual driver's facial orientation in the second display unit 200 of the gaze calibration system 1 according to an embodiment of the present invention and the driver's facial orientation and gaze direction detected from the captured image.

[0026] Figure 5 The diagram illustrates a dynamic image displayed on the second display unit 200 in the gaze calibration system 1 according to an embodiment of the present invention.

[0027] Figure 6 The diagram illustrates a dynamic image displayed on the second display unit 200 in the gaze calibration system 1 according to an embodiment of the present invention.

[0028] Figure 7 The diagram illustrates a dynamic image displayed on the second display unit 200 in the gaze calibration system 1 according to an embodiment of the present invention.

[0029] Figure 8 The diagram illustrates a dynamic image displayed on the second display unit 200 in the gaze calibration system 1 according to an embodiment of the present invention.

[0030] Figure 9 The diagram illustrates a dynamic image displayed on the second display unit 200 in the gaze calibration system 1 according to an embodiment of the present invention.

[0031] Figure 10This is a flowchart illustrating the main processing flow in the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0032] Figure 11 This diagram illustrates the data collection and processing flow in the first display unit 20 of the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0033] Figure 12 This diagram illustrates the data collection and processing flow in the second display unit 200 of the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0034] Figure 13 This is a diagram illustrating the calibration process in the line-of-sight calibration system 1 according to an embodiment of the present invention.

[0035] Figure 14 The image shown is an example of an image displayed on the first display unit 20 in a modified embodiment of the present invention.

[0036] Symbol Explanation

[0037] 1: Eye alignment system

[0038] 10: Filming Department

[0039] 20: First Display Section

[0040] 30: CPU

[0041] 31: Calibration Data Generation Department

[0042] 32: Calibration Department

[0043] 33: Data Storage Department

[0044] 34: Display Image Control Unit

[0045] 100: Driving Status Monitoring Device

[0046] 200: Second Display Unit Detailed Implementation

[0047] <Implementation Method>

[0048] use Figures 1 to 13 The line-of-sight calibration system 1 of this embodiment will be described.

[0049] <Composition of the line-of-sight calibration system 1>

[0050] like Figure 1 As shown, the line-of-sight calibration system 1 of this embodiment is configured to include a driving state monitoring device 100 and a second display unit 200.

[0051] like Figure 2As shown, the driving state monitoring device 100 is a device that is disposed in the center of the vehicle width direction in front of the driver's seat, and acquires the driver's facial image through the camera unit 10 described later, extracts feature points of the acquired facial image, and monitors the driver's driving state based on the state of the extracted feature points (facial orientation and / or eye opening, etc.).

[0052] It should be noted that a detailed description of the composition will be provided separately.

[0053] like Figure 2 As shown, the second display unit 200 is a display unit provided in front of the driver's seat, such as a head-up display (HUD) that can display basic information images facing the driver on the windshield.

[0054] It should be noted that, in Figure 2 The example shown is a HUD projected onto the windshield, but it could also be a HUD projected onto a combiner located on the dashboard.

[0055] Furthermore, in this embodiment, it is not limited to HUD, but also includes a display unit that presents an image in front of the driver's seat.

[0056] <Composition of the Driving Status Monitoring Device 100>

[0057] like Figure 1 As shown, the driving status monitoring device 100 is configured to include a camera unit 10, a first display unit 20, and a CPU 30.

[0058] The imaging unit 10 has built-in imaging elements such as CCD (Charge Coupled Device) or CIS (CMOS Image Sensor) and outputs images of the driver inside the vehicle captured by the imaging elements.

[0059] like Figure 2 As shown, the camera unit 10 is located on the upper outer edge of the first display unit 20, which will be described later, and acquires an image of the driver's face.

[0060] The imaging unit 10 is, for example, a near-infrared camera or an RGB camera, which captures the driver's face by receiving reflected light from the near-infrared light that has been irradiated. The imaging unit 10 then sends the captured image signal to the CPU 30, which will be described later.

[0061] like Figure 2 As shown, the first display unit 20 is, for example, a liquid crystal display located in the center of the vehicle width direction in front of the driver's seat, for example, displaying information related to the driver's personal registration, etc.

[0062] It should be noted that the first display unit 20 may also serve as the display unit of a navigation system mounted on a vehicle.

[0063] The CPU 30 includes a known CPU, RAM, ROM, I / O bus, etc. (not shown), and controls the entire driving state monitoring device 100 according to the control program stored in the ROM.

[0064] In addition, in the present embodiment, particularly, the display images of the first display unit 20 and the second display unit 200 are controlled. In addition, the CPU 30 detects the facial orientation and line-of-sight direction of the driver based on the facial image of the driver input from the imaging unit 10. In addition, the CPU 30 performs a line-of-sight calibration process based on the information related to the detected facial orientation and line-of-sight direction of the driver.

[0065] It should be noted that the detailed information on the configuration of the CPU 30 will be described later.

[0066] <Configuration of CPU 30>

[0067] As Figure 1 shown, the CPU 30 is configured to include: a calibration data generation unit 31, a calibration unit 32, a data storage unit 33, and a display image control unit 34.

[0068] The calibration data generation unit 31 detects the facial orientation and line-of-sight direction when the driver visually recognizes the first display unit 20 and the facial orientation and line-of-sight direction when the driver visually recognizes the second display unit 200 from the images captured by the imaging unit 10, and generates calibration data.

[0069] The calibration data generation unit 31 detects the facial orientation of the driver by analyzing the captured image of the driver taken by the imaging unit 10.

[0070] It should be noted that as a method for detecting the facial orientation, a known method can be used.

[0071] In addition, the calibration data generation unit 31 detects the line-of-sight direction of the driver by analyzing the captured image of the driver taken by the imaging unit 10.

[0072] It should be noted that as a method for detecting the line-of-sight direction, the line-of-sight direction can be detected from the detected facial orientation, or a method of detecting the line-of-sight direction based on the position of a moving point relative to a reference point by setting the reference point as the inner canthus and the moving point as the center of the iris (inner canthus - iris method) can be used, or a method of detecting the line-of-sight direction based on the position of a moving point relative to a reference point by setting the reference point as the corneal reflection point (Purkinje image) and the moving point as the center of the pupil (pupil - corneal reflection method) can be used.

[0073] For example, calibration data generation unit 31, such as Figure 3 , Figure 4 The calibration data is generated as shown.

[0074] here, Figure 3 (A) Figure 4 (A) is a diagram showing the view from the driver's head at an angle. Figure 3 (A) shows the driver's view of the first display unit 20. Figure 4 (A) shows the driver observing the second display unit 200. In the figure, D represents the driver, M represents the marker, the dotted line represents the orientation of the driver D's face relative to the marker M when looking directly at the marker M (the actual face orientation), the solid line represents the orientation of the driver D's face detected from the image when observing the marker M, and the dashed line represents the direction of the driver D's gaze detected from the image when observing the marker M.

[0075] exist Figure 3 In (B), the straight line corresponding to the direction of driver D's face is ( Figure 3 The intersection of the solid line (A) and the display surface of the first display unit 20 is set as F, and the straight line corresponding to the line of sight of the driver D at this time is ( Figure 3 The intersection of the dashed line (A) with the display surface of the first display unit 20 is shown as RE (right eye) and LE (left eye).

[0076] It should be noted that in this embodiment, since the driver visually recognizes the first display unit 20 from an oblique direction, F, RE, and LE are in an oblique straight line shape relative to M, but... Figure 3 In the diagram, for ease of understanding, F, RE, and LE are shown in a vertical arrangement relative to M.

[0077] The position information of F, RE, and LE in the display surface can be obtained by using the coordinates of the display surface of the first display unit 20.

[0078] Based on the location information, the calibration data generation unit 31 calculates the error (Δ11) of the driver's facial orientation, the error (Δ12) of the right eye, and the error (Δ13) of the left eye when the driver is observing the first display unit 20.

[0079] Similarly, the calibration data generation unit 31 calculates the error (Δ21) of the driver's facial orientation, the error of the right eye (Δ22), and the error of the left eye (Δ23) when the driver is observing the second display unit 200.

[0080] The calibration data generation unit 31 generates, based on the error data, the facial orientation gain and gaze direction gain of the driver's D vision recognition first display unit 20 and the facial orientation gain and gaze direction gain of the driver's D vision recognition second display unit 200 as calibration data.

[0081] It should be noted that the calibration data generated in the calibration data generation unit 31 is stored in the data storage unit 33, which will be described later.

[0082] The calibration unit 32 performs calibration based on the calibration data generated in the calibration data generation unit 31, on the gaze direction corresponding to the reference facial orientation when the driver recognizes the first display unit 20 and on the gaze direction corresponding to the reference facial orientation when the driver recognizes the second display unit 200.

[0083] The calibration unit 32 performs calibration at a time when the data storage unit 33, described later, stores a predetermined amount of calibration data.

[0084] The data storage unit 33 stores the calibration data generated in the calibration data generation unit 31.

[0085] In addition, the data storage unit 33 stores the display data used by the display image control unit 34, which will be described later.

[0086] The display image control unit 34 controls the display images displayed on the first display unit 20 and the second display unit 200.

[0087] The display image control unit 34 causes, for example, the first display unit 20 to display a personal registration screen.

[0088] Furthermore, the image display control unit 34, for example, causes the second display unit 200 to display an image after the CPU 30 acquires a facial image of the driver for personal registration of driver D, based on a control signal from the CPU 30.

[0089] In addition, the display image control unit 34 causes the second display unit 200 to display a dynamic image.

[0090] It should be noted that the moving image is preferably one that uniquely and statically represents the driver D's actions. For example, a moving image could be a continuous moving image in which markers that the driver D can visually recognize are sequentially shifted.

[0091] As a series of dynamic images displayed to sequentially transfer markers for driver D's visual recognition, it is possible to illustrate Figures 5 to 9 The dynamic image shown.

[0092] For example, Figure 5(A) is a diagram in which marker M is displayed alternately from left to right in a straight line along a horizontal column, then the direction is changed and the display proceeds from right to left. Figure 5 (B) is a diagram in which marker M is displayed alternately from top to bottom in a straight line along a vertical column, then changes direction and is displayed from bottom to top.

[0093] For example, Figure 6 (A) is a diagram in which marker M is displayed alternately from the top left to the bottom right in a straight line along a diagonal column, then the direction is changed and the display moves from the bottom right to the top left. Figure 6 (B) is a diagram in which marker M is displayed alternately from the upper right to the lower left in a straight line along a diagonal column, and then changes direction to be displayed from the lower left to the upper right.

[0094] For example, Figure 7 (A) is a diagram in which multiple columns of markers M are displayed alternately from left to right in a straight line along the horizontal direction, then move to the next column and change direction, displaying from right to left. Figure 7 (B) is a diagram in which multiple columns of marker M are displayed alternately from top to bottom in a straight line along the vertical direction, then move to the adjacent column on the right and change orientation, displaying from bottom to top.

[0095] For example, Figure 8 This is a diagram with multiple columns arranged and the marker M shifting diagonally from the top left to the bottom right. It should be noted that the shifting direction can also be set as from the bottom right to the top left, from the bottom left to the top right, from the top right to the bottom left, from the top left to the bottom left, from the bottom left to the top left, from the top right to the bottom right, and from the bottom right to the top right.

[0096] For example, Figure 9 This is a diagram with multiple columns arranged and the marker M moving sequentially in a zigzag pattern from the top left to the bottom left. It should be noted that the direction of the movement can also be set as from the bottom left to the top left, from the top right to the bottom right, from the top left to the bottom right, from the top left to the top right, from the top right to the bottom left, from the bottom right to the bottom left, and from the bottom right to the top right.

[0097] <Processing of line-of-sight calibration system 1>

[0098] use Figures 10 to 13 The processing of the line-of-sight calibration system 1 in this embodiment will be described.

[0099] <Main Processor of Line-of-Sight Calibration System 1>

[0100] like Figure 10As shown, the main processing of the line-of-sight calibration system 1 includes: calibration data collection processing from the first display unit 20 (step S100), calibration data collection processing from the second display unit 200 (step S200), and calibration processing (step S300).

[0101] The following is a detailed explanation of each processing step.

[0102] <Collection data collection and processing from the first display unit 20>

[0103] use Figure 11 The collection and processing of calibration data from the first display unit 20 will be explained.

[0104] First, the display image control unit 34 displays a personal registration screen for using the driving status monitoring device 100 on the first display unit 20 based on the control signal from the CPU 30 (step S110).

[0105] At this time, the imaging unit 10 captures a facial image of the driver D based on the control signal from the CPU 30 (step S120).

[0106] If the imaging unit 10 completes the imaging, it outputs the facial image data of the driver D to the calibration data generation unit 31.

[0107] The calibration data generation unit 31 extracts the driver D's facial orientation and gaze direction from the input facial image data of the driver D.

[0108] Then, the calibration data generation unit 31 calculates the error (Δ11), the error (Δ12) of the driver D's face orientation, the direction of his gaze, and the face orientation of the driver D when visually recognizing the marker M, based on the extracted facial orientation of the driver D.

[0109] Furthermore, the calibration data generation unit 31 generates the facial orientation gain and gaze direction gain of the driver's D visual recognition first display unit 20 as calibration data based on the above error data (step S130).

[0110] Then, the calibration data generated in the calibration data generation unit 31 is stored in the data storage unit 33 (step S140).

[0111] <Calibration data collection and processing from the second display unit 200>

[0112] use Figure 12 The collection and processing of calibration data from the second display unit 200 will be explained.

[0113] First, based on the control signal from the CPU 30, the display image control unit 34 displays a moving image on the second display unit 200 at the moment when the facial image of the driver D of the visual recognition first display unit 20 is captured (step S210).

[0114] It should be noted that, as a moving image displayed on the second display unit 200, examples can be shown Figures 5 to 9 The display shown is a continuous dynamic image of markers that are sequentially shifted for the driver's visual recognition.

[0115] At this time, the imaging unit 10 captures a facial image of the driver D who is looking at the dynamic image displayed on the second display unit 200 based on the control signal from the CPU 30 (step S220).

[0116] If the shooting unit 10 completes the shooting, it outputs the facial image data of the driver D to the calibration data generation unit 31.

[0117] The calibration data generation unit 31 extracts the driver D's facial orientation and gaze direction from the input facial image data of the driver D.

[0118] Then, the calibration data generation unit 31 calculates the error (Δ21), the error (Δ22) of the driver D's facial orientation, the direction of his gaze, and the facial orientation of the driver D when visually recognizing the mark M, based on the extracted facial orientation of the driver D.

[0119] Furthermore, the calibration data generation unit 31 generates facial orientation gain and gaze direction gain when the driver's D vision recognition second display unit 200 is used as calibration data based on the above error data (step S230).

[0120] Then, the calibration data generated in the calibration data generation unit 31 is stored in the data storage unit 33 (step S240).

[0121] <Calibration Process>

[0122] use Figure 13 The calibration process is explained.

[0123] CPU30 determines whether a predetermined amount of calibration data is stored in data storage unit 33 (step S310).

[0124] At this point, if the CPU30 determines that the predetermined amount of calibration data is not stored in the data storage unit 33 (No in step S310), it switches to waiting mode.

[0125] On the other hand, if the CPU30 determines that a predetermined amount of calibration data is stored in the data storage unit 33 ("Yes" in step S310), the process is transferred to step S320.

[0126] At this time, the calibration unit 32 performs calibration based on the calibration data generated in the calibration data generation unit 31, on the gaze direction corresponding to the reference face orientation when the driver recognizes the first display unit 20 and on the gaze direction corresponding to the reference face orientation when the driver recognizes the second display unit 200 (step S320).

[0127] <Functions and Effects>

[0128] As described above, the gaze calibration system 1 of this embodiment is configured to include: a first display unit 20 disposed in the center of the vehicle width direction in front of the driver's seat; a second display unit 200 disposed in front of the driver's seat; a camera unit 10 disposed on the upper outer edge of the first display unit; a display image control unit 34 that controls the display images displayed by the first display unit 20 and the second display unit 200; a calibration data generation unit 31 that detects the driver's facial orientation and gaze direction when visually recognizing the first display unit 20 and the driver's facial orientation and gaze direction when visually recognizing the second display unit 200 based on the images captured by the camera unit 10, and generates calibration data; and a calibration unit 32 that calibrates the gaze direction corresponding to the reference facial orientation when visually recognizing the first display unit 20 and the gaze direction corresponding to the reference facial orientation when visually recognizing the second display unit 200 based on the calibration data generated in the calibration data generation unit 31.

[0129] That is, the line-of-sight calibration system 1 of this embodiment performs line-of-sight calibration in a first display unit 20 located in the center of the vehicle width direction in front of the driver's seat and a second display unit 200 located on the front of the driver's seat.

[0130] Generally, it is known that calibration using images captured near the optical axis of the imaging unit 10 (calibration based on the first display unit 20) has high accuracy, while calibration using images captured off the optical axis of the imaging unit 10 (calibration in the second display unit 200) has low accuracy.

[0131] However, in the line-of-sight calibration system 1 of this embodiment, since calibration is performed based on calibration data when the driver's D vision recognizes the first display unit 20, and calibration is performed based on calibration data when the driver's D vision recognizes the second display unit 200, the accuracy of calibration in the second display unit 200 can also be improved.

[0132] That is, since calibration data is acquired when the difference between the driver D's facial orientation and the optical axis of the imaging unit 10 is small, and calibration data is acquired when the difference between the driver D's facial orientation and the optical axis of the imaging unit 10 is large, the accuracy of the line of sight calibration can be improved in a wide area including the region deviating from the optical axis of the imaging unit 10.

[0133] Specifically, if we take Figure 2 For example, when the driver uses the air conditioning 62 and / or navigation system 61 with D vision, the calibration data obtained for the first display unit 20 is used; when the driver uses the instrument panel 63 and / or door rearview mirror 64 with D vision, the calibration data obtained for the second display unit 200 is used. This can improve the accuracy of the line of sight calibration in a wide area, including the area that is off the optical axis of the imaging unit 10.

[0134] Furthermore, in this embodiment, after the gaze calibration system 1 acquires a facial image of the driver D for personal registration using a driving state monitoring device 100 equipped with a first display unit 20, a shooting unit 10, a display image control unit 34, a calibration data generation unit 31, and a calibration unit 32, the display image control unit 34 displays the image on the second display unit 200.

[0135] That is, the calibration data used by the first display unit 20 is acquired when the personal registration in the driving status monitoring device 100 is triggered, and after the facial image of the driver D for personal registration is acquired, the display image control unit 34 displays the image on the second display unit 200 and acquires the calibration data.

[0136] Therefore, it can reduce the burden on driver D and shorten the time for obtaining calibration data.

[0137] Furthermore, in the gaze calibration system 1 of this embodiment, the display image control unit 34 causes the second display unit 200 to display a dynamic image.

[0138] That is, by displaying dynamic images on the second display unit 200, the driver's actions can be uniquely fixed, thus shortening the time until eye-tracking data useful for eye-tracking calibration is obtained.

[0139] Furthermore, by displaying dynamic images on the second display unit 200, the driver's actions can be uniquely fixed, thereby improving the accuracy of the calibration data.

[0140] Furthermore, in the gaze calibration system 1 of this embodiment, the display image control unit 34 displays a continuous dynamic image of markers that are sequentially shifted for driver D visual recognition on the entire area of ​​the second display unit 200.

[0141] Specifically, the display image control unit 34 makes Figures 5 to 9 The continuous dynamic images shown are displayed over the entire area of ​​the second display unit 200.

[0142] Therefore, it is possible to easily guide the driver's line of sight, and by suppressing deviations in the driver's actions, the time until obtaining line of sight data useful for line of sight calibration can be shortened.

[0143] Furthermore, in the line-of-sight calibration system 1 of this embodiment, the calibration unit 32 performs calibration at the moment when the data storage unit 33 stores a predetermined amount of calibration data.

[0144] That is, the calibration unit 32 performs calibration when the data storage unit 33 stores enough calibration data to perform calibration processing, thereby improving the accuracy of the line-of-sight calibration processing.

[0145] Furthermore, in the gaze calibration system 1 of this embodiment, the second display unit 200 is composed of a head-up display.

[0146] That is, since the driving status monitoring device 100 installed in the vehicle can be used to perform line-of-sight calibration, no additional device is required, and line-of-sight calibration can be performed easily.

[0147] Furthermore, since the head-up display with a large display area is set as the second display unit 200, even in the second display unit 200 which is offset from the optical axis of the imaging unit 10 which is usually formed with low precision, it is possible to improve the calibration accuracy by obtaining sufficient calibration data.

[0148] <Variation Example 1>

[0149] For example, in the above embodiment, a configuration is shown in which a calibration data generation unit 31, a calibration unit 32, and a data storage unit 33 are provided in the CPU 30. However, it can also be configured to transmit image information captured by the imaging unit 10 to a server connected to the vehicle via a network, and perform the processing of the calibration data generation unit 31 and the processing of the calibration unit 32 in the server.

[0150] This enables the rapid processing of large amounts of information.

[0151] <Variation Example 2>

[0152] Although not specifically mentioned in the above embodiments, dynamic images may also be displayed in the first display unit 20.

[0153] Specifically, the homepage of personal registration can display, for example... Figure 5The animated image shown can also be set to display along with messages between the homepage and the next screen of the personal registration page. Figures 6 to 9 The image shown is a dynamic picture.

[0154] In addition, a message such as "Please visually identify the blank space in the order of (A) to (D)" can be displayed between the first page and the next page of the personal registration. Figure 14 The image shown.

[0155] Thus, by displaying dynamic and / or still images in the first display unit 20, the driver D's line of sight is uniquely fixed, thereby improving the accuracy of the calibration data obtained when the driver D visually recognizes the first display unit 20.

[0156] It should be noted that the processing of the calibration data generation unit 31, calibration unit 32, and display image control unit 34 is recorded on a computer-readable recording medium. By having the calibration data generation unit 31, calibration unit 32, and display image control unit 34 read and execute the program recorded on the recording medium, the line-of-sight calibration system 1 of the present invention can be realized. The computer system referred to herein includes hardware such as an OS and / or peripheral devices.

[0157] Furthermore, in the case of using a WWW (World Wide Web) system, "computer system" also includes a homepage providing environment (or display environment). Additionally, the aforementioned program can be transmitted from a computer system storing the program in a storage device or similar device to other computer systems via a transmission medium or through transmission waves within the transmission medium. Here, "transmission medium" for transmitting the program refers to a medium capable of transmitting information, such as a network (communication network) like the Internet and / or a communication line (communication line) like a telephone line.

[0158] Furthermore, the aforementioned program can also be a program used to implement a portion of the functions described above. Further, it can also be a so-called difference file (difference program) capable of implementing the aforementioned functions through combination with programs already recorded in the computer system.

[0159] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the specific configuration is not limited to these embodiments and may include designs that do not depart from the spirit of the present invention.

Claims

1. A line of sight calibration system, characterized by, Possessing: a first display portion provided at a central portion in a vehicle width direction in front of a driver's seat; a second display portion provided at a front surface of the driver's seat; a photographing portion provided at an outer edge upper portion of the first display portion, for photographing a first facial image in a vicinity of an optical axis of the photographing portion when a driver visually recognizes the first display portion, and photographing a second facial image in a wide area including a region deviated from the optical axis of the photographing portion when the driver visually recognizes the second display portion; a display image control portion that controls a display image that causes the first display portion and the second display portion to display; a calibration data generation portion that detects both of a first facial orientation when the driver visually recognizes the first display portion and a first line-of-sight direction, and both of a second facial orientation when the driver visually recognizes the second display portion and a second line-of-sight direction, based on both of the first facial image and the second facial image photographed by the photographing portion, and generates calibration data based on the first facial orientation, the second facial orientation, the first line-of-sight direction, and the second line-of-sight direction; and a calibration portion that calibrates, based on the calibration data generated in the calibration data generation portion, a line-of-sight direction corresponding to a facial orientation that becomes a reference in a case where the first display portion is visually recognized, and a line-of-sight direction corresponding to a facial orientation that becomes a reference in a case where the second display portion is visually recognized.

2. The line-of-sight calibration system according to claim 1, wherein the first display portion, the photographing portion, the display image control portion, the calibration data generation portion, and the calibration portion are provided to a driving state monitoring device that monitors the driver.

3. The line-of-sight calibration system according to claim 2, wherein the display image control portion displays an image on the second display portion after the facial image of the driver for personal registration is acquired using the driving state monitoring device.

4. The line-of-sight calibration system according to claim 3, wherein the display image control portion causes the second display portion to display a dynamic image.

5. The line-of-sight calibration system according to claim 3, wherein the display image control portion causes a continuous dynamic image in which a mark that causes the driver to visually recognize is caused to shift sequentially to be displayed on an entire region of the second display portion.

6. The line-of-sight calibration system according to any one of claims 1 to 5, wherein the line-of-sight calibration system possesses a data storage portion that stores the calibration data generated in the calibration data generation portion, the calibration portion performs calibration at a timing at which a predetermined amount of calibration data is stored in the data storage portion.

7. The line-of-sight calibration system according to any one of claims 1 to 5, wherein the second display portion is constituted by a head-up display.

8. The line-of-sight calibration system according to claim 6, wherein the second display portion is constituted by a head-up display. ​

Images