Occupant state recognition device, occupant state recognition method, and occupant state recognition program

Abstract

To provide an occupant state recognition apparatus, method and program for determining a viewing state of an occupant while selecting whether to use a line-of-sight direction, in accordance with the accuracy of imaging information of an in-vehicle camera.SOLUTION: An occupant state recognition apparatus 1 which recognizes an occupant state by determining a viewing state of an occupant on a vehicle includes: an occupant information acquisition unit 11 which acquires a face direction and a line-of-sight direction of an occupant on the basis of imaging information of a driver monitor camera 2 in a vehicle; a distance acquisition unit 12 which acquires a distance from the driver monitor camera 2 to the head of the occupant, on the basis of ranging information of a ranging sensor 3 in the vehicle; and a viewing state determination unit 13 which determines a viewing state of the occupant on the basis of at least one of the distance, the face direction, and the line-of-sight direction. The viewing state determination unit 13 determines, when the distance is equal to or smaller than a predetermined far distance threshold regarding acquisition of a line-of-sight direction, a viewing state of the occupant on the basis of the line-of-sight direction, or determines, when the distance is larger than the far distance threshold, the viewing state of the occupant on the basis of the face direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an occupant state recognition device, an occupant state recognition method, and an occupant state recognition program. [Background technology]

[0002] BACKGROUND ART Conventionally, there is known a device that detects the direction of a driver's face and line of sight using an image captured by a capturing unit, and determines whether or not the driver is looking aside (for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-15549 Summary of the Invention [Problem to be solved by the invention]

[0004] In this technical field, it has been studied to determine the visual state of a vehicle occupant by more appropriately using the gaze direction of the occupant. Because the gaze direction is obtained by image processing of an image captured by an in-vehicle camera, there is room for consideration of the accuracy of the image information from the in-vehicle camera. [Means for solving the problem]

[0005] An occupant state recognition device according to one aspect of the present invention is an occupant state recognition device that recognizes the occupant state by determining the visual state of the occupant of a vehicle, and includes an occupant information acquisition unit that acquires the facial direction and gaze direction of the occupant based on image information from an in-vehicle camera of the vehicle, a distance acquisition unit that acquires the distance from the in-vehicle camera to the occupant's head based on ranging information from an in-vehicle sensor of the vehicle, and a visual state determination unit that determines the visual state of the occupant based on the distance and at least one of the facial direction and gaze direction, wherein the visual state determination unit determines the visual state of the occupant based on the gaze direction if the distance is equal to or less than a predetermined long-distance threshold for acquiring the gaze direction, and determines the visual state of the occupant based on the facial direction if the distance is greater than the long-distance threshold.

[0006] In an occupant state recognition device according to one aspect of the present invention, when the accuracy of the in-vehicle camera's image information affects the accuracy of acquiring the gaze direction due to the distance from the in-vehicle camera to the occupant's head, the occupant's visual state is determined based on the facial orientation rather than the gaze direction. Thus, the occupant state recognition device according to one aspect of the present invention can determine the occupant's visual state by switching whether or not to use the gaze direction depending on the accuracy of the in-vehicle camera's image information.

[0007] In one embodiment, the visual state determination unit may calculate a reliability of the acquired gaze direction based on the imaging information, and when the distance is greater than a long distance threshold and the reliability is equal to or greater than a predetermined reliability threshold, determine the visual state of the occupant based on the gaze direction. In this way, even if the distance from the in-vehicle camera to the occupant's head is long, if the reliability of the acquired gaze direction is equal to or greater than the reliability threshold, the visual state of the occupant is determined based on the gaze direction. Therefore, the visual state of the occupant can be determined by appropriately using the gaze direction depending on the accuracy of the imaging information from the in-vehicle camera.

[0008] In one embodiment, the visual state determination unit may determine the visual state of the occupant based on the facial direction when the distance is smaller than a predetermined short-distance threshold for acquiring the gaze direction, and may determine the visual state of the occupant based on the gaze direction when the distance is equal to or greater than the short-distance threshold. In this way, when the accuracy of the in-vehicle camera's image information affects the accuracy of acquiring the gaze direction due to the short distance from the in-vehicle camera to the occupant's head, the visual state of the occupant is determined based on the facial direction rather than the gaze direction. Therefore, the visual state of the occupant can be determined by switching whether or not to use the gaze direction depending on the accuracy of the image information from the in-vehicle camera.

[0009] Another aspect of the present invention is an occupant status recognition method that recognizes an occupant status by determining the visual state of a vehicle occupant, and includes acquiring the facial direction and gaze direction of the occupant based on image capture information from an in-vehicle camera of the vehicle, acquiring the distance from the in-vehicle camera to the occupant's head based on ranging information from an in-vehicle sensor of the vehicle, and performing a visual status determination to determine the occupant's visual status based on the distance and at least one of the facial direction and gaze direction.In the visual status determination, if the distance is equal to or less than a predetermined long-distance threshold for acquiring the gaze direction, the occupant's visual status is determined based on the gaze direction, and if the distance is greater than the long-distance threshold, the occupant's visual status is determined based on the facial direction.

[0010] In the occupant state recognition method according to another aspect of the present invention, when the accuracy of the in-vehicle camera's image information affects the accuracy of the gaze direction acquisition due to the distance from the in-vehicle camera to the occupant's head, the occupant's visual state is determined based on the facial orientation rather than the gaze direction. Thus, according to the occupant state recognition method according to another aspect of the present invention, the occupant's visual state can be determined by switching whether or not to use the gaze direction depending on the accuracy of the in-vehicle camera's image information.

[0011] An occupant status recognition program according to another aspect of the present invention is an occupant status recognition program that operates a vehicle's ECU to recognize the occupant status by determining the visual state of the vehicle occupant, and operates the ECU as an occupant information acquisition unit that acquires the facial orientation and gaze direction of the occupant based on image information from the vehicle's in-vehicle camera, a distance acquisition unit that acquires the distance from the in-vehicle camera to the occupant's head based on ranging information from the vehicle's in-vehicle sensor, and a visual status determination unit that determines the occupant's visual status based on the distance and at least one of the facial orientation and gaze direction, and the visual status determination unit determines the occupant's visual status based on the gaze direction if the distance is equal to or less than a predetermined long-distance threshold for acquiring the gaze direction, and determines the occupant's visual status based on the facial direction if the distance is greater than the long-distance threshold.

[0012] In the occupant state recognition program according to another aspect of the present invention, when the accuracy of the in-vehicle camera's image information affects the accuracy of acquiring the gaze direction because the distance from the in-vehicle camera to the occupant's head is long, the occupant's visual state is determined based on the facial orientation rather than the gaze direction. Thus, according to the occupant state recognition program according to another aspect of the present invention, the occupant's visual state can be determined by switching whether or not to use the gaze direction depending on the accuracy of the image information from the in-vehicle camera. [Effects of the Invention]

[0013] According to the present invention, whether or not to use the line of sight direction can be switched depending on the accuracy of the image information captured by the in-vehicle camera, thereby making it possible to determine the visual state of the occupant. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration of an occupant state recognition device according to an embodiment; [Figure 2] 5 is a flowchart illustrating an example of an occupant state recognition process. [Figure 3] 10 is a flowchart illustrating an example of a visual inspection state determination process. DETAILED DESCRIPTION OF THE INVENTION

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same or equivalent elements will be designated by the same reference numerals, and redundant description will be omitted.

[0016] FIG. 1 is a block diagram showing the configuration of an occupant state recognition device according to an embodiment. The occupant state recognition device 1 shown in FIG. 1 is mounted on a vehicle such as a passenger car. The occupant state recognition device 1 recognizes the occupant state by determining the visual state of a vehicle occupant (e.g., a driver) while the vehicle is traveling, for example. The occupant state recognition device 1 determines whether the occupant is looking aside based on the visual state of the occupant, and issues a warning to the occupant according to the determination result. "Looking aside" means that the driver is looking outside a predetermined range, such as straight ahead.

[0017] The occupant state recognition device 1 includes an ECU (Electronic Control Unit) 10 that controls the device in an integrated manner. The ECU 10 is an electronic control unit having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a CAN (Controller Area Network) communication circuit, and the like. The ECU 10, for example, loads a program stored in the ROM into the RAM and executes the program loaded into the RAM with the CPU, thereby realizing various functions related to occupant state recognition. The ECU 10 may be composed of multiple ECUs. Some of the functions of the ECU 10 may be executed by a server that can communicate with the vehicle.

[0018] The ECU 10 is connected to a driver monitor camera (in-vehicle camera) 2, a distance measurement sensor (in-vehicle sensor) 3, and an HMI 4.

[0019] The driver monitor camera 2 is an imaging device that captures images of occupants. The driver monitor camera 2 is, for example, provided on the cover of the steering column of the vehicle, and captures images of at least the face of the driver. Multiple driver monitor cameras 2 may be provided to capture images of the driver from multiple directions. The driver monitor camera 2 may also capture images of the faces of occupants other than the driver. The driver monitor camera 2 transmits image information of the occupants to the ECU 10.

[0020] The distance measurement sensor 3 is a distance measurement device that acquires the distance from the driver monitor camera 2 to the head of the occupant. The distance measurement sensor 3 may be, for example, a TOF (Time of Flight) sensor that uses sound waves or radio waves. If the driver monitor camera 2 has a distance measurement function, the distance measurement sensor 3 may be the driver monitor camera 2. The distance measurement sensor 3 is provided, for example, on a cover of the steering column of the vehicle. The distance measurement sensor 3 transmits distance measurement information for acquiring the distance from the driver monitor camera 2 to the head of the occupant to the ECU 10.

[0021] The HMI 4 is an interface for outputting information from at least the occupant state recognition device 1 to the occupant. The HMI 4 includes output units such as a display and a speaker. The HMI 4 outputs images on the display or sounds from the speaker in response to a control signal from the ECU 10. The HMI 4 notifies the occupant, for example, by issuing an alarm, a warning, or the like, using images or sounds.

[0022] Next, a description will be given of the functional configuration of the ECU 10. The ECU 10 has an occupant information acquisition unit 11, a distance acquisition unit 12, a visual state determination unit 13, and a notification unit .

[0023] The occupant information acquisition unit 11 acquires occupant information including the facial orientation and line of sight direction of the occupant, and information as to whether or not these can be acquired, based on the image information captured by the driver monitor camera 2.

[0024] The occupant information acquisition unit 11 acquires the facial orientation of the occupant using an image of the occupant captured by the driver monitor camera 2 of the vehicle. The "facial orientation" refers to the direction in which the occupant's face is facing. The facial orientation of the occupant can be acquired by a known image processing method such as pattern matching, for example, by using the arrangement of the eyes, nose, and mouth in an area corresponding to the occupant's face in the image captured by the driver monitor camera 2. The occupant information acquisition unit 11 may acquire angle information of the facial orientation for each of the yaw angle direction and the pitch angle direction.

[0025] The occupant information acquisition unit 11 acquires the gaze direction of the occupant using an image captured by the driver monitor camera 2 of the vehicle. "Gaze direction" means the direction in which the occupant's gaze is directed. The gaze direction of the occupant can be acquired by a known image processing method, for example, using the relative positional relationship of multiple points set with respect to the occupant's eyes in the image captured by the driver monitor camera 2. The multiple points may include a reference point set at the inner corner of the eye, etc., and a moving point set at the iris, etc. The reference point is a point that serves as a reference for the relative position of the moving point. The occupant information acquisition unit 11 may acquire angle information of the gaze direction for each of the yaw angle direction and the pitch angle direction.

[0026] The distance acquisition unit 12 acquires the distance from the driver monitor camera 2 to the occupant's head based on the distance measurement information from the distance measurement sensor 3. The distance acquisition unit 12 acquires the distance from the driver monitor camera 2 to the occupant's head based on, for example, the relative distance measured from the distance measurement sensor 3 to the occupant's head, the mounting position of the driver monitor camera 2, and the mounting position of the distance measurement sensor 3. The "distance to the occupant's head" is one of the indices used to determine whether it is appropriate to use the occupant's line of sight when determining the occupant's visual state.

[0027] The "distance to the occupant's head" may be the distance from the driver monitor camera 2 to the occupant's face. The "distance to the face" may include the distance to a part of the face. The "distance to a part of the face" may be, for example, the distance to the eyes, nose, chin, forehead, cheeks, and mouth, or the distance to a predetermined position sandwiched between two or more of these parts. Furthermore, the "distance to the occupant's head" may be the distance from the driver monitor camera 2 to the occupant's hair. The "distance to the occupant's head" may include the distance from the driver monitor camera 2 to the occupant's neck.

[0028] The visual state determination unit 13 recognizes the occupant state by determining the occupant's visual state based on the distance from the driver monitor camera 2 to the occupant's head and at least one of the facial orientation and gaze direction. The occupant state includes, for example, the occupant's visual state and a face-lost state of the occupant. The occupant's visual state may include a frontal determination of whether the occupant is facing forward. The face-lost state is a state in which the occupant's visual state cannot be obtained from the image captured by the driver monitor camera 2 of the vehicle.

[0029] When the facial direction can be acquired and the gaze direction can be acquired, the visual state determination unit 13 performs visual state determination to determine the visual state of the occupant based on the facial direction or gaze direction. The visual state determination unit 13 determines whether or not it is possible to acquire the facial direction based on the captured image. When the facial direction of the occupant can be acquired, the visual state determination unit 13 determines that the facial direction can be acquired. When the facial direction of the occupant cannot be acquired, the visual state determination unit 13 determines that the facial direction cannot be acquired. The visual state determination unit 13 determines whether or not it is possible to acquire the gaze direction based on the captured image. When the gaze direction of the occupant can be acquired, the visual state determination unit 13 determines that the gaze direction can be acquired. When the gaze direction of the occupant cannot be acquired, the visual state determination unit 13 determines that the gaze direction cannot be acquired.

[0030] The visual state determination unit 13 determines, for example, whether the distance from the driver monitor camera 2 to the occupant's head is equal to or less than a predetermined long-distance threshold for acquiring the gaze direction. The long-distance threshold is a threshold for the distance from the driver monitor camera 2 to the occupant's head, and is used to determine whether the occupant's gaze direction becomes inappropriate for determining the visual state as the occupant moves away from the driver monitor camera 2. The long-distance threshold can be, for example, a threshold such that the positional accuracy of the reference point and moving point set for the occupant's eyes in the captured image becomes equal to or less than a certain level as the occupant moves away from the driver monitor camera 2. The long-distance threshold may be, for example, a parameter that is predetermined depending on the specifications of the driver monitor camera 2. The long-distance threshold may be, for example, a parameter that varies depending on the brightness inside the vehicle.

[0031] When the distance from the driver monitor camera 2 to the occupant's head is equal to or less than the long distance threshold, the visual state determination unit 13 may determine the visual state of the occupant based on the gaze direction, assuming that the gaze direction is available in accordance with the accuracy of the image information of the driver monitor camera 2. When the distance from the driver monitor camera 2 to the occupant's head is greater than the long distance threshold, the visual state determination unit 13 may determine that the gaze direction is unavailable in accordance with the accuracy of the image information of the driver monitor camera 2, and may determine the visual state of the occupant based on the facial direction.

[0032] The visual state determination unit 13 determines, for example, whether the distance from the driver monitor camera 2 to the occupant's head is equal to or greater than a predetermined short-distance threshold for acquiring the gaze direction. The short-distance threshold is a threshold for the distance from the driver monitor camera 2 to the occupant's head, used to determine whether the occupant's approach to the driver monitor camera 2 makes it inappropriate to use the occupant's gaze direction for determining the visual state. The short-distance threshold can be, for example, a threshold such that the positional accuracy of the reference point and moving point set for the occupant's eyes in the captured image becomes equal to or less than a certain level as the occupant approaches the driver monitor camera 2. The short-distance threshold may be, for example, a parameter determined in advance depending on the specifications of the driver monitor camera 2. The short-distance threshold may be, for example, a parameter that varies depending on the brightness inside the vehicle.

[0033] When the distance from the driver monitor camera 2 to the occupant's head is shorter than the short distance threshold, the visual state determination unit 13 may determine the visual state of the occupant based on the facial direction, assuming that the gaze direction is unavailable in accordance with the accuracy of the imaging information of the driver monitor camera 2. When the distance from the driver monitor camera 2 to the occupant's head is equal to or greater than the short distance threshold, the visual state determination unit 13 may determine the visual state of the occupant based on the gaze direction, assuming that the gaze direction is available in accordance with the accuracy of the imaging information of the driver monitor camera 2.

[0034] The visual state determination unit 13 may calculate the reliability of the acquired gaze direction based on the image capture information. "Reliability" refers to an index that indicates whether the gaze direction can be acquired from the image captured by the driver monitor camera 2 with enough accuracy to appropriately determine the visual state of the occupant. "Reliability" is one of the indices used to determine whether it is appropriate to use the gaze direction of the occupant when determining the visual state of the occupant. The visual state determination unit 13 calculates the reliability according to, for example, the image quality of the image captured by the driver monitor camera 2. The reliability may be, for example, the degree of degradation (degree of blur) of the image quality of the captured image. The degree of degradation of the image quality can be calculated using, for example, a known technique used for camera autofocus or the like. The reliability may also be calculated according to the brightness, contrast, etc. of the captured image.

[0035] The visual state determination unit 13 may determine whether the reliability is equal to or greater than a predetermined reliability threshold. When the distance from the driver monitor camera 2 to the occupant's head is greater than a long distance threshold and the reliability is equal to or greater than the predetermined reliability threshold, the visual state determination unit 13 may determine the occupant's visual state based on the line of sight.

[0036] For example, the visual state determination unit 13 may determine that the reliability is equal to or greater than the reliability threshold when the occupant's face direction is detected as being forward and both of the occupant's eyes are detected. The forward direction refers to the face direction when the occupant's face is facing the driver monitor camera 2. For example, the visual state determination unit 13 may determine that the reliability is less than the reliability threshold when the occupant's face direction is detected as being sideways and one of the occupant's eyes is detected. For example, the visual state determination unit 13 may determine that the reliability is less than the reliability threshold when the captured image contains a certain amount of noise. For example, the visual state determination unit 13 may determine that the reliability is less than the reliability threshold when the occupant is wearing glasses and the eyes are detected through the glasses. This is because the eyes may not be detected through the glasses depending on the degree of reflection on the glasses.

[0037] The visual state determination unit 13 may determine whether the driver is looking aside using a known method based on the acquired visual state of the occupant. The visual state determination unit 13 may determine that the driver is looking aside when the direction of the driver's face is deviated in the yaw angle direction or the pitch angle direction by a predetermined angle or more from the direction directly ahead of the driver. The visual state determination unit 13 may determine that the driver is looking aside when the direction of the driver's face is deviated in the yaw angle direction or the pitch angle direction by a predetermined angle or more from the direction directly ahead of the driver for a predetermined period of time.

[0038] The notification unit 14 outputs the determination result of the visual state determination unit 13. The notification unit 14 may notify the occupant by outputting the determination result of the visual state of the occupant determined by the visual state determination unit 13 to the HMI 4. For example, when the visual state determination result indicates that the driver is looking aside, the notification unit 14 outputs a control signal to the HMI 4 to issue an alarm or a warning to the driver. The alarm may be a warning in a more emphasized manner than a warning. For example, the alarm may be an alarm sound or an alarm announcement output as audio from a speaker of the HMI 4. The alarm may be an alarm message displayed as an image on a display of the HMI 4. The warning may be an attention warning announcement output as audio from a speaker of the HMI 4. The warning may be an attention warning message displayed as an image on a display of the HMI 4.

[0039] The results of the determination of the occupant's visual state may be used for determination by other systems such as an autonomous driving system.

[0040] [Example of ECU processing] Next, an example of the arithmetic processing (occupant state recognition method and occupant state recognition program) by the ECU 10 will be described. Fig. 2 is a flowchart showing an example of the occupant state recognition processing. The processing shown in Fig. 2 is repeatedly performed at a predetermined interval while the vehicle is running, for example.

[0041] 2, in step S01, the ECU 10 acquires occupant information using the occupant information acquisition unit 11. The occupant information acquisition unit 11 acquires occupant information including the facial orientation and line of sight of the occupant and information on whether or not these can be acquired, using an image of the occupant captured by the driver monitor camera 2 of the vehicle.

[0042] In step S02, the ECU 10 determines whether or not it is possible to acquire the facial direction of the occupant using the visual state determination unit 13. The visual state determination unit 13 determines whether or not it is possible to acquire the facial direction of the occupant based on the captured image. If the visual state determination unit 13 determines that it is possible to acquire the facial direction (step S02: YES), the ECU 10 proceeds to step S03.

[0043] In step S03, the ECU 10 determines the visual state (visual state determination) using the visual state determination unit 13. The visual state determination unit 13 specifically performs the process shown in FIG.

[0044] 3 is a flowchart showing an example of a visual state determination process. As shown in FIG. 3, in step S11, the ECU 10 determines whether or not it is possible to acquire the line-of-sight direction of the occupant using the visual state determination unit 13. The visual state determination unit 13 determines whether or not it is possible to acquire the line-of-sight direction of the occupant based on the captured image.

[0045] If the visual state determination unit 13 determines that it is possible to acquire the line of sight direction (step S11: YES), the ECU 10 proceeds to step S12. In step S12, the ECU 10 acquires the distance to the occupant's head using the distance acquisition unit 12. The distance acquisition unit 12 acquires the distance from the distance measurement sensor 3 to the occupant's head based on the distance measurement information from the distance measurement sensor 3 of the vehicle.

[0046] In step S13, the ECU 10 calculates the reliability of the acquired gaze direction using the visual observation state determination unit 13. The visual observation state determination unit 13 calculates the reliability of the acquired gaze direction using an image captured by the driver monitor camera 2 of the vehicle.

[0047] In step S14, the ECU 10 determines whether the distance to the occupant's head is equal to or greater than the short-distance threshold using the visual state determination unit 13. The visual state determination unit 13 determines whether the distance to the occupant's head is equal to or greater than the short-distance threshold, for example, based on the distance acquired by the distance acquisition unit 12 and a preset short-distance threshold.

[0048] If the visual state determination unit 13 determines that the distance to the occupant's head is equal to or greater than the short distance threshold (step S14: YES), the ECU 10 proceeds to step S15. In step S15, the ECU 10 determines whether the distance to the occupant's head is equal to or less than the long distance threshold using the visual state determination unit 13. The visual state determination unit 13 determines whether the distance to the occupant's head is equal to or less than the long distance threshold, for example, based on the distance acquired by the distance acquisition unit 12 and a preset long distance threshold.

[0049] If the visual state determination unit 13 determines that the distance to the occupant's head is equal to or less than the long distance threshold (step S15: YES), the ECU 10 proceeds to step S17. If the visual state determination unit 13 determines that the distance to the occupant's head is not equal to or less than the long distance threshold (step S15: NO), the ECU 10 proceeds to step S16. In step S16, the ECU 10 determines whether the reliability is equal to or greater than the reliability threshold using the visual state determination unit 13. The visual state determination unit 13 determines whether the reliability is equal to or greater than the reliability threshold, for example, based on the calculated reliability and a preset reliability threshold. If the visual state determination unit 13 determines that the reliability is equal to or greater than the reliability threshold (step S16: YES), the ECU 10 proceeds to step S17.

[0050] In step S17, the ECU 10 causes the visual state determination unit 13 to determine the visual state based on the line of sight. The visual state determination unit 13 determines the visual state of the occupant based on the acquired line of sight. After that, the ECU 10 ends the process in FIG. 3 and proceeds to step S05 in FIG. 2.

[0051] On the other hand, if the visual state determination unit 13 determines that it is not possible to obtain the gaze direction (step S11: NO), if the visual state determination unit 13 determines that the distance to the occupant's head is not greater than the short distance threshold (step S14: NO), or if the visual state determination unit 13 determines that the reliability is not greater than the reliability threshold (step S16: NO), the ECU 10 proceeds to step S18.

[0052] In step S18, the ECU 10 determines the visual state based on the facial direction using the visual state determination unit 13. The visual state determination unit 13 determines the visual state of the occupant based on the acquired facial direction. After that, the ECU 10 ends the process in FIG. 3 and proceeds to step S05 in FIG. 2.

[0053] 2, when the visual state determination unit 13 determines that it is not possible to acquire the face direction (step S02: NO), the ECU 10 proceeds to step S04. In step S04, the ECU 10 causes the visual state determination unit 13 to determine that face has been lost. The visual state determination unit 13 determines that the occupant's face cannot be acquired from the image captured by the driver monitor camera 2 of the vehicle, and determines that face has been lost. Thereafter, the ECU 10 proceeds to step S05.

[0054] In step S05, the ECU 10 outputs the determination result by the notification unit 14. The notification unit 14 notifies the occupant of the determination result determined by the visual condition determination unit 13 by outputting it to the HMI 4. Thereafter, the ECU 10 ends the processing of FIG. 2.

[0055] [Occupant Status Recognition Program] The occupant state recognition program causes the ECU 10 to function (operate) as the above-mentioned occupant information acquisition unit 11, distance acquisition unit 12, visual state determination unit 13, and notification unit 14. The occupant state recognition program is provided by a non-transitory recording medium such as a ROM or a semiconductor memory. Alternatively, the occupant state recognition program may be provided via communication such as a network.

[0056] According to the occupant state recognition device 1, occupant state recognition method, and occupant state recognition program described above, when the distance from the driver monitor camera 2 to the occupant's head is so great that the accuracy of the imaging information from the driver monitor camera 2 affects the accuracy of obtaining the gaze direction, the occupant's visual state is determined based on the facial direction rather than the gaze direction. Therefore, according to the occupant state recognition device 1, occupant state recognition method, and occupant state recognition program, it is possible to determine the occupant's visual state by switching whether or not to use the gaze direction depending on the accuracy of the imaging information from the driver monitor camera 2.

[0057] The reliability of the gaze direction acquisition is calculated based on the imaging information, and when the distance from the driver monitor camera 2 to the occupant's head is greater than a long distance threshold and the reliability is equal to or greater than a predetermined reliability threshold, the occupant's visual state is determined based on the gaze direction. In this way, even if the distance from the driver monitor camera 2 to the occupant's head is long, if the reliability of the gaze direction acquisition is equal to or greater than the reliability threshold, the occupant's visual state is determined based on the gaze direction. Therefore, the visual state can be determined by appropriately using the gaze direction depending on the accuracy of the imaging information from the driver monitor camera 2.

[0058] When the distance from the driver monitor camera 2 to the occupant's head is shorter than the short distance threshold, the occupant's visual state is determined based on the facial direction, and when the distance is equal to or greater than the short distance threshold, the occupant's visual state is determined based on the gaze direction. In this way, when the accuracy of the imaging information from the driver monitor camera 2 affects the accuracy of obtaining the gaze direction due to the short distance from the driver monitor camera 2 to the occupant's head, the occupant's visual state is determined based on the facial direction rather than the gaze direction. Therefore, the occupant's visual state can be determined by switching whether or not to use the gaze direction depending on the accuracy of the imaging information from the driver monitor camera 2.

[0059] [Variations] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. The present invention can be embodied in various forms, including the above-described embodiments, with various modifications and improvements made based on the knowledge of those skilled in the art.

[0060] In the above embodiment, the visual observation state determination unit 13 calculates the reliability of the acquired gaze direction based on the imaging information, but the calculation of the reliability and the determination of the visual observation state using the reliability are not essential and may be omitted. In this case, steps S13 and S16 in FIG. 3 may be omitted.

[0061] In the above embodiment, the visual state determination unit 13 determines the visual state of the occupant based on the comparison result between a predetermined short-distance threshold for acquiring the line-of-sight direction and the distance from the driver monitor camera 2 to the head of the occupant. However, determining the visual state using the short-distance threshold is not essential and may be omitted. In this case, step S14 in FIG. 3 may be omitted. Alternatively, instead of determining the visual state using the long-distance threshold, the visual state may be determined using the short-distance threshold. In this case, step S15 in FIG. 3 may be omitted. [Explanation of symbols]

[0062] 1...occupant state recognition device, 2...driver monitor camera (in-vehicle camera), 3...distance measurement sensor (in-vehicle sensor), 10...ECU, 11...occupant information acquisition unit, 12...distance acquisition unit, 13...visual state determination unit.

Claims

1. An occupant state recognition device that recognizes an occupant state by determining a visual state of a vehicle occupant, an occupant information acquisition unit that acquires a facial orientation and a line of sight direction of the occupant based on image information captured by an in-vehicle camera of the vehicle; a distance acquisition unit that acquires a distance from the in-vehicle camera to the head of the occupant based on distance measurement information from a distance measurement sensor of the vehicle; a visual state determination unit that determines a visual state of the occupant based on the distance and at least one of the facial orientation and the line of sight, The visual inspection state determination unit If the distance is equal to or less than a predetermined long distance threshold for acquiring the gaze direction, determining a visual observation state of the occupant based on the gaze direction; If the distance is greater than the long distance threshold, the occupant state recognition device determines the visual state of the occupant based on the facial orientation.

2. The visual inspection state determination unit Calculating a reliability of the acquisition of the gaze direction based on the imaging information; The occupant state recognition device according to claim 1 , wherein when the distance is greater than the long distance threshold and the reliability is equal to or greater than a predetermined reliability threshold, the visual state of the occupant is determined based on the line of sight direction.

3. The visual inspection state determination unit If the distance is smaller than a predetermined short distance threshold for acquiring the gaze direction, determining a visual state of the occupant based on the facial orientation; The occupant state recognition device according to claim 1 , further comprising: a step of: determining a visual state of the occupant based on the line of sight when the distance is equal to or greater than the short distance threshold value;

4. An occupant state recognition method for recognizing an occupant state of a vehicle by determining a visual state of an occupant, comprising: acquiring a facial orientation and a line of sight direction of the occupant based on image information captured by an in-vehicle camera of the vehicle; acquiring a distance from the in-vehicle camera to the head of the occupant based on distance measurement information from a distance measurement sensor of the vehicle; performing a visual state determination for determining a visual state of the occupant based on the distance and at least one of the face direction and the line of sight direction; In the visual inspection state determination, If the distance is equal to or less than a predetermined long distance threshold for acquiring the gaze direction, determining a visual observation state of the occupant based on the gaze direction; and determining a visual state of the occupant based on the facial orientation when the distance is greater than the long distance threshold.

5. An occupant state recognition program that operates an ECU of a vehicle to recognize an occupant state by determining a visual state of the occupant of the vehicle, an occupant information acquisition unit that acquires the facial orientation and line of sight direction of the occupant based on image information captured by an in-vehicle camera of the vehicle; a distance acquisition unit that acquires a distance from the in-vehicle camera to the head of the occupant based on distance measurement information from a distance measurement sensor of the vehicle; and a visual state determination unit that determines a visual state of the occupant based on the distance and at least one of the facial orientation and the line of sight; and operate the ECU as follows: In the visual inspection state determination unit, If the distance is equal to or less than a predetermined long distance threshold for acquiring the gaze direction, determining a visual observation state of the occupant based on the gaze direction; an occupant state recognition program that determines a visual state of the occupant based on the facial direction when the distance is greater than the long distance threshold;

Images