Crew status monitoring device

Abstract

To provide an occupant state monitoring apparatus capable of accurately monitoring a state of an occupant who is in a vehicle.SOLUTION: An occupant state monitoring apparatus 11 includes: an imaging section 21 that captures an image of an occupant 18 who is in a vehicle 10; a storage section 25 that stores vehicle member coordinates 16 that are position information on a vehicle member of the vehicle 10; and a calculation control section 26. The calculation control section 26 calculates gaze coordinates 17 indicating a position at which the occupant 18 gazes on the basis of the image obtained by capturing the occupant 18, and determines a state of the occupant 18 on the basis of the gaze coordinates 17 and the vehicle member coordinates 16.SELECTED DRAWING: Figure 5A

Description

Technical Field

[0001] The present invention relates to an occupant state monitoring device.

Background Art

[0002] As a device for monitoring the state of an occupant in a conventional vehicle, for example, the structures described in the following patent documents are known.

[0003] In the driving support device described in Patent Document 1, based on the line of sight detected by the line-of-sight detection unit, a display target to be displayed on the monitor is determined. Further, the monitor displays an image of the display target captured by an outside vehicle camera.

[0004] In Patent Document 2, based on the type distribution of each fixation region of the face image of the frames included in at least one sliding time window in the video, the monitoring result of the driver's attention is determined.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in the inventions described in each of the above-mentioned patent documents, there is room for improvement from the viewpoint of more accurately monitoring the state of the occupant.

[0007] Specifically, in the inventions described in each of the above-mentioned patent documents, although the state of the occupant is estimated from imaging means such as a video or a camera, information regarding the three-dimensional viewpoint of the occupant is not accurately grasped. Therefore, when attempting to monitor the occupant's side glances using such inventions, there was a problem that it was difficult to accurately monitor the side glances.

[0008] This invention has been made in view of these problems, and the object of this invention is to provide an occupant condition monitoring device that can accurately monitor the condition of occupants riding in a vehicle. [Means for solving the problem]

[0009] The occupant status monitoring device of the present invention comprises a shooting unit for photographing occupants riding in a vehicle, a storage unit for storing vehicle member coordinates, which are positional information of the vehicle's vehicle members, and a calculation control unit. The vehicle component coordinates include three-dimensional coordinate information indicating the interior surface of the dashboard. The calculation control unit calculates gaze coordinates indicating the position the occupant is looking at based on the image of the occupant, and based on the gaze coordinates and the vehicle component coordinates, If the occupant's line of sight is directed toward the interior surface of the dashboard, and the gaze coordinate is determined to be on the occupant's side rather than the interior surface of the dashboard, then it is determined that the occupant is looking away. It is characterized by the following: [Effects of the Invention]

[0010] According to the occupant status monitoring device of the present invention, the gaze coordinates are calculated three-dimensionally from the image captured by the camera unit, and the occupant's state can be determined by comparing the gaze coordinates with the vehicle component coordinates. For example, it is possible to easily detect when an occupant is looking away. In other words, even if the occupant's gaze is directed forward, if they are staring at portable devices or the like placed inside the vehicle, this state can be detected and reported, thereby improving safety during vehicle operation. [Brief explanation of the drawing]

[0011] [Figure 1A] A side view showing a vehicle equipped with an occupant condition monitoring device according to an embodiment of the present invention. [Figure 1B] This figure shows the front of the interior of a vehicle equipped with an occupant condition monitoring device according to an embodiment of the present invention. [Figure 2] This is a block diagram showing the connection configuration of an occupant condition monitoring device according to an embodiment of the present invention. [Figure 3] This is a schematic diagram showing the state of the occupants in an occupant state monitoring device according to an embodiment of the present invention. [Figure 4] It is a flowchart showing a method for monitoring the state of an occupant according to an embodiment of the present invention. [Figure 5A] In the occupant state monitoring device according to an embodiment of the present invention, it is a schematic diagram showing vehicle member coordinates. [Figure 5B] In the occupant state monitoring device according to an embodiment of the present invention, it is a schematic diagram showing vehicle member coordinates. [Figure 6] In the occupant state monitoring device according to an embodiment of the present invention, it is a schematic diagram for detailing the relationship between the fixation coordinates and the vehicle member coordinates. [Figure 7] In the occupant state monitoring device according to an embodiment of the present invention, it is a table showing the relationship between the line of sight and the presence or absence of notification. [Figure 8] In the occupant state monitoring device according to an embodiment of the present invention, it is a diagram showing the weighting of regions.

Mode for Carrying Out the Invention

[0012] Hereinafter, the occupant state monitoring device 11 according to an embodiment of the present invention will be described in detail based on the drawings. In the following description, the same members are basically denoted by the same reference numerals, and repeated descriptions are omitted.

[0013] FIG. 1A is a side view showing a vehicle 10 equipped with an occupant state monitoring device 11. FIG. 1B is a view showing the front part of the vehicle interior of the vehicle 10 equipped with the occupant state monitoring device 11.

[0014] The occupant state monitoring device 11 is a device for monitoring the state of an occupant 18 boarding the vehicle 10. The occupant state monitoring device 11 mainly includes a photographing unit 21, a storage unit 25, and an arithmetic control unit 26. The specific functions and configurations of the occupant state monitoring device 11 will be described later with reference to the figures after FIG. 2.

[0015] The vehicle 10 is, for example, an automobile, a train, or the like. The vehicle 10 is, for example, a vehicle equipped with an engine, an EV (Electrical Vehicle), an HEV (Hybrid Electrical Vehicle), or a PHEV (Plug-in Hybrid Electrical Vehicle).

[0016] In the seat 13 which is the driver's seat, the occupant 18 who drives the vehicle 10 is seated. The occupant 18 drives the vehicle 10 by operating the steering wheel 14, a brake pedal (not shown), an accelerator pedal, and the like.

[0017] A windshield 12 is disposed at the front portion of the vehicle body 28.

[0018] As shown in FIG. 1B, a display 15 is disposed near the center in the left-right direction of the dashboard 19. The display 15 is a multi-function display that collectively displays information regarding the vehicle 10 such as the outside air temperature, the operating status of the air conditioner, and also functions as a notification unit that displays an alarm described later. Here, as the notification unit, a speaker that emits an alarm sound may be adopted.

[0019] On the dashboard 19 above the display 15, a photographing unit 21 for photographing the head 30 etc. of the occupant 18 who drives the vehicle 10 is disposed. The photographing unit 21 is called a DMS (Driver Monitor System). The photographing unit 21 is, for example, an infrared camera or the like.

[0020] Gauges 23 are disposed on the back side of the upper portion of the steering wheel 14. The gauges 23 display the vehicle speed, the engine rotation speed, and the like.

[0021] The occupant status monitoring device 11 monitors the status of the occupant 18 driving the vehicle 10 using the imaging unit 21, and if it determines that the occupant 18 is distracted, it notifies the occupant 18 accordingly. In this way, the occupant 18 can recognize their own status through the notification from the display 15, etc., and return from a distracted state to a normal state of concentration on driving. This operation will be described later with reference to the flowchart.

[0022] Figure 2 is a block diagram showing the connection configuration of the crew condition monitoring device 11.

[0023] The crew condition monitoring device 11 mainly comprises a calculation control unit 26, an imaging unit 21, a storage unit 25, a display 15, and a notification unit 27.

[0024] The arithmetic control unit 26 includes a CPU, RAM, and ROM. The input terminals of the arithmetic control unit 26 are connected to the imaging unit 21 and the storage unit 25. The output terminals of the arithmetic control unit 26 are connected to the display 15 and the notification unit 27. As will be described later, the arithmetic control unit 26 calculates gaze coordinates 17 indicating the position that the occupant 18 is looking at, based on the image of the occupant 18, and determines the state of the occupant 18 based on the gaze coordinates 17 and the vehicle component coordinates 16.

[0025] The camera unit 21 photographs the crew members 18 riding in the vehicle 10. The camera unit 21 is, for example, an infrared camera and can photograph the heads 30 of the crew members 18 during the day and at night.

[0026] The memory unit 25 stores vehicle component coordinates 16, which are position information of vehicle components. The vehicle component position information refers to information indicating the three-dimensional position of the dashboard 19 shown in Figure 1A and the surfaces of each component provided thereon.

[0027] As mentioned above, the display 15 is a display device that shows various information related to the vehicle 10.

[0028] The notification unit 27 is, for example, a speaker installed on the vehicle body 28, and if it determines that the crew member 18 is distracted, it notifies the crew member 18 by broadcasting a message to that effect.

[0029] Figure 3 is a schematic diagram showing the state of the crew member 18 as captured by the imaging unit 21. The aforementioned imaging unit 21 captures the eyeballs 20 located in the head 30 of the crew member 18. Furthermore, the aforementioned calculation control unit 26 calculates the gaze coordinates 17, which are the three-dimensional positions where the crew member 18 is fixating, from the images captured of the head 30 or the eyeballs 20. For example, the calculation control unit 26 calculates the gaze coordinates 17 from the position of the pupil of the eyeball 20.

[0030] Figure 4 is a flowchart showing a method for monitoring the status of crew member 18. Referring to Figure 4, the method for monitoring distraction as a status of crew member 18 using the crew member status monitoring device 11 will be explained.

[0031] In step S10, the vehicle 10 begins to move based on the instructions of the occupant 18.

[0032] In step S11, the calculation control unit 26 uses the imaging unit 21 to photograph the head 30 and eyeballs 20 of the occupant 18. The calculation control unit 26 continues the imaging by the imaging unit 21 until the vehicle 10 has finished moving.

[0033] In step S12, the calculation control unit 26 calculates the three-dimensional coordinates of the gaze coordinates 17 from the image captured by the imaging unit 21. Specifically, as shown in Figure 5B, the calculation control unit 26 performs calculations to identify the gaze coordinates 17 from the image of the occupant 18. More specifically, as shown in Figure 6, the calculation control unit 26 calculates the line of sight angle and focal length from the lines of sight of the left and right eyeballs 20, calculates the line of sight coordinate angle from the position of the eyeballs 20, calculates the absolute value of the viewpoint coordinate (relative to the vehicle body) from the position and tilt of the head 30, and calculates the absolute value of the viewpoint coordinate (relative to the vehicle body) from the posture of the occupant 18. In this embodiment, the three-dimensional position of the gaze coordinates 17 can be accurately determined by calculating the gaze coordinates 17 based on information about the occupant 18's eyeballs 20, etc.

[0034] In step S13, the calculation control unit 26 compares the gaze coordinates 17 and the vehicle member coordinates 16. Referring to Figure 5A, the vehicle member coordinates 16 are three-dimensional coordinates that represent the surface of the vehicle members that make up the interior of the vehicle 10. For example, the vehicle member coordinates 16 are three-dimensional coordinates that represent the inner surface of the windshield 12, the inner surface of the dashboard 19, etc. For example, the calculation control unit 26 compares the gaze coordinates 17 and the vehicle member coordinates 16 by superimposing them.

[0035] In step S14, the arithmetic control unit 26 monitors the state of the crew member 18 and determines, for example, whether the crew member 18 is distracted or not.

[0036] As a concrete example of calculation, the gaze coordinate 17 and the vehicle component coordinate 16 can be compared. For example, referring to Figure 6, the calculation control unit 26 can determine that the occupant 18 is distracted if the gaze coordinate 17 is on the occupant 18's side rather than the surface of the vehicle component. More specifically, if the distance L10 between the vehicle component coordinate 16, which indicates the rear surface of the dashboard 19, and the gaze coordinate 17 is greater than or equal to a predetermined threshold, it can be determined that the occupant 18 is distracted. In this case, it is determined that the occupant 18 is not looking at the front of the vehicle 10 through the windshield 12, but is distracted by a portable device 22, such as a smartphone, located on the rear side of the dashboard 19. In this case, as will be described later, the occupant 18 can be notified not to distract themselves.

[0037] Referring to the table in Figure 7, in step S14, it is possible to determine whether or not the crew member 18 is distracted or whether notification is necessary based on the type of object that the crew member 18 is looking at.

[0038] Specifically, if the angle of the occupant's gaze is directed towards the windshield 12 or a mirror (side mirror or rearview mirror), and the gaze coordinate 17 is outside the vehicle, the calculation control unit 26 determines that the occupant 18 is not looking away and therefore no notification is necessary.

[0039] Furthermore, if the angle of the occupant's gaze is directed towards the windshield 12 or a mirror (side mirror or rearview mirror), and the gaze coordinate 17 is inside the vehicle, the calculation control unit 26 considers the relationship between the vehicle component coordinate 16 and the gaze coordinate 17. That is, if the gaze coordinate 17 is on or near the surface of the vehicle component coordinate 16, the calculation control unit 26 determines that the occupant 18 is temporarily looking at condensation or dust on the glass, and that they are not distracted and therefore no notification is necessary. On the other hand, if the gaze coordinate 17 is closer to the occupant 18 than the vehicle component coordinate 16, i.e., closer to the occupant 18, the calculation control unit 26 determines that the occupant 18 is distracted by looking at a portable device 22 such as a smartphone, and therefore a notification is necessary.

[0040] Furthermore, if the occupant's (18) line of sight is directed towards an interior surface such as the dashboard (19) and the focus is within the vehicle interior, the calculation control unit (26) considers the relationship between the vehicle component coordinates (16) and the gaze coordinates (17). Specifically, if the gaze coordinates (17) are located on or near the surface of the vehicle component coordinates (16), the calculation control unit (26) determines that the occupant (18) is temporarily looking at the infotainment system or interior dirt, is not distracted, and therefore no notification is necessary. On the other hand, if the gaze coordinates (17) are located closer to the occupant (18) than the vehicle component coordinates (16), i.e., closer to the occupant (18), the calculation control unit (26) determines that the occupant (18) is distracted, looking at a portable device (22) such as a smartphone, and therefore a notification is necessary. Also, if the gaze coordinates (17) are located further away from the occupant (18) than the vehicle component coordinates (16), i.e., further away from the occupant (18), the calculation control unit (26) determines that the occupant (18) is temporarily looking at a detached or damaged part, is not distracted, and therefore no notification is necessary.

[0041] As described above, by determining whether the occupant 18 is distracted, distraction can be accurately detected and reported according to the internal conditions of the vehicle 10 and the driving conditions of the occupant 18.

[0042] Referring to Figure 8, in step S14, weighting processing can be performed so that the degree to which the area where the portable device 22 is placed is judged to be distraction increases. Specifically, on the dashboard 19, the area where the portable device 22, such as a smartphone, is placed is limited. Typically, the portable device 22 is placed near the left or right side of the steering wheel 14, taking into consideration the convenience and visibility of the occupant 18. In Figure 8, the area 29 enclosed by the dotted line is where the portable device 22 is frequently placed.

[0043] Referring to Figure 6, one example of weighting is to set the threshold for comparison with L10 within region 29 to be shorter than in other regions. This makes it easier to reliably detect distraction within region 29.

[0044] In this way, the arithmetic control unit 26 can perform weighting processing in region 29 to increase the likelihood of it being judged as distracted. That is, if the crew member 18's gaze is directed towards region 29, it is determined that there is a high probability that the crew member 18 is distracted. On the other hand, if the crew member 18's gaze is directed away from region 29, it is determined that there is a low probability that the crew member 18 is distracted. In this way, distractedness caused by the crew member 18 staring at the portable device 22 can be detected more accurately.

[0045] If the answer in step S14 is YES, i.e., if crew member 18 is distracted, proceed to step S15.

[0046] If the answer in step S14 is NO, that is, if crew member 18 is not distracted, proceed to step S18 and no notification will be given.

[0047] In step S15, the calculation control unit 26 notifies the crew member 18. Specifically, the calculation control unit 26 emits voice or warning sounds from a speaker or other device, which is the notification unit 27, to correct the distracted driving. Alternatively, a display 15 can be used as the notification unit 27, in which case an image to correct the distracted driving will be displayed on the display 15. By notifying the crew member 18 via the notification unit 27, the crew member 18's distracted driving can be corrected.

[0048] In step S16, the arithmetic control unit 26 determines whether the crew member 18 is continuing to look away. The processing in step S16 is the same as in steps S13 and S14 described above.

[0049] If the answer in step S16 is YES, that is, if the crew member 18 continues to look away, the calculation control unit 26 returns to step S15 and continues the notification.

[0050] If the answer in step S16 is NO, that is, if the crew member 18 stops looking away, the calculation control unit 26 proceeds to step S17.

[0051] In step S17, the arithmetic control unit 26 terminates the notification by the notification unit 27.

[0052] In step S18, the occupant 18 terminates the driving of the vehicle 10. That is, the occupant 18 stops the vehicle 10.

[0053] Although embodiments of the present invention have been described above, the present invention is not limited thereto, and modifications are possible without departing from the spirit of the invention. Furthermore, the above-described embodiments can be combined with each other.

[0054] For example, in the embodiment described above, distraction was adopted as the state monitored by the calculation control unit 26, but other states can also be monitored. For example, a state in which the crew member 18 is extremely unbalanced can also be set as a state monitored by the calculation control unit 26. [Explanation of symbols]

[0055] 10 vehicles 11 Crew condition monitoring device 12 Windshield 13 seats 14. Steering wheel 15 displays 16 Vehicle component coordinates 17 Gaze Coordinates 18 crew members 19 Dashboard 20 Eyeball 21 Photography Department 22. Portable equipment 23 Meters 25 Memory section 26. Arithmetic Control Unit 27 Hochi Department 28 car bodies 29 areas 30 head

Claims

1. The camera crew was responsible for filming the occupants of the vehicle, A storage unit that stores vehicle member coordinates, which are position information of the vehicle members of the aforementioned vehicle, It comprises a calculation control unit and, The vehicle component coordinates include three-dimensional coordinate information indicating the interior surface of the dashboard. The calculation control unit, Based on the image of the crew member, the gaze coordinates indicating the position the crew member is looking at are calculated. An occupant status monitoring device characterized in that, based on the gaze coordinates and the vehicle component coordinates, it is determined that the occupant is looking away when it is determined that the occupant's line of sight is directed toward the interior surface of the dashboard and the gaze coordinates are on the occupant's side relative to the interior surface of the dashboard.

2. The calculation control unit, The occupant status monitoring device according to claim 1, characterized in that it calculates the gaze coordinates from information about the occupant's eyeballs included in the aforementioned image.

3. Further comprising a notification unit, The calculation control unit, The occupant status monitoring device according to claim 1 or 2, characterized in that, if it is determined that the occupant is distracted based on the gaze coordinates and the vehicle member coordinates, the device notifies the occupant via the notification unit.

4. The occupant condition monitoring device according to any one of claims 1 to 3, characterized in that the calculation control unit performs a weighting process such that the degree to which the interior surface of the dashboard is judged to be distracted is higher than the degree to which the interior surface of the dashboard is judged to be distracted in a second area excluding the first area.

5. The occupant condition monitoring device according to claim 4, characterized in that the predetermined first region is located near the left or right side of the steering wheel.

6. The calculation control unit shall If the occupant's line of sight is directed toward the first region, and the gaze coordinate is at least a first distance away from the interior surface of the dashboard toward the occupant, it is determined that the occupant is looking away. The occupant status monitoring device according to claim 4 or 5, characterized in that, when the occupant's line of sight angle is directed toward the second region, it is determined that the occupant is looking away if the gaze coordinate is at least a second distance greater than the first distance from the interior surface of the dashboard toward the occupant.

Images