Occupant Monitoring System

Abstract

To provide an imaging apparatus for a crew member monitoring system, capable of imaging data for creating a clear detection image even in the case where a crew member moves in a photographing and a camera swings, and provide a member monitoring system having the imaging apparatus.SOLUTION: The present invention provides an imaging apparatus 1 for a crew member monitoring system, comprising: a first light source 11 that irradiates a light of a first wavelength to a crew member 33 in a vehicle; a second light source 12 that irradiates the light of a second wavelength different from the first wavelength from a position different from that of the first light source 11 to the crew member 33; and a spectroscopic camera 13 that images the crew member 33 to which the light of the first wavelength and the light of the second wavelength, and acquires a data cube 20 which can fetch out a first image data 21 having information on the wavelength region containing the first wavelength and a second image data 22 having information on the wavelength region containing a second wavelength.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging device for an occupant monitoring system used in a vehicle cabin, and to an occupant monitoring system. [Background technology]

[0002] Conventionally, there is known an image capturing device that detects the facial direction and line of sight of a driver from an image captured of the driver operating a vehicle, and that can remove light reflected from glasses worn by the driver from the image (see Patent Document 1).

[0003] When light is reflected on the lens or frame of the glasses, there is a risk that the image information of the driver's eyes cannot be obtained, and the detection accuracy of the driver's face direction and line of sight may decrease. The image capturing device described in Patent Document 1 is equipped with two light illuminators arranged at different positions to illuminate the driver, and can capture an image of the driver captured when light is illuminated from one light illuminator and an image of the driver captured when light is illuminated from the other light illuminator. These images are then synthesized to form an image in which the reflection of light on the glasses worn by the driver has been removed, thereby improving the detection accuracy of the driver's face direction and line of sight. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2010-124043 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, according to the image capturing device described in Patent Document 1, since it is necessary to capture an image of the driver when light is illuminated from one light illuminator and an image of the driver when light is illuminated from the other light illuminator at different times, if the driver moves between capturing these two images or if the camera shakes, a difference in the image of the driver will occur between the two images. Therefore, in the image obtained by combining these two images and removing the reflection of light on the glasses, the image of the driver becomes unclear, and there is a risk that information such as the direction of the face and the line of sight for determining the state of the driver cannot be detected accurately.

[0006] The object of the present invention is to provide an imaging device for an occupant monitoring system that acquires image data used to create detection images for detecting information such as the facial direction, line of sight, and eye openness of an occupant in a vehicle cabin, and that can acquire image data for creating clear detection images even if the occupant moves or the camera shakes during shooting, and an occupant monitoring system equipped with such an imaging device. [Means for solving the problem]

[0007] In order to achieve the above object, one aspect of the present invention provides an imaging device for an occupant monitoring system as described below in [1] and [2], and an occupant monitoring system as described in [3].

[0008] [1] An imaging device for an occupant monitoring system, comprising: a first light source that irradiates an occupant in a vehicle with light of a first wavelength; a second light source that irradiates the occupant with light of a second wavelength different from the first wavelength from a position different from the first light source; and a spectroscopic camera that images the occupant irradiated with light of the first wavelength and light of the second wavelength, and obtains a data cube from which first image data having information of a wavelength range including the first wavelength and second image data having information of a wavelength range including the second wavelength can be extracted as different data. [2] The imaging device for an occupant monitoring system described in [1] above, wherein the vehicle is an automobile. [3] An occupant monitoring system comprising: an imaging device for an occupant monitoring system as described in [1] or [2] above; and an image processing unit which uses the first image data and the second image data to create a detection image used for detecting information for determining the state of the occupant, in which reflections of light of the first wavelength and light of the second wavelength are suppressed. Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device for an occupant monitoring system that acquires image data used to create detection images for detecting information such as the facial direction, line of sight, and eye openness of an occupant in the vehicle cabin, and that can acquire image data for creating clear detection images even if the occupant moves or the camera shakes during shooting, and an occupant monitoring system equipped with the imaging device. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of the interior of an automobile, showing the locations of a first light source, a second light source, and a spectroscopic camera that constitute an imaging device according to the present embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing how the image capturing device captures an image of the area around the face of a passenger sitting in the driver's seat. [Diagram 3] FIG. 3 is a block diagram showing an example of the configuration of an occupant monitoring system including an imaging device mounted on a vehicle. [Figure 4] FIG. 4 is a flowchart that illustrates a flow of occupant monitoring by the occupant monitoring system. [Diagram 5] FIG. 5 is a schematic diagram of the structure of a data cube acquired by a spectroscopic camera. [Figure 6] 6(a) and (b) are schematic diagrams of the first image data and the second image data, respectively. [Figure 7] FIG. 7 is a schematic diagram showing how passengers inside a bus are monitored. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The imaging device 1 for an occupant monitoring system according to an embodiment of the present invention is an imaging device for an occupant monitoring system that acquires image data used to create a detection image for detecting the state of an occupant in a vehicle cabin.

[0012] The photographing device 1 includes a first light source 11 that irradiates a passenger in a vehicle with light of a first wavelength, a second light source 12 that irradiates the passenger with light of a second wavelength different from the first wavelength from a position different from the first light source 11, and a spectroscopic camera 13 that photographs the passenger irradiated with light of the first wavelength and light of the second wavelength and acquires a data cube from which first image data having information on a wavelength range including the first wavelength and second image data having information on a wavelength range including the second wavelength can be extracted as different data.

[0013] 1 is a schematic diagram of the interior of an automobile to show installation locations of a first light source 11, a second light source 12, and a spectroscopic camera 13 which constitute an imaging device 1. P0 shown in FIG. 1 shows an example of an installation location of the spectroscopic camera 13, and P1 and P2 show examples of installation locations of the first light source 11 and the second light source 12.

[0014] 1, the spectroscopic camera 13 is installed, for example, on a pad portion (P0) of a steering wheel 31. The first light source 11 and the second light source 12 are installed, for example, on a rim portion (P1) of the steering wheel 31 or on an A-pillar (front pillar) 32 (P2). For example, the spectroscopic camera 13 is placed within 1 m from the face of an occupant to be photographed, and the first light source 11 and the second light source 12 are placed within 2 m from the face of the occupant.

[0015] 2 is a schematic diagram showing how the imaging device 1 captures an image of the area around the face of an occupant 33 sitting in a driver's seat 30. In the example shown in FIG. 2, the first light source 11 and the second light source 12 are installed on the rim of the steering wheel 31, and the spectroscopic camera 13 is installed on the pad of the steering wheel 31.

[0016] The light of the first wavelength emitted from the first light source 11 and the light of the second wavelength emitted from the second light source 12 illuminate the area around the face of an occupant 33 sitting in the driver's seat 30, and the spectroscopic camera 13 photographs the area around the face of the occupant 33 in the illuminated state, and a data cube is obtained.

[0017] The spectroscopic camera 13 is a camera capable of acquiring a spectroscopic spectrum (brightness vs. wavelength) for each pixel, such as a multispectral camera or a hyperspectral camera. The image data acquired by the spectroscopic camera 13 through shooting is three-dimensional (or four or more dimensional) data that includes wavelength information in addition to two-dimensional planar data, and is called a data cube.

[0018] The first light source 11 and the second light source 12 are, for example, lighting devices equipped with light-emitting elements such as LEDs. The first wavelength, which is the emission wavelength of the first light source 11, and the second wavelength, which is the emission wavelength of the second light source 12, are determined by the spectroscopic performance of the spectroscopic camera 13. That is, it is only necessary that the first wavelength and the second wavelength are separated from each other to such an extent that the first image data having information on a wavelength range including the first wavelength and the second image data having information on a wavelength range including the second wavelength can be extracted as different data from the data cube acquired by the spectroscopic camera 13.

[0019] Typically, the first light source 11 and the second light source 12 each emit light in a single wavelength region in the near-infrared region. That is, the first wavelength and the second wavelength are each included in the near-infrared region. In order to reliably acquire the first image data and the second image data as different data, it is preferable that the first wavelength and the second wavelength (their peak wavelengths) are apart by 50 nm or more, for example.

[0020] 3 is a block diagram showing an example of the configuration of an occupant monitoring system 100 mounted on a vehicle and including an image capturing device 1. The occupant monitoring system 100 is a system that monitors the area around the face of an occupant 33 in the vehicle, and can suppress distracted driving and drowsy driving by issuing a warning according to the state of the driver determined by monitoring, for example.

[0021] The occupant monitoring system 100 includes an image capturing device 1 having a first light source 11, a second light source 12, and a spectroscopic camera 13, a storage unit 14 for storing a data cube acquired by the spectroscopic camera 13, an image processing unit 15 for creating a detection image in which reflections of light of a first wavelength and light of a second wavelength are suppressed, the detection image being used for detecting information (referred to as information for determination) such as the facial direction, line of sight, and degree of eye opening for determining the state of an occupant 33, by using the first image data and the second image included in the data cube stored in the storage unit 14, and an image processing unit 16 for processing the image. The vehicle information processing system includes a memory unit 16 that stores a detection image created by the processing unit 15, an image analysis unit 17 that analyzes the detection image stored in the memory unit 16 and detects information for judgment, a state judgment unit 18 that judges the state of the occupant 33 based on the information for judgment detected by the image analysis unit 17, a warning unit 19 that issues a warning to the occupant 33 in accordance with the state of the occupant 33 judged by the state judgment unit 18, and a control unit 10 that controls the operation of the first light source 11, the second light source 12, the spectroscopic camera 13, the image processing unit 15, the image analysis unit 17, the state judgment unit 18, the warning unit 19, etc.

[0022] The control unit 10, the memory unit 14, the image processing unit 15, the memory unit 16, the image analysis unit 17, and the state determination unit 18 constituting the occupant monitoring system 100 are included in, for example, an MCU (microcontroller unit) implemented in an ECU (electronic control unit) mounted on a vehicle. The warning unit 19 is, for example, an in-vehicle device that issues a warning by sound or light. The occupant monitoring system 100 may not include the warning unit 19 in its configuration. In this case, the warning unit 19 is connected to the occupant monitoring system 100 as an external device.

[0023] 4 is a flow chart that shows a schematic flow of monitoring of an occupant 33 by the occupant monitoring system 100. First, light of a first wavelength from a first light source 11 and light of a second wavelength from a second light source 12 are simultaneously irradiated onto the vicinity of the face of the occupant 33, and the vicinity of the face of the occupant 33 in a state where the first light and the second light are irradiated is photographed by the spectroscopic camera 13 (step S1). A data cube acquired by the spectroscopic camera 13 through the photographing is stored in the storage unit 14.

[0024] Fig. 5 is a schematic diagram of the structure of a data cube acquired by the spectroscopic camera 13. Fig. 5 shows a state in which a plurality of images each including information of a different wavelength range are included in a data cube 20. The plurality of images constituting the data cube 20 include first image data 21 having information of a wavelength range including the emission wavelength of the first light source 11 and second image data 22 having information of a wavelength range including the emission wavelength of the second light source 12.

[0025] For example, if the wavelength range of the first image data 21 is λ1 to λ2, the luminance of wavelengths outside the range of λ1 to λ2 is zero in each pixel included in the first image data 21. Since the second wavelength, which is the emission wavelength of the second light source 12, is outside the range of λ1 to λ2, the light emitted from the second light source 12 is not reflected in the first image data 21. Similarly, since the first wavelength, which is the emission wavelength of the first light source 11, is outside the wavelength range of the second image data 22, the light emitted from the first light source 11 is not reflected in the second image data 22.

[0026] 6(a) and (b) are schematic diagrams of the first image data 21 and the second image data 22, respectively. As shown in Fig. 6(a) and (b), light 351 having a first wavelength emitted from the first light source 11 and reflected by the glasses of the occupant 33 is reflected in the first image data 21, and light 352 having a second wavelength emitted from the second light source 12 and reflected by the glasses of the occupant 33 is reflected in the second image data 22. Here, since the first light source 11 and the second light source 12 are installed at different positions, the position where the light 351 of the first image data 21 is reflected and the position where the light 352 of the second image data 22 is reflected (coordinates on the image) are different.

[0027] Next, the image processing unit 15 extracts the first image data and the second image data from the data cube stored in the storage unit 14, and creates an image for detection using the first image data and the second image data (step S2). The image for detection created by the image processing unit 15 is stored in the storage unit 16.

[0028] After the spectroscopic camera 13 acquires the data cube, the first image data and the second image data extracted from the data cube may be stored in the storage unit 14. In this case, the first image data and the second image data stored in the storage unit 14 are used in step S2.

[0029] The detection image in step S2 is created by image synthesis (additive synthesis, subtractive synthesis, etc.) using the first image data 21 and the second image data 22. Specifically, for example, the first image data 21 and the second image data 22 are subtractively synthesized to extract a difference, and differential image data in which the light 351 and the light 352 are extracted (the luminance of the light 351 and the light 352 is selectively increased) is created, and the first image data 21 and the second image data 22 are additively synthesized to create added image data in which the information of the first image data 21 and the second image data 22 is added (including both the light 351 and the light 352), and the added image data is subtractively synthesized with the difference image data to obtain a detection image in which the light 351 and the light 352 are removed or almost removed. Note that the method of synthesizing the first image data 21 and the second image data 22 for creating the detection image is not particularly limited, and various known methods can be used.

[0030] The first image data 21 and the second image data 22 used to create the detection image in step S2 are acquired simultaneously by the spectroscopic camera 13, so even if the occupant 33 is moving or the spectroscopic camera 13 is shaking when the image is captured, no image shift due to a time lag occurs between the first image data 21 and the second image data 22. Therefore, even if the occupant 33 is moving or the spectroscopic camera 13 is shaking when the image is captured, a clear detection image can be created.

[0031] Next, the image analysis unit 17 analyzes the detection image stored in the storage unit 16 to detect information for determination such as the facial direction, line of sight, and eye opening degree of the occupant 33 (step S3). At this time, since the light 351 and the light 352 are removed or almost removed from the detection image in step S2, information on the part where the light 351 and the light 352 are reflected (such as the state of the eyes of the occupant 33) can be detected with high accuracy. Furthermore, even if the occupant 33 is moving or the spectroscopic camera 13 is shaking when the image is captured, the information for determination can be detected with high accuracy from the clear detection image.

[0032] Next, the state determination unit 18 determines the state of the occupant 33 based on the determination information detected by the image analysis unit 17 (step S4). For example, if the occupant 33's face direction or line of sight is not facing forward, it is determined that the occupant is looking away, and if the degree of eye opening is low, it is determined that the occupant is drowsy.

[0033] Next, the warning unit 19 issues a warning to the occupant 33 according to the state of the occupant 33 determined by the state determination unit 18 (step S5). For example, if the state determination unit 18 determines that the occupant 33 is looking away, it issues a voice warning to drive without looking away, and if it determines that the occupant is drowsy, it issues a sound or light warning to encourage the occupant to wake up.

[0034] In the above, a case has been described in which the driver in the vehicle cabin is monitored using the occupant monitoring system 100 equipped with the image capturing device 1, but the subject of monitoring may be an occupant other than the driver, or an occupant of a vehicle other than an automobile, such as a train or an airplane. When monitoring an occupant other than the driver, for example, the condition of the occupant, such as an abnormal physical condition, can be determined.

[0035] Fig. 7 is a schematic diagram showing a state in which passengers inside a bus are monitored (the first light source 11 and the second light source 12 are not shown). In the example shown in Fig. 7, a plurality of passengers 33a to 33e seated in seats 42 are monitored by a spectroscopic camera 13 of an imaging device 1 installed on the ceiling of the bus.

[0036] In such a case, for example, the condition of the occupant, such as abnormal physical condition or abnormal behavior, can be determined. In addition, since the spectroscopic camera 13 is used, the influence of external light and interior lighting can be reduced, and the accuracy of condition detection in the morning and at night can be improved. Furthermore, by using near-infrared light sources as the first light source 11 and the second light source 12, the reflection of the occupants 33a to 33e on the window glass 41 can be reduced, and false detection can be suppressed.

[0037] (Effects of the embodiment) According to the above embodiment of the present invention, by simultaneously acquiring the first image data 21 and the second image data 22 by the spectroscopic camera 13, a clear detection image can be created even if the occupant 33 is moving or the spectroscopic camera 13 is shaking when photographing. This makes it possible to accurately detect information for determination such as the face direction, line of sight, and degree of eye opening of the occupant 33, and to accurately determine the state of the occupant 33. For this reason, it is very useful for, for example, suppressing the driver from looking away or falling asleep, and for stable state determination in the transfer of driving authority at autonomous driving levels 3 to 4 (for example, determining whether the driver is in a state where he or she can drive when authority is transferred to the driver).

[0038] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the invention. Furthermore, the components of the above embodiment can be combined in any manner without departing from the spirit of the invention.

[0039] In addition, the above-described embodiments do not limit the scope of the invention according to the claims. Also, it should be noted that not all of the combinations of features described in the embodiments are necessarily essential to the means for solving the problems of the invention. [Explanation of symbols]

[0040] 1. Imaging device 10 Control section 11 First Light Source 12 Second Light Source 13 Spectroscopic camera 14, 16 Storage section 15 Image processing section 17 Image Analysis Section 18 Status determination section 19 Warning section 100 Occupant Monitoring System

Claims

1. an imaging device including a first light source that irradiates a passenger in a vehicle with light of a first wavelength, a second light source that irradiates the passenger with light of a second wavelength different from the first wavelength from a position different from that of the first light source, and a spectroscopic camera that images the passenger irradiated with the light of the first wavelength and the light of the second wavelength and obtains a data cube from which first image data having information of a wavelength range including the first wavelength and second image data having information of a wavelength range including the second wavelength can be extracted as different data; an image processing unit that uses the first image data and the second image data to create a detection image in which reflection of the light of the first wavelength and the light of the second wavelength is suppressed, and the detection image is used to detect information for determining a state of the occupant; Equipped with Occupant monitoring system.

2. The vehicle is an automobile; The occupant monitoring system of claim 1 .

Images