Imaging device and biological site detection device

Abstract

To suppress deterioration of the detection accuracy of a portion of a human body based on a photographed image by using a simplified structure even under such an environment that reflected light as external light exists.SOLUTION: An imaging device includes an imaging unit 200 that includes light emission parts 202a, 202b that emit illumination light, light reception parts 204a, 204b that receive imaging light, and a control part 220 that controls the operations of the light emission parts and light reception parts. When a representative light path of the illumination light is an illumination light path LI, a representative light path of the imaging light is an imaging light path Li, an angle formed by the illumination light path LI and a reference straight line Lc is θIc, and an angle formed by the imaging light path Li and the reference straight line Lc is θic, the light emission parts and the light reception parts are arranged so as to satisfy such a first condition that θic<θIc and such a second condition that they are located in a space on the same side in a case of dividing a space of a vehicle coordinate system into two pieces by a plane Scx including a reference eye point Pc or a plane Scy.SELECTED DRAWING: Figure 3

Description

Technical Field

[0004] , , , , , , , ,

[0005] , , , , , , , ,

[0001] The present invention relates to an imaging device, a biological part detection device, and the like.

Background Art

[0002] Patent Document 1 discloses a configuration in which an imaging device using infrared light is mounted on a head-up display (HUD) device.

[0003] Patent Document 2 describes that when irradiating a user of a HUD device with infrared light to image the user and detecting the position of the user's eyes (such as the position of the pupil) based on the captured image, even when strong external light such as sunlight or streetlight enters the HUD device, it is possible to detect the position of the eyes by image processing.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] As a result of the study by the present inventor, the following matters have become clear. (1) When detecting a part such as the face of a vehicle occupant using infrared light or the like, if external light enters the imaging device, the detection accuracy decreases. (2) Examples of external light include reflected light generated by reflection of infrared light, which is illumination light, etc., by the occupant's glasses (including sunglasses), and reflected light generated by reflection of sunlight, etc., by the windshield. <00(3) It is preferable to take measures against both reflected light from illumination and reflected light from sunlight. However, using complex image processing systems, etc., is undesirable as it would lead to increased costs or an increased processing load on the equipment.

[0006] One of the objectives of the present invention is to provide an imaging device that can suppress a decrease in the accuracy of detecting parts of the human body based on captured images, even in environments where reflected light as ambient light is present, using a simplified configuration.

[0007] Other objects of the present invention will become apparent to those skilled in the art by referring to the embodiments and best embodiments described below, as well as the accompanying drawings. [Means for solving the problem]

[0008] The following are examples of embodiments of the present invention to facilitate understanding of its outline.

[0009] In the first embodiment, the imaging device is An imaging device used to image parts of a subject that is a vehicle occupant, Of the light used to detect the body parts of the subject, which are the occupants of the vehicle, The light reflected by the reflector and illuminating the part of the subject is used as illumination light. When the illumination light is reflected by the part of the subject that is the target of imaging, and further reflected by the reflector and returned to the imaging device, the resulting light is used as the imaging light. A light-emitting unit that emits the aforementioned illumination light, A light-receiving unit that receives the aforementioned imaging light, A control unit that controls the operation of the light-emitting unit and the light-receiving unit, It has an imaging unit equipped with, The coordinate axis of the vehicle body coordinate system, which is positioned in the longitudinal direction of the aforementioned vehicle, is defined as the X-axis. The Y-axis is a coordinate axis that is perpendicular to the X-axis and is positioned in the left-right direction of the vehicle. The coordinate axis that is orthogonal to the X and Y time axes and is positioned in the height direction of the vehicle is defined as the Z axis. The reference eye point Pc is defined as a point in the vehicle body coordinate system that is determined to be the typical eye position of the occupant of the vehicle as the subject, The reference line Lc is defined as a straight line passing through the reference eye point Pc of the occupant of the vehicle, which is the subject of the photograph, and parallel to the X-axis. The representative optical path of the illumination light between the reference eye point Pc and the reflector is defined as the illumination light path LI. The representative optical path of the imaging light between the reference eye point Pc and the reflector is defined as the imaging optical path Li. Let θIc be the angle that the illumination path LI makes with the reference straight line Lc. When the angle that the imaging optical path Li makes with the reference straight line Lc is θic, The light-emitting unit and the light-receiving unit are arranged to satisfy the first and second conditions. The first condition mentioned above is, θic < θIc The second condition mentioned above is, The light-emitting unit and the light-receiving unit are located parallel to the ZX plane in the vehicle body coordinate system and are situated in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scx that includes the reference eye point Pc. Or, The light-emitting unit and the light-receiving unit are located in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scy that includes the reference eye point Pc, parallel to the XY plane in the vehicle body coordinate system.

[0010] According to the first embodiment, the illumination light, such as infrared light, is directed towards the subject, the occupant (driver, etc.), from outside the optical axis of the infrared camera or other light-receiving unit (in other words, the imaging optical path Li, which is the representative optical path of the imaging light). Therefore, for example, the reflected light of the illumination light that is reflected by the lenses of the occupant's glasses (including sunglasses) does not go towards the infrared camera or other light-receiving unit.

[0011] In other words, the reflected light from lighting fixtures, such as those reflected by glasses, can be used to suppress the effect on the crew's vision, for example, their eyes, and the surrounding areas during imaging.

[0012] For example, assume a situation where the subject, the occupant, is facing substantially forward, that is, the occupant is looking ahead and the line of sight of the occupant is substantially parallel to the X-axis (front-rear direction axis) in the vehicle body coordinate system.

[0013] From the above second condition, the light emitting part and the light receiving part are located in the same space (here, the first space) which is divided into two spaces by the plane Scx or the plane Scy.

[0014] Therefore, the illumination optical path LI, which is a representative optical path of the illumination light between the reference eye point Pc and the reflector (such as the windshield), and which can be regarded as the optical axis of the light emitting part such as an infrared LED, and the imaging optical path Li, which is a representative optical path of the imaging light between the reference eye point Pc and the reflector (such as the windshield), and which can be regarded as the optical axis of the infrared camera or the like, are also located in the same space (the first space).

[0015] The illumination light irradiated from the light emitting part is reflected by the reflector (such as the windshield), travels through the above illumination optical path LI in the first space, and reaches the part of the occupant (for example, the face of a person, which can include eyes, nose, ears, and mouth), and is reflected at that part.

[0016] The reflected light (this is regarded as the regular reflected light) travels through the above imaging optical path Li (the optical axis of an infrared camera or the like constituting the light receiving part) in the first space, is reflected by the reflector (such as the windshield), and is incident on the light receiving part. Based on the electrical signal obtained by that light reception, detection of the part of the person (the part of the living body) is performed.

[0017] On the other hand, for example, when a person is wearing glasses (including sunglasses) or a mask, infrared light or the like, which is the imaging light, may be reflected by the glasses or the like, and reflected light (reflected light as unnecessary external light) may occur.

[0018] Here, from the first condition described above, the illumination light enters the eyeglasses, etc., from outside the optical axis of the infrared camera, etc., which constitutes the light-receiving part (in other words, the imaging optical path Li). Unwanted reflection occurs here, but since the illumination light enters the eyeglasses, etc., at a shallow angle of incidence, the unwanted reflected light caused by total internal reflection will go towards the second space, which is on the opposite side of the first space, and will not go towards the first space.

[0019] Therefore, unwanted reflected light is suppressed from superimposing on the regular reflected light used to detect human body parts, thus preventing a decrease in detection accuracy.

[0020] By arranging the light-emitting unit and the light-receiving unit in such a way that they satisfy the first and second conditions described above, the incidence of reflected light as unwanted ambient light into the imaging device can be suppressed, thereby preventing a decrease in the detection accuracy of human body parts.

[0021] In a second embodiment dependent on the first embodiment, The light-emitting unit that emits the illumination light is provided with a first light-emitting unit and a second light-emitting unit, The control unit can switch which of the first and second light-emitting units is used to illuminate the subject. The second light-emitting part may be arranged to satisfy the third condition. The third condition mentioned above is, The second light-emitting unit is located in the space opposite to the space where the first light-emitting unit is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that includes the reference eye point Pc, parallel to the ZX plane in the vehicle body coordinate system. Or, The second light-emitting part is located in the space opposite to the space where the first light-emitting part is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc.

[0022] In the second embodiment, a first and a second light-emitting unit are provided as light-emitting units. The second light-emitting unit is located in the space opposite to the first light-emitting unit when the space of the vehicle body coordinate system is divided into two by a plane Scx or Scy including the reference eye point Pc, and the control unit can switch the illumination (on and off) of each light-emitting unit.

[0023] In the first embodiment described above, it is possible to suppress unwanted reflected light from being directed towards the imaging unit, but it is still possible that some of the unwanted reflected light may enter the imaging unit.

[0024] For example, if a crew member changes the direction of their face, the illumination light emitted from one of the light-emitting parts may be reflected by the lenses of the glasses or other eyewear worn by the crew member, creating unwanted reflected light. This unwanted reflected light may then enter the imaging unit (or its light-receiving part), resulting in unwanted reflection patterns in the captured image.

[0025] In such cases, the control unit can switch between the first and second light-emitting units, for example, by turning off one light-emitting unit and turning on the other.

[0026] By switching between the first and second light-emitting units, the direction and angle of incidence of the illumination light on the eyeglasses, etc., are changed. Therefore, it is possible to suppress the reflection of the illumination light off the lenses of the eyeglasses, etc., which generates imaging light, from being directed towards the imaging device (the light-receiving unit in the imaging unit of the imaging device). Therefore, it is possible to suppress a decrease in detection accuracy when detecting, for example, the eyes of an occupant or their surroundings based on the captured image.

[0027] In a third embodiment dependent on the first embodiment, The imaging unit includes a first imaging unit and a second imaging unit. The second imaging unit may be arranged to satisfy the fourth condition. The fourth condition is, The second imaging unit is located in the space opposite to the space where the first imaging unit is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc. Or, The second imaging unit is located in the space opposite to the space where the first imaging unit is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc. and, The light-emitting unit and light-receiving unit in the first imaging unit are referred to as the first light-emitting unit and the first light-receiving unit, When the light-emitting unit and light-receiving unit in the second imaging unit are made into a second light-emitting unit and a second light-receiving unit, The first light-emitting unit and the first light-receiving unit are arranged to satisfy the first condition and the second condition. The second light-emitting unit and the second light-receiving unit are also arranged to satisfy the first and second conditions. The control unit may be configured to select which of the first light-emitting unit, the second light-emitting unit, the first light-receiving unit, or the second light-receiving unit to use.

[0028] In the third embodiment, the imaging unit is provided with first and second imaging units. The first imaging unit has a first light-emitting unit and a first light-receiving unit, and the second imaging unit has a second light-emitting unit and a second light-receiving unit. The control unit can appropriately select which of the first and second light-emitting units and the first and second light-receiving units to use for imaging parts of the human body.

[0029] For example, even when switching the light-emitting part using the configuration of the second embodiment described above, it may not be possible to completely prevent reflection from eyeglass lenses, etc.

[0030] In such cases, by appropriately selecting (switching) the devices used, including not only the light-emitting part but also the light-receiving part, it is possible to suppress the decrease in the accuracy of detecting human body parts due to unwanted reflected light.

[0031] For example, if the first imaging unit (first light-emitting unit and first light-receiving unit) is used, an unwanted reflection pattern may appear in the captured image. However, even in this case, using the second imaging unit (second light-emitting unit and second light-receiving unit) is likely to reduce the influence of the unwanted reflection pattern in the captured image.

[0032] For example, unwanted reflected light from eyeglass lenses, etc., may not enter the second light-receiving unit in the second imaging unit.

[0033] Furthermore, even if unwanted reflected light is incident on the second light-receiving unit, the position of the unwanted reflection pattern on the captured image will be different from the position of the unwanted reflection pattern on the captured image when the first imaging unit is used.

[0034] Therefore, when using the first imaging unit, even if there are areas where unwanted reflection patterns are superimposed and feature detection is not possible (for example, the left eye may not be detected), using the second imaging unit will shift the position of the unwanted reflected light patterns. As a result, areas where feature detection was not possible in the image captured by the first imaging unit (in this case, the left eye) will be detectable in the image captured by the second imaging unit.

[0035] Therefore, for example, by using the imaging data obtained from the first and second imaging units together (for example, by performing correction processing such as supplementing areas that could not be detected in one imaging image with the corresponding area in the other imaging image, or by performing image synthesis processing, etc.), the loss of information in the entire subject can be sufficiently suppressed.

[0036] Furthermore, if strong ambient light (such as sunlight) passes through a reflector (windshield) and enters, for example, the first imaging unit (first light receiving unit), and overlaps with the image of the subject captured by the reflector (windshield, etc.), it may interfere with imaging the occupant's eyes and their surroundings.

[0037] Even in this case, when the second imaging unit is used, it is positioned in the space opposite to the first imaging unit, and because of the difference in position, the optical path of strong ambient light and the optical path for imaging the subject (imaging optical path) are misaligned. Therefore, it is possible to image the occupants' eyes and the surrounding area.

[0038] Furthermore, various modifications exist in this embodiment. For example, a modified version could be considered in which the first light-emitting unit and the first light-receiving unit in the first imaging unit, and the second light-receiving unit in the second imaging unit, are utilized. In this modified version, the first light-emitting unit is used as a common light-emitting unit, and the imaging light is received by the first and second light-receiving units (in other words, two light-receiving units located in different positions). Furthermore, by using the image data obtained from each light-receiving unit in combination and performing image correction and image synthesis, the influence of unwanted reflected light patterns can be reduced, enabling accurate detection of the entire human body part (such as the face). In the example above, the first light-emitting unit and the first and second imaging units were used, but it is also possible to use the second light-emitting unit and the first and second imaging units. In this case as well, the same effect can be obtained.

[0039] In a fourth embodiment dependent on any one of the first to third embodiments, The light-receiving unit has a filter that blocks visible light and transmits the illumination light and the imaging light. It has, The light-emitting part may be positioned in a location where it cannot be seen, at least directly, by the occupants of the vehicle.

[0040] In the fourth embodiment, a light-receiving unit with a filter is used as the light-receiving unit of the imaging unit. For the filter section, for example, a multilayer long-pass filter (such as an IR filter) made of a material that absorbs visible light and transmits infrared light can be used. This makes it possible to suppress sunlight, streetlights, or reflected light generated when these lights are reflected by reflectors (such as windshields) from entering the light-receiving section of an infrared camera, thereby improving imaging accuracy.

[0041] Furthermore, the filter section provided in the light-receiving unit also has the visual effect of preventing the light-receiving unit from being directly visible to the occupants of the vehicle being photographed. This helps to prevent the design of the imaging unit from being compromised.

[0042] On the other hand, the light-emitting part is positioned in a location where it cannot be seen, at least not directly, by the vehicle's occupants.

[0043] As described above, the light-receiving unit is equipped with a filter, so the crew cannot normally see it, and as described above, the light-emitting unit is located in a position that the crew cannot normally see. Therefore, both the light-receiving unit and the light-emitting unit are difficult to see. Thus, it is possible to suppress any deterioration in the overall design of the imaging unit (imaging device).

[0044] Furthermore, adopting this configuration improves the flexibility of the arrangement (layout) of the light-receiving unit (infrared camera, etc.) and the light-emitting unit (infrared LED, etc.).

[0045] For example, if the above-mentioned filter unit is installed in the window portion of the imaging device (the part through which infrared light passes), then a constraint arises that both the light-receiving unit and the light-emitting unit must be installed in the area where visible light is blocked by the filter unit.

[0046] In contrast, in this embodiment, the light-receiving unit (infrared camera, etc.) itself is equipped with a filter, thus providing protection against visible light such as sunlight. Therefore, the light-emitting unit (infrared LED, etc.) is not subject to the same placement constraints as described above, and the degree of freedom in placement is improved.

[0047] In other words, within the range where the first and second conditions described in the first embodiment above are met, the light-receiving unit (infrared camera, etc.) and the light-emitting unit (infrared LED, etc.) can each be positioned in an optimal location with a certain degree of freedom. This improves the freedom of placement and simplifies the design of the imaging device.

[0048] In a fifth embodiment dependent on the first embodiment, The imaging device is An image display device is provided within the vehicle, which illuminates the reflector of the vehicle with display light to show an image to the occupant of the vehicle, Or, It may be provided integrally with the image display device.

[0049] According to the fifth embodiment, an imaging device can be provided within the image display device, thereby promoting miniaturization of the entire device (system).

[0050] In the sixth embodiment, the biological site detection device is A biological body part detection device that detects the body parts of occupants of a vehicle as a subject based on captured images, An imaging device in any of the first to fifth embodiments, A biological site detection processing unit that detects a part of a living organism based on the image acquired by the aforementioned imaging device, It holds.

[0051] In the sixth embodiment, the biological site detection device includes, in addition to the imaging device of each embodiment described above, a biological site detection processing unit that detects a part of a living organism (such as a human face) by image processing.

[0052] The biological part detection processing unit can, for example, if a reflected light pattern exists in an image obtained from one imaging unit, and a part such as an eye superimposed on that reflected light pattern cannot be detected, perform image processing (image correction, image synthesis, etc.) using images of the undetected part (such as an eye) from other images obtained from other imaging units, thereby enabling the detection of the entire part, such as a face. This improves detection accuracy.

[0053] Those skilled in the art will readily understand that the embodiments of the present invention illustrated can be further modified without departing from the spirit of the invention. [Brief explanation of the drawing]

[0054] [Figure 1] This figure shows an example of the configuration of an imaging device that uses infrared light. [Figure 2] This figure shows an example of the relative positional relationship between a reference eye point, a reference straight line passing through the reference eye point and parallel to the X-axis of the vehicle coordinate system, an illumination path which is a representative optical path of illumination light between the reference eye point and the reflector, an imaging path which is a representative optical path of imaging light between the reference eye point and the reflector, a plane parallel to the ZX plane in the vehicle coordinate system and containing the reference eye point, and a plane parallel to the XY plane in the vehicle coordinate system and containing the reference eye point. [Figure 3] This diagram shows examples of the arrangement of the light-emitting and light-receiving units that constitute each imaging unit when using the first and second imaging units, the optical paths of the illumination light and imaging light between the windshield (which is a reflector) and the reference eye point, and the positional relationships of each optical path. [Figure 4] This figure shows examples of captured images obtained by capturing the faces of crew members using the first and second imaging units, respectively. [Figure 5] This figure shows an example of the configuration of a biological site detection device. [Figure 6] This figure shows an example of the timing of the illumination of the light-emitting unit and the timing of the imaging by the light-receiving unit in each imaging unit when imaging a subject using the first and second imaging units. [Figure 7] This figure shows other examples of the timing of the illumination of the light-emitting unit and the timing of the imaging by the light-receiving unit in each imaging unit when imaging a subject using the first and second imaging units. [Figure 8] This figure shows an example configuration of an in-vehicle display device (in this case, a head-up display (HUD) device) equipped with an imaging device (and biological site detection device) using infrared light, and an example arrangement of the first and second imaging units in the in-vehicle display device. [Modes for carrying out the invention]

[0055] The best embodiments described below are used to facilitate understanding of the present invention. Therefore, those skilled in the art should note that the present invention is not unduly limited by the embodiments described below.

[0056] (First embodiment) Figure 1 shows an example of the configuration of an imaging device that uses infrared light. In Figure 1, the X, Y, and Z axes are the coordinate axes of the vehicle body coordinate system (Cartesian coordinate system). Here, the X axis is the coordinate axis of the vehicle body coordinate system that is positioned in the longitudinal direction of vehicle 1, the Y axis is the coordinate axis that is perpendicular to the X axis and is positioned in the lateral direction of vehicle 1, and the Z axis is the coordinate axis that is perpendicular to both the X and Y coordinate axes and is positioned in the height direction of vehicle 1.

[0057] Furthermore, in Figure 1, the point in the vehicle coordinate system that is defined as the typical eye position of the occupant 5 of vehicle 1, which is the subject of the photograph, is defined as the reference eye point Pc. Note that the "reference eye point" can be rephrased as the "center eye point located at the midpoint of both eyes." Furthermore, the reference straight line Lc is defined as a straight line parallel to the X-axis that passes through the reference eye point Pc of the occupant 5 of vehicle 1, which is the subject of the photograph. Furthermore, the representative optical path of the illumination light between the reference eye point Pc and the reflector 2, such as the windshield, is defined as the illumination light path LI. Note that the illumination light itself may also be referred to as illumination light LI.

[0058] Furthermore, the representative optical path of the imaging light between the reference eye point Pc and the reflector is defined as the imaging light path Li. Note that the imaging light itself is sometimes referred to as imaging light Li. Furthermore, PrI is defined as the point of reflection of the illumination path (or illumination light) L1 in the reflector (windshield) 2. Furthermore, Pri is defined as the reflection point of the imaging light path (or imaging light) Li in the reflector (windshield) 2. The optimal relative positions of each of the above-mentioned parts will be explained using Figures 2 and 3, etc.

[0059] Furthermore, in the following explanation, the light used to image parts of the subject (in this case, infrared light) that is reflected by a reflector (windshield 2, etc.) and illuminates the part of the subject (in Figure 1, the face 11 of occupant 5 of vehicle 1) will be referred to as "illumination light".

[0060] Furthermore, the light that is reflected by the part of the subject being imaged from the illumination light, and then reflected again by the reflector 2 and returned to the imaging device 201, is called "imaging light."

[0061] In Figure 1A-1, the imaging device 201 has an imaging unit 200 which includes a light-emitting unit (illumination unit) 202 consisting of, for example, an infrared LED that emits near-infrared light, and a light-receiving unit 204 consisting of an infrared camera.

[0062] The imaging device (reflective imaging device) 201 is equipped with a control unit (not shown in Figure 1, but indicated by reference numeral 220 in Figures 3, 5 to 7) that controls the operation of the light-emitting unit (infrared LED, etc.) 202 and the light-receiving unit (infrared camera, etc.) 204 in the imaging unit 200.

[0063] Furthermore, in the example shown in Figure 1, a windshield 2 is used as a reflector, possessing both reflectivity and transmittance of light (for example, display light for displaying images on a HUD device).

[0064] In the example shown in Figure 1, the subject being imaged is the driver 5, an occupant of vehicle 1, operating the steering wheel 8.

[0065] In Figure 1, reference numeral 6 indicates the viewpoint (eyes) of driver 5. Driver 5 is also wearing eyeglasses (including sunglasses) 9.

[0066] Infrared light, which is the illumination light, may be reflected by the lenses of the glasses 9 (or a mask, etc.), generating reflected light RF1. Also, sunlight Lt from the sun 3 may be reflected by a reflector (windshield) 2, for example, generating reflected light RF2.

[0067] Reflected light from sunlight, for example, in the case where the imaging device 201 is mounted on an in-vehicle display device such as a HUD, as in the example in Figure 8, may indirectly enter the in-vehicle display device via a window through which the display light passes, and then enter the imaging device 201 as stray light.

[0068] The imaging device 201, for example, images the face 11 (which may include the eyes, nose, ears, mouth, etc.), which is a part of the driver 5. Furthermore, by observing, for example, the face 11, it becomes possible to estimate the current state of the driver 5 (e.g., normal concentration, fatigue, drowsiness, etc.).

[0069] Furthermore, the imaging device 201 is housed, for example, in the instrument panel (or dashboard) 4 of the vehicle 1.

[0070] In addition, in the example shown in Figure 1, the imaging device 201 is housed in a housing 116 that has a window (an opening in the example shown in Figure 1) 12.

[0071] Furthermore, a filter section 13 is provided on the light-incident side of the light-receiving section (infrared camera, etc.) 204 in the imaging section 200. The filter section 13 has the characteristic of transmitting infrared light (in other words, illumination light and imaging light) and blocking (for example, absorbing) visible light. This filter section 13 has the effect of preventing ambient light from entering the imaging device 201.

[0072] This prevents sunlight, streetlights, or reflected light from reflective surfaces (such as windshields) from entering the light-receiving section of an infrared camera, thereby improving imaging accuracy.

[0073] Furthermore, the filter section 13 provided on the light-receiving section 204 also has the visual effect of preventing the light-receiving section 204 from being directly visible to the occupants 5 of the vehicle 1, which are the subject of the photograph. This helps to prevent the design of the imaging section from being compromised.

[0074] Furthermore, "direct viewing" means excluding methods such as "viewing indirectly using a mirror, etc." or "unnatural visual viewing such as forcing oneself into an awkward posture." This point is also true in the following explanation.

[0075] As the filter section 13, for example, a multilayer long-pass filter made of a material that absorbs visible light and transmits infrared light, such as an IR filter, can be used.

[0076] Furthermore, as shown in Figure 1A-2, the light-emitting part (infrared LED, etc.) 202 in the imaging unit 200 is positioned so that it cannot be seen, at least directly, by the occupants 5 of the vehicle 1.

[0077] In other words, the light-emitting unit 202 is located outside the expected field of view (or field of view range) FR when a person (occupant) assumed to be standard is seated and looking forward in a normal posture, and therefore is positioned so that the occupant 5 of the vehicle 1 cannot see it directly.

[0078] As described above, the light-receiving unit 204 is equipped with a filter unit 13, so that the crew member 5 cannot normally see it. Also, as described above, the light-emitting unit 202 is positioned so that the crew member 5 cannot normally see it.

[0079] Therefore, both the light-receiving unit 204 and the light-emitting unit 202 are difficult to see with the naked eye. Thus, it is possible to prevent the overall design of the imaging unit (imaging device) from being compromised.

[0080] Furthermore, adopting the configuration of the imaging unit 200 shown in A-1 and A-2 of Figure 1 improves the flexibility of the arrangement (layout) of the light-receiving unit (infrared camera, etc.) 204 and the light-emitting unit (infrared LED, etc.) 202.

[0081] For example, if the above-mentioned filter unit is to be installed in the window portion of the imaging device 201 (the portion through which infrared light passes), then a layout constraint arises in which both the light-receiving unit and the light-emitting unit must be installed in the area where visible light is blocked by the filter unit.

[0082] In contrast, in the example shown in Figure 1, the light-receiving unit (infrared camera, etc.) 204 itself is equipped with a filter unit 13, thus providing protection against visible light such as sunlight. Therefore, the light-emitting unit (infrared LED, etc.) 202 is not subject to the layout constraints described above, and the degree of freedom in placement is improved.

[0083] In other words, within the range that satisfies predetermined imaging conditions (which will be explained using Figures 2 and 3), the light-receiving unit (infrared camera, etc.) 204 and the light-emitting unit (infrared LED, etc.) 202 can each be positioned in an optimal location with a certain degree of freedom. Therefore, according to the configuration of the example in Figure 1, the degree of freedom of placement is improved compared to conventional designs, and the design of the imaging device is simplified.

[0084] The following provides a more detailed explanation of the preferred configuration. As the light-receiving unit (imaging unit) 204, for example, an infrared camera equipped with a filter unit 13 that acts as an optical filter that transmits only a specific wavelength band W of infrared light can be used. Here, the wavelength band W may be, for example, about 950 ± 25 nm.

[0085] Furthermore, an LED or the like that emits light in the above-mentioned wavelength band W can be used as the light-emitting unit (illumination unit) 202.

[0086] Furthermore, as described above, an automobile windshield 2 can be used as a reflector to reflect illumination light and imaging light. However, an interlayer that reflects at least 50% of infrared light in the wavelength band W may be provided.

[0087] Furthermore, the light-emitting unit 202 is positioned and angled such that the illumination light is reflected by point PrI on the reflector 2 and illuminates parts of the subject, such as the face of the occupant (the driver in the example of Figure 1) 5.

[0088] Furthermore, the light-receiving unit 204 is positioned and angled to capture imaging light that has been reflected by the subject and then reflected again by point Pri on the reflector 2.

[0089] As described above, the light-receiving unit 204 and the light-emitting unit 202 are installed within the instrument panel 4 and receive imaging light and emit illumination light, for example, through the surface of the housing 116 or through a window (an opening in the example of Figure 1) provided on the surface of the instrument panel 4.

[0090] Furthermore, regarding the arrangement (layout) of the light-receiving unit 204 and the light-emitting unit 202, for example, if we assume that the light-receiving unit 204 is installed on the end side in the left-right direction of the vehicle as seen from the occupant (driver) (in other words, on the side of the front pillar (also called the A-pillar) that serves as a support for the windshield 2), then it is preferable that the light-emitting unit 202 be installed on the end side of the vehicle 1 as seen from the occupant (driver) 5 (further outward, closer to the front pillar (A-pillar)) (see, for example, the layout example A-2 in Figure 8).

[0091] However, the above layout is merely an example and is not limited to it. The light-emitting unit 202 does not necessarily have to be positioned on the outside of the vehicle as seen from the occupant (driver) relative to the light-receiving unit 204; various arrangements are possible as long as the "predetermined conditions" are met. The "predetermined conditions" will be explained using Figures 2 and 3. For example, as shown in Figure 2, it is preferable to have an arrangement in which a cone obtained by rotating the optical path Lic of the illumination light is provided on the outside of a cone obtained by rotating the optical path Lic of the imaging light around a straight line Lc as an axis.

[0092] Next, refer to Figure 2. Figure 2 is a diagram illustrating an example of the relative positional relationship between a reference eye point, a reference straight line passing through the reference eye point and parallel to the X-axis of the vehicle coordinate system, an illumination path which is a representative optical path of illumination light between the reference eye point and the reflector, an imaging path which is a representative optical path of imaging light between the reference eye point and the reflector, a plane parallel to the ZX plane in the vehicle coordinate system and containing the reference eye point, and a plane parallel to the XY plane in the vehicle coordinate system and containing the reference eye point. Note that in Figure 2, parts common to Figure 1 are denoted by the same reference numerals (this is also the case in other drawings).

[0093] In Figure 2, the symbol 6L indicates the left eye of crew member 5, and the symbol 6R indicates the right eye of crew member 5.

[0094] To explain the relative positions of light rays, for example, it is necessary to define the spatial location of each ray. Therefore, in Figure 2A-1, the space of the vehicle body coordinate system is divided into two halves, left and right, from the perspective of the occupant (driver) 5, by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc. The space on the left from the perspective of the occupant (driver) 5 is designated as the "first space QL," and the space on the right is designated as the "second space QR."

[0095] As shown in Figure 1, when the light-receiving unit 204 and light-emitting unit 202 of the imaging unit 200 are both located within the instrument panel 4 and are arranged in the left-right direction of the vehicle 1, the positional relationship of the light rays can be explained using the space shown in A-1 of Figure 2.

[0096] On the other hand, if the light-receiving unit 204 and the light-emitting unit 202 are arranged in a divided manner in the height direction of the vehicle 1, for example, if the light-emitting unit 202 is located inside the instrument panel 4 and the light-receiving unit 204 is located on the ceiling of the vehicle 1 or on the upper part of the front pillar (A-pillar), the positional relationship of the light rays can be explained using the space shown in A-2 of Figure 2.

[0097] In Figure 2A-2, the space of the vehicle body coordinate system is divided vertically from the perspective of the occupant (driver) 5 by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc. The lower space from the perspective of the occupant (driver) 5 is designated as the "third space QD," and the upper space is designated as the "fourth space QU."

[0098] The following explanation describes the case of A-1 in Figure 2 (the following explanation can also be applied to A-2 in Figure 2).

[0099] In Figure 2A-1, the cone obtained by rotating the optical path Li of the illumination light is located outside the cone obtained by rotating the optical path Li of the imaging light with respect to the straight line Lc as the axis (central axis).

[0100] The cone for imaging light is depicted as having the reference eye point Pc as its apex and the circle Di (a smaller diameter circle) obtained by rotating the optical path Li of the imaging light as its base.

[0101] The cone for illumination light is depicted as having the reference eye point Pc as its vertex and the circle DI (a circle with a larger diameter) obtained by rotating the optical path LI of the illumination light as its base.

[0102] Earlier, in the explanation for Figure 1, it was stated that "the light-emitting unit 202 is positioned and angled such that the illumination light is reflected at point PrI on the reflector 2 and illuminates parts of the subject, such as the face of the occupant (driver) 5, and the light-receiving unit 204 is positioned and angled such that it can capture the imaging light that is reflected by the subject and then reflected again at point Pri on the reflector 2." In other words, the light-emitting unit 202 and the light-receiving unit 204 are positioned such that the relative positional relationship of the cone shown in A-1 (and A-2) of Figure 2 is established.

[0103] Next, refer to Figure 3. This figure shows examples of the arrangement of the light-emitting and light-receiving units that constitute each imaging unit when using the first and second imaging units, the optical paths of the illumination light and imaging light between the windshield (which is a reflector) and the reference eye point, and the positional relationships of each optical path.

[0104] In the example A-1 in Figure 3, first and second imaging units 200a and 200b are provided. The first imaging unit 200a has a first light-emitting unit 202a and a first light-receiving unit 204a, and the second imaging unit 200b has a first light-emitting unit 202b and a second light-receiving unit 204b. Reference numerals 16R and 16L indicate the front pillar (A-pillar).

[0105] The first imaging unit 200a (first light-emitting unit 202a and first light-receiving unit 204a) is located in the "first space QL," which is the space to the left of the occupant 5 when the space of the vehicle body coordinate system is divided into left and right halves by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc, as explained earlier in Figure 2A-1.

[0106] The second imaging unit 200b (the second light-emitting unit 202b and the second light-receiving unit 204b) is located in the "second space QR (the space located on the opposite side of the first space QL with respect to plane Scx, relative to plane Scx)" which is the space on the right side as seen from the perspective of the occupant 5, when the space of the vehicle body coordinate system is divided into left and right halves by plane Scx, which is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc, as explained earlier in Figure 2A-1.

[0107] As shown in Figure 3A-2, the imaging control unit 220 can individually control the operation of the first light-emitting unit 202a, the second light-emitting unit 202b, and the second light-receiving unit 204b. In other words, it is possible to individually select which light-emitting unit and light-receiving unit to use when imaging.

[0108] Next, refer to A-3 in Figure 3. As explained earlier, the point in the vehicle coordinate system that is determined to be the typical eye position of the occupant 5 of vehicle 1 as the subject is defined as the reference eye point Pc, the straight line Lc is defined as the reference straight line passing through the reference eye point Pc of the occupant 5 of vehicle 1 as the subject and parallel to the X axis, the illumination light path LI is defined as the typical optical path of the illumination light between the reference eye point Pc and the reflector (windshield, etc.) 2, the imaging light path Li is defined as the typical optical path of the imaging light between the reference eye point Pc and the reflector (windshield, etc.) 2, and the angle that the illumination light path LI makes with the reference straight line Lc is defined as θIc, and the angle that the imaging light path Li makes with the reference straight line Lc is defined as θic.

[0109] However, in the example shown in Figure 3, since there are two imaging units 200a and 200b, the angles θIc are described separately as θIc1 and θIc2 to correspond to each imaging unit. Similarly, the angles θic are also described separately as θic1 and θic2.

[0110] In the figure, PrI1 and PrI2 indicate the reflection points on the reflector (windshield, etc.) 2 of the illumination light emitted from the first and second light-emitting units 202a and 202b, respectively. Also, Pri1 and Pri2 indicate the reflection points on the reflector (windshield, etc.) 2 of the imaging light incident on the first and second light-receiving units 204a and 204b, respectively.

[0111] In Figure 3A-3, the relationship θic < θIc is satisfied. More specifically, the relationships θic1 < θIc1 and θic2 < θIc2 hold. Let's call the above relationship the "first condition".

[0112] Furthermore, as shown in A-1 of Figure 3, the first light-emitting unit 202a and the first light-receiving unit 204a of the first imaging unit 200a are parallel to the ZX plane in the vehicle body coordinate system and are located in the same space (the first space QL in the example of A-1 in Figure 3) when the space of the vehicle body coordinate system is divided into two by the plane Scx which includes the reference eye point Pc. Furthermore, the second light-emitting unit 202b and the second light-receiving unit 204b in the second imaging unit 200b are parallel to the ZX plane in the vehicle body coordinate system and are located in the same space (the second space Qr in the example of A-1 in Figure 3) when the space of the vehicle body coordinate system is divided into two by the plane Scx which includes the reference eye point Pc. Let's call this relationship the "second condition."

[0113] Regarding this second condition, when the space of the vehicle body coordinate system is divided into two by a plane Scy (see A-2 in Figure 2) that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc, the first light-emitting unit 202a and the first light-receiving unit 204a of the first imaging unit 200a are located in the third space QD shown in A-2 in Figure 2, for example, and the second light-emitting unit 202b and the second light-receiving unit 204b of the second imaging unit 200b are located in the fourth space QU shown in A-2 in Figure 2, for example.

[0114] Thus, the light-emitting unit and light-receiving unit (the first light-emitting unit 202a and the first light-receiving unit 204a, or the second light-emitting unit 202b and the second light-receiving unit 204b) in each imaging unit 200a and 200b are arranged to satisfy the first and second conditions. The first condition is, θic < θIc (θic1 < θIc1 or θic2 < θIc2) The second condition is, The light-emitting unit and light-receiving unit (first light-emitting unit 202a and first light-receiving unit 204a, or second light-emitting unit 202b and second light-receiving unit 204b) in each imaging unit are located parallel to the ZX plane in the vehicle body coordinate system and are situated in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scx that includes the reference eye point Pc. Or, The light-emitting unit and light-receiving unit (the first light-emitting unit 202a and the first light-receiving unit 204a, or the second light-emitting unit 202b and the second light-receiving unit 204b) in each imaging unit are located parallel to the XY plane in the vehicle body coordinate system and are situated in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scy that includes the reference eye point Pc.

[0115] As a result, the illumination light, such as infrared light, is directed towards the subject, the occupant (driver, etc.) 5, from outside the optical axis of the infrared camera or other light-receiving unit (in other words, the imaging optical path Li, which is the representative optical path of the imaging light). Therefore, for example, the reflected illumination light generated by reflection from the lenses of the glasses (including sunglasses) 9 worn by the occupant 5 does not go towards the light-receiving unit 204 (204a, 204b) consisting of the infrared camera or the like.

[0116] In other words, the reflected light from the illumination, which is reflected by the glasses 9, can be used to suppress the effect on the crew member's 5, for example, their eyes and the surrounding area during imaging.

[0117] For example, let's assume a situation where the subject, occupant 5, is facing approximately forward; in other words, occupant 5 is looking ahead, and the occupant's line of sight is approximately parallel to the X-axis (longitudinal axis) in the vehicle coordinate system.

[0118] Based on the second condition above, the light-emitting part and the light-receiving part are located in the same space, which is divided into two parts by the plane Sx or plane Scy.

[0119] Therefore, the illumination light path LI, which is the representative optical path of illumination light between the reference eye point Pc and the reflector (windshield, etc.) 2 (this can be considered as the optical axis of the light-emitting part of an infrared LED, etc.), and the imaging light path Li, which is the representative optical path of imaging light between the reference eye point Pc and the reflector (windshield, etc.) (this can be considered as the optical axis of an infrared camera, etc.), are located in the same space.

[0120] Illumination light emitted from the light-emitting unit 202 (202a, 202b) is reflected by the reflector (windshield, etc.) 2, travels through the illumination light path LI to a part of the occupant 5 (for example, a person's face (which may include eyes, nose, ears, and mouth)), and is reflected at that part.

[0121] The reflected light (referred to as the normal reflected light) travels through the imaging optical path Li (the optical axis of an infrared camera or the like that constitutes the light-receiving unit), is reflected by the reflector (windshield, etc.) 2, and enters the light-receiving unit 204 (204a, 204b). Based on the electrical signal obtained from this light reception, the detection of human body parts is performed.

[0122] On the other hand, for example, if crew member 5 is wearing glasses (including sunglasses) 9 or a mask, the infrared light used for imaging may be reflected by the glasses 9, resulting in reflected light (reflected light as unwanted ambient light).

[0123] Here, from the first condition described above, the illumination light enters the glasses 9 etc. from outside the optical axis (in other words, the imaging optical path Li) of the infrared camera etc. that constitutes the light receiving unit 204 (204a, 204b).

[0124] Unwanted reflection occurs here, but since the illumination light is incident on the glasses 9 etc. at a shallow angle of incidence, the unwanted reflected light resulting from total internal reflection is directed towards the space opposite to the space to which the light-emitting part 202 (202a or 202b) and the light-receiving part 204 (204a or 204b) belong.

[0125] Therefore, unwanted reflected light is suppressed from superimposing on the regular reflected light used to detect human body parts, thus preventing a decrease in detection accuracy.

[0126] By arranging the light-emitting unit and the light-receiving unit in such a way that they satisfy the first and second conditions described above, the incidence of reflected light as unwanted ambient light into the imaging device can be suppressed, thereby preventing a decrease in the detection accuracy of human body parts.

[0127] Next, refer to Figure 4. Figure 4 shows examples of captured images obtained by capturing the faces of the crew members using the first imaging unit and the second imaging unit, respectively.

[0128] Figure 4 explains the effects of using multiple (in this case, two) light-emitting units by switching between them, as well as the effects of using multiple (in this case, two) imaging units as shown in A-1 and A-2 of Figure 3.

[0129] Figure 4A-1 is the same as Figure 3A-1. Figure 4A-2 shows an example of an image captured by the first imaging unit 200a. Near the right eye of crew member 5, there is a reflection pattern of illumination light on the lens of the eyeglasses 9, making it difficult to detect the features of the right eye.

[0130] On the other hand, the image captured by the second imaging unit 200b does not contain any reflection patterns of illumination light, making it possible to detect the entire face 11.

[0131] Therefore, by using the two captured images shown in Figure 4A-2 and A-3 in combination, and by supplementing or correcting the image near the area where feature detection is difficult (in this case, the right eye) with other captured images that are not affected, it becomes possible to detect faces 11, etc., without any missing parts.

[0132] The following provides a detailed explanation. In the example of A-1 in Figure 4, a first light-emitting unit 202a and a second light-emitting unit 202b are provided as light-emitting units that emit illumination light, and the control unit 220 can appropriately switch which of the first and second light-emitting units 202a and 202b is used to illuminate the occupant 5 as the subject.

[0133] In this case, the second light-emitting unit 202b may be arranged to satisfy the third condition. The third condition is, The second light-emitting unit 202b is located in the space opposite to the space where the first light-emitting unit 202a is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that includes the reference eye point Pc, parallel to the ZX plane in the vehicle body coordinate system. Or, The second light-emitting unit 202b is located in the space opposite to the space where the first light-emitting unit 202a is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc.

[0134] In other words, if first and second light-emitting units 202a and 202b are provided as light-emitting units, the second light-emitting unit 202b is positioned in the space opposite to the first light-emitting unit 202a when the space of the vehicle body coordinate system is divided into two by a plane Scx or Scy including the reference eye point Pc, and the control unit 220 can switch the illumination (on and off) of each light-emitting unit 202a and 202b.

[0135] For example, if crew member 5 changes the direction of their face, as in the example of A-2 in Figure 4, the illumination light emitted from one of the light-emitting units (in this case, the first light-emitting unit 202a) may be reflected by the lens of the glasses 9 worn by crew member 5, generating unwanted reflected light. This unwanted reflected light may also enter the imaging unit (in this case, the light-receiving unit 204a of the first imaging unit 200a), resulting in unwanted reflection patterns in the captured image.

[0136] In such cases, the control unit 220 can switch between the first and second light-emitting units 202a and 202b, for example, by turning off one light-emitting unit and turning on the other.

[0137] By switching between the first and second light-emitting units 202a and 202b, the direction and angle of incidence of the illumination light to the glasses 9, etc., are changed. Therefore, it is possible to suppress the reflection of the illumination light from the lenses of the glasses 9, etc., which generates imaging light, towards the imaging device (the light-receiving unit in the imaging unit of the imaging device). Therefore, it is possible to suppress a decrease in detection accuracy when detecting, for example, the eyes of the occupant 5 or their surroundings based on the captured image.

[0138] Next, we will describe an example in which multiple (in this case, two) imaging units are provided, and when imaging is performed, the light-emitting unit and light-receiving unit to be used for imaging are appropriately selected from among the light-emitting unit and light-receiving unit contained in each imaging unit.

[0139] As shown in Figure 4A-1, if the imaging unit includes a first imaging unit 200a and a second imaging unit 200b, the second imaging unit 200b shall be arranged to satisfy the fourth condition. The fourth condition is that the second imaging unit 200b is located in the space opposite to the space where the first imaging unit 200a is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc. Or, The second imaging unit 200b is located in the space opposite to the space where the first imaging unit 200a is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc. and, The light-emitting unit and light-receiving unit in the first imaging unit 200a are defined as the first light-emitting unit 202a and the first light-receiving unit 204a, When the light-emitting unit and light-receiving unit in the second imaging unit 200b are made up of a second light-emitting unit 202b and a second light-receiving unit 204b, The first light-emitting unit and the light-receiving unit 1 are arranged to satisfy the first condition and the second condition, The second light-emitting unit 202b and the second light-receiving unit 204b are also arranged to satisfy the first and second conditions described above. The control unit (imaging control unit) 220 may be configured to individually select which of the first light-emitting unit 202a, the second light-emitting unit 202b, the first light-receiving unit 204a, and the second light-receiving unit 204b to use.

[0140] For example, even if the light-emitting unit is switched, it may not be possible to completely prevent reflections from the lenses of the glasses 9. In such cases, by appropriately selecting (switching) the device to be used, including not only the light-emitting unit but also the light-receiving unit, the effect of suppressing the decrease in the accuracy of detecting human body parts due to unwanted reflected light can be further enhanced.

[0141] For example, if the first imaging unit 200a (the first light-emitting unit 202a and the first light-receiving unit 204a) is used, an unwanted reflection pattern may appear in the captured image. However, even in this case, using the second imaging unit 200b (the second light-emitting unit 202b and the second light-receiving unit 204b) is highly likely to reduce the influence of the unwanted reflection pattern in the captured image (see examples A-1 and A-2 in Figure 4).

[0142] For example, unwanted reflected light from the lenses of the eyeglasses 9 may not enter the second light-receiving unit 204b of the second imaging unit 200b.

[0143] Furthermore, even if unwanted reflected light is incident on the second light-receiving unit 204b, the position of the unwanted reflection pattern on the captured image will be different from the position of the unwanted reflection pattern on the captured image when the first imaging unit 200b is used.

[0144] Therefore, when using the first imaging unit 200a, even if there are areas where unwanted reflection patterns are superimposed and feature detection is not possible (for example, the right eye may not be detectable), using the second imaging unit 200b will shift the position of the unwanted reflected light patterns. As a result, areas where feature detection was not possible in the image captured by the first imaging unit 200a (in this case, the right eye) will be detectable in the image captured by the second imaging unit 200b.

[0145] Therefore, for example, by using the imaging data obtained by the first and second imaging units 200a and 200b together (for example, by performing correction processing such as supplementing areas that could not be detected in one imaging image with the corresponding area in the other imaging image, or by performing image synthesis processing, etc.), the loss of information in the entire subject can be sufficiently suppressed.

[0146] Furthermore, if strong ambient light (such as sunlight) passes through the reflector (windshield, etc.) 2 and enters, for example, the first imaging unit 200a (or its first light-receiving unit 204a), a situation may arise where it overlaps with the image of the subject captured by the reflector (windshield, etc.) 2, causing interference with imaging of the occupant's eyes and surrounding areas.

[0147] Even in this case, when the second imaging unit 200b is used, the second imaging unit 200b is located in the space opposite to the first imaging unit 200a, and because its position is different, the optical path of strong ambient light and the optical path for imaging the subject (imaging optical path) are misaligned. Therefore, it is possible to image the eyes of the occupant 5 and the surrounding area. Thus, by using each captured image in combination to detect faces, etc., the detection accuracy can be improved.

[0148] Next, refer to Figure 5. Figure 5 is a diagram showing an example of the configuration of a biological site detection device. The biological site detection device (e.g., a face detection device) 110 includes an imaging device (reflective imaging device) 201 and a biological site detection processing unit 230.

[0149] The imaging device 201 includes first and second imaging units 200a and 200b, and an imaging control unit (control unit) 220.

[0150] The first imaging unit 200a includes a first light-emitting unit (infrared LED, etc.) 202a and a first light-receiving unit (infrared camera, etc.) 204a.

[0151] The second imaging unit 200b includes a second light-emitting unit (infrared LED, etc.) 202b and a second light-receiving unit (infrared camera, etc.) 204b.

[0152] The biological part detection processing unit 230 includes an image acquisition unit 231, a reflected light pattern extraction unit 232 that extracts reflected light patterns having a predetermined pattern, a reflected light pattern detection unit 233 that detects the position and type (whether it is a reflection of illumination light or sunlight, etc.) of the reflected light pattern based on, for example, training data, a biological part detection unit (e.g., a face detection unit) 236 that detects the part of the living body to be detected (a part of the occupant that is the subject) based on the detection results, and a result output unit 238.

[0153] As explained earlier, the biological part detection processing unit 230 can, for example, if a reflected light pattern exists in the image captured from one imaging unit, and a part such as the eyes superimposed on that reflected light pattern cannot be detected, perform image processing (image correction, image synthesis, etc.) using the image of the undetected part (eyes, etc.) from another image captured by another imaging unit, thereby enabling the detection of the entire part, such as the face. This improves detection accuracy.

[0154] Next, refer to Figure 6. Figure 6 shows an example of the timing of the illumination of the light-emitting unit and the imaging timing by the light-receiving unit in each imaging unit when imaging a subject using the first and second imaging units.

[0155] Figure 6A-1 is the same as Figure 3A-2 shown earlier. Figure 6A-2 shows an example of the timing of the illumination of the light-emitting unit and the imaging timing by the light-receiving unit in each imaging unit.

[0156] In Figure 6A-2, the code D202a is the drive signal for the first light-emitting unit 202a, the code D204a is the drive signal for the first light-receiving unit 204a, the code D202b is the drive signal for the second light-emitting unit 202b, and the code D204b is the drive signal for the second light-receiving unit 204b. The same applies to Figure 7A-2.

[0157] In the example shown in Figure 6A-2, imaging is performed by the first imaging unit 200a (first light-emitting unit 202a, first light-receiving unit 204a) during periods T1 and T3, and imaging is performed by the second imaging unit 200b (second light-emitting unit 202b, second light-receiving unit 204b) during periods T2 and T4.

[0158] In the example shown in Figure 6A-2, the first and second light-receiving units 204a and 204b are driven so that their respective exposure periods do not overlap. Furthermore, the light-emitting periods (illumination periods) of the first and second light-emitting units 202a and 202b are synchronized with the exposure periods of the first and second light-receiving units 204a and 204b.

[0159] Next, refer to Figure 7. Figure 7 shows another example of the timing of the illumination of the light-emitting unit and the imaging timing by the light-receiving unit in each imaging unit when imaging a subject using the first and second imaging units.

[0160] Figure 7A-1 is the same as Figure 3A-2 shown earlier. Figure 7A-2 shows an example of the timing of the illumination of the light-emitting unit and the imaging timing by the light-receiving unit in each imaging unit.

[0161] In Figure 7A-2, the first light-emitting unit 202a is off, and the second light-emitting unit 202b is lit during each of the periods T1 to T4.

[0162] Furthermore, during periods T1 and T3, imaging is performed by the first light-receiving unit 204a in the first imaging unit 200a. Furthermore, during periods T2 and T4, imaging is performed by the second light-receiving unit 204b in the second imaging unit 200b.

[0163] Whether to adopt the driving example shown in Figure 6 or the driving example shown in Figure 7 can be selected depending on the presence or absence of a reflected light pattern, or, if a reflected light pattern is present, its position and type.

[0164] For example, when the reflected light pattern detection unit 233 of the biological site detection processing unit 230 shown in Figure 5 detects a reflected light pattern and identifies its location and type, this information is supplied to the imaging control unit (control unit) 220. Based on the supplied information, the imaging control unit (control unit) 220 appropriately selects the light-emitting unit and light-receiving unit to be used for imaging and drives each unit at the appropriate timing.

[0165] Furthermore, the biological part detection unit (face detection unit) 236 in the biological part detection processing unit 230 shown earlier in Figure 5 may, for example, synthesize image data acquired by both the first and second light receiving units 204a and 204b to grasp the three-dimensional position of the driver and detect (identify) with high accuracy the position of the face and eyes, the direction of the gaze, and the forward lean of the posture due to drowsiness or poor physical condition.

[0166] This detection result (identification result) may be communicated to the occupants (driver, etc.) by display on an in-vehicle display device, such as a head-up display (HUD). Furthermore, if necessary, an alert may be issued to the occupants (driver, etc.).

[0167] Furthermore, the imaging control unit (control unit) 220 may constantly monitor whether the occupant 5 is wearing glasses or the like, or the direction of the occupant's face.

[0168] If it is determined that the illumination light from the first light-emitting unit 202a may be reflected by lenses such as eyeglasses, making it impossible to image the eye or its surroundings with the first light-receiving unit 204a, then, as shown in the example of A-2 in Figure 7, the first light-emitting unit 202a may be turned off, the second light-emitting unit 202b may be turned on, and imaging may be performed using the first and second light-receiving units 204a and 204b.

[0169] Furthermore, the imaging control unit (control unit) 220 may, while both the first and second light-receiving units 204a and 204b are able to continuously image the occupant's face without being affected by disturbances such as reflections from eyeglass lenses, learn, for example, the distance to the occupant's face and the positional relationship between feature points of the occupant's face on the captured image, and use the learning results to perform a three-dimensional position estimation of the occupant's eyes, etc., based on the imaging data obtained from only one of the light-receiving units.

[0170] In this case, the light-receiving unit on the side affected by the disturbance may suspend imaging for the period during which it is estimated that it continues to be affected by the disturbance.

[0171] (Second embodiment) Refer to Figure 8. Figure 8 shows an example configuration of an in-vehicle display device (in this case, a head-up display (HUD) device) equipped with an imaging device (and biological site detection device) using infrared light, and an example arrangement of the first and second imaging units in the in-vehicle display device.

[0172] The imaging device 201 described earlier (see A-2 in Figure 1) may be installed within a HUD (onboard display device) that illuminates a reflector (windshield, etc.) 2 of the vehicle 1 with image display light to display an image for the occupants 5 of the vehicle 1, or it may be installed integrally with an image display device (onboard display device). This makes it possible to promote miniaturization of the entire device (system).

[0173] In Figure 8, the HUD device 100 includes a housing 116, a light-transmitting cover 112 provided on the housing, the housing 116 (here, the housing 116 in the imaging device 201 shown earlier in Figure 1 is also used as the housing for the HUD device 100, and the reference numerals are the same), a display unit 102 for displaying images (equipped with a display panel 104 such as a liquid crystal display device), a curved mirror 108 such as a concave mirror that irradiates display light L10 toward the windshield 2, which acts as a reflector (reflective light-transmitting member), via the light-transmitting cover 112, and a flat mirror 106. Note that the curved mirror 108 and the flat mirror 106 can be rephrased as an optical system and optical elements.

[0174] Figure 8A-2 shows an example of the arrangement of the first and second imaging units 200a and 200b when they are incorporated into the HUD device 100.

[0175] In Figure 2A-2, region Z1 is the region from which the display light L10 is emitted. The first imaging unit 200a is positioned to the left of region Z1 as viewed from the crew member 5, and the second imaging unit 200b is positioned to the right of region Z1 as viewed from the crew member 5.

[0176] As explained earlier, the first and second light-emitting units 202a and 202b, and the first and second light-receiving units 204a and 204b are positioned in locations that cannot be seen by the occupant 5 in a normal posture, thus ensuring a practical design for the HUD device 100 as a whole.

[0177] Furthermore, because the image display device (in-vehicle display device) and the biological site detection device using infrared light, etc., are integrated, it is possible to obtain an image display device (in-vehicle display device) that is simple in configuration, multi-functional, compact, and suitable for installation inside a vehicle.

[0178] Furthermore, as explained earlier, in a preferred embodiment of the present invention, the light-receiving unit (infrared camera, etc.) itself is provided with a filter unit, thereby providing protection against visible light such as sunlight. Therefore, layout constraints are relaxed compared to the conventional design. In other words, the freedom of placement for the light-emitting unit (infrared LED, etc.) is improved.

[0179] Therefore, within the range where the first and second conditions described above are met, the light-receiving unit (infrared camera, etc.) and the light-emitting unit (infrared LED, etc.) can each be placed in an optimal position with a certain degree of freedom, thereby improving the freedom of placement and simplifying the design of the imaging device.

[0180] Furthermore, the imaging device may be integrated with the image display device or incorporated as an accessory.

[0181] Furthermore, the light-emitting unit and the light-receiving unit in the imaging device may be placed at positions far apart from each other. For example, the light-emitting unit may be placed (installed) at a position far from the imaging means, in a position that can directly illuminate the occupant's face (for example, on the ceiling of the vehicle or above or below the front pillar).

[0182] Although the present invention has been described above using several embodiments, the present invention is not limited to these and can be modified and applied in various ways.

[0183] In this specification, the term "vehicle" can be interpreted broadly as a means of transportation. Furthermore, "living organism" includes not only humans (human bodies) but also animals (animal bodies). In addition, dolls and models with structures similar to those of the human body can also be included as living organisms. Furthermore, the parts of a living organism to be detected (face, eyes, etc.) shall also be interpreted flexibly and broadly. For example, parts of a living organism can also include the entire living organism (e.g., the overall shape of the living organism).

[0184] Image display devices (vehicle-mounted display devices, or more broadly, display devices) include those used as simulators (for example, aircraft simulators, simulators as game devices, etc.).

[0185] As described above, according to this embodiment, even in environments where reflected light as ambient light is present, a decrease in the detection accuracy of biological (human body) parts based on captured images can be suppressed using a simplified configuration.

[0186] The present invention is not limited to the exemplary embodiments described above, and those skilled in the art will be able to easily modify the exemplary embodiments described above to the extent included in the claims. [Explanation of Symbols]

[0187] 1...Vehicle, 2...Reflector (windshield, etc.), 4...Instrument panel (or dashboard), 5...Occupant (driver, etc., person), 6...Eyes (viewpoint), 8...Steering wheel, 9...Eyeglasses, 12...Window section (opening), 13...Filter section (RF filter, etc.), 16L, 16R...Front pillar (A-pillar), 100...HUD device ( Image display device, in-vehicle display device), 102...Display unit, 104...Display panel such as liquid crystal display device, 106...Flat mirror, 108...Curved mirror (concave mirror, etc.), 110...Biological part detection device (face detection device), 112...Translucent cover, 200...Imaging unit, 201...Imaging device (reflective imaging device), 202 (202a, 202b)...Light-emitting unit (illumination unit, illumination unit, Infrared LED, etc.), 204 (204a, 204b)... Light receiving unit (light receiving unit, imaging unit, infrared camera, etc.), 220... Imaging control unit (control unit), 230... Biological site detection processing unit, 231... Image acquisition unit, 232... Reflected light pattern extraction unit, 233... Reflected light pattern detection unit, 236... Biological site detection unit (face detection unit), 238... Result output unit, Pc... Reference Eye point (center eye point located at the midpoint of both eyes), Lc...Reference straight line (straight line), LI...Illumination light path, Li...Imaging light path, FR...Field of view (field of view range), PrI (PrI1, PrI2)...Reflection point of illumination light on the reflector, Pri (Pri1, Pri2)...Reflection point of imaging light on the reflector, L10...Display light, Z1...Region from which display light (visible light) is emitted.

Claims

1. An imaging device used to image parts of a subject that is a vehicle occupant, Of the light used to detect the body parts of the subject, which are the occupants of the vehicle, The light reflected by the reflector and illuminating the part of the subject is used as illumination light. When the illumination light is reflected by the part of the subject that is the target of imaging, and further reflected by the reflector and returned to the imaging device, the resulting light is used as the imaging light. A light-emitting unit that emits the aforementioned illumination light, A light-receiving unit that receives the aforementioned imaging light, A control unit that controls the operation of the light-emitting unit and the light-receiving unit, It has an imaging unit equipped with, The coordinate axis of the vehicle body coordinate system, which is positioned in the longitudinal direction of the aforementioned vehicle, is defined as the X-axis. The Y-axis is a coordinate axis that is perpendicular to the X-axis and is positioned in the left-right direction of the vehicle. The coordinate axis that is orthogonal to the X and Y time axes and is positioned in the height direction of the vehicle is defined as the Z axis. The reference eye point Pc is defined as a point in the vehicle body coordinate system that is determined to be the typical eye position of the occupant of the vehicle as the subject, The reference straight line Lc is defined as a straight line parallel to the X-axis that passes through the reference eye point Pc of the occupant of the vehicle, which is the subject of the photograph, The representative optical path of the illumination light between the reference eye point Pc and the reflector is defined as the illumination optical path LI. The representative optical path of the imaging light between the reference eye point Pc and the reflector is defined as the imaging optical path Li. Let θIc be the angle that the illumination light path LI makes with the reference straight line Lc. When the angle that the imaging optical path Li makes with the reference straight line Lc is θic, The light-emitting unit and the light-receiving unit are arranged to satisfy the first and second conditions. The first condition is, θic < θic The second condition mentioned above is, The light-emitting unit and the light-receiving unit are located in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scx that includes the reference eye point Pc, parallel to the ZX plane in the vehicle body coordinate system. Or, The light-emitting unit and the light-receiving unit are located in the same space when the space of the vehicle body coordinate system is divided into two by a plane Scy that includes the reference eye point Pc, parallel to the XY plane in the vehicle body coordinate system. Imaging device.

2. The light-emitting unit that emits the illumination light is provided with a first light-emitting unit and a second light-emitting unit, The control unit can switch which of the first and second light-emitting units is used to illuminate the subject. The second light-emitting part is arranged to satisfy the third condition, The third condition mentioned above is, The second light-emitting part is located in the space opposite to the space where the first light-emitting part is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc. Or, The second light-emitting part is located in the space opposite to the space where the first light-emitting part is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc. The imaging apparatus according to claim 1.

3. The imaging unit includes a first imaging unit and a second imaging unit. The second imaging unit is arranged to satisfy the fourth condition, The fourth condition is, The second imaging unit is located in the space opposite to the space where the first imaging unit is located, when the space of the vehicle body coordinate system is divided into two by a plane Scx that is parallel to the ZX plane in the vehicle body coordinate system and includes the reference eye point Pc, Or, The second imaging unit is located in the space opposite to the space where the first imaging unit is located, when the space of the vehicle body coordinate system is divided into two by a plane Scy that is parallel to the XY plane in the vehicle body coordinate system and includes the reference eye point Pc. and, The light-emitting unit and light-receiving unit in the first imaging unit are referred to as the first light-emitting unit and the first light-receiving unit, When the light-emitting unit and light-receiving unit in the second imaging unit are made into a second light-emitting unit and a second light-receiving unit, The first light-emitting unit and the first light-receiving unit are arranged to satisfy the first condition and the second condition. The second light-emitting unit and the second light-receiving unit are also arranged to satisfy the first and second conditions. The control unit can select which of the first light-emitting unit, the second light-emitting unit, the first light-receiving unit, or the second light-receiving unit to use. The imaging apparatus according to claim 1.

4. The light-receiving unit has a filter unit that blocks visible light and transmits the illumination light and the imaging light. The light-emitting part is positioned so that it cannot be seen, at least directly, by the occupants of the vehicle. The imaging apparatus according to claim 1.

5. The imaging device is An image display device is provided within the vehicle, which illuminates the reflector of the vehicle with display light to show an image to the occupant of the vehicle, Or, The image display device is provided integrally with the image display device, The imaging apparatus according to claim 1.

6. A biological body part detection device that detects the body parts of occupants of a vehicle as a subject based on captured images, An imaging device as described in any one of paragraphs 1 to 5, A biological site detection processing unit that detects a part of a living organism based on the image acquired by the aforementioned imaging device, Having, A device for detecting biological tissue parts.

Images