Shooting control device

Abstract

The present invention provides a photographing control device that reduces the cost of the device and achieves condition detection such as object detection in a photographing target area with high accuracy. The photographing control device includes, for example, an exposure control unit that controls the exposure length of a photographing unit capable of photographing a passenger-accessible area in a vehicle compartment to at least a first exposure period and a second exposure period longer than the first exposure period; an image acquisition unit that acquires first photographing image data photographed during the first exposure period and second photographing image data photographed during the second exposure period; and a processing unit that processes the first photographing image data and the second photographing image data, respectively.

Description

Technical Field

[0001] The invention relates to a shooting control device. Background Art

[0002] In the past, it is known that there is a technology that uses a captured image obtained by capturing a target area with a capturing unit (camera) to detect objects reflected in the target area, the posture of the object, the surrounding conditions, etc., and to use it for situation grasping and control. When the captured image is used for situation grasping and control in this way, its accuracy is easily affected by the quality of the captured image. For example, due to the surrounding environment (brightness difference in the case of a visible light camera, temperature difference in the case of an infrared camera, etc.), there is a situation where "brightness saturation" or "underexposure" occurs in a part of the captured image, and the content of the captured image cannot be correctly recognized. Therefore, various technologies have been proposed to capture bright images and dim images, generate composite images, obtain dynamic range, and improve recognition.

[0003] Patent Document 1: Japanese Patent Application No. 2018-503515

[0004] Patent Document 2: Japanese Patent Application Publication No. 2019-16893

[0005] However, when performing image synthesis as proposed in the above-mentioned literature, the number of bits of the synthesized image increases, which increases the processing load, thereby requiring a high-performance processing device and a heat dissipation measure to cope with the increase in heat caused by the high processing load, etc., thereby causing a problem of increasing the overall cost of the device. Summary of the invention

[0006] Therefore, one of the objects of the present invention is to provide an imaging control device that can reduce the device cost and achieve situation detection such as object detection in an imaging target area with high accuracy.

[0007] The imaging control device according to the embodiment of the present invention includes, for example, an exposure control unit that controls the exposure length of an imaging unit capable of imaging a passenger-seating area in a vehicle compartment to at least a first exposure period and a second exposure period longer than the first exposure period; an image acquisition unit that acquires first image data captured during the first exposure period and second image data captured during the second exposure period; and a processing unit that processes the first image data and the second image data, respectively. According to this configuration, for example, by performing imaging during the first exposure period, which is relatively short in exposure period, there is a possibility that an image with insufficient brightness may be obtained in an area relatively far from the imaging unit, but imaging in an area relatively close to the imaging unit in which the image is not too bright, so-called "brightness saturation", can be performed. On the other hand, by performing imaging during the second exposure period, which is relatively longer in exposure period than the first exposure period, there is a possibility that an image with excessive brightness may be obtained in an area relatively close to the imaging unit, but imaging in an area relatively far from the imaging unit in which the image is not too dark, so-called "underexposure", can be performed. In other words, by switching the exposure period and processing the captured image data separately, it is possible to capture an area that is not too bright and an area that is not too dark without the need for the composite processing of the captured images. As a result, it is possible to reduce the cost of the device due to the reduction of the composite processing, and to obtain with high accuracy either the captured image data that is easy to identify the interior conditions of the area close to the imaging unit or the captured image data that is easy to identify the interior conditions of the area far from the imaging unit, and to use them for control.

[0008] The exposure control unit of the shooting control device according to the embodiment of the present invention may be controlled to alternately switch between the first exposure period and the second exposure period. According to this structure, for example, the first captured image data and the second captured image data can be efficiently acquired, and the state recognition of the first area and the state recognition of the second area can be performed at the same time, so that the state recognition can be performed smoothly.

[0009] The exposure control unit of the shooting control device according to the embodiment of the present invention may, for example, correct the length of the first exposure period in this shooting based on the average brightness value of the first area close to the shooting unit in the first shooting image data captured during the first exposure period in the previous shooting, and correct the length of the second exposure period in this shooting based on the average brightness value of the second area farther from the shooting unit than the first area in the second shooting image data captured during the second exposure period in the previous shooting. According to this structure, for example, it is possible to easily set the exposure period corresponding to the change in the surrounding conditions during shooting, and obtain the first shooting image data and the second shooting image data with appropriate brightness expression that makes it easier to recognize the shooting content.

[0010] The exposure control unit of the shooting control device according to the embodiment of the present invention can, for example, control the length of the exposure period of the infrared shooting unit that shoots infrared images as the shooting unit. According to this structure, for example, the first shot image data and the second shot image data that are appropriately expressed in brightness and are easier to identify the shot content can be obtained in the vehicle compartment without being recognized by passengers, regardless of day or night (bright state or dark state).

[0011] The exposure control unit of the shooting control device according to the embodiment of the present invention may, for example, synchronize the irradiation period of infrared light irradiated to the area that the infrared shooting unit can shoot with the first exposure period and the second exposure period. According to this structure, for example, the infrared light irradiation can be performed only during shooting, thereby contributing to the reduction of shooting costs, and the reduction of heat generation can be achieved, thereby contributing to the simplification of the heat countermeasure structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is an illustrative and schematic diagram for explaining the structure of a vehicle on which the imaging control device according to the embodiment can be mounted and the imaging range of the vehicle.

[0013] Figure 2 These are illustrative and schematic diagrams for explaining how a passenger sitting on a front seat is photographed in a vehicle equipped with the photographing control device according to the embodiment and the arrangement of the photographing unit.

[0014] Figure 3 This is an exemplary and schematic block diagram showing a brief hardware configuration of a monitoring control system including a photography control device (photography control unit) according to an embodiment.

[0015] Figure 4These are illustrative and schematic diagrams for explaining the exposure period of the front seat and the rear seat and the driving period of the illumination (infrared LED) of the imaging control device according to the embodiment.

[0016] Figure 5 This is an illustrative and schematic diagram showing an image of the interior of a vehicle cabin captured when exposure is performed in the first exposure period in the imaging control device according to the embodiment.

[0017] Figure 6 This is a diagram schematically illustrating an example of a captured image of the interior of a vehicle cabin captured when exposure is performed during the second exposure period in the imaging control device according to the embodiment.

[0018] Figure 7 This is an illustrative and schematic flowchart for explaining an operation mode of the imaging control device including a series of processes executed by the imaging control device according to the embodiment.

[0019] Description of Reference Numerals

[0020] 10…vehicle; 10a…compartment; 14…front seats; 16…rear seats; 18, 18a, 18b, 18c, 18d…passengers; 20…seat belts; 22…infrared imaging unit; 24…infrared lighting unit; 26…ECU; 28…equipment control unit; 32…output device; 34…on-board equipment; 36…seat belt sensor; 38…CPU; 38a…exposure control unit; 38b…image acquisition unit; 38c…processing unit; 100…monitoring control system; M1…first captured image; M2…second captured image. DETAILED DESCRIPTION

[0021] Hereinafter, embodiments of the present disclosure will be described based on the drawings. The configuration of the embodiments described below and the functions and effects brought about by the configuration are only examples and are not limited to the following descriptions.

[0022] Figure 1 1 and 2 are illustrative and schematic diagrams for explaining the structure of a vehicle 10 to which the imaging control device according to the embodiment is applied and the imaging range of an imaging unit of the vehicle 10 .

[0023] The imaging control device of this embodiment performs control for acquiring imaging image data for detecting (recognizing) the condition in the cabin 10a of the vehicle 10. The condition in the cabin 10a includes, for example, whether a passenger is present in the cabin 10a (riding condition), the posture of the passenger, the use condition of the equipment in the cabin 10a by the passenger, etc. In addition, the condition in the cabin 10a may also include the presence or absence of objects (luggage, living things, etc.) brought into the cabin 10a, etc.

[0024] In the present embodiment, the vehicle 10 equipped with the shooting control device (shooting control unit) may be, for example, a car with an internal combustion engine (not shown) as a driving source, i.e., an internal combustion engine car, or a car with an electric motor (not shown) as a driving source, i.e., an electric car, a fuel cell car, etc. In addition, it may be a hybrid car with both an internal combustion engine and an electric motor as driving sources, or a car with other driving sources.

[0025] like Figure 1 As shown in the example, the vehicle 10 is configured as a vehicle compartment 10a for passengers (driver, fellow passengers) not shown in the figure. In the vehicle compartment 10a, for example, a front seat 14 (front seat) is provided with a driver's seat 14a (driver's seat) opposite to the steering wheel 12, and a passenger seat 14b (front passenger seat) next to the seat 14a. In addition, for example, a rear seat 16 (rear seat) is provided with passenger seats 16a, 16b, and 16c. Figure 1 , an example of a five-seat vehicle (five-person vehicle) is shown in which the front row of seats 14 is two seats and the rear row of seats 16 is three seats. The seat layout can be appropriately selected according to the vehicle model, etc. For example, the rear row of seats 16 can also be composed of two seats (together with the front row of seats 14, it can be a four-person vehicle). In addition, it can be a three-row seat structure or a structure with more than three rows. In addition, it can also be a structure in which the front row of seats 14 for passengers is a single row, and the rear of the front row of seats 14 is a luggage space.

[0026] Figure 2 These are illustrative and schematic diagrams for explaining how to photograph a passenger 18 sitting in a front seat 14 (seat 14 a ) in a vehicle 10 equipped with the photographing control device according to the embodiment and the arrangement of photographing units.

[0027] The front seat 14 (seat 14a) is composed of a seat cushion 14c, a seat back 14d, a headrest 14e, etc., and a frame supporting the seat cushion 14c is fixed to the floor surface of the vehicle compartment 10a by a guide rail (not shown) so that the position can be adjusted along the front-rear direction of the vehicle. In addition, various adjustment mechanisms such as a reclining mechanism are provided in the front seat 14, and the posture of the passenger 18 (driver) wearing the seat belt 20 and holding the steering wheel 12 can be adjusted in a manner that is easy to maintain in a state suitable for driving.

[0028] As described above, the shooting control device of the present embodiment is configured as a shooting unit, for example, an infrared shooting unit 22, at a position where the inside of the car 10a can be seen, in order to detect (recognize) the situation inside the car 10a. In addition, an infrared lighting unit 24 is configured near the infrared shooting unit 22. In the case of the present embodiment, the infrared shooting unit 22 and the infrared lighting unit 24 are configured at the approximately central part of the inner upper part of the front window of the vehicle 10 (for example, near the rearview mirror). In addition, the infrared shooting unit 22 can also be configured at other positions as long as it is a position where the inside of the car 10a can be seen. In addition, as long as infrared light can be irradiated to the entire shooting area of ​​the infrared shooting unit 22, the position of the infrared lighting unit 24 can be appropriately changed, and the infrared shooting unit 22 and the infrared lighting unit 24 can also be configured at different positions. In addition, the infrared shooting unit 22 and the infrared lighting unit 24 can also be configured as an integrated camera unit.

[0029] The infrared camera unit 22 is, for example, a digital camera that has built-in imaging elements such as CCD (Charge Coupled Device) and CIS (CMOS image sensor) that correspond to infrared imaging. The infrared camera unit 22 can sequentially output animation data (photographed image data) at a predetermined frame rate. The infrared lighting unit 24 can be, for example, an LED (Light Emitting Diode) lamp that emits infrared rays. Figure 1 As indicated by the hatched lines in the figure, for example, a single camera can be used to capture an area in the vehicle 10a where passengers 18 can sit (a certain area of ​​the front seats 14 and the rear seats 16). In addition, the infrared lighting unit 24 sets the irradiation angle and irradiation intensity in such a way that infrared light can be irradiated to an area at least including the imaging range of the infrared camera 22.

[0030] The infrared imaging unit 22 adjusts and fixes the viewing angle and posture so that the imaging area includes at least each passenger 18 sitting in the front seat 14 and the rear seat 16. Figure 2 As shown, when the passenger 18 sits in the seat 14a (driver's seat), the face and at least the upper body of the passenger 18 are included in the photographing area of ​​the infrared photographing unit 22. Similarly, when the passenger 18 sits in the seat 14b (passenger seat), the face and at least the upper body of the passenger 18 are included in the photographing area of ​​the infrared photographing unit 22. In addition, when the passenger 18 sits in the seats 16a to 16c of the rear seats 16, the face and at least the upper body of the passenger 18 are included in the photographing area of ​​the infrared photographing unit 22.

[0031] The infrared lighting unit 24 irradiates infrared light to the passenger 18 sitting on the seat 14a, seat 16a, etc., but the infrared light is not recognized as light by the human eye, so even if the infrared light is irradiated in the direction of the face of the passenger 18, the passenger 18 will not feel dazzled. Therefore, the comfort of the passenger 18 in the vehicle compartment 10a can be ensured, and the infrared imaging unit 22 can easily realize the imaging of the passenger 18, etc.

[0032] However, if Figure 1 As shown, when the infrared camera 22 and the infrared lighting unit 24 are arranged in the front part of the vehicle cabin 10a, there is a case where the image quality of the infrared image captured cannot be obtained to the extent that the imaged content can be recognized for the entire area of ​​the front seat 14 and the rear seat 16. For example, the distance from the infrared camera 22 to the rear seat 16 is greater than the distance from the infrared camera 22 to the front seat 14, so there is a tendency that the captured infrared image becomes darker due to the increased attenuation of the reflected light (infrared light) on the rear seat 16 side compared with the front seat 14 side. In this case, it is assumed that the infrared image is captured in a manner that can obtain an infrared image with a brightness that can be used by control or the like near the area of ​​the front seat 14 relatively close to the infrared camera 22 (hereinafter, there is a case of being referred to as the first area). In this case, in the captured infrared image, the attenuation of reflected light increases near the area of ​​the rear seats 16 relatively far from the infrared camera unit 22 (hereinafter referred to as the second area), thereby increasing the tendency of "underexposure". As a result, the second area becomes a captured image that is difficult to use through control, etc. On the contrary, it is assumed that an infrared image with brightness that can be used through control, etc. is captured near the second area including the rear seats 16. In this case, in the captured image, the attenuation of reflected light is small near the first area including the front seats 14 relatively close to the infrared camera unit 22, so the amount of light received increases, thereby increasing the tendency of "brightness saturation". As a result, the first area becomes a captured image that is difficult to use through control, etc.

[0033] Therefore, in the imaging control device of the present embodiment, when the infrared imaging unit 22 sequentially captures the interior of the vehicle compartment 10a and outputs infrared captured image data (hereinafter, simply referred to as captured image data) at a predetermined frame rate, exposure control is performed in a manner such that, for example, first captured image data captured by exposure for a first area including the front seats 14 and second captured image data captured by exposure for a second area including the rear seats 16 are alternately output. In other words, the first captured image data captured by infrared that can express the first area with appropriate brightness is obtained for the front seats 14, and the second captured image data captured by infrared that can express the second area with appropriate brightness is obtained for the rear seats 16, and each captured image data is processed independently, for example, and is used for various monitoring controls. The monitoring control using the first captured image data and the second captured image data includes, for example, detection control of the presence or absence of the passenger 18 and the object carried, recognition control thereof, detection control of the presence or absence of the seat belt 20 being worn and warning control when the seat belt 20 is not worn, and gesture detection control when the passenger 18 operates the vehicle-mounted equipment. In addition, it can also be used for detection control of forgotten items in the vehicle compartment 10a. In addition, the first captured image data and the second captured image data are further analyzed, thereby, for example, determining the degree of drowsiness of the driver, controlling the airbag, and identifying the passenger 18 sitting in the front seat 14 and the rear seat 16, such as adult male, adult female, child, etc., so that it is also possible to perform appropriate control for each. In addition, in these cases, the synthesis processing of the first captured image data and the second captured image data is not performed, and each captured image data is used for control separately. As a result, the synthesis processing of the image is not required, the processing load can be reduced, and it can contribute to the reduction of the device cost.

[0034] Figure 3 1 is an exemplary and schematic block diagram showing the structure of a photographing control device (photographing control unit) that performs the above-mentioned photographing control. Figure 3 A monitoring control system 100 is shown, which includes a device control unit 28, which performs various monitoring controls in the vehicle 10 when an ECU 26 (Electronic Control Unit) functions as a shooting control device (shooting control unit) uses the first shooting image data and the second shooting image data obtained by the shooting control device.

[0035] like Figure 3As shown, ECU26 controls the infrared camera unit 22 and the infrared lighting unit 24 to perform photography, and provides the acquired first captured image data and second captured image data to the device control unit 28. Then, the device control unit 28 controls the output device 32 and the vehicle-mounted device 34 which are electrically connected via the in-vehicle network 30. The in-vehicle network 30 is configured as a CAN (Controller Area Network), for example. In addition, the in-vehicle network 30 is electrically connected to various sensors that are referenced when the control is performed by the device control unit 28. Figure 3 In the case of , seat belt sensors 36 ( 36 a to 36 e ) that detect whether the seat belts 20 provided in the front seats 14 and the rear seats 16 are worn are shown as an example.

[0036] For example, ECU26 has ROM40, RAM42, SSD44 (flash memory), etc. in addition to the device control unit 28 and CPU38. For example, CPU38 can read out a program installed and stored in a non-volatile storage device such as ROM40, and perform calculation processing according to the program. RAM42 temporarily stores various data used in the calculation of CPU38. SSD44 is a rewritable non-volatile storage unit that can store data even when the power of ECU26 is disconnected. In addition, the device control unit 28, CPU38, ROM40, RAM42, etc. can be integrated in the same package. In addition, ECU26 can also be a structure that uses other logical operation processors, logic circuits, etc. such as DSP (Digital Signal Processor) instead of CPU38. In addition, HDD can be set instead of SSD44, and SSD44 and HDD can also be set separately from ECU26. In addition, ECU26 that functions as a shooting control device (shooting control unit) can also be implemented in the CPU of multiple other ECUs that control the vehicle 10. In addition, in Figure 3 , an example is shown in which the device control unit 28 is installed as an independent module in the ECU 26. In other embodiments, the device control unit 28 may be realized by executing a program installed in the ROM 40 or the like in the CPU 38, or may be provided independently of the ECU 26.

[0037] The CPU 38 reads out a program installed and stored in a storage device such as the ROM 40 and executes the program, thereby realizing the exposure control unit 38 a , the image acquisition unit 38 b , the processing unit 38 c , and the like.

[0038] The exposure control unit 38a controls the exposure length of the infrared imaging unit 22 that can capture the area in the vehicle 10a where the passengers 18 can sit (a first area including the front seats 14 and a second area including the rear seats 16) to at least a first exposure period and a second exposure period that is longer than the first exposure period. In addition, the image acquisition unit 38b acquires the first captured image data captured during the first exposure period and the second captured image data captured during the second exposure period, and temporarily stores them in a storage device such as the RAM 42 in sequence. The processing unit 38c processes the first captured image data and the second captured image data, respectively, and provides them to the device control unit 28 to perform various controls.

[0039] For example, the exposure control unit 38a controls the infrared imaging unit 22 so that the first exposure period and the second exposure period are alternately switched. In other words, the image data captured by the infrared imaging unit 22 constitutes animation data in which frames representing (capturing) the first area including the front seats 14 with easily recognizable brightness and frames representing (capturing) the second area including the rear seats 16 with easily recognizable brightness alternately exist. For example, odd-numbered frames are captured during the first exposure period, and even-numbered frames are captured during the second exposure period. In this case, for example, 256 grayscales of 0 to 255 ("0" is darker, "255" is brighter) are used to represent the brightness in the captured image. In addition, the brightness of the captured image suitable for recognizing (detecting) the situation in the vehicle cabin 10a is set to "128" in advance through experiments or the like. In this case, when capturing odd-numbered frames, the first exposure period is set so that the average brightness of the first area is, for example, "128". In addition, when capturing even-numbered frames, the second exposure period is set so that the average brightness of the second area is, for example, "128". By performing the shooting in the first exposure period with odd-numbered frames and performing the shooting in the second exposure period with even-numbered frames, the time interval between the shooting in the first exposure period and the shooting in the second exposure period can be minimized, so that the conditions of the first area including the front seats 14 and the conditions of the second area including the rear seats 16 can be regarded as being obtained at the same time. In addition, the installation posture of the infrared imaging unit 22 is fixed, so the position corresponding to the first area and the position corresponding to the second area in the captured image data are known. Therefore, it is easy to calculate the average brightness value of the first area and the second area in the captured image data.

[0040] As described above, the first area is relatively close to the infrared imaging unit 22, so when infrared light is irradiated from the infrared illumination unit 24, the attenuation of the reflected light is relatively small, so the first area becomes relatively bright. Figure 4As shown in FIG. 1 , the first exposure period for making the average brightness of the first area on the front seat 14 side "128" is controlled so that the ON period is relatively short (line E1). On the other hand, the second area is relatively far from the infrared imaging unit 22 than the first area, so when the infrared light is irradiated from the infrared lighting unit 24, the attenuation of the reflected light is relatively large, so the second area is relatively darker than the first area. Therefore, as shown in FIG. Figure 4 As shown, the ON period of the second exposure period for making the average brightness of the second area on the rear seat 16 side "128", for example, is controlled to be relatively longer than the first exposure period (line E2).

[0041] In addition, the infrared camera 22 makes infrared rays emitted from an object or infrared rays reflected by an object visible. Therefore, there is a case where the brightness of the captured image changes with the change of the temperature of the captured object and the surrounding temperature. Therefore, the exposure control unit 38a corrects the length of the first exposure period in this shooting based on the average brightness value of the first area close to the infrared camera 22 in the first captured image data captured during the first exposure period in the previous shooting, which is stored in the RAM 42, etc. Similarly, the exposure control unit 38a corrects the length of the second exposure period in this shooting based on the average brightness value of the second area farther from the infrared camera 22 than the first area in the second captured image data captured during the second exposure period in the previous shooting, which is stored in the RAM 42, etc. In this way, the exposure period is corrected based on the captured image data captured in the previous shooting, so that, for example, even if the surrounding conditions (such as temperature conditions) at the time of shooting change, it is easy to cope with the change, and the first captured image data and the second captured image data expressed with appropriate brightness that make it easier to recognize the captured content can be obtained. In addition, for the captured image data referenced for correction, for example, it is sufficient to refer to odd frames during the first exposure period and even frames during the second exposure period. The exposure period can also be fine-tuned by referring to odd frames or even frames multiple times in the past.

[0042] In addition, if Figure 4 As shown, the exposure control unit 38a can drive the infrared ray illumination unit 24 to irradiate infrared rays during the first illumination driving period (line F1) in synchronization with the first exposure period, and can drive the infrared ray illumination unit 24 to irradiate infrared rays during the second illumination driving period (line F2) in synchronization with the second exposure period. In this case, infrared light irradiation is performed only during shooting, which can contribute to a reduction in shooting costs, and can reduce heat generation, which helps simplify the heat countermeasure structure.

[0043] As described above, exemplary and schematic captured image data (captured image) of a case where the exposure period is changed in the first area including the front seat 14 and the second area including the rear seat 16 is shown in FIG. Figure 5 as well as Figure 6 . Figure 5 The first captured image data (first captured image M1) is mainly the result of capturing the first area including the front seat 14 during the first exposure period in a manner that can be easily controlled and expressed with brightness. Figure 6 Mainly, the second captured image data (second captured image M2 ) is the result of capturing the image during the second exposure period so that the second area including the rear seat 16 is expressed with brightness that is easily controlled.

[0044] like Figure 1 As shown, the infrared camera unit 22 and the infrared lighting unit 24 are arranged in the approximate center of the inner upper part of the front window, so that in the case of the front seat 14 (the first area), the reflected light of the infrared light irradiated by the infrared lighting unit 24 reaches the infrared camera unit 22 in a state with less attenuation. Therefore, the exposure control unit 38a causes the infrared camera unit 22 to shoot during a relatively short first exposure period, so that the front seat 14 side can shoot the first shot image M1 with brightness that is easy to identify and suppresses brightness saturation. Therefore, it is easy to identify (detect) the passenger 18a sitting in the seat 14a (driver's seat) of the front seat 14, the seat belt 20a worn by the passenger 18a, etc. Similarly, it is easy to identify (detect) the passenger 18b sitting in the seat 14b (passenger seat) of the front seat 14, the seat belt 20b worn by the passenger 18b, etc. As a result, the first captured image M1 (first captured image data) captured during the first exposure period can well perform detection control of the presence or absence of the passenger 18 in the front seat 14 and the carried objects (luggage, etc.), recognition control thereof, detection control of the presence or absence of the seat belt 20, and gesture detection control when the passenger 18 in the front seat 14 operates the vehicle-mounted equipment. In addition, the storage box 48 between the seat 14a and the seat 14b and its surroundings are also easy to recognize without brightness saturation. Therefore, the first captured image M1 (first captured image data) captured during the first exposure period can also be used for detection of forgotten items in the front seat 14, etc.

[0045] On the other hand, in the case of the rear seat 16 (second area), the reflected light of the infrared light irradiated by the infrared lighting unit 24 is attenuated compared with the front seat 14, and the image is captured during the relatively short first exposure period. Therefore, if the rear seat 16 side is visible based on the infrared reflected light captured (received) by the infrared imaging unit 22 during the first exposure period, underexposure is likely to occur. As a result, it is difficult to identify the passenger 18c sitting in the seat 16a (seat behind the driver's seat) of the rear seat 16, the passenger 18d sitting in the seat 16b (seat behind the passenger seat) of the rear seat 16, and the like. In addition, it becomes difficult to distinguish whether the passenger 18c and the passenger 18d are wearing the seat belt 20. However, as described above, in the present embodiment, the first captured image M1 is used for the detection control of the passenger 18 and the load, the recognition control thereof, the detection control of the wearing of the seat belt 20 and the warning control when the seat belt 20 is not worn, the gesture detection control when the passenger 18 operates the vehicle-mounted equipment, and the like. In other words, the purpose is not to allow the passenger 18 and the like to directly recognize the first captured image M1 (first captured image data). Therefore, the first captured image M1 (first captured image data) for recognizing (detecting) the situation on the front seat 14 side has an underexposed area on the rear seat 16 side, and even if the rear seat 16 side is difficult to recognize, there will be no problem.

[0046] The exposure control unit 38a causes the infrared imaging unit 22 to perform imaging during the second exposure period which is relatively longer than the first exposure period, so that the infrared imaging unit 22 can receive infrared reflected light for a longer period of time. Figure 6 As shown, the second captured image M2 with easily recognizable brightness and suppressed underexposure can be captured on the rear seat 16 side. Therefore, it is easy to recognize (detect) the passenger 18c sitting in the seat 16a of the rear seat 16, the seat belt 20c worn by the passenger 18c, and the like. Similarly, it is easy to recognize (detect) the passenger 18d sitting in the seat 16b of the rear seat 16, the seat belt 20d worn by the passenger 18d, and the like. As a result, the second captured image M2 (second captured image data) captured during the second exposure period can well perform detection control of the passenger 18 in the rear seat 16 and the carried objects (luggage, etc.), recognition control thereof, detection control of the wearing of the seat belt 20, and gesture detection control when the passenger 18 in the rear seat 16 operates the vehicle-mounted equipment. In addition, the seat 16c between the seat 16a and the seat 16b and the object 50 (for example, clothes, etc.) placed around it can also be easily recognized without underexposure. Therefore, the second captured image M2 (second captured image data) captured during the second exposure period can be used for detection of a forgotten item in the rear seat 16 or the like.

[0047] On the other hand, in the case of the front seat 14 (first area), the reflected light of the infrared light irradiated by the infrared lighting unit 24 is in a state with a smaller attenuation than the rear seat 16, and is photographed during the second exposure period longer than the first exposure period that becomes an appropriate brightness. Therefore, if the front seat 14 side is visible based on the infrared reflected light photographed (received) by the infrared photographing unit 22 during the second exposure period, brightness saturation is likely to occur. As a result, it is difficult to identify the passenger 18a sitting in the seat 14a of the front seat 14, the passenger 18b sitting in the seat 14b of the front seat 14, etc., and it is also difficult to distinguish whether the passenger 18a and the passenger 18b are wearing the seat belt 20. However, as described above, in the present embodiment, the second photographed image M2 is used for the detection control of the passenger 18 and the load, the recognition control thereof, the detection control of the wearing of the seat belt 20 and the warning control when the seat belt 20 is not worn, the gesture detection control when the passenger 18 operates the vehicle-mounted equipment, etc. In other words, the purpose is not to allow the passenger 18 to directly see the second captured image M2 (second captured image data). Therefore, in the second captured image M2 (second captured image data) for recognizing (detecting) the situation on the rear seat 16 side, there is a saturated brightness area on the front seat 14 side, and even if the front seat 14 side is difficult to recognize, it will not cause any problem. In addition, since the seat 14a, seat 14b, storage box 48, etc. of the front seat 14 have less infrared radiation than the passenger 18, Figure 6 In the example, the state in which brightness saturation does not occur is represented.

[0048] return Figure 3The device control unit 28 obtains the first captured image data and the second captured image data captured by the switching control of the first exposure period and the second exposure period through the processing unit 38c, thereby performing various controls. For example, in the first captured image data (first captured image) obtained, the device control unit 28 determines that the passenger 18a is not wearing the seat belt 20a, regardless of whether the passenger 18a is detected in the seat 14a (driver's seat) or not, when the signal of the seat belt sensor 36a indicating the wearing of the seat belt 20a cannot be received. Therefore, the device control unit 28 can cause the output device 32 (for example, a speaker, a display light, a display device, etc.) to output an alarm indicating that the seat belt is not worn. In addition, when the device control unit 28 detects that the passenger 18c in the rear seat 16 performs a predetermined gesture (for example, an operation of raising the left hand above the head and moving it forward and backward) based on the second captured image data (second captured image), it determines that the passenger 18c performs a gesture of opening and closing the sunroof. In this case, the device control unit 28 can drive the vehicle-mounted device 34 (e.g., the sunroof opening and closing motor, etc.) to open and close the sunroof. In this case, even the passenger 18 sitting in a seat where the sunroof opening and closing operation is difficult can smoothly and easily open and close the sunroof.

[0049] The control of the device control unit 28 described above is an example. For example, the seat belt sensors 36b to 36e can be used for other passengers 18 to output an alarm when the seat belt 20 is not worn. In this case, the seat belt 20 is not judged as not worn for seats where the passengers 18 are not detected (recognized) or seats where luggage is loaded. In addition, the device control unit 28 can also make audio devices and navigation systems as vehicle-mounted devices 34, so that other passengers other than the specific passengers 18 can operate the vehicle-mounted devices 34 that are difficult to operate, thereby improving convenience.

[0050] Figure 7 This is an exemplary and schematic flowchart of a series of processes executed by the monitoring control system 100 including the imaging control device according to the embodiment.

[0051] The ECU 26 always monitors whether a status confirmation start request signal in the vehicle cabin 10a is received (S100), and if the status confirmation start request signal is not received (No in S100), the process is temporarily terminated. The status confirmation start request signal may be generated when the ignition switch of the vehicle 10 is turned on and received by the ECU 26, or may be generated by a switch operation of the passenger 18 and received by the ECU 26.

[0052] In S100, when ECU 26 receives the status confirmation start request signal (Yes in S100), coefficient n for specifying the frame number during imaging is reset to "n=1" (S102). The frame rate of infrared imaging unit 22 in this embodiment is, for example, "30".

[0053] Next, the exposure control unit 38a shoots the 2n-1th frame (the first frame: an odd-numbered frame) during the first exposure period (initial value) for the front seat 14 determined in advance by experiments or the like (S104), and temporarily stores it in the RAM 42. As described above, depending on the environment during shooting, brightness saturation and underexposure may occur even during the same exposure period. Therefore, the image acquisition unit 38b determines whether the first captured image data captured (acquired) is data that can confirm the condition of the front seat (front seat 14) (S106). For example, it is determined whether the average brightness of the first area corresponding to the front seat 14 is a predetermined value such as "128" (for example, including the allowable value ±2). Then, if the average brightness value does not fall within the range of "128±2", it is determined that the condition of the front seat (front seat 14) cannot be recognized ("No" in S106). Then, the exposure control unit 38a calculates a correction value (e.g., -1ms, etc.) of the exposure period that enables confirmation of the status of the front seat (first area) in the current shooting environment (S108), and determines to correct the first exposure period of the 2n+1th frame (third frame: odd-numbered frame). In addition, at the same time, the illumination period of the infrared illumination unit 24 is determined corresponding to the corrected first exposure period (S110).

[0054] In addition, in S106, when the average brightness value enters the range of "128±2", it is determined that the condition of the front seat (front seat 14) can be identified ("Yes" in S106), and the processing unit 38c provides the first captured image data to the equipment control unit 28, and according to the condition of the first area including the front seat 14, the output control of the output device 32 and the control of the vehicle-mounted equipment 34 are executed as front seat side control (S112).

[0055] Next, the exposure control unit 38a shoots the 2nth frame (the second frame: an even-numbered frame) during the second exposure period (initial value) for the rear seat 16 determined in advance by experiments or the like (S114), and temporarily stores it in the RAM 42. The image acquisition unit 38b determines whether the captured (acquired) second captured image data is data that can confirm the condition of the rear seat (rear seat 16) (S116). For example, it is determined whether the average brightness of the second area corresponding to the rear seat 16 is a predetermined value such as "128" (for example, it may also include an allowable value of ±2). Then, if the average brightness value does not fall within the range of "128±2", it is determined that the condition of the rear seat (rear seat 16) cannot be recognized (No in S116). Then, the exposure control unit 38a calculates a correction value (e.g., +3ms, etc.) of the exposure period that enables confirmation of the status of the rear seat (second area) in the current shooting environment (S118), and determines the second exposure period corrected for the 2n+2th frame (the fourth frame: an even-numbered frame). In addition, at the same time, the illumination period of the infrared illumination unit 24 is determined corresponding to the corrected second exposure period (S120).

[0056] In addition, in S116, when the average brightness value enters the range of "128±2", it is determined that the condition of the rear seat (rear seat 16) can be identified (S116 is), and the processing unit 38c provides the second captured image data to the equipment control unit 28, and according to the condition of the second area including the rear seat 16, the output control of the output device 32 and the control of the vehicle-mounted equipment 34 are executed as rear seat side control (S122).

[0057] When executing the shooting control processing of the output device 32, the ECU 26 always monitors whether a request signal for the end of the status confirmation in the vehicle cabin 10a is received (S124). Then, when the ECU 26 does not receive the request signal for the end of the status confirmation ("No" in S124), the exposure control unit 38a increments the coefficient n of the frame number used for specific shooting by (n+1) (S126) and moves to S104. Then, the exposure control unit 38a performs shooting of the third frame and shooting of the fourth frame. That is, the shooting of the first shot image data of the odd frame and the second shot image data of the even frame by the infrared shooting unit 22 is repeatedly performed. In addition, the output control of the output device 32 of the device control unit 28 and the control of the vehicle-mounted device 34 are repeatedly performed.

[0058] In addition, in S124, when ECU26 receives the status confirmation end request signal ("Yes" in S124), the process is temporarily terminated. The status confirmation end request signal can be generated when the ignition switch of the vehicle 10 is turned off and received by ECU26, or it can be generated by the switch operation of the passenger 18 and received by ECU26.

[0059] In addition, Figure 7 The flowchart of FIG. 1 shows an example of executing the control of the device control unit 28 each time the status of the front seat and the rear seat can be confirmed by the first captured image data and the second captured image data obtained. In other embodiments, the processing of S112 and S122 for each acquisition of the first captured image data and the second captured image data may be omitted, and the shooting during the first exposure period and the second exposure period may be continuously performed, and the control of the device control unit 28 may be executed when the shooting of a predetermined amount of frames is completed.

[0060] As described above, in the photographing control device, the exposure control unit 38a controls the exposure length of the infrared photographing unit 22 capable of photographing the area in the vehicle 10a where the passengers 18 can sit to at least the first exposure period and the second exposure period longer than the first exposure period. In addition, the image acquisition unit 38b acquires the first photographing image data photographed during the first exposure period and the second photographing image data photographed during the second exposure period. Then, the processing unit 38c processes the first photographing image data and the second photographing image data, respectively, and provides them to the device control unit 28, for example. According to this structure, for example, although there is a possibility that the brightness of the image is insufficient in the second area relatively far from the infrared photographing unit 22 by photographing during the first exposure period, the so-called "brightness saturation" photographing in which the image is less likely to be too bright can be performed in the first area relatively close to the infrared photographing unit 22. On the other hand, by shooting during the second exposure period which is relatively longer than the first exposure period, although there is a possibility that the image may be too bright and saturated in brightness in the first area which is relatively close to the infrared camera unit 22, it is possible to shoot so-called "underexposure" in which the image is less likely to be too dark in the second area which is relatively far from the infrared camera unit 22. In other words, by switching the exposure period, it is possible to shoot an area that is not too bright and an area that is not too dark without the need for synthesis processing of the shot image. As a result, it is possible to reduce the cost of the device due to the reduction in synthesis processing, and to obtain with high precision either the first shot image data which is easy to identify the condition inside the vehicle cabin 10a in the first area close to the infrared camera unit 22 or the second shot image data which is easy to identify the condition inside the vehicle cabin 10a in the second area far from the infrared camera unit 22 for control.

[0061] In addition, the above-mentioned shooting control device 32a can also be controlled to alternately switch between the first exposure period and the second exposure period. According to this structure, for example, the first captured image data and the second captured image data can be efficiently acquired, and the state recognition of the first area and the state recognition of the second area can be performed at the same time, so that the state recognition can be smoothly performed.

[0062] In addition, the above-mentioned shooting control device 32a can also correct the length of the first exposure period in this shooting, for example, based on the average brightness value of the first area close to the infrared shooting unit 22 in the first shooting image data captured during the first exposure period in the previous shooting. In addition, the length of the second exposure period in this shooting can also be corrected based on the average brightness value of the second area farther from the infrared shooting unit 22 than the first area in the second shooting image data captured during the second exposure period in the previous shooting. According to this structure, for example, it is easy to perform the setting of the exposure period corresponding to the change of the surrounding conditions during shooting, and obtain the first shooting image data and the second shooting image data with appropriate brightness expression that makes it easier to recognize the shooting content.

[0063] In addition, the above-mentioned imaging control device 32a controls the length of the exposure period of the infrared imaging unit 22. According to this structure, for example, the first captured image data and the second captured image data with appropriate brightness that make it easier to recognize the captured content can be obtained in the vehicle 10a without being recognized by the passengers 18, regardless of day or night (bright state or dark state).

[0064] In addition, the above-mentioned shooting control device 32a can also synchronize the irradiation period of the infrared light of the infrared lighting unit 24, which irradiates the area that can be photographed by the infrared shooting unit 22, with the first exposure period and the second exposure period. According to this structure, for example, it is possible to implement infrared light irradiation only during shooting, which can help reduce shooting costs, and can achieve a reduction in heat generation, thereby helping to simplify the heat countermeasure structure.

[0065] In the above-mentioned embodiment, the case where the infrared imaging unit 22 is used for imaging is described, but as the imaging unit, for example, a TOF camera (Time-of-Flight Camera) may be used, and the switching control between the first exposure period and the second exposure period may be performed in the same manner. In this case, it is also possible to obtain an image with suppressed brightness saturation and underexposure, which allows good recognition of the conditions of the first area including the front seats 14 and the conditions of the second area including the rear seats 16, and thus it is possible to obtain the same effect as when the infrared imaging unit 22 is used.

[0066] In addition, in the above example, an example is shown in which the length of the exposure period is changed in the first area including the front seat 14 and the second area including the rear seat 16. In other embodiments, for example, the shooting control device can also be applied to a vehicle with three rows of seats. In this case, the first exposure period for the first area including the front seat, the second exposure period for the second area including the middle seat, and the third exposure period for the third area including the rear seat are set respectively, and the switching control of the exposure period is performed in a manner that the farther away from the infrared shooting unit 22, the longer the exposure time. As a result, it is possible to obtain a captured image that suppresses brightness saturation and underexposure, which can well identify the conditions of each area, and thus it is possible to obtain the same effect as in the above-mentioned case of two rows of seats. The same is true even in vehicles with more than four rows of seats.

[0067] The imaging control program executed by the CPU 38 of this embodiment may be provided as an installable or executable file recorded on a computer-readable recording medium such as a CD-ROM, a floppy disk (FD), a CD-R, or a DVD.

[0068] Alternatively, the imaging control program may be stored in a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the imaging control program executed in the present embodiment may be provided or distributed via a network such as the Internet.

[0069] The embodiments and variations of the present invention have been described, but these embodiments and variations are provided as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope and purpose of the invention, and are included in the invention described in the claims and their equivalents.

Claims

1. A shooting control device, characterized in that: have: an exposure control unit that controls the exposure length of the imaging unit capable of imaging the passenger-accessible area in the vehicle compartment to at least a first exposure period and a second exposure period longer than the first exposure period; an image acquisition unit that acquires first captured image data captured during the first exposure period and second captured image data captured during the second exposure period; as well as a processing unit for processing the first captured image data and the second captured image data respectively; The exposure control unit corrects the length of the first exposure period in this shooting based on the average brightness value of the first area close to the shooting unit in the first exposure period captured in the previous shooting, and corrects the length of the second exposure period in this shooting based on the average brightness value of the second area farther from the shooting unit than the first area in the second exposure period captured in the second exposure period in the previous shooting.

2. The shooting control device according to claim 1, characterized in that: The exposure control unit controls so that the first exposure period and the second exposure period are alternately switched.

3. The shooting control device according to claim 1 or 2, characterized in that: The exposure control unit controls the length of an exposure period of an infrared imaging unit that serves as the imaging unit and captures an infrared image.

4. The shooting control device according to claim 3, characterized in that: The exposure control unit synchronizes a period of irradiation of infrared light to a region that can be imaged by the infrared imaging unit with the first exposure period and the second exposure period.

Images