Occupant monitoring device for a vehicle and occupant protection system for a vehicle

Abstract

The present invention relates to a passenger monitoring device for a vehicle and a passenger protection system for a vehicle, and aims to improve the passenger monitoring device for a vehicle. A passenger monitoring device (60) for a vehicle (1) is capable of irradiating light from a light emitting device (66) to a passenger who gets on the vehicle (1) and taking an image by an imaging device (65) to monitor the position of the passenger, including a plurality of imaging modules (61, 62) each having the imaging device (65) and the light emitting device (66); and a control section (17) which controls imaging of the plurality of imaging modules (61, 62). The control section (17) causes the plurality of imaging modules (61, 62) to each irradiate light from the light emitting device (66) at the time of imaging by the imaging device (65), and when a collision of the vehicle (1) is predicted or detected, irradiates light from the light emitting devices (66) of the plurality of imaging modules (61, 62) to the passenger at the same time and takes an image to monitor the position of the passenger.

Description

Technical Field

[0001] This invention mainly relates to a vehicle occupant monitoring device. Background Technology

[0002] In automobiles and other vehicles, it is common to photograph the occupants sitting in the vehicle and determine their condition based on the photographed images (Patent Documents 1 and 2).

[0003] When passengers cannot coordinate well, their gaze may shift frequently, causing discomfort.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 11-043009

[0007] Patent Document 2: Japanese Patent Application Publication No. 2016-038793 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, it is desirable that, in a vehicle, not only can the status of the occupants be monitored individually by the occupant monitoring device, but also that the monitoring results, such as the position of the occupants monitored by the vehicle's occupant monitoring device, be used to allow other devices in the vehicle, such as occupant protection devices, to appropriately control the occupants.

[0010] Furthermore, when it is necessary to accurately monitor the position of occupants using occupant monitoring devices, it is desirable to use luminous devices to illuminate the occupants with relatively strong light. In vehicles, the interior brightness sometimes varies significantly depending on the driving environment, or external light may illuminate the occupants. In these situations, it is also desirable for luminous devices to illuminate the occupants with relatively strong light so that their outlines can be clearly captured.

[0011] However, to illuminate the occupants with such intense light, the light-emitting device needs to be relatively large for high output and requires a relatively large current. Power consumption increases with the square of the current emitted by the light-emitting device. The heat sinks used to cool the light-emitting device also become larger. Furthermore, the higher the beam of light emitted by the light-emitting device, the earlier it diminishes.

[0012] Thus, there is a need to improve vehicle occupant monitoring devices.

[0013] Technical solutions for solving the problem

[0014] This invention provides a vehicle occupant monitoring device, which can illuminate occupants sitting in the vehicle with light from a light-emitting device and capture images with a shooting device to monitor the occupant's position. The device includes: multiple shooting modules, each having the shooting device and the light-emitting device; and a control unit that controls the shooting of the multiple shooting modules. The control unit causes each of the multiple shooting modules to illuminate light from the light-emitting device when the shooting device captures images. When a collision of the vehicle is predicted or detected, the light-emitting devices of the multiple shooting modules simultaneously illuminate the occupants and capture images to monitor the occupant's position.

[0015] Preferably, the control unit can cause the multiple shooting modules to operate at staggered shooting times in the absence of predicting or detecting a collision with the vehicle, so that the light emission times of these multiple light-emitting devices are staggered.

[0016] Preferably, the control unit can enable the shooting devices of the multiple shooting modules to shoot at high speed when a collision of the vehicle is predicted or detected, compared to the case where a collision of the vehicle is not predicted or detected.

[0017] Preferably, the light-emitting devices of the plurality of imaging modules can illuminate the occupant with infrared light, and the control unit will suppress the light-emitting devices of the plurality of imaging modules to emit light at a first amount of light that is sufficient to capture the occupant's retina or blood vessels, at least in the event that a collision of the vehicle is not predicted or detected.

[0018] Preferably, when the control unit predicts or detects a collision with the vehicle, it causes the light-emitting devices of the plurality of imaging modules to emit light at a second light intensity below the first light intensity.

[0019] Preferably, the control unit can monitor the occupant's state based on multiple images captured by the imaging devices of the multiple imaging modules, monitor the position of the occupant's head in the absence of a predicted or detected collision of the vehicle, and monitor the occupant's state based on the occupant's retina or blood vessels captured in at least one of the multiple imaging devices. In the event of a predicted or detected collision of the vehicle, the monitoring based on the occupant's retina or blood vessels is stopped, and the position of the occupant's head is monitored based on an image captured at high speed by one of the imaging devices.

[0020] Preferably, the plurality of shooting modules may include: a center shooting module, which is disposed in the front center of the vehicle and is capable of simultaneously illuminating and photographing the driver of the vehicle; and a driver shooting module, which is disposed in front of the driver.

[0021] This invention provides a vehicle occupant protection system comprising: an occupant monitoring device capable of illuminating an occupant in the vehicle with light from a light-emitting device and capturing an image with a shooting device to monitor the occupant's position; and an occupant protection device that performs protective control on the occupant based on the occupant's position monitored by the occupant monitoring device. The occupant monitoring device includes: multiple shooting modules, each having the shooting device and the light-emitting device; and a control unit that controls the shooting of the multiple shooting modules. The control unit causes each of the multiple shooting modules to illuminate light from the light-emitting device when the shooting device captures an image, and when a collision of the vehicle is predicted or detected, to simultaneously illuminate the occupant with light from the light-emitting devices of the multiple shooting modules and capture an image to monitor the occupant's position.

[0022] Invention Effects

[0023] In this invention, multiple imaging modules, each equipped with an imaging device and a light-emitting device, are each illuminated by light from their respective light-emitting devices while the imaging device is capturing images. Furthermore, when a vehicle collision is predicted or detected, the multiple imaging modules are activated to simultaneously capture images, illuminating the occupants simultaneously from the light-emitting devices of all the imaging modules. Thus, in the event of a predicted or detected vehicle collision, the occupants are brightly illuminated by the light from the multiple light-emitting devices. By capturing images of the occupants in this state, the outlines of the occupants are clearly captured in the images. As a result, in this invention, the position of the occupants can be accurately determined based on the captured images. Based on the correct occupant position determined by the vehicle occupant monitoring device of this invention, other vehicle devices, such as occupant protection devices, can appropriately perform protective controls on the occupants according to their correct positions.

[0024] However, in this invention, for example, even without increasing the luminous intensity in each individual light-emitting device, the total amount of light illuminating the occupants can be increased. Therefore, after returning from a state of predicted or detected vehicle collision to an undetected state, degradation due to increased light intensity can be avoided in each light-emitting device. Alternatively, degradation due to increased light intensity can be largely avoided. As a result, the light-emitting devices can continue to be used even after a vehicle collision has been predicted or detected. The light-emitting devices can continue to be used for a long period regardless of whether a vehicle collision is predicted or detected.

[0025] Furthermore, in this invention, under normal circumstances where a vehicle collision is not predicted or detected, multiple imaging modules operate by staggering their respective imaging timings so that the emission timings of these multiple light-emitting devices are staggered. Thus, under normal circumstances, the multiple imaging modules can photograph the occupant at their respective times, while the occupant is only illuminated by the light from their respective light-emitting devices. Additionally, it is difficult to continue illuminating the occupant with strong light. The condition of the occupant's retina, blood vessels, etc., can be accurately captured. Attached Figure Description

[0026] Figures 1(A)-(B) are schematic illustrations of a car according to an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram illustrating the control system of the car shown in Figures 1(A)-(B).

[0028] Figure 3 The following are luminous characteristics diagrams of an example of an infrared LED shown in Figures 1(A)-(B).

[0029] Figure 4 yes Figure 2 The flowchart shows the switching process of the occupant monitoring ECU's operating mode.

[0030] Figure 5 yes Figure 4 The flowchart shows the typical processing mode.

[0031] Figure 6 yes Figure 4 The flowchart for processing in the shrink mode.

[0032] Figure 7 yes Figure 2 The flowchart shows the processing of the occupant protection ECU.

[0033] Figure 8 This is a timing diagram illustrating the operational status of the driver's camera module and the central camera module.

[0034] Symbol Explanation

[0035] 1…Automobile (vehicle), 6…Instrument panel, 10…Control system (occupant protection system), 17…Occupant monitoring ECU, 18…Occupant protection ECU, 60…Occupant monitoring device, 61…Central camera module, 62…Driver camera module, 63…Occupant monitoring memory, 65…Camping device, 66…Infrared LED, 70…Occupant protection device, 71…Airbag device, 72…Seat belt device Detailed Implementation

[0036] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0037] Figures 1(A)-(B) are schematic illustrations of a car 1 according to an embodiment of the present invention.

[0038] Figure 1(A) is a top view of the car 1. Figure 1(B) is a schematic diagram of the front of the passenger compartment 3 of the car 1.

[0039] The car 1 in Figures 1(A)-(B) is an example of a vehicle. The car 1 has a passenger compartment 3 located in the center of its body 2. A seat 4 for passengers to sit on is located in the passenger compartment 3. A windshield 5 is located at the front of the passenger compartment 3. An instrument panel 6 is located below the windshield 5. The instrument panel 6 is positioned in front of the upper body of the passenger seat 4. On the instrument panel 6, a steering wheel 31 for the driver to operate the car 1 is mounted protruding rearward from the front side of the driver's seat 4. A brake pedal 32 and an accelerator pedal 33 for the driver to operate the car 1 are mounted side-by-side below the instrument panel 6.

[0040] Figure 2 This is a schematic diagram illustrating the control system 10 of the car 1 shown in Figures 1(A)-(B).

[0041] Figure 2 In the system 10 that constitutes the vehicle 1, multiple control modules are represented by separately programmed control ECUs (Electronic Control Units).

[0042] In car 1, in automatic driving that does not rely on the driver or other passengers, or in manual driving that is performed by the passengers, the passengers can be identified and monitored by the passenger monitoring device 60.

[0043] In addition, in the car 1, the occupants can also be protected from the impact of a collision by the occupant protection device 70.

[0044] By organically combining these technologies, it is possible to achieve appropriate occupant protection corresponding to the occupants' seating position. The aim is to improve the convenience and safety of vehicle 1.

[0045] The control system 10 of this vehicle 1 can function as an occupant protection system of the vehicle 1, which has an occupant monitoring device 60.

[0046] Figure 2 In this context, the multiple control devices constituting the control system 10 of the vehicle 1 are represented by separately programmed control ECUs (Electronic Control Units).

[0047] exist Figure 2Specifically, the control system 10 of the vehicle 1 includes: a drive ECU 11, a steering ECU 12, a braking ECU 13, an automatic driving / driving assistance ECU 14, a driving operation ECU 15, a detection ECU 16, an occupant monitoring ECU 17 of the occupant monitoring device 60, an occupant protection ECU 18 of the occupant protection device 70, an external communication ECU 19, and a UI ECU 20. These multiple control ECUs are directly connected to a cable 26 and connected to a central gateway (CGW) 27, which acts as a relay device, via a vehicle network 25, such as CAN (Controller Area Network) or LIN (Local Interconnect Network), used in the vehicle 1. Each control ECU can communicate with each other using encrypted data with its own ID as the source and the ID of the destination device attached. Each control ECU can also simultaneously broadcast encrypted data without a specified destination to multiple other control ECUs. Additionally, each control ECU receives encrypted data with its own ID as the destination for its own control. Furthermore, a control ECU can also receive encrypted data with a specific ID other than its own as the destination for its own control. When the central gateway 27 receives encrypted data from multiple cables 26 for use with a control ECU connected to other cables 26, it outputs the received encrypted data to the other cables 26. Thus, Figure 2 The multiple control ECUs shown are able to send and receive encrypted data to each other.

[0048] The drive ECU 11 receives encrypted data through the vehicle network 25 and controls the drive source and transmission device (not shown) such as an engine or motor installed in the vehicle 1. As a result, the vehicle 1 can accelerate.

[0049] The steering ECU 12 receives encrypted data via the vehicle network 25 and controls a steering device (not shown) installed in the vehicle 1. This allows the vehicle 1 to change its direction of travel.

[0050] The braking ECU 13 receives encrypted data via the vehicle network 25 and controls a braking device (not shown) installed in the vehicle 1. This allows the vehicle 1 to decelerate and stop. Additionally, the braking ECU 13 can also communicate with the drive ECU 11 via the vehicle network 25 to suppress the rotation of the drive source, change the gear ratio of the transmission, and further decelerate the vehicle 1.

[0051] The driving operation ECU 15 is connected to operating components such as the steering wheel 31, brake pedal 32, accelerator pedal 33, and gearshift lever 34 used to operate the vehicle 1. The driving operation ECU 15 outputs encrypted data containing operating information related to the operated components to the autonomous driving / driving assistance ECU 14, etc., via the vehicle network 25.

[0052] The detection ECU 16 is connected to, for example, a speed sensor 41, an acceleration sensor 42, and a stereo camera 43. The speed sensor 41 detects the speed of the vehicle 1. The acceleration sensor 42 detects the acceleration of the vehicle 1. The stereo camera 43 captures images of the surrounding area of ​​the vehicle 1. The stereo camera 43 can also capture images of the front, sides, or rear of the vehicle 1. The detection ECU 16 obtains physical quantities from the connected sensors as physical quantities detected in the vehicle 1 regarding its movement, and outputs them via the vehicle network 25 to the autonomous driving / driving assistance ECU 14, the occupant protection ECU 18, etc.

[0053] For example, the detection ECU16 analyzes the external images captured by the stereo camera 43, detects the presence or absence of other moving objects around the vehicle 1, and outputs the information about the types of other moving objects, their relative distance from the vehicle, and their direction to the autonomous driving / driving assistance ECU14, occupant protection ECU18, etc., through the vehicle network 25.

[0054] The detection ECU 16 analyzes the external images captured by the stereo camera 43, predicts collisions with other moving objects around the vehicle 1, and outputs this information to the autonomous driving / driving assistance ECU 14 and the occupant protection ECU 18 via the vehicle network 25.

[0055] When the acceleration sensor 42 detects a large collision acceleration above a specified value, the detection ECU 16 can also output the collision detection to the occupant protection ECU 18.

[0056] External communication ECU 19 communicates with the external environment of vehicle 1. For example, external communication ECU 19 communicates with communication relay station 101 of traffic information system 100, and sends and receives communication data with server device 102. In addition, external communication ECU 19 also sends and receives communication data with other vehicles 110 and pedestrians (not shown) traveling near vehicle 1, for example, via V2X communication. External communication ECU 19 can also output navigation data, traffic information, and information about the surrounding environment of vehicle 1 to autonomous driving / driving assistance ECU 14, etc., via vehicle network 25 when it receives such information through these communications.

[0057] The external communication ECU 19 acquires information about the predicted collisions in other vehicles 110 and outputs this information to the autonomous driving / driving assistance ECU 14 and the occupant protection ECU 18 via the vehicle network 25.

[0058] UIECU 20 is connected to, for example, display device 51 and operating device 52. Operating device 52 may also have a touch panel overlapping the display screen of display device 51 and multiple keys. When receiving encrypted data regarding the display, UIECU 20 causes an image to be displayed on display device 51. UIECU 20 may also generate, for example, navigation data based on the operation of operating device 52 corresponding to the display on display device 51, and output it to autonomous driving / driving assistance ECU 14, etc., via vehicle network 25.

[0059] The autonomous driving / driving assistance ECU 14 switches between autonomous driving and driving assistance modes to control the movement of the vehicle 1. The autonomous driving / driving assistance ECU 14 controls the movement of the vehicle 1 based on various information obtained through the vehicle network 25. For example, in autonomous driving mode, the autonomous driving / driving assistance ECU 14 outputs control information to the drive ECU 11, steering ECU 12, and braking ECU 13 to confirm the safety of the surroundings based on navigation data and simultaneously drive to the destination. In driving assistance mode, the autonomous driving / driving assistance ECU 14 outputs control information, adjusted according to the safety of the surroundings, to the drive ECU 11, steering ECU 12, and braking ECU 13 based on operational information.

[0060] The occupant monitoring ECU17 is connected to the central imaging module 61, the driver imaging module 62, and the occupant monitoring memory 63.

[0061] As shown in Figure 1(B), the central imaging module 61 includes an imaging device 65 and an infrared LED 66 as a light-emitting device. The central imaging module 61 is located in the center of the instrument panel 6 in front of the occupants. The light-emitting device illuminates infrared light across the entire wide-angle field of view of the imaging device 65 shown in Figure 1(B). Thus, the imaging device 65 can illuminate the driver and other occupants within its field of view with infrared light and capture an infrared-based image. The infrared-based image can include image components such as the retina and blood vessels of the driver and other occupants. The illumination frequency of the infrared LED 66 can also be the frequency contained in natural light or external lights. In this case, the infrared LED 66 can also be distinguished from natural light by adjusting the illumination intensity, illumination pattern, etc. By subtracting the infrared image of the unilluminated state from the infrared image of the illuminated state, an image containing only the image components generated by the illumination of the infrared LED 66 can be obtained.

[0062] As shown in Figure 1(B), the driver imaging module 62 includes an imaging device 65 and an infrared LED 66 as a light-emitting device. The driver imaging module 62 is located on the right side of the dashboard 6, in front of the driver's seat 4, which is situated in front of the occupants. The driver imaging module 62 can capture images of the driver together with the central imaging module 61. The light-emitting device illuminates infrared light at a narrow angle to the entire narrow viewing angle of the imaging device 65 shown in Figure 1(B). Thus, the imaging device 65 can illuminate the driver and other occupants within its viewing angle with infrared light and capture an infrared-based image. The infrared-based image can include image components such as the retina and blood vessels of the driver and other occupants. The illumination frequency of the infrared LED 66 can also be the same as that of the central imaging module 61.

[0063] The occupant monitoring memory 63 stores programs, configuration data, etc., used for occupant monitoring. The configuration data may also include individual data for each occupant.

[0064] The occupant monitoring ECU 17 reads a program from the occupant monitoring memory 63 and executes it. Thus, the occupant monitoring ECU 17 functions as the control unit of the occupant monitoring device 60. As the control unit of the occupant monitoring device 60, the occupant monitoring ECU 17 controls the operation of the central imaging module 61 and the driver imaging module 62. For example, the occupant monitoring ECU 17 activates the central imaging module 61 and the driver imaging module 62 to repeatedly capture images at a predetermined cycle. The occupant monitoring ECU 17 acquires images from the central imaging module 61 and the driver imaging module 62. The occupant monitoring ECU 17 analyzes the acquired images to identify and monitor the occupant, including the driver, who is riding in the vehicle 1.

[0065] For example, the occupant monitoring ECU17 analyzes the captured images to identify the occupants, such as the driver, who are riding in the car 1.

[0066] The occupant monitoring ECU 17 monitors the position of the occupant's upper body and head as a status indicator. It also monitors movement based on changes in position. The occupant monitoring ECU 17 can also output information about the position and movement of the occupant's upper body and head to the occupant protection ECU 18, etc.

[0067] The occupant monitoring ECU 17 also monitors occupant states such as drowsiness, excitement, looking away, and death. The occupant monitoring ECU 17 can determine drowsiness by, for example, the degree of head tilt or the degree of eye opening captured in the photograph. It can also determine excitement, emergency status, or drowsiness by, for example, the pulse rate in the retina or blood vessels, or the amount of hemoglobin. The occupant monitoring ECU 17 can determine looking away based on, for example, the absence of a photograph of both eyes, or the eyes not being photographed symmetrically relative to the nose or mouth. The occupant monitoring ECU 17 can also execute separately prepared procedures for these states in parallel.

[0068] Furthermore, the occupant monitoring ECU 17 can also output these monitoring results to the autonomous driving / driving assistance ECU 14, occupant protection ECU 18, etc., via the vehicle network 25. When receiving such information, the autonomous driving / driving assistance ECU 14 can also perform control to change the destination of autonomous driving to a hospital or other similar location and continue driving, or to make an emergency stop at the side of the road or in a parking lot.

[0069] The occupant protection ECU18 is connected to the airbag device 71 and the seat belt device 72.

[0070] The airbag device 71 includes, for example, a front airbag that deploys in front of an occupant seated in seat 4, a side airbag that deploys outside the occupant, a distal airbag that deploys between adjacent occupants, and a curtain airbag that deploys inside the glass surface of the vehicle body 2.

[0071] The seat belt device 72 has a seat belt that is attached to the occupant sitting in the seat 4.

[0072] When the occupant protection ECU18 predicts or detects a collision with the vehicle 1, it activates the airbag device 71 and the seat belt device 72 to perform occupant protection control.

[0073] For example, when the occupant protection ECU18 obtains the current position of the occupant's head from the occupant monitoring ECU17, it can also change the settings of the airbag deployment start position, deployment direction, deployment speed, deployment intensity, seat belt restraint start time, restraint intensity (limit), restraint period, etc., to suit the current position of the occupant's head.

[0074] When a collision is detected, the occupant protection ECU18 can also activate the airbag device 71 and the seat belt device 72 by setting at that time.

[0075] However, in the control system 10 of such a car 1, the occupant monitoring device 60 uses a light-emitting device such as an infrared LED 66 to photograph the occupants.

[0076] Figure 3 The image shows the luminous characteristics of an example of the infrared LED 66 shown in Figures 1(A)-(B).

[0077] Figure 3 The horizontal axis represents the cumulative on-time corresponding to the lifespan of the infrared LED66. The vertical axis represents the maintenance rate of the beam that makes the infrared LED66 emit light.

[0078] like Figure 3 As shown, the higher the light output of the infrared LED66, the shorter the cumulative on-time that can be used at that light output. The usage limit at the maximum light output of a single unit shown in the figure is shorter than the usage limit when emitting a lower first light output. The first light output is within the range suitable for imaging the retina and blood vessels. The light output of the infrared LED66 should preferably be minimized as much as possible.

[0079] The cumulative on-time refers to the total on-time of the infrared LED66. Therefore, it is best to turn off the infrared LED66 when it is not needed.

[0080] On the other hand, in car 1, the brightness inside the vehicle sometimes varies greatly depending on the driving environment, or there may be light shining outside the vehicle onto the occupants. Even in these situations, it is desirable for the infrared LED 66 to emit a relatively strong light onto the occupants so that the outlines of the occupants can be clearly captured. In order to accurately monitor the position of the occupants by capturing their outlines, it is desirable for the infrared LED 66 to emit a relatively strong light onto the occupants.

[0081] To increase the maximum light output of a single infrared LED66, the infrared LED66 has been enlarged. Larger infrared LED66s require a larger current to achieve high output compared to smaller ones. The power consumption and heat generation of the infrared LED66 increase proportionally to the square of the current. The heat sink used to cool the light-emitting device has also become larger.

[0082] Therefore, as Figure 3 As shown, to obtain the necessary light intensity for capturing occupant silhouettes and other details independently of the shooting environment, a high-output infrared LED66 is required, thus the aforementioned problem arises. Furthermore, if a mid-output infrared LED66 is used near the maximum light output of a single unit, insufficient light output occurs in the early stages of a short cumulative illumination time. In this case, the infrared LED66 must be replaced relatively frequently.

[0083] Furthermore, when the infrared LED66 illuminates the occupant with relatively strong light, it is difficult to capture the occupant's retina, blood vessels, etc. effectively. The range of light intensity suitable for capturing the retina, blood vessels, etc., is lower than the range of light intensity suitable for capturing the occupant's outline, regardless of the environment, and these ranges are more likely to not overlap.

[0084] In this embodiment, these problems are comprehensively addressed through design control, further improving the function of the occupant monitoring device 60 as a vehicle. This will be explained in detail below.

[0085] Figure 4 yes Figure 2 The flowchart shows the switching process of the operating mode of the occupant monitoring ECU17.

[0086] When the car 1 is started, the occupant monitoring ECU 17, as the control unit of the occupant monitoring device 60, repeatedly executes... Figure 4 The occupant monitoring ECU 17 can also be repeatedly executed while the vehicle 1 is in motion. Figure 4 The processing.

[0087] In step ST1, the occupant monitoring ECU 17 first activates the occupant monitoring device 60 in normal mode. In normal mode, the occupant monitoring ECU 17 performs all the functions of the occupant monitoring device 60. For example, the occupant monitoring ECU 17 monitors the position of the occupant's upper body and head based on images captured by the central imaging module 61, and monitors the occupant's sitting status, such as drowsiness, excitement, or looking to the side, based on images captured by the driver imaging module 62. The occupant monitoring ECU 17 may also monitor the occupant's status based on images captured of the occupant's retina or blood vessels.

[0088] In step ST2, the occupant monitoring ECU 17 determines whether a collision with vehicle 1 is predicted. The detection ECU 16 analyzes the external image captured by the stereo camera 43 to predict collisions with other moving objects around vehicle 1. If the occupant monitoring ECU 17 obtains this collision prediction, it determines that a collision with vehicle 1 is predicted, and the process proceeds to step ST3. If no collision is predicted, the occupant monitoring ECU 17 returns the process to step ST1. The occupant monitoring ECU 17 continues operating in normal mode until a collision is predicted.

[0089] In step ST3, the occupant monitoring ECU 17 switches the operating state of the occupant monitoring device 60 from the normal mode at startup to the retracted mode. In the retracted mode, the occupant monitoring ECU 17 performs only a portion of the functions of the occupant monitoring device 60. The occupant monitoring ECU 17 retracts to perform functions such as those required after collision prediction. Here, the occupant monitoring ECU 17, for example, performs only the function of monitoring and recording the position of the occupant's upper body and head based on images captured by the center imaging module 61. Additionally, as described later, in the retracted mode, the occupant monitoring ECU 17 captures images from the center imaging module 61 and the driver imaging module 62 at a higher speed and with shorter cycles than in the normal mode. In this case, the images captured by the driver imaging module 62 are not used. The occupant monitoring ECU 17 stops monitoring based on the state of the occupant's retina or blood vessels, and monitors only the occupant's head position with short cycles based solely on images captured at high speed by a single imaging device 65.

[0090] In step ST4, the occupant monitoring ECU 17 determines whether a collision has been detected in vehicle 1. When the acceleration sensor 42 detects a large collision acceleration exceeding a predetermined value, the detection ECU 16 detects a collision. Upon receiving this collision detection, the occupant monitoring ECU 17 determines that a collision has been detected in vehicle 1 and terminates the process. Figure 4 The occupant monitoring ECU 17 may also store the monitoring data of the occupants identified as having been involved in a collision in the occupant monitoring memory 63. Additionally, the occupant monitoring ECU 17 may monitor the occupants for a predetermined period after a collision detection and store the detected data in the occupant monitoring memory 63. If no collision detection is detected, the occupant monitoring ECU 17 proceeds to step ST5.

[0091] In step ST5, the occupant monitoring ECU 17 determines whether the collision of vehicle 1 has disappeared. For example, if the occupant monitoring ECU 17 does not obtain a new collision prediction from the detection ECU 16, it determines that the collision of vehicle 1 has disappeared and returns the process to step ST1. Alternatively, if the occupant monitoring ECU 17 continues to obtain a new collision prediction from the detection ECU 16, it determines that the collision of vehicle 1 has not disappeared and returns the process to step ST4.

[0092] Figure 5 yes Figure 4 The flowchart shows the typical processing mode.

[0093] When the occupant monitoring ECU 17 is activated and operated in normal mode, it repeatedly performs functions as the control unit of the occupant monitoring device 60. Figure 5 The processing.

[0094] Occupant monitoring ECU17 via Figure 5The processing is based on multiple images captured by the imaging device 65 with multiple imaging modules to monitor the status of the occupants.

[0095] In step ST11, the occupant monitoring ECU 17 performs initial settings for activating the occupant monitoring device 60 in normal mode. Here, the occupant monitoring ECU 17 is in a state where it is activated and performs all functions of the occupant monitoring device 60. The occupant monitoring ECU 17 configures the center imaging module 61 and the driver imaging module 62 so that they capture images at staggered times in a manner that does not overlap within a predetermined cycle for repeated capturing and monitoring. The occupant monitoring ECU 17 configures the infrared LED 66 of the center imaging module 61 and the infrared LED 66 of the driver imaging module 62 to emit light only during their respective capturing times, and sets their light emission levels to a first light intensity.

[0096] In step ST12, the occupant monitoring ECU 17 determines whether it is the first opportunity for the center imaging module 61 to perform imaging. If the first opportunity has not been reached, the occupant monitoring ECU 17 repeats this process. When the first opportunity is reached, the occupant monitoring ECU 17 causes the process to proceed to step ST13.

[0097] In step ST13, the occupant monitoring ECU 17 causes the central imaging module 61 to illuminate the infrared LED 66, executing the imaging process of the imaging device 65. The infrared LED 66 begins to emit light at a first light intensity and turns off when the imaging process of the imaging device 65 ends. The infrared LED 66 is controlled to emit light at a first light intensity sufficient to capture the occupant's retina or blood vessels.

[0098] In step ST14, the occupant monitoring ECU 17 acquires the image captured by the central imaging module 61 in step ST13 and performs monitoring processing. The occupant monitoring ECU 17 determines the current position of the occupant's head inside the vehicle based on the position of the occupant's head in the captured image. The occupant monitoring ECU 17 then sends the determined current position of the occupant's head to the occupant protection ECU 18.

[0099] In step ST15, the occupant monitoring ECU 17 determines whether it is the second opportunity for the driver imaging module 62 to perform imaging. If the second opportunity has not been reached, the occupant monitoring ECU 17 repeats this process. When the second opportunity is reached, the occupant monitoring ECU 17 causes the process to proceed to step ST16.

[0100] In step ST16, the occupant monitoring ECU 17 causes the driver imaging module 62 to illuminate the infrared LED 66 and execute the imaging process of the imaging device 65. The infrared LED 66 begins to emit light at a first light intensity and turns off when the imaging process of the imaging device 65 ends. The infrared LED 66 is controlled to emit light at a first light intensity capable of capturing images of the occupant's retina or blood vessels.

[0101] In step ST17, the occupant monitoring ECU 17 acquires the image captured by the driver imaging module 62 in step ST16 and performs monitoring processing. The occupant monitoring ECU 17 can also perform monitoring processing on the image captured by the center imaging module 61 in step ST13. The occupant monitoring ECU 17 determines the occupant's seating status based on the occupant's retina and blood vessels in the captured images. If the occupant is not in a seating state where they are capable of driving the vehicle 1, the occupant monitoring ECU 17 sends the determined seating status to the autonomous driving / driving assistance ECU 14, external communication ECU 19, UIECU 20, etc. The autonomous driving / driving assistance ECU 14 can also switch the driving of the vehicle 1 based on the occupant's seating status. The external communication ECU 19 can also send the occupant's seating status to the server device 102 or other vehicles 110. The UIECU 20 can also output alarms based on the occupant's seating status from the display device 51, etc.

[0102] In step ST18, the occupant monitoring ECU 17 determines whether monitoring has ended. For example, if vehicle 1 arrives at its destination and the occupants disembark, the occupant monitoring ECU 17 determines that monitoring has ended and terminates the monitoring. Figure 5 The occupant monitoring ECU 17 returns to step ST12 if the monitoring is not determined to be complete. In normal mode, the occupant monitoring ECU 17 repeats the process from steps ST12 to ST18 for each predetermined shooting cycle, causing the central imaging module 61 and the driver imaging module 62 to take pictures sequentially. Then, the infrared LED 66 of the central imaging module 61 illuminates only during the period when its imaging device 65 is capturing images. The infrared LED 66 of the driver imaging module 62 illuminates only during the period when its imaging device 65 is capturing images.

[0103] In this way, if the occupant monitoring ECU 17 does not predict or detect a collision with the vehicle 1, it causes multiple imaging modules to operate at staggered shooting times so that the lighting times of these multiple infrared LEDs 66 are staggered.

[0104] Figure 6 yes Figure 4 The flowchart for processing in the shrink mode.

[0105] When the occupant monitoring ECU 17 is activated in retracted mode, it acts as the control unit of the occupant monitoring device 60, repeatedly executing... Figure 6 The processing.

[0106] Occupant monitoring ECU17 via Figure 6 The processing is based on the image captured by the imaging device 65 of the central imaging module 61 to monitor the current position of the occupant's head.

[0107] In step ST21, the occupant monitoring ECU 17 performs initial settings for activating the occupant monitoring device 60 in retracted mode. Here, the occupant monitoring ECU 17 is in a state where it only performs a portion of the functions of the occupant monitoring device 60. Specifically, the occupant monitoring ECU 17 is in a state where it only monitors the current position of the occupant's head. The occupant monitoring ECU 17 configures the center imaging module 61 and the driver imaging module 62 to simultaneously capture images at the same time within a predetermined illumination shooting cycle shorter than in normal mode. The occupant monitoring ECU 17 configures the infrared LED 66 of the center imaging module 61 and the infrared LED 66 of the driver imaging module 62 to emit light only at their respective shooting times, and sets their emitted light intensity to a second light intensity below the first light intensity. The second light intensity is more than half the first light intensity, thereby ensuring that the total light intensity of the infrared LED 66 of the center imaging module 61 and the infrared LED 66 of the driver imaging module 62 is greater than or equal to the first light intensity. In addition, since the light intensity of each infrared LED66 is not increased compared to normal, even if the light intensity is switched to activate it during collision prediction detection, each infrared LED66 will not deteriorate and its lifespan will not be shortened.

[0108] In step ST22, the occupant monitoring ECU 17 determines whether it is a high-speed shooting opportunity for the driver's camera module 62 and the center camera module 61 to simultaneously capture images. If the high-speed shooting opportunity has not been reached, the occupant monitoring ECU 17 repeats this process. When the high-speed shooting opportunity is reached, the occupant monitoring ECU 17 causes the process to proceed to step ST23.

[0109] In step ST23, the occupant monitoring ECU 17 causes the infrared LEDs 66 of the driver imaging module 62 and the center imaging module 61 to illuminate and execute the imaging process of the imaging device 65. The infrared LEDs 66 of the driver imaging module 62 and the center imaging module 61 each begin illuminating with a first light intensity, and the lights are turned off when the imaging process of each imaging device 65 ends. The total amount of light illuminating the occupant is greater than or equal to the first light intensity, making it suitable for capturing the occupant's silhouette regardless of the environment.

[0110] In step ST24, the occupant monitoring ECU 17 obtains the image captured by the central imaging module 61 in step ST23 and performs monitoring processing. The occupant monitoring ECU 17 determines the current position of the occupant's head inside the vehicle based on the position of the occupant's head in the captured image. The occupant monitoring ECU 17 then sends the determined current position of the occupant's head to the occupant protection ECU 18.

[0111] In step ST25, the occupant monitoring ECU 17 determines whether monitoring has ended. For example, if vehicle 1 arrives at its destination and the occupants disembark, the occupant monitoring ECU 17 determines that monitoring has ended and terminates the monitoring. Figure 6 The process proceeds as follows: If the monitoring is not determined to be complete, the occupant monitoring ECU 17 returns the process to step ST22. In each high-speed shooting cycle in retracted mode, the occupant monitoring ECU 17 repeatedly performs the process from step ST22 to step ST25, causing the central imaging module 61 and the driver imaging module 62 to simultaneously capture images. Then, the infrared LEDs 66 of the central imaging module 61 and the driver imaging module 62 are simultaneously illuminated, illuminating the occupant with a total light intensity exceeding the first light dose. The imaging device 65 of the central imaging module 61 is capable of capturing images of the occupant under such intense light.

[0112] In this way, when the occupant monitoring ECU17 predicts or detects a collision with the vehicle 1, it causes the multiple infrared LEDs 66 to light up simultaneously, thereby activating multiple imaging modules.

[0113] Figure 7 yes Figure 2 The flowchart shows the processing of the occupant protection ECU18.

[0114] When the car 1 starts, the occupant protection ECU 18, as the control unit of the occupant protection device 70, repeatedly executes... Figure 7 The occupant protection ECU18 can also be repeatedly executed while the vehicle 1 is in motion. Figure 7 The processing.

[0115] In step ST31, the occupant protection ECU 18 determines whether a collision with vehicle 1 is predicted. The detection ECU 16 analyzes the external image from the stereo camera 43 to predict collisions with other moving objects around vehicle 1. If the occupant protection ECU 18 receives this collision prediction, it determines that a collision with vehicle 1 is predicted and proceeds to step ST32. If no collision is predicted, the occupant protection ECU 18 repeats this process.

[0116] In step ST32, the occupant protection ECU 18 obtains the latest head position of the occupant. The occupant monitoring ECU 17 obtains the latest head position of the occupant based on periodic occupant imaging. The occupant protection ECU 18 obtains the latest head position of the occupant from the occupant monitoring ECU 17.

[0117] In step ST33, the occupant protection ECU 18 sets the operation of the occupant protection device 70 based on the latest head position of the occupant. Based on the latest head position of the occupant from the occupant monitoring ECU 17, the occupant protection ECU 18 adjusts the airbag deployment start position, deployment direction, deployment speed, and deployment intensity settings to support the occupant's upper body and head from the direction in which the occupant falls due to the collision. Additionally, the occupant protection ECU 18 adjusts the seatbelt restraint start timing, restraint intensity (limit), and restraint duration settings.

[0118] Furthermore, the occupant protection ECU 18 can also execute the processing of step ST31 after the processing of steps ST32 and ST33. In this case, the occupant protection ECU 18 can also return the processing to step ST32 if a collision is not predicted.

[0119] In step ST34, the occupant protection ECU 18 determines whether a collision of vehicle 1 has been detected. The detection ECU 16 detects a collision of vehicle 1 based on the detection value from the acceleration sensor 42. If the occupant protection ECU 18 detects a collision, it determines that vehicle 1 has been in a collision and proceeds to step ST35. If no collision is detected, the occupant monitoring ECU 17 proceeds to step ST36.

[0120] In step ST35, the occupant protection ECU 18 performs occupant protection control. Based on these settings, the occupant protection ECU 18 activates the airbag device 71 and the seat belt device 72. The seat belts exert a restraining force on the occupant towards the seat 4, and the airbags deploy around the occupant. The impact on the occupant is absorbed by the seat belts and the airbags.

[0121] In step ST36, the occupant protection ECU 18 determines whether the collision of vehicle 1 has been resolved. For example, if the occupant protection ECU 18 does not obtain a new collision prediction from the detection ECU 16, it determines that the collision of vehicle 1 has been resolved and returns the process to step ST31. If the occupant monitoring ECU 17 continues to obtain a new collision prediction from the detection ECU 16, it determines that the collision of vehicle 1 has not been resolved and returns the process to step ST32.

[0122] Figure 8 This is a timing diagram illustrating the operational status of the central camera module 61 and the driver camera module 62.

[0123] Figure 8 (A) to (C) in the diagram are timing diagrams under normal conditions. Figure 8 (A) in the diagram represents the illumination shooting cycle in the normal mode. Figure 8 (B) in the diagram represents the illumination and shooting action of the central shooting module 61. Figure 8 (C) in the diagram represents the illumination and shooting action of the driver's shooting module 62. Time progresses from left to right.

[0124] Figure 8 (D) to (F) in the diagram are timing diagrams in the shrinking mode. Figure 8 (D) in the zoom-out mode is the illumination shooting cycle. Figure 8 (E) in the figure represents the illumination and shooting action of the central shooting module 61. Figure 8 (F) in the diagram represents the illumination and shooting action of the driver's shooting module 62. Time progresses from left to right.

[0125] In the normal mode, such as Figure 8 As shown in (A), the central imaging module 61 and the driver imaging module 62 repeatedly illuminate and capture images at relatively long intervals. Figure 8 As shown in (B), the central imaging module 61 illuminates and images the occupants starting from the very first moment of each cycle. Figure 8 As shown in (C), the driver imaging module 62 illuminates and photographs the occupant from a second opportune moment midway through each cycle. Here, the center imaging module 61 and the driver imaging module 62 capture two images in one shot for HDR shooting. By combining the two images, an HDR image with a wide dynamic range is obtained. In this way, the illumination period of the center imaging module 61 and the illumination period of the driver imaging module 62 do not overlap. In this case, the imaging device 65 of the center imaging module 61 and the imaging device 65 of the driver imaging module 62 can photograph the occupant using only the first amount of light illuminated by each infrared LED 66. It is possible to capture images suitable for judging the occupant's retina and blood vessels.

[0126] In the collapse mode, such as Figure 8 As shown in (D), the central imaging module 61 and the driver imaging module 62 repeatedly illuminate and capture images at high speed with a relatively short cycle. Figure 8 As shown in (E), the central imaging module 61 illuminates and photographs the occupants starting from the initial high-speed imaging moment of each cycle. As... Figure 8As shown in (F), the driver imaging module 62 illuminates and photographs the occupant from the initial high-speed imaging moment of each cycle. The illumination period of the center imaging module 61 coincides with the illumination period of the driver imaging module 62. In this case, the imaging device 65 of the center imaging module 61 and the imaging device 65 of the driver imaging module 62 photograph the occupant illuminated by these multiple infrared LEDs 66. The occupant illuminated with a first light intensity or higher is photographed. An image suitable for judging the occupant's outline can be captured.

[0127] Furthermore, in retractable mode, the center camera module 61 and the driver camera module 62 repeatedly capture images at a higher speed with a shorter cycle than in normal mode. This allows for near real-time capture and updating of the occupant's head position.

[0128] As described above, in this embodiment, the center imaging module 61 and the driver imaging module 62, each equipped with an imaging device 65 and an infrared LED 66 as a light-emitting device, are each illuminated by the infrared LED 66 when the imaging device 65 is capturing images. Furthermore, when a collision of the vehicle 1 is predicted or detected, multiple imaging modules equipped with the imaging device 65 and the infrared LED 66 are activated simultaneously, illuminating the occupants simultaneously from the infrared LEDs 66 of the multiple imaging modules. Thus, in the event of a predicted or detected collision of the vehicle 1, the occupants are brightly illuminated by the light from the multiple infrared LEDs 66. By capturing images of the occupants in this state, the outlines of the occupants are clearly captured in the captured images. As a result, in this embodiment, the position of the occupants can be accurately determined based on the captured images. Based on the correct occupant position determined by the occupant monitoring device 60 of the vehicle 1 according to this embodiment, other devices of the vehicle 1, such as the occupant protection device 70, can appropriately perform protective control on the occupants according to the correct occupant position.

[0129] Furthermore, in this embodiment, even without increasing the luminous intensity of each individual infrared LED 66, the total amount of light illuminating the occupants can be increased. Therefore, after returning from a state where a collision of the vehicle 1 is predicted or detected to a state where it is not detected, no degradation due to the increase in luminous intensity occurs in each individual infrared LED 66. Or, degradation due to the increase in luminous intensity is negligible. As a result, the infrared LEDs 66 can continue to be used even after a collision of the vehicle 1 is predicted or detected. The infrared LEDs 66 can continue to be used for a long period of time regardless of whether a collision of the vehicle 1 is predicted or detected.

[0130] Furthermore, in this embodiment, under normal circumstances where a collision with vehicle 1 is not predicted or detected, the multiple imaging modules operate at staggered times to ensure that the emission times of the multiple infrared LEDs 66 are staggered. Therefore, under normal circumstances, the multiple imaging modules can capture images of the occupant at their respective times, with only the light from their respective infrared LEDs 66 illuminating the occupant. Additionally, it is difficult to continue illuminating the occupant with strong light. This allows for effective imaging of the occupant's retina, blood vessels, and other vital signs.

[0131] In this embodiment, when a collision of vehicle 1 is predicted or detected, the imaging device 65 of the multiple imaging modules captures images at a high speed compared to the case where a collision of vehicle 1 is not predicted or detected. Even with such high-speed imaging to shorten the capture period of each image, the total amount of light illuminating the occupants increases, making it difficult to capture the occupants as dark in each image, and enabling the clear capture of the occupants' outlines. Then, by obtaining multiple images with clear capture of the occupants' outlines in each short capture period, the position of the occupants after the predicted or detected collision of vehicle 1 can be obtained in real time and accurately.

[0132] Because the position of the occupants of the vehicle 1 after a collision can be predicted or detected in real time and accurately, other devices of the vehicle 1, such as the occupant protection device 70, can appropriately perform protective controls on the occupants based on their real-time and accurate occupant positions.

[0133] The above embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or alterations can be made without departing from the spirit of the invention.

[0134] For example, in the above-described embodiment, the central camera module 61, which constitutes multiple camera modules, is located in the central part of the dashboard 6 in front of the occupants, and the driver camera module 62 is located in the right side of the dashboard 6 in front of the seat 4 where the driver sits.

[0135] Multiple camera modules can also be combined in locations other than these. For example, the driver camera module 62 can be positioned together with the center camera module 61 in the central part of the dashboard 6 in front of the occupants. Additionally, multiple camera modules can be used in situations where collision prediction is not frequently required. Figure 8 The actions are as described in (A) to (C). For example, even under normal circumstances, multiple shooting modules can perform shooting actions simultaneously. In addition, the images captured by multiple shooting modules are not necessarily used for individual processing; for example, they can be used as multiple images captured by a stereo camera for processing purposes such as determining the position of occupants.

Claims

1. A passenger monitoring apparatus of a vehicle capable of irradiating light from a light emitting device to a passenger seated on the vehicle and taking an image by a photographing device, monitoring a position of the passenger, the passenger monitoring apparatus of the vehicle comprising: a plurality of photographing modules having the photographing device and the light emitting device; and a control section that controls photographing of the plurality of photographing modules, the plurality of photographing modules including a center photographing module disposed at a center of a front of the vehicle capable of simultaneously irradiating and photographing the driver of the vehicle and a driver photographing module disposed in front of the driver, wherein the control section causes each of the plurality of photographing modules to irradiate light from the light emitting device when the photographing device photographs, and simultaneously irradiates light from the light emitting device of the plurality of photographing modules to the passenger and photographs when a collision of the vehicle is predicted or detected, monitoring a position of the passenger, the control section causes the photographing device of the plurality of photographing modules to photograph at a high speed when the collision of the vehicle is predicted or detected, compared to a case where the collision of the vehicle is not predicted or detected, the light emitting device of the plurality of photographing modules irradiates infrared light to the passenger, the control section causes the light emitting device of the plurality of photographing modules to emit light at a first light amount capable of photographing a retina or a blood vessel of the passenger at least in the case where the collision of the vehicle is not predicted or detected, the control section causes the light emitting device of the plurality of photographing modules to emit light at a second light amount below the first light amount and above half of the first light amount when the collision of the vehicle is predicted or detected.

2. The passenger monitoring apparatus of the vehicle according to claim 1, wherein the control section causes the plurality of photographing modules to operate with photographing timings of the plurality of photographing modules being staggered from each other in the case where the collision of the vehicle is not predicted or detected, so that the light emitting timings of the plurality of light emitting devices are staggered from each other.

3. The passenger monitoring apparatus of the vehicle according to claim 1 or 2, wherein the control section monitors a state of the passenger based on a plurality of photographed images photographed by the photographing device of the plurality of photographing modules, in the case where the collision of the vehicle is not predicted or detected, a head position of the passenger is monitored, and a state of the passenger is monitored based on a retina or a blood vessel of the passenger photographed in at least one of the photographed images by the photographing device, in the case where the collision of the vehicle is predicted or detected, monitoring of the state of the passenger based on the retina or the blood vessel of the passenger is stopped, and the head position of the passenger is monitored based on a photographed image photographed at a high speed by one of the photographing devices.

4. A passenger protection system of a vehicle comprising: a passenger monitoring apparatus of a vehicle capable of irradiating light from a light emitting device to a passenger seated on the vehicle and taking an image by a photographing device, monitoring a position of the passenger; and a passenger protection device that performs protection control of the passenger in accordance with the position of the passenger monitored by the passenger monitoring apparatus, wherein the passenger monitoring apparatus has: a plurality of photographing modules having the photographing device and the light emitting device; and a control section that controls photographing of the plurality of photographing modules, ​ ​ The plurality of photographing modules include a center photographing module disposed at a front center of the vehicle and capable of simultaneously irradiating and photographing a driver of the vehicle, and a driver photographing module disposed in front of the driver, wherein the control section causes each of the plurality of photographing modules to irradiate light from the light emitting device when the photographing device photographs, and when a collision of the vehicle is predicted or detected, simultaneously irradiates light from the light emitting devices of the plurality of photographing modules to the occupant and photographs, and monitors a position of the occupant, the control section causes the photographing devices of the plurality of photographing modules to photograph at a high speed when a collision of the vehicle is predicted or detected, compared to a case where a collision of the vehicle is not predicted or detected, the light emitting devices of the plurality of photographing modules irradiate infrared rays to the occupant, the control section causes the light emitting devices of the plurality of photographing modules to emit light at a first light amount capable of photographing a retina or a blood vessel of the occupant at least in a case where a collision of the vehicle is not predicted or detected, the control section causes the light emitting devices of the plurality of photographing modules to emit light at a second light amount that is less than or equal to the first light amount and more than half of the first light amount when a collision of the vehicle is predicted or detected.

Images