In-vehicle monitoring device for a vehicle

Abstract

The present invention relates to an in-vehicle monitoring device for a vehicle. In a vehicle, a new monitoring function for the state of an occupant including a driver is implemented. An in-vehicle monitoring device (15) for a vehicle (1) has a first photographing means (41) and a second photographing means (48) that photograph an occupant in a passenger compartment (3) of the vehicle (1), and a control unit (37) that can perform monitoring processing for a photographed image of the first photographing means (41) and a photographed image of the second photographing means (48). The first photographing means (41) and the second photographing means (48) are arranged side by side in a vehicle width direction at a central portion in the vehicle width direction of the vehicle (1), and at least a portion of the photographing ranges overlap.

Description

Technical Field

[0001] This invention relates to an in-vehicle monitoring device. Background Technology

[0002] In order to monitor the driver, occupant monitoring devices are installed in vehicles.

[0003] Patent document 1 discloses a vehicle occupant monitoring device that focuses on the driver's eyes to capture and monitor them.

[0004] Patent document 2 discloses a vehicle occupant monitoring device that shines a strong light on the head of the occupant being photographed during a collision.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-088647

[0008] Patent Document 2: Japanese Patent Application Publication No. 2020-050078 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] However, there is a need for further improvements in vehicle occupant monitoring. In particular, in autonomous driving and advanced driver assistance systems, there is a need to more accurately monitor whether occupants such as drivers are maintained in a state suitable for driving or movement.

[0011] Thus, a new monitoring function for the status of occupants, including the driver, is required in the vehicle.

[0012] Technical solutions for solving the problem

[0013] The present invention provides an in-vehicle monitoring device for a vehicle, comprising: a first imaging component and a second imaging component for imaging the occupants of the vehicle's cabin; and a control unit capable of performing monitoring processing on the images captured by the first imaging component and the images captured by the second imaging component, wherein the first imaging component and the second imaging component are arranged side by side along the vehicle width direction in the central portion of the vehicle's width direction, and at least a portion of their imaging ranges overlap.

[0014] Preferably, the first shooting component can capture the driver's upper body or head at a narrow angle, while the second shooting component can capture the interior of the vehicle cabin at a wide angle, capable of capturing the driver and other passengers.

[0015] Preferably, the first camera component that captures the driver's upper body or head at a narrow angle can be positioned side-by-side with the driver further away in the vehicle's width direction than the second camera component that captures the interior of the vehicle cabin at a wide angle.

[0016] Preferably, the first camera component that captures the driver's upper body or head at a narrow angle can shoot at a shorter cycle than the second camera component that captures the interior of the vehicle cabin at a wide angle, and can shoot synchronously with the second camera component at a ratio of multiple shots to one shot.

[0017] Preferably, it may include: a first light-projecting component that projects light into the shooting range of the first shooting component when the first shooting component is shooting; and a second light-projecting component that projects light into the shooting range of the second shooting component when the second shooting component is shooting.

[0018] Preferably, when the first shooting component is shooting at a narrow angle to capture the upper body or head of the driver, the first projection component that projects light into the shooting range of the first shooting component can be positioned further away from the driver in the vehicle width direction than the first shooting component.

[0019] Preferably, when the second shooting component captures a wide-angle shot of the interior of the vehicle cabin, the second projection component that projects light into the shooting range of the second shooting component can be positioned further away from the center of the vehicle in the width direction than the second shooting component.

[0020] Preferably, the second light-emitting component can output less light than the first light-emitting component, and the first light-emitting component reduces the amount of light output when the second light-emitting component projects light synchronously.

[0021] Preferably, the control unit can perform: monitoring processing based solely on the image captured by the first imaging component, monitoring processing based solely on the image captured by the second imaging component, and monitoring processing based on the correlation between the image captured by the first imaging component and the image captured by the second imaging component.

[0022] Preferably, as a monitoring process based on the correlation between the images captured by the first camera and the images captured by the second camera, the control unit can determine the occupant's seating position based on the difference between the occupant's position in the image captured by the first camera and the occupant's position in the image captured by the second camera.

[0023] Invention Effects

[0024] As a camera component for capturing images of occupants in the vehicle's interior, the in-vehicle monitoring device of the present invention includes multiple camera components, such as a first camera component and a second camera component. Furthermore, the control unit can perform monitoring processing on images captured by the first camera component and images captured by the second camera component.

[0025] In particular, in this invention, the first and second imaging components for photographing the occupants of the vehicle's interior are positioned side-by-side along the vehicle's width in the central portion of the vehicle's width direction, with at least a portion of their imaging ranges overlapping. This allows the same occupant to be captured in both the first and second imaging components, maintaining a certain correlation between the images captured by these multiple imaging components.

[0026] As a result, the control unit can not only perform monitoring processing based solely on the images captured by the first camera and solely on the images captured by the second camera, but also perform monitoring processing based on the correlation between the images captured by the first camera and the images captured by the second camera. As a monitoring process based on the correlation between the images captured by the first camera and the images captured by the second camera, the control unit can also determine the occupant's seating position based on, for example, the difference between the occupant's position in the image captured by the first camera and the position in the image captured by the second camera.

[0027] Thus, in this invention, since the first and second imaging components are arranged side by side along the vehicle width direction in the central part of the vehicle and at least a portion of their imaging ranges overlap, as a new monitoring function for the state of occupants such as the vehicle driver, the control unit can perform monitoring processing based on the correlation between the images captured by the first and second imaging components. Attached Figure Description

[0028] Figure 1 This is an explanatory diagram of a car to which the present invention is applied.

[0029] Figure 2 yes Figure 1 A diagram illustrating the control system of a car.

[0030] Figure 3 yes Figure 2 An illustration of one example of the types of monitoring devices used in automobiles for monitoring occupants.

[0031] Figure 4 yes Figure 3 An explanatory diagram of the occupant monitoring device.

[0032] Figure 5 It has Figure 4 A diagram illustrating the configuration of an LCD device with a display control panel.

[0033] Figure 6 yes Figure 4 A diagram illustrating the configuration of the first and second camera modules.

[0034] Figure 7 It is about Figure 4 The timing diagram of basic shooting and illumination of the first and second camera modules.

[0035] Figure 8 This is about the implementation method. Figure 4 The timing diagram of the shooting and emission of the first and second camera modules.

[0036] Figure 9 yes Figure 4 The flowchart of the main control performed by the monitoring and control department.

[0037] Figure 10 yes Figure 4 The flowchart shows the monitoring and control of multiple occupants by the monitoring and control department. Detailed Implementation

[0038] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0039] Figure 1 This is an explanatory diagram of a car 1 to which the present invention is applied.

[0040] Car 1 is an example of a vehicle. Car 1 can be powered by an internal combustion engine, powered by the electricity stored in a battery, or a combination of both.

[0041] In addition, the car 1 can drive according to the driver's driving operation, or it can drive through autonomous driving or driving assistance, or it can switch between the two.

[0042] In addition to automobiles 1, such vehicles may also include personal mobility vehicles, powered two-wheeled vehicles, powered three-wheeled vehicles, buses, trucks, aircraft, rail vehicles, and the like.

[0043] exist Figure 1 The car body 1 has a passenger compartment 3 within its body 2, which includes a driver and other passengers. The passenger compartment 3 contains multiple front seats 4 for individual seating of the driver and other passengers, and long rear seats 4 for side-by-side seating of multiple passengers. While in motion, the driver and other passengers are required to be seated properly in their seats 4 and to wear safety equipment such as seat belts.

[0044] A dashboard 5 extending along the width of the vehicle body 2 is provided at the front of the cabin 3, which is located in front of the multiple front seats 4.

[0045] Figure 2 yes Figure 1 Explanation diagram of the control device 10 of car 1.

[0046] Figure 2 The control device 10 includes: a door opening / closing sensor 11, a vehicle speed sensor 12, a parking sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driver assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 connecting them.

[0047] The in-vehicle network 21 can also be a wired communication network for the vehicle 1, such as one based on CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 can be a communication network such as a LAN, or a combination of both. A wireless communication network may also be included as a part of the in-vehicle network 21.

[0048] The door opening and closing sensor 11 detects the opening and closing of the doors of the car 1.

[0049] Vehicle speed sensor 12 detects the speed of the moving car 1. Vehicle speed sensor 12 can also detect a stopped state.

[0050] The parking sensor 13 detects a parking state that keeps the car 1 stationary. The parking sensor 13 can also detect, for example, the operation of a gear lever (not shown) to the parking position and the operation of a parking lever (not shown) to the brake position as parking states.

[0051] When the door opening / closing sensor 11 detects the opening or closing of a door, the occupant monitoring device 15 performs identification processing of the occupants sitting in the vehicle compartment 3, and then monitors the status of the occupants. If multiple occupants are identified, the occupant monitoring device 15 can also monitor each occupant separately.

[0052] The occupant monitoring device 15 can also output information about each identified occupant, and monitoring-based information, to various parts of the control device 10 via the in-vehicle network.

[0053] When the driver and other occupants are in a prescribed state, the occupant monitoring device 15 can also perform controls such as issuing warnings to attract the occupants' attention, or perform controls for hazard prevention or avoidance. At this time, the occupant monitoring device 15 can also output control information to various parts of the control device 10 via the in-vehicle network.

[0054] The speaker device 14 outputs sounds, warning sounds, etc. The speaker device 14 can also output warnings to occupants other than the driver monitored by the occupant monitoring device 15.

[0055] Position setting device 16 adjusts, for example, the fore-aft and vertical position of seat 4 and the angle of the backrest, and the steering wheel 7 (see below). Figure 4 The position setting device 16 adjusts the seat position based on the occupant information output by the occupant monitoring device 15, including the forward, backward, up, and down positions and angles of various pedals.

[0056] The driver assistance device 17 controls the driving operations of the vehicle 1 in manual driving to assist the driver, or controls the driving of the vehicle 1 through automatic driving. The driver assistance device 17 controls the acceleration, deceleration, stopping, and steering of the vehicle 1. The driver assistance device 17 can also perform driving assistance corresponding to the driver based on the driver information output by the occupant monitoring device 15.

[0057] The external communication device 18 establishes wireless communication paths, for example, with base stations of public wireless communication networks, base stations of commercial wireless communication networks, and base stations for high-traffic information, and performs data communication using the established wireless communication paths. The external communication device 18 can also perform bidirectional data communication, for example, with server devices for assisted autonomous driving. The external communication device 18 can also send occupant information, including the driver's information, output by the occupant monitoring device 15 to the server device as, for example, emergency assistance information.

[0058] In the event of a detected or predicted collision with the vehicle 1, the occupant protection device 19 performs controls to protect the occupants. The occupant protection device 19 protects the occupant seated in seat 4 by deploying, for example, an airbag (not shown), or by applying tension to the seatbelt. The occupant protection device 19 can also perform occupant protection based on occupant information output from the occupant monitoring device 15.

[0059] The air conditioning unit 20 controls the temperature and oxygen concentration of the vehicle compartment 3. The air conditioning unit 20 supplies, for example, cooled or heated air to the vehicle compartment 3, thereby adjusting the vehicle compartment 3 to a set temperature. The air conditioning unit 20 can also perform air conditioning based on occupant information output from the occupant monitoring device 15.

[0060] Through control based on the occupant information output by the occupant monitoring device 15, occupants can ride comfortably, for example, according to their own settings. The driver can concentrate, for example, on the driving of the car 1.

[0061] Figure 3 yes Figure 2 An illustration of an example of the type of monitoring device 15 for occupants of a car 1.

[0062] exist Figure 3Examples of surveillance methods include: strabismus, drowsiness, sitting posture, emergency events, seat belt wearing status, and child car seat installation status.

[0063] It is meaningful for the occupant monitoring device 15 to perform only one of these monitoring controls, but it is preferred as an in-vehicle monitoring device to perform multiple of these monitoring controls, or to perform multiple full monitoring controls.

[0064] In addition, among these types of surveillance, strabismus, drowsiness, and emergency situations are considered essential for drivers.

[0065] Regarding seating posture, emergency situations, and seatbelt wearing status, the aim is to monitor not only the driver but also other occupants.

[0066] Regarding the installation status of child seats, there are already cars 1 that have the function of detecting or setting the installation status, so it is possible to consider lowering the priority of monitoring compared to these.

[0067] By implementing these multiple monitoring methods, the monitoring capability of the occupant monitoring device 15 of vehicle 1 is enhanced. In autonomous driving or highly assisted driving, it is desirable to maintain the state of occupants such as drivers in a state suitable for driving, and it is desirable to be able to execute each of the multiple monitoring methods more accurately.

[0068] However, when installing the aforementioned multiple surveillance systems in the vehicle 1, the occupant monitoring device 15 may require a sensor for each type of surveillance and for each occupant. On the other hand, the space available for installing the sensors inside the vehicle compartment 3 is limited. When the installation of additional sensors becomes difficult due to the limited space in the vehicle 1, the multi-functionality of the occupant monitoring device 15 is limited, and the occupant monitoring device 15 may not be able to achieve a certain level of multi-functionality.

[0069] Thus, in vehicle 1, when multi-functionalizing the monitoring of multiple occupants, including the driver, the increase in the number of imaging sensors used for monitoring may be limited. Furthermore, as the number of images captured by the imaging sensors increases, their processing load also increases.

[0070] In this embodiment, an example is disclosed that appropriately avoids the limitations of the occupant monitoring device 15 of such a car 1 and improves the feasibility of various monitoring functions as described above.

[0071] Specifically, the occupant monitoring device 15 of this embodiment is based on images captured by two imaging sensors to achieve various controls such as driver strabismus, driver drowsiness, the seating position or posture of each occupant, emergency events of each occupant, the wearing status of each occupant's seat belt, the installation status of child seats, and image output.

[0072] Figure 4 This is an embodiment of the present invention as an in-vehicle monitoring device for automobile 1. Figure 3 Explanation diagram of the occupant monitoring device 15.

[0073] Figure 4 The occupant monitoring device 15 monitors not only the driver, but also multiple occupants other than the driver who are seated in multiple seats 4.

[0074] Figure 4 The occupant monitoring device 15 includes: a first camera module 31, a second camera module 32, an LCD device 33 with a display operation panel 34, an input / output device 35, a memory 36, and a monitoring and control unit 37 connected thereto.

[0075] The input / output device 35 is connected to the in-vehicle network 21. The input / output device 35 inputs and outputs data with other parts of the vehicle 1 via the in-vehicle network 21.

[0076] The LCD device 33 displays a screen on the display operation panel 34 for each occupant of the vehicle 1 to recognize. The display screen includes, for example, an operation screen operated by the occupants.

[0077] The display operation panel 34 is a transparent or semi-transparent panel that overlaps with the display surface of the liquid crystal device 33. The display operation panel 34 detects the occupant's operation on the display surface of the liquid crystal device 33 and outputs information such as the operation position to the monitoring and control unit 37.

[0078] The first camera module 31 has a first LED 43, a first shooting sensor 41, and a first device controller 45 connected to them.

[0079] The first imaging sensor 41 can also be a semiconductor optical sensor such as a CCD or CMOS sensor. The first imaging sensor 41 outputs imaging data containing the captured image to the monitoring and control unit 37.

[0080] A narrow-angle lens 42 is superimposed on the first imaging sensor 41. In order to suppress distortion in the peripheral part of the image, the narrow-angle lens 42 may also be composed of multiple optical lenses.

[0081] The first LED 43 can also be a semiconductor light-emitting element. A narrow-angle projection lens 44 is stacked on top of the first LED 43. The first LED 43 can also project, for example, infrared light. In this case, the first imaging sensor 41 outputs imaging data, including the infrared image, to the monitoring and control unit 37.

[0082] The first device controller 45 controls the operation of the first imaging sensor 41 and the first LED 43. During the period when the first imaging sensor 41 is capturing images, the first device controller 45 causes the first LED 43 to illuminate. The control cycle of the first device controller 45 can be set by the monitoring and control unit 37.

[0083] The second camera module 32 has a second LED 46, a second shooting sensor 48, and a second device controller 50 connected to them.

[0084] The second imaging sensor 48 can also be a semiconductor optical sensor such as a CCD or CMOS sensor. The second imaging sensor 48 outputs the imaging data, including the captured image, to the monitoring and control unit 37.

[0085] A wide-angle lens 49 is superimposed on the second imaging sensor 48. To suppress distortion at the periphery of the image, the wide-angle lens 49 may also be composed of multiple optical lenses.

[0086] The second LED 46 can also be a semiconductor light-emitting element. A wide-angle projection lens 47 is superimposed on the second LED 46. The second LED 46 can also project, for example, infrared light. In this case, the second imaging sensor 48 outputs the imaging data, including the infrared image, to the monitoring and control unit 37.

[0087] The second device controller 50 controls the operation of the second imaging sensor 48 and the second LED 46. During the period when the second imaging sensor 48 is capturing images, the second device controller 50 causes the second LED 46 to illuminate. The control cycle of the second device controller 50 can be set by the monitoring and control unit 37.

[0088] Moreover, such as Figure 4 As shown, the first LED 43 of the first camera module 31 projects light onto the driver's upper body or head at a narrow angle, and the first image sensor 41 captures images of the driver's upper body or head at a narrow angle. Thus, the first image sensor 41 can capture bright, high-resolution images of the driver's upper body or head. The images captured by the first image sensor 41 can capture the driver's eyes, eyeballs, etc., at high resolution.

[0089] In contrast, the second LED 46 of the second camera module 32 projects a wide-angle overall light onto the interior of the vehicle compartment 3, and the second imaging sensor 48 captures a wide-angle overall image of the interior of the vehicle compartment 3. Thus, the second imaging sensor 48 can capture a bright image including the driver and multiple occupants other than the driver. The image from the second imaging sensor 48 can capture the driver, the passenger in the front passenger seat (other than the driver), and the passengers in the rear seats.

[0090] The first imaging sensor 41 and the second imaging sensor 48, which are positioned side by side along the width direction of the vehicle 1 in the central part of the vehicle 1, capture images of the occupants of the vehicle 1's cabin 3 from different angles or shooting ranges.

[0091] In addition, the first shooting sensor 41 and the second shooting sensor 48 are configured to overlap a portion of their shooting ranges to jointly capture the driver.

[0092] Memory 36 stores programs and data. Memory 36 can also be composed of non-volatile memory and volatile memory. Non-volatile memory includes, for example, HDD, SSD, EEPROM, etc. Volatile memory includes, for example, RAM.

[0093] The memory 36 of the occupant monitoring device 15 can also manage and record data of multiple occupants riding in the vehicle 1 on a per-occupant basis. Figure 4 The data represents first crew data 61 concerning the first crew member and second crew data 62 concerning the second crew member. A database is constructed in memory 36 using this multiple crew data.

[0094] The occupant data, such as the first occupant data 61 and the second occupant data 62, may also include the unique identification information of each occupant, such as registration data captured by the first imaging sensor 41 or the second imaging sensor 48 of the occupant's head or eyes, and various setting data performed by the occupant. The setting data may also include information such as the occupant's seat position, the initial settings for the presence or absence of driver assistance, driving preference information under autonomous driving, information on the server device used, occupant protection settings, and air conditioning settings.

[0095] In this way, the memory 36 can record the shooting data captured by each occupant when the first shooting sensor 41 or the second shooting sensor 48 is facing the predetermined screen displayed on the liquid crystal device 33 from the front as the registered shooting data of each occupant.

[0096] Additionally, memory 36 can also record crew data about unregistered general crew members.

[0097] The monitoring and control unit 37 can also be, for example, an ECU, a CPU, or other microcomputer. The monitoring and control unit 37 reads and executes programs from the memory 36. Thus, the monitoring and control unit 37 serves as the control unit of the occupant monitoring device 15.

[0098] The monitoring and control unit 37 can perform processing on the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48.

[0099] The monitoring and control unit 37 can also perform various processing on the occupants in the vehicle compartment 3 based on images captured by the first imaging sensor 41 and the second imaging sensor 48 from different viewing angles or shooting ranges.

[0100] For example, the monitoring and control unit 37 can also perform controls for identifying or registering occupants riding in the car 1. The monitoring and control unit 37 can also perform controls, for example, corresponding to the operation content determined as the occupant's operation of the operating screen of the LCD device 33.

[0101] In addition, when an occupant is in the vehicle 1, the monitoring and control unit 37 identifies the occupant and performs monitoring and control of the occupant status.

[0102] In particular, for passengers who are drivers, the monitoring and control unit 37 can also use the respective registered shooting data registered in the memory 36 as the reference data for judgment, and judge the state of strabismus, drowsiness, emergency situation, etc. based on the state of each passenger's eyes.

[0103] Figure 5 It has Figure 4 A diagram illustrating the configuration of the LCD device 33 on the display operation panel 34.

[0104] exist Figure 5 The middle section indicates that there is an instrument panel 5 and a center console 6 located at the front of the vehicle compartment 3.

[0105] Moreover, it has Figure 4 The LCD device 33 of the display and operation panel 34 is located in the center of the vehicle 1 in the width direction, facing the rear of the vehicle 1. The LCD device 33 with the display and operation panel 34 is longitudinally embedded from the instrument panel 5 to the center console 6.

[0106] Figure 6 yes Figure 4 A diagram illustrating the configuration of the first camera module 31 and the second camera module 32.

[0107] like Figure 6 As shown, Figure 4 The first camera module 31 and the second camera module 32 are configured on the back side of the display operation panel 34 of the LCD device 33, with the first camera module 31 positioned above the LCD device 33.

[0108] The first LED 43, the first shooting sensor 41, the second shooting sensor 48, and the second LED 46 of the first camera module 31 and the second camera module 32 are arranged side by side along the width of the vehicle 1.

[0109] The first imaging sensor 41 is positioned on the opposite side of the driver, with the center position Y0 in the width direction of the vehicle 1 as a reference. The second imaging sensor 48 is positioned on the driver's side, with the center position Y0 in the width direction of the vehicle 1 as a reference. Thus, the first imaging sensor 41, which captures the driver's upper body or head at a narrow angle, is positioned side by side with the second imaging sensor 48, which captures the entire interior of the vehicle 3 at a wide angle, and is further away from the driver in the width direction of the vehicle 1. Figure 5 The steering wheel 7 shown is difficult to position between the first image sensor 41 and the driver. The steering wheel 7 is difficult to be captured in the image captured by the first image sensor 41.

[0110] Thus, the first imaging sensor 41 and the second imaging sensor 48 are arranged side by side in the width direction of the car 1, therefore, relative to Figure 6 The driver's position in the image captured by the first sensor 41 and the second sensor 48 will be offset by parallax. Assuming, for example, that the first sensor 41 and the second sensor 48 capture images from the same angle or with the same shooting range, the driver's position in the image captured by the first sensor 41 will also be offset by parallax. This parallax depends on the sensor spacing between the first sensor 41 and the second sensor 48.

[0111] The first LED 43 is positioned further away from the driver in the width direction of the car 1, making it farther away from the driver than the first imaging sensor 41. The light emitted by the first LED 43 to the driver is less likely to be blocked by the steering wheel 7.

[0112] The second LED 46 is positioned closer to the driver's side in the width direction of the vehicle 1 than the second imaging sensor 48. The second LED 46 is also positioned further away from the center position Y0 in the width direction of the vehicle 1 than the second imaging sensor 48. Therefore, the second imaging sensor 48 can be positioned close to the center position Y0 in the central portion of the width direction of the vehicle 1. By being positioned close to the center position Y0, the second imaging sensor 48 can capture a wide-angle, overall view of the interior of the vehicle cabin 3 from the central portion of the width direction of the vehicle 1. The second imaging sensor 48 can capture images not only of the occupants seated in the front seats 4 but also of the occupants seated in the rear seats 4.

[0113] Figure 7 It is about Figure 4 The timing diagram of basic shooting and illumination of the first camera module 31 and the second camera module 32.

[0114] exist Figure 7 In the middle, time moves from left to right.

[0115] Figure 7In diagram (A), the first image sensor 41 is in its capturing state. The first image sensor 41 captures images at regular intervals. Based on the images captured by the first image sensor 41, narrow-angle monocular processing (NAPMS) for driver monitoring can be performed. During periods of high voltage, the first image sensor 41 captures images. The same applies to other parameters.

[0116] Figure 7 (B) shows the projection state of the first LED 43. During the shooting period of the first shooting sensor 41, the first LED 43 projects light into the shooting range of the first shooting sensor 41 in a narrow angle.

[0117] Figure 7 (C) shows the shooting state of the second shooting sensor 48. The second shooting sensor 48 performs shooting at regular intervals. Based on the images captured by the second shooting sensor 48, wide-angle monocular processing can be performed to monitor the driver and other occupants. In wide-angle monocular processing, not only can the processing of monitoring the state of each of the multiple occupants included in the captured image be performed, but also the processing of transmitting the captured image for viewing on a mobile terminal, etc. In the observed image, all occupants, including the driver, can be captured. In this case, such as Figure 7 As shown in (C), the images captured by the second imaging sensor 48 can also be used alternately for monitoring and transmission processing. Even in this case, the first imaging sensor 41 captures and monitors the driver's state at short intervals, so it can be assumed that the movement of the car 1 will not be hindered.

[0118] Furthermore, the shooting cycle of the second imaging sensor 48 is twice that of the first imaging sensor 41. Additionally, the shooting timing of the second imaging sensor 48 is synchronized with that of the first imaging sensor 41. With the images captured by the first imaging sensor 41 and the second imaging sensor 48 synchronized, parallax based on factors such as the sensor spacing between the first and second imaging sensors 41 can be calculated with high precision based on the deviation of the driver's shooting position in these images. Furthermore, compound eye processing can be performed using the synchronized images captured by the first and second imaging sensors 41.

[0119] In this way, the first imaging sensor 41, which captures the driver's upper body or head at a narrow angle, captures images at a shorter cycle than the second imaging sensor 48, which captures the entire interior of the vehicle compartment 3 at a wide angle, and performs multiple captures simultaneously with the second imaging sensor 48. When the first imaging sensor 41 and the second imaging sensor 48 capture images simultaneously, the monitoring control unit 37 can perform monitoring processing based on the images captured by the first imaging sensor 41 and the second imaging sensor 48. When the first imaging sensor 41 and the second imaging sensor 48 capture images simultaneously, the monitoring control unit 37 can perform different processing than when the first imaging sensor 41 captures images alone or when the second imaging sensor 48 captures images alone.

[0120] Figure 7 In the middle (D), the second LED 46 is in its projection state. During the shooting period of the second shooting sensor 48, the second LED 46 projects light into the shooting range of the second shooting sensor 48 with a wide angle.

[0121] Figure 7 The middle (E) is the one being projected. Figure 7 The light of the first LED43 in (B) Figure 7 The amount of light projected by the driver of the second LED46 in the middle (D). For example... Figure 7 With the first LED43 and the second LED46 outputting light in the same manner, the amount of light projected onto the driver is increased or decreased at regular shooting intervals. The brightness of the driver in the captured image varies according to the amount of light projected.

[0122] Figure 8 This is about the implementation method. Figure 4 The timing diagram of the shooting and emission of the first camera module 31 and the second camera module 32.

[0123] Figure 8 Middle (A)~ Figure 8 The middle (E) is basically the same as Figure 7 Middle (A)~ Figure 7 The middle (E) corresponds to this.

[0124] However, as Figure 8 As shown in (D), the light emission of the second LED46 is suppressed to a lesser extent than that of the second LED46. Figure 8 The first LED 43 shown in (B) has a low light emission. The second LED 46 outputs less light than the first LED 43.

[0125] Furthermore, when the second LED 46 projects light synchronously, the amount of light emitted by the first LED 43 is suppressed to be lower than when the second LED 46 does not project light synchronously. When the second LED 46 projects light synchronously, the first LED 43 reduces the amount of light it outputs by an amount that roughly corresponds to the amount of light it outputs from the second LED 46.

[0126] Therefore, as Figure 8 As shown in (E), the amount of light emitted by the driver when the first LED 43 and the second LED 46 illuminate simultaneously is approximately the same as the amount of light emitted by the driver when only the first LED 43 illuminates. The brightness of the driver in the captured image can be constant. Basically, no brightness correction processing corresponding to the illumination conditions of the captured image is required.

[0127] Thus, in this embodiment, in order for the first imaging sensor 41 and the second imaging sensor 48 to capture images synchronously, even if the first LED 43 and the second LED 46 sometimes emit light synchronously, the amount of light emitted to the driver can be easily changed between the case of synchronous illumination and the case of the first LED 43 emitting light alone. The monitoring control unit 37 for monitoring occupants can continue to acquire images with stable brightness as images captured by the first imaging sensor 41.

[0128] Figure 9 yes Figure 4 The flowchart of the main control performed by the monitoring and control unit 37.

[0129] When a new occupant boards the vehicle 1, the monitoring and control unit 37 repeats the process. Figure 9 The processing.

[0130] In step ST1, the monitoring and control unit 37 determines whether a new occupant has entered the vehicle 1. The occupant opens a door (not shown) of the vehicle 1 and sits in seat 4. The monitoring and control unit 37 may also detect and determine whether a new occupant has entered the vehicle 1 based on, for example, the opening and closing detection signal of the door opening and closing sensor 11. Alternatively, for example, the monitoring and control unit 37 may also determine whether a new occupant has entered the vehicle 1 based on the image data captured by the first image sensor 41 or the second image sensor 48. If no new occupant has entered the vehicle 1, the monitoring and control unit 37 repeats this process. When a new occupant enters the vehicle 1, the monitoring and control unit 37 causes the process to proceed to step ST2.

[0131] In step ST2, the monitoring and control unit 37 determines the seating position of the newly arrived occupant. The monitoring and control unit 37 can also determine the seating position based on data captured by the second imaging sensor 48 inside the vehicle. When the monitoring and control unit 37 detects that the occupant is stably seated in seat 4, it determines that the occupant is in that position.

[0132] In step ST3, the monitoring and control unit 37 performs personal identification processing on the newly boarding occupant. The monitoring and control unit 37 may also perform personal identification processing on the newly boarding occupant based on the latest image data from the second imaging sensor 48 inside the vehicle.

[0133] The monitoring and control unit 37 can, for example, compare the occupant image contained in the captured data with the occupant data of multiple occupants recorded in the memory 36 to identify each occupant with high precision. In this case, the monitoring and control unit 37 can also compare the image components being observed only based on the registered captured data of each occupant registered in the memory 36. Alternatively, the monitoring and control unit 37 can compare the feature points extracted from each image instead of directly comparing the images. The monitoring and control unit 37 can also identify a passenger as the occupant of the registered captured data if the registered captured data contained in the occupant data registered in the memory 36 is consistent with the data at a certain level of accuracy. In this case, the monitoring and control unit 37 identifies the passenger as the occupant determined by comparing with the multiple occupant data registered in the memory 36. If there is no registered captured data consistent with the data at a certain level of accuracy among the multiple occupant data registered in the memory 36, the monitoring and control unit 37 can also identify the passenger as an unregistered occupant.

[0134] Furthermore, if there are occupants whose occupant data is recorded in the memory 36, the monitoring and control unit 37 can further use this occupant data to perform setting processing. In this case, the monitoring and control unit 37 outputs the setting data information to each part of the vehicle 1. This performs processing regarding occupant seating positions, initial settings for the presence or absence of driver assistance, driving preference information under automatic driving, information on the server devices used, occupant protection settings, and air conditioning settings. Additionally, based on the latest acquired image data, the monitoring and control unit 37 determines, for example, whether a child seat is installed in the front passenger seat 4. If a child seat is installed, it can also perform a setting to prevent the airbag from deploying in the front passenger seat 4.

[0135] In step ST4, the monitoring and control unit 37 determines whether a new passenger needs to be re-registered. The monitoring and control unit 37 displays a confirmation screen for registration processing on, for example, the LCD device 33. If a corresponding passenger operation is performed on the display operation panel 34, it is determined that re-registration is required, and the process proceeds to step ST5. If an operation that does not require registration is performed, the monitoring and control unit 37 proceeds the process to step ST6.

[0136] In step ST5, the monitoring and control unit 37 performs the process of re-registering the newly boarding passenger. The monitoring and control unit 37 appends the passenger data of the new passenger to the memory 36.

[0137] In step ST6, the monitoring and control unit 37 begins to monitor and control the newly boarded passenger.

[0138] In step ST7, the monitoring and control unit 37 determines whether to terminate occupant monitoring and control. For example, the monitoring and control unit 37 may determine to terminate occupant monitoring and control when the vehicle 1 stops and the ignition is turned off, when the vehicle reaches its destination and stops, or when an occupant disembarks. The monitoring and control unit 37 may also determine occupant disembarkation based on, for example, the opening and closing detection of the door opening and closing sensor 11, or images from the first imaging sensor 41 or the second imaging sensor 48. If occupant monitoring and control has not been terminated, the monitoring and control unit 37 repeats this process. If occupant monitoring and control has been terminated, the monitoring and control unit 37 proceeds to step ST8.

[0139] In step ST8, the monitoring and control unit 37 performs the end processing for occupant monitoring. The monitoring and control unit 37 obtains setting information from each part of the vehicle 1, such as when each occupant alights, and updates the occupant data recorded in the memory 36. Thus, the occupant data registered in the memory 36 becomes data adapted to the occupant's preferences. The latest settings for the occupants are automatically updated during the next ride. Additionally, the monitoring and control unit 37 can also temporarily record occupant data for unregistered occupants in the memory 36. Therefore, if a registration operation is performed later for that occupant, the settings can be immediately linked.

[0140] Then, the monitoring and control unit 37 ended. Figure 9 Monitoring and control.

[0141] Figure 10 yes Figure 4 The flowchart shows the monitoring and control of multiple occupants performed by the monitoring and control unit 37.

[0142] When the monitoring and control unit 37 determines that in Figure 9 When step ST6 begins monitoring and controlling the initial crew members, it is repeated. Figure 10 The monitoring and control continues until step ST7 determines that the last passenger has disembarked and the process ends.

[0143] In step ST11, the monitoring and control unit 37 determines whether the processing timing is based on the image capture.

[0144] A first imaging sensor 41 captures the driver's upper body or head at a narrow angle, and a second imaging sensor 48 captures the entire interior of the vehicle cabin 3 at a wide angle. Figure 8 Each shooting timer executes a shooting action, and the captured data is output to the monitoring and control unit 37. If new shooting data is acquired from the first shooting sensor 41 or the second shooting sensor 48, the monitoring and control unit 37 determines that it is based on the shooting processing timer and proceeds to step ST12. If no new shooting data is acquired, the monitoring and control unit 37 repeats this process and waits.

[0145] In step ST12, the monitoring and control unit 37 begins narrow-angle monocular processing based solely on images captured by the first imaging sensor 41, which essentially captures the driver's upper body or head.

[0146] In step ST13, the monitoring and control unit 37 determines whether the driver is strabismus based on the image captured by the first image sensor 41. The monitoring and control unit 37 extracts, for example, image components of the driver's eyes from the image captured by the first image sensor 41. The monitoring and control unit 37 determines whether the driver's gaze, inferred from the image components of the driver's eyes, is directed towards the direction of travel of the vehicle 1. If the driver's gaze is in the direction of travel of the vehicle 1, the monitoring and control unit 37 determines that the driver is not strabismus. If the driver's gaze is not in the direction of travel of the vehicle 1 multiple times consecutively, the monitoring and control unit 37 determines that the driver is strabismus.

[0147] In step ST14, the monitoring and control unit 37 determines whether the driver is drowsy based on the image captured by the first image sensor 41. The monitoring and control unit 37 extracts, for example, image components of the driver's eyes from the image captured by the first image sensor 41. If the driver's eyes are open, as inferred from the image components of the driver's eyes, the monitoring and control unit 37 determines that the driver is not drowsy. If the driver's eyes are closed multiple times consecutively, the monitoring and control unit 37 determines that the driver is drowsy.

[0148] In step ST15, the monitoring and control unit 37 determines an emergency for the driver based on the image captured by the first imaging sensor 41. The monitoring and control unit 37 extracts image components from the image captured by the first imaging sensor 41, such as the driver's seating posture. If the driver's seating posture is, for example, with their head lowered and not in a suitable driving position, the monitoring and control unit 37 determines that an emergency has occurred. Alternatively, for example, the monitoring and control unit 37 can obtain information such as the driver's pulse and blood flow from the image captured by the first imaging sensor 41. In this case, the monitoring and control unit 37 determines that an emergency has occurred if the driver's pulse is greater than a threshold or if the increase or decrease in blood flow is greater than a threshold.

[0149] Therefore, the monitoring and control unit 37 ends the narrow-angle monocular processing.

[0150] In step ST16, the monitoring and control unit 37 determines whether the current processing timing is the timing for wide-angle synchronous shooting.

[0151] like Figure 8As shown, the second imaging sensor 48, which captures a wide-angle overall view of the interior of the vehicle compartment 3, sometimes captures images simultaneously with the first imaging sensor 41, and outputs the captured data to the monitoring and control unit 37. During this synchronous capture timing, the monitoring and control unit 37 determines that the current processing timing is for wide-angle synchronous capture and proceeds to step ST17. If only new captured data is acquired from the first imaging sensor 41 during this processing timing, the monitoring and control unit 37 directly performs post-processing without executing other judgment processes, and proceeds to step ST28.

[0152] In step ST17, the monitoring and control unit 37 determines whether it is a timing requirement for compound eye processing related to synchronized shooting. For example... Figure 8 As shown, when the second imaging sensor 48 and the first imaging sensor 41 capture images synchronously, the monitoring and control unit 37 alternately performs compound eye processing (using both images captured by the first imaging sensor 41 and images captured by the second imaging sensor 48) and wide-angle monocular processing (processing essentially only images captured by the second imaging sensor 48). This is done before starting... Figure 10 In the case of initial synchronized shooting during monitoring and control, if the monitoring and control unit 37 determines that the timing for the previous wide-angle monocular processing was compound eye processing, the processing proceeds to step ST18. If the monitoring and control unit 37 determines that the timing for the previous compound eye processing was not compound eye processing but wide-angle monocular processing, the processing proceeds to step ST22. Information related to the previous processing content can also be recorded in memory 36. After the processing determination in step ST17, the monitoring and control unit 37 can also update the information of the previous processing recorded in memory 36.

[0153] In step ST18, the monitoring and control unit 37 begins compound eye processing of the images captured by the first imaging sensor 41 and the second imaging sensor 48. The monitoring and control unit 37 can also determine the driver's shooting position that is commonly included in the images captured by the first imaging sensor 41 and the second imaging sensor 48.

[0154] In step ST19, the monitoring control unit 37 performs a correction process for the viewing angle difference between the images captured by the first imaging sensor 41 and the second imaging sensor 48. The monitoring control unit 37 calculates the viewing angle difference corresponding to the sensor interval between the first and second imaging sensors 41, based on, for example, the driver's shooting position contained in the image captured by the first imaging sensor 41 and the driver's shooting position contained in the image captured by the second imaging sensor 48. The monitoring control unit 37 may also correct the shooting position information of each pixel used in narrow-angle monocular processing of the image captured by the first imaging sensor 41 and the shooting position information of each pixel used in wide-angle monocular processing of the image captured by the second imaging sensor 48, based on the viewing angle difference obtained through calculation. The monitoring control unit 37 may also record the obtained viewing angle difference information and the shooting position information of each pixel in the memory 36.

[0155] In step ST20, the monitoring and control unit 37 acquires the parallax of the first imaging sensor 41 and the second imaging sensor 48. The monitoring and control unit 37 may also calculate the parallax of the first imaging sensor 41 and the second imaging sensor 48 based on the placement position of the first imaging sensor 41, the placement position of the second imaging sensor 48, the sensor spacing, the viewing angle difference, etc.

[0156] In step ST21, the monitoring and control unit 37 determines the driver's seating position. Based on images captured by the first image sensor 41 and the second image sensor 48, the monitoring and control unit 37 calculates the direction and distance of parts of the driver, such as the head, neck, and waist, and determines the position of those parts within the seat 4. The monitoring and control unit 37 can also determine whether the determined position of the driver's parts is within the correct seat position.

[0157] Therefore, the monitoring and control unit 37 ends the compound eye processing. Then, for post-processing, the monitoring and control unit 37 causes the processing to proceed to step ST28.

[0158] In step ST22, the monitoring and control unit 37 begins wide-angle monocular processing, which is essentially only processed for images captured by the second imaging sensor 48.

[0159] In step ST23, the monitoring and control unit 37 determines the seating position (sitting posture) of each of the multiple occupants, including the driver, based on the images captured by the second imaging sensor 48. The monitoring and control unit 37 extracts, for example, image components of each occupant from the images captured by the second imaging sensor 48. The monitoring and control unit 37 determines whether the image components of each occupant are suitable as a seating position (sitting posture) for the moving vehicle 1. For example, if an occupant's seating position (sitting posture) is lateral, the monitoring and control unit 37 determines that the occupant's seating position (sitting posture) is not suitable as a seating position (sitting posture) for the moving vehicle 1.

[0160] In step ST24, the monitoring and control unit 37 determines an emergency for each of the multiple occupants, including the driver, based on the images captured by the second imaging sensor 48. The monitoring and control unit 37 extracts, for example, image components of each occupant from the images captured by the second imaging sensor 48. The monitoring and control unit 37 determines whether the image components of each occupant correspond to an emergency. For example, if an occupant's seating position (sitting posture) is lateral, the monitoring and control unit 37 determines that the occupant's seating position (sitting posture) corresponds to an emergency.

[0161] In step ST25, the monitoring and control unit 37 determines the seat belt wearing status of each of the multiple occupants, including the driver, based on the image captured by the second imaging sensor 48. The monitoring and control unit 37 extracts the seat belt image component for each of the multiple occupants, including the driver. If the seat belt image component does not cross in front of the upper body of each occupant, the monitoring and control unit 37 determines that the seat belt is not properly worn.

[0162] In step ST26, the monitoring and control unit 37 determines the installation status of the child seat based on the image captured by the second imaging sensor 48. The monitoring and control unit 37 attempts to extract image components of the child seat. If the image components of the child seat can be extracted, the monitoring and control unit 37 determines that a child seat is installed.

[0163] In step ST27, the monitoring and control unit 37 generates and outputs a monitoring image for observation on a mobile terminal or the like based on the image captured by the second imaging sensor 48.

[0164] Therefore, the monitoring and control unit 37 ends the wide-angle monocular processing. Then, for post-processing, the monitoring and control unit 37 causes the processing to proceed to step ST28.

[0165] In step ST28, the monitoring control unit 37 begins post-processing based on the judgments of the multiple monitoring types mentioned above. The monitoring control unit 37 first determines whether the judgments of the multiple monitoring types include content requiring a warning; if so, the process proceeds to step ST29. Otherwise, the monitoring control unit 37 proceeds to step ST30.

[0166] In step ST29, the monitoring and control unit 37 outputs a warning corresponding to the content of the monitoring type that requires a warning.

[0167] Warnings to the driver can also be, for example, a warning display on the LCD device 33 or a warning sound output from the speaker device 14. In addition, if the driver continues to drive for more than a specified period, if the driver's eyes open and close at a specified frequency, or if the head tends to be lowered, the monitoring and control unit 37 can also determine that control is needed and output instructions to the driver to prompt them to rest.

[0168] In step ST30, the monitoring and control unit 37 determines whether the determination of the multiple monitoring types includes control requiring an emergency. If so, the process proceeds to step ST31. Otherwise, the monitoring and control unit 37 terminates the process. Figure 10 Control.

[0169] In step ST31, the monitoring and control unit 37 performs emergency control corresponding to the type of monitoring content that requires emergency handling.

[0170] Even if a warning is issued and the driver does not look directly at the vehicle, i.e., if the monitoring and control unit 37 continuously determines that the driver's line of sight is not forward, it can switch the driving mode of the vehicle 1 to automatic driving and decelerate and stop the vehicle 1. When the vehicle 1 is decelerated and stopped, the monitoring and control unit 37 can also illuminate a hazard warning light (not shown) and transmit emergency information to the external communication device 18. For example, if it is determined that the driver is drowsy or has a high pulse, the monitoring and control unit 37 can also decelerate and stop the vehicle 1. When the vehicle 1 is decelerated and stopped, the monitoring and control unit 37 can also illuminate a hazard warning light (not shown) and transmit emergency information to the external communication device 18.

[0171] Then, the monitoring and control unit 37 ended. Figure 10 Control.

[0172] Thus, as a variety of processes for monitoring, the monitoring control unit 37 performs: narrow-angle monocular processing based on the image captured by the first imaging sensor 41, wide-angle monocular processing based on the image captured by the second imaging sensor 48 which captures the vehicle compartment 3 from a different angle or shooting range than the first imaging sensor 41, and compound-eye processing based on the image captured by the first imaging sensor 41 and the image captured by the second imaging sensor 48.

[0173] The monitoring and control unit 37 performs different types of processing when the first imaging sensor 41 and the second imaging sensor 48 capture images simultaneously and when only one of the first imaging sensor 41 and the second imaging sensor 48 captures images.

[0174] As part of the narrow-angle monocular processing, which involves various monitoring processes based solely on images captured by the first imaging sensor 41, the monitoring control unit 37 determines the driver's strabismus, the driver's drowsiness, and the driver's emergency. In the narrow-angle monocular processing, the monitoring control unit 37 can perform all of these processes, or it can perform only a portion of them.

[0175] As part of the compound eye processing, which involves various monitoring processes based on images captured by the first imaging sensor 41 (capturing the driver's upper body or head) and images captured by the second imaging sensor 48 (capturing the entire interior of the vehicle cabin 3), the monitoring control unit 37 performs correction processing to suppress the difference in viewing angle or shooting range between the first and second imaging sensors 41 and 48 during simultaneous shooting. This process acquires parallax information between the first and second imaging sensors 41 and determines the driver's seating position. As part of the monitoring processing based on the correlation between the images captured by the first and second imaging sensors 41 and 48, the monitoring control unit 37 determines the occupant's seating position based on the difference between the occupant's shooting position in the image captured by the first imaging sensor 41 and the shooting position in the image captured by the second imaging sensor 48. In the compound eye processing, the monitoring control unit 37 may perform other than these processes, or it may perform only some of these processes.

[0176] As part of the wide-angle monocular processing, which involves various monitoring processes based solely on images captured by the second imaging sensor 48, the monitoring control unit 37 determines the seating position or posture of each occupant, determines any emergency situation involving each occupant, determines the seatbelt wearing status of each occupant, determines the child seat installation status, and outputs the images captured by the second imaging sensor 48. In the wide-angle monocular processing, the monitoring control unit 37 can perform all other processing or only some of the processing.

[0177] As described above, in this embodiment, the imaging sensor for photographing the occupants of the vehicle 1's cabin 3 includes multiple imaging sensors, such as a first imaging sensor 41 and a second imaging sensor 48. Furthermore, the monitoring and control unit 37 can perform monitoring processing on the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48.

[0178] In particular, in this embodiment, the first imaging sensor 41 and the second imaging sensor 48, which capture images of the occupants of the vehicle 1's cabin 3, are arranged side-by-side along the width direction of the vehicle 1 in the central portion of the vehicle's width direction, and at least a portion of their imaging ranges overlap. Therefore, the same occupant can be captured in the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48, enabling a certain degree of correlation between the images captured by these multiple imaging sensors.

[0179] As a result, the monitoring and control unit 37 can not only perform monitoring processing based solely on the images captured by the first imaging sensor 41 and solely on the images captured by the second imaging sensor 48, but also perform monitoring processing based on the correlation between the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48. As a monitoring process based on the correlation between the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48, the monitoring and control unit 37 can also determine the occupant's seating position based, for example, the difference between the occupant's position in the image captured by the first imaging sensor 41 and the position in the image captured by the second imaging sensor 48.

[0180] Thus, in this embodiment, since the first imaging sensor 41 and the second imaging sensor 48 are arranged side by side along the width direction in the central part of the vehicle 1 and at least part of their imaging ranges overlap, as a new monitoring function for the state of occupants such as the driver of the vehicle 1, the control unit can perform monitoring processing based on the correlation between the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48.

[0181] Furthermore, in this embodiment, processing can be performed on a greater number of surveillance types than the number of imaging sensors. In this embodiment, the increase in the number of imaging sensors can also be suppressed, and the monitoring processing for occupants, including the driver, can be multifunctional.

[0182] The above embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or alterations can be made without departing from the spirit of the invention.

[0183] For example, in the embodiment described above, the imaging sensors are a first imaging sensor 41 and a second imaging sensor 48. There can also be three or more imaging sensors. The same applies to the LEDs, which are the light-emitting components. The number of LEDs can also be different from that of the imaging sensors.

[0184] Symbol Explanation

[0185] 1…car (vehicle), 2…body, 3…cabin, 4…seat, 5…instrument panel, 6…center console, 7…steering wheel, 10…control device, 11…door opening / closing sensor, 12…vehicle speed sensor, 13…parking sensor, 14…speaker equipment, 15…occupant monitoring device, 16…position setting device, 17…driver assistance device, 18…external communication device, 19…occupant protection device, 20…air conditioning device, 21…in-vehicle network, 31…first camera module, 32…second camera Module, 33… LCD device, 34… Display operation panel, 35… Input / output device, 36… Memory, 37… Monitoring and control unit, 41… First imaging sensor, 42… Narrow-angle lens, 43… First LED, 44… Light-projecting narrow-angle lens, 45… First device controller, 46… Second LED, 47… Light-projecting wide-angle lens, 48… Second imaging sensor, 49… Wide-angle lens, 50… Second device controller, 61… First occupant data, 62… Second occupant data.

Claims

1. An in-vehicle monitoring device for a vehicle, comprising: The first and second camera components are used to photograph the occupants of the vehicle's cabin. as well as The control unit is capable of performing monitoring processing on the images captured by the first imaging component and the images captured by the second imaging component. The first and second shooting components are positioned side-by-side along the central portion of the vehicle's width direction, and at least a portion of their shooting ranges overlap. The first imaging component captures the driver's upper body or head at a narrow angle at a first time interval, repeating in a first cycle; the second imaging component captures the interior of the vehicle cabin at a wide angle at a second time interval, repeating in a second cycle. The first imaging component captures images at a first cycle that is shorter than the second cycle of the second imaging component, and captures images synchronously with the second imaging component at a ratio of multiple captures per capture. The in-vehicle monitoring device further includes: a first light-projecting component that projects light onto the shooting range of the first shooting component when the first shooting component is shooting; and a second light-projecting component that projects light onto the shooting range of the second shooting component when the second shooting component is shooting. When the first light-projecting component projects light synchronously with the second light-projecting component, it reduces the amount of light output. The second light-emitting component outputs less light during the second timing compared to the asynchronous light output of the first light-emitting component.

2. The in-vehicle monitoring device according to claim 1, wherein, The second imaging component captures a wide-angle view of the interior of the vehicle compartment, enabling it to photograph the driver and other occupants. The first camera component is positioned side-by-side with the driver, further away from the driver than the second camera component in the vehicle's width direction.

3. The in-vehicle monitoring device according to claim 1 or 2, wherein, When the first shooting component captures the upper body or head of the driver at a narrow angle, the first projection component that projects light into the shooting range of the first shooting component is positioned further away from the driver than the first shooting component in the vehicle width direction.

4. The in-vehicle monitoring device according to claim 1 or 2, wherein, When the second shooting component takes a wide-angle shot of the interior of the vehicle, the second light-projecting component that projects light into the shooting range of the second shooting component is positioned further away from the center of the vehicle in the width direction than the second shooting component.

5. The in-vehicle monitoring device according to claim 1 or 2, wherein, The control unit performs: monitoring processing based solely on the image captured by the first imaging component, monitoring processing based solely on the image captured by the second imaging component, and monitoring processing based on the correlation between the image captured by the first imaging component and the image captured by the second imaging component.

6. The in-vehicle monitoring device according to claim 1 or 2, wherein, As a monitoring process based on the correlation between the images captured by the first imaging component and the images captured by the second imaging component, The control unit determines the occupant's seating position based on the difference between the occupant's position in the image captured by the first camera and the occupant's position in the image captured by the second camera.

Images