Vehicle interior monitoring device

Abstract

To realize a new monitoring function for the condition of passengers including a driver in a vehicle.SOLUTION: An in-vehicle monitoring device 15 of a vehicle 1 includes a first imaging member 41 and a second imaging member 48 for imaging a passenger in a cabin 3 of the vehicle 1, and a control unit 37 capable of executing monitoring processing on the captured image of the first imaging member 41 and the captured image of the second imaging member 48. The first imaging member 41 and the second imaging member 48 are arranged side by side in the vehicle width direction in the central portion of the vehicle 1 in the vehicle width direction, and are provided such that at least parts of the imaging ranges overlap.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to an in-vehicle monitoring device for a vehicle. [Background technology]

[0002] A vehicle is provided with an occupant monitoring device to monitor the driver who is in the vehicle. Patent Document 1 discloses a vehicle occupant monitoring device that focuses on the driver's eyes to capture and monitor the driver's eyes. Patent Document 2 discloses a vehicle occupant monitoring device that irradiates a strong light onto the head of an occupant, the image of which is captured during a collision. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-088647 A [Patent Document 2] JP 2020-050078 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a demand for further monitoring functions in vehicle occupant monitoring. In particular, in autonomous driving and advanced driving assistance, there is a demand for more accurate monitoring of whether the condition of occupants such as the driver is being maintained in a state suitable for driving or traveling.

[0005] Thus, there is a demand for vehicles to realize new monitoring functions for the status of occupants, including the driver. [Means for solving the problem]

[0006] The present invention One embodiment of the present invention The vehicle interior monitoring device according to the present invention is Narrow the inside First imaging member to be imaged and the first imaging member captures an image of the interior of the vehicle at a wider angle than the first imaging member.A second imaging member, an image captured by the first imaging member, and an image captured by the second imaging member. Based on a control unit capable of executing a monitoring process, the first imaging member and the second imaging member being: In front of the vehicle and beyond the steering of the vehicle The vehicle width direction of the vehicle In the center Leave The second imaging member and the first imaging member are arranged in this order from the driver's side where the steering wheel is in front, along the vehicle width direction so that at least a part of the imaging ranges overlap. Set up side by side do.

[0010] Preferably, the imaging device has a first light-projecting member that projects light into the imaging range of the first imaging member when the first imaging member captures an image, and a second light-projecting member that projects light into the imaging range of the second imaging member when the second imaging member captures an image.

[0011] Preferably, before The first light emitting member is arranged so as to be spaced apart from the first imaging member in a width direction of the vehicle. The driver in front of the steering wheel It is advisable to set it up far away from the

[0013] Preferably, the first imaging member captures images at a cycle shorter than that of the second imaging member, and captures images in synchronization with the second imaging member once every several captures; the first light-projecting member periodically projects light when the first imaging member captures images periodically, and when the second light-projecting member projects light in accordance with the periodic imaging of the second imaging member, the first light-projecting member projects light with a reduced output light amount; The second light emitting member emits a smaller amount of light than the first light emitting member. The second imaging member periodically projects light when it periodically captures images. , it is good.

[0014] Preferably, the control unit executes a monitoring process based only on the image captured by the first imaging member, a monitoring process based only on the image captured by the second imaging member, and a monitoring process based on the correlation between the image captured by the first imaging member and the image captured by the second imaging member.

[0015] Preferably, the control unit determines the seating position of the occupant based on the difference between the imaging position of the occupant in the image captured by the first imaging member and the imaging position of the occupant in the image captured by the second imaging member as a monitoring process based on the correlation between the image captured by the first imaging member and the image captured by the second imaging member. Preferably, the first imaging member images the driver, who is the occupant behind the steering wheel, at a narrow angle in the passenger compartment of the vehicle, and the second imaging member images the passenger compartment at a wide angle so as to image occupants other than the driver as well as the driver, the first imaging member and the second imaging member are provided in the central portion of the dashboard and center console of the vehicle in the vehicle width direction, the second imaging member is provided on the driver's side based on the center of the vehicle in the vehicle width direction, and the first imaging member is provided farther from the driver than the second imaging member so as to be on the opposite side to the driver based on the center of the vehicle in the vehicle width direction. Effect of the Invention

[0016] The vehicle interior monitoring device of the present invention has a plurality of imaging members, i.e., a first imaging member and a second imaging member, as imaging members for imaging occupants in a vehicle compartment of the vehicle, and the control unit is capable of executing a monitoring process for the images captured by the first imaging member and the images captured by the second imaging member. In particular, in the present invention, the first and second imaging members for capturing images of occupants in the vehicle cabin are arranged side by side in the vehicle width direction at the center of the vehicle in the vehicle width direction so that at least a part of their imaging ranges overlap, making it possible to capture images of the same occupant in both the image captured by the first imaging member and the image captured by the second imaging member, and to provide a certain correlation between the images captured by these multiple imaging members. As a result, the control unit can execute not only a monitoring process based only on the captured image of the first imaging member or a monitoring process based only on the captured image of the second imaging member, but also a monitoring process based on the correlation between the captured image of the first imaging member and the captured image of the second imaging member. As a monitoring process based on the correlation between the captured image of the first imaging member and the captured image of the second imaging member, for example, the control unit may determine the seating position of the occupant based on the difference between the imaging position of the occupant in the image captured by the first imaging member and the imaging position of the occupant in the image captured by the second imaging member. In this way, in the present invention, the control unit is able to execute monitoring processing based on the correlation between the image captured by the first imaging member and the image captured by the second imaging member, as a new monitoring function for the status of vehicle occupants, such as the driver, due to the first imaging member and the second imaging member being aligned along the vehicle width direction in the central part of the vehicle's width direction so that at least a portion of their imaging ranges overlap. [Brief description of the drawings]

[0017] [Figure 1] FIG. 1 is an explanatory diagram of an automobile to which the present invention is applied. [Diagram 2] FIG. 2 is an explanatory diagram of the control device of the automobile of FIG. [Diagram 3] FIG. 3 is an explanatory diagram of an example of the types of monitoring performed by the automobile occupant monitoring device of FIG. [Figure 4] FIG. 4 is an explanatory diagram of the occupant monitoring device of FIG. [Diagram 5] FIG. 5 is an explanatory diagram of the arrangement of a liquid crystal device having the display operation panel of FIG. [Figure 6] FIG. 6 is an explanatory diagram of the arrangement of the first camera module and the second camera module in FIG. [Figure 7] FIG. 7 is a basic timing chart of image capture and light emission for the first camera module and the second camera module of FIG. [Figure 8] FIG. 8 is a timing chart of image capture and light emission for the first camera module and the second camera module of FIG. 4 in this embodiment. [Figure 9] FIG. 9 is a flowchart of the main control by the monitoring control unit of FIG. [Figure 10] FIG. 10 is a flowchart of the monitoring control of a plurality of occupants by the monitoring control unit of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0019] FIG. 1 is an explanatory diagram of an automobile 1 to which the present invention is applied. The automobile 1 is an example of a vehicle. The automobile 1 may be one that uses an internal combustion engine as a power source for running, one that uses stored electric power in a battery as a power source for running, or one that combines these. Furthermore, the automobile 1 may be one that runs according to the driving operations of a driver, one that runs automatically or with driving assistance, or one that can be switched between these two. In addition to the automobile 1, such vehicles may include personal mobility vehicles, powered two-wheeled vehicles, powered three-wheeled vehicles, buses, trucks, aircraft, and railroad cars.

[0020] 1 has a vehicle body 2, which can accommodate multiple passengers including a driver. The vehicle compartment 3 is provided with multiple front row seats 4 for the multiple passengers, including the driver, to sit individually, and a long rear row seat 4 for the multiple passengers to sit side by side. While driving, the driver and other passengers are basically required to sit correctly in the seats 4 and wear safety equipment such as seat belts. A dashboard 5 extending along the vehicle width direction of the vehicle body 2 is provided in the front part of the vehicle compartment 3, which is in front of the plurality of front row seats 4.

[0021] FIG. 2 is an explanatory diagram of the control device 10 of the automobile 1 of FIG. The control device 10 in Figure 2 has a door opening / closing sensor 11, a vehicle speed sensor 12, a park sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driving assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 to which these are connected.

[0022] The in-vehicle network 21 may be a wired communication network for the automobile 1, for example, compliant with CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 may be a communication network such as a LAN, or a combination of these. A part of the in-vehicle network 21 may include a wireless communication network.

[0023] The door opening / closing sensor 11 detects whether the door of the automobile 1 is opened or closed. The vehicle speed sensor 12 detects the speed of the traveling automobile 1. The vehicle speed sensor 12 may detect a stopped state. The park sensor 13 detects a parking state in which the automobile 1 is kept stopped. The park sensor 13 may detect, for example, that a shift lever (not shown) is operated to a parking position or that a parking lever (not shown) is operated to a brake position as the parking state.

[0024] For example, when the door opening / closing sensor 11 detects the opening / closing of a door, the occupant monitoring device 15 executes a recognition process of an occupant in the vehicle compartment 3, and then monitors the state of the occupant. When multiple occupants are recognized, the occupant monitoring device 15 may monitor the multiple occupants individually. The occupant monitoring device 15 may output information about each recognized occupant and information based on the monitoring to each part of the control device 10 via the vehicle network. When a driver or other occupant is in a predetermined state, the occupant monitoring device 15 may execute control such as outputting a warning to alert the occupant or execute control for preventing or avoiding danger. In this case, the occupant monitoring device 15 may output control information, etc. to each part of the control device 10 through the vehicle network.

[0025] The speaker device 14 outputs voice, warning sounds, etc. The speaker device 14 may output warnings from an occupant monitoring device 15 to the driver and other occupants. The position setting device 16 adjusts, for example, the front-rear and up-down positions of the seat 4 and the angle of the backrest, the front-rear and up-down positions and angles of the handlebars, and the front-rear and up-down positions and angles of various pedals. The position setting device 16 changes the seat position based on the occupant information output by the occupant monitoring device 15. The driving assistance device 17 performs control to assist the driver in driving the automobile 1 through manual driving, and controls the driving of the automobile 1 through automatic driving. The driving assistance device 17 controls the acceleration, deceleration, stopping, and steering of the automobile 1. The driving assistance device 17 may perform driving assistance according to the driver based on the driver information output by the occupant monitoring device 15. The exterior communication device 18 establishes a wireless communication path with, for example, a base station of a public wireless communication network, a base station of a commercial wireless communication network, or a base station for advanced traffic information, and performs data communication using the established wireless communication path. The exterior communication device 18 may perform two-way data communication with, for example, a server device that supports autonomous driving. The exterior communication device 18 may transmit information on occupants including the driver output by the occupant monitoring device 15 to the server device, for example, as emergency support information. The occupant protection device 19 executes control to protect the occupant when it detects or predicts a collision of the automobile 1. The occupant protection device 19 protects the occupant seated in the seat 4 by, for example, deploying an airbag or applying tension to a seat belt (not shown). The occupant protection device 19 may execute occupant protection for the occupant according to the occupant information output by the occupant monitoring device 15. The air conditioner 20 controls the temperature and oxygen concentration in the vehicle interior 3. The air conditioner 20 supplies, for example, cooled air or superheated air to the vehicle interior 3, thereby adjusting the vehicle interior 3 to a set temperature. The air conditioner 20 may perform air conditioning in accordance with information about the occupants output by the occupant monitoring device 15. The passengers can ride comfortably under settings suited to them, for example, by the control based on the passenger information output by the passenger monitoring device 15. The driver can concentrate on driving the automobile 1, for example.

[0026] FIG. 3 is an explanatory diagram of an example of the types of monitoring performed by the occupant monitoring device 15 of the automobile 1 of FIG. FIG. 3 shows examples of types of monitoring, such as looking away from the vehicle, falling asleep, seating posture, emergency situations, whether the seat belt is fastened, and whether the child seat is fastened. Although it would be meaningful for the occupant monitoring device 15 to execute just one of these monitoring controls, it is preferable for the occupant monitoring device 15 to execute multiple or all of these monitoring controls as an in-vehicle monitoring device. Among these types of monitoring, attention to distraction and drowsiness are considered to be essential for drivers. It is desirable to monitor the seating posture, emergency situations, and seat belt fastening status of not only the driver but also other passengers. Some automobiles 1 already have the functionality to detect and set the installation status of a child seat, so it may be acceptable to give monitoring a lower priority compared to these.

[0027] By carrying out these multiple types of monitoring, the monitoring capability of the occupant monitoring device 15 of the automobile 1 is improved. In autonomous driving and advanced driving assistance, it is desirable to maintain the state of occupants such as the driver in a state suitable for driving, and it is desirable to be able to execute each of the multiple types of monitoring more accurately. However, when attempting to implement the above-mentioned multiple types of monitoring in the automobile 1, the occupant monitoring device 15 may require an image sensor for each monitoring type and for each occupant. Meanwhile, the space available for installing image sensors inside the passenger compartment 3 is limited. If it becomes difficult to add image sensors due to the constraints of the installation space in the automobile 1, this may result in restrictions on the multi-functionality of the occupant monitoring device 15 of the automobile 1, and the occupant monitoring device 15 may not be able to be multi-functional beyond a certain level. In this way, in the automobile 1, an increase in the number of image sensors used may be a limitation in multiplying the types of monitoring for multiple occupants including the driver. Also, as the number of captured images of the image sensors increases, the processing load also increases. In this embodiment, an example is disclosed that preferably avoids the limitations of the occupant monitoring device 15 of the automobile 1 and enhances the feasibility of the above-mentioned multiple types of monitoring. Specifically, the occupant monitoring device 15 of this embodiment is intended to realize multiple types of control, such as the driver's distraction, the driver's drowsiness, the seating position or seating posture of each occupant, an emergency involving each occupant, the fastening status of each occupant's seat belt, the fastening status of a child seat, and image output, based on images captured by two image sensors.

[0028] FIG. 4 is an explanatory diagram of the passenger monitoring device 15 of FIG. 3 as an in-vehicle monitoring device of the automobile 1 according to the embodiment of the present invention. The passenger monitoring device 15 in FIG. 4 monitors not only the driver but also a plurality of passengers seated in a plurality of seats 4 other than the driver. The occupant monitoring device 15 of FIG. 4 comprises a first camera module 31, a second camera module 32, a liquid crystal device 33 having a display operation panel 34, an input / output device 35, a memory 36, and a monitoring control unit 37 to which these are connected.

[0029] The input / output device 35 is connected to the in-vehicle network 21. The input / output device 35 inputs and outputs data between the input / output device 35 and other units provided in the automobile 1 through the in-vehicle network 21.

[0030] The liquid crystal device 33 displays, on the display operation panel 34, a screen to be visually recognized by each passenger in the automobile 1. The display screen includes, for example, an operation screen to be operated by the passenger. The display operation panel 34 is a transparent or semi-transparent panel that is provided over the display surface of the liquid crystal device 33. The display operation panel 34 detects an operation by an occupant on the display surface of the liquid crystal device 33, and outputs information such as the operation position to the monitoring control unit 37.

[0031] The first camera module 31 includes a first LED 43, a first imaging sensor 41, and a first device controller 45 to which these are connected.

[0032] The first imaging sensor 41 may be a semiconductor optical sensor such as a CCD or a CMOS sensor. The first imaging sensor 41 outputs imaging data including a captured image to the monitoring control unit 37. A narrow-angle lens 42 is provided overlapping the first imaging sensor 41. The narrow-angle lens 42 may be made up of a plurality of optical lenses in order to suppress distortion in the peripheral portion of the image.

[0033] The first LED 43 may be a semiconductor light emitting element. A light projecting lens 44 is provided overlapping the first LED 43. The first LED 43 may project infrared light, for example. In this case, the first imaging sensor 41 outputs imaging data including an image captured by infrared light to the monitoring control unit 37.

[0034] The first device controller 45 controls the operation of the first imaging sensor 41 and the operation of the first LED 43. The first device controller 45 causes the first LED 43 to emit light during a period when the first imaging sensor 41 captures an image. The control cycle of the first device controller 45 can be set by the monitoring control unit 37.

[0035] The second camera module 32 has a second LED 46, a second imaging sensor 48, and a second device controller 50 to which these are connected.

[0036] The second imaging sensor 48 may be a semiconductor optical sensor such as a CCD or a CMOS sensor. The second imaging sensor 48 outputs imaging data including the captured image to the monitoring control unit 37. A wide-angle lens 49 is provided over the second imaging sensor 48. The wide-angle lens 49 may be made up of multiple optical lenses in order to suppress distortion in the peripheral portions of the image.

[0037] The second LED 46 may be a semiconductor light emitting element. A light projecting wide angle lens 47 is provided so as to overlap the second LED 46. The second LED 46 may project infrared light, for example. In this case, the second imaging sensor 48 outputs imaging data including an image captured by infrared light to the monitoring control unit 37.

[0038] The second device controller 50 controls the operation of the second imaging sensor 48 and the operation of the second LED 46. The second device controller 50 causes the second LED 46 to emit light during a period in which the second imaging sensor 48 captures an image. A control cycle of the second device controller 50 can be set by the monitoring control unit 37.

[0039] 4, the first LED 43 of the first camera module 31 projects light onto the driver's upper body or head at a narrow angle, and the first imaging sensor 41 captures the driver's upper body or head at a narrow angle. This allows the first imaging sensor 41 to capture a bright, high-resolution image of the driver's upper body or head. The image captured by the first imaging sensor 41 can capture the driver's eyes, eyeballs, etc. at a high resolution. In contrast, the second LED 46 of the second camera module 32 projects light generally at a wide angle onto the interior of the vehicle compartment 3, and the second imaging sensor 48 captures an image of the interior of the vehicle compartment 3 generally at a wide angle. This allows the second imaging sensor 48 to capture a bright image including the driver as well as multiple passengers other than the driver. In the image captured by the second imaging sensor 48, the driver as well as a passenger in the front passenger seat and passengers in the rear seat may be captured. The first imaging sensor 41 and the second imaging sensor 48, which capture images of passengers in the passenger compartment 3 of the automobile 1, are arranged side by side along the vehicle width direction in the central part of the automobile 1 in the vehicle width direction, and capture images of the passenger compartment 3 at different angles of view or imaging ranges.

[0040] The first imaging sensor 41 and the second imaging sensor 48 are provided so that their imaging ranges partially overlap each other to commonly capture an image of the driver.

[0041] The memory 36 stores programs and data. The memory 36 may be composed of a non-volatile memory and a volatile memory. The non-volatile memory may be, for example, a HDD, an SSD, an EEPROM, etc. The volatile memory may be, for example, a RAM.

[0042] Data on multiple occupants registered in the automobile 1 may be managed and recorded for each occupant in the memory 36 of the occupant monitoring device 15. Fig. 4 shows first occupant data 61 for a first occupant and second occupant data 62 for a second occupant. A database is formed in the memory 36 from the multiple occupant data.

[0043] Each occupant data such as the first occupant data 61 and the second occupant data 62 may include identification information unique to each occupant, for example, registered image data obtained by capturing an image of the occupant's head or eyes using an image sensor, and various setting data set by the occupant. The setting data may include, for example, information such as the seat position of the occupant, initial setting of whether or not to use driving assistance, driving preference information in autonomous driving, information on the server device to be used, occupant protection settings, and air conditioning settings. In this way, the memory 36 can record the imaging data captured by the first imaging sensor 41 or the second imaging sensor 48 from the front of each occupant while a specified screen is displayed on the liquid crystal device 33 as registered imaging data for each occupant. Additionally, the memory 36 may store occupant data for generic occupants that are not registered.

[0044] The monitoring control unit 37 may be, for example, an ECU, a CPU, or another microcomputer. The monitoring control unit 37 reads and executes a program from the memory 36. In this way, the monitoring control unit 37 serving as a control unit of the occupant monitoring device 15 is realized. The monitoring control unit 37 is capable of executing processing on the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48 . The monitoring control unit 37 may perform multiple types of processing as described above regarding occupants in the vehicle cabin 3 based on images captured by the first imaging sensor 41 and the second imaging sensor 48 with different angles of view or imaging ranges. For example, the monitoring control unit 37 may execute control for identifying or registering a passenger in the automobile 1. For example, the monitoring control unit 37 may execute control according to the operation content determined as an operation by the passenger on the operation screen of the liquid crystal device 33. Furthermore, when an occupant gets into the automobile 1, the monitoring control unit 37 identifies the occupant and performs monitoring control of the occupant's condition. In particular, for occupants who are drivers, the monitoring control unit 37 may use each occupant's registered imaging data registered in the memory 36 as reference data for making judgments, and judge whether each occupant is looking away, falling asleep, or in an emergency, based on the state of their eyes.

[0045] FIG. 5 is an explanatory diagram of the arrangement of the liquid crystal device 33 having the display operation panel 34 of FIG. FIG. 5 shows a dashboard 5 and a center console 6 provided at the front of the vehicle interior 3. 4 is provided in the center of the vehicle 1 in the vehicle width direction, facing toward the rear of the vehicle 1. The liquid crystal device 33 having the display operation panel 34 is embedded vertically from the dashboard 5 to the center console 6.

[0046] FIG. 6 is an explanatory diagram of the arrangement of the first camera module 31 and the second camera module 32 in FIG. As shown in FIG. 6, the first camera module 31 and the second camera module 32 in FIG. 4 are disposed on the back side of the display operation panel 34 of the liquid crystal device 33 so as to be disposed above the liquid crystal device 33. The first LED 43, the first imaging sensor 41, the second imaging sensor 48, and the second LED 46 for the first camera module 31 and the second camera module 32 are arranged side by side in the vehicle width direction of the automobile 1.

[0047] The first imaging sensor 41 is provided on the opposite side of the driver with respect to the center position Y0 of the automobile 1 in the vehicle width direction. The second imaging sensor 48 is provided on the driver's side with respect to the center position Y0 of the automobile 1 in the vehicle width direction. As a result, the first imaging sensor 41, which captures an image of the driver's upper body or head at a narrow angle, is arranged farther from the driver in the vehicle width direction of the automobile 1 than the second imaging sensor 48, which captures an image of the interior of the passenger compartment 3 at a wide angle as a whole. The steering wheel 7 shown in FIG. 5 is less likely to be interposed between the first imaging sensor 41 and the driver. The steering wheel 7 is less likely to be reflected in the image captured by the first imaging sensor 41. In this way, the first imaging sensor 41 and the second imaging sensor 48 are arranged side by side in the vehicle width direction of the automobile 1, so that a parallax occurs for the driver in Fig. 6. Even if the first imaging sensor 41 and the second imaging sensor 48 capture images of the same angle of view or the same imaging range, the imaging position of the driver in the image captured by the first imaging sensor 41 and the imaging position of the driver in the image captured by the second imaging sensor 48 will be shifted in accordance with the parallax. This parallax depends on the sensor distance between the first imaging sensor 41 and the second imaging sensor 48.

[0048] The first LED 43 is provided farther away from the driver than the first imaging sensor 41 in the vehicle width direction of the automobile 1. The light output from the first LED 43 toward the driver is less likely to be blocked by the steering wheel 7. The second LED 46 is provided on the driver's side of the second imaging sensor 48 in the vehicle width direction of the automobile 1. The second LED 46 is provided farther from the central position Y0 in the vehicle width direction of the automobile 1 than the second imaging sensor 48 in the vehicle width direction of the automobile 1. This allows the second imaging sensor 48 to be disposed closer to the central position Y0 in the central portion of the vehicle 1 in the vehicle width direction. By being disposed closer to the central position Y0, the second imaging sensor 48 can capture an image of the entire interior of the passenger compartment 3 at a wide angle from the central portion of the vehicle 1 in the vehicle width direction. The second imaging sensor 48 can capture images of not only the passengers seated in the multiple seats 4 in the front row, but also the multiple passengers seated in the seats 4 in the rear row.

[0049] FIG. 7 is a basic timing chart of imaging and light emission for the first camera module 31 and the second camera module 32 in FIG. In FIG. 7, time flows from left to right.

[0050] 7A shows the imaging state of the first imaging sensor 41. The first imaging sensor 41 captures images at regular intervals. Based on the captured images of the first imaging sensor 41, narrow-angle monocular processing for monitoring the driver is possible. During a high level period, the first imaging sensor 41 captures images. The same applies to the others. 7B shows a light-projecting state of the first LED 43. The first LED 43 projects light toward the imaging range of the first imaging sensor 41 at a narrow angle during the period when the first imaging sensor 41 captures an image.

[0051] FIG. 7(C) shows the imaging state of the second imaging sensor 48. The second imaging sensor 48 performs imaging at regular intervals. Based on the image captured by the second imaging sensor 48, wide-angle monocular processing for monitoring the driver and other occupants is possible. In the wide-angle monocular processing, not only the processing for monitoring the state of each of the multiple occupants included in the captured image, but also the processing for transmitting the captured image for viewing on a mobile terminal or the like may be performed. All occupants including the driver may be captured in the image to be viewed. In this case, as shown in FIG. 7(C), the captured image of the second imaging sensor 48 may be used alternately for the monitoring processing and the transmission processing. Even in this case, since the state of the driver is monitored by capturing an image by the first imaging sensor 41 at short intervals, it can be assumed that the movement of the automobile 1 will not be hindered. Moreover, the imaging period of the second imaging sensor 48 is twice that of the first imaging sensor 41. Moreover, the imaging timing of the second imaging sensor 48 is synchronized with the imaging timing of the first imaging sensor 41. When the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48 are synchronized, it is possible to calculate the parallax based on the sensor interval between the first imaging sensor 41 and the second imaging sensor 48 with high accuracy based on the deviation of the imaging position of the driver in those captured images. Moreover, compound eye processing is possible using the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48 that are synchronized. In this way, the first imaging sensor 41, which images the upper body or head of the driver at a narrow angle, images the interior of the vehicle compartment 3 at a shorter period than the second imaging sensor 48, which images the interior of the vehicle compartment 3 as a whole at a wide angle, and images the interior of the vehicle compartment 3 in synchronization with the second imaging sensor 48 once every several times. When the first imaging sensor 41 and the second imaging sensor 48 image the interior of the vehicle compartment 3 in synchronization with each other, the monitoring control unit 37 can execute monitoring processing based on the images captured by the first imaging sensor 41 and the images captured by the second imaging sensor 48. When the first imaging sensor 41 and the second imaging sensor 48 image the interior of the vehicle compartment 3 in synchronization with each other, the monitoring control unit 37 can execute processing different from that executed when the first imaging sensor 41 image the interior of the vehicle compartment 3 at a narrower period than that executed when the second imaging sensor 48 image the interior of the vehicle compartment 3 at a narrower period than ... 7D shows a light-projecting state of the second LED 46. The second LED 46 projects light at a wide angle toward the imaging range of the second imaging sensor 48 during the period in which the second imaging sensor 48 captures an image.

[0052] Fig. 7(E) shows the amount of light emitted by the driver, which emits the light of the first LED 43 in Fig. 7(B) and the light of the second LED 46 in Fig. 7(D). When the first LED 43 and the second LED 46 output light in the same manner as in Fig. 7, the amount of light emitted by the driver increases and decreases at each imaging timing. The brightness of the driver in the captured image changes according to the amount of light emitted.

[0053] FIG. 8 is a timing chart of imaging and light emission for the first camera module 31 and the second camera module 32 of FIG. 4 in this embodiment. 8(A) to 8(E) basically correspond to FIGS. 7(A) to 7(E).

[0054] 8(D), the amount of light emitted by the second LED 46 is suppressed to be lower than the amount of light emitted by the first LED 43 shown in FIG. In addition, when the second LED 46 synchronously emits light, the amount of light emitted by the first LED 43 is suppressed to be lower than in normal times when the second LED 46 does not synchronously emit light. When the second LED 46 synchronously emits light, the amount of light output by the first LED 43 is reduced by an amount substantially corresponding to the amount of light output by the second LED 46.

[0055] As a result, as shown in Fig. 8(E), the amount of light projected by the driver when the first LED 43 and the second LED 46 project light synchronously becomes substantially the same as the amount of light projected by the driver when only the first LED 43 projects light. The brightness of the driver in the captured image can be constant. The process of brightness correction according to the light projection situation for the captured image is basically unnecessary. In this manner, in this embodiment, since the first imaging sensor 41 and the second imaging sensor 48 capture images synchronously, even if the first LED 43 and the second LED 46 may project light synchronously, the amount of light projected onto the driver, for example, is unlikely to change between the synchronized projection of light and other independent projection of light. The monitoring control unit 37 that monitors the occupants can continuously obtain images of stable brightness as images captured by the first imaging sensor 41.

[0056] FIG. 9 is a flowchart of the main control by the monitoring control unit 37 of FIG. When a new passenger gets into the automobile 1, the monitoring and control unit 37 repeatedly executes the process of FIG.

[0057] In step ST1, the monitoring control unit 37 judges whether or not a new occupant gets into the automobile 1. The occupant opens a door (not shown) of the automobile 1 and sits in the seat 4. The monitoring control unit 37 may detect and judge that a new occupant gets into the automobile 1, for example, based on an opening / closing detection signal of the door opening / closing sensor 11. In addition, for example, the monitoring control unit 37 may judge whether or not a new occupant has gotten into the automobile 1, based on the fact that the new occupant is captured in the imaging data of the imaging sensor. If a new occupant has not gotten into the automobile 1, the monitoring control unit 37 repeats this process. When a new occupant gets into the automobile 1, the monitoring control unit 37 advances the process to step ST2.

[0058] In step ST2, the monitoring control unit 37 determines the riding position of the new occupant. The monitoring control unit 37 may determine the riding position of the new occupant based on the imaging data of the second imaging sensor 48 that captures an image of the entire interior of the vehicle. When the monitoring control unit 37 detects that the occupant is stable in a seated state in the seat 4, it may determine that the occupant is riding in that position.

[0059] In step ST3, the monitoring control unit 37 performs a personal identification process for the new occupant. The monitoring control unit 37 may perform the personal identification process for the new occupant based on the latest imaging data of the second imaging sensor 48 that captures an image of the entire interior of the vehicle. For example, the monitoring control unit 37 may compare the image of the occupant included in the image of the imaging data with the occupant data of the multiple occupants recorded in the memory 36 to identify each occupant with high accuracy. In this case, the monitoring control unit 37 may perform matching based only on the image components of the frontal view in the registered imaging data of each occupant registered in the memory 36. Also, the monitoring control unit 37 may compare feature points extracted from each image, instead of directly comparing the images. If the registered imaging data included in the occupant data registered in the memory 36 matches with a certain degree of accuracy or higher, the monitoring control unit 37 may identify the occupant as the occupant of the registered imaging data. In this case, the monitoring control unit 37 identifies the occupant as the occupant identified by matching with the multiple occupant data registered in the memory 36. If there is no registered imaging data that matches with a certain degree of accuracy or higher in the multiple occupant data registered in the memory 36, the monitoring control unit 37 may identify the occupant as an unregistered occupant. Furthermore, if there is an occupant whose occupant data is recorded in the memory 36, the monitoring control unit 37 may further execute a setting process using the occupant data. In this case, the monitoring control unit 37 outputs information of the setting data to each unit of the automobile 1. This executes processes such as the seat position for the occupant, the initial setting of whether or not driving assistance is provided, driving preference information in automatic driving, information on the server device to be used, occupant protection settings, and air conditioning settings. Furthermore, the monitoring control unit 37 may determine, for example, whether a child bucket is installed on the passenger seat 4 based on the latest captured image data, and execute a setting to prohibit the deployment of an airbag toward the passenger seat 4 if a child bucket is installed.

[0060] In step ST4, the monitoring control unit 37 judges whether or not to newly register the newly boarded passenger. For example, the monitoring control unit 37 displays a confirmation screen for the registration process on the liquid crystal device 33, and when the passenger performs an operation corresponding to the confirmation screen on the display operation panel 34, it judges that the passenger should be newly registered and proceeds to step ST5. If an operation not requiring registration is performed, the monitoring control unit 37 proceeds to step ST6.

[0061] In step ST5, the monitor control unit 37 executes a process of newly registering the new occupant. The monitor control unit 37 adds and records the occupant data of the new occupant in the memory .

[0062] In step ST6, the monitoring control unit 37 starts monitoring and controlling the newly boarded occupant.

[0063] In step ST7, the monitoring control unit 37 judges whether or not to end the monitoring control of the occupants. For example, when the automobile 1 stops and the ignition is turned off, when the automobile 1 arrives at the destination and stops, or when the occupants get off, the monitoring control unit 37 may judge to end the monitoring control of the occupants. The monitoring control unit 37 may judge to end the monitoring control of the occupants based on, for example, the detection of opening and closing by the door opening and closing sensor 11 and an image from the imaging sensor. If the monitoring control of the occupants is not to be ended, the monitoring control unit 37 repeats this process. If the monitoring control of the occupants is to be ended, the monitoring control unit 37 advances the process to step ST8.

[0064] In step ST8, the monitoring control unit 37 executes a process of ending monitoring of the occupants. The monitoring control unit 37 acquires, for example, setting information when each occupant gets off the vehicle from each part of the automobile 1, and updates the occupant data of each occupant recorded in the memory 36. As a result, the occupant data registered in the memory 36 is adapted to the preferences of the occupant. The latest settings for the occupant will be automatically performed the next time the occupant gets in the vehicle. In addition, the monitoring control unit 37 may temporarily record occupant data for an unregistered occupant in the memory 36. As a result, if the occupant performs a registration operation thereafter, the settings can be immediately linked. Thereafter, the monitoring and control unit 37 ends the monitoring and control of FIG.

[0065] FIG. 10 is a flowchart of the monitoring control of a plurality of occupants by the monitoring control unit 37 of FIG. When the monitoring control unit 37 determines in step ST6 of FIG. 9 that monitoring control for the first occupant should be started, it repeatedly executes the monitoring control of FIG. 10 until it determines in step ST7 that the last occupant has disembarked and the process is terminated.

[0066] In step ST11, the monitor control unit 37 determines whether or not it is time for processing based on imaging. The first imaging sensor 41, which captures an image of the driver's upper body or head at a narrow angle, and the second imaging sensor 48, which captures an image of the interior of the vehicle compartment 3 at a wide angle, capture images at the respective imaging timings shown in Fig. 8 and output the captured image data to the monitoring control unit 37. If new captured image data has been acquired from the first imaging sensor 41 or the second imaging sensor 48, the monitoring control unit 37 determines that it is time for processing based on the captured image, and proceeds to the process at step ST12. If new captured image data has not been acquired, the monitoring control unit 37 waits by repeating this process.

[0067] In step ST12, the monitoring control unit 37 starts a narrow-angle monocular process based only on the image captured by the first imaging sensor 41 which basically captures an image of the upper body or head of the driver.

[0068] In step ST13, the monitoring control unit 37 judges whether the driver is looking away based on the image captured by the first imaging sensor 41. The monitoring control unit 37 extracts, for example, image components of the driver's eyes from the image captured by the first imaging sensor 41. The monitoring control unit 37 judges whether the line of sight estimated from the image components of the driver's eyes is directed in the traveling direction of the automobile 1. If the driver's line of sight is in the traveling direction of the automobile 1, the monitoring control unit 37 judges that the driver is not looking away. If the driver's line of sight is not in the traveling direction of the automobile 1 for multiple consecutive times, the monitoring control unit 37 judges that the driver is looking away.

[0069] In step ST14, the monitoring control unit 37 determines whether the driver is dozing off based on the image captured by the first imaging sensor 41. The monitoring control unit 37 extracts, for example, image components of the driver's eyes from the image captured by the first imaging sensor 41. If the open / closed state estimated from the image components of the driver's eyes is open, the monitoring control unit 37 determines that the driver is not dozing off. If the driver's eyes are closed multiple times in succession, the monitoring control unit 37 determines that the driver is dozing off.

[0070] In step ST15, the monitoring control unit 37 judges whether the driver has experienced an emergency event based on the captured image of the first imaging sensor 41. The monitoring control unit 37 extracts, for example, an image component of the driver's seating posture from the captured image of the first imaging sensor 41. If the driver's seating posture is not suitable for driving, for example, with the head lowered, the monitoring control unit 37 judges whether the driver has experienced an emergency event. In addition, for example, the monitoring control unit 37 may obtain information such as the driver's pulse rate and blood flow rate from the captured image of the first imaging sensor 41. In this case, the monitoring control unit 37 judges whether the driver has experienced an emergency event if the driver's pulse rate is higher than a threshold value and if the increase or decrease in blood flow rate is equal to or greater than a threshold value. This causes the monitoring control unit 37 to end the narrow-angle monocular processing.

[0071] In step ST16, the monitor control unit 37 judges whether or not the current processing timing is the timing for wide-angle synchronous imaging. 8, the second imaging sensor 48, which captures an overall wide-angle image of the interior of the vehicle cabin 3, may capture images simultaneously with the first imaging sensor 41 and output the captured image data to the monitoring control unit 37. At the timing of this synchronous imaging, the monitoring control unit 37 determines that the current processing timing is the timing for wide-angle simultaneous imaging, and proceeds to step ST17. If the current processing timing is for acquiring new image data only from the first imaging sensor 41, the monitoring control unit 37 proceeds to step ST28 for post-processing without performing any other determination processing.

[0072] In step ST17, the monitoring control unit 37 judges whether or not it is the timing for compound eye processing for synchronous imaging. When the second imaging sensor 48 captures images in synchronization with the first imaging sensor 41 as shown in FIG. 8, the monitoring control unit 37 alternately executes compound eye processing using both the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48, and wide-angle monocular processing that basically processes only the captured image of the second imaging sensor 48. In the case of the first synchronous imaging after starting the monitoring control of FIG. 10, if the wide-angle monocular processing was executed last time, the monitoring control unit 37 judges that it is the timing for compound eye processing and proceeds to step ST18. If the compound eye processing was executed last time, the monitoring control unit 37 judges that it is the timing for wide-angle monocular processing, not the timing for compound eye processing, and proceeds to step ST22. Information regarding the previous processing content may be recorded in the memory 36. The monitoring control unit 37 may update the information of the previous processing recorded in the memory 36 after the processing judgment in step ST17.

[0073] In step ST18, the monitoring control unit 37 starts compound eye processing using both the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48. The monitoring control unit 37 may identify the imaging position of the driver included in both the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48 in the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48.

[0074] In step ST19, the monitoring control unit 37 executes a correction process for the angle of view difference between the image captured by the first imaging sensor 41 and the image captured by the second imaging sensor 48. For example, the monitoring control unit 37 calculates the angle of view difference according to the sensor interval between the first imaging sensor 41 and the second imaging sensor 48 based on the imaging position of the driver included in the image captured by the first imaging sensor 41 and the imaging position of the driver included in the image captured by the second imaging sensor 48. The monitoring control unit 37 may correct the imaging position information for each pixel used in the narrow-angle monocular processing for the image captured by the first imaging sensor 41 and the imaging position information for each pixel used in the wide-angle monocular processing for the image captured by the second imaging sensor 48 based on the angle of view difference acquired by the calculation. The monitoring control unit 37 may record the acquired information on the angle of view difference and the imaging position information for each pixel in the memory 36.

[0075] In step ST20, the monitoring control unit 37 acquires the parallax between the first imaging sensor 41 and the second imaging sensor 48. The monitoring control unit 37 may calculate the parallax between the first imaging sensor 41 and the second imaging sensor 48 based on the installation position of the first imaging sensor 41, the installation position of the second imaging sensor 48, the sensor interval, the difference in angle of view, and the like.

[0076] In step ST21, the monitoring control unit 37 determines the seating position of the driver. Based on the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48, the monitoring control unit 37 calculates the direction and distance of the driver's body parts, such as the head, neck, and waist, and identifies the positions of those body parts on the seat 4. The monitoring control unit 37 may determine whether the identified positions of the driver's body parts are the positions in the correct seat position. This causes the monitor control unit 37 to end the compound eye process. After that, the monitor control unit 37 advances the process to step ST28 for post-processing.

[0077] In step ST22, the monitoring control unit 37 starts wide-angle monocular processing, which basically processes only the image captured by the second imaging sensor .

[0078] In step ST23, the monitoring control unit 37 determines the seating position (seating posture) of each of the multiple occupants including the driver based on the captured image of the second imaging sensor 48. The monitoring control unit 37 extracts, for example, image components of each occupant from the captured image of the second imaging sensor 48. The monitoring control unit 37 determines whether or not the image components of each occupant are suitable as a seating position (seating posture) while the automobile 1 is moving. For example, if the seating position (seating posture) of the occupant is lying down, the monitoring control unit 37 determines that the seating position (seating posture) of the occupant is not suitable for the automobile 1 while it is moving.

[0079] In step ST24, the monitoring control unit 37 judges an emergency event for each of the multiple occupants including the driver based on the captured image of the second imaging sensor 48. The monitoring control unit 37 extracts, for example, an image component of each occupant from the captured image of the second imaging sensor 48. The monitoring control unit 37 judges whether or not the image component of each occupant corresponds to an emergency event. For example, if the seating position (seating posture) of the occupant is lying down, the monitoring control unit 37 judges that the seating position (seating posture) of the occupant corresponds to an emergency event.

[0080] In step ST25, the monitoring control unit 37 determines the wearing state of the seat belt for each of the multiple occupants including the driver based on the captured image of the second imaging sensor 48. The monitoring control unit 37 extracts the image component of the seat belt for each of the multiple occupants including the driver. If the image component of the seat belt is not crossed in front of the upper body of each occupant, the monitoring control unit 37 determines that the seat belt is not worn correctly.

[0081] In step ST26, the monitoring control unit 37 determines whether the child seat is installed based on the image captured by the second imaging sensor 48. The monitoring control unit 37 attempts to extract the image components of the child seat. If the image components of the child seat can be extracted, the monitoring control unit 37 determines that the child seat is installed.

[0082] In step ST27, the monitoring control unit 37 generates a monitoring image for viewing on a mobile terminal or the like based on the image captured by the second imaging sensor 48, and outputs the generated monitoring image. This causes the monitor control unit 37 to end the wide-angle monocular processing. After that, the monitor control unit 37 advances the processing to step ST28 for post-processing.

[0083] In step ST28, the monitoring control unit 37 starts post-processing based on the above-mentioned multiple monitoring types. The monitoring control unit 37 first determines whether the above-mentioned multiple monitoring types include any that require a warning, and if so, proceeds to step ST29. Otherwise, the monitoring control unit 37 proceeds to step ST30.

[0084] In step ST29, the monitoring control unit 37 outputs a warning according to the contents of the monitoring type that requires the warning. The warning to the driver may be, for example, a warning display on the liquid crystal device 33 or a warning sound output from the speaker device 14. If the driver has been driving continuously for a predetermined period of time or longer, if the driver's eyes are opening and closing at a predetermined frequency, or if the driver's head is tilted downward, the monitoring control unit 37 may determine that control is necessary and execute an output to urge the driver to take a break, etc.

[0085] In step ST30, the monitoring control unit 37 judges whether the above-mentioned multiple monitoring types include any that require emergency control, and if so, proceeds to step ST31. Otherwise, the monitoring control unit 37 ends the control of FIG.

[0086] In step ST31, the monitoring control unit 37 executes emergency control according to the type of monitoring that requires emergency processing. When the driver does not look straight ahead even after outputting a warning, that is, when the monitoring control unit 37 determines that the driver's gaze is not continuously looking straight ahead, the monitoring control unit 37 may switch the driving mode of the automobile 1 to automatic driving and decelerate and stop the driving of the automobile 1. When decelerating and stopping the driving of the automobile 1, the monitoring control unit 37 may turn on hazard lights (not shown) and transmit emergency information by the external vehicle communication device 18. For example, when it is determined that the driver is dozing or has a high pulse rate, the monitoring control unit 37 may decelerate and stop the driving of the automobile 1. When decelerating and stopping the driving of the automobile 1, the monitoring control unit 37 may turn on hazard lights (not shown) and transmit emergency information by the external vehicle communication device 18. Thereafter, the monitor and control unit 37 ends the control shown in FIG.

[0087] In this way, the surveillance control unit 37 performs multiple types of surveillance processing, including narrow-angle monocular processing based on the image captured by the first imaging sensor 41, wide-angle monocular processing based on the image captured by the second imaging sensor 48, which captures the vehicle interior 3 at an angle of view or imaging range different from that of the first imaging sensor 41, and compound eye processing based on the image captured by the first imaging sensor 41 and the image captured by the second imaging sensor 48. The monitoring control unit 37 executes different types of processing depending on whether the first imaging sensor 41 and the second imaging sensor 48 capture images synchronously or only one of the first imaging sensor 41 and the second imaging sensor 48 captures images. The monitoring control unit 37 determines whether the driver is looking away, whether the driver is falling asleep, and whether the driver is in an emergency, as multiple types of monitoring processing based only on the image captured by the first imaging sensor 41 in the narrow-angle monocular processing. The monitoring control unit 37 may execute processing other than those described above, or may execute only a part of the processing, in the narrow-angle monocular processing. The monitoring control unit 37 performs a correction process to suppress a difference in the angle of view or the imaging range between the first imaging sensor 41 and the second imaging sensor 48 in the synchronized imaging as a plurality of types of monitoring processes based on the captured image of the first imaging sensor 41 capturing an image of the upper body or head of the driver in the compound eye process and the captured image of the second imaging sensor 48 capturing an image of the entire interior of the vehicle compartment 3, acquires parallax information between the first imaging sensor 41 and the second imaging sensor 48, and determines the seating position of the driver. The monitoring control unit 37 performs a monitoring process based on the correlation between the captured image of the first imaging sensor 41 and the captured image of the second imaging sensor 48, and determines the seating position of the occupant based on the difference between the captured image position of the occupant in the image captured by the first imaging sensor 41 and the captured image of the second imaging sensor 48. The monitoring control unit 37 may perform other processes or a part of the processes in the compound eye process. As multiple types of monitoring processing based only on the image captured by the second imaging sensor 48 in the wide-angle monocular processing, the monitoring control unit 37 determines the seating position or seating posture of each occupant, determines an emergency event for each occupant, determines the fastening state of the seat belt of each occupant, determines the fastening state of the child seat, and outputs the image captured by the second imaging sensor 48. The monitoring control unit 37 may execute processing other than those described above or may execute only a part of the processing in the wide-angle monocular processing.

[0088] As described above, in this embodiment, the vehicle 1 has a plurality of image capturing sensors, namely, the first image capturing sensor 41 and the second image capturing sensor 48, as image capturing sensors for capturing images of occupants in the vehicle interior 3. The control unit can execute a monitoring process for the images captured by the first image capturing sensor 41 and the second image capturing sensor 48. Particularly in this embodiment, the first imaging sensor 41 and the second imaging sensor 48 that capture images of occupants in the passenger compartment 3 of the automobile 1 are arranged side by side in the vehicle width direction at the center of the vehicle width direction of the automobile 1, and are provided so that at least a part of their imaging ranges overlap. This makes it possible to capture images of the same occupant in the images captured by the first imaging sensor 41 and the second imaging sensor 48, and a certain correlation can be established between the images captured by these multiple imaging sensors. As a result, the control unit can execute not only a monitoring process based only on the image captured by the first imaging sensor 41 or only on the image captured by the second imaging sensor 48, but also a monitoring process based on the correlation between the images captured by the first imaging sensor 41 and the second imaging sensor 48. As the monitoring process based on the correlation between the images captured by the first imaging sensor 41 and the second imaging sensor 48, the control unit may determine the seating position of the occupant based on the difference between the imaging position of the occupant in the image captured by the first imaging sensor 41 and the imaging position of the occupant in the image captured by the second imaging sensor 48, for example. In this manner, in this embodiment, the control unit is able to execute monitoring processing based on the correlation between the image captured by the first imaging sensor 41 and the image captured by the second imaging sensor 48, as a new monitoring function for the status of occupants, such as the driver of the automobile 1, due to the first imaging sensor 41 and the second imaging sensor 48 being aligned along the vehicle width direction in the central part of the vehicle width direction so that at least a portion of their imaging ranges overlap. In addition, in the present embodiment, it is possible to execute processes for a greater number of monitoring types than the number of image sensors, and it is also possible to multi-function the monitoring process for passengers including the driver while suppressing an increase in the number of image sensors.

[0089] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.

[0090] For example, in the embodiment described above, there are two image sensors, the first image sensor 41 and the second image sensor 48. There may be three or more image sensors. The same applies to the LEDs as light-emitting members. The number of LEDs provided may differ from the number of image sensors. [Explanation of symbols]

[0091] 1...Automobile (vehicle), 2...Vehicle body, 3...Vehicle interior, 4...Seat, 5...Dashboard, 6...Center console, 7...Steering, 10...Control device, 11...Door opening / closing sensor, 12...Vehicle speed sensor, 13...Park sensor, 14...Speaker device, 15...Occupant monitoring device, 16...Position setting device, 17...Driving assistance device, 18...External communication device, 19...Occupant protection device, 20...Air conditioning device, 21...In-vehicle network, 31...First camera module, 32...Second Camera module, 33... liquid crystal device, 34... display operation panel, 35... input / output device, 36... memory, 37... monitoring control unit, 41... first imaging sensor, 42... narrow-angle lens, 43... first LED, 44... narrow-angle light-projecting lens, 45... first device controller, 46... substantially second LED, 46... second LED, 47... wide-angle light-projecting lens, 48... second imaging sensor, 49... wide-angle lens, 50... second device controller, 61... first occupant data, 62... second occupant data

Claims

1. a first imaging member for imaging an interior of a vehicle at a narrow angle; a second imaging member that captures an image of the interior of the vehicle cabin at a wider angle than the first imaging member so that at least a part of an imaging range of the second imaging member overlaps with that of the first imaging member; a control unit capable of executing a monitoring process based on an image captured by the first imaging member and an image captured by the second imaging member; having The first imaging member and the second imaging member are arranged in the vehicle width direction in the forward direction of the vehicle and at the center of the vehicle width direction of the vehicle, in the order of the second imaging member and the first imaging member from the driver's side where the steering wheel is in front. An in-vehicle monitoring device.

2. a first light emitting member that emits light into an imaging range of the first imaging member when the first imaging member captures an image; and a second light emitting member that emits light into an imaging range of the second imaging member when the second imaging member captures an image.

2. The vehicle interior monitoring device according to claim 1.

3. The first light-projecting member is provided farther from a driver in front of the steering wheel than the first imaging member in a vehicle width direction of the vehicle.

3. The vehicle interior monitoring device according to claim 2.

4. The first imaging member captures images at a cycle shorter than the cycle at which the second imaging member captures images, and captures images synchronously with the second imaging member at a rate of once every several captures; the first light-projecting member periodically projects light when the first imaging member periodically captures an image, and when the second light-projecting member projects light in accordance with the periodic imaging of the second imaging member, the first light-projecting member projects a reduced amount of light to be output; The second light emitting member periodically emits light with a smaller amount of light than the first light emitting member when the second imaging member periodically captures images.

4. The vehicle interior monitoring device according to claim 2 or 3.

5. The control unit executes a monitoring process based only on an image captured by the first imaging member, a monitoring process based only on an image captured by the second imaging member, and a monitoring process based on a correlation between the image captured by the first imaging member and the image captured by the second imaging member. The vehicle interior monitoring device according to any one of claims 1 to 4.

6. The control unit performs monitoring processing based on a correlation between an image captured by the first imaging member and an image captured by the second imaging member, determining a seating position of the occupant based on a difference between an imaging position of the occupant in the image captured by the first imaging member and an imaging position of the occupant in the image captured by the second imaging member; 6. The vehicle interior monitoring device according to claim 5.

7. The first imaging member images a driver behind the steering wheel in the passenger compartment of the vehicle at a narrow angle, the second imaging member captures an image of the vehicle interior at a wide angle so as to capture an image of the driver as well as passengers other than the driver; the first imaging member and the second imaging member are provided at central portions of a dashboard and a center console of the vehicle in a vehicle width direction, the second imaging member is provided on the driver's side with respect to the center of the vehicle in a vehicle width direction, The first imaging member is provided farther from the driver than the second imaging member so as to be on the opposite side of the driver with respect to the center of the vehicle in the vehicle width direction. The vehicle interior monitoring device according to any one of claims 1 to 6.

Images