Occupant monitoring device for a vehicle

Abstract

The present application relates to a passenger monitoring device for a vehicle which can monitor the state of a plurality of passengers in the vehicle well. A passenger monitoring device (15) for a vehicle (1) has a photographing means (41) for photographing the interior of the vehicle (1) in order to monitor the state of a plurality of passengers in the vehicle (1), a display means (32) for displaying a screen for the passengers in the vehicle (1) to recognize, and a determination means (36) for determining the state of the passengers in the vehicle (1) using photographing data photographed by the photographing means (41) of the passengers when the screen is displayed in the display means (32). In a panel (33) which displays the screen of the display means (32), a non-display portion (33b) is provided inside a rectangular frame (33a) which circumscribes the screen display area. The photographing means (41) is disposed on the back side of the non-display portion (33b) of the panel (33) of the display means (32).

Description

Technical Field

[0001] This invention relates to a vehicle occupant monitoring device. Background Technology

[0002] Inventions concerning driver monitoring devices in vehicles have been proposed (Patent Documents 1 and 2).

[0003] In Patent Document 1, a camera specifically installed in front of the driver is used solely for the purpose of monitoring the driver.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-014359

[0007] Patent Document 2: Japanese Patent Application Publication No. 2019-014360 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, sometimes there are passengers in the vehicle besides the driver.

[0010] When passengers are also monitored in the same way as the driver, it is difficult to capture them using a camera specifically placed in front of the driver.

[0011] Furthermore, for example, as in Patent Document 2, it is considered to move the location of the driver-specific monitoring device of Patent Document 1 toward the center of the vehicle in the width direction.

[0012] However, if the camera is simply moved towards the center of the vehicle in the width direction, the driver cannot be properly filmed, and the monitoring function for the driver may be reduced.

[0013] In addition, occupant monitoring devices for each passenger, including the driver, have been considered, but this is unlikely to be a practical option when considering factors such as cost.

[0014] Therefore, further improvements are expected in order to better monitor the condition of multiple occupants in the vehicle.

[0015] Technical solutions for solving the problem

[0016] One embodiment of the present invention provides a vehicle occupant monitoring device, comprising: a camera unit that captures images of the vehicle interior for monitoring the status of multiple occupants in the vehicle; a display unit that displays an image for the occupants to identify; and a judgment unit that determines the status of the occupants in the vehicle using data captured by the camera unit of the occupants while the image is displayed on the display unit. The non-display portion is provided inside a rectangular frame circumscribed to the display area of ​​the image on the display unit panel, and the camera unit is disposed on the back side of the non-display portion of the display unit panel.

[0017] Invention Effects

[0018] This invention includes: a camera that captures images of the vehicle's interior to monitor the status of multiple occupants; a display unit that displays images for each occupant to identify; and a determination unit that uses data captured by the camera to measure the status of each occupant while the display unit shows a predetermined image. Therefore, in this embodiment, the status of each occupant can be determined using the data captured by the camera to measure the status of each occupant while the display unit shows a predetermined image, along with the current data captured by the camera. This suppresses the influence of factors such as the occupant's viewing position and body characteristics in the captured data, resulting in a more accurate determination of the occupant's status.

[0019] In contrast, if the state of each occupant in a vehicle is determined solely based on current shooting data captured by the shooting component, the state of each occupant may not be accurately determined due to factors such as their viewing position in the shooting data and their physical characteristics. In this invention, shooting data captured by the shooting component of each occupant while a predetermined image is displayed on the display component is used; therefore, these influences are suppressed, and the state of each occupant can be determined more accurately.

[0020] In particular, in this invention, the capturing component is positioned on the back side of the panel displaying the image captured by the capturing component, that is, further positioned on the back side of the non-display portion, which is configured to be the inner side of a rectangular frame circumscribed outside the image display area. Therefore, the capturing component can capture an image equivalent to an image captured from the back side of the image displayed on the display component. The capturing component can capture the heads of each occupant observing the image from its front. Moreover, by using the capture data of each occupant's head captured from the front, the state of each occupant contained in the current capture data captured by the capturing component can be determined more accurately.

[0021] As a result, in this invention, the display panel and the camera are positioned offset from the center of the vehicle width direction in the central part of the vehicle width direction. The camera has a wide-angle lens that can capture the head view of multiple occupants sitting in the front row of the vehicle when it is centrally positioned in the central part of the vehicle width direction. Even if the determination unit determines at least the opening and closing state of the driver's eyes or the line of sight as the state of each of the multiple occupants in the vehicle, it can more accurately determine the state of each of the multiple occupants, including the driver. Attached Figure Description

[0022] Figure 1 This is an explanatory diagram of a car using an occupant monitoring device according to an embodiment of the present invention.

[0023] Figure 2 yes Figure 1 A diagram illustrating the control system of a car.

[0024] Figure 3 yes Figure 2 An explanatory diagram of a car occupant monitoring device.

[0025] Figure 4A This is a frontal view of the head of the driver, who is the primary occupant of the car.

[0026] Figure 4B yes Figure 4A A three-dimensional view of the first driver's head.

[0027] Figure 4C This is a front view of the head of the second driver, who is also a passenger in the car.

[0028] Figure 4D yes Figure 4C A three-dimensional view of the second driver's head.

[0029] Figure 5A It is related to having Figure 3 An illustrative diagram showing the configuration of the LCD display control panel and the camera module with a shooting sensor in the vehicle cabin.

[0030] Figure 5B yes Figure 5A An explanatory diagram of the camera module 31, which has a shooting sensor 41 and a shooting wide-angle lens 42, showing the shooting range R1 of the vehicle interior.

[0031] Figure 6A This is an explanatory diagram of the display and operation panel of an LCD device located at an offset position in the central part of the vehicle body in the width direction.

[0032] Figure 6B Looking from the width of vehicle body 2 Figure 6A The side view of the display operation panel.

[0033] Figure 7 yes Figure 3 The flowchart shows the occupant registration control process performed by the monitoring and control department.

[0034] Figure 8 yes Figure 3 The flowchart shows the occupant monitoring and control process performed by the monitoring and control department.

[0035] Figure 9 It is about Figure 6A An explanatory diagram illustrating the configuration of a modified camera module with a shooting sensor relative to the display operation panel of an LCD device.

[0036] Figure 10 It is about Figure 9 An explanatory diagram illustrating the configuration of a camera module with a shooting sensor relative to the display operation panel of an LCD device, representing another variation of the example. Detailed Implementation

[0037] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0038] Figure 1 This is an explanatory diagram of a car 1 in which the occupant monitoring device 15 of an embodiment of the present invention is applied.

[0039] Car 1 is an example of a vehicle. Car 1 can use an internal combustion engine, battery power, or a combination thereof as its power source.

[0040] exist Figure 1 A passenger compartment 3 is formed on the body 2 of a car 1. Multiple front seats 4 for the driver and front passenger, and a long rear seat 4 are provided in the passenger compartment 3. Occupants, including the driver, sit in the seats 4. A dashboard 5 is positioned in front of the front seats 4, at the front of the passenger compartment 3, extending along the width of the body 2.

[0041] Figure 2 yes Figure 1 Explanation diagram of the control device 10 of car 1.

[0042] Figure 2 The control device 10 includes: a door switch sensor 11, a vehicle speed sensor 12, a parking sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driver assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 connecting them.

[0043] The in-vehicle network 21 can also be a wired communication network for the vehicle 1, such as one based on CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 can be a communication network such as a LAN, or a combination of both. A wireless communication network can also be included as a part of the in-vehicle network 21.

[0044] The door switch sensor 11 detects the opening and closing of the doors of the car 1.

[0045] Vehicle speed sensor 12 detects the speed of the moving car 1. Vehicle speed sensor 12 can also detect a stationary state.

[0046] The parking sensor 13 detects a parking state that keeps the car 1 stationary. The parking sensor 13 can also detect parking states by, for example, the gear lever (not shown) being moved to the parking position and the parking lever (not shown) being moved to the braking position.

[0047] When the occupant monitoring device 15 detects, for example, the door switch sensor 11, opening or closing of the door, it performs identification processing of the occupant sitting in the cabin 3.

[0048] When multiple occupants are identified, the occupant monitoring device 15 can also monitor each individual occupant individually.

[0049] The occupant monitoring device 15 can also output the information of each identified occupant and the monitoring-based information to various parts of the control device 10 through the in-vehicle network 21.

[0050] In particular, the occupant monitoring device 15 identifies, for example, the occupant seated in the driver's seat 4 as the driver, and monitors the identified driver for squinting or drowsiness. Furthermore, when the driver is in a prescribed state, the occupant monitoring device 15 performs controls to attract the driver's attention and controls to prevent hazards. At this time, the occupant monitoring device 15 can also output driver information, information for attracting attention, and information for preventing hazards to various parts of the control device 10 via the in-vehicle network 21.

[0051] The speaker device 14 outputs sounds, warning sounds, etc. The speaker device 14 can also output warnings to the driver and other occupants issued by the occupant monitoring device 15.

[0052] The position setting device 16 adjusts, for example, the fore-aft and vertical position and backrest angle of the seat 4, the fore-aft and vertical position and angle of the steering wheel, and the fore-aft and vertical position and angle of various pedals. The position setting device 16 changes the seat position based on the occupant information output by the occupant monitoring device 15.

[0053] The driving assistance device 17 assists in driving operations of the vehicle 1 performed manually by the driver, or assists in driving operations of the vehicle 1 performed automatically. The driving assistance device 17 controls the acceleration, deceleration, stopping, and steering of the vehicle 1. Based on the driver information output by the occupant monitoring device 15, the driving assistance device 17 performs driving assistance corresponding to the driver.

[0054] The external communication device 18 establishes a wireless communication path with, for example, a base station of a public wireless communication network, a base station of a commercial wireless communication network, or a base station for high-traffic information, and performs data communication using the established wireless communication path. The external communication device 18 can also perform bidirectional data communication, for example, with a server device for assisted autonomous driving. The external communication device 18 can also send occupant information, including the driver, output by the occupant monitoring device 15 as, for example, emergency assistance information, to the server device.

[0055] Occupant protection device 19 performs controls to protect occupants in the event of a detected or predicted collision with vehicle 1. Occupant protection device 19 protects the occupant seated in seat 4 by deploying, for example, an airbag (not shown), or by applying tension to the seatbelt. Occupant protection device 19 can also perform occupant protection based on occupant information output from occupant monitoring device 15.

[0056] The air conditioning unit 20 controls the temperature and oxygen concentration of the vehicle compartment 3. The air conditioning unit 20 supplies, for example, cooled or superheated air to the vehicle compartment 3, thereby adjusting the vehicle compartment 3 to a set temperature. The air conditioning unit 20 can also perform air conditioning corresponding to occupant information output from the occupant monitoring device 15.

[0057] By controlling the vehicle based on occupant information output from these occupant monitoring devices 15, occupants can ride comfortably, for example, according to their own settings. The driver can concentrate on, for example, driving the vehicle 1.

[0058] Figure 3 yes Figure 2 Explanation diagram of the occupant monitoring device 15 of car 1. Figure 3 The occupant monitoring device 15 monitors not only the driver, but also multiple occupants, such as those seated in the front seats 4. Figure 3 The occupant monitoring device 15 includes: a camera module 31, an LCD device 32 with a display operation panel 33, an input / output device 34, a memory 35, and a monitoring and control unit 36 ​​connected thereto.

[0059] For example, Figure 1As shown, the camera module 31 is located in the center of the dashboard 5 in the vehicle width direction. The camera module 31 is used to capture images of multiple occupants seated in the front seats 4. The camera module 31 includes: a capture sensor 41, a wide-angle lens 42, a first LED 43, a first projection lens 44, a second LED 45, and a second projection lens 46.

[0060] The image sensor 41 is a semiconductor optical sensor such as a CCD or CMOS sensor. The image sensor 41 may also have a generally quadrilateral light-receiving surface, for example, with multiple light-receiving elements arranged thereon. The image sensor 41 may also output image data containing the captured image to the monitoring and control unit 36.

[0061] A wide-angle lens 42 is superimposed on the image sensor 41. The wide-angle lens 42 can also be, for example, a lens that can capture the upper body or head of multiple occupants seated in the front seats 4, when the image sensor 41 is located in the center of the dashboard 5 in the vehicle width direction. The wide-angle lens 42 can also be composed of multiple optical lenses to suppress image distortion at the periphery.

[0062] In order to monitor the status of multiple occupants in the car 1, the imaging sensor 41 and the wide-angle lens 42 are used as imaging components to capture images of the car 1's interior.

[0063] The first LED 43 and the second LED 45 can also be semiconductor light-emitting elements. The first LED 43 and the second LED 45 serve as light-projecting components, projecting light onto at least the driver sitting in the car 1. The first LED 43 and the second LED 45 can also project light, for example, infrared light. In this case, the imaging sensor 41 outputs imaging data, including the infrared image, to the monitoring and control unit 36. A first projection lens 44 is disposed superimposed on the first LED 43. The first projection lens 44 primarily projects the light from the first LED 43 onto the driver sitting in the driver's seat 4. A second projection lens 46 is disposed superimposed on the second LED 45. The second projection lens 46 primarily projects the light from the second LED 45 onto the occupant sitting in the front passenger seat 4. Furthermore, the second projection lens 46 can also diffuse light and project it onto both the occupant sitting in the front passenger seat 4 and the driver sitting in the driver's seat 4.

[0064] Furthermore, in the camera module 31, the shooting sensor 41 and the wide-angle lens 42 are positioned in the central part along the vehicle width direction. The first LED 43 and the first projection lens 44 are positioned at the end part on the passenger side. The second LED 45 and the second projection lens 46 are positioned at the end part on the driver side. Thus, even if there are objects such as the steering wheel 7 between the driver and the instrument panel 5, the camera module 31 can project light and take pictures without being obstructed by the steering wheel 7 or the like.

[0065] The liquid crystal device 32 typically has a roughly quadrilateral display surface. The liquid crystal device 32 displays images on the display surface of the display operation panel 33.

[0066] The LCD device 32 serves as a display component, displaying images for each passenger in the car 1 to identify.

[0067] The display operation panel 33 is a transparent or semi-transparent operation detection panel disposed superimposed on the display surface of the liquid crystal device 32. The component formed by superimposing the display operation panel 33 and the display surface of the liquid crystal device 32 is a display component. The display operation panel 33 detects the occupant's operation on the display surface of the liquid crystal device 32. The display operation panel 33 can also output the occupant's operation position on the display surface of the liquid crystal device 32 to the monitoring and control unit 36.

[0068] Input / output device 34 is connected to in-vehicle network 21. Input / output device 34 inputs and outputs data with other parts of the vehicle 1 via in-vehicle network 21.

[0069] Memory 35 stores programs and data. Memory 35 can also be composed of non-volatile memory and volatile memory. Non-volatile memory may include, for example, HDD, SSD, EEPROM, etc. Volatile memory may include, for example, RAM.

[0070] Data of multiple occupants registered in vehicle 1 can be managed and recorded in the memory 35 of occupant monitoring device 15 for each occupant. Figure 3 The data represents first crew data 51 concerning the first crew member and second crew data 52 concerning the second crew member. A database is constructed using multiple crew data sets.

[0071] The occupant data, such as the first occupant data 51 and the second occupant data 52, may also include identification information inherent to each occupant, such as registration data of the occupant's head and eyes captured by the camera sensor 41, and various setting data of the occupant. The setting data may also include information such as the occupant's seat position, initial settings for the presence or absence of driver assistance, driving preference information in autonomous driving, information on the server device used, occupant protection settings, and air conditioning settings.

[0072] In this way, the memory 35, as a recording component, can record the shooting data captured by each occupant when the shooting sensor 41 is facing the display component from the front and the specified screen is displayed, as the registered shooting data of each occupant.

[0073] In addition, the memory 35 can also record crew data related to unregistered general crew members.

[0074] The monitoring and control unit 36 ​​may be, for example, an ECU, a CPU, or other microcomputer. The monitoring and control unit 36 ​​reads a program from the memory 35 and executes it. Thus, the monitoring and control unit 36, which serves as the control unit of the occupant monitoring device 15, is implemented.

[0075] The monitoring and control unit 36 ​​performs, for example, registration processing and monitoring processing of occupants riding in the vehicle 1. In the monitoring processing, the monitoring and control unit 36 ​​can also identify occupants riding in the vehicle 1 and perform monitoring processing for each occupant.

[0076] In particular, regarding the occupants, at least the driver, the monitoring and control unit 36, as a judgment unit, uses the respective registered shooting data registered in the memory 35 as the reference data for judgment, and judges the state of each occupant, such as strabismus, drowsiness, emergency situation, etc.

[0077] but, Figure 3 The camera module 31's image sensor 41 is positioned in the central part of the vehicle body 2 in the width direction, and it not only captures the driver, but also the upper body or head of multiple occupants seated in the front seats 4. In the car 1, in addition to the driver, there are sometimes other passengers.

[0078] When monitoring passengers in the same way as the driver, it is difficult to photograph passengers using the camera sensor 41 that is specially set in front of the driver.

[0079] Additionally, in cases where a single camera sensor 41 is needed to capture images of multiple occupants, including the driver, such as... Figure 3 Similar to the camera module 31, a wide-angle lens 42 is required in the image sensor 41 to expand the field of view. However, when the wide-angle lens 42 is aligned with the image sensor 41, the occupant captured in the peripheral portion of the image (excluding the center) will be distorted. When the driver is distorted in the image, the driver's eyes will also be distorted, potentially reducing the accuracy of judgments based on eye opening / closing or line of sight, such as drowsiness or strabismus. The monitoring function for the driver itself may be compromised.

[0080] Figure 4A to Figure 4D This is an explanatory diagram showing the driver's position as an occupant of car 1.

[0081] Figure 4A This is a front view of the first driver's head. Figure 4B yes Figure 4A A three-dimensional view of the first driver's head.

[0082] The camera sensor 41, located in the center of the vehicle 1 in the width direction, does not detect objects when the driver is facing forward of the vehicle 1. Figure 4AThe camera shot directly at the driver's head, rather than... Figure 4B That's how you shoot the driver's head from an angle.

[0083] Figure 4C This is a front view of the second driver's head. Figure 4D yes Figure 4C A three-dimensional view of the second driver's head.

[0084] The camera sensor 41, located in the center of the vehicle 1 in the width direction, does not detect objects when the driver is facing forward of the vehicle 1. Figure 4C The camera shot directly at the driver's head, rather than... Figure 4D That's how you shoot the driver's head from an angle.

[0085] and, Figure 4D In the shooting data, with Figure 4B The shooting data is different; the driver's right eye is obscured by the driver's high nose, and only the pupil is visible.

[0086] In this case, the monitoring and control unit 36 ​​only based on Figure 4D The captured data makes it difficult to determine the opening and closing status of the driver's right eye. Especially, such as... Figure 4D Therefore, in the case where an image is captured not in a distortion-free state, but as a distorted image, the monitoring and control unit 36 ​​only... Figure 4D The captured data makes it extremely difficult to determine the open or closed state of the driver's right eye.

[0087] Furthermore, since the image does not include the white of the driver's right eye, the monitoring and control unit 36 ​​may only be based on... Figure 4D The captured data cannot be extracted from the right eye, nor can it be extracted from the right eye. Figure 4D The driver's head. For example, in cases where the image capture area of ​​the head is determined based on the configuration relationship between the eyes and nose, or the configuration relationship between the eyes and mouth, the monitoring and control unit 36 ​​only bases on... Figure 4D The shooting data cannot be determined. Figure 4D The driver's head. Without data on the driver's head as a reference, the monitoring and control unit 36 ​​cannot correctly infer that the driver's right eye is not included in the image components.

[0088] When the camera module 31's image sensor 41 is positioned in the center of the vehicle 1 along its width, this can be easily generated. Figure 4D That kind of state.

[0089] Thus, there are individual differences among passengers, such as people with large eyes, small eyes, protruding eyes, and deep-set eyes.

[0090] The actual position of the driver's head may shift not only in the direction of the view along the width of the vehicle, but also in the vertical direction. When these shifts are taken into account, it is not easy to reliably determine the driver's head based solely on the current image data from the image sensor 41.

[0091] Figure 5A This is about the implementation method. Figure 3 An explanatory diagram showing the configuration of the LCD device 32 of the display operation panel 33 and the camera module 31 with the shooting sensor 41 in the vehicle cabin.

[0092] exist Figure 5A The middle section indicates that there is an instrument panel 5 and a center console 6 located at the front of the vehicle compartment 3.

[0093] The LCD device 32 with a display operation panel 33 is arranged longitudinally from the instrument panel 5 to the center console 6 in the central part of the vehicle body 2 in the width direction.

[0094] In detail, the liquid crystal device 32 with the display operation panel 33 is positioned at an offset position slightly off-center from the center position Y0 in the width direction of the vehicle body 2 towards the side opposite to the driver, i.e., the passenger side, thus moving away from the driver. Furthermore, the liquid crystal device 32 with the display operation panel 33, positioned at this offset position, is tilted at an angle relative to the width direction in the center of the vehicle body 2. The liquid crystal device 32 with the display operation panel 33 is tilted towards the driver in the offset position relative to the longitudinal direction of the vehicle body 2. Moreover, although the liquid crystal device 32 with the display operation panel 33 is tilted towards the driver, because it is positioned slightly off-center from the driver, away from the driver, the passenger can obtain the same operability as in the case of a centrally located configuration.

[0095] Figure 6A to Figure 6B This is an explanatory diagram of the display operation panel 33 of the LCD device 32, which is located at an offset position in the central part of the vehicle body 2 in the width direction. Figure 6A This is the main view from the three sides of the vehicle compartment. Figure 6B This is a side view taken from the width direction of vehicle body 2. Figure 6A In the figure, the central position Yd of the display operation panel 33 of the LCD device 32 is offset from the central position Y0 in the width direction of the vehicle body 2 to the left, i.e., the side of the passenger.

[0096] The liquid crystal device 32 with a display operation panel 33 and the camera module 31 are positioned offset from the center Y0 position in the width direction of the vehicle body 2. The liquid crystal device 32 with the display operation panel 33 uses resin retaining members 61 to hold the peripheral portion of the display operation panel 33 at its four corners.

[0097] The display area of ​​the liquid crystal device 32, which displays the image, is not a quadrilateral display surface like that of a typical monitor, but rather has a roughly concave shape with a central upper edge lacking a quadrilateral. A non-display portion 33b, forming a notch, is formed inside the rectangular frame 33a that circumscribes the roughly concave display area of ​​the liquid crystal device 32. The display areas of the liquid crystal device 32 are located on the left and right sides of the non-display portion 33b. The non-display portion 33b is formed symmetrically along a line with reference to the central position Yd of the display operation panel 33 of the liquid crystal device 32.

[0098] Thus, the non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32 as a notch in the center of the upper edge of the display area, inside the rectangular frame 33a that is circumscribed to the display area. The non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32 as a notch along the upper edge of the display area. Furthermore, the non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32, extending at least from the center position Yd in the vehicle width direction of the display area to the center position Y0 in the vehicle width direction of the vehicle 1. The non-display portion 33b is formed in the display operation panel 33 in a symmetrical shape in the vehicle width direction of the vehicle 1, based on the center position Yd in the vehicle width direction of the display area.

[0099] The camera module 31, equipped with a shooting sensor 41, is located on the back side of the display operation panel 33, on the back side of the non-display portion 33b formed above the display area of ​​the liquid crystal device 32. Specifically, the camera module 31 is configured such that the shooting sensor 41 and the wide-angle lens 42 are positioned between the central position Yd of the display operation panel 33 of the liquid crystal device 32 and the central position Y0 of the vehicle body 2 in the vehicle width direction. Thus, the shooting sensor 41 and the wide-angle lens 42 are configured such that the central positions of these components in the vehicle width direction are located between the central position Yd of the display operation panel 33 of the liquid crystal device 32 and the central position Y0 of the vehicle body 2 in the vehicle width direction. Furthermore, the shooting sensor 41 and the wide-angle lens 42 are positioned at an offset position slightly deviated from the central position Y0 of the vehicle body 2 in the vehicle width direction towards the side opposite to the driver, and are tilted relative to the longitudinal direction of the vehicle body 2, so as to face the driver, just like the display operation panel 33.

[0100] The first LED 43 and the second LED 45 of the camera module 31 are arranged together with the shooting sensor 41 on the back side of the non-display unit 33b.

[0101] Furthermore, with the camera module 31 positioned on the back of the display operation panel 33, the wide-angle lens 42, the first projection lens 44, and the second projection lens 46 can also be configured by processing the display operation panel 33 itself.

[0102] Figure 5B This is an explanatory diagram of the shooting range R1 of the vehicle interior generated by a camera module 31 equipped with a shooting sensor 41 and a wide-angle lens 42, which is obliquely positioned at an offset position in the central part of the vehicle width direction of the vehicle 1.

[0103] The wide-angle lens 42 coincides with the image sensor 41. Therefore, the image sensor 41... Figure 5B As shown in the shooting range R1, the entire vehicle interior 3 can be photographed. The shooting sensor 41 is positioned in the central part of the vehicle 1 in the width direction and can photograph the head of the driver and the heads of the passengers sitting in the multiple front seats 4.

[0104] However, when the wide-angle lens 42 is combined with the imaging sensor 41 to capture multiple occupants, the central portion of the image within the shooting range R1 becomes a high-quality area Rp1 with minimal distortion. On the other hand, the peripheral portion of the shooting range R1 becomes a distortion-generating area Rp2 compared to the central portion. Therefore, when an occupant is seated on the right or left side in the vehicle width direction, it becomes difficult to capture the occupant's head without distortion. When judging based on factors such as the opening or closing of the occupant's eyes, the possibility of inaccurate judgment increases.

[0105] In this embodiment, the display operation panel 33 is tilted relative to the front-rear direction of the vehicle body 2, facing the driver. The image sensor 41 and wide-angle lens 42, located on the back side of the display operation panel 33, are positioned offset from the driver's side, with the center position Y0 in the width direction of the vehicle body 2 as a reference. Therefore, the high-quality area Rp1 in the center of the image captured by the image sensor 41 is closer to the driver's side. The driver is easily captured in the high-quality area Rp1 of the image captured by the image sensor 41, and is difficult to capture in the distortion-prone area Rp2 in the periphery. It is expected that the driver will be captured as an image with distortion suppressed.

[0106] Thus, the camera module 31 is positioned offset from the center Y0 in the vehicle width direction on the back side of the non-display portion 33b of the LCD device 32, which is related to the display control panel 33. The LCD device 32 with the display control panel 33 can also be positioned in a tilted posture with its back facing down, following the inclination of the interior surface from the dashboard 5 to the center console 6. As a result, the camera module 31, like the LCD device 32 with the display control panel 33, is tilted relative to the front-rear direction of the vehicle body 2, facing the driver, and is also positioned in a tilted posture with its back facing down.

[0107] Furthermore, the camera module 31 is positioned inside the retaining member 61, so that it does not protrude outwards from the retaining members 61 at the four corners of the display operation panel 33. In addition, as shown by the dashed circles in FIG6, the camera module 31 is configured such that the capturing sensor 41, the first LED 43, and the second LED 45 are located inside the outer circle of the image display area.

[0108] By configuring the image captured by the camera module 31's sensor 41 in this restricted manner, the center position of the image in the vehicle width direction is located between the center position Yd of the image display area of ​​the liquid crystal device 32 with the display operation panel 33 and the center position Y0 of the vehicle body 2 in the vehicle width direction. Furthermore, the camera module 31's sensor 41 is positioned inside the holding member 61, as if it were positioned within the liquid crystal device 32 itself. As a result, the camera module 31's sensor 41 captures images of the occupants from the back side of the liquid crystal device 32 with the display operation panel 33. The camera module 31's sensor 41 is offset from the back side of the display operation panel 33, moving from the center position Yd of the image display area in the vehicle width direction towards the center position Y0 of the vehicle body 1, thus occupying a range from the center position Yd of the image display area in the vehicle width direction to the center position Y0 of the vehicle body 1.

[0109] Figure 7 yes Figure 3 The flowchart of the occupant registration control performed by the monitoring and control unit 36.

[0110] When a new passenger boards the vehicle 1, the monitoring and control unit 36 ​​can also repeat the process. Figure 7 Registration control.

[0111] In step ST1, the monitoring and control unit 36 ​​first determines whether an occupant is registered.

[0112] If, for example, an occupant selects and operates the occupant registration button on a menu screen displayed on the LCD device 32, the monitoring and control unit 36 ​​determines that the occupant is registered and proceeds to step ST2. Otherwise, the monitoring and control unit 36 ​​determines that the occupant is not registered and terminates the process. Figure 7 Registration control.

[0113] Starting from step ST2, the monitoring and control unit 36 ​​executes a new occupant registration process. The monitoring and control unit 36 ​​first displays a guide to begin occupant registration on the LCD device 32. Occupants who want to register by pressing the occupant registration button in the menu screen are assumed to be facing the guide display on the LCD device 32 directly. Alternatively, the monitoring and control unit 36 ​​can also provide a direct facing instruction in the guide to direct the face towards the LCD device 32. Furthermore, the monitoring and control unit 36 ​​illuminates the first LED 43 and the second LED 45 of the camera module 31. This illuminates infrared light onto the upper body or head of the occupant who is facing the guide display on the LCD device 32.

[0114] In step ST3, the monitoring and control unit 36 ​​acquires the current capture data from the capture sensor 41 and acquires the image components contained in the capture data, which are captured from the front of the occupant of the newly registered object, as a frontal image. If multiple occupants can be extracted from the images captured by the capture sensor 41, the monitoring and control unit 36 ​​can select one as an occupant. It is expected that the frontal image will contain image components formed by infrared light concerning the occupant's eyes, nose, and mouth with high accuracy. The image components formed by infrared light may also include vein patterns of the head or eyeballs. The vein patterns of the head or eyeballs depend on the number of feature points extracted, but can be used to identify an individual. Furthermore, the vein patterns of the head or eyeballs are less affected by the brightness and darkness patterns caused by the surface shape of the head or the concavity and convexity of the face. The vein pattern of the eyeballs generally extends from the sclera (white of the eye) towards the iris (black of the eye). The vein pattern of the eyelids covering the eyeballs is a different pattern from the vein pattern of the eyeballs. The monitoring and control unit 36 ​​can also extract and acquire information on these vein patterns of the head, eyeballs, eyelids, etc., from the capture data. Moreover, it is expected that in the image components that capture the occupant's head facing forward, the high-quality image components include the various parts of the head.

[0115] In addition, the monitoring and control unit 36 ​​can repeatedly perform guidance or repeatedly acquire the current shooting data from the shooting sensor 41 until it can confirm that the image in the facing image contains image components formed by infrared light about the occupant's eyes, nose, and mouth.

[0116] Starting from step ST4, the monitoring and control unit 36 ​​displays a guide on the LCD device 32 showing the occupant's forward-looking (front-view) perspective of the vehicle 1. It is assumed that the occupant is observing the front of the vehicle 1 based on the guide displayed on the LCD device 32. Additionally, the monitoring and control unit 36 ​​illuminates the first LED 43 and the second LED 45 of the camera module 31. This illuminates infrared light onto the upper body or head of the occupant observing the front of the vehicle 1.

[0117] In step ST5, the monitoring and control unit 36 ​​acquires current shooting data from the shooting sensor 41, and acquires the image components of the occupant looking forward contained in the shooting data, as a forward-looking image (frontal view image). In the forward-looking image, image components formed by infrared light about the occupant's eyes, nose, and mouth can be included at an angle substantially different from the frontal view. When the angle at which the occupant is photographed changes, the relative configuration in the image, such as the spacing between the eyes, changes. The forward-looking image may also include vein patterns of the head or eyeballs formed by infrared light. The monitoring and control unit 36 ​​can also extract and obtain information about these vein patterns of the head, eyeballs, eyelids, etc., from the shooting data.

[0118] Furthermore, it is expected that in the image components capturing such a forward-looking occupant's head, high-quality images will include those of the head itself, showing the occupants looking forward without drowsiness or glancing sideways. These forward-looking head components can be appropriately correlated with the image components of the head facing directly ahead. As a result, in the case of an image of, for example, an occupant looking forward while drowsy, by performing the same processing as converting the image components of the occupants' forward-looking head (where they are neither drowsy nor glancing sideways) to the image components of the head facing directly ahead, image components of the head facing directly ahead while drowsy can be obtained. These image components of the head facing directly ahead ensure that the image represents the occupant's drowsy state. While performing uniform processing regardless of the occupant's characteristics, there is a possibility that the occupant is not drowsy, but this possibility can be reliably reduced.

[0119] In step ST6, the monitoring and control unit 36 ​​performs the user registration process.

[0120] During user registration, the monitoring and control unit 36 ​​records the images of the two occupants captured as registration data, along with the unique identification information inherent in each occupant. Thus, occupant data containing the registration data for the new occupant is recorded in the memory 35. The memory 35, as a recording unit, records the capture data taken by the image sensor 41 of each occupant while a predetermined image is displayed on the liquid crystal device 32, as the capture data for each occupant. The registration data recorded for each occupant in the memory 35 includes data captured from a frontal view of the occupant and data captured from a forward-looking view of the occupant.

[0121] Figure 8 yes Figure 3 The flowchart of the occupant monitoring and control performed by the monitoring and control unit 36.

[0122] The monitoring and control unit 36 ​​can also repeat operations, for example, while the occupants are riding in the car 1. Figure 8 Monitoring and control.

[0123] In step ST11, the monitoring and control unit 36 ​​determines whether an occupant has boarded the vehicle. The monitoring and control unit 36 ​​can also determine this based on, for example, the detection of a door switch sensor 11 opening or an image captured by the sensor 41. If an occupant has boarded the vehicle, the monitoring and control unit 36 ​​proceeds to step ST12. If no occupant has boarded the vehicle, the monitoring and control unit 36 ​​repeats this process. Alternatively, if no occupant has boarded the vehicle, the monitoring and control unit 36 ​​can terminate the process. Figure 8 Monitoring and control.

[0124] Starting from step ST12, the monitoring and control unit 36 ​​begins monitoring the passengers. The monitoring and control unit 36 ​​first identifies the passengers.

[0125] The monitoring and control unit 36 ​​acquires new captured images from the imaging sensor 41, extracts image components of the occupants, and compares them with multiple occupant data registered in the memory 35.

[0126] At this time, the monitoring and control unit 36 ​​can also compare the images based solely on the forward-looking image components from the registered image data of each occupant registered in the memory 35. Forward-looking images typically contain prominent feature points such as the nose of the face. Even when comparing only the forward-looking image components, occupant matching can be determined with high accuracy. Alternatively, the monitoring and control unit 36 ​​can compare the feature points extracted from each image instead of directly comparing the images.

[0127] If the registered shooting data contained in the occupant data registered in the memory 35 is matched with a certain degree of accuracy or higher, the monitoring and control unit 36 ​​can also identify the occupant of the vehicle as the occupant of the registered shooting data.

[0128] In this case, the monitoring and control unit 36 ​​identifies the passenger as the passenger determined by comparing it with multiple passenger data registered in the memory 35.

[0129] If no registered video data matches with a certain degree of accuracy among the multiple occupant data registered in the memory 35, the monitoring and control unit 36 ​​can also identify the occupant as unregistered.

[0130] In step ST13, the monitoring and control unit 36 ​​acquires the occupant data of the identified occupant.

[0131] In step ST14, the monitoring and control unit 36 ​​performs setting processing using the acquired occupant data. If the occupant data includes various setting data for occupants, the monitoring and control unit 36 ​​performs setting processing based on that setting data. The monitoring and control unit 36 ​​outputs the setting data information to each part of the vehicle 1. This performs processing regarding occupant seating positions, initial settings for the presence or absence of driver assistance, driving preference information during automatic driving, information about the server devices used, occupant protection settings, and air conditioning settings.

[0132] In addition, based on the latest captured data, the monitoring and control unit 36 ​​determines, for example, whether a child seat is installed in the front passenger seat 4, and if a child seat is installed, executes a setting to prevent the airbag from deploying in the front passenger seat 4.

[0133] Starting from step ST15, the monitoring and control unit 36 ​​begins monitoring the identified occupants. The monitoring and control unit 36 ​​first acquires the latest image data from the imaging sensor 41.

[0134] In step ST16, the monitoring and control unit 36 ​​determines the image components of the upper body and head of the occupants contained in the latest captured data. The monitoring and control unit 36 ​​uses the image components of the registered captured data in the memory 35 for identified occupants as a reference to determine the image components of the upper body and head of the occupants contained in the latest captured data. Here, the monitoring and control unit 36 ​​can also infer the data captured from the front of the occupants contained in the current captured data captured by the capturing sensor 41 based on, for example, the difference between data captured from the front of each occupant registered in the memory 35 and data captured from the occupants looking forward, and determine the state of each occupant riding in the car 1 based on the image components inferred from the data captured from the front of the occupants. The monitoring and control unit 36 ​​can also use, for example, data captured from the front of the occupants already registered in the memory 35 to correct lens distortion and orientation of the occupant images contained in the current captured data, and infer the data captured from the front of the occupants based on the current captured data. Furthermore, in the absence of registered shooting data, the monitoring and control unit 36 ​​can determine the image components of the occupant's upper body and head contained in the latest shooting data by using the image components of the standard registered shooting data in the memory 35 as a reference.

[0135] The monitoring and control unit 36 ​​determines factors such as the opening and closing of the eyes, the direction of the gaze, and the orientation of the head as the occupant's condition in the latest captured data. The monitoring and control unit 36 ​​can also detect venous pulse. Furthermore, for example, if the occupant (driver) has their eyes closed, their gaze is not forward, their head is not facing forward, or their pulse is high, the monitoring and control unit 36 ​​determines that the driver is not fit to drive. In other cases, the monitoring and control unit 36 ​​can determine that the driver is fit to drive.

[0136] In this way, the monitoring and control unit 36 ​​acts as a judgment unit, using the registered shooting data recorded in the memory 35 as reference data to determine the status of each occupant riding in the car 1.

[0137] Furthermore, the monitoring and control unit 36 ​​at least determines the opening and closing state of the driver's eyes or the driver's gaze as the state of the driver riding in the car 1. At this time, the monitoring and control unit 36 ​​can use high-quality registration and shooting data as a reference, and therefore, based on information such as the opening and closing state of each occupant's eyes, as well as changes in the shooting state of the pupils between the upper and lower eyelids and changes in the shooting state of the sclera, it can accurately determine the occupant's gaze direction and other eye expressions.

[0138] In contrast, when making the same judgments without using the registered video data of each occupant, it's difficult to accurately determine eye expressions such as gaze direction, taking into account individual differences like eye size. This can lead to excessive warnings based on the occupant's physical characteristics, causing discomfort for that occupant.

[0139] In step ST17, the monitoring and control unit 36 ​​determines whether control regarding the movement of the vehicle 1 is needed based on the status determination results of the occupants, such as the driver. For example, if it is determined that the driver is not in a state suitable for driving, the monitoring and control unit 36 ​​determines that control is needed and proceeds to step ST18. Otherwise, if all occupants, including the driver, are in a state suitable for movement, the monitoring and control unit 36 ​​determines that control is not needed, skips step ST18, and proceeds to step ST19.

[0140] In step ST18, the monitoring and control unit 36 ​​performs control over the driving of the vehicle 1, etc.

[0141] For example, if the monitoring and control unit 36 ​​determines that the driver's gaze is not forward or that their head is not facing forward, it will issue a warning to the driver. The warning to the driver can also be, for example, a warning display on the LCD device 32 or an audible warning from the speaker device 14. Even if a warning is issued, if the driver does not look forward, i.e., if the monitoring and control unit continuously determines that the driver's gaze is not forward, it can also switch the vehicle 1's driving mode to automatic driving and decelerate the vehicle 1 to a stop. When the vehicle 1 is decelerated and stopped, the monitoring and control unit 36 ​​can also activate hazard lights (not shown) and transmit emergency information via the external communication device 18.

[0142] For example, if the system determines that the driver is drowsy or has a high pulse, the monitoring and control unit 36 ​​will issue a warning to the driver or slow down and stop the vehicle 1. When slowing down and stopping the vehicle 1, the monitoring and control unit 36 ​​may also activate hazard lights (not shown) or send emergency information via the external communication device 18.

[0143] Furthermore, if driving continues for more than the prescribed period, if the eyes open and close at a prescribed frequency, or if the head is tilted slightly downward, the monitoring and control unit 36 ​​may determine in step ST17 that control is needed and execute outputs such as prompting the driver to rest.

[0144] In step ST19, the monitoring and control unit 36 ​​determines whether to continue occupant monitoring and control. The monitoring and control unit 36 ​​may also determine occupant disembarkation based on, for example, the opening and closing detection of the door switch sensor 11 or the image captured by the sensor 41. If the occupant disembarks, the monitoring and control unit 36 ​​determines that occupant monitoring and control should not continue, and the process proceeds to step ST20. Otherwise, the monitoring and control unit 36 ​​determines that occupant monitoring and control should continue, and the process returns to step ST15. The monitoring and control unit 36 ​​repeats the process from step ST15 to step ST19 until it determines that occupant monitoring and control should not continue.

[0145] In step ST20, the monitoring and control unit 36 ​​updates the occupant data registered in the memory 35 as occupant identification data, processing the process when an occupant disembarks. The monitoring and control unit 36 ​​obtains setting information for occupant disembarkation from various parts of the vehicle 1 and updates the occupant data registered in the memory 35. Thus, the occupant data registered in the memory 35 becomes data tailored to the occupant's preferences. The latest settings regarding the occupant are automatically updated during the next ride.

[0146] Additionally, the monitoring and control unit 36 ​​can temporarily record passenger data for unregistered passengers in the memory 35. This allows for immediate association with the data should the passenger subsequently register.

[0147] As described above, this embodiment includes: a camera sensor 41 that captures images of the vehicle 1's interior to monitor the status of multiple occupants riding in the vehicle 1; a liquid crystal device 32 that displays images for each occupant to identify; a memory 35 that records the captured data from the camera sensor 41 of each occupant when a predetermined image is displayed on the liquid crystal device 32 as registered captured data for each occupant; and a determination unit that uses the registered captured data recorded in the memory 35 as reference data to determine the status of each occupant riding in the vehicle 1. Therefore, in this embodiment, the status of each occupant riding in the vehicle 1 can be determined using the current captured data from the camera sensor 41 and the registered captured data from the camera sensor 41 of each occupant when a predetermined image is displayed on the liquid crystal device 32, which serves as reference data. The influence of the occupant's viewing position and body characteristics in the captured data can be suppressed, resulting in a more accurate determination of the status of each occupant riding in the vehicle 1.

[0148] In contrast, if the state of each occupant in the car 1 is determined solely based on the current shooting data captured by the shooting sensor 41, the state of each occupant may not be accurately determined due to the influence of their viewing position in the shooting data and their physical characteristics. In this embodiment, registered shooting data captured by the shooting sensor 41 of each occupant while a predetermined image is displayed on the liquid crystal device 32 is used as reference data. Therefore, these influences can be suppressed, and the state of each occupant can be determined more accurately.

[0149] In particular, in this embodiment, the image sensor 41 is disposed on the back side of the display operation panel 33 displaying the screen, that is, further disposed on the back side of the non-display portion 33b which is set inside the rectangular frame 33a circumscribed outside the screen display area. Therefore, the image sensor 41 can capture an image equivalent to the image captured from the back side of the screen displayed in the liquid crystal device 32. The image sensor 41 can capture the heads of each occupant observing the screen from its front. Moreover, by using the registered capture data of each occupant's head captured from the front as reference data, the state of each occupant contained in the current capture data captured by the image sensor 41 can be determined more accurately.

[0150] As a result, in this embodiment, the display operation panel 33 of the liquid crystal device 32 and the shooting sensor 41 are disposed offset from the center of the vehicle width direction in the central part of the vehicle 1. The shooting sensor 41 has a wide-angle lens 42 that can capture the head view of multiple occupants sitting in the front row of the vehicle 1 when it is centrally disposed in the central part of the vehicle width direction of the vehicle 1. The determination unit determines the state of each of the multiple occupants sitting in the vehicle 1, at least the open and closed state of the driver's eyes or the line of sight, and can also more accurately determine the state of each of the multiple occupants sitting in the vehicle 1, including the driver.

[0151] In particular, in this embodiment, a non-display portion 33b is provided inside the rectangular frame 33a that is circumscribed outside the display area in the display operation panel 33 of the display liquid crystal device 32. The non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32 along the upper edge of the display area as a notch in the center of the upper edge of the display area. At this time, the non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32, covering at least the range from the center of the display area in the vehicle width direction to the center of the vehicle width direction of the car 1, and is formed with the center of the display area in the vehicle width direction as a reference, forming a shape that is symmetrical in the vehicle width direction of the car 1. Furthermore, the image sensor 41 is offsetly disposed on the back side of the display operation panel 33 of the liquid crystal device 32, moving from the center of the display area in the vehicle width direction toward the center of the vehicle width direction of the car 1, so as to occupy the range from the center of the display area in the vehicle width direction to the center of the vehicle width direction of the car 1. Therefore, the field of view of the imaging sensor 41 is not obstructed by the display area of ​​the liquid crystal device 32, and even though the display operation panel 33 of the liquid crystal device 32 is positioned offset from the center of the vehicle width direction in the central part of the vehicle 1, it can still capture images of the heads of multiple occupants seated in the front row of the vehicle 1 from the central part of the vehicle 1. Regardless of whether the same occupant is seated on the right or left side of the vehicle width direction, images can be captured under the same conditions. This improves the accuracy of occupant status assessment regardless of their seating position.

[0152] In addition, in the display operation panel 33 of the display liquid crystal device 32, the non-display unit 33b sets the center of the display area in the vehicle width direction as the reference and forms a shape that is symmetrical in the vehicle width direction of the car 1. As a result, the shape of the displayed screen can be set to a stable shape that is symmetrical in the left and right direction.

[0153] The above embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or alterations can be made without departing from the spirit of the invention.

[0154] For example, in the above embodiment, the liquid crystal device 32 is arranged longitudinally.

[0155] In addition, LCD devices 32 can also be configured horizontally.

[0156] Figure 9 It is about Figure 6A An explanatory diagram of the configuration of a modified camera module 31 with a shooting sensor 41 relative to the display operation panel 33 of an LCD device 32.

[0157] Even in this case, by having at least the central position of the imaging sensor 41 located on the back side of the display operation panel 33 of the liquid crystal device 32, the display area of ​​the image of the liquid crystal device 32 is shifted towards the passenger side, and located between the central position Yd of the display area and the central position Y0 of the vehicle width direction of the vehicle body 2, an image equivalent to that captured from the back side of the screen displayed on the liquid crystal device 32 can be captured, achieving the same effect as the above-described embodiment.

[0158] Additionally, in this configuration, the capturing sensor 41 or camera module 31 is positioned inside the holding member 61 so as not to protrude beyond the holding member 61 at the periphery of the display operation panel 33 of the liquid crystal device 32. Furthermore, the capturing sensor 41, the first LED 43, and the second LED 45 of the camera module 31 are configured to be located inside the outer circle of the image display area shown by the dashed circle in the figure.

[0159] Figure 10 It is about Figure 9 An explanatory diagram of the configuration of a camera module 31 with a shooting sensor 41 relative to the display operation panel 33 of the LCD device 32, as another variation.

[0160] Figure 10 The liquid crystal device 32 has a generally quadrilateral display area. In this case, by arranging the camera module 31 along the upper edge of the quadrilateral display area, a similar effect to the embodiment described above can be achieved. Furthermore, the image sensor 41, the first LED 43, and the second LED 45 of the camera module 31 are arranged inside the outer circle of the image display area shown by the dashed circle in the figure. Moreover, when the display operation panel 33 of the liquid crystal device 32 is offset towards the passenger side, by tilting the image sensor 41 towards the driver side and setting the center position of the image sensor 41 between the center position Yd of the quadrilateral display area and the center position Y0 of the vehicle width direction of the vehicle body 2, a similar effect to the embodiment described above can be achieved.

[0161] In the above embodiment, the display operation panel 33 of the liquid crystal device 32 and the image sensor 41 are centrally located in the middle part of the vehicle 1 in the width direction. The image sensor 41 has a wide-angle lens 42 that can capture the head view of multiple occupants sitting in the front row of the vehicle 1 when centrally located in the middle part of the vehicle 1 in the width direction. The monitoring and control unit 36 ​​determines at least the open / closed state of the occupants' eyes or their line of sight as the state of the occupants sitting in the vehicle 1.

[0162] The occupant monitoring device may not fall into any of these categories. For example, the occupant monitoring device may only monitor the driver. Even in this case, it is expected that the accuracy of the monitoring judgment can be improved, for example, by configuring the image sensor 41 on the back side of the display operation panel 33 of the liquid crystal device 32 to be inside the outer circle of the image display area, or by the monitoring control unit 36 ​​using the registered image data recorded in the memory 35 as reference data to determine the state of the occupant in the car 1.

[0163] Symbol Explanation

[0164] 1…car (vehicle), 2…body, 3…cabin, 4…seat, 5…instrument panel, 6…center console, 7…steering wheel, 10…control device, 11…door switch sensor, 12…vehicle speed sensor, 13…parking sensor, 14…speaker equipment, 15…occupant monitoring device, 16…position setting device, 17…driver assistance device, 18…external communication device, 19…occupant protection device, 20…air conditioning device, 21…in-vehicle network, 31…camera module, 32…LCD Device (display component), 33…display operation panel (panel), 33a…rectangular frame, 33b…non-display unit, 34…input / output device, 35…memory (recording component), 36…monitoring and control unit (judgment unit), 41…image sensor (image component), 42…wide-angle lens, 43…first LED, 44…first projection lens, 45…second LED, 46…second projection lens, 51…first occupant data, 52…second occupant data, 61…retention component.

Claims

1. A vehicle occupant monitoring device, comprising: The camera component captures images of the vehicle's interior in order to monitor the status of multiple occupants, including the driver. Display components that display images for identification by occupants of the vehicle; as well as The judgment unit uses the image data captured by the imaging component of the occupants while the image is displayed on the display component to determine the state of the occupants riding in the vehicle. In this configuration, a non-display portion is provided inside a rectangular frame circumscribed to the display area of ​​the display component on the panel displaying the image. The camera component is disposed on the back side of the non-display portion of the panel of the display component. The display panel and the camera component are disposed offset from the center of the vehicle in the width direction towards the side opposite to the driver. The non-display portion is formed in the panel displaying the image of the display component, extending at least from the center of the display area of ​​the image in the vehicle width direction to the center of the vehicle width direction. The shooting component is offsetly configured on the back side of the panel of the display component, moving from the center of the display area of ​​the image in the vehicle width direction toward the center of the vehicle width direction, so as to be located within the range from the center of the display area of ​​the image in the vehicle width direction to the center of the vehicle width direction. The display component is configured at an angle relative to the front-rear direction of the vehicle body, facing the driver.

2. The occupant monitoring device for a vehicle according to claim 1, wherein, The non-display portion is formed as a notch along the upper edge of the display area of ​​the screen in the panel displaying the image of the display component.

3. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, The non-display portion is formed in the panel displaying the image of the display component, as a notch in the center of the upper edge of the display area of ​​the image.

4. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, The non-display unit, in the panel displaying the image of the display component, sets the center of the display area of ​​the image in the vehicle width direction as a reference, and forms a shape that is symmetrical in the vehicle width direction.

5. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, It has a retaining member that holds the periphery of the panel of the display component. The shooting component is positioned inside the holding component.

6. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, It is equipped with a camera module, which includes the shooting component and a lighting component for projecting light onto at least the driver riding in the vehicle. The light-projecting component and the shooting component of the camera module are together disposed on the back side of the non-display section.

7. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, The wide-angle lens is aligned with the shooting component, and the wide-angle lens has a field of view capable of capturing the heads of multiple occupants seated in the front row of the vehicle.

8. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, It has a recording unit that records the data captured by the camera unit on the screen displayed on the display unit as the individual occupant's video data. The judgment unit uses the shooting data recorded in the recording component to determine the status of the occupants riding in the vehicle.

9. The occupant monitoring device for a vehicle according to claim 8, wherein, The recording unit records the occupant's facing image and forward-looking image as the shooting data captured on camera. The judgment unit infers the data captured by the shooting unit that was taken from the front of the occupant, based on the difference between the data recorded in the recording unit and the data taken from the occupant looking forward. The state of the occupants in the vehicle is determined based on image components inferred from data captured from the front of the occupants.

10. The occupant monitoring device for a vehicle according to claim 1 or 2, wherein, The judgment unit at least judges the state of the driver's eyes or the driver's line of sight as the state of the driver riding in the vehicle.

Images