Occupant monitoring device for a vehicle

Abstract

The present application relates to an occupant monitoring apparatus of a vehicle. Even in a case where a display member that displays an operation screen of an occupant is used together with a photographing member for occupant monitoring in a vehicle, occupant ease of use of the display member and state monitoring of the occupant are favorably combined. An occupant monitoring apparatus (15) of a vehicle (1) has a display member (32) that displays an operation screen of an occupant of the vehicle (1), a control section (36) that performs control corresponding to an operation content judged as an operation of the occupant on the operation screen of the display member (32), and a photographing member (41) that can photograph the occupant of the vehicle (1). The control section (36) does not perform control corresponding to the judged operation content in a case where the operation content of the occupant is an operation performed at least with a movement toward the photographing member (36).

Description

Technical Field

[0001] This invention relates to a vehicle occupant monitoring device. Background Technology

[0002] In vehicles, LCD devices with touch panels have begun to be used as user interfaces for display purposes such as vehicle settings. These LCD devices with touch panels display operation objects such as buttons.

[0003] In this situation, the vehicle occupants operate objects such as buttons displayed on an LCD device with a touch panel. The vehicle's control unit performs controls corresponding to the operation.

[0004] Furthermore, LCD devices with touch panels, such as mobile terminals, allow for various operations beyond simply clicking on the object being touched. These include sliding operations, such as moving the device along its surface. Passengers are accustomed to these operations when using mobile terminals and similar devices.

[0005] Furthermore, in vehicles, it is believed that future requirements will include monitoring the occupant's status during driver assistance or autonomous driving (Patent Documents 1, 2).

[0006] In this case, in order to monitor the occupants of the vehicle, the vehicle may install a camera on the dashboard or other surface in front of the occupants, and determine and monitor the occupants' status based on the images captured by the camera.

[0007] Existing technical documents

[0008] Patent documents

[0009] Patent Document 1: Japanese Patent Application Publication No. 2019-014359

[0010] Patent Document 2: Japanese Patent Application Publication No. 2019-014360 Summary of the Invention

[0011] The problem that the invention aims to solve

[0012] To meet these various requirements, consider placing the shooting equipment near an LCD device with a touch panel installed in the vehicle.

[0013] Therefore, a predetermined display can be made on the LCD device, and the camera can capture images of the occupants observing the display from the front. If pre-existing data captured from the front of the occupants is available, for example, comparing the data from that front view with current data can more accurately determine the state of each occupant, such as the driver. In this case, it is desirable for the camera to be positioned near the LCD device.

[0014] However, if the camera is placed near the LCD display, the camera's field of view may be obstructed by the fingers, hands, wrists, or elbows of the occupant performing a swiping operation on the LCD. In particular, when monitoring the status of multiple front-seat occupants, including the driver, using data captured by a single camera, it is desirable to position the camera and the LCD with a touch panel together in the center of the vehicle's width. In this case, the camera's field of view may easily be obstructed by the fingers, hands, wrists, or elbows of the occupant performing the swiping operation. If the occupant cannot be photographed, their status cannot be monitored.

[0015] Therefore, in LCD devices with touch panels installed in vehicles, it is considered to restrict all swipe operations and allow only click operations. However, if swipe operations are restricted entirely, occupants will repeatedly perform click operations. Occupants will be unable to operate the LCD devices in vehicles using their familiar swipe operations. Occupants will not be able to use the familiar and comfortable swipe operations found on mobile devices as the operating screens in vehicles. The ease of use of LCD devices with touch panels installed in vehicles will decrease.

[0016] Thus, in vehicles, it is required that even when the display unit that displays the occupant's operating screen is used together with the camera unit for occupant monitoring, the ease of use of the display unit for occupant and the monitoring of the occupant's status can be well balanced.

[0017] Technical solutions for solving the problem

[0018] The present invention provides a vehicle occupant monitoring device comprising: a display unit that displays an operation screen of a vehicle occupant; a control unit that performs control corresponding to the operation content determined to be an operation performed by an occupant on the operation screen of the display unit; and a camera unit capable of capturing images of the vehicle occupant, wherein the control unit does not perform control corresponding to the determined operation content if the occupant's operation content is an operation accompanied at least by a movement toward the camera unit.

[0019] Optionally, the camera component may be configured adjacent to the display area of ​​the display component, and the control unit may not perform control corresponding to the determined operation if the occupant's operation involves moving towards the camera component in the peripheral area of ​​the display area.

[0020] Optionally, the display component may have a panel larger than the display area of ​​the operation screen as the panel for displaying the operation screen, and the camera component may be disposed on the back side of the panel of the display component.

[0021] Optionally, the control unit may not perform control corresponding to the determined operation content if the operation accompanying the movement toward the shooting component is an operation following the surface of the display component toward the shooting component.

[0022] Optionally, the display unit may display multiple slidable objects in the operation screen, and the control unit shall not perform control corresponding to the determined sliding operation when the sliding operation of the objects displayed around the shooting unit moves toward the shooting unit.

[0023] Optionally, the control unit may not perform control corresponding to the determined operation content when the direction of operation following the surface of the display unit is toward the shooting unit, but may perform control corresponding to the determined operation content when the direction of operation following the surface of the display unit is toward a direction different from that of the shooting unit.

[0024] Optionally, the control unit may not perform control corresponding to the determined operation if the occupant's operation is at least an operation performed while the vehicle is in motion and the occupant is moving towards the camera component.

[0025] Optionally, the display unit and the camera unit may be located in the central part of the vehicle in the width direction, and the control unit monitors the status of the vehicle's occupants based on the images captured by the camera unit.

[0026] Invention Effects

[0027] In this invention, the control unit does not perform control corresponding to the determined operation content when the occupant's operation involves at least an action accompanied by movement towards the camera component. Therefore, the occupant cannot operate the display unit showing the vehicle occupant's operation screen while moving towards the camera component. As a result, in this invention, it is expected that the camera component's field of view will not be obstructed by the occupant's fingers, hands, wrists, elbows, etc., during operation. Furthermore, if the occupant is not facing the camera component, they can perform unrestricted operations accompanying the action; therefore, operations accompanying the action displayed on the display unit are possible.

[0028] Thus, in this invention, the operation accompanying the occupant's actions is not uniformly restricted. Therefore, even when the display component that displays the operation screen is used together with the camera component for occupant monitoring, the ease of use of the display component for the occupant and the monitoring of the occupant's status can be well achieved. Attached Figure Description

[0029] Figure 1This is an explanatory diagram of a car to which the occupant monitoring device according to an embodiment of the present invention is applied.

[0030] Figure 2 yes Figure 1 A diagram illustrating the control system of a car.

[0031] Figure 3 yes Figure 2 An explanatory diagram of a car occupant monitoring device.

[0032] Figures 4(A)-(B) are about Figure 3 An illustration of the in-vehicle configuration of the occupant monitoring system, which includes an LCD device with a display control panel and a camera module with a shooting sensor.

[0033] Figures 5(A)-(D) are explanatory diagrams of the driver's shooting position as a car occupant.

[0034] Figure 6 This is a detailed diagram illustrating the LCD device and camera module of the occupant monitoring system located in the central part of the vehicle body in the width direction.

[0035] Figure 7 yes Figure 3 The flowchart of the occupant registration control performed by the monitoring and control department.

[0036] Figure 8 yes Figure 3 The flowchart of the occupant monitoring and control performed by the monitoring and control department.

[0037] Figure 9 It refers to the central portion of the vehicle body located in the width direction and capable of being monitored and controlled. Figure 3 The flowchart shows the response control related to the operation of the display panel.

[0038] Figure 10 It is about Figure 6 An explanatory diagram of a first modified example of an occupant monitoring device consisting of an LCD device and a camera module located in the central part of the vehicle body in the width direction.

[0039] Figure 11 It is about Figure 6 A second modified example of an occupant monitoring device consisting of an LCD device and a camera module located in the central part of the vehicle body in the width direction is illustrated in the diagram. Detailed Implementation

[0040] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0041] Figure 1 This is an explanatory diagram of a car 1 in which the occupant monitoring device 15 of an embodiment of the present invention is applied.

[0042] Car 1 is an example of a vehicle. Car 1 can be powered by an internal combustion engine, by the electrical energy stored in a battery, or by a combination of these.

[0043] exist Figure 1 A passenger compartment 3 is formed on the body 2 of a car 1. Multiple front seats 4 for the driver and front passenger are provided in the passenger compartment 3, along with a long rear seat 4. Occupants, including the driver, sit in the seats 4. A dashboard 5 is positioned in front of the multiple front seats 4, at the front of the passenger compartment 3, extending along the width of the body 2.

[0044] Figure 2 yes Figure 1 Explanation diagram of the control device 10 of car 1.

[0045] Figure 2 The control device 10 includes: a door switch sensor 11, a vehicle speed sensor 12, a parking sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driver assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 connecting them.

[0046] The in-vehicle network 21 can also be a wired communication network for the vehicle 1, such as one based on CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 can be a communication network such as a LAN, or a combination of both. A wireless communication network can also be included as a part of the in-vehicle network 21.

[0047] The door switch sensor 11 detects the opening and closing of the doors of the car 1.

[0048] Vehicle speed sensor 12 detects the speed of the moving car 1. Vehicle speed sensor 12 can also detect a stationary state.

[0049] The parking sensor 13 detects a parking state that keeps the car 1 stationary. The parking sensor 13 can also detect parking states by, for example, the gear lever (not shown) being moved to the parking position and the parking lever (not shown) being moved to the braking position.

[0050] When the occupant monitoring device 15 detects, for example, the door switch sensor 11, opening or closing of the door, it performs identification processing of the occupant sitting in the cabin 3.

[0051] When multiple occupants are identified, the occupant monitoring device 15 can also monitor each individual occupant individually.

[0052] The occupant monitoring device 15 can also output the information of each identified occupant and the monitoring-based information to various parts of the control device 10 through the in-vehicle network.

[0053] In particular, the occupant monitoring device 15 identifies, for example, the occupant seated in the driver's seat 4 as the driver, and monitors the identified driver for squinting or drowsiness. Furthermore, when the driver is in a prescribed state, the occupant monitoring device 15 performs controls to attract the driver's attention and controls to prevent hazards. At this time, the occupant monitoring device 15 can also output driver information, information for attracting attention, and information for preventing hazards to various parts of the control device 10 via the in-vehicle network.

[0054] The speaker device 14 outputs sounds, warning sounds, etc. The speaker device 14 can also output warnings to the driver and other occupants issued by the occupant monitoring device 15.

[0055] The position setting device 16 adjusts, for example, the fore-aft and vertical position and backrest angle of the seat 4, the fore-aft and vertical position and angle of the steering wheel, and the fore-aft and vertical position and angle of various pedals. The position setting device 16 changes the seat position based on the occupant information output by the occupant monitoring device 15.

[0056] The driving assistance device 17 assists in driving operations of the vehicle 1 performed manually by the driver, or assists in driving operations of the vehicle 1 performed automatically. The driving assistance device 17 controls the acceleration, deceleration, stopping, and steering of the vehicle 1. Based on the driver information output by the occupant monitoring device 15, the driving assistance device 17 performs driving assistance corresponding to the driver.

[0057] The external communication device 18 establishes a wireless communication path with, for example, a base station of a public wireless communication network, a base station of a commercial wireless communication network, or a base station for high-traffic information, and performs data communication using the established wireless communication path. The external communication device 18 can also perform bidirectional data communication, for example, with a server device for assisted autonomous driving. The external communication device 18 can also send occupant information, including the driver, output by the occupant monitoring device 15 as, for example, emergency assistance information, to the server device.

[0058] Occupant protection device 19 performs controls to protect occupants in the event of a detected or predicted collision with vehicle 1. Occupant protection device 19 protects the occupant seated in seat 4 by deploying, for example, an airbag (not shown), or by applying tension to the seatbelt. Occupant protection device 19 can also perform occupant protection based on occupant information output from occupant monitoring device 15.

[0059] The air conditioning unit 20 controls the temperature and oxygen concentration of the vehicle compartment 3. The air conditioning unit 20 supplies, for example, cooled or superheated air to the vehicle compartment 3, thereby adjusting the vehicle compartment 3 to a set temperature. The air conditioning unit 20 can also perform air conditioning corresponding to occupant information output from the occupant monitoring device 15.

[0060] By controlling the vehicle based on occupant information output by these occupant monitoring devices 15, occupants can ride comfortably, for example, according to their own settings. The driver can concentrate on, for example, driving the vehicle 1.

[0061] Figure 3 yes Figure 2 Explanation diagram of the occupant monitoring device 15 of car 1. Figure 3 The occupant monitoring device 15 monitors not only the driver, but also, for example, multiple occupants seated in the front seats 4. Figure 3 The occupant monitoring device 15 includes: a camera module 31, an LCD device 32 with a display operation panel 33, an input / output device 34, a memory 35, and a monitoring and control unit 36 ​​connected thereto.

[0062] Input / output device 34 is connected to in-vehicle network 21. Input / output device 34 inputs and outputs data with other parts of the vehicle 1 via in-vehicle network 21.

[0063] The liquid crystal device 32 displays images on the display surface of the display operation panel 33. The liquid crystal device 32, as a display component, displays images that are recognizable by each occupant of the vehicle 1. The displayed images include, for example, operation screens operated by the occupants.

[0064] The display operation panel 33 is a transparent or semi-transparent panel that is superimposed on the display surface of the liquid crystal device 32. The component formed by superimposing the display operation panel 33 and the display surface of the liquid crystal device 32 is a display component. The display operation panel 33 detects the occupant's operation on the display surface of the liquid crystal device 32. The display operation panel 33 can also output the occupant's operation position on the display surface of the liquid crystal device 32 to the monitoring and control unit 36.

[0065] The camera module 31 is used to capture images of multiple occupants seated in multiple seats 4 in the front row. The camera module 31 includes: a capture sensor 41, a wide-angle lens 42, a first LED 43, a first projection lens 44, a second LED 45, and a second projection lens 46.

[0066] The image sensor 41 is a semiconductor optical sensor such as a CCD or CMOS sensor. The image sensor 41 may also have a generally quadrilateral light-receiving surface, for example, with multiple light-receiving elements arranged thereon. The image sensor 41 may also output image data containing the captured image to the monitoring and control unit 36.

[0067] The wide-angle lens 42 is stacked on top of the image sensor 41. The wide-angle lens 42 may also be composed of multiple optical lenses to suppress image distortion at the periphery.

[0068] The imaging sensor 41 and the wide-angle lens 42 are imaging components, which can capture images of the occupants of the car 1 in order to monitor the status of multiple occupants in the car 1.

[0069] The wide-angle lens 42 can also be, for example, a lens that can capture the upper body or head of multiple occupants sitting in multiple seats 4 in the front row, with the shooting sensor 41 set in the center of the dashboard 5 in the vehicle width direction.

[0070] The first LED 43 and the second LED 45 can also be semiconductor light-emitting elements. The first LED 43 and the second LED 45 act as light-projecting components, projecting light onto the occupants of the car 1, which is the target of the imaging sensor 41. The first LED 43 and the second LED 45 can also project light, for example, infrared light. In this case, the imaging sensor 41 outputs imaging data, including the infrared image, to the monitoring and control unit 36. The first projection lens 44 is superimposed on the first LED 43. The first projection lens 44 can also primarily illuminate the driver, seat 4, with the light from the first LED 43. The second projection lens 46 is superimposed on the second LED 45. The second projection lens 46 can also primarily illuminate the occupant, seat 4, with the light from the second LED 45. Furthermore, the second projection lens 46 can also diffuse light and project it onto both the occupant, seat 4, and the driver, seat 4.

[0071] Memory 35 stores programs and data. Memory 35 can also be composed of non-volatile memory and volatile memory. Non-volatile memory may include, for example, HDD, SSD, EEPROM, etc. Volatile memory may include, for example, RAM.

[0072] Data on multiple occupants registered in vehicle 1 can also be managed and recorded in the memory 35 of the occupant monitoring device 15 for each occupant. Figure 3 The data represents first crew data 51 concerning the first crew member and second crew data 52 concerning the second crew member. A database is constructed in memory 35 using this multiple crew data.

[0073] Each occupant data, such as the first occupant data 51 and the second occupant data 52, may also include unique identification information for each occupant, such as registered image data of the occupant's head and eyes captured by the camera sensor 41, and various setting data for the occupant. The setting data may also include information such as the occupant's seat position, initial settings for the presence or absence of driver assistance, driving preference information during autonomous driving, information about the server device being used, occupant protection settings, and air conditioning settings.

[0074] In this way, the memory 35, as a recording component, can record the shooting data captured by each occupant when the shooting sensor 41 faces the predetermined screen displayed on the liquid crystal device 32 from the front, and use it as the registration shooting data of each occupant.

[0075] In addition, the memory 35 can also record crew data of unregistered general crew members.

[0076] The monitoring and control unit 36 ​​may be, for example, an ECU, a CPU, or other microcomputer. The monitoring and control unit 36 ​​reads a program from the memory 35 and executes it. Thus, the monitoring and control unit 36, serving as the control unit of the occupant monitoring device 15, is implemented. The monitoring and control unit 36 ​​can also perform controls corresponding to operations determined to be performed by the occupant on the operation screen of the LCD device 32.

[0077] Furthermore, the monitoring and control unit 36 ​​monitors the status of multiple occupants of the vehicle 1 based on data from images captured by the imaging sensor 41. To perform occupant monitoring, the monitoring and control unit 36 ​​performs, for example, registration processing and monitoring processing for the occupants of the vehicle 1. During monitoring processing, the monitoring and control unit 36 ​​can also identify the occupants of the vehicle 1 and perform monitoring processing for each occupant. In particular, regarding the occupant at least as the driver, the monitoring and control unit 36, acting as a control unit, uses the respective registered image data registered in the memory 35 as reference data for judgment, determining the occupant's state such as strabismus, drowsiness, or emergency situation.

[0078] Figures 4(A)-(B) are about Figure 3 An illustration of the in-vehicle configuration of the occupant monitoring device 15, which includes an LCD device 32 with a display operation panel 33 and a camera module 31 with a shooting sensor 41.

[0079] Figure 4(A) shows an instrument panel 5 and a center console 6 located at the front of the cabin 3.

[0080] The liquid crystal device 32 and the camera module 31 in this embodiment are, for example, Figure 1As shown, the camera module 31 is longitudinally positioned in the central part of the vehicle 1 along its width, extending from the dashboard 5 to the center console 6. Therefore, the camera sensor 41 of the camera module 31 can capture a wide-angle shot of the driver and passenger. The camera sensor 41 can capture the upper body including the driver's head and the upper body including the passenger's head.

[0081] The liquid crystal device 32 can also be centrally positioned in the width direction of the vehicle 1, so that the central position Yd of the displayed image in the width direction coincides with the central position Y0 of the vehicle 1. However, in this embodiment, the central position Yd of the displayed image in the width direction is offset slightly towards the side opposite to the driver compared to the central position Y0 of the vehicle 1. Furthermore, the liquid crystal device 32 is tilted relative to the width direction, facing the driver from its position. The displayed image of the liquid crystal device 32 can improve the driver's visibility while ensuring the visibility of passengers.

[0082] Figure 4(B) is an explanatory diagram of the in-vehicle shooting range generated by a camera module 31 with a shooting sensor 41 and a shooting wide-angle lens 42, which is obliquely set at an offset position in the central part of the vehicle width direction of the vehicle 1.

[0083] A wide-angle lens 42 is superimposed on the imaging sensor 41. Thus, the imaging sensor 41, as shown in FIG4(B) as the imaging range, can capture the entire vehicle interior 3. The imaging sensor 41 is positioned in the central part of the vehicle 1 in the width direction and can capture the heads of the driver and passengers seated in the multiple front seats 4.

[0084] However, when the wide-angle lens 42 is combined with the image sensor 41 to capture multiple occupants, the central portion of the image range becomes a high-quality area with minimal distortion. On the other hand, the peripheral portion of the image range becomes a distortion-generating area compared to the central portion. Therefore, when an occupant is seated on the right or left side in the vehicle width direction, it becomes difficult to capture the occupant's head without distortion. When using the opening or closing state of the occupant's eyes for judgment, the possibility of inaccurate judgment increases.

[0085] In this embodiment, the display operation panel 33 is arranged at an angle relative to the front-rear direction of the vehicle body 2, facing the driver. As a result, the high-quality area in the central portion of the image captured by the image sensor 41 is closer to the driver. The driver is easily captured in the high-quality area in the central portion of the image captured by the image sensor 41, and is less likely to be captured in the distortion-prone areas at the periphery. It is expected that the driver will be captured in an image with reduced distortion.

[0086] Figures 5(A) to (D) are explanatory diagrams showing the driver's position as an occupant of car 1.

[0087] Figure 5(A) is a front view of the head of the first driver. Figure 5(B) is a perspective view of the head of the first driver in Figure 5(A).

[0088] Regarding the camera sensor 41 located in the center of the vehicle width direction of the car 1, when the driver is facing forward of the car 1, it does not photograph the driver's head from the frontal direction as shown in Figure 5(A), but rather photographs the driver's head from an oblique direction as shown in Figure 5(B).

[0089] Figure 5(C) is a front view of the second driver's head. Figure 5(D) is a perspective view of the second driver's head in Figure 5(C).

[0090] Regarding the camera sensor 41 located in the center of the vehicle width direction of the car 1, when the driver is facing forward of the car 1, it does not photograph the driver's head from the frontal direction as shown in Figure 5(C), but rather photographs the driver's head from an oblique direction as shown in Figure 5(D).

[0091] Moreover, in the shooting data of Figure 5(D), unlike the shooting data of Figure 5(B), the driver's right eye is obscured by the driver's high nose, and only the black pupil can be seen.

[0092] In this situation, the monitoring and control unit 36 ​​cannot easily determine the opening and closing state of the driver's right eye based solely on the captured data in FIG5(D). In particular, as in FIG5(D), when the image is captured as a distorted image rather than an undistorted one, the monitoring and control unit 36 ​​finds it extremely difficult to determine the opening and closing state of the driver's right eye based solely on the captured data in FIG5(D).

[0093] Furthermore, the image data in Figure 5(D) does not include the sclera (white of the driver's right eye) to a level sufficient for state determination. Therefore, the monitoring and control unit 36 ​​cannot extract the state of the right eye based solely on the image data in Figure 5(D). Consequently, the monitoring and control unit 36 ​​may also be unable to determine the driver's head orientation, etc., if it cannot determine the right eye. For example, in cases where the image area of ​​the head in the image is determined based on the arrangement of the eyes and nose, or the arrangement of the eyes and mouth, the monitoring and control unit 36 ​​cannot determine the driver's head orientation, etc., based solely on the image data in Figure 5(D). Without data regarding the driver's head as a reference, the monitoring and control unit 36 ​​is more likely to be unable to accurately infer the driver's state when the image components do not include the driver's right eye.

[0094] When the camera module 31's shooting sensor 41 is positioned in the center of the vehicle 1 in the width direction, the state shown in Figure 5(D) is likely to occur.

[0095] Thus, there are individual differences among passengers, such as people with large eyes, small eyes, protruding eyes, and deep-set eyes.

[0096] The actual position of the driver's head may move not only in the direction of the view in the width of the vehicle, but also in the up and down direction. When taking these movements into account, it is not easy to accurately infer the state of the driver's head and eyes based solely on the current shooting data of the shooting sensor 41.

[0097] Figure 6 This is a detailed explanatory diagram of the LCD device 32 and camera module 31 of the occupant monitoring device 15 located in the central part of the vehicle body 2 in the width direction. Figure 6 This is a front view from three sides of the vehicle compartment. Figure 6 In the figure, the central position Yd of the display operation panel 33 of the LCD device 32 is offset from the central position Y0 in the width direction of the vehicle body 2 to the left, i.e., the side of the passenger.

[0098] The liquid crystal device 32, which has a display operation panel 33, uses resin retaining members 61, which serve as pressing parts, to hold the periphery of the display operation panel 33 at its four corners. The display operation panel 33 has a shape that is slightly larger than that of the liquid crystal device 32.

[0099] The liquid crystal device 32, serving as the display area for displaying images, is not a quadrilateral display surface like that of a typical monitor, but rather has a roughly concave shape with a central portion lacking the upper edge of a quadrilateral. A non-display portion 33b with a notch is formed inside the rectangular frame 33a that circumscribes the roughly concave display area of ​​the liquid crystal device 32. The display areas of the liquid crystal device 32 are located on the left and right sides of the non-display portion 33b. The non-display portion 33b is formed symmetrically along a line with reference to the central position Yd of the display operation panel 33 of the liquid crystal device 32.

[0100] Thus, the non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32 as a notch in the center of the upper edge of the display area of ​​the screen, inside the rectangular frame 33a that is circumscribed to the display area. The non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32 as a notch along the upper edge of the display area of ​​the screen. Furthermore, the non-display portion 33b is formed in the display operation panel 33 of the display liquid crystal device 32, extending at least from the center position Yd in the vehicle width direction of the screen display area to the center position Y0 in the vehicle width direction of the vehicle 1. The non-display portion 33b is formed in the display operation panel 33 in a symmetrical shape in the vehicle width direction of the vehicle 1, based on the center position Yd in the vehicle width direction of the screen display area.

[0101] Furthermore, in the display area of ​​the display operation panel 33, there are restriction areas 62 on the left, right and lower sides of the non-display section 33b where the camera module 31 is configured, which prohibit sliding operations in the direction of the camera module 31.

[0102] The camera module 31, which has a shooting sensor 41, is located on the back side of the display operation panel 33, on the back side of the non-display portion 33b formed above the display area of ​​the liquid crystal device 32. The camera module 31 is positioned inside the holding member 61 so as not to protrude outward from the holding members 61 at the four corners of the display operation panel 33.

[0103] Specifically, the camera module 31 is configured such that the shooting sensor 41 and the wide-angle lens 42 are located between the center position Yd of the display operation panel 33 of the LCD device 32 and the center position Y0 of the vehicle body 2 in the width direction. Thus, the shooting sensor 41 and the wide-angle lens 42 are configured such that their center positions in the width direction are located between the center position Yd of the display operation panel 33 of the LCD device 32 and the center position Y0 of the vehicle body 2 in the width direction.

[0104] Therefore, the image sensor 41 of the camera module 31 is positioned as if it were located within the LCD device 32 itself. The image sensor 41 is arranged adjacent to the operation screen display area of ​​the operation panel 33 on the LCD device 32, which displays the operation screen.

[0105] As a result, the camera module 31's image sensor 41 can capture images of the occupants from the rear side of the LCD device 32, which has a display operation panel 33.

[0106] Furthermore, in the camera module 31, the shooting sensor 41 and the wide-angle lens 42 are positioned in the central part along the vehicle width direction. The first LED 43 and the first projection lens 44 are positioned at the end part on the passenger side. The second LED 45 and the second projection lens 46 are positioned at the end part on the driver side. Thus, even if there are objects such as the steering wheel 7 between the driver and the instrument panel 5, the camera module 31 can project light and take pictures without being obstructed by the steering wheel 7 or the like.

[0107] Furthermore, with the camera module 31 positioned on the back of the display operation panel 33, the wide-angle lens 42, the first projection lens 44, and the second projection lens 46 can also be configured by processing the display operation panel 33 itself.

[0108] Figure 7 yes Figure 3 The flowchart of the occupant registration control performed by the monitoring and control unit 36.

[0109] When a new passenger boards the vehicle 1, the monitoring and control unit 36 ​​can also repeat the process. Figure 7 Registration control.

[0110] In step ST1, the monitoring and control unit 36 ​​first determines whether an occupant is registered.

[0111] If, for example, an occupant selects and operates the occupant registration button on a menu screen displayed on the LCD device 32, the monitoring and control unit 36 ​​determines that the occupant is registered and proceeds to step ST2. Otherwise, the monitoring and control unit 36 ​​determines that the occupant is not registered and terminates the process. Figure 7 Registration control.

[0112] Starting from step ST2, the monitoring and control unit 36 ​​performs the registration process for new occupants. The monitoring and control unit 36 ​​first displays a guide to begin occupant registration on the LCD device 32. Considering that occupants who operate the occupant registration button in the menu screen will generally be facing the guide display on the LCD device 32 directly, the monitoring and control unit 36 ​​can also provide a direct facing instruction in the guide to ensure the face is facing the LCD device 32. Additionally, the monitoring and control unit 36 ​​illuminates the first LED 43 and the second LED 45 of the camera module 31. This illuminates infrared light onto the upper body or head of the occupant who is facing the guide display on the LCD device 32.

[0113] In step ST3, the monitoring and control unit 36 ​​acquires the current shooting data from the shooting sensor 41 and acquires the image component of the occupant captured from the front of the newly registered object contained in the shooting data as a frontal image. If multiple occupants can be extracted from the images captured by the shooting sensor 41, the monitoring and control unit 36 ​​can also select one as an occupant. It is expected that the frontal image will contain image components formed by infrared light regarding the occupant's eyes, nose, and mouth with high accuracy. The image components formed by infrared light may also include vein patterns of the head or eyeballs. The vein patterns of the head or eyeballs depend on the number of feature points extracted, but can be used to identify an individual. Furthermore, the vein patterns of the head or eyeballs are less affected by the brightness and darkness patterns caused by the surface shape of the head or the concavity and convexity of the face. The vein pattern of the eyeballs generally extends from the sclera (white of the eye) towards the iris (black of the eye). The vein pattern of the eyelids covering the eyeballs is a different pattern from the vein pattern of the eyeballs. The monitoring and control unit 36 ​​can also extract and obtain information about these vein patterns of the head, eyeballs, eyelids, etc., from the shooting data. Moreover, it is expected that in the image components that capture the occupant's head facing forward, the high-quality image components include the various parts of the head.

[0114] In addition, the monitoring and control unit 36 ​​can repeatedly perform guidance or repeatedly acquire the current shooting data from the shooting sensor 41 until it can confirm that the image in the facing image contains image components formed by infrared light about the occupant's eyes, nose, and mouth.

[0115] Starting from step ST4, the monitoring and control unit 36 ​​displays a forward-looking (front-view) orientation on the liquid crystal device 32, guiding the occupant to observe the front of the vehicle 1. Considering that the occupant observes the front of the vehicle 1 based on the orientation display on the liquid crystal device 32, the monitoring and control unit 36 ​​also illuminates the first LED 43 and the second LED 45 of the camera module 31. This illuminates infrared light onto the upper body or head of the occupant observing the front of the vehicle 1.

[0116] In step ST5, the monitoring and control unit 36 ​​acquires current image data from the imaging sensor 41, and obtains the image components captured of the occupant looking forward within the image data, as a forward-looking image. In the forward-looking image, image components formed by infrared light about the occupant's eyes, nose, and mouth can be included at an angle substantially different from the front-facing image. The forward-looking image may also include vein patterns formed by infrared light on the forward-looking head or eyeballs. The monitoring and control unit 36 ​​can also extract and obtain information about these vein patterns (head, eyeballs, eyelids, etc.) from the image data.

[0117] Furthermore, it is expected that in the image component capturing the forward-looking occupant's head, high-quality images will include each part of the head looking forward in a state of neither drowsiness nor glancing sideways. The image components of the forward-looking head can appropriately correspond to the image components of the head facing directly. As a result, in the case of, for example, an image of an occupant looking forward in a drowsy state, by performing the same processing as converting the image components of the occupant's head looking forward in a state of neither drowsiness nor glancing sideways to the image components of the head facing directly, image components of the head facing directly while drowsy can be obtained. These image components of the head facing directly ensure that the image represents the occupant's drowsy state. While performing uniform processing regardless of the occupant's characteristics may result in the occupant not being drowsy, this possibility can be reliably reduced.

[0118] In step ST6, the monitoring and control unit 36 ​​performs the user registration process.

[0119] During user registration, the monitoring and control unit 36 ​​records the images of the two occupants captured as registration data, along with the unique identification information inherent in each occupant. Thus, occupant data containing the registration data for the new occupant is recorded in the memory 35. The memory 35, as a recording unit, records the capture data taken by the image sensor 41 of each occupant while a predetermined image is displayed on the liquid crystal device 32, as capture data for each occupant. The registration data recorded for each occupant in the memory 35 includes data captured from a frontal view of the occupant and data captured from a forward-looking view of the occupant.

[0120] Figure 8 yes Figure 3 The flowchart of the occupant monitoring and control performed by the monitoring and control unit 36.

[0121] The monitoring and control unit 36 ​​can also repeat operations, for example, while the occupants are riding in the car 1. Figure 8 Monitoring and control.

[0122] In step ST11, the monitoring and control unit 36 ​​determines whether an occupant has boarded the vehicle. The monitoring and control unit 36 ​​can also determine this based on, for example, the detection of a door switch sensor 11 opening or an image captured by the sensor 41. If an occupant has boarded the vehicle, the monitoring and control unit 36 ​​proceeds to step ST12. If no occupant has boarded the vehicle, the monitoring and control unit 36 ​​repeats this process. Alternatively, if no occupant has boarded the vehicle, the monitoring and control unit 36 ​​can terminate the process. Figure 8 Monitoring and control.

[0123] Starting from step ST12, the monitoring and control unit 36 ​​begins monitoring the passengers. The monitoring and control unit 36 ​​first identifies the passengers.

[0124] The monitoring and control unit 36 ​​acquires new captured images from the imaging sensor 41, extracts image components of the occupants, and compares them with multiple occupant data registered in the memory 35.

[0125] At this time, the monitoring and control unit 36 ​​can also compare the images based solely on the forward-looking image components from the registered image data of each occupant registered in the memory 35. Forward-looking images typically contain prominent feature points such as the nose of the face. Even when comparing only the forward-looking image components, occupant matching can be determined with high accuracy. Alternatively, the monitoring and control unit 36 ​​can compare the feature points extracted from each image instead of directly comparing the images.

[0126] If the registered shooting data contained in the occupant data registered in the memory 35 is matched with a certain degree of accuracy or higher, the monitoring and control unit 36 ​​can also identify the occupant of the vehicle as the occupant of the registered shooting data.

[0127] In this case, the monitoring and control unit 36 ​​identifies the passenger as the passenger determined by comparing it with multiple passenger data registered in the memory 35.

[0128] If no registered video data matches with a certain degree of accuracy among the multiple occupant data registered in the memory 35, the monitoring and control unit 36 ​​can also identify the occupant as unregistered.

[0129] In step ST13, the monitoring and control unit 36 ​​acquires the occupant data of the identified occupant.

[0130] In step ST14, the monitoring and control unit 36 ​​performs setting processing using the acquired occupant data. If the occupant data includes various setting data for occupants, the monitoring and control unit 36 ​​performs setting processing based on that setting data. The monitoring and control unit 36 ​​outputs the setting data information to each part of the vehicle 1. This performs processing regarding occupant seating positions, initial settings for the presence or absence of driver assistance, driving preference information during automatic driving, information about the server devices used, occupant protection settings, and air conditioning settings.

[0131] In addition, based on the latest captured data, the monitoring and control unit 36 ​​determines, for example, whether a child seat is installed in the front passenger seat 4, and if a child seat is installed, it prevents the airbag from deploying in the front passenger seat 4.

[0132] Starting from step ST15, the monitoring and control unit 36 ​​begins monitoring the identified occupants. The monitoring and control unit 36 ​​first acquires the latest image data from the imaging sensor 41.

[0133] In step ST16, the monitoring and control unit 36 ​​determines the image components of the upper body and head of the occupants contained in the latest captured data. The monitoring and control unit 36 ​​uses the image components of the registered captured data in the memory 35 for identified occupants as a reference to determine the image components of the upper body and head of the occupants contained in the latest captured data. Here, the monitoring and control unit 36 ​​can also infer the data captured from the front of the occupants contained in the current captured data captured by the capturing sensor 41 based on, for example, the difference between data captured from the front of each occupant registered in the memory 35 and data captured from the occupants looking forward, and determine the state of each occupant in the car 1 based on the image components inferred from the data captured from the front of the occupants. The monitoring and control unit 36 ​​can also use, for example, data captured from the front of the occupants to correct lens distortion and orientation in the occupant images contained in the current captured data, and infer the data captured from the front of the occupants based on the current captured data. Furthermore, in the event that registration cannot be identified, the monitoring and control unit 36 ​​can determine the image components of the occupant's upper body and head contained in the latest shooting data, even if the image components of the standard registered shooting data in the memory 35 are used as a reference, without using the image components of the registered shooting data.

[0134] The monitoring and control unit 36 ​​determines factors such as the opening and closing of the eyes, the direction of the gaze, and the orientation of the head as the occupant's condition in the latest captured data. The monitoring and control unit 36 ​​can also detect venous pulse. Furthermore, for example, if the occupant (driver) has their eyes closed, their gaze is not forward, their head is not facing forward, or their pulse is high, the monitoring and control unit 36 ​​determines that the driver is not fit to drive. In other cases, the monitoring and control unit 36 ​​can determine that the driver is fit to drive.

[0135] In this way, the monitoring and control unit 36, acting as a control unit, uses the registered shooting data recorded in the memory 35 as reference data to determine the status of each occupant riding in the car 1.

[0136] In addition, the monitoring and control unit 36 ​​at least determines the opening and closing state of the driver's eyes or the driver's gaze as the state of the driver riding in the car 1. At this time, the monitoring and control unit 36 ​​can use high-quality registration and shooting data as a reference, so it can accurately determine the occupant's gaze direction and other eye expressions based on information such as the opening and closing state of each occupant's eyes, as well as changes in the shooting state of the pupils between the upper and lower eyelids and changes in the shooting state of the sclera.

[0137] In contrast, when making the same judgments without using the registered video data of each occupant, it's difficult to accurately determine eye expressions such as gaze direction, taking into account individual differences like eye size. This can lead to excessive warnings based on the occupant's physical characteristics, causing discomfort for that occupant.

[0138] In step ST17, the monitoring and control unit 36 ​​determines whether control regarding the movement of the vehicle 1 is needed based on the status determination results of the occupants, such as the driver. For example, if it is determined that the driver is not in a state suitable for driving, the monitoring and control unit 36 ​​determines that control is needed and proceeds to step ST18. Otherwise, if all occupants, including the driver, are in a state suitable for movement, the monitoring and control unit 36 ​​determines that control is not needed, skips step ST18, and proceeds to step ST19.

[0139] In step ST18, the monitoring and control unit 36 ​​performs control over the driving of the vehicle 1, etc.

[0140] For example, if the monitoring and control unit 36 ​​determines that the driver's gaze is not forward or that their head is not facing forward, it will issue a warning to the driver. The warning to the driver can also be, for example, a warning display on the LCD device 32 or an audible warning from the speaker device 14. Even if a warning is issued, if the driver does not look forward, i.e., if the monitoring and control unit continuously determines that the driver's gaze is not forward, it can also switch the vehicle 1's driving mode to automatic driving and decelerate the vehicle 1 to a stop. When the vehicle 1 is decelerated and stopped, the monitoring and control unit 36 ​​can also activate hazard warning lights (not shown) and transmit emergency information via the external communication device 18.

[0141] For example, if the system determines that the driver is drowsy or has a high pulse, the monitoring and control unit 36 ​​will issue a warning to the driver or slow down and stop the vehicle 1. When slowing down and stopping the vehicle 1, the monitoring and control unit 36 ​​may also activate hazard warning lights (not shown) or send emergency information via the external communication device 18.

[0142] Furthermore, if driving continues for more than the prescribed period, if the eyes open and close at a prescribed frequency, or if the head is tilted slightly downward, the monitoring and control unit 36 ​​may determine in step ST17 that control is needed and execute outputs such as prompting the driver to rest.

[0143] In step ST19, the monitoring and control unit 36 ​​determines whether to continue occupant monitoring and control. The monitoring and control unit 36 ​​may also determine occupant disembarkation based on, for example, the opening and closing detection of the door switch sensor 11 or the image captured by the sensor 41. If the occupant disembarks, the monitoring and control unit 36 ​​determines that occupant monitoring and control should not continue, and the process proceeds to step ST20. Otherwise, the monitoring and control unit 36 ​​determines that occupant monitoring and control should continue, and the process returns to step ST15. The monitoring and control unit 36 ​​repeats the process from step ST15 to step ST19 until it determines that occupant monitoring and control should not continue.

[0144] In step ST20, the monitoring and control unit 36 ​​updates the occupant data registered in the memory 35 as occupant identification data, processing the process when an occupant disembarks. The monitoring and control unit 36 ​​obtains setting information for occupant disembarkation from various parts of the vehicle 1 and updates the occupant data registered in the memory 35. Thus, the occupant data registered in the memory 35 becomes data tailored to the occupant's preferences. The latest settings regarding the occupant are automatically updated during the next ride.

[0145] Additionally, the monitoring and control unit 36 ​​can temporarily record passenger data for unregistered passengers in the memory 35. This allows for immediate association with the data should the passenger subsequently register.

[0146] Thus, in the occupant monitoring device 15 of this embodiment, the camera module 31 with the shooting sensor 41 is disposed in the center of the vehicle 1 in the width direction of the vehicle and on the back side of the display operation panel 33 of the liquid crystal device 32. Therefore, it is possible to use a single shooting sensor 41 to capture images of the driver and passengers and monitor their respective states. Moreover, since the camera module 31 with the shooting sensor 41 is disposed near the display operation panel 33 of the liquid crystal device 32, in addition to using the current shooting data of the occupants, it also uses data previously captured from the front of the occupants to determine their states. Therefore, it is possible to suppress the influence of the shooting direction and shooting position of the occupants, and to make a more accurate judgment compared to judging the occupants' states solely based on the current shooting data.

[0147] Next, the response control of the liquid crystal device 32, which has such a display operation panel 33 that can be used for occupant monitoring, to occupant operation will be described.

[0148] Figure 9 Is with regard to Figure 3 The flowchart shows the response control related to the operation of the display operation panel 33, which is located in the central part of the vehicle body 2 in the vehicle width direction and can be used for monitoring and control.

[0149] The monitoring and control unit 36 ​​of the occupant monitoring device 15 repeatedly executes commands to detect the occupant's operation of the display operation panel 33 of the LCD device 32. Figure 9 The processing.

[0150] In addition, the ECUs of other devices in the vehicle connected to the occupant monitoring device 15, such as... Figure 2 The ECUs of the other devices shown can also obtain information about the occupant's operation of the display operation panel 33 of the LCD device 32 through the monitoring and control unit 36, and repeat the operation. Figure 9 The processing.

[0151] Moreover, such as Figure 6 As shown, in the display area of ​​the display operation panel 33, there are restriction areas 62 on the left, right and lower sides of the non-display part 33b of the camera module 31. These restriction areas 62 prohibit sliding operations in the direction of the camera module 31.

[0152] In step ST31, the monitoring and control unit 36 ​​determines whether the occupant has operated the display operation panel 33 of the LCD device 32. If no occupant operation is detected, the monitoring and control unit 36 ​​continues the current process. Then, when occupant operation is detected, the monitoring and control unit 36 ​​causes the process to proceed to step ST32.

[0153] In step ST32, the monitoring and control unit 36, acting as the operating environment, determines whether the vehicle 1 is in motion. If the vehicle 1 is in motion, the monitoring and control unit 36 ​​returns the process to step ST31 to prevent operation of the display control panel 33. If the vehicle 1 is not in motion, the monitoring and control unit 36 ​​proceeds to step ST33.

[0154] Here, the monitoring and control unit 36, as the operating environment, can also determine whether the driver has performed an operation. In this case, if the driver performs an operation, the monitoring and control unit 36 ​​can return the processing to step ST31, and if a passenger performs an operation, the processing can proceed to step ST33.

[0155] In step ST33, the monitoring and control unit 36 ​​determines the location of the operation performed by the occupant on the display area of ​​the display operation panel 33 of the liquid crystal device 32. For example, Figure 6 As shown, the display area of ​​the operation panel 33 includes a main display area AM and a secondary display area AS. The secondary display area AS can be set on the left and right sides of the non-display unit 33b. The main display area AM can display operation objects 65 such as multiple operation buttons for various settings. The secondary display area AS can display operation objects 65 related to various states of the vehicle 1 at the current time.

[0156] The LCD device 32 can display multiple operable objects 65 on the operation screen. The monitoring and control unit 36 ​​can determine, in determining the operation position, whether a particular operable object 65 has been operated.

[0157] In step ST34, the monitoring and control unit 36 ​​determines the occupant's operation. The occupant can perform actions such as clicking (pressing), sliding (moving up, down, left, or right) on the object 65 displayed on the display area of ​​the LCD device 32's display panel 33. The monitoring and control unit 36 ​​can use the operation position or the position of the object 65 determined in step ST33 as a reference to determine whether a click or slide operation has been performed, thus using this as the occupant's operation. Additionally, the monitoring and control unit 36 ​​can also determine the operation direction of a slide operation that follows the surface of the display panel 33 towards the camera's mounting position.

[0158] In step ST35, the monitoring and control unit 36 ​​determines whether the operation is a sliding operation following the surface of the display operation panel 33. If it is determined to be a sliding operation in step ST35, the monitoring and control unit 36 ​​proceeds to step ST36. If it is not determined to be a sliding operation in step ST35, the monitoring and control unit 36 ​​proceeds to step ST38.

[0159] In step ST36, the monitoring and control unit 36 ​​determines whether the sliding operation is an operation in the direction of the restricted sliding operation within the restricted sliding operation area. For example... Figure 6 As shown, in the display area of ​​the display operation panel 33, restriction areas 62 are provided on the left, right, and bottom sides of the non-display section 33b where the camera module 31 is located, prohibiting sliding operations towards the camera module 31. The monitoring and control unit 36 ​​determines whether the sliding operation is towards the camera module 31. If the sliding operation is towards the camera module 31, the monitoring and control unit 36 ​​proceeds to step ST37. Otherwise, the monitoring and control unit 36 ​​proceeds to step ST38.

[0160] Here, as Figure 6 For example, for a portion of the operation object 65 that overlaps with the restricted area 62, it can be judged in the same way as the operation object 65 contained in the restricted area 62, or it can be judged in the same way as the operation object 65 not contained in the restricted area 62. For example, if a sufficiently wide restricted area 62 is provided around the camera module 31, it can also be judged in the same way as the operation object 65 not contained in the restricted area 62.

[0161] Furthermore, the setting of restricted area 62 is not limited to Figure 6 The pattern.

[0162] In step ST37, the monitoring and control unit 36 ​​does not perform response control corresponding to the occupant's operating position and operation content. If the occupant's operation is a sliding operation involving movement towards the image sensor 41 in the peripheral area of ​​the display area, the monitoring and control unit 36 ​​does not perform control corresponding to the operation content. The monitoring and control unit 36 ​​may also output a warning window indicating that response control is not performed to the operation to the liquid crystal device 32. Then, the monitoring and control unit 36 ​​terminates this control.

[0163] In step ST38, the monitoring and control unit 36 ​​performs response control corresponding to the occupant's operating position and operation content. For example, if the occupant's operation is a click operation on a displayed object, the monitoring and control unit 36 ​​performs a response control pre-defined to that operation. The monitoring and control unit 36 ​​may also display, for example, an execution screen regarding the operated display object on the liquid crystal device 32.

[0164] For example, if the occupant's operation is a swipe operation on the displayed object, the monitoring and control unit 36 ​​executes a response control pre-defined to that operation. The monitoring and control unit 36 ​​can, for example, display a setting screen on the operated display object on the liquid crystal device 32.

[0165] Thus, when the monitoring and control unit 36 ​​performs a sliding operation on an object 65 displayed around the image sensor 41 that follows the setting position of the image sensor 41, it does not execute control corresponding to the sliding operation. Conversely, when a sliding operation is performed on an object 65 outside the periphery of the image sensor 41, or even if the object 65 is outside the periphery of the image sensor 41, the monitoring and control unit 36 ​​executes control corresponding to the sliding operation.

[0166] As described above, in this embodiment, the monitoring and control unit 36 ​​does not perform control corresponding to the determined operation content when the occupant's operation involves at least a movement towards the camera sensor 41. Therefore, the occupant cannot operate the LCD device 32 displaying the occupant's operation screen while moving towards the camera sensor 41. As a result, in this embodiment, the field of view of the camera sensor 41 is less likely to be obstructed by the occupant's fingers, hands, wrists, elbows, etc., during operation. Furthermore, if the occupant is not facing the camera sensor 41, there are no restrictions on the operation accompanying the movement; therefore, the occupant can perform the operation accompanying the movement on the operation screen displayed on the LCD device 32.

[0167] Thus, in this embodiment, the sliding operation accompanying the occupant's movements is not uniformly restricted, thereby achieving a high balance between occupant ease of use and occupant status monitoring.

[0168] Thus, in the automobile 1, it is required to achieve a good balance between the LCD device 32 used for occupant monitoring, occupant usability, and occupant status monitoring.

[0169] The above embodiments are examples of preferred embodiments of the present invention, but the present invention is not limited thereto, and various modifications or alterations can be made without departing from the spirit of the invention.

[0170] For example, in the above embodiment, the liquid crystal device 32 is arranged longitudinally.

[0171] In addition, LCD devices 32 can also be configured horizontally.

[0172] Figure 10 It is about Figure 6 An explanatory diagram of a first modified example of an occupant monitoring device 15, comprising an LCD device 32 and a camera module 31, located in the central part of the vehicle body 2 in the width direction.

[0173] Figure 10 The display area of ​​the LCD device 32 is horizontal.

[0174] Additionally, the camera module 31, which includes the image sensor 41, is located on the back side of the display operation panel 33, on the back side of the non-display portion 33b formed above the display area of ​​the liquid crystal device 32. The camera module 31 is positioned inside the holding member 61 in a manner that prevents it from protruding outwards from the holding members 61 at the four corners of the display operation panel 33.

[0175] Even in this case, by setting restriction areas 62 on the left, right, and bottom sides of the non-display portion 33b where the camera module 31 is configured in the display area of ​​the display operation panel 33, which prohibit sliding operations in the direction toward the camera module 31, the same effect as the above-described embodiment can be obtained.

[0176] Figure 11 It is about Figure 6 The illustration shows a second modified example of the occupant monitoring device 15, which is located in the central part of the vehicle body 2 in the width direction, and includes an LCD device 32 and a camera module 31.

[0177] Figure 11 The display area of ​​the LCD device 32 is horizontal.

[0178] Furthermore, the camera module 31, which has a shooting sensor 41, is configured on the back side of the display operation panel 33, forming the upper side of the quadrilateral display area of ​​the liquid crystal device 32. The camera module 31 is positioned inside the holding member 61 so that it does not protrude outwards from the holding members 61 at the four corners of the display operation panel 33.

[0179] Even in this case, by setting a restriction area 62 that prohibits sliding operations toward the camera module 31 at the upper center of the display area of ​​the display operation panel 33, adjacent to the camera module 31, the same effect as the above-described embodiment can be obtained.

[0180] In the above embodiment, the display operation panel 33 of the liquid crystal device 32 and the image sensor 41 are centrally located in the middle part of the vehicle 1 in the width direction. The image sensor 41 has a wide-angle lens 42 that can capture the head view of multiple occupants sitting in the front row of the vehicle 1 when centrally located in the middle part of the vehicle 1 in the width direction. The monitoring and control unit 36 ​​determines, at least the open / closed state of the occupants' eyes or their line of sight, as a state of occupants sitting in the vehicle 1.

[0181] The occupant monitoring device may not fall into any of these categories. For example, the occupant monitoring device may only monitor the driver. Even in this case, it is expected that the accuracy of the monitoring judgment can be improved by, for example, configuring the image sensor 41 on the back side of the display operation panel 33 of the liquid crystal device 32, inside the outer circle of the image display area, or by the monitoring control unit 36 ​​using the registered image data recorded in the memory 35 as reference data to determine the state of the occupant in the car 1.

[0182] Symbol Explanation

[0183] 1…Automobile (vehicle), 2…Body, 3…Cabin, 4…Seat, 5…Instrument panel, 6…Center console, 7…Steering, 10…Control device, 11…Door switch sensor, 12…Vehicle speed sensor, 13…Parking sensor, 14…Speaker equipment, 15…Occupant monitoring device, 16…Position setting device, 17…Driver assistance device, 18…External communication device, 19…Occupant protection device, 20…Air conditioning device, 21…In-vehicle network, 31…Camera module, 32…LCD device (display component), 33…Display operation panel, 33a…Rectangular frame, 33b…Non-display 34…Input / output device, 35…Memory, 36…Monitoring and control unit, 41…Image sensor (image capture component), 42…Wide-angle image capture lens, 43…First LED, 44…First projection lens, 45…Second LED, 46…Second projection lens, 51…First occupant data, 52…Second occupant data, 61…Holding component, 62…Restricted area, 65…Operating object, AM…Main display area (display area), AS…Sub-display area (display area), Y0…Center position in the vehicle width direction, Yd…Center position in the vehicle width direction of the liquid crystal device.

Claims

1. A vehicle occupant monitoring device, comprising: The display component shows the operating screen for the vehicle's occupants; The control unit performs control corresponding to the operation content, which is determined to be the operation performed by the occupant on the operation screen of the display unit; as well as The camera component is capable of capturing images of the occupants of the vehicle. Specifically, if the occupant's operation involves at least a movement towards the camera component, the control unit will not perform control corresponding to the determined operation. The shooting component and the display area of ​​the display component are arranged adjacent to each other. If the occupant's operation involves moving towards the camera component while the display area is located in the periphery of the camera component, the control unit does not perform control corresponding to the determined operation.

2. The occupant monitoring device for a vehicle according to claim 1, wherein, The display component has a panel larger than the display area of ​​the operation screen, serving as the panel for displaying the operation screen. The camera component is located on the back side of the panel of the display component.

3. The occupant monitoring device for a vehicle according to claim 1, wherein, If the operation performed by the control unit accompanying the movement toward the shooting component is an operation in which the surface of the display component moves toward the shooting component, then the control unit does not perform control corresponding to the determined operation content.

4. The occupant monitoring device for a vehicle according to claim 1, wherein, The display component shows multiple operable objects that can be swiped in the operation screen. When the control unit performs a sliding operation towards the camera component for the object being operated on and displayed around the camera component, the control unit does not perform the control corresponding to the determined sliding operation.

5. The occupant monitoring device for a vehicle according to claim 3 or 4, wherein, When the control unit is oriented towards the camera unit in the direction of operation following the surface of the display unit, it does not perform control corresponding to the determined operation content. When the direction of operation following the surface of the display component is different from the direction of the shooting component, control is performed according to the determined operation content.

6. The occupant monitoring device for a vehicle according to claim 1, wherein, If the occupant's operation is at least an operation performed while the vehicle is in motion and the occupant is moving towards the camera, the control unit will not perform control corresponding to the determined operation.

7. The occupant monitoring device for a vehicle according to claim 1, wherein, The display component and the camera component are located in the central part of the vehicle in the width direction. The control unit monitors the status of the vehicle's occupants based on the images captured by the camera.

Images