Vehicle occupant monitoring device

Abstract

To provide an occupant monitoring device provided in a vehicle in order to monitor occupants, in which arrangements of a photographing member and the like are improved.SOLUTION: An occupant monitoring device 15 of a vehicle 1 comprises: a user interface module 38 provided with a display member 32 for displaying a screen for occupants in the vehicle 1 and at least a photographing member 41 for monitoring the occupants in the vehicle 1; an actuator 37 that is driven to change a direction of the user interface module 38; and a control part 36 that executes control by which the occupants in the vehicle 1 are identified or registered. The control part 36 controls the actuator 37 so that a direction of the photographing member 41 approaches a direction of an occupant who newly gets on the vehicle 1, so as to change the direction of the user interface module 38; and executes identification control or registration control by which the occupants in the vehicle 1 are monitored, using data on images photographed by the photographing member 41, with the direction of the user interface module 38 changed.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to a vehicle occupant monitoring device. [Background technology]

[0002] Liquid crystal devices having touch panels are beginning to be adopted as user interfaces in vehicles, for example as display members for performing vehicle settings, etc. Liquid crystal devices having touch panels display operation objects such as buttons. In this case, a vehicle occupant operates an operation object, such as a button, displayed on a liquid crystal device having a touch panel, and a control unit of the vehicle executes control according to the operation content. A liquid crystal device having a touch panel, for example in a mobile terminal, allows a variety of operations, such as a swipe operation by moving the screen as if tracing the surface of the liquid crystal device, in addition to a click operation by touching an object. Passengers are accustomed to these operations when operating a mobile terminal, etc.

[0003] In addition, it is considered that in the future, there will be a demand for vehicles to monitor the state of occupants during driving assistance or autonomous driving (Patent Documents 1 and 2). In this case, in order to monitor the vehicle occupants, an imaging device is provided, for example, on the dashboard in front of the occupants, and the condition of the occupants is determined and monitored based on images captured by the imaging device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-014359 A [Patent Document 2] JP 2019-014360 A Summary of the Invention [Problem to be solved by the invention]

[0005] In order to meet these various requirements, it is conceivable that the imaging device is provided near a liquid crystal device having a touch panel provided in the vehicle. This allows the imaging device to capture an image of an occupant from the front who is looking at a predetermined display on the liquid crystal device. If there is imaging data of an occupant previously captured from the front, for example, it is possible to more accurately determine the state of each occupant, such as the driver, by comparing the imaging data from the front with the current imaging data. In addition, the liquid crystal device and the imaging device may be arranged, for example, in the center of the vehicle in the vehicle width direction. This allows not only the driver but also passengers to view and operate the liquid crystal device having a touch panel. In this case, it is not desirable to tilt the liquid crystal device and the imaging device toward the driver in the center of the vehicle width direction, since this would hinder the convenience of the passengers. On the other hand, if the liquid crystal display and the imaging device are fixedly installed facing backward in the center of the vehicle width direction, the driver and passengers will be captured in the peripheral parts of the image, where distortion is likely to occur, rather than in the center of the image in the imaging data of the imaging device. Then, when performing personal recognition of the driver and other passengers based only on images with large distortion, it may be difficult to capture detailed facial features. There is a possibility that the performance of personal recognition may be degraded.

[0006] In such a passenger monitoring device provided in a vehicle to monitor passengers, there is a demand for improvements in the arrangement of the imaging member and the like. [Means for solving the problem]

[0007] The vehicle occupant monitoring device according to the present invention includes a user interface module provided with an imaging member for monitoring at least a vehicle occupant as well as a display member for displaying a screen to a vehicle occupant, an actuator for driving the user interface module to change the orientation of the user interface module, and a control unit for executing control for identifying or registering an occupant riding in the vehicle, the control unit comprising: a vehicle seat that detects a new occupant getting on the vehicle; a seating position of the detected new occupant in the vehicle; and a seating position of the detected new occupant that is in a stable posture in the seating position; and after that, a relative direction of a face or head of the detected new occupant is obtained, and an orientation of the imaging member is adjusted to the obtained relative direction; The actuator is controlled to change the orientation of the user interface module, and the imaging data of the imaging member in the changed orientation of the user interface module is used to perform identification control for identifying individual occupants of the vehicle or registration control for such identification.

[0009] Preferably, the user interface module is movably provided in a central portion of the vehicle in a vehicle width direction, and the control unit monitors the state of an occupant of the vehicle based on an image captured by the imaging member.

[0010] Preferably, the imaging member provided in the user interface module captures wide-angle images of the interior of the vehicle while facing from the center of the vehicle's width direction toward the rear of the vehicle, in order to monitor multiple occupants in the vehicle.

[0011] Preferably, after executing identification control or registration control, the control unit controls the actuator to return the orientation of the user interface module, and monitors occupants in the vehicle using the imaging data of the imaging member in the returned orientation.

[0012] Preferably, when monitoring occupants in the vehicle, the control unit uses imaging data of the occupants acquired during identification control or registration control together with imaging data of the imaging member in the returned orientation.

[0013] Preferably, the display member and the imaging member are movably provided in a central portion of the vehicle in a vehicle width direction, and the control unit monitors the condition of the occupants of the vehicle based on an image captured by the imaging member. Effect of the Invention

[0014] In the present invention, an imaging member for monitoring a vehicle occupant is provided in a user interface module together with a display member so as to be close to a display member for displaying a screen to the vehicle occupant, and a control unit that executes control for identifying or registering an occupant who rides in the vehicle controls an actuator to change the orientation of the user interface module so that the orientation of the imaging member approaches the direction of a new occupant who will ride in the vehicle. In particular, in a state where the imaging member of the user interface module is in a home position where a plurality of occupants of the vehicle can be monitored, after detecting that a new occupant will ride in the vehicle, determining the riding position of the new occupant who will ride in the vehicle, and determining that the new occupant has a stable posture at the riding position, a relative direction of the face or head of the new occupant is acquired, and the actuator is controlled so that the orientation of the imaging member becomes the acquired relative direction, thereby making it possible to orient the imaging member toward the face or head of the new occupant who will ride in the vehicle. Then, the control unit can execute identification control or registration control for monitoring an occupant who rides in the vehicle based on an image in which distortion of the occupant such as the driver is suppressed, using imaging data of the imaging member in a state where the orientation of the user interface module is changed. As a result, in the present invention, the arrangement of the imaging member and other components in the occupant monitoring device provided in the vehicle for monitoring the occupants can be made suitable for monitoring the occupants. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is an explanatory diagram of an automobile to which an occupant monitoring device according to an embodiment of the present invention is applied. [Diagram 2] FIG. 2 is an explanatory diagram of the control device of the automobile of FIG. [Diagram 3]FIG. 3 is an explanatory diagram of the vehicle occupant monitoring device of FIG. [Figure 4] FIG. 4 is an explanatory diagram of the normal interior layout of a liquid crystal device having a display operation panel and a camera module having an image sensor in the occupant monitoring device of FIG. [Diagram 5] FIG. 5 is an explanatory diagram of an image capturing state of a driver who is an occupant of a vehicle. [Figure 6] FIG. 6 is a detailed explanatory diagram of a liquid crystal device and a camera module of an occupant monitoring device provided in the center portion of the vehicle body in the vehicle width direction. [Figure 7] FIG. 7 is a flowchart of the monitoring control of the occupant by the monitoring control unit of FIG. [Figure 8] FIG. 8 is a flowchart of the new registration process of FIG. [Figure 9] FIG. 9 is an explanatory diagram of a modification of the liquid crystal device and the camera module of the passenger monitoring device provided in the center portion in the width direction of the vehicle body of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0017] FIG. 1 is an explanatory diagram of an automobile 1 to which an occupant monitoring device 15 according to an embodiment of the present invention is applied. The automobile 1 is an example of a vehicle. The automobile 1 may be one that uses an internal combustion engine as a power source for running, one that uses stored electric power in a battery as a power source for running, or one that combines these. A vehicle interior 3 is formed in a vehicle body 2 of an automobile 1 in Fig. 1. A plurality of front row seats 4 for a driver's seat and a passenger seat, and a long rear row seat 4 are provided in the vehicle interior 3. Passengers including the driver are seated in the seats 4. A dashboard 5 is provided in the front part of the vehicle interior 3, which is in front of the plurality of front row seats 4, so as to extend along the vehicle width direction of the vehicle body 2.

[0018] FIG. 2 is an explanatory diagram of the control device 10 of the automobile 1 of FIG. The control device 10 in Figure 2 has a door opening / closing sensor 11, a vehicle speed sensor 12, a park sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driving assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 to which these are connected.

[0019] The in-vehicle network 21 may be a wired communication network for the automobile 1, for example, compliant with CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 may be a communication network such as a LAN, or a combination of these. A part of the in-vehicle network 21 may include a wireless communication network.

[0020] The door opening / closing sensor 11 detects whether the door of the automobile 1 is opened or closed. The vehicle speed sensor 12 detects the speed of the traveling automobile 1. The vehicle speed sensor 12 may detect a stopped state. The park sensor 13 detects a parking state in which the automobile 1 is kept stopped. The park sensor 13 may detect, for example, that a shift lever (not shown) is operated to a parking position or that a parking lever (not shown) is operated to a brake position as the parking state.

[0021] The occupant monitoring device 15 executes a recognition process for an occupant in the vehicle compartment 3 when, for example, the door opening / closing sensor 11 detects the opening or closing of a door. When the occupant monitoring device 15 recognizes multiple occupants, it may monitor each of the multiple occupants individually. The occupant monitoring device 15 may output information about each recognized occupant and information based on the monitoring to each part of the control device 10 via the vehicle network. In particular, the occupant monitoring device 15 recognizes, for example, an occupant seated in the driver's seat 4 as the driver, and monitors the recognized driver for looking away or falling asleep. When the driver is in a predetermined state, the occupant monitoring device 15 executes control for calling the driver's attention and control for preventing danger. At this time, the occupant monitoring device 15 may output information about the driver, information for calling the driver's attention, and information for preventing danger to each part of the control device 10 via the vehicle network.

[0022] The speaker device 14 outputs voice, warning sounds, etc. The speaker device 14 may output warnings from an occupant monitoring device 15 to the driver and other occupants. The position setting device 16 adjusts, for example, the front-rear and up-down positions of the seat 4 and the angle of the backrest, the front-rear and up-down positions and angles of the handlebars, and the front-rear and up-down positions and angles of various pedals. The position setting device 16 changes the seat position based on the occupant information output by the occupant monitoring device 15. The driving assistance device 17 assists the driver in manually driving the automobile 1, and assists the driver in automatically driving the automobile 1. The driving assistance device 17 controls the acceleration, deceleration, stopping, and steering of the automobile 1. Based on the driver information output by the occupant monitoring device 15, the driving assistance device 17 executes driving assistance according to the driver. The exterior communication device 18 establishes a wireless communication path with, for example, a base station of a public wireless communication network, a base station of a commercial wireless communication network, or a base station for advanced traffic information, and performs data communication using the established wireless communication path. The exterior communication device 18 may perform two-way data communication with, for example, a server device that supports autonomous driving. The exterior communication device 18 may transmit information on occupants including the driver output by the occupant monitoring device 15 to the server device, for example, as emergency support information. The occupant protection device 19 executes control to protect the occupant when it detects or predicts a collision of the automobile 1. The occupant protection device 19 protects the occupant seated in the seat 4 by, for example, deploying an airbag or applying tension to a seat belt (not shown). The occupant protection device 19 may execute occupant protection for the occupant according to the occupant information output by the occupant monitoring device 15. The air conditioner 20 controls the temperature and oxygen concentration in the vehicle interior 3. The air conditioner 20 supplies, for example, cooled air or superheated air to the vehicle interior 3, thereby adjusting the vehicle interior 3 to a set temperature. The air conditioner 20 may perform air conditioning in accordance with information about the occupants output by the occupant monitoring device 15. The passengers can ride comfortably under settings suited to them, for example, by the control based on the passenger information output by the passenger monitoring device 15. The driver can concentrate on driving the automobile 1, for example.

[0023] Fig. 3 is an explanatory diagram of the occupant monitoring device 15 of the automobile 1 of Fig. 2. The occupant monitoring device 15 of Fig. 3 monitors not only the driver but also a plurality of occupants seated, for example, in a plurality of seats 4 in the front row. The occupant monitoring device 15 of Fig. 3 has a camera module 31, a liquid crystal device 32 having a display operation panel 33, an input / output device 34, a memory 35, an actuator 37, and a monitoring control unit 36 ​​to which these are connected.

[0024] The input / output device 34 is connected to the in-vehicle network 21. The input / output device 34 inputs and outputs data between the input / output device 34 and other units provided in the automobile 1 through the in-vehicle network 21.

[0025] The liquid crystal device 32 displays an image on a display surface of the display operation panel 33. As a display member, the liquid crystal device 32 displays a screen to be visually recognized by each occupant in the automobile 1. The display screen includes, for example, an operation screen operated by the occupant. The display operation panel 33 is a transparent or semi-transparent panel that is superimposed on the display surface of the liquid crystal device 32. The display operation panel 33 superimposed on the display surface of the liquid crystal device 32 constitutes a display member. The display operation panel 33 detects an operation by the occupant on the display surface of the liquid crystal device 32. The display operation panel 33 may output the operation position of the occupant on the display surface of the liquid crystal device 32 to the monitoring control unit 36.

[0026] The camera module 31 is for capturing images of multiple passengers seated in multiple front row seats 4. The camera module 31 has an image sensor 41, an imaging wide-angle lens 42, a first LED 43, a first light-projecting lens 44, a second LED 45, and a second light-projecting lens 46.

[0027] The imaging sensor 41 is a semiconductor optical sensor such as a CCD or a CMOS sensor. The imaging sensor 41 may have a substantially rectangular light receiving surface on which a plurality of light receiving elements are arranged. The imaging sensor 41 may output imaging data including a captured image to the monitoring control unit 36. The wide-angle imaging lens 42 is provided over the imaging sensor 41. The wide-angle imaging lens 42 may be made up of a plurality of optical lenses so as to suppress distortion in the peripheral portion of the image. The imaging sensor 41 and the imaging wide-angle lens 42, as imaging members, are capable of capturing images of the occupants of the automobile 1 so as to enable monitoring of the status of the multiple occupants riding in the automobile 1. The wide-angle imaging lens 42 may be a lens capable of capturing images of the upper bodies or heads of multiple occupants seated in multiple seats 4 in the front row when the imaging sensor 41 is provided in the center of the dashboard 5 in the vehicle width direction, for example.

[0028] The first LED 43 and the second LED 45 may be semiconductor light emitting elements. The first LED 43 and the second LED 45, as light emitting members, project light toward the occupant of the automobile 1 that is the image pickup target of the image pickup sensor 41. The first LED 43 and the second LED 45 may project infrared light, for example. In this case, the image pickup sensor 41 outputs imaging data including an image captured by infrared light to the monitoring control unit 36. The first light emitting lens 44 is provided overlapping the first LED 43. The first light emitting lens 44 may irradiate the light of the first LED 43 mainly toward the driver seated in the seat 4 as the driver's seat. The second light emitting lens 46 is provided overlapping the second LED 45. The second light emitting lens 46 may irradiate the light of the second LED 45 mainly toward the occupant seated in the seat 4 as the passenger seat. Second light projecting lens 46 may project diffused light toward a driver seated in seat 4 serving as a driver's seat as well as an occupant seated in seat 4 serving as a passenger seat.

[0029] The actuator 37 drives the liquid crystal device 32 having the display operation panel 33, and the camera module 31. The camera module 31 is disposed integrally on the rear side of the display operation panel 33 of the liquid crystal device 32, and constitutes a user interface module 38. The actuator 37 may drive the liquid crystal device 32 having the display operation panel 33, and the camera module 31, by driving the user interface module 38 to change its orientation.

[0030] The memory 35 stores programs and data. The memory 35 may be composed of a non-volatile memory and a volatile memory. Examples of the non-volatile memory include a HDD, an SSD, and an EEPROM. Examples of the volatile memory include a RAM.

[0031] Data on multiple occupants registered in the automobile 1 may be managed and recorded for each occupant in the memory 35 of the occupant monitoring device 15. First occupant data 51 for a first occupant and second occupant data 52 for a second occupant are shown in Fig. 3. A database is formed in the memory 35 from the multiple occupant data.

[0032] Each occupant data such as the first occupant data 51 and the second occupant data 52 may include identification information unique to each occupant, for example, registered image data obtained by capturing an image of the occupant's head or eyes by the image sensor 41, and various setting data set by the occupant. The setting data may include, for example, information such as the seat position of the occupant, an initial setting for whether or not to use driving assistance, driving preference information in autonomous driving, information on the server device to be used, occupant protection settings, and air conditioning settings. In this way, the memory 35, as a recording member, can record image data captured by the image sensor 41 from the front of each occupant while a specified screen is being displayed on the liquid crystal device 32, as registered image data for each occupant. Additionally, the memory 35 may record occupant data for unregistered generic occupants.

[0033] The monitoring control unit 36 ​​may be, for example, an ECU, a CPU, or other microcomputer. The monitoring control unit 36 ​​reads and executes a program from the memory 35. This realizes the monitoring control unit 36 ​​as a control unit of the occupant monitoring device 15. The monitoring control unit 36 ​​executes control for identifying or registering an occupant riding in the automobile 1. The monitoring control unit 36 ​​may execute control according to the operation content determined as an operation by the occupant on the operation screen of the liquid crystal device 32, for example. Furthermore, when an occupant gets in the automobile, the monitoring control unit 36 ​​executes monitoring control of the occupant's state. At this time, the monitoring control unit 36 ​​may operate the actuator 37 as necessary. The monitoring control unit 36 ​​monitors the state of the multiple occupants of the automobile 1 based on the image data captured by the imaging sensor 41. For occupant monitoring, the monitoring control unit 36 ​​executes, for example, a registration process and a monitoring process for the occupants who get in the automobile 1. In the monitoring process, the monitoring control unit 36 ​​may identify the occupant who gets in the automobile 1 and execute a monitoring process for each occupant. In particular, for at least the occupant who is the driver, the monitoring control unit 36, as a judgment unit, judges the state of each occupant, such as distraction, dozing, or emergency, using the registered image data of each occupant registered in the memory 35 as reference data for judgment.

[0034] FIG. 4 is an explanatory diagram of the normal interior arrangement of the liquid crystal device 32 having the display operation panel 33 and the camera module 31 having the image sensor 41 in the occupant monitoring device 15 of FIG.

[0035] FIG. 4(A) shows a dashboard 5 and a center console 6 provided at the front of a vehicle interior 3. 1, the liquid crystal device 32 and the camera module 31 of this embodiment are disposed vertically from the dashboard 5 to the center console 6 in the center portion of the vehicle 1 in the vehicle width direction toward the rear of the vehicle 1. This allows the image sensor 41 of the camera module 31 to capture an image of the entire interior of the vehicle at a wide angle, including the driver and passengers. The image sensor 41 can capture an image of the driver's upper body including the head and the passenger's upper body including the head into a single piece of image data. The liquid crystal device 32 is disposed in the center in the vehicle width direction of the automobile 1 so that the vehicle width direction central position Yd of the display screen coincides with the vehicle width direction central position Y0 of the automobile 1. The display screen of the liquid crystal device 32 can ensure visibility for the driver while ensuring visibility for passengers.

[0036] FIG. 4B is an explanatory diagram of an image capturing range inside the vehicle captured by the camera module 31 having the image capturing sensor 41 and the wide-angle image capturing lens 42 provided toward the rear in the center portion of the vehicle 1 in the vehicle width direction. A wide-angle imaging lens 42 is placed over the imaging sensor 41. This allows the imaging sensor 41 to capture an image of the entire interior of the vehicle at a wide angle, including the driver and passengers, as shown in the imaging range in Fig. 4(B). The imaging sensor 41 is disposed in the center of the vehicle 1 in the vehicle width direction, and can capture images of the heads of the driver and passengers seated in the multiple front row seats 4. However, when the imaging wide-angle lens 42 is combined with the imaging sensor 41 to image multiple occupants, the image in the center of the imaging range is a high-quality area with less distortion, while the peripheral area of ​​the imaging range is a distortion-generating area where distortion occurs compared to the center. Therefore, if an occupant sits toward the right or left edge in the vehicle width direction, it tends to be difficult to image the occupant's head without distortion. When using the open / closed state of the occupant's eyes to judge, there is a high possibility that the state cannot be judged correctly based on the imaging data alone.

[0037] Unlike FIG. 4, the liquid crystal device 32 may be offset such that the center position Yd of the display screen in the vehicle width direction is slightly shifted from the center position Y0 of the automobile 1 in the vehicle width direction toward the opposite side of the driver. In this case, the camera module 31 may be offset in the same manner as the liquid crystal device 32, or the center position of the screen of the image sensor 41 may be located between the center position Yd of the display screen of the liquid crystal device 32 in the vehicle width direction and the center position Y0 of the automobile 1 in the vehicle width direction. In addition, the liquid crystal device 32 and the image sensor 41 may be tilted toward the driver in the offset position with respect to the front-rear direction of the vehicle body 2. As a result, the high image quality area in the center of the image captured by the image sensor 41 is closer to the driver. The driver is more likely to be captured in the high image quality area in the center of the image captured by the image sensor 41, and less likely to be captured in the distortion-generating area in the peripheral portion. It is expected that the driver will be captured in an image with reduced distortion.

[0038] FIG. 5 is an explanatory diagram of an image capturing state of a driver who is an occupant of the automobile 1. In FIG.

[0039] Fig. 5(A) is a front view of the head of the first driver, and Fig. 5(B) is a perspective view of the head of the first driver of Fig. 5(A). When the driver faces forward of the automobile 1, the image sensor 41 provided in the center of the automobile 1 in the vehicle width direction images the driver's head not from the front as in FIG. 5(A) but from an oblique direction as in FIG. 5(B).

[0040] Fig. 5(C) is a front view of the head of the second driver, and Fig. 5(D) is a perspective view of the head of the second driver of Fig. 5(C). When the driver faces forward of the automobile 1, the image sensor 41 provided in the center of the automobile 1 in the vehicle width direction images the driver's head from an oblique direction as shown in FIG. 5(D) rather than from the front as shown in FIG. 5(C).

[0041] In the image data of FIG. 5(D), unlike the image data of FIG. 5(B), the driver's right eye is hidden by the driver's high nose and only the pupil of the eye is visible. In this case, it is not easy for the monitoring control unit 36 ​​to determine whether the driver's right eye is open or closed from only the imaging data of Fig. 5(D). In particular, when the image is not free of distortion as in Fig. 5(D) but is captured as a distorted image, it is extremely difficult for the monitoring control unit 36 ​​to determine whether the driver's right eye is open or closed from only the imaging data of Fig. 5(D). In addition, since the image data of FIG. 5(D) does not include the image components of the white of the driver's right eye to a degree that allows the state to be determined, it is difficult for the monitoring control unit 36 ​​to extract the state of the right eye based only on the image data of FIG. 5(D). If the monitoring control unit 36 ​​cannot identify the right eye, it may not be able to determine the direction of the driver's head as a result. For example, when identifying the imaging area of ​​the head in the image based on the positional relationship between both eyes and the nose and the positional relationship between both eyes and the mouth, the monitoring control unit 36 ​​cannot identify the direction of the head of the driver in FIG. 5(D) based only on the image data of FIG. 5(D). If there is no reference data for the driver's head, the monitoring control unit 36 ​​is more likely to be unable to correctly estimate the state of the driver if the image components do not include the driver's right eye. When the imaging sensor 41 of the camera module 31 is provided in the center of the automobile 1 in the vehicle width direction, the situation as shown in FIG. 5(D) is likely to occur.

[0042] Thus, there are individual differences among passengers, such as some with large eyes, some with small eyes, some with eyes that protrude forward, and some with eyes that are set back. The actual position of the driver's head may move not only in the direction of the field of view in the vehicle width direction, but also in the vertical direction. Considering the occurrence of these movements, it is not easy to correctly estimate the state of the driver's head and eyes based only on the current image data of the image sensor 41.

[0043] Fig. 6 is a detailed explanatory diagram of the liquid crystal device 32 and the camera module 31 of the occupant monitoring device 15 provided in the central portion in the vehicle width direction of the vehicle body 2. Fig. 6(A) is a front view seen from the rear side of the vehicle compartment 3. In Fig. 6, the central position of the display operation panel 33 of the liquid crystal device 32 coincides with the central position Y0 in the vehicle width direction of the vehicle body 2. The liquid crystal device 32 and the camera module 31 are integrated as a user interface module 38.

[0044] The liquid crystal device 32 has a display area for displaying a screen that is not a rectangular display surface like a typical monitor, but has a generally concave shape with the center of the upper edge of the rectangle cut out. The camera module 31 having the image sensor 41 is disposed on the rear side of the display operation panel 33 , behind a recess formed above the display area of ​​the liquid crystal device 32 . The central positions of the imaging sensor 41 and the imaging wide-angle lens 42 in the vehicle width direction coincide with the central position of the display operation panel 33 of the liquid crystal device 32 and the central position Y0 of the vehicle body 2 in the vehicle width direction. This makes it appear as if the imaging sensor 41 of the camera module 31 is provided in the liquid crystal device 32 itself. As a result, the imaging sensor 41 of the camera module 31 can capture an image of the occupant from the back side of the liquid crystal device 32 having the display operation panel 33. In addition, in the camera module 31, the imaging sensor 41 and the imaging wide-angle lens 42 are disposed in the center portion in the vehicle width direction. The first LED 43 and the first light-projecting lens 44 are disposed in the end portion on the passenger seat side. The second LED 45 and the second light-projecting lens 46 are disposed in the end portion on the driver side. This allows the camera module 31 to project light and capture images without being blocked by an object such as the steering wheel 7, even if there is an object between the driver and the dashboard 5. In addition, when the camera module 31 is thus provided on the back side of the display / operation panel 33, the wide-angle imaging lens 42, the first light-projecting lens 44, and the second light-projecting lens 46 may be provided by processing the display / operation panel 33 itself.

[0045] The user interface module 38, in which the liquid crystal device 32 and the camera module 31 are integrally provided, is rotationally driven by the actuator 37 around an axis along, for example, the vertical direction of the automobile 1. As a result, the orientations of the liquid crystal device 32 and the camera module 31 are changed in the same manner. Note that the actuator 37 may rotationally drive the user interface module 38 so as to rotate the passenger side edge of the user interface module 38 around, for example, the driver side edge, rather than around an axis at the central position Y0 of the user interface module 38 in the vehicle width direction. The user interface module 38 is movably provided in the central portion of the automobile 1 in the vehicle width direction. In such a user interface module 38, the liquid crystal device 32 can display not only a setting screen for occupant monitoring, but also other screens, such as a navigation setting screen, a guidance screen, and a content display screen, to the driver and passengers.

[0046] FIG. 7 is a flowchart of the monitoring control of the occupant by the monitoring control unit of FIG. When a new passenger gets into the automobile 1, the monitoring control unit 36 ​​may repeatedly execute the monitoring control of FIG.

[0047] In step ST1, the monitoring control unit 36 ​​judges whether or not a new occupant has boarded the automobile 1. The occupant opens a door (not shown) of the automobile 1 and sits in the seat 4. The monitoring control unit 36 ​​may detect and judge that a new occupant has boarded the automobile 1, for example, based on an opening / closing detection signal of the door opening / closing sensor 11. In addition, for example, the monitoring control unit 36 ​​may judge whether or not a new occupant has boarded the automobile 1, based on the fact that the new occupant has been captured in the image data of the image sensor 41. If a new occupant has not boarded the automobile 1, the monitoring control unit 36 ​​repeats this process. When a new occupant has boarded the automobile 1, the monitoring control unit 36 ​​advances the process to step ST2.

[0048] In step ST2, the monitoring control unit 36 ​​determines the riding position of the occupant who has newly boarded the automobile 1. The monitoring control unit 36 ​​may determine the riding position of the new occupant based on the fact that the new occupant is imaged in the imaging data of the imaging sensor 41.

[0049] In step ST3, the monitoring control unit 36 ​​judges whether the newly boarded passenger is in a stable posture in the riding position. If the image capturing position of the new passenger does not change significantly based on a plurality of pieces of image data captured continuously by the image sensor 41, the monitoring control unit 36 ​​may judge that the newly boarded passenger is in a stable posture in the riding position. The monitoring control unit 36 ​​repeats this process until it is judged that the new passenger is in a stable posture. If it is judged that the new passenger is in a stable posture, the monitoring control unit 36 ​​advances the process to step ST4.

[0050] In step ST4, the monitoring control unit 36 ​​determines the position of the face or head of the newly boarded occupant based on the latest imaging data captured by the imaging sensor 41, and obtains, by calculation, a control amount for the actuator 37 for capturing an image of the face or head of the newly boarded occupant satisfactorily in the central portion of the image captured by the imaging sensor 41. The control amount for the actuator 37 includes a control amount in the width direction of the automobile 1. The control amount for the actuator 37 may further include a control amount in the up-down direction of the automobile 1. In this way, the monitoring control unit 36 ​​obtains the relative direction of the face or head of the newly boarded occupant.

[0051] In step ST5, the monitoring control unit 36 ​​operates the actuator 37 based on the acquired control amount. As a result, the image sensor 41 and the liquid crystal device 32 provided in the user interface module 38 are oriented so as to face the face of the new occupant. The actuator 37 is controlled, and the orientation of the user interface module 38 is changed so that the orientation of the image sensor 41 approaches the direction of the new occupant who will board the automobile 1 and becomes the acquired relative direction.

[0052] In step ST6, the monitoring control unit 36 ​​executes a personal identification process for the newly boarded occupant based on the imaging data after control by the imaging sensor 41. The monitoring control unit 36 ​​executes identification control for monitoring the occupant in the automobile 1, using the imaging data of the imaging sensor 41 with the orientation of the user interface module 38 changed. The monitoring control unit 36 ​​may compare an image of the occupant that is included in the center part of the image of the imaging data and has reduced distortion with the occupant data of multiple occupants recorded in the memory 35, and identify each occupant with high accuracy. The monitoring control unit 36 ​​may turn on the first LED 43 and the second LED 45 of the camera module 31. This causes ultraviolet light to be irradiated onto the upper body or head of an occupant facing the guidance display on the liquid crystal device 32 so as to view the guidance display. In addition, in order to identify the passenger in the vehicle, the monitoring control unit 36 ​​may obtain a new captured image from the imaging sensor 41, extract image components of the passenger in the vehicle, and compare them with the passenger data registered in the memory 35. In this case, the monitoring control unit 36 ​​may compare only the image components of the frontal view in the registered image data of each passenger registered in the memory 35. A frontal view image often clearly contains characteristic points such as a nose on a face. Even if the comparison is made based only on the image components of the frontal view, it is possible to determine whether the passengers match with high accuracy. In addition, the monitoring control unit 36 ​​may compare the characteristic points extracted from each image, rather than directly comparing the images. When the registered imaging data included in the occupant data registered in the memory 35 matches with a certain degree of accuracy or higher, the monitoring control unit 36 ​​may identify the occupant as the occupant of the registered imaging data. In this case, the monitoring control unit 36 ​​identifies the occupant as an occupant identified by checking against the multiple occupant data registered in the memory 35. When there is no registered imaging data that matches with a certain degree of accuracy or higher among the multiple occupant data registered in the memory 35, the monitoring control unit 36 ​​may identify the occupant as an unregistered occupant. Furthermore, if there is an occupant whose occupant data is recorded in the memory 35, the monitoring control unit 36 ​​may further execute a setting process using the occupant data. The monitoring control unit 36 ​​outputs information of the setting data to each unit of the automobile 1. This executes processes such as the seat position for the occupant to sit, the initial setting of whether or not to use driving assistance, driving preference information in automatic driving, information of the server device to be used, occupant protection settings, and air conditioning settings. Furthermore, the monitoring control unit 36 ​​determines, for example, whether a child bucket is installed on the passenger seat 4 based on the latest captured image data, and executes a setting to prohibit the deployment of an airbag toward the passenger seat 4 if a child bucket is installed.

[0053] In step ST7, the monitoring control unit 36 ​​judges whether or not to newly register the newly boarded passenger. For example, the monitoring control unit 36 ​​displays a confirmation screen for the registration process on the liquid crystal device 32, and when the passenger performs an operation corresponding to the confirmation screen on the display operation panel 33, it judges that the passenger should be newly registered, and proceeds to step ST8. If an operation not requiring registration is performed, the monitoring control unit 36 ​​proceeds to step ST9.

[0054] In step ST8, the monitoring control unit 36 ​​executes a new registration process for the newly boarded passenger. The monitoring control unit 36 ​​adds and records passenger data of the new passenger in the memory 35. The monitoring control unit 36 ​​executes registration control for the passenger who gets into the automobile 1, using the imaging data of the imaging sensor 41 with the orientation of the user interface module 38 changed.

[0055] In step ST9, the monitoring control unit 36 ​​returns the actuator 37 operated in step ST5 with the acquired control amount, so that the image sensor 41 and the liquid crystal device 32 provided in the user interface module 38 face toward the rear of the automobile 1.

[0056] In step ST10, the monitoring control unit 36 ​​starts monitoring control of the state of the occupants riding in the automobile 1. After executing the identification control and the registration control, the monitoring control unit 36 ​​controls the actuator 37 to return the orientation of the user interface module 38, and monitors the occupants riding in the automobile 1 using the imaging data of the imaging sensor 41 in the returned state. When there are multiple occupants riding in the automobile 1, the monitoring control unit 36 ​​determines and monitors the state of each of the multiple occupants using the latest imaging data of the imaging sensor 41 in the returned state as well as the registered occupant data recorded in the memory 35 for each occupant. The imaging sensor 41 provided in the user interface module 38 can capture a wide-angle image of the interior of the automobile 1 when facing backward from the center of the automobile 1 in the vehicle width direction. The imaging sensor 41 can capture imaging data for monitoring the multiple occupants riding in the automobile 1. For example, the monitoring control unit 36 ​​may identify image components of the upper body and head of the occupant included in the latest imaging data. The monitoring control unit 36 ​​may use image components of the registered imaging data in the memory 35 for the occupant already identified as a reference to determine image components of the upper body and head of the occupant included in the latest imaging data. Here, the monitoring control unit 36 ​​may estimate an image of the occupant from the front included in the current imaging data captured by the imaging sensor 41 based on a difference (difference) between an image of each occupant registered in the memory 35 captured from the front and an image of the occupant viewed from the front, and may determine the state of each occupant riding in the automobile 1 based on the image components estimated as an image of the occupant captured from the front. The monitoring control unit 36 ​​may use an image of the occupant captured from the front to correct lens distortion and orientation of the image of the occupant included in the current imaging data, and estimate an image of the occupant captured from the front based on the current imaging data. In addition, if the registration cannot be identified, the monitoring control unit 36 ​​may use the image components of the standard registered imaging data in the memory 35 as a reference, or may determine the image components of the occupant's upper body and head contained in the latest imaging data without using the image components of the registered imaging data. The monitoring control unit 36 ​​may determine the state of the occupant in the latest imaging data, for example, the open / closed state of the eyes, the direction of the line of sight, the direction of the head, etc. The monitoring control unit 36 ​​may determine the pulsation of the veins. Then, for example, when the driver's eyes are closed, the line of sight is not looking straight ahead, the head direction is not looking straight ahead, or the pulse rate is high, the monitoring control unit 36 ​​determines that the driver is not in a state suitable for driving. In other cases, the monitoring control unit 36 ​​may determine that the driver is in a state suitable for driving. In this way, the monitoring control unit 36, as a determination unit, may determine the state of each occupant in the automobile 1 using the registered imaging data recorded in the memory 35 as reference data. The monitoring control unit 36 ​​may determine at least the open / closed state of the driver's eyes or the line of sight of the driver as the state of the driver riding in the automobile 1. In this case, the monitoring control unit 36 ​​can use high-quality registered image data as a reference, so that not only the open / closed state of the eyes but also information on the change in the image state of the black eye between the upper eyelid and the lower eyelid and the change in the image state of the white eye can be acquired for each occupant, and the occupant's eye expression such as the line of sight can be determined with high accuracy. In contrast, if a similar determination is made without using the registered image data for each occupant, the occupant's eye expression such as the line of sight will be determined including individual differences such as the size of the eyes. It is not easy to determine the eye expression such as the line of sight for each individual with high accuracy. This results in excessive warnings being output based on the physical characteristics of the occupant. This is uncomfortable for the occupant. The monitoring control unit 36 ​​may also determine whether or not control is required for the driving of the automobile 1, based on the state determination result of the occupant such as the driver. For example, if it is determined that the driver is not in a state suitable for driving, the monitoring control unit 36 ​​determines that control is required and executes control for the driving of the automobile 1. For example, if it is determined that the driver is not looking straight ahead or the head is not facing forward, the monitoring control unit 36 ​​issues a warning to the driver. The warning to the driver may be, for example, a warning display on the liquid crystal device 32 or an output of a warning sound from the speaker device 14. If the driver does not look straight ahead even after the warning is output, that is, if it is determined that the driver is not looking straight ahead continuously, the monitoring control unit 36 ​​may switch the driving mode of the automobile 1 to automatic driving and decelerate and stop the driving of the automobile 1. When decelerating and stopping the driving of the automobile 1, the monitoring control unit 36 ​​may execute the lighting of hazard lights (not shown) and the transmission of emergency information by the external vehicle communication device 18. For example, if it is determined that the driver is dozing or has a high pulse rate, the monitoring control unit 36 ​​issues a warning to the driver or decelerates and stops the traveling of the automobile 1. When decelerating and stopping the traveling of the automobile 1, the monitoring control unit 36 ​​may turn on hazard lights (not shown) and transmit emergency information by the external vehicle communication device 18. Note that if the vehicle has been traveling continuously for a predetermined period of time or longer, if the eyes are opening and closing at a predetermined frequency, or if the head is tilted downward, the monitoring control unit 36 ​​may determine that control is necessary and execute an output to urge the driver to take a break, etc.

[0057] In step ST11, the monitoring control unit 36 ​​judges whether or not to end the occupant status monitoring control. The monitoring control unit 36 ​​may judge to end the occupant status monitoring control when the automobile 1 stops and the ignition is turned off, when the automobile 1 arrives at the destination and stops, or when the occupant gets off. The monitoring control unit 36 ​​may judge that the occupant gets off, for example, based on the detection of opening and closing by the door opening and closing sensor 11, or the image of the imaging sensor 41. If the occupant status monitoring control is not to be ended, the monitoring control unit 36 ​​repeats this process. If the occupant status monitoring control is to be ended, the monitoring control unit 36 ​​advances the process to step ST12.

[0058] In step ST12, the monitoring control unit 36 ​​executes post-processing of the monitoring control of the occupants. The monitoring control unit 36 ​​acquires, for example, setting information of each occupant when they get off the vehicle from each part of the automobile 1, and updates the occupant data of each occupant recorded in the memory 35. As a result, the occupant data registered in the memory 35 is adapted to the preferences of the occupant. The latest settings for the occupant will be automatically performed the next time the occupant gets in the vehicle. In addition, the monitoring control unit 36 ​​may temporarily record occupant data for an unregistered occupant in the memory 35. As a result, if the occupant performs a registration operation thereafter, the settings can be immediately linked. Thereafter, the monitoring and control unit 36 ​​ends the monitoring and control of FIG.

[0059] FIG. 8 is a flowchart of the new registration process of FIG.

[0060] In step ST21, the monitor control unit 36 ​​starts the registration process for the new occupant, and displays a front-facing guidance on the liquid crystal device 32. The new occupant faces the liquid crystal device 32 and faces forward.

[0061] In step ST22, the monitoring control unit 36 ​​acquires new imaging data from the imaging sensor 41, the orientation of which has been changed in the same manner as the liquid crystal device 32, and acquires a captured image of the new occupant viewed from the front. The monitoring control unit 36 ​​may turn on the first LED 43 and the second LED 45 of the camera module 31. As a result, the upper body or head of the occupant facing the guidance display on the liquid crystal device 32 so as to view the guidance display is irradiated with ultraviolet light. It is expected that the front-facing image contains image components of the eyes, nose, and mouth of the occupant due to infrared light with a high probability. The image components of infrared light may contain vein patterns of the head and eyeballs. The vein patterns of the head and eyeballs can be used to identify an individual, depending on the number of feature points extracted therefrom. In addition, the vein patterns of the head and eyeballs are less susceptible to light and dark patterns caused by the surface shape of the head or the unevenness and shape of the face. The vein patterns of the eyeballs are basically directed from the white of the eye at the periphery to the black of the eye at the center. The vein patterns of the eyelids covering the eyeballs are different from the vein patterns of the eyeballs. The monitoring control unit 36 ​​may extract and acquire information on the vein patterns of the head, eyeballs, eyelids, etc. from the imaging data. It is expected that the image components capturing the head of the occupant facing the vehicle in this way contain image components of each part of the head with high image quality. In addition, the monitoring control unit 36 ​​may repeat the guidance or repeat acquisition of current image data from the image sensor 41 until it confirms that the front-on image contains infrared light image components of the occupant's eyes, nose, and mouth.

[0062] In step ST23, the monitor control unit 36 ​​displays a front view guide on the liquid crystal device 32. The new occupant faces the front of the automobile 1.

[0063] In step ST24, the monitoring control unit 36 ​​acquires new imaging data from the imaging sensor 41, and acquires a captured image of the new occupant looking forward. The monitoring control unit 36 ​​turns on the first LED 43 and the second LED 45 of the camera module 31. As a result, the upper body or head of the occupant looking forward from the front of the automobile 1 is irradiated with ultraviolet light. The front view image may include infrared image components of the eyes, nose, and mouth of the occupant in a position basically different from that of the front-on image. The front view image may also include vein patterns of the head and eyes viewed from the front using infrared light. The monitoring control unit 36 ​​may extract and acquire information on the vein patterns of the head, eyes, eyelids, etc. from the imaging data. It is expected that the image components capturing the head of the occupant looking forward in this way contain high-quality image components of each part of the head looking forward when the occupant is not dozing or looking aside. The image components of each part of the head looking forward can be suitably made to correspond to the image components of each part of the head directly opposite. As a result, when there is an image of the head of one occupant looking forward in a dozing state, for example, it is possible to obtain the image components of each part of the head looking forward in a dozing state by performing a process similar to that in the case of converting the image components of each part of the head looking forward when the occupant is not dozing or looking aside into the image components of each part of the head looking forward. Such image components of each part of the head looking forward can be considered to be likely as the occupant is dozing. If a uniform process is performed regardless of the characteristics of the occupant, there is a possibility that the occupant will not be dozing, but such a possibility can be reliably reduced.

[0064] In step ST25, the monitoring control unit 36 ​​additionally registers in the memory 35 the occupant data of the new occupant, including the imaging data acquired through the processes of steps ST21 to ST24. As a result, the memory 35 stores a captured image of the new occupant viewed from the front and a captured image of the new occupant viewed from the front.

[0065] As described above, in this embodiment, the image sensor 41 for monitoring the occupants of the automobile 1 is provided in the user interface module 38 together with the liquid crystal device 32 so as to be close to the liquid crystal device 32 for displaying a screen to the occupants of the automobile 1, and when control for identifying or registering an occupant riding in the automobile 1 is executed, the actuator 37 is controlled to change the orientation of the user interface module 38 so that the orientation of the image sensor 41 approaches the direction of the occupant who will newly ride in the automobile 1. In particular, when the image sensor 41 of the user interface module 38 is in a home position where it can monitor a plurality of occupants of the automobile 1, after detecting that a new occupant will ride in the automobile 1, determining the riding position of the occupant who will newly ride in the automobile 1, and determining that the new occupant is in a stable posture at the riding position, the relative direction of the face or head of the new occupant is acquired, and the actuator 37 is controlled so that the orientation of the image sensor 41 becomes the acquired relative direction, thereby making it possible to orient the image sensor 41 toward the face or head of the new occupant who will newly ride in the automobile 1. The monitoring control unit 36 ​​can then use the imaging data from the imaging sensor 41 when the orientation of the user interface module 38 is changed to perform identification control or registration control to monitor occupants in the automobile 1 based on images with reduced distortion of occupants such as the driver. This allows the monitoring control unit 36 ​​to execute identification control or registration control for monitoring the occupant with the orientation of the image sensor 41 moving closer to the direction of the occupant who is newly getting into the automobile 1. For example, even if the user interface module 38 is provided in the center of the automobile 1 in the vehicle width direction and the image sensor 41 provided in the user interface module 38 faces rearward from the center of the automobile 1 in the vehicle width direction to capture a wide-angle image of the interior of the automobile 1 to capture and monitor a plurality of occupants who are riding in the automobile 1, the occupant involved in the identification control or registration control can be captured closer to the center of the screen with less distortion. The monitoring control unit 36 ​​can more reliably identify the occupant involved in the identification control or registration control by using data obtained by capturing an image of the occupant near the center of the image, preferably by using data obtained by capturing an image of the occupant facing the image. As a result, in this embodiment, the arrangement of the image sensor 41 and other components in the occupant monitoring device 15 provided in the automobile 1 for monitoring occupants can be made suitable for monitoring occupants. In addition, in this embodiment, since personal recognition of occupants such as a driver can be performed based on images with reduced distortion, it becomes easier to capture detailed facial features with high accuracy. Even if the image sensor 41 is provided in the center of the vehicle width direction, it is expected that the performance of personal recognition can be improved.

[0066] Moreover, in this embodiment, after executing the identification control or the registration control, the monitoring control unit 36 ​​controls the actuator 37 to return the orientation of the user interface module 38, and monitors the occupants in the automobile 1 using the imaging data of the imaging sensor 41 in the returned state. Here, the orientation of the user interface module 38 after being returned may be, for example, a state in which the imaging sensor 41 faces the rear direction of the automobile 1. As a result, in this embodiment, the imaging sensor 41 can capture and monitor multiple occupants in the automobile 1 based on imaging data obtained by capturing a wide-angle image of the interior of the automobile 1 from the center part in the width direction of the automobile 1. In this embodiment, multiple occupants in the automobile 1 can be captured and monitored by one imaging sensor 41. During occupant monitoring, the imaging sensor 41 faces stably in the rear direction, so that uncertainty and a sense of instability in operation caused by differences in the orientation of the imaging sensor 41 can be suppressed. Then, by using the imaging data of the imaging sensor 41 in the returned orientation together with the imaging data of the occupants acquired during identification control or registration control to monitor the occupants in the automobile 1, even if the imaging data of the imaging sensor 41 in the returned orientation does not properly capture, for example, all of the occupants' eyes, the condition of the occupants can be monitored more reliably by supplementing the information with the imaging data of the occupants acquired during identification control or registration control, for example.

[0067] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.

[0068] For example, in the above-described embodiment, the liquid crystal device 32 is arranged vertically. Alternatively, for example, the liquid crystal device 32 may be arranged horizontally.

[0069] FIG. 9 is an explanatory diagram of a modified example of the liquid crystal device 32 and the camera module 31 of the passenger monitoring device 15 provided in the central portion in the width direction of the automobile 1 of FIG. The camera module 31 having the image sensor 41 is disposed on the rear side of the display operation panel 33 so as to be above the rectangular display area of ​​the liquid crystal device 32 .

[0070] In the above-described embodiment, the display operation panel 33 of the liquid crystal device 32 and the imaging sensor 41 are disposed at the center in the width direction of the automobile 1. The imaging sensor 41 has a wide-angle imaging lens 42 with an angle of view capable of capturing images of the heads of multiple occupants seated in the front row of the automobile 1 while being disposed at the center in the width direction of the automobile 1. The monitoring control unit 36 ​​determines at least the open / closed state of the eyes of the occupants or their line of sight as the state of the occupants in the automobile 1. The passenger monitoring device may not fall into any of these categories. For example, the passenger monitoring device may monitor only the driver. Even in this case, it is expected that the reliability of the monitoring judgment can be improved by, for example, arranging the image sensor 41 on the back side of the display operation panel 33 of the liquid crystal device 32 so as to be located inside the circumscribing circle of the display area of ​​the image, or by the monitoring control unit 36 ​​using the registered image data recorded in the memory 35 as reference data to judge the state of the passenger in the automobile 1. [Explanation of symbols]

[0071] 1...Automobile (vehicle), 2...Vehicle body, 3...Vehicle interior, 4...Seat, 5...Dashboard, 6...Center console, 7...Steering, 10...Control device, 11...Door opening / closing sensor, 12...Vehicle speed sensor, 13...Park sensor, 14...Speaker device, 15...Occupant monitoring device, 16...Position setting device, 17...Driving assistance device, 18...External communication device, 19...Occupant protection device, 20...Air conditioning device, 21...In-vehicle network, 31...Camera module, 32...Liquid crystal device, 33...Display operation panel, 34...Input / output device, 35...Memory, 36...Monitoring control unit, 37...Actuator, 38...User interface module, 41...Imaging sensor, 42...Imaging wide-angle lens, 43...First LED, 44...First light projection lens, 45...Second LED, 46...Second light projection lens, 51...First occupant data, 52...Second occupant data

Claims

1. a user interface module provided with a display member for displaying a screen to a vehicle occupant and an imaging member for monitoring at least the vehicle occupant; an actuator for actuating the user interface module to change the orientation of the user interface module; A control unit that executes control for identifying or registering an occupant who rides in the vehicle; having The control unit is detecting a new occupant entering the vehicle; determining a riding position of a detected new occupant in the vehicle; and After determining that the detected new occupant is in a stable posture in the riding position, a relative direction of the detected new occupant's face or head is obtained; controlling the actuator to change the orientation of the user interface module so that the orientation of the imaging member coincides with the acquired relative direction; executing an identification control for identifying an individual occupant riding in the vehicle or a registration control for the identification, using the image data of the image capturing member in a state in which the orientation of the user interface module is changed; Vehicle occupant monitoring device.

2. The user interface module is movably provided in a central portion of the vehicle in a vehicle width direction, The control unit monitors the state of an occupant of the vehicle based on the captured image of the imaging member.

2. A vehicle occupant monitoring device according to claim 1.

3. the imaging member provided in the user interface module captures an image of the interior of the vehicle at a wide angle while facing from a center portion of the vehicle in a width direction of the vehicle toward a rear direction of the vehicle, and captures the image to monitor a plurality of occupants in the vehicle.

3. A vehicle occupant monitoring device according to claim 1 or 2.

4. After executing the identification control or the registration control, the control unit: Controlling the actuator to reorient the user interface module; monitoring a passenger in the vehicle using image data captured by the imaging member in a returned state; 4. A vehicle occupant monitoring device according to claim 1.

5. When monitoring an occupant in the vehicle, the control unit The imaging data of the imaging member in the returned state is used together with the imaging data of the occupant acquired in the identification control or the registration control.

5. A vehicle occupant monitoring device according to any one of claims 1 to 4.

6. The display member and the imaging member are movably provided in a central portion of the vehicle in a vehicle width direction, The control unit monitors the state of an occupant of the vehicle based on the captured image of the imaging member.

6. A vehicle occupant monitoring device according to any one of claims 1 to 5.

Images