Vehicle occupant monitoring device
By positioning the imaging device adjacent to the display and controlling operations that obstruct the view, the vehicle occupant monitoring device ensures reliable condition monitoring without restricting swipe operations, enhancing usability and accuracy.
Patent Information
- Application Number
- JP2021027735
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-02-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2041-02-24
AI Technical Summary
The installation of an imaging device near a liquid crystal device with a touch panel in a vehicle can obstruct the field of view due to movements such as swipe operations by occupants, particularly when monitoring multiple passengers, leading to unreliable condition monitoring and reduced usability.
The imaging member is positioned adjacent to the display area of the display member, and the control unit prevents control execution for operations involving movement towards the imaging member, allowing swipe operations while ensuring the imaging device's field of view is not obstructed.
This approach maintains ease of use by allowing swipe operations while effectively monitoring occupant conditions without obstructing the imaging device's view, achieving a balance between usability and monitoring reliability.
Smart Images

Figure 0007584322000001 
Figure 0007584322000002 
Figure 0007584322000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a vehicle occupant monitoring device. [Background technology]
[0002] Liquid crystal devices having touch panels are beginning to be adopted as user interfaces in vehicles, for example as display members for performing vehicle settings, etc. Liquid crystal devices having touch panels display operation objects such as buttons. In this case, a vehicle occupant operates an operation object, such as a button, displayed on a liquid crystal device having a touch panel, and a control unit of the vehicle executes control according to the operation content. A liquid crystal device having a touch panel, for example in a mobile terminal, allows a variety of operations, such as a swipe operation by moving the screen as if tracing the surface of the liquid crystal device, in addition to a click operation by touching an object. Passengers are accustomed to these operations when operating a mobile terminal, etc.
[0003] In addition, it is considered that in the future, there will be a demand for vehicles to monitor the state of occupants during driving assistance or autonomous driving (Patent Documents 1 and 2). In this case, in order to monitor the vehicle occupants, an imaging device is provided, for example, on the dashboard in front of the occupants, and the condition of the occupants is determined and monitored based on images captured by the imaging device. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2019-014359 A [Patent Document 2] JP 2019-014360 A Summary of the Invention [Problem to be solved by the invention]
[0005] In order to meet these various demands, it is conceivable that the imaging device may be provided near a liquid crystal device having a touch panel provided in the vehicle. This allows the imaging device to capture an image of the occupant from the front as he or she looks at a predetermined display on the liquid crystal device. If there is imaging data of the occupant captured from the front in advance, it is possible to more accurately determine the state of each occupant, such as the driver, by, for example, comparing the imaging data from the front with the current imaging data. In this case, it is desirable to provide the imaging device near the liquid crystal device. However, if the imaging device is provided near a liquid crystal device as a display member, when a passenger being monitored performs, for example, a swipe operation on the liquid crystal device, the field of view of the imaging device may be blocked by the fingers, hands, arms, and elbows of the passenger who moves when performing the operation. In particular, when attempting to monitor the status of multiple passengers in the front row including the driver using image data captured by one imaging device, it is desirable to provide the imaging device together with a liquid crystal device having a touch panel in the center part of the vehicle width direction of the vehicle body, and in such a case, the field of view of the imaging device may be easily blocked by the fingers, hands, arms, and elbows of the passenger when performing a swipe operation. If it is not possible to capture an image of the passenger, it is not possible to monitor the status of the passenger.
[0006] For this reason, it is conceivable that in a liquid crystal device having a touch panel provided in a vehicle, swipe operations and the like are uniformly restricted, and only click operations and the like are allowed to be used. However, if swipe operations and the like are uniformly restricted, the occupant will repeat click operations and the like. The occupant will not be able to operate the liquid crystal device provided in the vehicle using swipe operations and the like to which he is accustomed. The occupant will not be able to use swipe operations and the like, which he is accustomed to and which do not feel strange, as an operation screen of the vehicle. The usability of the liquid crystal device having a touch panel provided in the vehicle will be reduced.
[0007] In such a vehicle, even when a display member that displays an operation screen for the occupant in the vehicle is used together with an imaging member for occupant monitoring, it is necessary to achieve a good balance between ease of use for the occupant regarding the display member and monitoring of the occupant's status. [Means for solving the problem]
[0008] The vehicle occupant monitoring device of the present invention comprises a display member that displays an operation screen for a vehicle occupant, a control unit that executes control according to operation content determined as an operation by the occupant on the operation screen of the display member, and an imaging member capable of imaging the vehicle occupant, and the control unit does not execute control according to the determined operation content when the operation content by the occupant is an operation that at least involves movement toward the imaging member.
[0009] Preferably, the imaging member is disposed adjacent to the display area of the display member, Control Unit When the operation performed by the occupant is an operation involving a movement toward the imaging member in the peripheral portion of the imaging member in the display area, control according to the determined operation content may not be executed.
[0010] Preferably, the display member has a panel for displaying the operation screen that is larger than a display area of the operation screen, and the imaging member is provided on the back side of the panel of the display member.
[0011] Preferably, Control Unit When the operation involving a movement toward the imaging member is an operation of tracing the surface of the display member toward the imaging member, control according to the determined operation content may not be executed.
[0012] Preferably, the display member displays a plurality of operation objects that can be swiped on the operation screen, Control UnitWhen a swipe operation is performed on the operation object displayed around the imaging member by tracing the operation object toward the imaging member, control corresponding to the determined swipe operation may not be executed.
[0013] Preferably, Control Unit When the direction of the operation of tracing the surface of the display member is toward the imaging member, control according to the determined operation content is not executed, and when the direction of the operation of tracing the surface of the display member is toward a direction different from the imaging member, control according to the determined operation content is executed.
[0014] Preferably, when the operation performed by the occupant is an operation that involves at least a movement toward the imaging member while the vehicle is traveling, the control unit does not execute control according to the determined operation.
[0015] Preferably, the display member and the imaging member are provided in a central portion of the vehicle in a vehicle width direction, and the control unit monitors the state of an occupant of the vehicle based on an image captured by the imaging member. Effect of the Invention
[0016] In the present invention, when the operation content of the occupant is an operation involving at least a movement toward the imaging member, the control unit does not execute control according to the determined operation content. Therefore, the occupant may not perform an operation involving a movement toward the imaging member on the display member that displays the operation screen of the vehicle occupant. As a result, in the present invention, it is expected that the field of view of the imaging member is less likely to be obstructed by the movement of the occupant's fingers, hands, arms, elbows, etc. when operating. Moreover, the occupant can perform an operation involving a movement without being restricted as long as he or she does not move toward the imaging member, and therefore can perform an operation involving a movement on the operation screen displayed on the display member. In this way, with the present invention, operations involving movement by the occupant are not uniformly restricted, so even when a display member that displays an operation screen is used together with an imaging member for occupant monitoring, it is possible to satisfactorily achieve both ease of use for the occupant regarding the display member and monitoring of the occupant's condition. [Brief description of the drawings]
[0017] [Figure 1] FIG. 1 is an explanatory diagram of an automobile to which an occupant monitoring device according to an embodiment of the present invention is applied. [Diagram 2] FIG. 2 is an explanatory diagram of the control device of the automobile of FIG. [Diagram 3] FIG. 3 is an explanatory diagram of the vehicle occupant monitoring device of FIG. [Figure 4] FIG. 4 is an explanatory diagram of the in-vehicle arrangement of a liquid crystal device having a display operation panel and a camera module having an image sensor in the occupant monitoring device of FIG. [Diagram 5] FIG. 5 is an explanatory diagram of an image capturing state of a driver who is an occupant of a vehicle. [Figure 6] FIG. 6 is a detailed explanatory diagram of a liquid crystal device and a camera module of an occupant monitoring device provided in the center portion of the vehicle body in the vehicle width direction. [Figure 7] FIG. 7 is a flowchart of the passenger registration control by the monitor control unit of FIG. [Figure 8] FIG. 8 is a flowchart of the monitoring control of the occupant by the monitoring control unit of FIG. [Figure 9] FIG. 9 is a flowchart of response control for operations on the display / operation panel of FIG. 3 that is provided in the center portion of the vehicle body in the vehicle width direction and can be used for monitoring control. [Figure 10] FIG. 10 is an explanatory diagram of a first modified example of a liquid crystal device and a camera module of an occupant monitoring device provided in the center portion in the width direction of the vehicle body of FIG. [Figure 11] FIG. 11 is an explanatory diagram of a second modified example of a liquid crystal device and a camera module of an occupant monitoring device provided in the center portion in the width direction of the vehicle body of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0018] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0019] FIG. 1 is an explanatory diagram of an automobile 1 to which an occupant monitoring device 15 according to an embodiment of the present invention is applied. The automobile 1 is an example of an automobile 1. The automobile 1 may be an automobile that uses an internal combustion engine as a power source for running, an automobile that uses stored electric power in a battery as a power source for running, or a combination of these. A vehicle interior 3 is formed in a vehicle body 2 of an automobile 1 in Fig. 1. A plurality of front row seats 4 for a driver's seat and a passenger seat, and a long rear row seat 4 are provided in the vehicle interior 3. Passengers including the driver are seated in the seats 4. A dashboard 5 is provided in the front part of the vehicle interior 3, which is in front of the plurality of front row seats 4, so as to extend along the vehicle width direction of the vehicle body 2.
[0020] FIG. 2 is an explanatory diagram of the control device 10 of the automobile 1 of FIG. The control device 10 in Figure 2 has a door opening / closing sensor 11, a vehicle speed sensor 12, a park sensor 13, a speaker device 14, an occupant monitoring device 15, a position setting device 16, a driving assistance device 17, an external communication device 18, an occupant protection device 19, an air conditioning device 20, and an in-vehicle network 21 to which these are connected.
[0021] The in-vehicle network 21 may be a wired communication network for the automobile 1, for example, compliant with CAN (Controller Area Network) or LIN (Local Interconnect Network). The in-vehicle network 21 may be a communication network such as a LAN, or a combination of these. A part of the in-vehicle network 21 may include a wireless communication network.
[0022] The door opening / closing sensor 11 detects whether the door of the automobile 1 is opened or closed. The vehicle speed sensor 12 detects the speed of the traveling automobile 1. The vehicle speed sensor 12 may detect a stopped state. The park sensor 13 detects a parking state in which the automobile 1 is kept stopped. The park sensor 13 may detect, for example, that a shift lever (not shown) is operated to a parking position or that a parking lever (not shown) is operated to a brake position as the parking state.
[0023] The occupant monitoring device 15 executes a recognition process for an occupant in the vehicle compartment 3 when, for example, the door opening / closing sensor 11 detects the opening or closing of a door. When the occupant monitoring device 15 recognizes multiple occupants, it may monitor each of the multiple occupants individually. The occupant monitoring device 15 may output information about each recognized occupant and information based on the monitoring to each part of the control device 10 via the vehicle network. In particular, the occupant monitoring device 15 recognizes, for example, an occupant seated in the driver's seat 4 as the driver, and monitors the recognized driver for looking away or falling asleep. When the driver is in a predetermined state, the occupant monitoring device 15 executes control for calling the driver's attention and control for preventing danger. At this time, the occupant monitoring device 15 may output information about the driver, information for calling the driver's attention, and information for preventing danger to each part of the control device 10 via the vehicle network.
[0024] The speaker device 14 outputs voice, warning sounds, etc. The speaker device 14 may output warnings from an occupant monitoring device 15 to the driver and other occupants. The position setting device 16 adjusts, for example, the front-rear and up-down positions of the seat 4 and the angle of the backrest, the front-rear and up-down positions and angles of the handlebars, and the front-rear and up-down positions and angles of various pedals. The position setting device 16 changes the seat position based on the occupant information output by the occupant monitoring device 15. The driving assistance device 17 assists the driver in manually driving the automobile 1, and assists the driver in automatically driving the automobile 1. The driving assistance device 17 controls the acceleration, deceleration, stopping, and steering of the automobile 1. Based on the driver information output by the occupant monitoring device 15, the driving assistance device 17 executes driving assistance according to the driver. The exterior communication device 18 establishes a wireless communication path with, for example, a base station of a public wireless communication network, a base station of a commercial wireless communication network, or a base station for advanced traffic information, and performs data communication using the established wireless communication path. The exterior communication device 18 may perform two-way data communication with, for example, a server device that supports autonomous driving. The exterior communication device 18 may transmit information on occupants including the driver output by the occupant monitoring device 15 to the server device, for example, as emergency support information. The occupant protection device 19 executes control to protect the occupant when it detects or predicts a collision of the automobile 1. The occupant protection device 19 protects the occupant seated in the seat 4 by, for example, deploying an airbag or applying tension to a seat belt (not shown). The occupant protection device 19 may execute occupant protection for the occupant according to the occupant information output by the occupant monitoring device 15. The air conditioner 20 controls the temperature and oxygen concentration in the vehicle interior 3. The air conditioner 20 supplies, for example, cooled air or superheated air to the vehicle interior 3, thereby adjusting the vehicle interior 3 to a set temperature. The air conditioner 20 may perform air conditioning in accordance with information about the occupants output by the occupant monitoring device 15. The passengers can ride comfortably under settings suited to them, for example, by the control based on the passenger information output by the passenger monitoring device 15. The driver can concentrate on driving the automobile 1, for example.
[0025] Fig. 3 is an explanatory diagram of the occupant monitoring device 15 of the automobile 1 of Fig. 2. The occupant monitoring device 15 of Fig. 3 monitors not only the driver but also a plurality of occupants seated, for example, in a plurality of seats 4 in the front row. The occupant monitoring device 15 of Fig. 3 has a camera module 31, a liquid crystal device 32 having a display operation panel 33, an input / output device 34, a memory 35, and a monitoring control unit 36 to which these are connected.
[0026] The input / output device 34 is connected to the in-vehicle network 21. The input / output device 34 inputs and outputs data between the input / output device 34 and other units provided in the automobile 1 through the in-vehicle network 21.
[0027] The liquid crystal device 32 displays an image on a display surface of the display operation panel 33. As a display member, the liquid crystal device 32 displays a screen to be visually recognized by each occupant in the automobile 1. The display screen includes, for example, an operation screen operated by the occupant. The display operation panel 33 is a transparent or semi-transparent panel that is superimposed on the display surface of the liquid crystal device 32. The display operation panel 33 superimposed on the display surface of the liquid crystal device 32 constitutes a display member. The display operation panel 33 detects an operation by the occupant on the display surface of the liquid crystal device 32. The display operation panel 33 may output the operation position of the occupant on the display surface of the liquid crystal device 32 to the monitoring control unit 36.
[0028] The camera module 31 is for capturing images of multiple passengers seated in multiple front row seats 4. The camera module 31 has an image sensor 41, an imaging wide-angle lens 42, a first LED 43, a first light-projecting lens 44, a second LED 45, and a second light-projecting lens 46.
[0029] The imaging sensor 41 is a semiconductor optical sensor such as a CCD or a CMOS sensor. The imaging sensor 41 may have a substantially rectangular light receiving surface on which a plurality of light receiving elements are arranged. The imaging sensor 41 may output imaging data including a captured image to the monitoring control unit 36. The wide-angle imaging lens 42 is provided over the imaging sensor 41. The wide-angle imaging lens 42 may be made up of a plurality of optical lenses so as to suppress distortion in the peripheral portion of the image. The imaging sensor 41 and the imaging wide-angle lens 42, as imaging members, are capable of capturing images of the occupants of the automobile 1 so as to enable monitoring of the status of the multiple occupants riding in the automobile 1. The wide-angle imaging lens 42 may be a lens capable of capturing images of the upper bodies or heads of multiple occupants seated in multiple seats 4 in the front row when the imaging sensor 41 is provided in the center of the dashboard 5 in the vehicle width direction, for example.
[0030] The first LED 43 and the second LED 45 may be semiconductor light emitting elements. The first LED 43 and the second LED 45, as light emitting members, project light toward the occupant of the automobile 1 that is the image pickup target of the image pickup sensor 41. The first LED 43 and the second LED 45 may project infrared light, for example. In this case, the image pickup sensor 41 outputs imaging data including an image captured by infrared light to the monitoring control unit 36. The first light emitting lens 44 is provided overlapping the first LED 43. The first light emitting lens 44 may irradiate the light of the first LED 43 mainly toward the driver seated in the seat 4 as the driver's seat. The second light emitting lens 46 is provided overlapping the second LED 45. The second light emitting lens 46 may irradiate the light of the second LED 45 mainly toward the occupant seated in the seat 4 as the passenger seat. Second light projecting lens 46 may project diffused light toward a driver seated in seat 4 serving as a driver's seat as well as an occupant seated in seat 4 serving as a passenger seat.
[0031] The memory 35 stores programs and data. The memory 35 may be composed of a non-volatile memory and a volatile memory. Examples of the non-volatile memory include a HDD, an SSD, and an EEPROM. Examples of the volatile memory include a RAM.
[0032] Data on multiple occupants registered in the automobile 1 may be managed and recorded for each occupant in the memory 35 of the occupant monitoring device 15. First occupant data 51 for a first occupant and second occupant data 52 for a second occupant are shown in Fig. 3. A database is formed in the memory 35 from the multiple occupant data.
[0033] Each occupant data such as the first occupant data 51 and the second occupant data 52 may include identification information unique to each occupant, for example, registered image data obtained by capturing an image of the occupant's head or eyes by the image sensor 41, and various setting data set by the occupant. The setting data may include, for example, information such as the seat position of the occupant, an initial setting for whether or not to use driving assistance, driving preference information in autonomous driving, information on the server device to be used, occupant protection settings, and air conditioning settings. In this way, the memory 35, as a recording member, can record image data captured by the image sensor 41 from the front of each occupant while a specified screen is being displayed on the liquid crystal device 32, as registered image data for each occupant. Additionally, the memory 35 may record occupant data for unregistered generic occupants.
[0034] The monitoring control unit 36 may be, for example, an ECU, a CPU, or another microcomputer. The monitoring control unit 36 reads and executes a program from the memory 35. This realizes the monitoring control unit 36 as a control unit of the occupant monitoring device 15. The monitoring control unit 36 may execute control according to the operation content determined as an operation by the occupant on the operation screen of the liquid crystal device 32. The monitoring control unit 36 also monitors the states of multiple occupants of the automobile 1 based on the image data captured by the imaging sensor 41. To monitor the occupants, the monitoring control unit 36 executes, for example, a registration process and a monitoring process for the occupants in the automobile 1. In the monitoring process, the monitoring control unit 36 may identify the occupants in the automobile 1 and execute a monitoring process for each occupant. In particular, for at least the occupants who are drivers, the monitoring control unit 36, as a determination unit, determines the state of each occupant, such as distraction, dozing, or emergency, using the registered image data of each occupant registered in the memory 35 as reference data for determination.
[0035] FIG. 4 is an explanatory diagram of the in-vehicle arrangement of the liquid crystal device 32 having the display operation panel 33 and the camera module 31 having the image sensor 41 in the occupant monitoring device 15 of FIG.
[0036] FIG. 4(A) shows a dashboard 5 and a center console 6 provided at the front of a vehicle interior 3. 1, the liquid crystal device 32 and the camera module 31 of this embodiment are disposed vertically in the center of the vehicle 1 in the vehicle width direction, from the dashboard 5 to the center console 6. This allows the image sensor 41 of the camera module 31 to capture an overall image of the driver and passengers at a wide angle. The image sensor 41 can capture an image of the driver's upper body including the head, and the passenger's upper body including the head. The liquid crystal device 32 may be disposed at the center in the vehicle width direction of the automobile 1 so that the center position Yd of the display screen in the vehicle width direction coincides with the center position Y0 of the automobile 1 in the vehicle width direction, but in this embodiment, the center position Yd of the display screen in the vehicle width direction is offset slightly toward the opposite side of the driver from the center position Y0 of the automobile 1 in the vehicle width direction. The liquid crystal device 32 is disposed tilted with respect to the vehicle width direction so as to face the driver from that position. The display screen of the liquid crystal device 32 can improve the visibility of the driver while ensuring the visibility of passengers.
[0037] FIG. 4B is an explanatory diagram of an image capturing range inside the vehicle by the camera module 31 having the image capturing sensor 41 and the imaging wide-angle lens 42, which are provided at an angle at an offset position in the center of the vehicle 1 in the vehicle width direction. A wide-angle imaging lens 42 is superimposed on the imaging sensor 41. This allows the imaging sensor 41 to capture an image of the entire vehicle interior 3, as shown as an imaging range in Fig. 4(B). The imaging sensor 41 is disposed in the center portion in the vehicle width direction of the automobile 1, and can capture images of the heads of the driver and passengers seated in the multiple front row seats 4. However, when the imaging wide-angle lens 42 is combined with the imaging sensor 41 to enable imaging of multiple occupants, the image in the center of the imaging range is a high-quality area with less distortion, while the peripheral area of the imaging range is a distortion-generating area where distortion occurs compared to the center. Therefore, if an occupant sits toward the right or left edge in the vehicle width direction, it tends to be difficult to image the occupant's head without distortion. When using the open / closed state of the occupant's eyes to judge, there is a high possibility that the judgment will not be made correctly. In this embodiment, the display operation panel 33 is disposed at an angle with respect to the front-rear direction of the vehicle body 2 so as to face the driver. As a result, the high image quality area in the center of the image captured by the image sensor 41 is closer to the driver. The driver is more likely to be captured in the high image quality area in the center of the image captured by the image sensor 41, and is less likely to be captured in the peripheral distortion area. It can be expected that the driver will be captured in an image with reduced distortion.
[0038] FIG. 5 is an explanatory diagram of an image capturing state of a driver who is an occupant of the automobile 1. In FIG.
[0039] Fig. 5(A) is a front view of the head of the first driver, and Fig. 5(B) is a perspective view of the head of the first driver of Fig. 5(A). When the driver faces forward of the automobile 1, the image sensor 41, which is provided in the center of the automobile 1 in the vehicle width direction, images the driver's head from an oblique direction as shown in FIG. 5(B) rather than from the front as shown in FIG. 5(A).
[0040] Fig. 5(C) is a front view of the head of the second driver, and Fig. 5(D) is a perspective view of the head of the second driver of Fig. 5(C). When the driver faces forward of the automobile 1, the image sensor 41 provided in the center of the automobile 1 in the vehicle width direction images the driver's head from an oblique direction as shown in FIG. 5(D) rather than from the front as shown in FIG. 5(C).
[0041] In the image data of FIG. 5(D), unlike the image data of FIG. 5(B), the driver's right eye is hidden by the driver's high nose and only the pupil of the eye is visible. In this case, it is not easy for the monitoring control unit 36 to determine whether the driver's right eye is open or closed from only the imaging data of Fig. 5(D). In particular, when the image is not free of distortion as in Fig. 5(D) but is captured as a distorted image, it is extremely difficult for the monitoring control unit 36 to determine whether the driver's right eye is open or closed from only the imaging data of Fig. 5(D). In addition, since the image data of FIG. 5(D) does not include the image components of the white of the driver's right eye to a degree that allows the state to be determined, it is difficult for the monitoring control unit 36 to extract the state of the right eye based only on the image data of FIG. 5(D). If the monitoring control unit 36 cannot identify the right eye, it may not be able to determine the direction of the driver's head as a result. For example, when identifying the imaging area of the head in the image based on the positional relationship between both eyes and the nose and the positional relationship between both eyes and the mouth, the monitoring control unit 36 cannot identify the direction of the head of the driver in FIG. 5(D) based only on the image data of FIG. 5(D). If there is no reference data for the driver's head, the monitoring control unit 36 is more likely to be unable to correctly estimate the state of the driver if the image components do not include the driver's right eye. When the imaging sensor 41 of the camera module 31 is provided in the center of the automobile 1 in the vehicle width direction, the situation as shown in FIG. 5(D) is likely to occur.
[0042] Thus, there are individual differences among passengers, such as some with large eyes, some with small eyes, some with eyes that protrude forward, and some with eyes that are set back. The actual position of the driver's head may move not only in the direction of the field of view in the vehicle width direction, but also in the vertical direction. Considering the occurrence of these movements, it is not easy to correctly estimate the state of the driver's head and eyes based only on the current image data of the image sensor 41.
[0043] Fig. 6 is a detailed explanatory diagram of the liquid crystal device 32 and the camera module 31 of the occupant monitoring device 15 provided in the central portion in the vehicle width direction of the vehicle body 2. Fig. 6(A) is a front view from the vehicle compartment 3. In Fig. 6, the central position Yd of the display operation panel 33 of the liquid crystal device 32 is shifted from the central position Y0 in the vehicle width direction of the vehicle body 2 toward the passenger side, which is the left side in the figure. The liquid crystal device 32 having the display / operation panel 33 is held at its square peripheral edge by a holding member 61 serving as a resin pressing part. The display operation panel 33 has an outer shape that is slightly larger than the liquid crystal device 32 .
[0044] The liquid crystal device 32 has a display area for displaying a screen that is not a rectangular display surface like a general monitor, but has a generally concave shape with the center of the upper edge of the rectangle cut out. A non-display area 33b that is a cutout is formed inside a rectangular frame 33a that circumscribes the generally concave display area of the liquid crystal device 32. The display area of the liquid crystal device 32 is located on both the left and right sides of the non-display area 33b. The non-display area 33b is formed linearly symmetrically with respect to the center position Yd of the display operation panel 33 of the liquid crystal device 32. In this way, the non-display section 33b is formed as a cutout portion in the center of the upper edge of the display area of the screen inside the rectangular frame 33a circumscribing the display area in the display operation panel 33 that displays the screen for the liquid crystal device 32. The non-display section 33b is formed as a cutout portion along the upper edge of the display area of the screen in the display operation panel 33 that displays the screen for the liquid crystal device 32. The non-display section 33b is formed in the display operation panel 33 that displays the screen for the liquid crystal device 32, at least over a range from the center position Yd in the vehicle width direction of the display area of the screen to the center position Y0 in the vehicle width direction of the automobile 1. The non-display section 33b is formed in a shape symmetrical in the vehicle width direction of the automobile 1 in the display operation panel 33, based on the center position Yd in the vehicle width direction of the display area of the screen. In the display area of the display operation panel 33, restriction areas 62 that prohibit swiping operations in the direction toward the camera module 31 are set on both the left and right sides and below the non-display section 33b in which the camera module 31 is arranged.
[0045] Camera module 31 having image sensor 41 is disposed on the rear side of non-display section 33b formed on the upper side of the display area of liquid crystal device 32, on the rear side of display operation panel 33. Camera module 31 is provided inside holding member 61 that holds the square corners of display operation panel 33 so as not to protrude outside of holding member 61. In detail, the camera module 31 is provided such that the imaging sensor 41 and the imaging wide-angle lens 42 are located between the central position Yd of the display operation panel 33 of the liquid crystal device 32 and the central position Y0 in the vehicle width direction of the vehicle body 2. As a result, the imaging sensor 41 and the imaging wide-angle lens 42 are provided such that their central positions in the vehicle width direction are located between the central position Yd of the display operation panel 33 of the liquid crystal device 32 and the central position Y0 of the vehicle body 2 in the vehicle width direction. This makes it appear as if the imaging sensor 41 of the camera module 31 is provided in the liquid crystal device 32 itself. The imaging sensor 41 is disposed adjacent to the display area of the operation screen of the display / operation panel 33 that displays the operation screen on the liquid crystal device 32. As a result, the image sensor 41 of the camera module 31 can capture an image of the occupant from the rear side of the liquid crystal device 32 having the display operation panel 33 . In addition, in the camera module 31, the imaging sensor 41 and the imaging wide-angle lens 42 are disposed in the center portion in the vehicle width direction. The first LED 43 and the first light-projecting lens 44 are disposed in the end portion on the passenger seat side. The second LED 45 and the second light-projecting lens 46 are disposed in the end portion on the driver side. This allows the camera module 31 to project light and capture images without being blocked by an object such as the steering wheel 7, even if there is an object between the driver and the dashboard 5. In addition, when the camera module 31 is thus provided on the back side of the display / operation panel 33, the wide-angle imaging lens 42, the first light-projecting lens 44, and the second light-projecting lens 46 may be provided by processing the display / operation panel 33 itself.
[0046] FIG. 7 is a flowchart showing the passenger registration control by the monitoring control unit 36 of FIG. When a new occupant gets into the automobile 1, the monitoring control unit 36 may repeatedly execute the registration control of FIG.
[0047] In step ST1, the monitoring control unit 36 first determines whether or not to register an occupant. For example, when the occupant selects and operates the occupant registration button on the menu screen displayed on the liquid crystal device 32, the monitoring control unit 36 determines that the occupant is to be registered and proceeds to step ST2. In other cases, the monitoring control unit 36 determines that the occupant is not to be registered and ends the registration control of FIG.
[0048] From step ST2, the monitoring control unit 36 executes the registration process for the new occupant. The monitoring control unit 36 first displays a guide for starting occupant registration on the liquid crystal device 32. It is considered that an occupant who is about to register by operating the occupant registration button on the menu screen basically looks directly at the guide display on the liquid crystal device 32 from the front. Furthermore, the monitoring control unit 36 may instruct the occupant to face the liquid crystal device 32 in the guide. Furthermore, the monitoring control unit 36 turns on the first LED 43 and the second LED 45 of the camera module 31. As a result, ultraviolet light is irradiated onto the upper body or head of the occupant who faces the liquid crystal device 32 so as to view the guide display.
[0049] In step ST3, the monitoring control unit 36 acquires current imaging data from the imaging sensor 41, and acquires image components of the newly registered occupant included in the imaging data, which are captured from the front, as a frontal image. When multiple occupants can be extracted from the image captured by the imaging sensor 41, the monitoring control unit 36 may have the occupant select one of them. It is expected that the frontal image contains image components of the occupant's eyes, nose, and mouth due to infrared light with a high probability. The image components of infrared light may contain vein patterns of the head and eyes. The vein patterns of the head and eyes can be used to identify individuals, depending on the number of feature points extracted therefrom. In addition, the vein patterns of the head and eyes are less susceptible to light and dark patterns caused by the surface shape of the head or the unevenness and shape of the face. The vein patterns of the eyes are basically directed from the white of the eye at the periphery to the black of the eye at the center. The vein pattern of the eyelids covering the eyeball is a different pattern from the vein pattern of the eyeball. The monitoring control unit 36 may extract and acquire information on the vein patterns of the head, eyeballs, eyelids, etc. from the imaging data. It is expected that the image components capturing the head of the occupant facing the vehicle in this way will contain high-quality image components of each part of the head. In addition, the monitoring control unit 36 may repeat the guidance or repeat acquisition of current image data from the image sensor 41 until it confirms that the front-on image contains infrared light image components of the occupant's eyes, nose, and mouth.
[0050] From step ST4, the monitoring control unit 36 displays a front view guidance for the occupant to view the front of the automobile 1 on the liquid crystal device 32. It is considered that the occupant views the front of the automobile 1 based on the guidance display on the liquid crystal device 32. In addition, the monitoring control unit 36 turns on the first LED 43 and the second LED 45 of the camera module 31. As a result, the upper body or head of the occupant who is viewing the front of the automobile 1 is irradiated with ultraviolet light.
[0051] In step ST5, the monitoring control unit 36 acquires current imaging data from the imaging sensor 41, and acquires image components of the occupant looking straight ahead, which are included in the imaging data, as a front view image. The front view image may include image components of the occupant's eyes, nose, and mouth, which are basically arranged differently from the front-on image, by infrared light. The front view image may also include vein patterns of the head and eyeballs viewed from the front by infrared light. The monitoring control unit 36 may extract and acquire information on the vein patterns of the head, eyeballs, eyelids, etc. from the imaging data. It is expected that the image components capturing the head of the occupant looking forward in this way contain high-quality image components of each part of the head looking forward when the occupant is not dozing or looking aside. The image components of each part of the head looking forward can be suitably made to correspond to the image components of each part of the head directly opposite. As a result, when there is an image of the head of one occupant looking forward in a dozing state, for example, it is possible to obtain the image components of each part of the head looking forward in a dozing state by performing a process similar to that in the case of converting the image components of each part of the head looking forward when the occupant is not dozing or looking aside into the image components of each part of the head looking forward. Such image components of each part of the head looking forward can be considered to be likely as the occupant is dozing. If a uniform process is performed regardless of the characteristics of the occupant, there is a possibility that the occupant will not be dozing, but such a possibility can be reliably reduced.
[0052] In step ST6, the monitor and control unit 36 executes a user registration process. In the user registration process, the monitoring control unit 36 records the two captured images of the occupant as registration imaging data together with identification information unique to each occupant. As a result, occupant data including the registration imaging data for the new occupant is recorded in the memory 35. The memory 35 serves as a recording member, and records imaging data captured by the imaging sensor 41 of each occupant while a predetermined screen is displayed on the liquid crystal device 32 as imaging data for each occupant. The registration imaging data recorded in the memory 35 for each occupant includes an image of the occupant taken from the front and an image of the occupant viewed from the front.
[0053] FIG. 8 is a flowchart of the monitoring control of the occupant by the monitoring control unit 36 of FIG. The monitoring control unit 36 may repeatedly execute the monitoring control of FIG.
[0054] In step ST11, the monitoring control unit 36 determines whether an occupant is in the vehicle. The monitoring control unit 36 may determine whether an occupant is in the vehicle based on, for example, the door opening / closing detection by the door opening / closing sensor 11 and an image from the imaging sensor 41. If an occupant is in the vehicle, the monitoring control unit 36 advances the process to step ST12. If an occupant is not in the vehicle, the monitoring control unit 36 repeats this process. Note that if an occupant is not in the vehicle, the monitoring control unit 36 may end the monitoring control of FIG. 8.
[0055] From step ST12, the monitoring control unit 36 starts the monitoring process of the passenger who is in the vehicle. The monitoring control unit 36 first identifies the passenger who is in the vehicle. The monitoring control unit 36 acquires a new captured image from the image sensor 41 , extracts image components of passengers currently in the vehicle, and compares them with a plurality of passenger data registered in the memory 35 . At this time, the monitoring control unit 36 may perform the comparison based only on the image components of the front view in the registered imaging data of each occupant registered in the memory 35. Front view images often contain clear feature points due to the unevenness of the face, such as the nose. Even if the comparison is performed based only on the image components of the front view, it is possible to determine the match of the occupants with high accuracy. Furthermore, the monitoring control unit 36 may compare feature points extracted from each image, rather than directly comparing the images. When the registered imaging data included in the occupant data registered in the memory 35 matches with a certain degree of accuracy or higher, the monitoring control unit 36 may identify the occupant in the vehicle as the occupant of the registered imaging data. In this case, the monitoring control unit 36 identifies the occupant who is in the vehicle as a specific occupant by comparing the occupant data with multiple occupant data registered in the memory 35. When there is no registered imaging data that matches with a certain degree of accuracy or higher among the multiple occupant data registered in the memory 35, the monitoring control unit 36 may identify the occupant as an unregistered occupant.
[0056] In step ST13, the monitoring control unit 36 acquires occupant data of the identified occupant.
[0057] In step ST14, the monitoring control unit 36 executes a setting process using the acquired occupant data. If the occupant data includes various setting data set by the occupant, the monitoring control unit 36 executes a setting process based on the setting data. The monitoring control unit 36 outputs information on the setting data to each part of the automobile 1. This executes processes such as the seat position of the occupant, initial setting of whether or not to use driving assistance, driving preference information in automatic driving, information on the server device to be used, occupant protection settings, and air conditioning settings. In addition, the monitoring control unit 36 determines, for example, whether a child bucket is installed on the passenger seat 4 based on the latest imaging data it has acquired, and if a child bucket is installed, executes a setting to prohibit the deployment of an airbag toward the passenger seat 4.
[0058] From step ST15, the monitoring control unit 36 starts the monitoring process for the identified occupant. The monitoring control unit 36 first obtains the latest image data captured by the image sensor 41.
[0059] In step ST16, the monitoring control unit 36 identifies the image components of the upper body and head of the occupant included in the latest imaging data. The monitoring control unit 36 uses the image components of the registered imaging data in the memory 35 for the occupant already identified as a reference to determine the image components of the upper body and head of the occupant included in the latest imaging data. Here, the monitoring control unit 36 may estimate the image of the occupant from the front included in the current imaging data captured by the imaging sensor 41 based on the difference (difference) between the image of each occupant registered in the memory 35 captured from the front and the image of the occupant viewed from the front, and may determine the state of each occupant riding in the automobile 1 based on the image components estimated as the image of the occupant captured from the front. The monitoring control unit 36 may use the image of the occupant captured from the front to correct the lens distortion and orientation of the image of the occupant included in the current imaging data, and estimate the image of the occupant captured from the front based on the current imaging data. In addition, if the registration cannot be identified, the monitoring control unit 36 may use the image components of the standard registered imaging data in the memory 35 as a reference, or may determine the image components of the occupant's upper body and head contained in the latest imaging data without using the image components of the registered imaging data. The monitoring control unit 36 judges the state of the occupant in the latest imaging data, for example, whether the eyes are open or closed, the direction of the line of sight, the direction of the head, etc. The monitoring control unit 36 may judge the pulsation of the veins. Then, for example, when the driver's eyes are closed, the line of sight is not looking straight ahead, the head direction is not looking straight ahead, or the pulse rate is high, the monitoring control unit 36 judges that the driver is not in a state suitable for driving. In other cases, the monitoring control unit 36 may judge that the driver is in a state suitable for driving. In this way, the monitoring control unit 36, as a determination unit, determines the state of each passenger in the automobile 1 by using the registered image data recorded in the memory 35 as reference data. Furthermore, the monitoring control unit 36 determines at least the open / closed state of the driver's eyes or the line of sight of the driver as the state of the driver in the automobile 1. In this case, the monitoring control unit 36 can use high-quality registered imaging data as a reference, so that it is possible to obtain information on not only the simple open / closed state of the eyes but also changes in the imaging state of the black eye between the upper eyelid and the lower eyelid and changes in the imaging state of the white eye for each occupant, and it is possible to determine the eye expression such as the line of sight of the occupant with high accuracy. On the other hand, if a similar judgment is made without using the registered image data for each occupant, the occupant's facial expression such as the gaze direction will be judged while taking into account individual differences such as eye size. It is not easy to judge the facial expression such as the gaze direction of each individual with high accuracy. This will result in excessive warnings being output based on the physical characteristics of the occupant, which is unpleasant for the occupant.
[0060] In step ST17, the monitoring control unit 36 judges whether or not control of the driving of the automobile 1 is necessary based on the state judgment result of the occupants such as the driver. For example, if it is judged that the driver is not in a state suitable for driving, the monitoring control unit 36 judges that control is necessary and proceeds to step ST18. In other cases, for example, if all the occupants including the driver are in a state suitable for traveling, the monitoring control unit 36 judges that control is unnecessary, skips step ST18, and proceeds to step ST19.
[0061] In step ST18, the monitoring and control unit 36 executes control over the running of the automobile 1 and the like. For example, when it is determined that the driver is not looking straight ahead or the head is not facing forward, the monitoring control unit 36 issues a warning to the driver. The warning to the driver may be, for example, a warning display on the liquid crystal device 32 or an alarm sound output from the speaker device 14. When the driver does not look straight ahead even after the warning is output, that is, when it is determined that the driver is not looking straight ahead continuously, the monitoring control unit 36 may switch the driving mode of the automobile 1 to automatic driving and decelerate and stop the driving of the automobile 1. When decelerating and stopping the driving of the automobile 1, the monitoring control unit 36 may turn on hazard lights (not shown) and transmit emergency information by the external vehicle communication device 18. For example, if it is determined that the driver is dozing or has a high pulse rate, the monitoring control unit 36 issues a warning to the driver or decelerates and stops the traveling of the automobile 1. When decelerating and stopping the traveling of the automobile 1, the monitoring control unit 36 may turn on hazard lamps (not shown) and transmit emergency information via the exterior communication device 18. In addition, if the driver has been driving continuously for a predetermined period of time or longer, if the eyes are opening and closing at a predetermined frequency, or if the head is tilted downward, the monitoring control unit 36 may determine that control is necessary in step ST17 and execute an output to encourage the driver to take a break, etc.
[0062] In step ST19, the monitoring control unit 36 judges whether or not to continue monitoring control of the occupant. The monitoring control unit 36 may judge whether the occupant has dismounted based on, for example, the detection of opening and closing by the door opening and closing sensor 11 and the image of the imaging sensor 41. If the occupant has dismounted, the monitoring control unit 36 judges not to continue monitoring control of the occupant and proceeds to step ST20. Otherwise, the monitoring control unit 36 judges to continue monitoring control of the occupant and returns the process to step ST15. The monitoring control unit 36 repeats the processes from step ST15 to step ST19 until it is determined not to continue monitoring control of the occupant.
[0063] In step ST20, the monitoring control unit 36 updates the occupant data registered in the memory 35 for the recognized occupant as a process when the occupant gets out of the vehicle. The monitoring control unit 36 acquires setting information for each part of the automobile 1 when the occupant gets out of the vehicle, and updates the occupant data registered in the memory 35. As a result, the occupant data registered in the memory 35 becomes adapted to the preferences of the occupant. The next time the occupant gets in, the latest settings for the occupant will be automatically set. Furthermore, the monitoring control unit 36 may temporarily record occupant data for an unregistered occupant in the memory 35. This allows the settings to be immediately linked if the occupant subsequently performs a registration operation.
[0064] In this way, in the occupant monitoring device 15 of this embodiment, the camera module 31 having the image sensor 41 is disposed behind the display operation panel 33 of the liquid crystal device 32 in the center portion in the vehicle width direction of the automobile 1, so that the driver and the passengers can be imaged by the single image sensor 41 and their respective states can be monitored. Moreover, the camera module 31 having the image sensor 41 is disposed near the display operation panel 33 of the liquid crystal device 32, and the state is determined using data obtained by previously imaging the occupant from the front or the like in addition to the current imaging data of the occupant, so that the influence of the imaging direction and imaging position of the occupant can be suppressed, and the state of the occupant can be determined more reliably than when the state is determined based only on the current imaging data.
[0065] Next, a response control to an operation by an occupant for the liquid crystal device 32 having the display operation panel 33 that can be used for such occupant monitoring will be described.
[0066] FIG. 9 is a flowchart of response control with respect to operations on the display / operation panel 33 of FIG. 3, which is provided in the center portion in the vehicle width direction of the vehicle body 2 and can be used for monitoring control. The monitoring control unit 36 of the occupant monitoring device 15 repeatedly executes the process of FIG. 9 in order to detect the operation of the display operation panel 33 of the liquid crystal device 32 by the occupant. In addition, an ECU of another device in the vehicle connected to the occupant monitoring device 15, for example an ECU of the other device shown in Figure 2, may obtain information on occupant operations on the display operation panel 33 of the liquid crystal device 32 through the monitoring control unit 36 and repeatedly execute the processing of Figure 9. As shown in FIG. 6, in the display area of the display operation panel 33, restriction areas 62 that prohibit swipe operations in the direction toward the camera module 31 are set on both the left and right sides and below the non-display section 33b in which the camera module 31 is located.
[0067] In step ST31, the monitoring control unit 36 determines whether or not an occupant has operated the display operation panel 33 of the liquid crystal device 32. If an occupant operation is not detected, the monitoring control unit 36 continues this process. Then, if an occupant operation is detected, the monitoring control unit 36 advances the process to step ST32.
[0068] In step ST32, the monitoring control unit 36 determines whether or not the automobile 1 is moving as the operating environment. If the automobile 1 is moving, the monitoring control unit 36 returns the process to step ST31 to prohibit operation on the display operation panel 33. If the automobile 1 is not moving, the monitoring control unit 36 advances the process to step ST33. Here, the monitoring control unit 36 may further determine whether or not the driver has operated the vehicle as the operating environment. In this case, the monitoring control unit 36 may return the process to step ST31 if the driver has operated the vehicle, and may advance the process to step ST33 if the passenger has operated the vehicle.
[0069] In step ST33, the monitoring control unit 36 determines the operation position operated by the occupant in the display area of the display operation panel 33 of the liquid crystal device 32. The display area of the display operation panel 33 has a main display area AM and a sub-display area AS, as shown in FIG. 6 for example. The sub-display area AS may be set on both the left and right sides of the non-display section 33b. The main display area AM can display operation objects 65 such as a plurality of operation buttons for various settings. The sub-display area AS can display operation objects 65 relating to the current time and various states of the automobile 1. The liquid crystal device 32 can display a plurality of operation objects 65 that can be swiped on the operation screen. In determining the operation position, the monitoring control unit 36 may determine which of the plurality of operation objects 65 being displayed has been operated.
[0070] In step ST34, the monitoring control unit 36 determines the operation content of the occupant. The occupant may basically perform, for example, a pressing click operation or a swiping operation to move up, down, left, or right on the operation object 65 displayed in the display area of the display operation panel 33 of the liquid crystal device 32. The monitoring control unit 36 may determine, as the operation content of the occupant, whether a click operation or a swipe operation has been performed based on the operation position or the position of the operation object 65 identified in step ST33. In addition, the monitoring control unit 36 may determine the operation direction from the operation position for a swipe operation to trace the surface of the display operation panel 33 toward the installation position of the imaging camera.
[0071] In step ST35, the monitoring control unit 36 judges whether or not the operation content is a swipe operation of tracing the surface of the display operation panel 33. If it is judged to be a swipe operation in step ST34, the monitoring control unit 36 advances the process to step ST36. If it is not judged to be a swipe operation in step ST34, the monitoring control unit 36 advances the process to step ST38.
[0072] In step ST36, the monitoring control unit 36 judges whether or not the swipe operation is an operation in a direction in which the swipe operation is restricted in an area in which the swipe operation is restricted. As shown in FIG. 6, in the display area of the display operation panel 33, restriction areas 62 that prohibit a swipe operation in a direction toward the camera module 31 are set on both the left and right sides and below the non-display section 33b in which the camera module 31 is arranged. The monitoring control unit 36 judges whether or not the swipe operation is in a direction toward the camera module 31. If the swipe operation is in a direction toward the camera module 31, the monitoring control unit 36 advances the process to step ST37. Otherwise, the monitoring control unit 36 advances the process to step ST38. 6, the control object 65 that partially overlaps with the restricted area 62 may be determined to be the same as the control object 65 included in the restricted area 62, or may be determined to be the same as the control object 65 not included in the restricted area 62. For example, when a sufficiently large restricted area 62 is set around the camera module 31, the control object 65 may be determined to be the same as the control object 65 not included in the restricted area 62. Moreover, the setting of the restricted area 62 is not limited to the pattern shown in FIG.
[0073] In step ST37, the monitoring control unit 36 does not execute response control to the operation according to the operation position and operation content of the occupant. If the operation content of the occupant is a swipe operation involving a movement toward the image sensor 41 in the peripheral part of the image sensor 41 in the display area, the monitoring control unit 36 does not execute control according to the operation content. The monitoring control unit 36 may output a warning window or the like to the liquid crystal device 32 to the effect that response control to the operation will not be executed. After that, the monitoring control unit 36 ends this control.
[0074] In step ST38, the monitor and control unit 36 executes response control to the operation by the occupant in accordance with the operation position and the operation content. For example, when the operation of the occupant is a click operation on a display object, the monitoring control unit 36 executes a response control that is associated in advance with the operation. For example, the monitoring control unit 36 may display an execution screen for the operated display object on the liquid crystal device 32. For example, when the operation by the occupant is a swipe operation on a display object, the monitoring control unit 36 executes a response control that is associated in advance with the operation. For example, the monitoring control unit 36 may display a setting screen for the operated display object on the liquid crystal device 32.
[0075] In this way, when a swipe operation is performed on the operation object 65 displayed around the imaging sensor 41 toward the installation position of the imaging sensor 41, the monitoring control unit 36 does not execute control according to the swipe operation. On the other hand, when a swipe operation is performed on the operation object 65 displayed outside the periphery of the imaging sensor 41, or when a swipe operation is performed around the imaging sensor 41 toward a location other than the imaging sensor 41, the monitoring control unit 36 executes control according to the swipe operation.
[0076] As described above, in this embodiment, when the operation performed by the occupant is an operation involving at least a movement toward the image sensor 41, the monitoring control unit 36 does not execute control according to the determined operation. Therefore, the occupant may not perform an operation involving a movement toward the image sensor 41 on the liquid crystal device 32 that displays the operation screen for the occupant of the automobile 1. As a result, in this embodiment, the field of view of the image sensor 41 is less likely to be blocked by the movement of the occupant's fingers, hands, arms, elbows, etc. when performing an operation. Moreover, the occupant is not restricted from performing an operation involving a movement as long as the occupant does not move toward the image sensor 41, and can perform an operation involving a movement on the operation screen displayed on the liquid crystal device 32. In this way, in this embodiment, swipe operations involving movement by the occupant are not uniformly restricted, so that ease of use for the occupant and monitoring of the occupant's condition can be achieved in a highly balanced manner. Thus, in the automobile 1, the liquid crystal device 32 used for passenger monitoring is required to satisfactorily balance ease of use for passengers with monitoring of the passengers' states.
[0077] The above-described embodiment is an example of a preferred embodiment of the present invention, but the present invention is not limited to this, and various modifications and changes are possible without departing from the gist of the invention.
[0078] For example, in the above-described embodiment, the liquid crystal device 32 is arranged vertically. Alternatively, for example, the liquid crystal device 32 may be arranged horizontally.
[0079] FIG. 10 is an explanatory diagram of a first modified example of the liquid crystal device 32 and the camera module 31 of the passenger monitoring device 15 provided in the central portion in the vehicle width direction of the vehicle body 2 of FIG. The display area of the liquid crystal device 32 in FIG. 10 is horizontally long. Moreover, camera module 31 having image sensor 41 is disposed on the rear side of non-display section 33b formed on the upper side of the display area of liquid crystal device 32, on the rear side of display operation panel 33. Camera module 31 is provided inside holding member 61 so as not to protrude outward from holding member 61 which holds the square corners of display operation panel 33. Even in this case, by setting a restriction area 62 that prohibits a swipe operation in the direction toward the camera module 31 on both the left and right sides and below the non-display section 33b in which the camera module 31 is located in the display area of the display operation panel 33, the same effect as in the above-described embodiment can be obtained.
[0080] FIG. 11 is an explanatory diagram of a second modified example of the liquid crystal device 32 and the camera module 31 of the passenger monitoring device 15 provided in the central portion in the vehicle width direction of the vehicle body 2 in FIG. The display area of the liquid crystal device 32 in FIG. 11 is horizontally long. Furthermore, camera module 31 having image sensor 41 is disposed on the back side of display operation panel 33, above the rectangular display area defined by liquid crystal device 32. Camera module 31 is provided inside holding member 61 so as not to protrude outside holding member 61 which holds the rectangular corner of display operation panel 33. Even in this case, the same effect as in the above-described embodiment can be obtained by setting a restriction area 62 that prohibits swipe operations in the direction toward the camera module 31 in the upper center of the display area of the display operation panel 33, where the camera module 31 is located adjacent to the display area.
[0081] In the above-described embodiment, the display operation panel 33 of the liquid crystal device 32 and the imaging sensor 41 are disposed at the center in the width direction of the automobile 1. The imaging sensor 41 has a wide-angle imaging lens 42 with an angle of view capable of capturing images of the heads of multiple occupants seated in the front row of the automobile 1 while being disposed at the center in the width direction of the automobile 1. The monitoring control unit 36 determines at least the open / closed state of the eyes of the occupants or their line of sight as the state of the occupants in the automobile 1. The passenger monitoring device may not fall into any of these categories. For example, the passenger monitoring device may monitor only the driver. Even in this case, it is expected that the reliability of the monitoring judgment can be improved by, for example, arranging the image sensor 41 on the back side of the display operation panel 33 of the liquid crystal device 32 so as to be located inside the circumscribing circle of the display area of the image, or by the monitoring control unit 36 using the registered image data recorded in the memory 35 as reference data to judge the state of the passenger in the automobile 1. [Explanation of symbols]
[0082] 1...Automobile (vehicle), 2...Vehicle body, 3...Vehicle interior, 4...Seat, 5...Dashboard, 6...Center console, 7...Steering, 10...Control device, 11...Door opening / closing sensor, 12...Vehicle speed sensor, 13...Park sensor, 14...Speaker device, 15...Occupant monitoring device, 16...Position setting device, 17...Driving assistance device, 18...External communication device, 19...Occupant protection device, 20...Air conditioning device, 21...In-vehicle network, 31...Camera module, 32...Liquid crystal device (display member), 33...Display operation panel, 33a...Rectangular Frame, 33b...non-display section, 34...input / output device, 35...memory, 36...monitoring control section, 41...imaging sensor (imaging member), 42...imaging wide-angle lens, 43...first LED, 44...first light projection lens, 45...second LED, 46...second light projection lens, 51...first occupant data, 52...second occupant data, 61...holding member, 62...restriction area, 65...operation object, AM...main display area (display area), AS...sub-display area (display area), Y0...center position of vehicle width, Yd...center position of liquid crystal device in vehicle width
Claims
1. A display member that displays an operation screen for a vehicle occupant; a control unit that executes control in accordance with an operation content determined as an operation by an occupant on the operation screen of the display member; an imaging member capable of imaging an occupant of the vehicle; having The control unit is When the operation content of the occupant is an operation involving at least a movement toward the imaging member, a control corresponding to the determined operation content is not executed. Vehicle occupant monitoring device.
2. the imaging member is disposed adjacent to a display area of the display member; The control unit is When the operation content of the occupant is an operation involving a movement toward the imaging member in a peripheral portion of the display area of the imaging member, control corresponding to the determined operation content is not executed.
2. A vehicle occupant monitoring device according to claim 1.
3. the display member has a panel for displaying the operation screen that is larger than a display area of the operation screen, The imaging member is provided on the rear side of the panel of the display member.
3. A vehicle occupant monitoring device according to claim 1 or 2.
4. The control unit is When the operation involving a movement toward the imaging member is an operation of tracing the surface of the display member toward the imaging member, no control corresponding to the determined operation content is executed.
4. A vehicle occupant monitoring device according to claim 1.
5. the display member displays a plurality of operation objects that can be swiped on the operation screen; The control unit is When a swipe operation is performed on the operation object displayed around the imaging member by tracing the operation object toward the imaging member, no control corresponding to the determined swipe operation is executed.
4. A vehicle occupant monitoring device according to claim 1.
6. The control unit is When the direction of the operation of tracing the surface of the display member is toward the imaging member, the control according to the determined operation content is not executed, When the direction of the operation of tracing the surface of the display member is a direction different from the direction of the imaging member, a control corresponding to the determined operation content is executed.
6. A vehicle occupant monitoring device according to claim 4 or 5.
7. The control unit is When the operation content of the occupant is an operation involving at least a movement toward the imaging member while the vehicle is traveling, control according to the determined operation content is not executed.
7. A vehicle occupant monitoring device according to any one of claims 1 to 6.
8. the display member and the imaging member are provided in a central portion of the vehicle in a vehicle width direction, The control unit monitors the state of an occupant of the vehicle based on the captured image of the imaging member. A vehicle occupant monitoring device according to any one of claims 1 to 7.
Citation Information
Patent Citations
On-vehicle electronic device, and operation control method of same
JP2007255971A
Display device for vehicle
JP2007326410A
Operation supporting device and method for on-vehicle equipment
JP2008195141A
Operation auxiliary device
JP2009186189A
Driver imagine device
JP2019014359A
