Occupant monitoring device, occupant monitoring system, and occupant monitoring method

Abstract

To provide an occupant monitoring technique capable of preventing an alarm indicating that an occupant of a vehicle is in an abnormal state from being issued when in fact the occupant is in a normal state in which he or she can drive the vehicle.SOLUTION: An occupant monitoring device comprises: an image acquisition part 111 for acquiring a photographed image in which the interior of a vehicle is captured by an imaging device; a position adjustment part 116 for outputting an adjustment instruction for adjusting a position or an attitude of a control target device which is disposed inside the vehicle; an image processing part 115 which is capable of processing an image of an occupant included in the photographed image; a determination part 112 for determining whether or not a first face part is included in the image of the occupant; and a detection part 114 which, when it is determined that the first face part is not included in the photographed image, detects a position of a blocking object overlapping the first face part and blocking the first face part in the photographed image. On the basis of the detected position of the blocking object, it performs control using at least one of the position adjustment part and the image processing part and thereby adjusts the position of the control target device or processes the photographed image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] This disclosure relates to occupant monitoring technology.

Background Art

[0002] To monitor the state of vehicle occupants, an occupant monitoring device is mounted on a vehicle and used. For example, Patent Document 1 discloses an in-vehicle imaging device aimed at surely capturing and imaging the face of a driver. Specifically, Patent Document 1 discloses an in-vehicle imaging device that captures the face of a driver, includes an imaging unit installed inside the vehicle to capture the face of the driver, a wheel position adjustment unit installed inside the vehicle to adjust the position of the steering wheel, an imaging position adjustment unit that adjusts at least one of the imaging position and imaging angle of the imaging unit, and a position adjustment control unit that controls at least one of the wheel position adjustment unit and the imaging position adjustment unit based on the imaging result of the imaging unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the technology of Patent Document 1, even if an attempt is made to adjust the position of the steering wheel or the imaging position of the imaging unit in order to surely capture the face of the driver, that is, to image without the steering wheel overlapping the face of the driver, there may be a case where it is impossible to make the steering wheel not overlap the face of the driver at all. In such a case, due to a part of the image of the driver's face being missing in the captured image, there is a problem that an unnecessary alarm, that is, an alarm indicating an abnormal state is issued by the occupant monitoring device even though the driver is in a normal state where driving is possible.

[0005] This disclosure is made to solve these problems and aims to provide crew monitoring technology that can suppress the generation of unnecessary alarms. [Means for solving the problem]

[0006] The occupant monitoring device according to the embodiment of the present disclosure includes: an image acquisition unit that acquires an image of the interior of a vehicle captured by an imaging device; a position adjustment unit that outputs an adjustment instruction for adjusting the position or attitude of a control target device located in the vehicle; an image processing unit capable of processing the image; a determination unit that determines whether or not a first face portion is included in the image of an occupant included in the image; and, if it is determined that the first face portion is not included in the image of the occupant, a detection unit that detects the position in the image of an obstruction that overlaps with and obstructs the first face portion. Based on the position of the detected obstruction, control is performed using at least one of the position adjustment unit and the image processing unit to adjust the position of the control target device or to process the image. [Effects of the Invention]

[0007] According to the occupant monitoring device of the embodiment of this disclosure, the generation of unnecessary alarms can be suppressed. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic side view of the interior of a vehicle equipped with a crew monitoring system, including a crew monitoring device. [Figure 2] This figure shows an example configuration of an occupant monitoring device, an occupant monitoring system, and a vehicle system. [Figure 3A] This figure shows an example of the hardware configuration of a crew monitoring system. [Figure 3B] This figure shows an example of the hardware configuration of a crew monitoring system. [Figure 4] This figure shows an example of captured images of the crew. [Figure 5]This is a flowchart showing the operation of the crew monitoring system. [Modes for carrying out the invention]

[0009] Various embodiments of this disclosure will be described in detail below with reference to the attached drawings. Components that are denoted by the same or similar reference numerals in the drawings have the same or similar configuration or function, and redundant descriptions of such components will be omitted.

[0010] Embodiment 1. <Configuration of occupant monitoring device, occupant monitoring system, and vehicle system> Referring to Figure 1, an overview of the occupant monitoring device, occupant monitoring system, and vehicle system according to this disclosure will be described. As shown in Figure 1, the driver D, who is an occupant of the vehicle, is seated in the seat 4. The driver D steers the vehicle by operating the steering wheel 3 while looking at the information displayed on the meter panel 5. The steering wheel 3 has a steering column 31 that encloses the steering shaft, a steering wheel 32 located at the end of the steering shaft, and a horn pad 33 located at the end of the steering shaft and in the center of the steering wheel 32. An occupant monitoring system 1 is provided in the vehicle cabin to monitor the state of the driver D. The occupant monitoring system 1 includes an imaging device 12 (see Figure 2), which is positioned to capture images of the driver, who is an occupant of the vehicle. For example, the occupant monitoring system 1 may be located on the steering column 31 as shown in Figure 1, or on or within the meter panel 5. The imaging device 12 captures images of the interior of the vehicle and outputs the captured images to the occupant monitoring device 11.

[0011] The crew monitoring system 1 further includes a crew monitoring device 11 (see Figure 2), which will be described later. The imaging device 12 is configured to move in the front-rear direction along the double-ended arrow A1 parallel to the axial direction of the steering column 31, in the up-down direction along the double-ended arrow A2 perpendicular to the axial direction of the steering column 31, and clockwise / counterclockwise along the double-ended arrow A3, based on instructions from the crew monitoring device 11. In other words, the position or orientation of the imaging device 12 is adjusted based on instructions from the crew monitoring device 11.

[0012] Furthermore, the steering wheel 32 and horn pad 33, located at the end of the steering shaft, are configured to telescopically move in the front-to-back direction along the double-ended arrow A4 parallel to the axial direction of the steering column 31, and to tilt vertically along the double-ended arrow A5 perpendicular to the axial direction of the steering column 31, based on instructions from the occupant monitoring device 11. In other words, the position or orientation of the steering wheel 32 and horn pad 33 is adjusted based on instructions from the occupant monitoring device 11.

[0013] Furthermore, the seat 4 is configured to move in the longitudinal direction along the double-ended arrow A6 parallel to the longitudinal direction of the vehicle, in the vertical direction along the double-ended arrow A7 perpendicular to the longitudinal direction of the vehicle, and clockwise / counterclockwise along the double-ended arrow A8, based on instructions from the occupant monitoring device 11. In other words, the position or orientation of the seat 4 is adjusted based on instructions from the occupant monitoring device 11.

[0014] Furthermore, the meter panel 5 is configured to be movable clockwise / counterclockwise along the arrows A9 at both ends, based on instructions from the crew monitoring device 11. That is, the position or orientation of the meter panel 5 is adjusted based on instructions from the crew monitoring device 11. In addition, the size or arrangement of the content displayed in the display area of ​​the meter panel 5 may be configured to be changeable based on instructions from the crew monitoring device 11, according to the position or orientation of various controlled devices.

[0015] Next, referring to FIG. 2, a more detailed configuration of the occupant monitoring device 11, the occupant monitoring system 1, and the vehicle system will be described. As shown in FIG. 2, the occupant monitoring system 1 includes an imaging device 12 that captures an image inside the vehicle cabin, an occupant monitoring device 11 that outputs a control signal for controlling various devices based on an image of an occupant included in the image captured by the imaging device 12, a memory 13 that stores an image of the occupant or a general-purpose image, a CAN transceiver 14 that mediates the transmission of signals between the occupant monitoring device 11 and the vehicle-side ECU 21, and a projection device 15 that projects a predetermined image. Examples of the devices to be controlled include the imaging device 12, the steering wheel 3, the seat 4, and the meter panel 5.

[0016] As shown in FIG. 2, the vehicle system includes an occupant monitoring system 1, a vehicle-side ECU 21, a motor 22 that drives the imaging device 12, a steering wheel 3, a motor 23 that drives the steering wheel 3, a seat 4, a motor 24 that drives the seat 4, a meter panel 5, a display driver 25 that performs display control of the meter panel 5, and a motor 26 that mechanically drives the meter panel 5. Actuators other than motors may be used.

[0017] Further, the vehicle system includes an input device 27 that receives operation information related to the operability input by the user. The operation information is, for example, information regarding the positions of the steering wheel 3, the seat 4, the meter panel 5, or whether the display area of the content displayed on the meter panel 5 is appropriate. Information indicating how inappropriate it is may be included if it is not appropriate. The input device 27 outputs the received operation information to the vehicle-side ECU 21, and the operation information is transmitted to the occupant monitoring device 11.

[0018] <Configuration of the Occupant Monitoring Device> As shown in FIG. 2, the occupant monitoring device 11 includes an image acquisition unit 111, a determination unit 112, a vehicle information acquisition unit 113, a detection unit 114, an image processing unit 115, and a position adjustment unit 116.

[0019] (Image Acquisition Unit) The image acquisition unit 111 acquires the captured image captured by the imaging device 12 and supplies the acquired captured image to the determination unit 112.

[0020] (Determination unit) When a passenger is reflected in the captured image acquired by the image acquisition unit 111, the determination unit 112 determines whether or not a predetermined facial part of the passenger (hereinafter sometimes referred to as the "first facial part") is included. This determination may be made by the determination unit 112 extracting the feature amount of the face from the captured image using a known method, or by inputting the captured image into a learned model that has learned the relationship between the face image and the facial part. Examples of the predetermined facial part include all or part of the parts constituting the face such as eyes, mouth, nose, ears, etc. When it is determined that the predetermined facial part is not included in the captured image, the determination unit 112 outputs a determination result indicating that the predetermined facial part is not included in the captured image to the detection unit 114. As an example, as shown in FIG. 4, when the mouth of the passenger is shielded by the horn pad 33 which is an example of the shielding object and the mouth of the passenger is not included in the captured image IMG, the determination unit 112 outputs a determination result indicating that the mouth is not included in the captured image IMG to the detection unit 114. Other examples of the shielding object may include all kinds of objects such as parts related to the steering such as the steering boss, the steering spoke, the steering wheel 32, the arm of the driver D, the mask worn by the driver D, etc.

[0021] Also, the determination unit 112 may be configured to determine whether or not a predetermined facial part of the passenger is included in the processed image output from the image processing unit 115.

[0022] (Vehicle information acquisition unit) The vehicle information acquisition unit 113 acquires state information regarding the state of the control target device from the vehicle - side ECU 21 via the CAN transceiver 14, and outputs the acquired state information to the detection unit 114. Examples of the state information include position information indicating the position of the control target device, and angle information indicating the rotation angle or tilt angle of the control target device.

[0023] (Detection unit) If the detection unit 114 receives a determination result from the determination unit 112 indicating that a predetermined face area is not included in the captured image IMG, it detects the position in the captured image IMG of an obstruction that overlaps with and obscures the predetermined face area. In the example in Figure 4, the position of the horn pad 33 in the captured image IMG is detected.

[0024] Furthermore, the detection unit 114 determines whether to displace the controlled device based on the position of the detected obstruction. For example, it determines whether to displace the controlled device by determining whether displacing the controlled device based on the position of the obstruction will prevent the obstruction from appearing in the captured image.

[0025] When the detection unit 114 determines that the controlled device should be displaced, it outputs a displacement instruction to the position adjustment unit 116, along with information indicating the position of the obstruction.

[0026] On the other hand, if the detection unit 114 determines that the controlled device should not be displaced, it outputs a correction instruction to the image processing unit 115 to prevent any obstructing objects visible in the captured image from being displayed.

[0027] (Image Processing Department) When the image processing unit 115 receives a correction instruction from the detection unit 114, it extracts face image data corresponding to a predetermined face part that has been determined to be missing from a previously captured image of driver D and superimposes it onto the captured image. The previously captured image of driver D is obtained from the memory 13. Figure 4 shows a captured image IMG in which the lower part of driver D's face is missing due to a blind spot of the horn pad 33. The image processing unit 115 extracts face image data corresponding to the missing part from a previously captured image and superimposes it onto the captured image IMG to fill in the missing part. By estimating the size of driver D's face from the normally captured area and applying the interpolation data to the missing area before performing sensing, it is possible to suppress a decrease in the accuracy or robustness of occupant detection. Note that general-purpose face image data may be used as the interpolation data to be superimposed.

[0028] As another example, image data surrounding a specific facial area that is determined not to be included in the captured image may be extracted from previously captured images of driver D and superimposed onto the captured image. For example, if driver D is wearing a mask, it becomes difficult to determine driver D's facial orientation or whether they are looking away because it is not possible to sense both driver D's mouth and nose or the nose itself. In such cases, there is a technique that enables the determination of facial orientation or looking away by interpolating an area of ​​about 1-2 cm below driver D's chin. Therefore, instead of interpolating images of driver D's mouth and nose, image data of the area below the chin, which is the area around the mouth and nose that is determined not to be included in the captured image IMG, may be extracted from previously captured images of driver D and superimposed onto the captured image. Note that general-purpose image data of the area around the face may be used as the interpolated data to be superimposed.

[0029] Such image processing is performed when the vehicle is stopped or in motion. If the position or orientation of the controlled device changes while the vehicle is in motion, it will interfere with the driver's operation. Therefore, by performing image processing without changing the position or orientation of the controlled device while the vehicle is in motion, it is possible to maintain operability by the driver while the vehicle is in motion while suppressing the issuance of unnecessary warnings. When image processing is performed only while the vehicle is in motion, the image processing unit 115 acquires vehicle speed information from the vehicle-side ECU 21 via the vehicle information acquisition unit 113 to determine whether the vehicle is in motion or not.

[0030] As a simpler mechanism than the method of superimposing complementary data, a picture resembling the lower part of the driver's face may be drawn in advance on the back or side of the horn pad 33.

[0031] Furthermore, if the image processing unit 115 receives a correction instruction, it may perform projection control of the projection device 15 so as to project the face of driver D, which overlaps with the obstacle, onto the obstacle in the captured image IMG.

[0032] (Position adjustment part) When the position adjustment unit 116 receives a displacement instruction from the detection unit 114, it acquires status information of the controlled device from the vehicle information acquisition unit 113. Then, based on the position of the obstruction and the status information of the controlled device, the position adjustment unit 116 estimates the position of the occupant's face, calculates the controlled device to be displaced and the amount of displacement, generates an adjustment instruction to adjust the position or attitude of the controlled device, and outputs the generated adjustment instruction. The position of the occupant's face is estimated, for example, from the geometric relationship between the position of the steering wheel 3, the position of the imaging device 12, and the position of the seat 4, and the face area in the captured image. Furthermore, the controlled device to be displaced and the amount of displacement are calculated based on the position of the obstruction in the captured image and the estimated face position.

[0033] For example, in the case of the captured image IMG in Figure 4, the position adjustment unit 116 outputs all or part of the following adjustment instructions: an adjustment instruction to move the position of the imaging device 12 toward the driver D along the double-ended arrow A1; an adjustment instruction to raise the position of the imaging device 12 along the double-ended arrow A2; an adjustment instruction to move the imaging device 12 counterclockwise along the double-ended arrow A3; an adjustment instruction to shorten the steering shaft by telescopic movement along the double-ended arrow A4; an instruction to tilt the steering wheel 32 and horn pad 33 downward along the double-ended arrow A5; an adjustment instruction to move the seat 4 toward the rear of the vehicle along the double-ended arrow A6; an adjustment instruction to raise the seat 4 along the double-ended arrow A7; or an adjustment instruction to move the seat 4 counterclockwise along the double-ended arrow A8. If the imaging device 12 is installed on the meter panel 5, the position adjustment unit 116 may also output an adjustment instruction to move the meter panel 5 along the double-ended arrow A9.

[0034] Furthermore, the position adjustment unit 116 may output change instructions to change the size or arrangement of the content displayed in the display area of ​​the meter panel 5 according to the position or orientation of various controlled devices.

[0035] Such adjustment or modification instructions are output, for example, while the vehicle is stopped, such as immediately after turning on the vehicle's ignition. By adjusting the position or attitude of the controlled device based on the output of the adjustment instruction while the vehicle is stopped, it is possible to prevent interference with the driver D's operation. As another example, the adjustment instruction is output while the vehicle is in motion. Even when the adjustment instruction is output while the vehicle is in motion, by adjusting the position or attitude of the controlled device while the vehicle is stopped, it is possible to prevent interference with the driver D's operation. When the output of the adjustment or modification instruction is based on whether the vehicle is stopped or in motion, the position adjustment unit 116 obtains vehicle speed information from the vehicle-side ECU 21 via the vehicle information acquisition unit 113 to determine whether the vehicle is stopped or in motion. Note that since the position or attitude of the imaging device 12 does not interfere with the driver D's operation, the output of the adjustment instruction to adjust the position of the imaging device 12, and the adjustment of the position or attitude of the imaging device 12 based on this output, may be performed while the vehicle is in motion.

[0036] (CAN transceiver) The CAN transceiver 14 receives information regarding the status of the controlled device from the vehicle-side ECU 21 and transmits it to the vehicle information acquisition unit 113.

[0037] Furthermore, the CAN transceiver 14 transmits the adjustment instructions output from the position adjustment unit 116 to the vehicle-side ECU.

[0038] The vehicle-side ECU 21 controls the controlled device by driving motors 22-24, 26 or a display driver 26, which are provided in accordance with the controlled device, according to adjustment instructions received from the CAN transceiver 14. For example, while the vehicle is stationary, the vehicle-side ECU 21 adjusts the position or orientation of at least one of the steering wheel 3, seat 4, meter panel 5, and imaging device 12, or the size or arrangement of the content displayed in the display area of ​​the meter panel 5. The vehicle-side ECU 21 also controls the display driver 25 to display a question on the meter panel 5 asking whether the position of the controlled device after displacement is appropriate. The input device 27 receives the response to the question entered by the driver D as operability information and outputs the operability information to the vehicle-side ECU 21. The input device 27 may be a device that accepts touch input or a microphone that accepts voice input. The vehicle-side ECU 21 transmits the operability information received via the input device 27 to the occupant monitoring device 11. The crew monitoring device 11 terminates the displacement of the controlled device if the operability information indicates that the position of the controlled device after displacement is appropriate, and adjusts the amount of displacement of the controlled device and performs position adjustment of the controlled device again if the operability information indicates that the position of the controlled device after displacement is not appropriate.

[0039] Next, the hardware configuration of the crew monitoring device 11 will be described with reference to Figures 3A and 3B. As an example, as shown in Figure 3A, the crew monitoring device 11 is implemented by a processing circuit 100a. The processing circuit 100a implements the functions of the functional parts of the crew monitoring device 11. The processing circuit 100a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of each functional part of the crew monitoring device 11 may be implemented by separate processing circuits, or these functions may be implemented together by a single processing circuit.

[0040] As another example, as shown in Figure 3B, the crew monitoring device 11 is implemented by a processor 100b and a memory 100c. The functions of the functional parts of the crew monitoring device 11 are realized when a program stored in the memory 100c is read by the processor 100b and executed. The program is implemented as software, firmware, or a combination of software and firmware. Examples of memory 100c include, for example, non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically-EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidiscs, and DVDs.

[0041] <Operation> Next, the operation of the crew monitoring device 11 will be explained with reference to Figure 5.

[0042] In step ST1, the image acquisition unit 111 acquires the captured image from the imaging device 12.

[0043] In step ST2, the determination unit 112 determines whether the captured image acquired by the image acquisition unit 111 contains a predetermined facial area of ​​the occupant. If it is determined that the captured image contains a predetermined facial area of ​​the occupant, the process ends.

[0044] If it is determined that the captured image does not contain a predetermined facial area of ​​the occupant, the process proceeds to step ST3, and the detection unit 114 detects the position of the obstruction in the captured image.

[0045] Subsequently, the process proceeds to step ST4, where the detection unit 114 determines whether or not to displace the controlled device based on the position of the detected obstruction.

[0046] If the controlled device is not displaced, the process proceeds to step ST13, where the image processing unit 115 acquires face image data corresponding to a predetermined face part that was determined not to be included in the captured image from memory 13 as complementary data, and applies the acquired image data to the captured image. In other words, the acquired face image data is superimposed on the captured image. The complementary captured image is output to the determination unit 112 as a processed image, for example, and used for determination by the determination unit 112.

[0047] When displacing the controlled device, the process proceeds to step ST5, where the vehicle information acquisition unit 113 acquires status information regarding the state of the controlled device from the vehicle-side ECU 21.

[0048] Following step ST5, in step ST6, the position adjustment unit 116 estimates the position of the occupant's face based on the captured image acquired by the image acquisition unit 111 and the state information acquired by the vehicle information acquisition unit 113, and displaces the controlled device. Specifically, the position adjustment unit 116 calculates the controlled device to be displaced and the amount of displacement based on the position of the obstruction and the state information of the controlled device, generates an adjustment instruction to adjust the position or attitude of the controlled device, and outputs the generated adjustment instruction. Based on this adjustment instruction, the vehicle-side ECU 21 controls actuators such as the motor 23 to displace the controlled device.

[0049] Following step ST6, in step ST7, the vehicle information acquisition unit 113 acquires operability information, which is input via the input device 27, indicating whether the position of the controlled device after displacement is appropriate.

[0050] In step ST8, following step ST7, the detection unit 114 determines whether the operability information indicates that the position of the controlled device after displacement is appropriate. If the operability information indicates that the position of the controlled device after displacement is appropriate, the process proceeds to step ST9. In step ST9, the image acquisition unit 111 acquires images of the occupants taken by the imaging device 12 after the position of the controlled device has been displaced from the imaging device 12.

[0051] If the operability information indicates that the position of the controlled device after displacement is not appropriate, the process proceeds to step ST10, where the position adjustment unit 116 adjusts the amount of displacement of the controlled device and performs position adjustment of the controlled device again. After the process in step ST10 is completed, the process proceeds to step ST9.

[0052] In step ST11, following step ST9, the determination unit 112 determines whether a predetermined facial area of ​​the occupant is included in the image newly acquired by the image acquisition unit 111. That is, it determines whether a predetermined facial area of ​​the occupant is included in the image taken after the position of the controlled device has been displaced. If the predetermined facial area of ​​the occupant is included in the newly acquired image, the process in Figure 5 is terminated.

[0053] If the newly acquired image does not include a predetermined facial area of ​​the crew member, in step ST12, the image processing unit 115 acquires facial image data corresponding to the predetermined facial area that was determined to be missing from the image as supplementary data from the memory 13, and applies the acquired facial image data to the newly acquired image. The supplemented image is output to the determination unit 112 as a processed image, for example, and used for determination by the determination unit 112.

[0054] <Note> Some aspects of the various embodiments described above are summarized below.

[0055] (Note 1) The occupant monitoring device described in Appendix 1 includes an image acquisition unit (111) that acquires an image of the interior of a vehicle captured by an imaging device, a position adjustment unit (116) that outputs an adjustment instruction for adjusting the position or attitude of a control target device located inside the vehicle, an image processing unit (115) capable of processing the image, a determination unit (112) that determines whether or not a first face portion is included in the image of the occupant included in the image, and a detection unit (114) that, if it is determined that the first face portion is not included in the image of the occupant, detects the position in the image of an obstruction that overlaps with and obstructs the first face portion. Based on the position of the detected obstruction, control is performed using at least one of the position adjustment unit and the image processing unit to adjust the position or attitude of the control target device or to process the image.

[0056] With this configuration, which includes both a position adjustment unit (116) and an image processing unit (115), when an obstruction acts as a barrier to the imaging device, and the position adjustment unit (116) alone cannot prevent the obstruction from appearing in the captured image, especially when it is not possible to prevent the obstruction from appearing in the captured image while maintaining appropriate driver operability, the image processing unit (115) can process the area of ​​the obstruction from the image of the occupant used for occupant monitoring by the occupant monitoring device or occupant monitoring system to monitor the driver's condition. Therefore, it is possible to prevent the occupant monitoring device or occupant monitoring system from determining that the driver is in an abnormal state due to the presence of an obstruction in the image, thereby suppressing the issuance of unnecessary alarms by the occupant monitoring device or occupant monitoring system.

[0057] (Note 2) The occupant monitoring device in Appendix 2 is the occupant monitoring device described in Appendix 1, wherein the image processing unit processes the captured image while the vehicle is in motion.

[0058] According to the occupant monitoring device described in Appendix 2, the processing of the captured image is performed while the vehicle is in motion. This makes it possible to process the captured image without changing the position or orientation of the controlled device while the vehicle is in motion. This makes it possible to suppress the issuance of unnecessary alarms while maintaining operability by the occupant D while the vehicle is in motion. In particular, it can address changes in the occupant D's posture while the vehicle is in motion. That is, even if a facial image of the occupant D including a predetermined facial area is obtained while the vehicle is stopped, and the occupant D loses their posture while the vehicle is in motion, causing the predetermined facial area to enter the blind spot of the steering wheel 32 or horn pad 33, the issuance of an alarm can be suppressed by processing the captured image.

[0059] (Note 3) The occupant monitoring device in Appendix 3 is the occupant monitoring device described in Appendix 1 or 2, and the shielding is the horn pad of the vehicle.

[0060] According to the occupant monitoring device described in Appendix 3, the driver's condition can be monitored even if the horn pad is obstructing the imaging device.

[0061] (Note 4) The occupant monitoring device in Appendix 4 is an occupant monitoring device described in any one of Appendix 1 to 3, wherein the controlled device includes a steering wheel.

[0062] According to the occupant monitoring system described in Appendix 4, the system can recognize the driver's face by adjusting the position of the steering wheel.

[0063] (Note 5) The crew monitoring device in Appendix 5 is a crew monitoring device described in any one of Appendix 1 to 4, and the controlled device includes a seat.

[0064] According to the occupant monitoring device described in Appendix 5, the occupant monitoring device can recognize the driver's face by adjusting the position of the seat.

[0065] (Note 6) The crew monitoring device in Appendix 6 is a crew monitoring device described in any one of Appendix 1 to 5, and the controlled device includes the imaging device.

[0066] According to the occupant monitoring system described in Appendix 6, the occupant monitoring system can recognize the driver's face by adjusting the position of the imaging device.

[0067] (Note 7) The crew monitoring device in Appendix 7 is a crew monitoring device described in any one of Appendix 1 to 6, wherein the controlled device includes a meter panel.

[0068] According to the occupant monitoring device described in Appendix 7, the occupant monitoring device can recognize the driver's face by adjusting the position of the meter panel. Furthermore, the visibility requirements for the meter panel can also be met.

[0069] (Note 8) The crew monitoring device in Appendix 8 is the crew monitoring device described in Appendix 7, wherein the position adjustment unit instructs a change in the size or arrangement of the content displayed in the display area of ​​the meter panel according to the position or orientation of the controlled device.

[0070] According to the crew monitoring device in Appendix 8, even if the visibility of the content displayed in the meter panel display area is obstructed by displacing the position or orientation of the controlled device, visibility can be improved by changing the size or arrangement of the content displayed in the meter panel display area.

[0071] (Note 9) The occupant monitoring device in Appendix 9 is an occupant monitoring device described in any one of Appendix 3 to 5, 7, and 8, wherein the position adjustment unit outputs the adjustment instruction when the vehicle is stopped.

[0072] According to the occupant monitoring device in Appendix 9, the position or attitude of the controlled device is adjusted while the vehicle is stopped based on the outputted adjustment instructions, thereby preventing interference with the operation by occupant D.

[0073] (Note 10) The crew monitoring device in Appendix 10 is a crew monitoring device described in any one of Appendix 1 to 9, wherein the image processing unit is configured to project face image data corresponding to the first face portion so as to be superimposed on the position of the detected obstruction when it is determined that the image of the crew does not include the first face portion.

[0074] According to the occupant monitoring device described in Appendix 10, by projecting an image of the face area that overlaps with the horn pad (obstruction) onto the horn pad, the occupant monitoring device can recognize the driver's face even when the imaging range of the imaging device is obstructed by the horn pad.

[0075] (Note 11) The crew monitoring system described in Appendix 11 comprises the imaging device and a crew monitoring device described in any one of Appendix 1 to 10.

[0076] (Note 12) The occupant monitoring method described in Appendix 12 is an occupant monitoring method for an occupant monitoring device comprising an image acquisition unit, a position adjustment unit, an image processing unit, a determination unit, and a detection unit, comprising the steps of: the image acquisition unit acquiring an image of the interior of a vehicle captured by an imaging device (ST1); the position adjustment unit outputting adjustment instructions for adjusting the position or attitude of a control target device located inside the vehicle (ST6, ST10); the image processing unit processing the image (ST12, ST13); and the determination unit determining the image of an occupant included in the image. The system includes the steps of: determining whether or not a first face portion is included (ST2); detecting, if the detection unit determines that the first face portion is not included in the image of the occupant, detecting the position in the captured image of an obstruction that overlaps with and obstructs the first face portion; and, based on the position of the detected obstruction, outputting an adjustment instruction for the position of the controlled device by the position adjustment unit, or processing the captured image by the image processing unit (ST6, ST10, ST12, ST13).

[0077] Furthermore, it is possible to combine embodiments, or to modify or omit each embodiment as appropriate. [Industrial applicability]

[0078] The occupant monitoring device of this disclosure can be mounted on a vehicle and used as a device to detect whether or not the vehicle is capable of being driven by its occupants. [Explanation of Symbols]

[0079] 1 Occupant monitoring system, 3 Steering wheel, 4 Seat, 5 Meter panel, 11 Occupant monitoring device, 12 Imaging device, 13 Memory, 14 CAN transceiver, 15 Projection device, 21 Vehicle-side ECU, 22 Motor, 23 Motor, 24 Motor, 25 Display driver, 26 Motor, 27 Input device, 31 Steering column, 32 Steering wheel, 33 Horn pad, 100a Processing circuit, 100b Processor, 100c Memory, 111 Image acquisition unit, 112 Judgment unit, 113 Vehicle information acquisition unit, 114 Detection unit, 115 Image processing unit, 116 Position adjustment unit.

Claims

1. an image acquisition unit that acquires an image of the interior of the vehicle captured by an imaging device; a position adjustment unit that outputs an adjustment instruction for adjusting the position or attitude of a control target device disposed in the vehicle; an image processing unit capable of processing the captured image; a determination unit that determines whether a first facial region is included in an image of the occupant included in the captured image; a detection unit that detects a position in the captured image of an obstruction that overlaps with the first facial portion and obstructs the first facial portion when it is determined that the first facial portion is not included in the image of the occupant; Equipped with performing control using at least one of the position adjustment unit and the image processing unit based on the position of the detected obstruction, thereby adjusting the position or attitude of the control target device or processing the captured image; Occupant monitoring device.

2. The passenger monitoring device according to claim 1 , wherein the image processing unit processes the captured image while the vehicle is traveling.

3. 3. The passenger monitoring device according to claim 1, wherein the shielding object is a horn pad of the vehicle.

4. The occupant monitoring device according to claim 1 , wherein the device to be controlled includes a steering wheel.

5. The occupant monitoring device according to claim 1 , wherein the controlled device includes a seat.

6. The passenger monitoring device according to claim 1 , wherein the device to be controlled includes the imaging device.

7. The passenger monitoring system according to claim 1 , wherein the device to be controlled includes a meter panel.

8. The occupant monitoring device according to claim 7 , wherein the position adjustment unit instructs a change in size or arrangement of content displayed in the display area of ​​the meter panel in accordance with the position or attitude of the device to be controlled.

9. 9. The passenger monitoring device according to claim 3, wherein the position adjustment unit outputs the adjustment instruction when the vehicle is stopped.

10. the image processing unit is configured to project face image data corresponding to the first face region so as to be superimposed on a position of the detected obstruction when it is determined that the first face region is not included in the image of the occupant. An occupant monitoring device according to any one of claims 1 to 9.

11. the imaging device; An occupant monitoring device according to any one of claims 1 to 10; An occupant monitoring system comprising:

12. An occupant monitoring method for an occupant monitoring device including an image acquisition unit, a position adjustment unit, an image processing unit, a determination unit, and a detection unit, an image acquisition unit acquiring an image of the interior of the vehicle captured by an imaging device; a step of outputting an adjustment instruction by a position adjustment unit to adjust the position or attitude of a control target device disposed in the vehicle; an image processing unit capable of processing the captured image; a determination unit determining whether a first facial region is included in an image of the occupant included in the captured image; a step in which, when it is determined that the first facial region is not included in the image of the occupant, the detection unit detects a position in the captured image of an obstruction that overlaps with the first facial region and obstructs the first facial region; Equipped with The method further includes a step in which the position adjustment unit outputs an adjustment instruction for adjusting the position of the control target device based on the detected position of the obstruction, or the image processing unit processes the captured image. Occupant monitoring methods.

Images