INTANGIBLE MONITORING DEVICE, INTANGIBLE MONITORING PROCEDURES AND MEDIUM

Abstract

An occupant monitoring device (2), comprising: an image acquisition unit (3) for capturing a recording image in which an interior of a vehicle is recorded by an imaging unit (8) provided in a movable unit in the vehicle; a scanning unit (4) for performing a scanning process which is at least a determination of the state of an occupant or a personal authentication of the occupant by image processing of the captured image; a position estimation unit (5) for estimating whether or not there is a change in the position of the imaging unit (8); a calibration unit (6) for performing a calibration for image processing in the scanning unit (4); and a determination unit (7) to prevent scanning by the scanning unit (4) and to cause calibration by the calibration unit (6) when the position estimation unit (5) estimates that there is a change in the position of the imaging unit (8).

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an occupant monitoring device, an occupant monitoring method, and a medium. BACKGROUND

[0002] An occupant monitoring device acquires a captured image of a vehicle occupant and performs image processing on the acquired image, thereby performing various types of occupant scanning. For example, Patent Literature 1 describes detecting a driver's gaze direction from a captured image and determining whether distracted driving occurs. Patent Literature 1 describes performing calibration to correct a detection error in the gaze direction caused by individual differences of a driver. Furthermore, Patent Literature 2 describes measuring a position and a direction of an imaging unit and performing calibration according to the measured position and direction of the imaging unit. CITATION LISTPATENT LITERATURE Patent Literature 1: JP 2005-313843 A Patent Literature 2: JP 2010-181209 A SUMMARY OF THE INVENTION OBJECT

[0003] Here, the imaging unit is generally arranged in a steering column or a center console. However, since many devices are arranged on the steering column and center console in addition to the imaging unit, the layout of the imaging unit is limited. Accordingly, it is conceivable to arrange the imaging unit on a movable unit, such as a rearview mirror, which provides more layout space than the steering column and center console. In this case, even if calibration is performed by the method described in Patent Literature 1 or Patent Literature 2, the positional relationship between the imaging unit and an occupant changes when the occupant corrects the angle of the rearview mirror, resulting in a problem that the scanning performance deteriorates.

[0004] The present disclosure has been made to solve the problems described above, and an object thereof is to suppress deterioration of scanning performance due to a change in the position of an imaging unit in an occupant monitoring apparatus in which the imaging unit is provided in a movable unit in a vehicle. SOLUTION TO THE TASK

[0005] An occupant monitoring device according to the present disclosure includes an image acquisition unit for acquiring a captured image in which an interior of a vehicle is captured by an imaging unit provided in a movable unit in the vehicle, a scanning unit for performing scanning that is at least a state determination of an occupant or a personal authentication of the occupant by image processing of the captured image, a position estimation unit for estimating whether or not there is a change in a position of the imaging unit, a calibration unit for performing calibration for image processing in the scanning unit, and a determination unit for not causing the scanning unit to be performed and causing the calibration unit to be performed when the position estimation unit estimates,that there is a change in the position of the imaging unit.

[0006] An occupant monitoring method according to the present disclosure includes acquiring a captured image in which an interior of a vehicle is captured by an imaging unit provided in a movable unit in the vehicle; performing scanning that performs at least one of state determination of an occupant and personal authentication of the occupant by image processing of the captured image; estimating whether or not there is a change in a position of the imaging unit; performing calibration for the image processing; and performing the calibration without performing the scanning when it is estimated that there is a change in the position of the imaging unit.

[0007] Regarding a non-transitory computer-readable medium having an executable program stored thereon, the program instructs a computer to perform the following: acquiring a captured image in which an interior of a vehicle is captured by an imaging unit provided in a movable unit in the vehicle; performing scanning that is at least one of a state determination of an occupant and personal authentication of the occupant by image processing of the captured image; estimating whether or not there is a change in a position of the imaging unit; performing calibration for the image processing; and performing the calibration without performing the scanning when it is estimated that there is a change in the position of the imaging unit. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0008] According to the present disclosure, in an occupant monitoring device in which an imaging unit is provided in a movable unit in a vehicle, when it is estimated that there is a change in a position of the imaging unit, sampling is not performed and calibration is performed, and thereby it is possible to suppress a decrease in sampling performance caused by the change in the position of the imaging unit. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram illustrating a portion of a vehicle including an occupant monitoring device according to a first embodiment. Fig. 2A and Fig. 2B are block diagrams illustrating an example of a hardware configuration of the occupant monitoring device according to the first embodiment. Fig. 3 is a flowchart illustrating the processing procedure of the occupant monitoring device according to the first embodiment. Fig. 4 is a block diagram illustrating a portion of a vehicle including an occupant monitoring device according to a second embodiment. Fig. 5 is a flowchart illustrating the processing procedure of the occupant monitoring device according to the second embodiment. Fig. 6 is a flowchart illustrating the processing procedure of an occupant monitoring device according to a third embodiment. Fig. 7 is a block diagram illustrating a portion of a vehicle including an occupant monitoring device according to a fourth embodiment. Fig. 8 is a flowchart illustrating the processing procedure of the occupant monitoring device according to the fourth embodiment. DESCRIPTION OF EMBODIMENTSFirst embodiment.

[0009] An occupant monitoring device 2 according to a first embodiment will be described with reference to Fig. 1 described. Fig. Figure 1 is a block diagram illustrating a portion of a vehicle 1 with the occupant monitoring device 2 according to the first embodiment. The occupant monitoring device 2 is a driver monitoring system (DMS) and performs various scans for an occupant in the vehicle 1 by image processing for the occupant. Fig. The vehicle 1 shown in Figure 1 comprises an occupant monitoring device 2, an imaging unit 8, and a notification unit 9. The occupant monitoring device 2 comprises an image acquisition unit 3, a scanning unit 4, a position estimation unit 5, a calibration unit 6, and a determination unit 7. The occupant here is either a driver or a passenger of the vehicle 1.

[0010] The imaging unit 8 is a camera provided in a movable unit in the vehicle 1 and captures images of the interior of the vehicle 1. The movable unit in the vehicle 1 is configured to be changeable in position or angle by an operation of a passenger or an instruction from the vehicle 1, and is, for example, a rearview mirror or a telescopic steering column. The rearview mirror is suspended from a ceiling of a front portion in the vehicle 1, and a driver can change the angle of the rearview mirror to check a rear of the vehicle 1. The telescopic steering column is configured to change the position of the steering wheel, and the driver can optimize the driving posture by adjusting the telescopic steering column.

[0011] The imaging unit 8 captures an image of an image capture area extending within a preset camera view angle from an installation position. The imaging unit 8 is installed such that the face of the occupant in the vehicle 1 to be imaged is included in the imaging area. The imaging unit 8 captures images of an occupant in the vehicle 1 at intervals of, for example, 30 to 60 frames per second (fps) and outputs the captured images to the image acquisition unit 3.

[0012] The image acquisition unit 3 acquires a captured image from the imaging unit 8. Then, the image acquisition unit 3 outputs the acquired captured image to the position estimation unit 5, the scanning unit 4, and the calibration unit 6.

[0013] The scanning unit 4 performs scanning, which includes at least one condition determination of the occupant, such as dozing while driving and distracted driving, or personal authentication of the occupant through image processing of the captured image. The image processing of the captured image in the scanning unit 4 includes, for example, detecting a facial orientation, a degree of eye opening, a gaze direction, and the like.

[0014] The position estimation unit 5 estimates whether or not there is a change in the position of the imaging unit 8. As described above, the imaging unit 8 is provided on a movable unit, such as the rearview mirror and the telescopic steering column, in the vehicle 1. Accordingly, there is a possibility that the position of the imaging unit 8 is changed by angular adjustment of the rearview mirror or the telescopic steering column. A change in the position of the imaging unit 8 means that the position of at least one rotational direction or one translational direction of the imaging unit 8 changes compared to the last calibration.In addition, the position estimation unit 5 can estimate that a change in the position of the imaging unit 8 occurs in at least one case where a rotation amount of the imaging unit 8 is equal to or greater than a preset first threshold rotation amount, or in a case where a translation amount of the imaging unit 8 is equal to or greater than a preset first threshold translation amount. The first threshold rotation amount and the first threshold translation amount are each set to be greater than, for example, a change in the position of the imaging unit 8 caused by vibrations of the vehicle 1 generated by the driving of the vehicle 1.

[0015] Here, the rotation amount of the imaging unit 8 is the amount of change in the rotation direction of the imaging unit 8, the translation amount of the imaging unit 8 is the amount of change in the translation direction of the imaging unit 8, and a change amount of the position of the imaging unit 8 is the movement amount and the rotation amount of the imaging unit 8. Further, at least one of the cases where the rotation amount of the imaging unit 8 is equal to or greater than the preset first threshold of the rotation amount or a case where the translation amount of the imaging unit 8 is equal to or greater than the preset first threshold of the translation amount is expressed as a case where the change amount of the position of the imaging unit 8 is equal to or greater than the first threshold.Then, a case where the rotation amount of the imaging unit 8 is smaller than the preset first threshold of the rotation and a case where the translation amount of the imaging unit 8 is smaller than the preset first threshold of the translation amount are expressed as a case where the change amount of the position of the imaging unit 8 is smaller than the first threshold.

[0016] An example of a method for estimating whether or not there is a change in the position of the imaging unit 8 by the position estimation unit 5 will be described. The position estimation unit 5 calculates a coordinate position of a preset object in the captured image. The preset object is an object installed in the vehicle 1 that is located within the imaging range of the imaging unit 8, and is, for example, a vehicle seat, a headrest, or a window frame. The preset object is, for example, an installed object that is located at a position assumed to be included in the imaging range of the imaging unit 8 even if the position of the imaging unit 8 is changed due to adjustment of the angle of the rearview mirror or the telescopic steering column by the driver, and that is adjusted accordingly depending on the installation location of the imaging unit 8.The position estimation unit 5 calculates a coordinate position of the object in the captured image and stores the calculated coordinate position for a preset time. Then, the position estimation unit 5 compares the coordinate position of the object in a currently captured image with the coordinate position of the object in a previously captured image and calculates the rotation amount and the translation amount of the imaging unit 8. Next, the position estimation unit 5 compares the calculated rotation amount and translation amount of the imaging unit 8 with a first threshold value of the rotation amount and a first threshold value of the translation amount, respectively. Then, when the change amount of the position of the imaging unit 8 is equal to or greater than the first threshold value, the position estimation unit 5 estimates that there has been a change in the position of the imaging unit 8.Further, the position estimation unit 5 may also estimate the change amount in the imaging unit 8 using a captured image, for example, according to the method described in Patent Literature 2.

[0017] Furthermore, the method for estimating whether or not there is a change in the position of the imaging unit 8 by the position estimation unit 5 is not limited to the method described above. For example, when the position of the movable unit in the vehicle 1 in which the imaging unit 8 is provided changes, the position estimation unit 5 may estimate whether or not the imaging unit 8 has moved by acquiring information from the vehicle 1 indicating that the position of the movable unit in the vehicle 1 has changed. In addition, the position estimation unit 5 may acquire information indicating the amount of change in the movable unit in the vehicle 1 from the vehicle 1 and estimate the amount of change in the imaging unit 8 based on the acquired information.

[0018] The calibration unit 6 performs calibration for the image processing of the scanning unit 4. Calibration is processing for correcting an individual difference of an occupant who is a detection target in the image processing. Generally, a scanning result of the scanning unit 4 is accompanied by a detection error due to an individual difference of the occupant. Accordingly, the calibration unit 6 performs calibration to ensure the scanning accuracy in the scanning unit 4. A specific example of calibration is described below, but the calibration is not limited to this, and other known means may be used.

[0019] First, an example will be described in which the calibration unit 6 corrects a standard model used for detecting a gaze direction. In this case, the sensing unit 4 maintains a standard model for detecting a gaze direction in which a captured image in which the occupant's face is captured and a gaze direction of the occupant are linked. Then, the sensing unit 4 detects the occupant's gaze direction from the captured image using the standard model and determines, for example, whether the driver is distracted. Here, the standard model included in the sensing unit 4 needs to be corrected according to the individual difference of the occupant.

[0020] Accordingly, the calibration unit 6 corrects the standard model based on the occupant's actual gaze direction and the captured image in that state. For example, the calibration unit 6 detects the occupant's gaze direction and uses the result to estimate the in-vehicle device the occupant is looking at. Then, the calibration unit 6 determines the direction of the in-vehicle device viewed by the occupant as the reference gaze direction and corrects the standard model based on an error between the gaze direction as the reference and the detected gaze direction. In addition, the calibration unit 6 can correct the standard model by linking the line of sight when the occupant operates each device in the vehicle 1 with the position of the device where the operation is performed. Thus, the sensing unit 4 can accurately detect the gaze direction regardless of any individual difference of the occupant.

[0021] Furthermore, the calibration unit 6 can correct a standard model used for facial direction detection. In this case, the scanning unit 4 holds a 3D model that is a standard model, detects a facial direction of the occupant from a captured image using the 3D model, and determines, for example, whether the occupant is driving distractedly. For example, the 3D model is a 3D model that reflects a facial size and feature points of facial portions of an adult standard model. That is, the 3D model does not reflect the facial size and feature points of facial portions of the occupant.

[0022] Accordingly, the calibration unit 6 extracts feature points corresponding to the feature points of the 3D model of the occupant in the captured image. A method for extracting feature points of the occupant from the captured image is not particularly limited, and the feature points are extracted, for example, by a feature point extraction method using machine learning, an image recognition method such as edge extraction, or template matching. Then, the calibration unit 6 compares the positions of the feature points of the 3D model with the positions of the feature points of the occupant extracted from the captured image on the captured image.Next, the calibration unit 6 calibrates the positions of the feature points of the 3D model in accordance with the positions of the occupant's feature points extracted from the captured image, such that errors between the positions of the feature points of the 3D model and the positions of the occupant's feature points extracted from the captured image are within a preset threshold. Thus, the sensing unit 4 can accurately detect the gaze direction regardless of the occupant's individual differences.

[0023] Additionally, the calibration unit 6 can correct the standard model used for detecting the degree of eye opening. In this case, the sensing unit 4 maintains a standard model for detecting the degree of eye opening of the occupant. Then, the sensing unit 4 detects the occupant's eyes from the captured image and detects the degree of eye opening, which is a degree of opening of the occupant's eyes, using the standard model. The standard model for detecting the degree of eye opening includes, for example, information of a reference flatness, which is a value obtained by dividing a width of an eye when the eye is open by a height of the eye. However, the reference flatness varies depending on the individual difference of the occupant.

[0024] Accordingly, the calibration unit 6 calculates the widths and heights of the occupant's eyes from the captured image and calculates the occupant's reference flatness. Then, the calibration unit 6 corrects the standard model based on the error between the reference flatness of the standard model and the calculated reference flatness of the occupant. Thus, the sensing unit 4 can accurately detect the degree of eye opening regardless of the occupant's individual differences.

[0025] The determination unit 7 determines, depending on an estimation result of the position estimation unit 5, whether or not to perform sampling in the sampling unit 4 and calibration by the calibration unit 6. Specifically, when the determination unit 7 determines that the position of the imaging unit 8 has changed based on the estimation result of the position estimation unit 5, it causes the calibration unit 6 to perform calibration. Then, the determination unit 7 does not cause the sampling unit 4 to perform sampling during calibration and restarts sampling in the sampling unit 4 after the calibration is completed. Alternatively, the determination unit 7 does not cause the sampling result in the sampling unit 4 to be output during calibration and may cause the sampling result in the sampling unit 4 to be output after the calibration is completed.Hereinafter, not initiating scanning in the scanning unit 4 includes both not initiating scanning and initiating scanning but not outputting a scanning result. The detailed processing of the determination unit 7 will be described in an occupant monitoring method to be described later.

[0026] The notification unit 9 is, for example, a speaker or a display provided in the vehicle 1 and notifies the occupant of information. When the notification unit 9 acquires information indicating a detection result from the sensing unit 4, it notifies the occupant of the detection result. When the notification unit 9 acquires a signal indicating that sensing is not being performed by the determination unit 7, the notification unit 9 notifies the occupant that sensing is not being performed. When the notification unit 9 further acquires information indicating that the sensing result is not to be output from the sensing unit 4 or the determination unit 7, the notification unit 9 notifies the occupant that sensing is not being performed.

[0027] Next, a hardware configuration example of the occupant monitoring device 2 will be described. Fig. Figure 2 is a block diagram illustrating a hardware configuration example of the occupant monitoring device 2 of the first embodiment. The image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 of the occupant monitoring device 2 may be a processing circuit 12, which is dedicated hardware, as shown in Figure 1. Fig. 2A, or they may be a processor 13 executing a program stored in a memory 14, as shown in Fig. 2B.

[0028] As in Fig. As shown in Figure 2A, in a case where the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 are dedicated hardware, the processing circuit 12 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. Each of the functions of the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 can be implemented by one processing circuit, or the functions of the respective units can be implemented jointly by one processing circuit.

[0029] As in Fig. 2B, in a case where the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 are the processor 13, the functions of the respective units are implemented by software, firmware, or a combination of software and firmware. Software or firmware is described as a program and stored in the memory 14. The processor 13 reads and executes a program stored in the memory 14 to implement each function of the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7. That is, the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 include the memory 14 for storing a program that, when executed by the processor 13, is used to execute each function described in Fig. 3, Fig. 5, Fig. 6 or Fig. 8. Furthermore, it can also be said that these programs cause a computer to execute procedures or processes of the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7.

[0030] The processor 13 here is, for example, a central processing unit (CPU), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, a digital signal processor (DSP), or the like. The memory 14 can be, for example, a non-volatile or volatile semiconductor memory, such as a RAM (Random Access Memory), a ROM (Read-Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically Erasable Programmable ROM), a magnetic disk, such as a hard disk or a flexible disk, or an optical disk, such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

[0031] It should be noted that a portion of each of the functions of the image acquisition unit 3, the scanning unit 4, the position estimation unit 5, the calibration unit 6, and the determination unit 7 may be implemented by dedicated hardware, and a portion thereof may be implemented by software or firmware. As described above, the processing circuit 12 in the occupant monitoring device 2 may implement the above-described functions by hardware, software, firmware, or a combination thereof.

[0032] Next, an occupant monitoring method by the occupant monitoring device 2 will be described. Fig. Fig. 3 is a flowchart illustrating a processing procedure of the occupant monitoring device 2 of the first embodiment. The occupant monitoring device 2 starts operating, for example, at a time when the imaging unit 8 starts activating, and repeats the operation while the vehicle 1 is traveling. In the following description, it is assumed that the scanning unit 4 starts scanning at START in Fig. 3.

[0033] In step S1, the image acquisition unit 3 acquires a captured image from the imaging unit 8. Then, the image acquisition unit 3 outputs the acquired captured image to the position estimation unit 5, the scanning unit 4, and the calibration unit 6.

[0034] In step S2, the position estimation unit 5 determines whether the amount of change in the position of the imaging unit 8 is equal to or greater than the first threshold. Then, if the amount of change in the position of the imaging unit 8 is equal to or greater than the first threshold (YES in step S2), the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8 (step S3). If the amount of change in the position of the imaging unit 8 is less than the first threshold (NO in step S2), the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 (step S4).

[0035] If the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 (step S4), the determination unit 7 determines that scanning is being performed by the scanning unit 4 (step S5). That is, the determination unit 7 causes the scanning by the scanning unit 4 to continue.

[0036] When the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8 (step S3), the determination unit 7 determines not to perform scanning by the scanning unit 4 (step S6). That is, the determination unit 7 causes the scanning by the scanning unit 4 to stop.

[0037] In step S7, the determination unit 7 outputs a signal indicating that no scanning is being performed to the notification unit 9. Thus, the notification unit 9 notifies the occupant that no scanning is being performed. Step S7 may be performed concurrently with step S6. Furthermore, the notification unit 9 may continuously notify the occupant that no scanning is being performed until a processing operation in step S10, which will be described later, is performed.

[0038] In step S8, the determination unit 7 determines whether or not the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 for a preset time. If the position estimation unit 5 does not estimate that there is no change in the position of the imaging unit 8 for the preset time (NO in step S8), the determination unit 7 repeats step S8 until the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 for the preset time. The preset time is set, for example, to a time required for the position or angle of the movable unit in the vehicle 1 to be changed by an operation of a passenger or an instruction of the vehicle 1. The preset time is, for example, 30 seconds.

[0039] When the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 for the preset time (YES in step S8), the determination unit 7 causes the calibration to be performed by the calibration unit 6 (step S9).

[0040] After the calibration by the calibration unit 6 is completed, the determination unit 7 causes the scanning to be performed by the scanning unit 4 in step S10.

[0041] In step S11, the scanning unit 4 outputs the scanning result to the notification unit 9. Thus, the notification unit 9 notifies the occupant of the scanning result.

[0042] This completes the processing of the occupant monitoring device 2.

[0043] Next, the effects of the occupant monitoring device 2 of the present first embodiment will be described in comparison with a conventional occupant monitoring device. The conventional occupant monitoring device performs image processing on a captured image captured by the imaging unit 8 installed in a fixed portion in the vehicle 1, such as a steering column or a center console, and performs a scan for an occupant. Then, the conventional occupant monitoring device performs calibration, for example, when a preset time has elapsed.In the conventional occupant monitoring device, since the imaging unit 8 is installed in the fixed portion in the vehicle 1, the position of the imaging unit 8 does not change easily, and therefore, it is possible to maintain the scanning performance by performing calibration when a preset time has elapsed. However, since many devices other than the imaging unit 8 are arranged in the fixed portion in the vehicle 1, such as the steering column and the center console, the arrangement of the imaging unit 8 is limited.

[0044] Generally, the movable unit in the vehicle 1 has more space in the arrangement than the fixed portion. Accordingly, the occupant monitoring device 2 of the present embodiment performs image processing on a captured image captured by the imaging unit 8 provided in a movable unit in the vehicle 1, such as a rearview mirror and a telescopic steering column, and performs sensing for the occupant. Here, in the occupant monitoring device 2 of the present embodiment, a case is considered where calibration is performed when a preset time has elapsed, similarly to the conventional occupant monitoring device.In this case, for example, the occupant monitoring device 2 continues scanning without performing calibration until a preset time elapses, even if the angle of the movable unit provided with the imaging unit 8 is changed and the positional relationship between the imaging unit 8 and the occupant changes. Thus, there is a problem that a state in which the scanning performance is deteriorated continues.

[0045] Accordingly, the occupant monitoring device 2 of the present embodiment includes the image acquisition unit 3 for acquiring a captured image in which the interior of the vehicle 1 is captured by the imaging unit 8 provided in a movable unit in the vehicle 1, the scanning unit 4 for performing scanning that is at least a state determination of an occupant or personal authentication of the occupant by image processing of the captured image, the position estimation unit 5 for estimating whether or not there is a change in a position of the imaging unit 8, the calibration unit 6 for performing calibration for image processing in the scanning unit 4, and the determination unit 7 for causing the scanning unit 4 not to be performed and for causing the calibration unit 6 to be performed when the position estimation unit 5 estimates,that there is a change in the position of the imaging unit 8. Since the occupant monitoring device 2 of the present embodiment performs calibration when it is estimated that there is a change in the position of the imaging unit 8, it is possible to suppress deterioration in the scanning performance due to the change in the position of the imaging unit 8.

[0046] If the position estimation unit 5 estimates that the position of the imaging unit 8 does not change continuously over a preset time, the determination unit 7 also causes the calibration unit 6 to perform the calibration. The preset time is a time required to change the position or angle of the movable unit in the vehicle 1, for example, by an operation of a passenger or an instruction of the vehicle 1. Thus, the determination unit 7 does not cause the calibration to be performed while the position of the imaging unit 8 is changing, so that the processing load can be reduced. Note that it is not necessary to perform the processing operation in step S8 described in Fig. 3. That is, when the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8, the determination unit 7 can cause the calibration to be performed by the calibration unit 6.

[0047] When the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8, the determination unit 7 causes the scanning unit 4 to perform scanning after the calibration by the calibration unit 6 ends. Thus, the scanning can be quickly restarted after the calibration is completed.

[0048] Furthermore, if the determination unit 7 does not initiate scanning by the scanning unit 4, it outputs a signal to the notification unit 9 indicating that scanning is not being performed and notifying the occupant of the information. Thus, it is possible to notify the occupant that scanning is not being performed in the occupant monitoring device 2.

[0049] In addition, vibrations are generated throughout the vehicle 1 when the vehicle 1 is traveling, and thus the position of the imaging unit 8 changes precisely when the vehicle 1 is traveling. Accordingly, the occupant monitoring device 2 repeatedly executes step S8 in Fig. 3 if the accuracy of the position estimation unit 5 is high enough to detect a change in the position of the imaging unit 8 due to the vibrations of the vehicle 1. Alternatively, the occupant monitoring device 2 repeatedly executes step S9 if step S8 of Fig. 3 is not executed. Thus, a period in which no scanning is performed by the scanning unit 4 becomes long.

[0050] Accordingly, the position estimation unit 5 estimates that a change in the position of the imaging unit 8 occurs in at least one of a case where the rotation amount of the imaging unit 8 is equal to or greater than a preset first threshold rotation amount, and a case where the translation amount of the imaging unit 8 is equal to or greater than a preset first threshold translation amount. The first threshold rotation amount and the first threshold translation amount are each greater than, for example, a change in the position of the imaging unit 8 caused by vibrations of the vehicle 1 generated by the traveling of the vehicle 1. Thus, the occupant monitoring device 2 can suppress deterioration in sensing performance due to a change in the position of the movable unit in the vehicle 1.It should be noted that a change in the position of the imaging unit 8 caused by a vibration of the vehicle 1 caused by the running of the vehicle 1 is smaller than a change in the position of the imaging unit 8 due to a change in the position of the movable unit in the vehicle 1. Accordingly, a decrease in the scanning performance due to vibration of the vehicle 1 caused by the running of the vehicle 1 is smaller than a decrease in the scanning performance due to a change in the position of the movable unit in the vehicle 1. Second embodiment.

[0051] An occupant monitoring device 2 according to a second embodiment will be described with reference to Fig. 4 described. Fig. 4 is a block diagram illustrating a portion of the vehicle 1 including the occupant monitoring device 2 according to the second embodiment. In the occupant monitoring device 2 of the second embodiment, the scanning unit 4 includes a storage unit 10. Other configurations of the occupant monitoring device 2 are similar to those of the first embodiment. The same reference numerals are used for configurations similar to those of the first embodiment.

[0052] As in the first embodiment, the scanning unit 4 performs scanning, which includes at least one occupant condition determination, such as dozing while driving and distracted driving, or personal authentication of the occupant through image processing of the captured image. The scanning unit 4 then includes the storage unit 10.

[0053] The storage unit 10 stores at least one result of the last scan. Then, in accordance with a signal from the determination unit 7, the result of the last scan is output to the notification unit 9. Note that the storage unit 10 may be provided separately from the scanning unit 4 in the occupant monitoring device 2.

[0054] Next, an occupant monitoring method by the occupant monitoring device 2 will be described. Fig. Fig. 5 is a flowchart illustrating the processing procedure of the occupant monitoring device 2 according to the second embodiment. The occupant monitoring device 2 starts the operation, for example, at a time when the imaging unit 8 starts the activation, and repeats the operation while the vehicle 1 is traveling. In the following description, it is assumed that the scanning unit 4 starts the scanning at START in Fig. 5.

[0055] Steps S1 to S6 and steps S8 to S10 are similar to the processing procedure of the occupant monitoring device 2 of the first embodiment, as shown in Fig. 3. If the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8 (step S3), the determination unit 7 determines that sampling by the scanning unit 4 should not be performed (step S6), and the process proceeds to step S20. Furthermore, in step S6, in a case where the scanning unit 4 performs sampling but does not output a sampling result, the scanning unit 4 does not store the sampling result in the storage unit 10 as valid data.

[0056] In step S20, the determination unit 7 causes the scanning unit 4 to output the result of the last scanning to the notification unit 9. Thus, the notification unit 9 notifies the occupant of the scanning result in the scanning unit 4 before the position of the imaging unit 8 is judged to have changed. Step S20 may be performed concurrently with step S6. In addition, the notification unit 9 may continuously notify the occupant of the scanning result in the scanning unit 4 before the position of the imaging unit 8 is judged to have changed until the processing operation in step S21 to be described later is performed. Next, the occupant monitoring device 2 executes the processing operations of steps S8 to S10.

[0057] In step S21, the determination unit 7 causes the scanning unit 4 to output the latest scanning result to the notification unit 9. Thus, the processing operation of the occupant monitoring device 2 is terminated.

[0058] As in the first embodiment, the occupant monitoring device 2 of the present embodiment includes the image acquisition unit 3 for acquiring a captured image captured by the imaging unit 8 provided in a movable unit in the vehicle 1, the scanning unit 4 for performing scanning that is at least a state determination of an occupant or personal authentication of the occupant by image processing of the captured image, the position estimation unit 5 for estimating whether or not there is a change in a position of the imaging unit 8, the calibration unit 6 for performing calibration for image processing in the scanning unit 4, and the determination unit 7 for not causing the scanning unit 4 to be performed and for causing the calibration unit 6 to be performed when the position estimation unit 5 estimates,that there is a change in the position of the imaging unit 8. Thus, it is possible to suppress deterioration of the scanning performance due to a change in the position of the imaging unit 8.

[0059] Furthermore, during the change of the position of the imaging unit 8 and during calibration, there is a high probability that the scanning unit 4 cannot ensure the scanning accuracy and will erroneously detect it. Accordingly, the scanning unit 4 stores at least a result of the last scanning, and when no scanning is performed by the scanning unit 4, the determination unit 7 causes the scanning unit 4 to output the result of the last scanning to the notification unit 9. Thus, the occupant monitoring device 2 outputs the scanning result of the scanning unit 4 before the position of the imaging unit 8 changes, while the position of the imaging unit 8 changes, and while calibration is being performed, and thus it is possible to suppress erroneous detection by the scanning unit 4. Third embodiment.

[0060] An occupant monitoring device 2 according to a third embodiment will be described with reference to Fig. 6 described. Fig. 6 is a flowchart illustrating the processing procedure of the occupant monitoring device 2 according to the third embodiment. The occupant monitoring device 2 of the third embodiment differs from the occupant monitoring devices 2 of the first and second embodiments in the processing procedure of the determination unit 7. Other configurations of the occupant monitoring device 2 are similar to those of the first embodiment or the second embodiment. The same reference numerals are used for configurations similar to those of the first embodiment.

[0061] Steps S1 to S8 are similar to the processing procedure of the occupant monitoring device 2 of the first embodiment, as shown in Fig. 3. In the occupant monitoring device 2 of the first embodiment, when the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 for the preset time (YES in step S8), the determination unit 7 causes the calibration to be performed by the calibration unit 6 (step S9).

[0062] On the other hand, in the occupant monitoring device 2 of the present embodiment, when the position estimation unit 5 estimates that there is no change in the position of the imaging unit 8 for the preset time (YES in step S8), the determination unit 7 determines whether the occupant who is a target of scanning by the scanning unit 4 is captured in the captured image (step S30). For example, the determination unit 7 may cause the scanning unit 4 or the calibration unit 6 to determine whether the occupant who is a target of scanning by the scanning unit 4 is captured in the captured image.

[0063] If the determination unit 7 determines that the occupant is included in the captured image (YES in step S30), the process proceeds to step S9. Steps S9 to S11 are similar to the processing procedure of the occupant monitoring device 2 of the first embodiment shown in Fig. 3 is shown.

[0064] If the determination unit 7 determines that the occupant is not included in the captured image (NO in step S30), the determination unit 7 repeats step S30 until it is determined that the occupant is included in the captured image. That is, the determination unit 7 does not cause calibration (step S9) and sampling (step S10) to be performed in a case where the occupant is not included in the captured image. Thus, the processing operation of the occupant monitoring device 2 is terminated.

[0065] As in the first embodiment, the occupant monitoring device 2 of the present embodiment includes the image acquisition unit 3 for acquiring a captured image in which the interior of the vehicle 1 is captured by the imaging unit 8 provided in a movable unit in the vehicle 1, the scanning unit 4 for performing scanning that is at least a state determination of an occupant or personal authentication of the occupant by image processing of the captured image, the position estimation unit 5 for estimating whether or not there is a change in a position of the imaging unit 8, the calibration unit 6 for performing calibration for image processing in the scanning unit 4, and the determination unit 7 for not causing the scanning unit 4 to be performed and for causing the calibration unit 6 to be performed.when the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8. Thus, it is possible to suppress deterioration of the scanning performance due to a change in the position of the imaging unit 8.

[0066] Furthermore, in a case where the occupant is not captured by the imaging unit 8, the calibration unit 6 and the scanning unit 4 cannot perform calibration and scanning, respectively. Accordingly, the determination unit 7 does not cause calibration and scanning to be performed in a case where the occupant is not captured in the captured image, so unnecessary processing can be suppressed. Fourth embodiment.

[0067] An occupant monitoring device 2 according to a fourth embodiment will be described. Fig. 7 is a block diagram illustrating a part of the vehicle 1 including the occupant monitoring device 2 according to the fourth embodiment. The vehicle 1 according to the fourth embodiment includes a vehicle speed sensor 11 that detects a speed of the vehicle 1.

[0068] As described above, the occupant monitoring device 2 performs sensing, which is at least one of a condition determination of an occupant, such as dozing while driving and distracted driving, or a personal authentication of the occupant by the sensing unit 4. Thus, the occupant monitoring device 2 can improve the safety and comfort of the occupant.

[0069] On the other hand, when the position of the movable unit provided with the imaging unit 8 is changed, the positional relationship between the imaging unit 8 and the occupant is changed, thus increasing the possibility of the sensing unit 4 performing erroneous detection, and deteriorating the occupant's safety and comfort. Accordingly, the occupant monitoring device 2 according to the first to third embodiments, when a change in the position of the imaging unit 8 is estimated, does not initiate sensing by the sensing unit 4, but instead initiates calibration by the calibration unit 6, thus making it possible to suppress a decrease in sensing accuracy.

[0070] Incidentally, the possibility that the sensing unit 4 performs erroneous detection depends on the amount of change in the position of the imaging unit 8. That is, the sensing accuracy decreases as the amount of change in the position of the imaging unit 8 increases. The occupant monitoring device 2 according to the fourth embodiment differs from the occupant monitoring devices 2 according to the first to third embodiments in that the position estimation unit 5 estimates an amount of change in a position of the imaging unit 8, and the determination unit 7 makes a determination depending on the amount of change in the position of the imaging unit 8. Other configurations of the occupant monitoring device 2 are similar to those of any of the first to third embodiments. The same reference numerals are used for configurations similar to those of the first embodiment.

[0071] Similar to the first embodiment, the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8 when the amount of change in the position of the imaging unit 8 is equal to or greater than the first threshold.

[0072] The determination unit 7 determines whether the rotation amount of the imaging unit 8 is equal to or greater than a second threshold of a preset rotation amount and whether the translation amount of the imaging unit 8 is equal to or greater than a second threshold of a preset translation amount. The second threshold of the rotation amount and the second threshold of the translation amount are set to values ​​greater than the first threshold of the rotation amount and the first threshold of the translation amount, respectively. The second threshold of the rotation amount and the second threshold of the translation amount are set according to the desired assurance of scanning accuracy by the scanning unit 4.For example, the second threshold value of the rotation amount and the second threshold value of the translation amount are respectively set to the change amount of the position of the imaging unit 8 by experimentally calculating a change amount of the position of the imaging unit 8 and a probability of erroneous detection by the scanning unit 4, and a possibility of erroneous detection of the scanning by the scanning unit 4 is 80%.

[0073] Hereinafter, at least one of the cases where the rotation amount of the imaging unit 8 is equal to or greater than the second threshold of the preset rotation amount, or a case where the translation amount of the imaging unit 8 is equal to or greater than the second threshold of the preset translation amount, is expressed as a case where the change amount of the position of the imaging unit 8 is equal to or greater than the second threshold. Further, a case where the rotation amount of the imaging unit 8 is smaller than the second threshold of the preset rotation amount, or a case where the translation amount of the imaging unit 8 is smaller than the second threshold of the preset translation amount, is expressed as a case where the change amount of the position of the imaging unit 8 is smaller than the second threshold.

[0074] Then, in a case where the position estimation unit 5 estimates that the change amount of the position of the imaging unit 8 is equal to or greater than the first threshold, and there is a change in the position of the imaging unit 8, the determination unit 7 makes the following determination. If it is determined that the change amount of the position of the imaging unit 8 is less than the second threshold, the determination unit 7 causes sampling to be performed by the sampling unit 4 and does not cause calibration to be performed by the calibration unit 6, for example, until the vehicle 1 stops.Further, when the determination unit 7 determines that the amount of change in the position of the imaging unit 8 is equal to or greater than the second threshold, it does not cause the sampling by the sampling unit 4 to be performed and causes the calibration to be performed regardless of the driving state of the vehicle 1.

[0075] Next, an occupant monitoring method by the occupant monitoring device 2 will be described. Fig. Fig. 8 is a flowchart illustrating the processing procedure of the occupant monitoring device 2 according to the fourth embodiment. The occupant monitoring device 2 starts the operation, for example, at a time when the imaging unit 8 starts the activation, and repeats the operation while the vehicle 1 is traveling. In the following description, it is assumed that the scanning unit 4 starts the scanning at START in Fig. 8.

[0076] The steps S1 to S4 are similar to the processing procedure of the occupant monitoring device 2 of the first embodiment shown in Fig. 3. When the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8 (step S3), the determination unit 7 determines whether the amount of change in the position of the imaging unit 8 is equal to or greater than the second threshold (step S40).

[0077] If the amount of change in the position of the imaging unit 8 is equal to or greater than the second threshold (YES in step S40), the process proceeds to step S6. If the amount of change in the position of the imaging unit 8 is less than the second threshold (NO in step S40), the determination unit 7 causes the sampling by the sampling unit 4 to continue (step S41). Then, the determination unit 7 acquires information indicative of the speed of the vehicle 1 from the vehicle speed sensor 11 and determines whether the vehicle 1 is stopped or not (step S42). If the vehicle 1 is stopped (YES in step S42), the process proceeds to step S5. If the vehicle 1 is not stopped (NO in step S42), step S42 is repeated. That is, if the vehicle 1 is not stopped (NO in step S42), the sampling continues until the vehicle 1 stops.Steps S6 to S11 are similar to the processing procedure of the occupant monitoring device 2 of the first embodiment shown in FIG. Fig. 3 is shown.

[0078] Thus, the processing operation of the occupant monitoring device 2 is terminated.

[0079] As in the first embodiment, the occupant monitoring device 2 of the present embodiment includes the image acquisition unit 3 for acquiring a captured image in which the interior of the vehicle 1 is captured by the imaging unit 8 provided in a movable unit in the vehicle 1, the scanning unit 4 for performing scanning that is at least a state determination of an occupant or personal authentication of the occupant by image processing of the captured image, the position estimation unit 5 for estimating whether or not there is a change in a position of the imaging unit 8, the calibration unit 6 for performing calibration for image processing in the scanning unit 4, and the determination unit 7 for not causing the scanning unit 4 to be performed and for causing the calibration unit 6 to be performed.when the position estimation unit 5 estimates that there is a change in the position of the imaging unit 8. Thus, it is possible to suppress deterioration in scanning performance due to a change in the position of the imaging unit 8.

[0080] Further, in the occupant monitoring device 2 according to the present embodiment, in a case where the rotation amount of the imaging unit 8 is smaller than the second threshold of the preset rotation amount, and in a case where the translation amount of the imaging unit 8 is smaller than the second threshold of the preset translation amount, that is, in a case where the change amount of the position of the imaging unit 8 is equal to or greater than the first threshold and smaller than the second threshold, the determination unit 7 causes the sampling by the sampling unit 4 to be performed and does not cause the calibration by the calibration unit 6 to be performed until the vehicle 1 stops.After the vehicle 1 stops, the determination unit 7 does not cause the scanning to be performed by the scanning unit 4 and causes the calibration to be performed by the calibration unit 6. The second threshold value can be appropriately set depending on how much the scanning accuracy by the scanning unit 4 is to be ensured. For example, consider a case where the change amount of the position of the imaging unit 8 and the probability of erroneous detection by the scanning unit 4 are experimentally calculated, and the second threshold value is set to the change amount of the position of the imaging unit 8 at which the possibility of erroneous detection of the scanning by the scanning unit 4 is 80%.

[0081] In this case, if the possibility of erroneous detection of the scanning is less than 80%, the determination unit 7 determines that the scanning accuracy has been ensured and causes the scanning to continue until the vehicle 1 stops. Thus, the occupant monitoring device 2 can improve the safety and comfort of the occupant by performing scanning while the vehicle 1 is traveling.

[0082] When the possibility of erroneous detection of the sampling is 80% or more, the determination unit 7 determines that the sampling accuracy cannot be ensured and causes the calibration to be performed regardless of the running state of the vehicle 1. In this way, priority can be given to improving the sampling accuracy when the sampling accuracy cannot be ensured.

[0083] Note that in the present embodiment, the example in which the determination unit 7 acquires the information indicating the speed of the vehicle 1 from the vehicle speed sensor 11 and determines whether the vehicle 1 is stopped or not was described, but the present invention is not limited to this. For example, the determination unit 7 may acquire information indicating an open / closed state of a door and the on / off of an ignition switch of the vehicle 1 and determine whether the vehicle 1 is stopped or not.

[0084] Furthermore, each embodiment disclosed in the present specification can be freely combined within its scope, and each embodiment can be appropriately modified or omitted. LIST OF REFERENCE SYMBOLS 1 vehicle, 2 occupant monitoring device, 3 image acquisition unit, 4 scanning unit, 5 Position estimation unit, 6 Calibration unit, 7 Determination unit, 8 imaging unit, 9 Notification unit, 10 storage units, 11 Vehicle speed sensor, 12 processing circuit, 13 processor, 14 storage

Claims

[1] An occupant monitoring device (2) comprising: an image capturing unit (3) for capturing a captured image in which an interior of a vehicle is captured by an imaging unit (8) provided in a movable unit in the vehicle; a scanning unit (4) for performing a scanning which is at least a state determination of an occupant or a personal authentication of the occupant by image processing of the captured image; a position estimation unit (5) for estimating whether or not there is a change in a position of the imaging unit (8); a calibration unit (6) for performing a calibration for the image processing in the scanning unit (4); and a determination unit (7) for not causing the scanning to be performed by the scanning unit (4) and for causing the calibration to be performed by the calibration unit (6) when the position estimation unit (5) estimates that there is a change in the position of the imaging unit (8). [2] The occupant monitoring device according to claim 1, wherein, when the position estimation unit (5) estimates that the position of the imaging unit (8) does not change continuously for a preset time, the determination unit (7) causes the calibration to be performed by the calibration unit (6). [3] The occupant monitoring device according to claim 1 or 2, wherein, when the position estimation unit (5) estimates that there is a change in the position of the imaging unit (8), the determination unit (7) causes the scanning by the scanning unit (4) to be performed after completion of the calibration performed by the calibration unit (6). [4] An occupant monitoring device according to any one of claims 1 to 3, wherein, when the scanning is not performed by the scanning unit (4), the determination unit (7) outputs a signal indicating that the scanning is not performed to a notification unit (9) to notify the occupant of information. [5] Occupant monitoring device according to one of claims 1 to 3, wherein the scanning unit (4) stores at least one result of the last scanning and, if the scanning is not performed by the scanning unit (4), the determination unit (7) causes the scanning unit (4) to output the result of the last scanning to a notification unit (9) to notify the occupant of information. [6] The occupant monitoring device according to any one of claims 1 to 5, wherein when the occupant is not recorded on the captured image, the determination unit (7) does not cause the calibration and the scanning to be performed. [7] The occupant monitoring device according to any one of claims 1 to 6, wherein the position estimation unit (5) estimates that there is a change in the position of the imaging unit (8) in at least a case where a rotation amount of the imaging unit (8) is equal to or greater than a preset first threshold value of the rotation amount, or in a case where a translation amount of the imaging unit (8) is equal to or greater than a preset first threshold value of the translation amount. [8] The occupant monitoring device according to claim 7, wherein each of the first threshold value of the rotation amount and the first threshold value of the translation amount is greater than a change in the position of the imaging unit (8) caused by the vibration of the vehicle generated by the running of the vehicle. [9] The occupant monitoring device according to claim 7 or 8, wherein in a case where the rotation amount of the imaging unit (8) is less than a second threshold of a preset rotation amount, and in a case where the translation amount of the imaging unit (8) is less than a second threshold of a preset translation amount, the determination unit (7) causes the scanning by the scanning unit (4) to be performed and does not cause the calibration by the calibration unit (6) to be performed until the vehicle stops, and does not cause the scanning by the scanning unit (4) to be performed and causes the calibration by the calibration unit (6) to be performed after the vehicle stops, and in at least one case where the rotation amount of the imaging unit (8) is equal to or greater than the second threshold of the rotation amount or a case,in which the translation amount of the imaging unit (8) is equal to or greater than the second threshold value of the translation amount, the determination unit (7) does not cause the scanning to be performed by the scanning unit (4) and causes the calibration to be performed by the calibration unit (6). [10] An occupant monitoring method comprising: Capturing a captured image in which an interior of a vehicle is captured by an imaging unit (8) provided in a movable unit in the vehicle; Performing a scan that is at least a condition determination of an occupant or a personal authentication of the occupant by image processing of the captured image; Estimating whether or not there is a change in a position of the imaging unit (8); Performing calibration for image processing; and Performing calibration without performing scanning when it is estimated that there is a change in the position of the imaging unit (8). [11] A non-transitory computer-readable medium having stored thereon an executable program, the program instructing a computer to do the following: Capturing a captured image in which an interior of a vehicle is captured by an imaging unit (8) provided in a movable unit in the vehicle; Performing a scan that is at least a condition determination of an occupant or a personal authentication of the occupant by image processing of the captured image; Estimating whether or not there is a change in a position of the imaging unit (8); Performing calibration for image processing; and Performing calibration without performing scanning when it is estimated that there is a change in the position of the imaging unit (8).

Images