OCCLUSION DETECTION DEVICE, OCCUPANTION MONITORING DEVICE, AND OCCLUSION DETECTION METHOD

Abstract

This blocking determination device (100) selects a plurality of reference pixels (12) that are different for each frame in a photographed image P, and determines, on the basis of a luminance value of the plurality of selected reference pixels (12), whether there is a blocking object (9) that interferes with photographing by a photographing unit (1). In addition, the blocking determination device is characterized by determining that the photographing unit (1) is in a blocked state where the photographing unit (1) is blocked by the blocking object (9) when the blocking object (9) is determined to be present in two or more of a preset number of frames, and that the photographing unit (1) is not in the blocked state when the blocking object (9) is determined not to be present in a preset number of frames. In this way, the determination accuracy for the blocked state can be improved by determining whether there is the blocking object (9) on the basis of the luminance value of the plurality of reference pixels (12) of which pixel positions are different for each frame, and by determining whether the photographing unit is in the blocked state on the basis of whether the blocking object (9) is in the plurality of frames.

Description

[Technical field]

[0001] The present disclosure relates to an occlusion determination device, an occlusion determination device, and an occlusion determination method. [Background technology]

[0002] There is a technology that uses an imaging unit to capture an image of an occupant inside a vehicle, and monitors whether or not the occupant is dozing while driving or looking away from the road. Conventionally, when an obstacle between the occupant and the imaging unit prevents the occupant from being imaged, the occupant is notified that the imaging unit is in a blocked state where the imaging unit is blocked by the obstacle. Whether or not the imaging unit is in a blocked state where the imaging unit is blocked by the obstacle is determined based on the luminance value of pixels in a specific area in the captured image that contains a specific part such as the occupant's face (for example, see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-159346 A Summary of the Invention [Problem to be solved by the invention]

[0004] In general, the imaging unit performs imaging control such as exposure control and gain control of the entire captured image so that the average luminance value of a specific region in the captured image approaches a preset range. For example, when a local high-luminance object such as the sun is captured in a part of the specific region, the imaging unit prevents whiteout by reducing the luminance value of the entire captured image. However, when such imaging control is performed, the luminance value of the pixels in the specific region in the captured image is reduced, so there is a problem that the occlusion state cannot be correctly determined based on the luminance value of the pixels in the specific region.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an occlusion determination device and an occlusion determination method that can improve the accuracy of determining an occlusion state. [Means for solving the problem]

[0006] The occlusion determination device according to the present disclosure includes an image acquisition unit that acquires a plurality of frames of captured images from an imaging unit that captures an image of the inside of a vehicle, and each The imaging device includes a pixel selection unit which selects a plurality of reference pixels whose pixel positions differ depending on the frame; an obstruction determination unit which determines whether or not there is an obstruction that hinders imaging by the imaging unit based on the luminance values ​​of the reference pixels; and an obstruction state determination unit which determines that the imaging unit is in an obstructed state where it is obstructed by an obstruction when the obstruction determination unit determines that there is an obstruction in two or more predetermined number of frames, and determines that the imaging unit is not in an obstructed state when the obstruction determination unit does not determine that there is an obstruction in the predetermined number of frames.

[0007] The occupant monitoring device according to the present disclosure includes an imaging unit that images the interior of a vehicle, an image acquisition unit that acquires a plurality of frames of captured images from the imaging unit, and each The imaging device includes a pixel selection unit which selects a plurality of reference pixels whose pixel positions differ depending on the frame; an obstruction determination unit which determines whether or not there is an obstruction that hinders imaging by the imaging unit based on the luminance value of the reference pixel; an obstruction state determination unit which determines that the imaging unit is in an obstructed state where the imaging unit is obstructed by an obstruction when the obstruction determination unit determines that there is an obstruction in two or more preset number of frames, and determines that the imaging unit is not in an obstructed state when the obstruction determination unit does not determine that there is an obstruction in the preset number of frames; and a monitoring processing unit which performs a monitoring process of at least one of determining the state of the occupant of the vehicle by image recognition processing and personal authentication of the occupant when the obstruction state determination unit determines that there is an obstruction state, and does not perform the monitoring process when the obstruction state determination unit determines that there is an obstruction state.

[0008] The occlusion determination method according to the present disclosure includes steps of acquiring a plurality of frames of captured images from an imaging unit that captures an image of the inside of a vehicle, and selecting from the captured images a frame of captured images that includes a plurality of frames of captured images. each The method includes the steps of: selecting a plurality of reference pixels whose pixel positions differ depending on the frame; determining whether or not there is an obstruction that prevents imaging by the imaging unit based on the luminance values ​​of the reference pixels; and determining that the imaging unit is in an obstructed state in which it is obstructed by an obstruction if it is determined that an obstruction is present in two or more predetermined number of frames; and determining that the imaging unit is not in an obstructed state if it is not determined that an obstruction is present in two or more predetermined number of frames. Effect of the Invention

[0009] According to the present disclosure, whether or not an occluding object is present is determined based on the luminance values ​​of a plurality of reference pixels whose pixel positions differ from frame to frame. Then, whether or not an occluding object is present is determined based on whether or not an occluding object is determined to be present in a preset number of frames that is equal to or greater than two. This can improve the accuracy of determining whether or not an occluding object is present. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a part of a DMS including an occlusion determination device according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram of the DMS of the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram showing an example of a captured image according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of a hardware configuration of the occlusion determination device according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing the obstruction state determination method according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of a captured image according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram illustrating the processing operation in step S20 in the first embodiment. [Figure 8] FIG. 8 is a flowchart showing a method for determining an obstruction state by an obstruction determination device according to a comparative example. [Figure 9] FIG. 9 is a schematic diagram for explaining the processing operation of an occlusion determination device according to a comparative example. [Figure 10] FIG. 10 is a schematic diagram for explaining the processing operation of the occlusion determination device according to the first embodiment. [Figure 11] FIG. 11 is a block diagram showing a part of a DMS including an occlusion determination device according to the second embodiment. [Figure 12] FIG. 12 is a flowchart showing a method for determining an obstructed state according to the second embodiment. [Figure 13] FIG. 13 is a block diagram showing a part of a DMS including an occlusion determination device according to the third embodiment. [Figure 14] FIG. 14 is a flowchart showing a method for determining an obstructed state according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Embodiment 1 The occlusion determination device 100 in the first embodiment will be described with reference to Figs. 1 and 2. Fig. 1 is a block diagram showing a part of a driver monitoring system (DMS) including the occlusion determination device 100 in the first embodiment. The DMS is an occupant monitoring device, and performs various monitoring processes for the occupant 8 in the vehicle by image recognition processing of the occupant 8. The monitoring processing includes, for example, determining the state of the occupant 8, such as drowsy driving and inattentive driving, and personal authentication of the occupant 8. The DMS shown in Fig. 1 includes an imaging unit 1, an occlusion determination device 100, a monitoring processing unit 2, and a notification unit 3. Fig. 2 is a schematic diagram of the DMS in the first embodiment. Fig. 2A shows a case where there is an obstruction 9 that prevents the imaging unit 1 from capturing an image of the occupant 8. Fig. 2B shows a case where there is no obstruction 9 that prevents the imaging unit 1 from capturing an image of the occupant 8, and the sun, which is a local high-luminance object 10, is present in the imaging range of the imaging unit 1.

[0012] Here, occupant 8 is a general term for the driver and passengers of the vehicle. In Fig. 2A, for example, obstruction 9 is a towel placed on the dashboard, which hangs down over the center display and obstructs the imaging range of imaging element 1b, which will be described later. Although the sun is shown as local high-luminance object 10 in Fig. 2B, it may be a headlight, a street lamp, or the like.

[0013] As shown in FIG. 2B, the imaging unit 1 is a camera having a light-emitting element 1a and an imaging element 1b. The imaging unit 1 captures an image of an occupant 8 inside a vehicle at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured image P to an image acquisition unit 4 of the occlusion determination device 100. The light-emitting element 1a and the imaging element 1b are installed on the instrument panel, dashboard, center display, or rearview mirror of the vehicle. The light-emitting element 1a and the imaging element 1b may be provided close to each other or may be provided separately. FIG. 2 shows an example in which the light-emitting element 1a and the imaging element 1b are arranged separately on both sides of the center display.

[0014] The light-emitting element 1a is, for example, an infrared light-emitting diode that emits near-infrared light. The light-emitting element 1a irradiates the near-infrared light to the occupant 8. The light-emitting element 1a is not limited to an infrared light-emitting diode, and may be one that irradiates the occupant 8 with visible light or far-infrared light.

[0015] The imaging element 1b is, for example, an image sensor, and captures an image of the occupant 8 illuminated with near-infrared light or the like by the light-emitting element 1a. The imaging element 1b performs imaging control such as exposure control and gain control of the entire captured image P so that the average luminance value of a specific region 11 in the captured image P approaches a preset range. The specific region 11 is set so as to accommodate a specific portion of the captured image P, such as the face of the occupant 8, which is used when performing image recognition processing on the occupant 8.

[0016] An example of imaging control by the imaging element 1b will be described with reference to Fig. 3. Fig. 3 is a schematic diagram showing an example of a captured image P in the first embodiment. Fig. 3A shows a captured image P in a frame f_1, and Fig. 3B shows a captured image P in a frame f_2 captured at a time later than the frame f_1. The interval between frames f_1 and f_2 is not limited. Hereinafter, a region in the captured image P where an occupant 8 is captured will be referred to as an occupant region P_8, and a region in the captured image P where a local high-luminance object 10 is captured will be referred to as a high-luminance object region P_10.

[0017] When the image sensor 1b captures the captured image P shown in FIG. 3A in frame f_1, it calculates an average luminance value of a specific region 11 of the captured image P. The specific region 11 in FIG. 3 is an area surrounded by a dashed line, and is an area assumed to be the head area of ​​the occupant 8. Then, the exposure and gain of the entire captured image P from the next frame onwards are determined so that the calculated average luminance value of the specific region 11 approaches a preset range. The preset range may be set to a range of luminance values ​​in which a specific portion of the occupant 8 can be detected with high accuracy.

[0018] For example, as shown in FIG. 3A, when a locally high-luminance object 10 is captured in the specific region 11, the average luminance value of the specific region 11 is likely to be higher than the preset range. In this case, the image sensor 1b performs image capture control such as exposure control and gain control of the entire captured image P so that the average luminance value of the specific region 11 in the next frame and thereafter approaches the preset range. As a result, the luminance value of the entire captured image P is reduced in the frame f_2 as shown in FIG. 3B. In this manner, the image sensor 1b performs image capture control so that the specific region 11 of the captured image P has a luminance value suitable for performing image recognition processing on the occupant 8. By such image capture control, it is difficult to determine the state of the occupant 8 in the captured image P of FIG. 3A because the area around the right eye of the occupant in the region P_8 is blown out due to the locally high-luminance object 10, but the luminance value is optimized in the captured image P of FIG. 3B, and the state of the occupant 8 can be determined.

[0019] Returning to Fig. 1, the occlusion determination device 100 determines whether or not the imaging unit 1 is in a occluded state where it is occluded by an occluding object 9. The occlusion determination device 100 includes an image acquisition unit 4, a pixel selection unit 5, an occluding object determination unit 6, and an occlusion state determination unit 7. Each component of the occlusion determination device 100 will be described below.

[0020] The image acquisition unit 4 acquires captured images P of a plurality of frames from the imaging unit 1. Then, the image acquisition unit 4 outputs the acquired captured images P of a plurality of frames to the pixel selection unit 5. As described above, the captured image P acquired by the image acquisition unit 4 is an image in which the entire captured image P is subjected to imaging control so that the luminance value of the specific region 11 approaches a preset range.

[0021] The pixel selection unit 5 selects a plurality of reference pixels 12 from within a specific region 11 of the captured image P acquired from the image acquisition unit 4. The pixel selection unit 5 also selects a plurality of reference pixels 12 whose pixel positions differ depending on the frame. Then, the pixel selection unit 5 outputs position information of the plurality of reference pixels 12 to the obstructing object determination unit 6.

[0022] The obstructing object determination unit 6 determines whether or not an obstructing object 9 is present based on the luminance values ​​of the plurality of reference pixels 12 selected by the pixel selection unit 5. Then, the obstructing object determination unit 6 outputs the determination result to the obstructed state determination unit 7.

[0023] When the obstructing object determination unit 6 determines that an obstructing object 9 is present in a preset number of frames equal to or greater than two, the obstructed state determination unit 7 determines that the imaging unit 1 is in an obstructed state, obstructed by the obstructing object 9. Then, the obstructed state determination unit 7 outputs a signal indicating that the imaging unit 1 is in an obstructed state. On the other hand, if the obstructing object determination unit 6 does not determine that an obstructing object 9 is present in a preset number of frames equal to or greater than two, the obstructed state determination unit 7 determines that the imaging unit 1 is not in an obstructed state where it is obstructed by an obstructing object 9. Then, the obstructed state determination unit 7 outputs a signal indicating that the imaging unit 1 is not in an obstructed state. The detailed processing operations of each component of the occlusion determination device 100 will be described later in the description of the occlusion determination method.

[0024] Next, a description will be given of an example of a hardware configuration of the occlusion determination device 100. Fig. 4 is a schematic diagram showing an example of a hardware configuration of the occlusion determination device 100 of embodiment 1. The image acquisition unit 4, pixel selection unit 5, occlusion object determination unit 6, and occlusion state determination unit 7 of the occlusion determination device 100 may be a processing circuit 13 which is dedicated hardware as shown in Fig. 4A, or may be a processor 14 which executes a program stored in a memory 15 as shown in Fig. 4B.

[0025] 4A, when the image acquisition unit 4, pixel selection unit 5, shielding object determination unit 6, and shielding state determination unit 7 are dedicated hardware, the processing circuit 13 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-programmable Gate Array), or a combination thereof. The functions of each of the image acquisition unit 4, pixel selection unit 5, shielding object determination unit 6, and shielding state determination unit 7 may be realized by a processing circuit, or the functions of each unit may be realized by a single processing circuit.

[0026] As shown in FIG. 4B, when the image acquisition unit 4, the pixel selection unit 5, the shielding object determination unit 6, and the shielding state determination unit 7 are the processor 14, the functions of each unit are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 15. The processor 14 realizes the functions of the image acquisition unit 4, the pixel selection unit 5, the shielding object determination unit 6, and the shielding state determination unit 7 by reading and executing the program stored in the memory 15. That is, the image acquisition unit 4, the pixel selection unit 5, the shielding object determination unit 6, and the shielding state determination unit 7 are provided with the memory 15 for storing a program that, when executed by the processor 14, results in the execution of each step shown in FIG. 5 described later. In addition, it can be said that these programs cause a computer to execute the procedures or methods of the image acquisition unit 4, the pixel selection unit 5, the shielding object determination unit 6, and the shielding state determination unit 7.

[0027] Here, the processor 14 is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor), etc. The memory 15 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM), or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

[0028] Note that the functions of the image acquisition unit 4, pixel selection unit 5, occlusion determination unit 6, and occlusion state determination unit 7 may be partially realized by dedicated hardware and partially realized by software or firmware. In this manner, the processing circuit 13 in the occlusion determination device 100 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

[0029] 1, when a signal indicating that the vehicle is not in an obstructed state is input from the obstruction state determination unit 7 of the obstruction determination device 100, the monitoring processing unit 2 performs various monitoring processes for the vehicle occupant 8 by image recognition processing of the vehicle occupant 8. Specifically, the monitoring processing unit 2 performs at least one of a monitoring process of determining the state of the occupant 8, such as drowsy driving and inattentive driving, and a personal authentication of the occupant 8.

[0030] The notification unit 3 notifies the occupant 8 that the imaging unit 1 is in an obstructed state when a signal indicating that the imaging unit 1 is in an obstructed state is input from the obstruction state determination unit 7 of the occlusion determination device 100. The notification unit 3 is, for example, a speaker and a display provided in the vehicle, and outputs at least one of a sound and an image that prompts the occupant 8 to remove the obstruction 9. In addition, an ECU (Electronic Control Unit) (not shown) installed in the vehicle may implement driving assistance such as following the vehicle in front, lane keep assist, rear side warning, and damage mitigation braking when a signal indicating an obstructed state is input from the obstruction state determination section 7 of the obstruction determination device 100.

[0031] Next, a description will be given of a method for determining a occlusion state by the occlusion determination device 100. Fig. 5 is a flowchart showing the occlusion state determination method according to the embodiment 1. The occlusion determination device 100 starts operation, for example, when the imaging unit 1 starts imaging, and repeats the operation while the vehicle is traveling.

[0032] First, in step S10, the image acquisition unit 4 acquires captured images P of a plurality of frames that are image-controlled from the imaging unit 1. FIG. 6 is a schematic diagram showing an example of the captured image P of the first embodiment. FIG. 6 shows a captured image P acquired by the image acquisition unit 4 when there is an obstruction 9 that prevents the imaging unit 1 shown in FIG. 2A from capturing an image of the occupant 8. The area in the captured image P where the obstruction 9 is captured is referred to as the obstruction area P_9. When the obstruction 9 is present in front of the imaging element 1b as shown in FIG. 2A, the obstruction area P_9 has a lower luminance than other areas as shown in FIG. 6. This is because the obstruction 9 is not irradiated by the light-emitting element 1a and is captured darker than other areas.

[0033] In step S10, the image acquisition unit 4 may acquire all captured images P captured by the imaging unit 1, or may acquire every predetermined number of frames to reduce the processing load. In the following, an example will be described in which the imaging unit 1 captures images at intervals of 60 fps, and the image acquisition unit 4 acquires captured images P every 15 frames. That is, an example will be described in which the image acquisition unit 4 acquires captured images P of 4 frames (f_15, f_30, f_45, f_60) per second.

[0034] In step S20, the pixel selection unit 5 selects multiple reference pixels 12 whose pixel positions vary from frame to frame from within the specific region 11 of the captured image P acquired from the image acquisition unit 4. When determining whether or not an obstruction 9 is present based on the luminance values ​​of the reference pixels 12 in steps S30 and S40 described below, calculating the luminance values ​​of all pixels in the entire captured image P or the specific region 11 would result in a large processing load. Therefore, the pixel selection unit 5 selects multiple reference pixels 12 by thinning out a predetermined number of pixels from all pixels in the specific region 11 of the captured image P.

[0035] Here, an example of a method for selecting reference pixels 12 by pixel selection unit 5 will be described. Fig. 7 is a schematic diagram for explaining the processing operation in step S20 in embodiment 1. Figs. 7A to 7D show specific regions 11 in frames f_15, f_30, f_45, and f_60, respectively, and white dots indicate reference pixels 12. The pixel selection unit 5 divides each of the width w and height h of the specific region 11 into four. A preset number of reference pixels 12 are selected. Here, an example will be described in which there are a total of nine reference pixels 12, three vertically and three horizontally. The interval between adjacent reference pixels 12 is w / 4 in width and 4 / h in height. Then, the pixel selection unit 5 selects a plurality of reference pixels 12 whose pixel positions differ depending on the frame. For example, in the frame f_15 shown in FIG. 7A, the pixel selection unit 5 selects the nine pixels at the top left of the specific region 11 as the reference pixels 12. In the frame f_30 shown in FIG. 7B, the pixel selection unit 5 selects the pixels at positions shifted by w / 2 to the right from the positions of the nine reference pixels 12 selected in the frame f_15 shown in FIG. 7A as the reference pixels 12. In the frame f_45 shown in FIG. 7C, the pixel selection unit 5 selects the pixels at positions shifted by h / 2 downward from the positions of the nine reference pixels 12 selected in the frame f_30 shown in FIG. 7B as the reference pixels 12. In the frame f_60 shown in FIG. 7D, the pixel selection unit 5 selects the pixels at positions shifted by w / 2 to the left from the positions of the nine reference pixels 12 selected in the frame f_45 shown in FIG. 7C as the reference pixels 12. For the frames f_60 and after, the above-mentioned operations in the frames f_15, f_30, f_45, and f_60 are repeated.

[0036] The method of selecting the reference pixels 12 by the pixel selection unit 5 is not limited to the above-mentioned method. Although an example has been shown in which the reference pixels 12 are selected from the pixels of the specific region 11 in order to reduce the processing load, they may be selected from the pixels of the entire captured image P. Furthermore, it is preferable that the pixel selection unit 5 selects a plurality of reference pixels 12 whose pixel positions differ depending on the frame such that the plurality of reference pixels 12 are scattered throughout the entire captured image P or the specific region 11 in a preset number of frames equal to or greater than two in step S70 described below.

[0037] Returning to FIG. 5, in step S30, the obstructing object determination unit 6 calculates the luminance values ​​of the plurality of reference pixels 12 selected by the pixel selection unit 5.

[0038] In step S40, the obstructing object determination unit 6 determines whether or not an obstructing object 9 that blocks imaging by the imaging unit 1 is present, based on the luminance value of the reference pixel 12 calculated in step S30.

[0039] Here, an example of a method for determining whether or not there is an obstruction 9 that obstructs imaging by the imaging unit 1 based on the luminance value of the reference pixel 12 in step S40 will be described. For example, if the number of reference pixels 12 whose luminance values ​​of the reference pixels 12 calculated in step S30 are less than the preset threshold is equal to or greater than a predetermined number, the obstruction determination unit 6 determines that there is an obstruction 9 that obstructs imaging by the imaging unit 1 (YES in step S40). On the other hand, if the number of reference pixels 12 whose luminance values ​​of the reference pixels 12 calculated in step S30 are less than the preset threshold is less than a predetermined number, the obstruction determination unit 6 determines that there is no obstruction 9 that obstructs imaging by the imaging unit 1 (NO in step S40). The preset threshold is appropriately set according to the pixel range of the image sensor that is the imaging element 1b, and the predetermined number is appropriately set according to the number of reference pixels 12 and the area of ​​the specific region 11.

[0040] More specifically, the following description will be given with reference to FIG. 7. The pixels in the region P_9 of the obstructing object are low-luminance pixels less than the threshold, and the pixels in the specific region 11 other than the region P_9 of the obstructing object are high-luminance pixels equal to or greater than the threshold, and the predetermined number is four. In this case, the number of reference pixels 12 less than the threshold is nine in the frame f_15 shown in FIG. 7A, nine in the frame f_30 shown in FIG. 7B, seven in the frame f_45 shown in FIG. 7C, and nine in the frame f_60 shown in FIG. 7D. In each of the frames f_15, f_30, f_45, and f_60, the number of reference pixels 12 less than the threshold is equal to or greater than the predetermined number. Therefore, the obstructing object determination unit 6 determines that the obstructing object 9 exists in all of the frames f_15, f_30, f_45, and f_60 (YES in step S40).

[0041] 5, when the obstructing object determination unit 6 determines that there is an obstructing object 9 interfering with imaging by the imaging unit 1 (YES in step S40), it outputs a signal indicating the number of frames in which the obstructing object 9 is present to the obstructing state determination unit 7 (step S50). On the other hand, when the obstructing object determination unit 6 determines that there is no obstructing object 9 interfering with imaging by the imaging unit 1 (NO in step S40), it outputs a signal indicating the number of frames in which the obstructing object 9 is not present to the obstructing state determination unit 7 (step S60). When the sum of the number of frames in which the obstructing object 9 is present in step S50 and the number of frames in which the obstructing object 9 is not present in step S60 becomes a preset number of frames equal to or greater than two in step S70 described later, the obstructing object determination unit 6 outputs a signal to the obstructing state determination unit 7 (steps S50 and S60).

[0042] In step S70, the shielding state determination unit 7 determines whether the imaging unit 1 is in a shielding state where it is shielded by a shielding object 9. Specifically, the shielding state determination unit 7 determines that the imaging unit 1 is in a shielding state when the shielding object determination unit 6 determines that the shielding object 9 is present in a preset number of frames that is two or more. Also, the shielding state determination unit 7 determines that the imaging unit 1 is not in a shielding state when the shielding object determination unit 6 does not determine that the shielding object 9 is present in a preset number of frames that is two or more.

[0043] The preset number of frames, which is equal to or greater than two, may be appropriately set depending on the number of frames of the captured images P acquired by the image acquisition section 4 in step S10, for example. Here, the preset number of frames equal to or greater than 2 is set to 4. In Fig. 7, the obstructing object determination unit 6 determines that the obstructing object 9 is present in four frames (frames f_15, f_30, f_45, and f_60) in one second, and therefore outputs a signal indicating that the number of frames in which the obstructing object 9 is present is four to the obstructing state determination unit 7 (step S50). Therefore, the obstructing state determination unit 7 determines YES in step S70.

[0044] If the result of the determination in step S70 is YES, the shielded state determination unit 7 outputs a signal indicating that the imaging unit 1 is in a shielded state where it is shielded by the shielding object 9 (step S80). On the other hand, if the result of the determination in step S70 is NO, the shielded state determination unit 7 outputs a signal indicating that the imaging unit 1 is not in a shielded state where it is shielded by the shielding object 9 (step S90). This completes the processing operation of the occlusion determination device 100.

[0045] Furthermore, in step S80, the obstructed state determination unit 7 may output a signal indicating that the imaging unit 1 is in a obstructed state, that is, obstructed by the obstruction 9, to the notification unit 3 and the ECU. Then, the notification unit 3 notifies the occupant 8 that the imaging unit 1 is in a obstructed state, that is, obstructed by the obstruction 9. Furthermore, the ECU performs driving assistance.

[0046] Furthermore, in step S90, the obstructed state determination unit 7 may output a signal indicating that the imaging unit 1 is not in an obstructed state where it is obstructed by the obstructing object 9 to the monitoring processing unit 2. Then, the monitoring processing unit 2 performs various monitoring processes targeting the occupant 8 of the vehicle by image recognition processing of the occupant 8.

[0047] Next, the effect of the occlusion determination device 100 in this embodiment will be described with reference to Figs. 8 to 10. Fig. 8 is a flowchart showing a method for determining an occlusion state of the occlusion determination device according to a comparative example. Fig. 9 is a schematic diagram for explaining the processing operation of the occlusion determination device according to the comparative example. Fig. 10 is a schematic diagram for explaining the processing operation of the occlusion determination device 100 according to the first embodiment. In the occlusion determination device according to the comparative example, the same components as those in the occlusion determination device 100 according to this embodiment are given the same reference numerals. Figs. 9 and 10 show a specific region 11 in a captured image P, and white dots indicate reference pixels 12. Figs. 9A to D and Figs. 10A to D show the specific region 11 in frames f_15, f_30, f_45, and f_60, respectively. Figs. 9 and 10 show a case where there is no occlusion object 9 that prevents the imaging unit 1 from imaging the occupant 8 shown in Fig. 2B, and a local high-luminance object 10 is present in the imaging range of the imaging unit 1. When a locally high-luminance object 10 is present in the imaging range of the imaging unit 1, the luminance value of the captured image P is reduced by imaging control by the imaging unit 1 as shown in Fig. 3. Then, the obstructing object determination unit 6 determines whether or not an obstructing object 9 is present based on the luminance values ​​of a plurality of reference pixels 12, not on whether or not the presence of the obstructing object 9 has actually been recognized. Therefore, when the luminance value of the entire captured image P is reduced by imaging control by the imaging unit 1, there is a possibility that the obstructing object determination unit 6 cannot correctly determine whether or not an obstructing object 9 is present. For example, in the processing operation of the obstructing object determination device of the comparative example described later, although an obstructing object 9 does not actually exist, it is erroneously determined to be in an obstructed state.

[0048] The processing operation of the occlusion determination device of the comparative example will be described with reference to Figures 8 and 9. The processing operation of the occlusion determination device 100 of the present embodiment in Figure 5 and the processing operation of the occlusion determination device of the comparative example in Figure 8 differ in step 20 in Figure 5 and step 21 in Figure 8. In step S21 in Figure 8, the pixel selection unit 5 selects multiple reference pixels 12 at the same pixel positions in all frames. That is, the positions of the multiple reference pixels 12 are fixed, and Figure 9 shows an example in which nine reference pixels 12 in the center of the specific region 11 are selected.

[0049] In step S30, the obstructing object determination unit 6 calculates the luminance values ​​of the multiple reference pixels 12. When a localized high-luminance object 10 is captured in a part of the specific region 11, the luminance of the entire captured image P is significantly reduced by imaging control, but the pixels in the region P_10 of the high-luminance object are brighter than the other regions.

[0050] Then, in step S40, the shielding object determination unit 6 determines whether or not a shielding object 9 that obstructs the imaging unit 1 is present, based on the luminance value of the reference pixel 12 calculated in step S30. As in the shielding determination device 100 of the first embodiment described above, the shielding object determination unit 6 determines that a shielding object 9 is present when the number of reference pixels 12 whose luminance values ​​calculated in step S30 are less than a preset threshold is equal to or greater than a predetermined number (YES in step S40). On the other hand, the shielding object determination unit 6 determines that a shielding object 9 is not present when the number of reference pixels 12 whose luminance values ​​calculated in step S30 are less than a preset threshold is less than a predetermined number (NO in step S40). Here, the pixels in the high-luminance object region P_10 are high-luminance pixels that are equal to or greater than the threshold, and the pixels in the specific region 11 other than the high-luminance object region P_10 are low-luminance pixels that are less than the threshold, and the predetermined number is set to four. In this case, the number of reference pixels 12 whose luminance values ​​are less than the threshold in FIGS. 9A to 9D is five. Therefore, the obstructing object determination unit 6 erroneously determines that the obstructing object 9 is present in all of frames f_15, f_30, f_45, and f_60 (YES in step S40). Then, the obstructing object determination unit 6 outputs a signal indicating that the number of frames in which the obstructing object 9 is present is 4 to the obstructing state determination unit 7 (step S50).

[0051] Next, in step S70, the shielding state determination unit 7 determines whether the imaging unit 1 is in a shielding state where the imaging unit 1 is shielded by the shielding object 9. Specifically, the shielding state determination unit 7 determines that the imaging unit 1 is in a shielding state when the shielding object determination unit 6 determines that the shielding object 9 exists in a preset number of frames that is two or more. Also, the shielding state determination unit 7 determines that the imaging unit 1 is not in a shielding state when the shielding object determination unit 6 does not determine that the shielding object 9 exists in a preset number of frames that is two or more. As in the shielding determination device 100 of the above-mentioned first embodiment, the preset number is set to four. In FIG. 9, the shielding object determination unit 6 determines that the shielding object 9 exists in four frames, f_15, f_30, f_45, and f_60, as described above, and outputs a signal indicating that the number of frames in which the shielding object 9 exists is four to the shielding state determination unit 7 (step S50). Therefore, the obstructed state determination unit 7 determines YES in step S70, and outputs a signal indicating that the imaging unit 1 is in a obstructed state where it is obstructed by the obstructing object 9 (step S80).

[0052] In this way, in the occlusion determination device according to the comparative example, when a localized high-luminance object 10 is captured in a part of the specific region 11, the entire captured image P is controlled to be captured. This significantly reduces the luminance of the entire specific region 11. As a result, even when there is no obstruction 9 preventing the imaging unit 1 from capturing an image of the occupant 8, the occlusion determination device may erroneously determine that the imaging unit 1 is in an obstructed state due to the obstruction 9.

[0053] The process of the occlusion determination device 100 in this embodiment to solve this problem will be described with reference to Fig. 5 and Fig. 10. First, in step S20, the pixel selection unit 5 selects a plurality of reference pixels 12 from within a specific region 11 of the captured image P acquired from the image acquisition unit 4. The method of selecting the reference pixels 12 by the pixel selection unit 5 is the same as that shown in Fig. 7 above. As shown in Figure 10, the pixel selection unit 5 selects multiple reference pixels 12 whose pixel positions differ depending on the frame, so that there is a frame f_15 in which many pixels in the high-brightness object area P_10 are selected as reference pixels 12, and frames f_30, f_45, and f_60 in which few pixels are selected.

[0054] In step S30, the obstructing object determination unit 6 calculates the luminance values ​​of the plurality of reference pixels 12, respectively.

[0055] Then, in step S40, the shielding object determination unit 6 determines whether or not a shielding object 9 that obstructs the imaging unit 1 from capturing an image is present based on the luminance value of the reference pixel 12 calculated in step S30. The determination method is the same as that of the shielding determination device according to the comparative example described above. The number of reference pixels 12 that are less than the threshold is 1 in the frame f_15 shown in FIG. 10A, 7 in the frame f_30 shown in FIG. 10B, 8 in the time frame f_45 shown in FIG. 10C, and 6 in the time frame f_60 shown in FIG. 10D. In f_30, f_45, and f_60, the number of reference pixels 12 that are less than the preset threshold is equal to or greater than a predetermined number, so that the shielding object determination unit 6 erroneously determines that a shielding object 9 is present (YES in step S40). Then, the shielding object determination unit 6 outputs a signal indicating that the number of frames in which a shielding object 9 is present is 3, that is, f_30, f_45, and f_60, to the shielding state determination unit 7 (step S50). On the other hand, in frame f_15 where many pixels in the high-luminance object region P_10 are selected as reference pixels 12, the number of reference pixels 12 that are less than the preset threshold is less than the predetermined number. Therefore, the obstructing object determination unit 6 correctly determines that an obstructing object 9 does not exist in frame f_15 (NO in step S40). Then, the obstructing object determination unit 6 outputs a signal indicating that the number of frames in which an obstructing object 9 does not exist is 1 in f_15 to the obstructing state determination unit 7 (step S60).

[0056] Next, in step S70, the shielding state determination unit 7 determines whether the imaging unit 1 is in a shielding state in which it is shielded by the shielding object 9. Specifically, when the shielding object determination unit 6 determines that the shielding object 9 exists in a preset number of frames of 2 or more, the shielding state determination unit 7 determines that the imaging unit 1 is in a shielding state in which it is shielded by the shielding object 9. Also, when the shielding object determination unit 6 does not determine that the shielding object 9 exists in a preset number of frames of 2 or more, the shielding state determination unit 7 determines that the imaging unit 1 is not in a shielding state. As in the shielding determination device 100 of the above-mentioned embodiment 1 and the shielding determination device according to the comparative example, the preset number of frames of 2 or more is set to 4. In the case shown in FIG. 10, the shielding object determination unit 6 outputs a signal indicating that the number of frames in which the shielding object 9 exists is 3 and a signal indicating that the number of frames in which the shielding object 9 does not exist is 1 to the shielding state determination unit 7. Therefore, the shielding object determination unit 6 does not determine that the shielding object 9 exists in 4 frames. Therefore, the obstructed state determination unit 7 determines NO in step S70, and outputs a signal indicating that the imaging unit 1 is not in an obstructed state where it is obstructed by the obstructing object 9 (step S90).

[0057] In this manner, the occlusion determination device 100 in the present embodiment is characterized by including a pixel selection unit 5 that selects a plurality of reference pixels 12 having different pixel positions from the captured image P depending on the frame, an occlusion determination unit 6 that determines whether or not there is an occlusion 9 that hinders imaging by the imaging unit 1 based on the luminance values ​​of the selected plurality of reference pixels 12, and an occlusion state determination unit 7 that determines that the imaging unit 1 is in an occlusion state in which the occlusion 9 is occluded by the occlusion 9 when it is determined that the occlusion 9 is present in two or more preset number of frames, and determines that the imaging unit 1 is not in an occlusion state in which the occlusion 9 is occluded by the occlusion 9 when it is not determined that the occlusion 9 is present in two or more preset number of frames. In this manner, the accuracy of determining the occlusion state can be improved by determining whether or not there is an occlusion 9 based on the luminance values ​​of the plurality of reference pixels 12 having different pixel positions depending on the frame, and determining whether or not there is an occlusion state based on whether or not it is determined that the occlusion 9 is present in two or more preset number of frames.

[0058] Furthermore, the pixel selection unit 5 selects a plurality of reference pixels 12 by thinning out a predetermined number of pixels from all pixels in the captured image P or the specific region 11 acquired from the image acquisition unit 4 (for example, step S20 in FIG. 5). This allows a reduction in the processing load when determining whether or not an obstruction 9 exists based on the luminance values ​​of the reference pixels 12 (for example, steps S30 and S40 in FIG. 5) compared to the case of calculating the luminance values ​​of all pixels in the captured image P or the specific region 11.

[0059] On the other hand, compared to when all pixels in the captured image P or the specific region 11 are used as the reference pixels 12, when pixels obtained by thinning out a predetermined number of pixels from all pixels in the captured image P or the specific region 11 are used as the multiple reference pixels 12, there is a risk that the accuracy of determining whether the state is blocked will be lower. Therefore, the pixel selection unit 5 selects multiple reference pixels 12 whose pixel positions differ depending on the frame (for example, step S20 in FIG. 5). In addition, the pixel selection unit 5 selects multiple reference pixels 12 whose pixel positions differ depending on the frame so that the multiple reference pixels 12 are scattered throughout the captured image P or the specific region 11 in a preset number of frames of two or more. Then, in step S70 in FIG. 5, the blocking state determination unit 7 determines whether the state is blocked based on whether the blocking object determination unit 6 has determined that a blocking object 9 exists in a preset number of frames of two or more. As a result, it is possible to determine whether the state is blocked based on the luminance values ​​of the multiple reference pixels 12 scattered throughout the captured image P or the specific region 11 in a preset number of frames of two or more. Therefore, it is possible to reduce the processing load while improving the accuracy of determining the occlusion state.

[0060] In addition, in step S40 of Figure 5, an example is shown in which it is determined that an obstructing object 9 exists when, among the multiple reference pixels 12, the number of reference pixels 12 whose luminance values ​​are less than a threshold is a predetermined number or more, and it is determined that an obstructing object 9 does not exist when the number of reference pixels 12 whose luminance values ​​are less than the threshold is less than the predetermined number. Here, consider a case in which it is determined that an obstructing object 9 is present when the average luminance value of the plurality of reference pixels 12 is less than a preset threshold, and it is determined that an obstructing object 9 is not present when the average luminance value of the plurality of reference pixels 12 is equal to or greater than the threshold. In this case, summation and division are performed to calculate the average luminance value of the plurality of reference pixels 12. In this way, counting the number of reference pixels 12 whose luminance values ​​are less than the threshold can reduce the processing load, rather than calculating the average luminance value of the plurality of reference pixels 12.

[0061] 5, the obstructed state determination unit 7 outputs a signal indicating that the imaging unit 1 is in a obstructed state where it is obstructed by the obstruction 9 to, for example, the notification unit 3 provided in the vehicle. Then, the notification unit 3 notifies the occupant 8 that the imaging unit 1 is in a obstructed state where it is obstructed by the obstruction 9. This enables the occupant 8 to take measures to remove the obstruction 9.

[0062] 5, the shielding state determination unit 7 outputs a signal indicating that the imaging unit 1 is not in a shielded state where it is shielded by a shielding object 9 to the monitoring processing unit 2. That is, in the DMS, which is an occupant monitoring device including the shielding determination device 100 of this embodiment, when the shielding state determination unit 7 determines that the imaging unit 1 is not in a shielded state, the monitoring processing unit 2 performs a monitoring process targeting the occupant 8, and when the shielding state determination unit 7 determines that the imaging unit 1 is in a shielded state, the monitoring processing unit 2 does not perform the monitoring process. As a result, when image recognition processing for the vehicle occupant 8 can be performed, the monitoring process is performed, and when image recognition processing for the vehicle occupant 8 cannot be performed, the monitoring process is not performed, thereby reducing the processing load.

[0063] In addition, if an increase in processing load is acceptable, in step S40 of Figure 5, the occluding object determination unit 6 may determine that an occluding object 9 is present if the average luminance value of the multiple reference pixels 12 is less than a predetermined threshold, and may determine that an occluding object 9 is not present if the average luminance value of the multiple reference pixels 12 is equal to or greater than the threshold.

[0064] 2 shows an example in which the light-emitting element 1a and the imaging element 1b are disposed apart on both sides of the center display. When the light-emitting element 1a and the imaging element 1b are disposed apart in this manner, there is a high possibility that the region P_9 of the obstruction will be a low-luminance region in the captured image P. This is because the obstruction 9 placed so as to cover the imaging element 1b is not illuminated by the light-emitting element 1a, or the obstruction 9 illuminated by the light-emitting element 1a is not within the imaging range of the imaging element 1b.

[0065] The light emitting element 1a and the imaging element 1b are not limited to the above-mentioned positions, and the light emitting element 1a and the imaging element 1b may be arranged in close proximity. For example, the light emitting element 1a and the imaging element 1b may be arranged in close proximity on the driver's side of the center display. In this case, the obstruction 9 is often irradiated by the light emitting element 1a and imaged, and the area P_9 of the obstruction in the captured image P is likely to be a high-luminance area. Therefore, in step S40 of FIG. 5, the obstruction determination unit 6 may determine that the obstruction 9 exists (YES in step S40) if the number of reference pixels 12 whose luminance values ​​of the reference pixels 12 calculated in step S30 are equal to or greater than a preset threshold is equal to or greater than a predetermined number. Then, the obstruction determination unit 6 may determine that the obstruction 9 does not exist (NO in step S40) if the number of reference pixels 12 whose luminance values ​​of the reference pixels 12 calculated in step S30 are equal to or greater than a preset threshold is not equal to or greater than a predetermined number. In this way, the conditions for determining whether or not the obstructing object 9 is present by the obstructing object determining unit 6 in step S40 may be set appropriately in accordance with the design of the imaging unit 1.

[0066] Embodiment 2 The occlusion determination device 100 in the second embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing a part of a DMS including the occlusion determination device 100 in the second embodiment. The occlusion determination device 100 in the second embodiment is characterized by including a traveling determination unit 17. The other configurations of the occlusion determination device 100 are the same as those in the first embodiment. The same reference numerals are used for the same configurations as those in the first embodiment.

[0067] The traveling determination unit 17 acquires traveling information indicating whether or not the vehicle is traveling from, for example, a control unit 18 provided in the vehicle that controls the vehicle speed and shift. Then, the traveling determination unit 17 judges whether or not the vehicle is traveling based on the traveling information. When the traveling determination unit 17 determines that the vehicle is traveling, it outputs a signal to the image acquisition unit 4 to acquire a plurality of frames of captured images P from the imaging unit 1, and when it determines that the vehicle is not traveling, it outputs a signal to end the processing operation of the occlusion determination device 100.

[0068] Next, a description will be given of a method for determining an occlusion state by the occlusion determination device 100. Fig. 12 is a flowchart showing a method for determining an occlusion state according to the second embodiment.

[0069] In step S01, the traveling determination unit 17 acquires traveling information indicating whether the vehicle is traveling or not, and determines whether the vehicle is traveling or not. If the traveling determination unit 17 determines that the vehicle is traveling (YES in step S01), the process proceeds to step S10. Steps S10 to S90 are the same as the obstruction state determination method of the first embodiment. If the traveling determination unit 17 determines that the vehicle is not traveling (NO in step S01), the process ends the processing operation of the obstruction determination device 100.

[0070] 12, when the traveling determination unit 17 determines that the vehicle is traveling (YES in step S01), the shielded state determination unit 7 determines whether or not the vehicle is in the shielded state in step S70. When the traveling determination unit 17 determines that the vehicle is not traveling (NO in step S01), the shielded state determination unit 7 does not determine whether or not the vehicle is in the shielded state in step S70.

[0071] As in the first embodiment, the occlusion determination device 100 in the present embodiment is characterized by including a pixel selection unit 5 that selects a plurality of reference pixels 12 having different pixel positions from the captured image P depending on the frame, an occlusion determination unit 6 that determines whether or not there is an occlusion object 9 that hinders imaging by the imaging unit 1 based on the luminance values ​​of the selected plurality of reference pixels 12, and an occlusion state determination unit 7 that determines that the imaging unit 1 is in an occlusion state in which the occlusion object 9 is occluded by the occlusion object 9 when it is determined that the occlusion object 9 is present in two or more preset number of frames, and determines that the imaging unit 1 is not in an occlusion state in which the occlusion object 9 is occluded by the occlusion object 9 when it is determined that the occlusion object 9 is present in two or more preset number of frames. In this way, the accuracy of determining the occlusion state can be improved by determining whether or not there is an occlusion object 9 based on the luminance values ​​of the plurality of reference pixels 12 having different pixel positions depending on the frame, and determining whether or not there is an occlusion state based on whether or not it is determined that the occlusion object 9 is present in two or more preset number of frames.

[0072] Moreover, the occlusion determination device 100 in this embodiment includes a traveling determination unit 17 that acquires traveling information indicating whether the vehicle is traveling or not, and determines whether the vehicle is traveling or not based on the traveling information. Then, the occlusion state determination unit 7 determines whether the vehicle is in an occlusion state when the traveling determination unit 17 determines that the vehicle is traveling, and does not determine whether the vehicle is in an occlusion state when the traveling determination unit 17 determines that the vehicle is not traveling. When the vehicle is not traveling, there is no need to perform monitoring processing for the occupant 8, and therefore there is no problem even if the imaging of the occupant 8 by the imaging unit 1 is hindered by an obstruction 9. Therefore, when the vehicle is not traveling, the processing operation of determining whether or not the vehicle is in an obstructed state in step S70 of Fig. 12 is not performed, thereby reducing the processing load of the obstruction determination device 100.

[0073] 12 before step S10, the present invention is not limited to this, and may be performed at least before step S70. In the flowchart shown in FIG. 12, performing the processing operation of step S01 at an upstream stage has a greater effect of reducing the processing load of the occlusion determination device 100.

[0074] Embodiment 3 The occlusion determination device 100 in the third embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing a part of a DMS including the occlusion determination device 100 in the third embodiment. The occlusion determination device 100 in the third embodiment is characterized by including a face area determination unit 19. The rest of the configuration of the occlusion determination device 100 is the same as in the first embodiment. The same reference numerals are used for the same configuration as in the first embodiment.

[0075] The facial area determination unit 19 acquires facial area detection information indicating whether or not the detection of the facial area of ​​the occupant 8 has been successful from the facial area detection unit 20, which detects the facial area of ​​the occupant 8 from the captured image P output from the imaging unit 1. Then, the facial area determination unit 19 determines whether or not the detection of the facial area of ​​the occupant 8 has been successful based on the facial area detection information. If the facial area determination unit 19 determines that the detection of the facial area of ​​the occupant 8 has been successful, it outputs a signal to terminate the processing operation of the occlusion determination device 100, and if it determines that the detection of the facial area of ​​the occupant 8 has failed, it outputs a signal to the image acquisition unit 4 to acquire a plurality of frames of captured images P from the imaging unit 1.

[0076] Next, a description will be given of a method for determining an occlusion state by the occlusion determination device 100. Fig. 14 is a flowchart showing a method for determining an occlusion state according to the second embodiment.

[0077] In step S02, the facial area determination unit 19 acquires facial area detection information indicating whether or not the detection of the facial area of ​​the occupant 8 has been successful, and determines whether or not the detection of the facial area of ​​the occupant 8 has been successful. If the facial area determination unit 19 determines that the detection of the facial area of ​​the occupant 8 has been successful (YES in step S02), it terminates the processing operation of the occlusion determination device 100. If the facial area determination unit 19 determines that the detection of the facial area of ​​the occupant 8 has failed (NO in step S02), it proceeds to step S10. Steps S10 to S90 are the same as the occlusion state determination method of the first embodiment.

[0078] 14, if the facial area determination unit 19 determines that the detection of the facial area of ​​the occupant 8 has been successful (YES in step S02), the occluded state determination unit 7 does not determine whether or not the occluded state exists in step S70. On the other hand, if the facial area determination unit 19 determines that the detection of the facial area of ​​the occupant 8 has failed (NO in step S01), the occluded state determination unit 7 determines whether or not the occluded state exists in step S70.

[0079] As in the first embodiment, the occlusion determination device 100 in the present embodiment is characterized by including a pixel selection unit 5 that selects a plurality of reference pixels 12 having different pixel positions from the captured image P depending on the frame, an occlusion determination unit 6 that determines whether or not there is an occlusion object 9 that hinders imaging by the imaging unit 1 based on the luminance values ​​of the selected plurality of reference pixels 12, and an occlusion state determination unit 7 that determines that the imaging unit 1 is in an occlusion state in which the occlusion object 9 is occluded by the occlusion object 9 when it is determined that the occlusion object 9 is present in two or more preset number of frames, and determines that the imaging unit 1 is not in an occlusion state in which the occlusion object 9 is occluded by the occlusion object 9 when it is determined that the occlusion object 9 is present in two or more preset number of frames. In this way, the accuracy of determining the occlusion state can be improved by determining whether or not there is an occlusion object 9 based on the luminance values ​​of the plurality of reference pixels 12 having different pixel positions depending on the frame, and determining whether or not there is an occlusion state based on whether or not it is determined that the occlusion object 9 is present in two or more preset number of frames.

[0080] Moreover, the occlusion determination device 100 in this embodiment includes a face area determination unit 19 that acquires face area detection information indicating whether or not detection of the face area of ​​the occupant 8 has been successful, and determines whether or not detection of the face area of ​​the occupant 8 has been successful based on the face area detection information. If the face area determination unit 19 determines that detection of the face area of ​​the occupant 8 has been successful, the occlusion state determination unit 7 does not determine whether or not the occlusion state exists, and if the face area determination unit 19 determines that detection of the occupant 8 has failed, it determines whether or not the occlusion state exists. When the detection of the face area of ​​the occupant 8 is successful, the imaging of the occupant 8 by the imaging unit 1 is not hindered by the obstruction 9. Therefore, when the detection of the face area of ​​the occupant 8 is successful, the processing operation of determining whether or not the occupant is obstructed in step S70 of Fig. 14 is not performed, thereby reducing the processing load of the occlusion determination device 100.

[0081] 14 before step S10, the present invention is not limited to this, and may be performed at least before step S70. In the flowchart shown in FIG. 14, performing the processing operation of step S01 at an upstream stage has a greater effect of reducing the processing load of the occlusion determination device 100.

[0082] Also, an example has been shown in which the facial area determination unit 19 acquires facial area detection information indicating whether or not the detection of the facial area of ​​the occupant 8 has been successful, and determines whether or not the detection of the facial area of ​​the occupant 8 has been successful, but this is not limiting. The facial area determination unit 19 may acquire information indicating whether or not a specific part relating to the organs of the body of the occupant 8 that needs to be detected from the captured image P in order to execute image recognition processing on the occupant 8 in the monitoring processing unit 2 has been detected, and may determine whether or not the detection of the specific part has been successful based on that information.

[0083] Furthermore, the embodiments disclosed in this specification can be freely combined within the scope of the specification, and each embodiment can be modified or omitted as appropriate. [Explanation of symbols]

[0084] REFERENCE SIGNS LIST 1 imaging unit, 1a light emitting element, 1b imaging element, 2 monitoring processing unit, 3 notification unit, 4 image acquisition unit, 5 pixel selection unit, 6 obstruction determination unit, 7 obstruction state determination unit, 8 occupant, 9 obstruction, 10 high-brightness object, 11 specific area, 12 reference pixel, 13 processing circuit, 14 processor, 15 memory, 17 driving determination unit, 18 control unit, 19 face area determination unit, 20 face area detection unit, 100 obstruction determination device

Claims

1. an image acquisition unit that acquires a plurality of frames of captured images from an imaging unit that images the interior of a vehicle; a pixel selection unit that selects, from the captured image, a plurality of reference pixels whose pixel positions differ depending on the frame; an obstruction determination unit that determines whether or not an obstruction that obstructs imaging by the imaging unit is present based on a luminance value of the reference pixel; a blocking state determination unit that determines that the imaging unit is in a blocking state where the imaging unit is blocked by the blocking object when the blocking object determination unit determines that the blocking object exists in two or more preset number of frames, and that the imaging unit is not in the blocking state when the blocking object determination unit does not determine that the blocking object exists in the preset number of frames; An occlusion determination device comprising:

2. The pixel selection unit selects the plurality of reference pixels by thinning out a predetermined number of pixels from all pixels of the captured image or a specific area in which a specific portion of the occupant is included in the captured image. The occlusion determination device according to claim 1 .

3. The pixel selection unit selects the plurality of reference pixels having different pixel positions depending on the frame such that the plurality of reference pixels are scattered throughout the captured image or the specific region in the predetermined number of frames. The occlusion determination device according to claim 2 .

4. The occluding object determination unit determines that the occluding object is present when a number of the reference pixels whose luminance values ​​are less than a preset threshold value is equal to or greater than a predetermined number among the plurality of reference pixels, and determines that the occluding object is not present when a number of the reference pixels whose luminance values ​​are less than the threshold value is less than the predetermined number. The occlusion determination device according to any one of claims 1 to 3.

5. The vehicle further includes a travel determination unit that acquires travel information indicating whether the vehicle is traveling or not, and determines whether the vehicle is traveling or not based on the travel information, The obstructed state determination unit determines whether or not the vehicle is in the obstructed state when the movement determination unit determines that the vehicle is moving, and does not determine whether or not the vehicle is in the obstructed state when the movement determination unit determines that the vehicle is not moving. The occlusion determination device according to any one of claims 1 to 4.

6. a face area determination unit that obtains face area detection information indicating whether or not a face area of ​​an occupant has been successfully detected in the captured image, and determines whether or not the detection of the face area of ​​the occupant has been successfully detected based on the face area detection information, The occluded state determination unit does not determine whether or not the occluded state exists when the face area determination unit determines that the detection of the face area of ​​the occupant has been successful, and determines whether or not the occluded state exists when the face area determination unit determines that the detection of the face area of ​​the occupant has failed. The occlusion determination device according to any one of claims 1 to 5.

7. An imaging unit that captures an image of the inside of a vehicle; an image acquisition unit that acquires captured images of a plurality of frames from the imaging unit; a pixel selection unit that selects, from the captured image, a plurality of reference pixels whose pixel positions differ depending on the frame; an obstruction determination unit that determines whether or not an obstruction that obstructs imaging by the imaging unit is present based on a luminance value of the reference pixel; a blocking state determination unit that determines that the imaging unit is in a blocking state where the imaging unit is blocked by the blocking object when the blocking object determination unit determines that the blocking object exists in two or more preset number of frames, and that the imaging unit is not in the blocking state when the blocking object determination unit does not determine that the blocking object exists in the preset number of frames; a monitoring processing unit that performs a monitoring process of at least one of determining a state of an occupant of the vehicle by image recognition processing of the occupant and authenticating the occupant when the occupant is not in the blocked state by the blocking state determination unit, and does not perform the monitoring process when the occupant is in the blocked state by the blocking state determination unit; An occupant monitoring device comprising:

8. An occlusion determination method for an occlusion determination device including an image acquisition unit, a pixel selection unit, an occlusion object determination unit, and an occlusion state determination unit, comprising: The image acquisition unit acquires a plurality of frames of captured images from an imaging unit that captures an image of the inside of a vehicle; a pixel selection unit selecting, from the captured image, a plurality of reference pixels having pixel positions that differ depending on the frames; a step of the obstructing object determination unit determining whether or not there is an obstructing object that obstructs imaging by the imaging unit based on a luminance value of the reference pixel; and determining, when the blocking state determination unit determines that the blocking object is present in two or more of the preset number of frames, that the imaging unit is in a blocking state in which the imaging unit is blocked by the blocking object, and when the blocking state determination unit does not determine that the blocking object is present in two or more of the preset number of frames, that the imaging unit is not in the blocking state. How to judge occlusion.

Images