Anomaly detection system

Abstract

To provide an anomaly detection system capable of specifying a cause of a malfunction of an occupant monitoring apparatus more concretely.SOLUTION: An anomaly detection system 1 includes: an imaging unit 10 having an imaging surface 10a and a shielding section 30; and a control unit 20 controlling the imaging unit 10. The shielding section 30 is configured to be displaced between a shielding position and an open position, and has shielding section detection patterns PA1, PA2 formed on an inside surface thereof. The control unit 20 includes: a shield determination unit 21 configured to perform first anomaly determination processing to determine whether there is an object between the imaging surface 10a and an occupant; and an optical component anomaly determination unit 22 configured to perform second anomaly determination processing to determine, when the presence of the object is determined between the imaging surface 10a and the occupant in the first anomaly determination processing, whether an optical component included in the imaging unit 10 is faulty or not, after the first anomaly determination processing.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an abnormality detection system.

Background Art

[0002] Conventionally, there is an abnormality detection system for identifying the cause of failure in obtaining the face information of an occupant by an occupant monitoring device. In Patent Document 1, based on the position of an in-vehicle component detected by in-vehicle component position detection and the state of the driver's face estimated by a face state estimation means, a cause analysis means for analyzing whether the cause of failure in obtaining face information is due to an in-vehicle component, the state of the driver's face, or a factor different from these is disclosed in a technique related to an information processing device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Regarding identifying the cause of failure in obtaining the face information of an occupant by an occupant monitoring device, there is still room for improvement. For example, in an occupant monitoring device, due to a defect in the imaging unit itself that images the occupant, failure in obtaining the face information of the occupant may occur. An abnormality detection system that can more specifically identify the cause of the abnormal operation of the occupant monitoring device is desired.

[0005] An object of the present invention is to provide an abnormality detection system that can more specifically identify the cause of abnormal operation of an occupant monitoring device.

Means for Solving the Problems

[0006] The present invention provides an anomaly detection system that is mounted on a vehicle and positioned facing the occupants of the vehicle, comprising: an imaging unit having an imaging surface and a shielding portion capable of shielding the imaging surface; and a control unit that controls the imaging unit, wherein the shielding portion is configured to be displaceable between a shielding position that shields the imaging surface facing it and an open position that is retracted from the shielding position, and has a shielding portion detection pattern on the inner surface facing the imaging surface when positioned in the shielding position, and the control unit is configured to perform a first anomaly determination process to determine the presence or absence of an object between the imaging surface and the occupants, and in the first anomaly determination process, The optical component abnormality determination unit is configured to perform a second abnormality determination process after the first abnormality determination process to determine whether or not there is an abnormality in the optical component included in the imaging unit when it is determined that there is an object between the imaging surface and the occupant, wherein in the second abnormality determination process, the optical component abnormality determination unit performs imaging with the imaging unit while the shielding unit is positioned at the shielding position, determines that there is no abnormality in the optical component if the similarity between the captured image and the image corresponding to the shielding unit detection pattern is greater than or equal to a predetermined threshold, and determines that there is an abnormality in the optical component if the similarity is less than the predetermined threshold. [Effects of the Invention]

[0007] The anomaly detection system according to the present invention has the effect of being able to more specifically identify the cause of the malfunction of the occupant monitoring device. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing an anomaly detection system according to an embodiment. [Figure 2] Figure 2 is an exploded perspective view showing the imaging unit in the embodiment. [Figure 3] Figure 3 is a perspective view showing the imaging unit in the embodiment. [Figure 4] Figure 4 is a schematic diagram showing the first anomaly determination process in the anomaly detection system according to the embodiment. [Figure 5]Figure 5 shows the pattern for detecting the shielding portion in the embodiment. [Figure 6] Figure 6 shows an image of the occlusion detection pattern captured by the imaging unit in the embodiment. [Figure 7] Figure 7 is a diagram showing the fourth anomaly determination process in the anomaly detection system according to the embodiment. [Figure 8] Figure 8 is a diagram showing the correspondence between the anomaly detection process and the malfunction event in the anomaly detection system according to the embodiment. [Figure 9] Figure 9 is a flowchart illustrating the operation of the crew monitoring device in the embodiment. [Figure 10] Figure 10 is a flowchart illustrating the operation of the anomaly detection system according to the embodiment. [Figure 11] Figure 11 is a flowchart illustrating an example of the first anomaly determination process in the anomaly detection system according to the embodiment. [Figure 12] Figure 12 is a flowchart illustrating an example of the second anomaly determination process in the anomaly detection system according to the embodiment. [Figure 13] Figure 13 shows the pattern for detecting the shielded portion in the second modified example. [Figure 14] Figure 14 shows an image of the occlusion detection pattern captured by the imaging unit in the second modified example. [Figure 15] Figure 15 is a perspective view showing the imaging unit in the third modified example. [Modes for carrying out the invention]

[0009] An anomaly detection system according to an embodiment of the present invention will be described in detail below with reference to the drawings. However, this embodiment does not limit the present invention. Furthermore, the components in the following embodiments include those that are easily conceivable by those skilled in the art or that are substantially identical.

[0010] [Embodiment] An embodiment will be described with reference to Figures 1 to 12. This embodiment relates to an anomaly detection system. Figure 1 is a schematic configuration diagram of the anomaly detection system according to the embodiment, Figure 2 is an exploded perspective view showing the imaging unit in the embodiment, Figure 3 is a perspective view showing the imaging unit in the embodiment, Figure 4 is a schematic diagram showing the first anomaly determination process in the anomaly detection system according to the embodiment, Figure 5 is a diagram showing the pattern for detecting the shielding unit in the embodiment, Figure 6 is a diagram showing the image of the shielding unit detection pattern captured by the imaging unit in the embodiment, Figure 7 is a diagram showing the fourth anomaly determination process in the anomaly detection system according to the embodiment, Figure 8 is a diagram showing the correspondence between the anomaly determination process and malfunction events in the anomaly detection system according to the embodiment, Figure 9 is a flowchart showing the operation of the occupant monitoring device in the embodiment, Figure 10 is a flowchart showing the operation of the anomaly detection system according to the embodiment, Figure 11 is a flowchart showing an example of the first anomaly determination process in the anomaly detection system according to the embodiment, and Figure 12 is a flowchart showing an example of the second anomaly determination process in the anomaly detection system according to the embodiment.

[0011] The abnormality detection system 1 according to the embodiment shown in Figure 1 is included in the occupant monitoring device mounted on the vehicle and is a system for detecting abnormalities that occur in the occupant monitoring device. The occupant monitoring device monitors the monitored person (occupant) DR inside the vehicle's cabin and performs various processes if the state of the monitored person DR is abnormal. The occupant monitoring device is, for example, a Driver Monitoring System (DMS).

[0012] As shown in FIG. 1, the abnormality detection system 1 according to the embodiment includes an imaging unit 10 and a control unit 20. The imaging unit 10 captures a still image or a moving image (hereinafter simply referred to as an "image"). In the embodiment, the imaging unit 10 is mounted on a vehicle and captures an image including a driver DR who is a monitoring target in the vehicle interior. For example, the imaging unit 10 is mounted on the roof portion of the vehicle. Further, the imaging unit 10 may be built in a pillar, an in-vehicle meter, or an in-vehicle display of the vehicle. The imaging unit 10 is activated when the ACC (accessory) power supply or the IG (ignition) power supply of the vehicle is turned on, and captures an image of the vehicle's monitoring target DR until these power supplies are turned off.

[0013] As shown in FIG. 2, the imaging unit 10 includes a camera board 11, a front cover 12a, a back cover 12b, an optical filter 16, and a shielding unit 30.

[0014] The camera board 11 has various electronic components and optical components mounted thereon and constitutes an electronic circuit that electrically connects the electronic components, and is a so-called printed circuit board (Printed Circuit Board). The base material of the camera board 11 is, for example, an insulating layer made of an insulating material such as an epoxy resin, a glass epoxy resin, a paper epoxy resin, or a ceramic, on which a wiring pattern (printed pattern) is formed (printed) by a conductive member such as a copper foil. In the embodiment, the camera board 11 is formed in a rectangular plate shape.

[0015] In the imaging unit 10 of this embodiment, a camera module 11a including optical components and a light source (infrared LED) 11b are mounted on one main surface of the camera substrate 11, and are electrically connected to other electronic components mounted on the camera substrate 11. The camera module 11a is, for example, a near-infrared camera and is mounted approximately in the center of one main surface of the camera substrate 11, and the light source 11b is mounted near the camera module 11a on one main surface of the camera substrate 11. For example, the camera module 11a is connected to the control unit 20 via a connector (not shown) provided on the camera module 11a. The imaging unit 10 receives reflected near-infrared light irradiated from the light source 11b toward the monitored person DR and images the monitored person DR.

[0016] The front cover 12a and the back cover 12b are components that constitute the housing (camera module case) of the imaging unit 10. The front cover 12a covers one main surface of the camera substrate 11 on which the camera module 11a and light source 11b are mounted, and the back cover 12b is positioned to cover the other main surface of the camera substrate 11. The housing of the imaging unit 10 is formed when the front cover 12a and the back cover 12b are assembled together. The front cover 12a has openings formed at positions corresponding to the camera module 11a and light source 11b, exposing the camera module 11a and light source 11b to the outside of the housing.

[0017] The optical filter 16 is an optical component that selectively transmits light of a specific wavelength. The optical filter 16 is provided in front of the camera module 11a and the light source 11b, and selectively transmits light of a specific wavelength incident on the camera module 11a. The optical filter 16 in this embodiment is a visible light cut filter that transmits near-infrared light and absorbs light of wavelengths in the visible light region. The optical filter 16 is a rectangular plate-shaped filter, and the front cover 12a has a recess into which the optical filter 16 can be fitted. In this recess, there is an opening in the front cover 12a that exposes the camera module 11a and the light source 11b to the outside of the housing of the imaging unit 10. The optical filter 16 is attached to the recess in the front cover 12a and fixed to the front cover 12a so as to cover the camera module 11a and the light source 11b.

[0018] The imaging unit 10 has an imaging surface 10a and is mounted on the vehicle so that the imaging surface 10a faces the monitored person (occupant) DR inside the vehicle's cabin, in order to capture images of the monitored person (occupant) DR inside the vehicle's cabin using the camera module 11a. In this embodiment, the imaging surface 10a is the surface of the optical filter 16 (see Figure 3).

[0019] The shielding portion 30 is a plate-shaped member that shields the imaging surface 10a. The shielding portion 30 is located on the front surface of the front cover 12a. As shown in Figure 3, the shielding portion 30 is configured to be displaceable between a shielding position that faces the imaging surface 10a and shields the imaging surface 10a, and an open position that is retracted from the shielding position.

[0020] For example, the shielding section 30 is a sliding shutter and is mounted on the front cover 12a so as to be displaceable between a shielded position and an open position by guide rails 15a and 15b provided on the front cover 12a. Guide rail 15a is a rail-shaped member provided on the upper part of the main surface of the front cover 12a and extending in the width direction of the imaging section 10. Guide rail 15b is a rail-shaped member provided on the lower part of the main surface of the front cover 12a and extending in the width direction of the imaging section 10. The shielding section 30 is mounted between guide rails 15a and 15b.

[0021] The shielding section 30 is provided to protect the privacy of the monitored person (DR) when the imaging unit 10 is in operation. The monitored person (DR) can move the position of the shielding section 30 between the open position and the shielded position at their own discretion. In addition, the shielding section 30 can protect the imaging surface 10a by being positioned in the shielded position when the imaging unit 10 is not in use.

[0022] As shown in Figure 3, in this embodiment, the shielding portion 30 in the open position is positioned at one end in the width direction of the imaging portion 10 so as not to overlap with the imaging surface 10a.

[0023] In this embodiment, the movement of the shielding portion 30 from the open position to the shielded position is performed manually by the vehicle's monitored person DR. The shielding portion 30 is provided with a pinch portion 30a so that the driver DR can manually move the shielding portion 30. The pinch portion 30a is formed at the upper end of the shielding portion 30 by causing a part of the shielding portion 30 to protrude upward. The driver DR can move the shielding portion 30 from the open position to the shielded position by pinching the pinch portion 30a with their fingertips and pulling the shielding portion 30 in the direction of arrow Y1 shown in Figure 3. When the shielding portion 30 reaches the shielded position, the entire imaging surface 10a is covered by the shielding portion 30. The monitored person DR can also move the shielding portion 30 from the imaging surface 10a by moving the shielding portion 30 from the shielded position to the open position in the opposite direction to arrow Y1 shown in Figure 3. With this configuration, the shielding portion 30 in the embodiment is displaceable between a shielded position and an open position. In this embodiment, the movement of the shielding portion 30 from the open position to the shielded position may be performed automatically by a switching unit 14, which will be described later.

[0024] The control unit 20 is a control board equipped with a program for controlling the occupant monitoring device. The control unit 20 may also be integrated into the vehicle's onboard meter or onboard display.

[0025] The control unit 20 includes an obstruction detection unit 21 and an optical component abnormality detection unit 22. The obstruction detection unit 21 is configured to perform a first abnormality detection process to determine whether or not an object SH exists between the imaging surface 10a and the monitored person DR. The optical component abnormality detection unit 22 is configured to perform a second abnormality detection process after the first abnormality detection process if it is determined in the first abnormality detection process that an object SH exists between the imaging surface 10a and the monitored person DR, to determine whether or not there is an abnormality in the optical components included in the imaging unit 10.

[0026] In this embodiment, the first anomaly determination process is performed when the control unit 20 fails to detect the face of the monitored person DR from the image captured by the imaging unit 10. In the first anomaly determination process, for example, the obstruction determination unit 21 detects edges contained in the image from the brightness gradient value of the image captured by the imaging unit 10, and determines that there is an object SH between the imaging surface 10a and the monitored person DR when the ratio of edges in the image is smaller than a predetermined first threshold.

[0027] On the other hand, if the ratio of edges in the image is greater than a predetermined first threshold, the occlusion determination unit 21 further compares the first image P1, which was taken when the face of the monitored person DR could be detected from the image of the monitored person DR taken by the imaging unit 10, with the second image P2, which was taken when the face of the monitored person DR could not be detected from the image of the monitored person DR taken by the imaging unit 10, to determine whether or not there is an object SH between the imaging surface 10a and the monitored person DR.

[0028] Specifically, in the first anomaly detection process, the occlusion detection unit 21 divides the first image P1 into multiple regions RA1. For example, as shown in Figure 4, the occlusion detection unit 21 divides the first image P1 into multiple regions RA1 by matrixing. Then, similar to the first image P1, the occlusion detection unit 21 divides the second image P2 into multiple regions RA2 by matrixing. The occlusion detection unit 21 detects edges contained in each region RA1 and RA2 from the brightness gradient values ​​within each region RA1 and RA2 and obtains the respective edge distribution (a value indicating how many edges are present in the region). The occlusion detection unit 21 calculates the ratio of the difference in edge distributions of the corresponding regions RA1 and RA2, and determines that there is an object SH between the imaging surface 10a and the monitored person DR if the ratio of the difference in edge distributions of the corresponding regions RA1 and RA2 is greater than a predetermined second threshold. Furthermore, if the ratio of the difference in edge distribution between the corresponding regions RA1 and RA2 is less than or equal to a predetermined second threshold, it is determined that there is no object SH between the imaging surface 10a and the monitored person DR.

[0029] The method for the first anomaly detection process described above is merely an example and is not limited thereto. For example, the obstruction detection unit 21 may perform the process of detecting the face of the monitored person DR from the image captured by the imaging unit 10 as the first anomaly detection process. In this case, the obstruction detection unit 21 determines that there is no object SH between the imaging surface 10a and the monitored person DR if the face of the monitored person DR is detected from the image captured by the imaging unit 10. On the other hand, the obstruction detection unit 21 determines that there is an object SH between the imaging surface 10a and the monitored person DR if the face of the monitored person DR cannot be detected from the image captured by the imaging unit 10. Furthermore, in the anomaly detection system 1, a known method different from the method for the first anomaly detection process described above may be adopted as the first anomaly detection process to determine the presence or absence of an object SH between the imaging surface 10a and the monitored person DR.

[0030] The second abnormality determination process in the embodiment is a process that determines whether or not there is an abnormality in the optical components included in the imaging unit 10 based on whether or not the shielding portion 30 located in the shielding position can be detected. In the embodiment, the shielding portion 30 has a shielding portion detection pattern PA1 on the inner surface facing the imaging surface 10a when the shielding portion 30 is positioned in the shielding position. As shown in Figure 5, the shielding portion detection pattern PA1 in the embodiment is a pattern (black and white checkerboard pattern) in which two white rectangular patterns and two black rectangular patterns are arranged alternately in a 2x2 matrix. In the shielding portion detection pattern PA1 shown in Figure 5, black rectangular patterns are placed in the upper left and lower right of the four regions divided in a 2x2 matrix, and black rectangular patterns are placed in the upper right and lower left.

[0031] In the second abnormality determination process, the optical component abnormality determination unit 22 performs imaging with the imaging unit 10 while the shielding unit 30 is positioned in the shielding position. Even if the imaging unit 10 used in the crew monitoring device does not have the function to focus on the shielding detection pattern PA1 located close to the imaging surface 10a, when the imaging unit 10 images the shielding detection pattern PA1, a blurred image P3 is captured, as shown in Figure 6. Here, in the captured image, a difference occurs between the brightness of the parts where the black rectangular pattern is captured and the brightness of the parts where the white rectangular pattern is captured. For example, as shown in Figure 6, the parts corresponding to the black rectangular pattern are captured with relatively low brightness, and the parts where the white rectangular pattern is captured are captured with relatively high brightness.

[0032] The optical component abnormality determination unit 22 compares, for example, an image corresponding to the occlusion detection pattern PA1 pre-stored in the control unit 20 with an image P3 of the occlusion detection pattern PA1 actually captured by the imaging unit 10, and calculates the similarity between the image corresponding to the occlusion detection pattern PA1 pre-stored in the control unit 20 and the image P3 of the occlusion detection pattern PA1 actually captured by the imaging unit 10. Here, similarity is a numerical representation of how similar the two images are. For example, the optical component abnormality determination unit 22 calculates similarity by comparing the brightness distribution between the images. For example, the optical component abnormality determination unit 22 may calculate similarity as the sum of the differences in brightness values ​​for each corresponding pixel between the images.

[0033] The optical component abnormality determination unit 22 determines that there is no abnormality in the optical component if the calculated similarity is equal to or greater than a predetermined threshold. On the other hand, the optical component abnormality determination unit 22 determines that there is an abnormality in the optical component if the calculated similarity is less than a predetermined threshold. Here, the predetermined threshold for similarity is set to a value that can sufficiently distinguish between an image P3 in which the occlusion detection pattern PA1 is captured and an image in which something other than the occlusion detection pattern PA1 is captured, such as a monitored person DR inside the vehicle. In other words, the predetermined threshold for similarity is set so that when the comparison target with the image corresponding to the occlusion detection pattern PA1 stored in the control unit 20 is an image in which the occlusion detection pattern PA1 is captured, the similarity is equal to or greater than the threshold, and when the comparison target with the image corresponding to the occlusion detection pattern PA1 stored in the control unit 20 is an image in which the occlusion detection pattern PA1 is captured, the similarity is less than the threshold.

[0034] In the second abnormality determination process, for the optical component abnormality determination unit 22 to image the shielding detection pattern PA1, the shielding part 30 must be positioned in the shielding position. In this embodiment, the movement of the shielding part 30 is performed manually by the monitored person DR. Therefore, the control unit 20 in this embodiment includes a shielding part movement notification unit 23. In the second abnormality determination process, the shielding part movement notification unit 23 is configured to notify the monitored person DR of its intention to move the shielding part 30 to the shielding position before imaging the shielding detection pattern PA1. After the notification from the shielding part movement notification unit 23, the control unit 20 detects that the monitored person DR has moved the shielding part 30 to the shielding position, and then images the shielding detection pattern PA1.

[0035] For example, the imaging unit 10 has a sensor that detects whether or not the shielding unit 30 is in the shielding position, and this sensor detects when the shielding unit 30 has been moved to the shielding position. However, if the sensor has already detected that the shielding unit 30 is in the shielding position, no notification is sent to the monitored person DR with the intention of prompting them to move the shielding unit 30 to the shielding position.

[0036] In this embodiment, the control unit 20 is configured to perform a third abnormality determination process and a fourth abnormality determination process, in addition to the first and second abnormality determination processes. The third abnormality determination process determines whether or not data transmission and reception are occurring normally between the control unit 20 and the imaging unit 10. The fourth abnormality determination process determines whether or not there is an abnormality in the light source 11b built into the imaging unit 10. In this case, the control unit 20 further includes a data transmission / reception determination unit 24 and a light source abnormality determination unit 25.

[0037] The data transmission / reception determination unit 24 is configured to perform a third anomaly determination process. The light source anomaly determination unit 25 is configured to perform a fourth anomaly determination process. In this embodiment, the third and fourth anomaly determination processes are performed after the first anomaly determination process and before the second anomaly determination process, if the first anomaly determination process determines that there is an object SH between the imaging surface 10a and the monitored person DR. In this embodiment, the fourth anomaly determination process is performed after the third anomaly determination process. However, the fourth anomaly determination process may be performed before the third anomaly determination process.

[0038] In the third abnormality determination process, the data transmission / reception determination unit 24 transmits a signal from the control unit 20 to the imaging unit 10 requesting a response. The imaging unit 10 transmits a pre-set response signal to the control unit 20 in response to this request for a response. If the control unit 20 receives a response signal from the imaging unit 10 within a predetermined period (for example, a period of about 3 seconds) after the control unit 20 has transmitted a signal requesting a response to the imaging unit 10, the data transmission / reception determination unit 24 determines that data transmission and reception between the control unit 20 and the imaging unit 10 are functioning normally. On the other hand, if the control unit 20 does not receive a response signal from the imaging unit 10 within a predetermined period (for example, a period of about 3 seconds) after the control unit 20 has transmitted a signal requesting a response to the imaging unit 10, the data transmission / reception determination unit 24 determines that there is an abnormality in data transmission and reception between the control unit 20 and the imaging unit 10.

[0039] In the fourth abnormality detection process, the light source abnormality detection unit 25 flashes the light source 11b multiple times at regular intervals. The light source abnormality detection unit 25 captures an image using the imaging unit 10 and detects whether the brightness in the image (for example, the average value of the brightness in the image) has changed in accordance with the flashing of the light source 11b. At this time, the imaging unit 10 captures the image with image correction functions such as auto gain and white balance adjustment turned OFF. Note that changes in brightness in the image may also be detected, for example, by comparing the distribution trend of brightness histograms between images.

[0040] As shown in the diagram in Figure 7, when the actual state of the light source (LED) 11b is "on," the detection result of the light source abnormality determination unit 25 is determined to be "no abnormality" if it is "on," and determined to be "off" if it is "an abnormality" in the light source 11b system. Also, when the actual state of the light source (LED) 11b is "off," the detection result of the light source abnormality determination unit 25 is determined to be "another infrared light source" if it is "on," and determined to be "no abnormality" if it is "off." Here, "another infrared light source" refers to, for example, an infrared light source provided in a device other than the crew monitoring device, including the abnormality detection system 1 according to the embodiment.

[0041] In the anomaly detection system 1 of this embodiment, the cause of the malfunction of the crew monitoring device is identified by the first to fourth anomaly determination processes. Figure 8 shows the correspondence between the anomaly determination processes in the anomaly detection system 1 and examples of malfunction events in the crew monitoring device. As shown in Figure 8, the "major categories" of anomalies in the crew monitoring device that the anomaly detection system 1 can detect include "obstructions," "optical filter malfunctions," "camera module malfunctions," "imaging unit failures," and "light source malfunctions."

[0042] "Malfunctions" related to "obstructions" include the imaging surface 10a being "hidden by a hand", the imaging surface 10a being "hidden by an object", and the imaging surface 10a being "covered with a cloth". In addition, "Malfunctions" related to "optical filter malfunctions" include "water droplets on the optical filter 16", "graffiti on the optical filter 16", "cracks on the surface of the optical filter 16", "dirt on the optical filter 16", "cloudy on the optical filter 16", and "a sticker attached to the optical filter 16".

[0043] Regarding "camera module malfunctions," the "malfunctions" include the "lens being detached" and the "lens being loose" of camera module 11a. Furthermore, regarding "imaging unit malfunctions," the "malfunctions" include the "image sensor malfunction" of imaging unit 10, "disconnection" of the circuit wiring of imaging unit 10, and the "failure to power on" of imaging unit 10. Regarding "light source malfunctions," the "malfunctions" include the "bulb burnout or other malfunctions" of light source 11b, and "interference with other light sources."

[0044] The first anomaly detection process can detect malfunctions related to "obstructions," "optical filter malfunctions," "camera module malfunctions," and "imaging unit failures," as well as "light source malfunctions," specifically "bulb failures." However, if these anomalies are detected in the first anomaly detection process, it is determined that there is an obstruction (an object SH existing between the imaging surface 10a and the monitored person DR), and it is not possible to identify which specific anomaly the detected anomaly corresponds to.

[0045] The third anomaly detection process can detect malfunction events related to "imaging unit failure." The fourth anomaly detection process can detect "malfunction events" related to "imaging unit failure" and "light source failure." However, if these anomalies are detected in the fourth anomaly detection process, it will be determined that there is an anomaly in the light source, but it will not be possible to identify which specific anomaly the detected anomaly corresponds to.

[0046] The second anomaly detection process can detect malfunctions related to "optical filter malfunctions," "camera module malfunctions," and "imaging unit failures," as well as "light source malfunctions," specifically "bulb failures." However, if these anomalies are detected in the second anomaly detection process, it determines that there is an anomaly in the optical component, but it cannot identify which specific anomaly the detected anomaly corresponds to.

[0047] The anomaly detection system 1 according to this embodiment can detect anomalies in the crew monitoring device by combining the first to fourth anomaly determination processes, thereby specifically identifying malfunction events in the crew monitoring device.

[0048] An example of the operation of the anomaly detection system 1 will be explained using Figures 9 to 12. As shown in Figure 9, when the power of the occupant monitoring device is turned ON, the control unit 20 performs initial settings such as reading data necessary for controlling the imaging unit 10 (step S1). After that, the anomaly detection system 1 executes an anomaly detection mode (step S2) in which it performs the first to fourth anomaly determination processing. If any of the first to fourth anomaly determination processes in the anomaly detection mode is determined to be "no anomaly", the occupant monitoring device executes a normal mode in which the imaging unit 10 monitors the occupants (step S3). Also, in the anomaly detection mode of step S2, if it is determined that there is an anomaly that makes it impossible to monitor the monitored person DR, for example, the control unit 20 requests a power OFF and turns off the power of the occupant monitoring device.

[0049] Furthermore, if the monitored person DR cannot be detected during the execution of the normal mode (step S3), if the control unit 20 receives a signal from the in-vehicle equipment such as the imaging unit 10 requesting a transition to the abnormal detection mode, or if the vehicle operation ends, the control unit 20 returns from the normal mode (step S3) to the abnormal detection mode (step S2). Also, if the vehicle operation ends, the control unit 20 detects an abnormality in the occupant monitoring device in the abnormal detection mode (step S2), requests a power OFF, and turns off the power to the occupant monitoring device.

[0050] Figure 10 shows an example of the operation of the anomaly detection system 1 according to the embodiment. The operation of the anomaly detection system 1 shown in Figure 10 corresponds to the operation performed in the anomaly detection mode of Figure 9. As shown in Figure 10, the anomaly detection system 1 first performs a first anomaly determination process (step S11). If the first anomaly determination process determines that there is no obstruction (object SH existing between the imaging surface 10a and the monitored person DR), the system determines that the crew monitoring device is functioning normally and continues monitoring the monitored person (crew) DR (step S12). On the other hand, if the first anomaly determination process determines that there is an obstruction (object SH existing between the imaging surface 10a and the monitored person DR) (i.e., if an obstruction is detected), the anomaly detection system 1 performs a third anomaly determination process (step S13). In the third anomaly determination process, if the control unit 20 is unable to properly receive a response signal from the imaging unit 10 (i.e., there is no response), the anomaly detection system 1 determines that the imaging unit 10 is malfunctioning (step S14). On the other hand, in the third anomaly determination process, if the control unit 20 is able to properly receive a response signal from the imaging unit 10 (i.e., a response is received), the anomaly detection system 1 performs the fourth anomaly determination process (step S15).

[0051] In the fourth abnormality determination process, if an abnormality is determined to exist, the abnormality detection system 1 determines that there is an abnormality in the light source 11b (step S16). On the other hand, if the fourth abnormality determination process determines that there is no abnormality, the abnormality detection system 1 performs the second abnormality determination process (step S17). In the second abnormality determination process, if an abnormality is determined to exist (the similarity between images is below a predetermined threshold and matching is NG), the abnormality detection system 1 determines that there is a defect in the optical component of the imaging unit 10 (step S18). On the other hand, if the second abnormality determination process determines that there is no abnormality (the similarity between images is above a predetermined threshold and matching is OK), the system determines that there is an obstruction (object SH existing between the imaging surface 10a and the monitored person DR).

[0052] As described above, the anomaly detection system 1 can more specifically identify the cause of the malfunction of the crew monitoring device by detecting an anomaly in the crew monitoring device in the order of the first anomaly determination procedure, the third anomaly determination procedure, the fourth anomaly determination procedure, and the second anomaly determination procedure.

[0053] Figure 11 shows the first abnormality determination process in the abnormality detection system 1 according to the embodiment. As shown in Figure 11, when the occupant monitoring device (DMS) is activated, the occupant monitoring device uses the imaging unit 10 to capture an image of the person to be monitored (for example, the driver of the vehicle) DR and detects the face of the person to be monitored DR from the captured image (step S21). If a face is detected in step S21, the occupant monitoring device saves information regarding the edge distribution of the captured image in a storage unit provided in the control unit 20, for example. After that, the occupant monitoring device determines the state of the person to be monitored DR (step S23) and takes appropriate action according to the result (step 24). For example, if in step S23 it is determined that the driver, who is the person to be monitored DR, is in a state where driving is difficult due to illness or other reasons, in step S24 it takes action such as notifying the person to be monitored DR to stop driving the vehicle.

[0054] In step S21, if the face of the monitored person DR cannot be detected, the occupant monitoring device issues an alarm intended to prompt the monitored person DR to face forward (step S25). Subsequently, the occupant monitoring device uses the imaging unit 10 to image the monitored person DR again and detects the face of the monitored person DR from the captured image (step S26). If the face of the monitored person DR can be detected in step S26, the process proceeds to step S22. On the other hand, if the face of the monitored person DR cannot be detected in step S26, the device detects whether the monitored person DR has manually dismissed the alarm intended to prompt them to face forward (step S27). If step S27 detects that the monitored person DR has not manually dismissed the alarm intended to prompt them to face forward, the occupant monitoring device determines that the monitored person DR, who is the driver, is in a state where driving is difficult due to illness or other reasons (step S28), and takes action such as notifying the monitored person DR to stop driving the vehicle (step S29).

[0055] In step S27, if the alarm intended to prompt the monitored person DR to manually face forward is dismissed, the obstruction detection unit 21 performs edge detection on the image captured in step S26 (step S30) and calculates the ratio of edges in the image (step S31). In step S31, if the ratio of edges is below a predetermined threshold, the obstruction detection unit 21 determines that the imaging surface 10a is obstructed (step S32), and the control unit 20 causes the occupant monitoring device to issue an alarm or warning indicating that the imaging surface 10a is obstructed (step S33). In step S33, the control unit 20 may also cancel the operation of functions that require output information from the occupant monitoring device (for example, functions equivalent to automatic driving level 2 that require monitoring ahead). Here, the predetermined threshold for the ratio of edges in step S31 is set to a value that allows for distinction between an image of the monitored person DR and an image captured when the imaging surface 10a is obstructed by the obstruction part 30, etc. The edge ratio of an image captured while the imaging surface 10a is obscured for some reason is likely to be lower than the edge ratio of an image captured of the monitored person DR. Therefore, the predetermined threshold for the edge ratio in step S31 is set to a value lower than the edge ratio of an image captured of the monitored person DR, and higher than the edge ratio of an image captured while the imaging surface 10a is obscured by the occlusion part 30 or the like.

[0056] In step 31, if the ratio of edges is higher than a predetermined threshold, the obstruction detection unit 21 compares the edge distribution saved in step S22 with the edge distribution of the image from which edge detection was performed in step S30 (step S34), calculates the value of the difference ratio of the edge distributions, and determines whether the calculated value of the difference ratio of the edge distributions is higher than a predetermined threshold. Here, the comparison of the edge distributions and the calculation of the difference ratio of the edge distributions in steps S34 and S34 are performed using the same method as described above with reference to Figure 4. In step S35, if the value of the difference ratio of the edge distributions is higher than a predetermined threshold, the obstruction detection unit 21 determines that the imaging surface 10a is obstructed (step S36), and causes the occupant monitoring device to issue a warning or alarm to that effect (step S37). In step S37, the control unit 20 may also cancel the operation of functions that require output information from the occupant monitoring device (for example, functions equivalent to automatic driving level 2 that must monitor the area ahead). On the other hand, in step S35, if the ratio of the difference in edge distribution is less than or equal to a predetermined threshold, the obstruction detection unit 21 determines that there is some kind of malfunction in the occupant monitoring device (step S38) and takes action according to the determination result (step S39). For example, depending on the determination result, the obstruction detection unit 21 causes a warning or alarm to be issued indicating that a malfunction has occurred in the occupant monitoring device. Also, in step S39, the control unit 20 may cancel the operation of functions that require output information from the occupant monitoring device (for example, functions equivalent to autonomous driving level 2 that must monitor the area ahead).

[0057] In the anomaly detection system 1 of this embodiment, at least a part of the process shown in Figure 11 described above can be adopted as the first anomaly detection process.

[0058] Figure 12 shows an example of the second abnormality determination process in the abnormality detection system 1 according to the embodiment. As shown in Figure 12, in the second abnormality determination process, the optical component abnormality determination unit 22 detects whether or not the shielding part 30 is in the shielding position (step S41). If it is determined that the shielding part 30 is in the shielding position, the control unit 20 reads an image (matching image) corresponding to the shielding part detection pattern PA1 (step S42), and reads an image of the monitored person DR captured by the imaging unit 10 (step S43). Subsequently, the optical component abnormality determination unit 22 calculates the similarity value between the matching image and the image of the monitored person DR captured by the imaging unit 10, and if the similarity value is above a predetermined threshold, it determines that there is no abnormality in the optical component (matching OK) (step S47). On the other hand, if the optical component abnormality determination unit 22 determines that there is an abnormality in the optical component (matching NG) if the similarity value is below a predetermined threshold (step S48). Furthermore, if it is determined in step S41 that the shielding portion 30 is not in the shielding position, the result of the previous check to determine whether or not there is an abnormality in the optical component is used. For example, as the initial check result, a second abnormality determination process may be performed in advance when the occupant monitoring device is installed in the vehicle, and the check result of "check OK" may be saved. Also, if the previous check result is not saved, the second abnormality determination process may be configured to notify the monitored person DR that an error has occurred.

[0059] In the anomaly detection system according to the embodiment, the configuration shown in Figure 12, as described above, can also be adopted as the second anomaly determination process.

[0060] In the above embodiment, a configuration was described that performs all of the first to fourth anomaly detection processes, but the system is not limited to this configuration. For example, the anomaly detection system 1 according to the embodiment may not perform at least one of the processes from the third anomaly detection process and the fourth anomaly detection process among the first anomaly detection process, the third anomaly detection process, the fourth anomaly detection process, and the second anomaly detection process.

[0061] Furthermore, the shielding portion 30 may be configured to be automatically displaceable between a shielded position and an open position. In this case, as shown in Figures 2 and 3, the imaging unit 10 includes a switching unit 14 as a shielding portion movement mechanism that displaces the position of the shielding portion 30 between a shielded position and an open position. In this case, the control unit 20 may have a shielding portion movement unit instead of a shielding portion movement notification unit 23. If a shielding portion movement unit is provided, in the second abnormality determination process, the switching unit 14 may be configured to perform a shielding portion movement process to place the shielding portion 30 in the shielded position before imaging is performed by the imaging unit 10 with the shielding portion 30 in the shielded position. For example, the system may be configured to automatically move the shielding portion 30 to the open position when the power of the crew monitoring device is turned ON, and to automatically move the shielding portion 30 to the shielded position when the power of the crew monitoring device is turned OFF. Furthermore, in this embodiment, a switch or the like may be provided by the switching unit 14 that allows the position of the shielding unit 30 to be displaced between the shielding position and the open position, so that the monitored person DR can displace the position of the shielding unit 30 between the shielding position and the open position at their own discretion.

[0062] As described above, the anomaly detection system 1 according to the embodiment comprises an imaging unit 10 having an imaging surface 10a mounted on a vehicle and positioned facing the vehicle occupant (surveillance target DR), and a shielding part 30 capable of shielding the imaging surface 10a, and a control unit 20 that controls the imaging unit 10, wherein the shielding part 30 is configured to be displaceable between a shielding position that shields the imaging surface 10a facing it and an open position that is retracted from the shielding position, and has shielding part detection patterns PA1 and PA2 on the inner surface facing the imaging surface 10a when positioned in the shielding position, and the control unit 20 is configured to perform a first anomaly determination process to determine the presence or absence of an object between the imaging surface 10a and the occupant. The system includes an object detection unit 21 and an optical component abnormality determination unit 22 configured to perform a second abnormality determination process to determine whether or not there is an abnormality in the optical components included in the imaging unit 10 after the first abnormality determination process, if the first abnormality determination process determines that there is an object between the imaging surface 10a and the occupant. The optical component abnormality determination unit 22 is characterized in that, in the second abnormality determination process, the imaging unit 10 takes an image with the occluding unit 30 positioned in the occluding position, and determines that there is no abnormality in the optical components if the similarity between the captured image and the images corresponding to the occluding unit detection patterns PA1 and PA2 is above a predetermined threshold, and determines that there is an abnormality in the optical components if the similarity is below the predetermined threshold.

[0063] If the first anomaly detection process determines that an object SH is present between the imaging surface 10a and the monitored person (crew member) DR, a second anomaly detection process is performed after the first anomaly detection process to confirm whether the determination in the first anomaly detection process that an object SH is present between the imaging surface 10a and the monitored person DR is due to an abnormality in the optical components of the imaging unit 10. This configuration makes it possible to more specifically identify the cause of the crew monitoring device not functioning correctly.

[0064] Furthermore, in the abnormality detection system 1 according to the embodiment, in the second abnormality determination process, the control unit 20 includes a shielding unit movement notification unit 23 configured to perform a shielding unit movement notification process to notify the occupant of the intention to move the shielding unit 30 to the shielding position before the imaging unit 10 performs imaging while the shielding unit 30 is positioned in the shielding position.

[0065] In the embodiment, the abnormality detection system 1, in the second abnormality determination process, notifies the monitored person DR of the intention to prompt them to move the shielding unit 30 to the shielding position. This allows the system to determine whether there is an abnormality in the optical components of the imaging unit 10, even in configurations where it is necessary to manually position the shielding unit 30 in the shielding position, and to more specifically identify the cause of the occupant monitoring device not functioning properly.

[0066] Furthermore, in the abnormality detection system 1 according to the embodiment, the control unit 20 includes a data transmission / reception determination unit 24 configured to perform a third abnormality determination process to determine whether or not data transmission and reception are being performed normally between the control unit 20 and the imaging unit 10, and a light source abnormality determination unit 25 configured to perform a fourth abnormality determination process to determine whether or not there is an abnormality in the light source built into the imaging unit 10. The third and fourth abnormality determination processes are performed after the first abnormality determination process and before the second abnormality determination process, if the first abnormality determination process determines that there is an object between the imaging surface 10a and the occupant.

[0067] The anomaly detection system 1 according to this embodiment can more specifically identify the cause of the anomaly in the imaging unit 10 by performing a third anomaly determination process and a fourth anomaly determination process in addition to the second anomaly determination process.

[0068] Furthermore, in the anomaly detection system 1 according to the embodiment, the imaging unit 10 includes a shielding unit movement mechanism that displaces the position of the shielding unit 30 between a shielded position and an open position, and in the second anomaly determination process, the control unit 20 may include a shielding unit movement unit configured to perform a shielding unit movement process to position the shielding unit 30 in the shielded position using the shielding unit movement mechanism before imaging is performed by the imaging unit 10 with the shielding unit 30 positioned in the shielded position.

[0069] In this case, the anomaly detection system 1 can automatically position the shielding part in the shielding position using the shielding part movement mechanism, and the second anomaly determination process can be performed smoothly.

[0070] [Modified Example of Embodiment 1] A first modified example of the embodiment will now be described. Figure 13 shows the shielding detection pattern of the shielding portion in the first modified example, and Figure 14 shows an image of the shielding detection pattern captured by the imaging unit in the first modified example. As shown in Figure 13, in the anomaly detection system 1 of the first modified example, the shielding detection pattern PA2 of the shielding portion 30 is different from the shielding detection pattern PA1 of the above embodiment.

[0071] Specifically, the occlusion detection pattern PA2 is a rectangular pattern with the left half being black and the right half being white. As shown in Figure 14, when the imaging unit 10 images the occlusion detection pattern PA2, an image P4 is captured in which the brightness of the right half is relatively higher than the brightness of the left half. The second anomaly detection process can be performed even when using such an occlusion detection pattern PA2. The other configurations in the first modified example are the same as in the embodiment described above.

[0072] [Second modified example of the embodiment] A second modified example of the embodiment will now be described. Figure 15 is a perspective view showing the imaging unit in the second modified example. As shown in Figure 15, the anomaly detection system 1 of the second modified example differs from the configuration of the above embodiment in that the shielding unit 30 is mounted so as to be openable and closable in the vertical direction indicated by arrow Y2. In addition, in the anomaly detection system 1 of the second modified example, a connector 11c is provided on the side of the housing of the imaging unit 10, and the camera module 11a (see Figure 2) is connected to the control unit 20 via the connector 11c.

[0073] In the second modified example, the shielding portion 30 is connected to the imaging portion 10 at its upper end by a connecting member such as a hinge. By lowering the free end of the shielding portion 30 in the open position along arrow Y2, the position of the shielding portion 30 can be displaced to the shielded position. Conversely, by lifting the free end of the shielding portion 30 in the shielded position along the direction opposite to arrow Y2, the position of the shielding portion 30 can be displaced to the open position. The other configurations in the second modified example are the same as in the above embodiment.

[0074] The embodiments and modifications disclosed above can be combined and implemented as appropriate. [Explanation of Symbols]

[0075] 1: Anomaly detection system 10: Imaging unit, 10a: Imaging surface, 20: Control unit 21: Obstruction detection unit 22: Optical component abnormality detection unit 23: Shielding unit movement notification unit 25: Light source abnormality determination unit 30: Shielding section PA1, PA2: Pattern for detecting the shielding section

Claims

1. An imaging unit having an imaging surface mounted on a vehicle and positioned facing the occupants of the vehicle, and a shielding portion capable of shielding the imaging surface, The system comprises a control unit for controlling the imaging unit, The shielding portion is configured to be displaceable between a shielding position that shields the imaging surface in opposition to the imaging surface and an open position that is retracted from the shielding position, and has a pattern for detecting the shielding portion on the inner surface facing the imaging surface when positioned in the shielding position. The control unit includes an obstruction determination unit configured to perform a first abnormality determination process to determine whether or not there is an object between the imaging surface and the occupant, and an optical component abnormality determination unit configured to perform a second abnormality determination process to determine whether or not there is an abnormality in the optical components included in the imaging unit after the first abnormality determination process, if the first abnormality determination process determines that there is an object between the imaging surface and the occupant. An anomaly detection system characterized in that, in the second anomaly determination process, the optical component anomaly determination unit performs imaging with the imaging unit while the shielding unit is positioned at the shielding position, determines that there is no anomaly in the optical component if the similarity between the captured image and the image corresponding to the shielding unit detection pattern is greater than or equal to a predetermined threshold, and determines that there is an anomaly in the optical component if the similarity is less than the predetermined threshold.

2. In the second abnormality determination process, the control unit includes a shielding unit movement notification unit configured to perform a shielding unit movement notification process that notifies the occupant of the intention to move the shielding unit to the shielding position before the imaging unit performs imaging while the shielding unit is positioned at the shielding position. The anomaly detection system according to claim 1.

3. The imaging unit includes a shielding unit moving mechanism that displaces the position of the shielding unit between the shielding position and the open position. In the second abnormality determination process, the control unit includes a shielding unit moving unit configured to perform a shielding unit moving process to position the shielding unit at the shielding position using the shielding unit moving mechanism before imaging is performed by the imaging unit while the shielding unit is positioned at the shielding position. The anomaly detection system according to claim 1.

4. The control unit includes a data transmission / reception determination unit configured to perform a third abnormality determination process to determine whether data transmission and reception between the control unit and the imaging unit are functioning correctly, and a light source abnormality determination unit configured to perform a fourth abnormality determination process to determine whether there is an abnormality in the light source built into the imaging unit. The third and fourth anomaly determination processes are performed after the first anomaly determination process and before the second anomaly determination process, if the first anomaly determination process determines that the object is located between the imaging surface and the occupant. An anomaly detection system according to any one of claims 1 to 3.

Images