Notification device and notification system

Abstract

To provide a notification device and a notification system that shorten the processing time for determining the non-temporary malfunction state after the activation of the vehicle ECU.SOLUTION: The notification device includes: an image acquisition part that acquires an image; a steady-state determination part that determines presence or absence of a non-temporary malfunction state occurring in a camera system; a storage processing part that stores malfunction information in a storage area; a startup determination part that determines presence or absence of the non-temporary malfunction state during a period until the steady-state determination part becomes capable of making a determination; and an alarm output processing part that outputs a warning when each determination part determines that the non-temporary malfunction state is present. The startup determination part includes: a previous information acquisition part that retrieves the malfunction information stored in the storage area when a control device of a vehicle transitions from a previous power-on state to a power-off state; and a provisional decision part that determines that the non-temporary malfunction state is present if the malfunction information acquired by the previous information acquisition part indicates presence of the non-temporary malfunction state, and determines that the non-temporary malfunction state is absent if the malfunction information acquired by the previous information acquisition part indicates absence of the non-temporary malfunction state.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present embodiment relates to a notification device and a notification system. [Background technology]

[0002] In recent years, vehicles have been equipped with a function for capturing images of the interior of the vehicle using an on-board camera. Images captured by the on-board camera may be used to estimate whether the driver is looking away or has their eyes closed using facial image recognition technology, and may also be recorded.

[0003] For example, if a foreign object adheres to the camera lens of an in-vehicle camera, the captured image may not be continuously usable. Therefore, a technique for detecting a continuous, i.e., non-temporary malfunction of the in-vehicle camera, such as the adhesion of a foreign object, has been proposed.

[0004] For example, Patent Document 1 discloses an object detection device that includes a division unit that divides an image captured by an imaging device into a plurality of partial regions, a calculation unit that calculates an average value of a first value and a variation in the first value for each partial region based on color information of pixels included in the partial regions divided by the division unit, and a detection unit that detects the partial region in which the average value and the variation calculated by the calculation unit each satisfy a predetermined adhesion condition as an object area. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-106644 Summary of the Invention [Problem to be solved by the invention]

[0006] If a user intends to use video captured by an in-vehicle camera while driving, it is preferable to notify the driver before the start of driving that a non-temporary malfunction has occurred, i.e., that the in-vehicle camera cannot be used. However, determining whether the in-vehicle camera is in a non-temporary malfunction requires observation for approximately 5 to 30 minutes. Therefore, after the vehicle system is started, it is difficult to determine whether the in-vehicle camera is in a non-temporary malfunction state and notify the driver before the vehicle starts driving. For example, there is a risk that the driver will be notified of the non-temporary malfunction state some time after the vehicle starts driving. In this case, it is difficult for the driver to immediately resolve the non-temporary malfunction state.

[0007] Therefore, a notification device and a notification system are provided that reduce the processing time required to determine whether a vehicle is in a non-temporary malfunction state after the vehicle control device is started. [Means for solving the problem]

[0008] The notification device of the present disclosure includes an image acquisition unit that acquires images from a camera that captures images of the interior of the vehicle's cabin; a steady-state determination unit that determines whether or not a non-temporary malfunction state has occurred in a camera system including the camera in a steady state based on the images acquired by the image acquisition unit; a memory processing unit that stores in a memory area malfunction information that is information about the non-temporary malfunction state determined by the steady-state determination unit in response to the vehicle's ignition changing from an on state to an off state; a startup determination unit that determines whether or not a non-temporary malfunction state has occurred in the camera system during the period from when the ignition changes from an off state to an on state until the steady-state determination unit becomes able to make a determination; and an alarm output processing unit that outputs a warning to notify the driver when the steady-state determination unit or the startup determination unit determines that the camera system has a non-temporary malfunction state. The startup determination unit includes a previous information acquisition unit that retrieves the malfunction information stored in the memory area when the vehicle's control device last changed from a power-on state to an off state, and a provisional decision unit that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that no non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that no non-temporary malfunction state exists.

[0009] According to the notification device of the present disclosure, a non-temporary malfunction state is determined not by using an image captured by a camera after the vehicle control device is started, but by using malfunction information stored the previous time the vehicle was dismounted, thereby shortening the processing time required to determine a non-temporary malfunction state when the vehicle is currently in the vehicle. Therefore, even when the steady-state determination unit cannot determine whether a non-temporary malfunction state exists during the period leading up to the start of driving, such as when driving is started within a short period of time after the ECU power is turned on, the driver can be notified of a non-temporary malfunction state that has been provisionally determined with a high probability. This improves driver convenience and vehicle driving safety.

[0010] The notification system of the present disclosure includes an image acquisition unit that acquires images from a camera that captures images of the interior of the vehicle's cabin; a steady-state determination unit that determines whether or not a non-temporary malfunction has occurred in a camera system including the camera in a steady state based on the images acquired by the image acquisition unit; a memory area that stores malfunction information, which is information regarding a non-temporary malfunction that has occurred in the camera system determined by the steady-state determination unit in a manner that will not be erased even if the power to the vehicle's control device is turned off; a memory processing unit that stores the malfunction information in the memory area in response to the ignition of the vehicle changing from an on state to an off state; a startup determination unit that determines whether or not a non-temporary malfunction has occurred in the camera system during the period from when the ignition changes from an off state to an on state until the steady-state determination unit becomes able to make a determination; and a warning output processing unit that notifies the driver when the steady-state determination unit or the startup determination unit determines that a non-temporary malfunction has occurred. The startup determination unit includes a previous information acquisition unit that retrieves the malfunction information stored in the memory area when the control device last changed from a power-on state to an off state, and a provisional decision unit that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that no non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that no non-temporary malfunction state exists.

[0011] According to the notification system of the present disclosure, a non-temporary malfunction state is determined not by using an image captured by a camera after the vehicle's control device is started, but by using malfunction information stored the last time the vehicle was dismounted, thereby shortening the processing time required to determine a non-temporary malfunction state after the control device is started. Therefore, even if the steady-state determination unit cannot determine whether a non-temporary malfunction state exists during the period until the start of driving, such as when driving is started within a short period of time after the ECU power is turned on, the driver can be notified of a non-temporary malfunction state that has been provisionally determined with a high probability. This improves driver convenience and vehicle driving safety. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of a notification system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing a schematic configuration of a camera system including a camera and a projector; [Figure 3] A diagram showing examples of unobstructed and obscured states [Figure 4] Figures showing other examples of occlusion states [Figure 5] 1 is a flowchart illustrating a process in a steady state of the notification device according to the first embodiment; [Figure 6] 1 is a flowchart illustrating a process performed when the notification device according to the first embodiment changes from an IG-ON state to an IG-OFF state; [Figure 7] 1 is a flowchart illustrating a process performed when the ECU power supply of the notification device according to the first embodiment is switched from an OFF state to an ON state; [Figure 8] FIG. 10 is a block diagram illustrating an example of the configuration of a notification device according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a method for calculating similarity / dissimilarity by a similarity determination unit according to a second embodiment; [Figure 10] FIG. 10 is a diagram illustrating another example of a method for calculating similarity / dissimilarity by the similarity determination unit according to the second embodiment; [Figure 11]10 is a flowchart illustrating a process performed when the notification device according to the second embodiment changes from an IG-ON state to an IG-OFF state. [Figure 12] 10 is a flowchart illustrating a process performed when the ECU power supply of the notification device according to the second embodiment is switched from an OFF state to an ON state; [Figure 13] FIG. 10 is a block diagram illustrating an example of the configuration of a notification device according to a third embodiment. [Figure 14] 10A and 10B are diagrams showing examples of image states according to imaging modes of a camera system according to a third embodiment; [Figure 15] 10 is a flowchart illustrating a process performed when the notification device according to the third embodiment changes from an IG-ON state to an IG-OFF state. [Figure 16] 10 is a flowchart illustrating a process performed when the ECU power supply of the notification device according to the third embodiment is switched from an OFF state to an ON state; [Figure 17] 10 is a flowchart illustrating a process performed when the ECU power supply of the notification device according to the fourth embodiment is switched from an OFF state to an ON state; [Figure 18] 10 is a flowchart illustrating a process performed when the ECU power supply of the notification device according to the fourth embodiment is switched from an OFF state to an ON state; DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, notification systems and notification devices according to several embodiments will be described with reference to the drawings. The same reference numerals will be used to designate substantially the same components in several embodiments, and the description thereof will be omitted.

[0014] (First embodiment) A first embodiment will be described with reference to Figures 1 to 7. The notification system 1 includes a notification device 10. The notification device 10 is a device mounted on a moving object such as a vehicle. The notification device 10 is connected to a camera 20, a floodlight 30, a display device group 40, a door opening / closing switch 201, and an ignition (IG) button 202 mounted on the vehicle via an in-vehicle network such as a CAN (Controller Area Network).

[0015] The notification system 1 or notification device 10 has a function of notifying the user, in this case the driver, whether or not images captured by the camera 20 can be used. The camera 20 and the floodlight 30 constitute a camera system that captures images of the interior of a vehicle. In this specification, a non-temporary malfunction state refers to a state in which the camera system is unable to capture images normally for a predetermined period of time, such as when a foreign object adheres to the lens of the camera 20 or when the camera 20 or the floodlight 30 is covered. The notification system 1 or notification device 10 has a function of notifying the user, in this case the driver, via an alarm output means such as the display device group 40 when a non-temporary malfunction state occurs.

[0016] The camera 20 has a function of capturing an image of an object inside the vehicle cabin. The image of the object may be, for example, the driver's face or the upper body of the driver. The floodlight 30 has a function of illuminating the object when the camera 20 captures an image, thereby assisting the camera 20 in capturing a good image. The display device group 40 includes, for example, meters, in-vehicle infotainment (IVI), etc.

[0017] As shown in FIG. 2 , the camera 20 includes a lens 21, a lens barrel 22, and an image sensor 23. The lens 21 is positioned and oriented so that the image capture target (e.g., the driver's face) is included in the capture range. The lens 21 is located at the tip of the lens barrel 22, and the image sensor 23 is located on the opposite side of the lens barrel 22 from the lens 21. The image sensor 23 may be configured, for example, as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The image sensor 23 receives light collected by the lens. The projector 30 includes an infrared light source such as an LED (light-emitting device) and projects light onto the image capture target to obtain a more appropriate image depending on the brightness of the vehicle interior. The projector 30 is configured to project light in synchronization with the shutter opening timing of the camera 20. The camera 20 sends the captured image to the notification device 10.

[0018] (non-temporary malfunction) An example of a non-temporary malfunction state that may occur in a camera system is described below. In this specification, a non-temporary malfunction state refers to a state in which the lens 21 of the camera 20 and / or the projection surface of the projector 30 are continuously obstructed, making it impossible to capture an image of the target. FIG. 3 shows examples of images captured depending on the presence and type of obstruction in the camera 20. In all of the examples shown in FIG. 3, it is assumed that the projector 30 is not obstructed. (a-1) shows an unobstructed state in the camera 20. For example, if there is no foreign matter attached to the lens 21 and no obstruction covering the lens 21, light from the target O, e.g., the driver's face, reaches the image sensor 23 without being obstructed. Therefore, as shown in (a-2), the target O is captured. As shown in (b-1), if the camera 20 is obstructed, for example, if there is an obstruction S covering the entire imaging surface of the lens 21, all light from the target O is obstructed and does not reach the image sensor 23. Therefore, as shown in (b-2), the captured image is entirely blacked out. In this case, the distance between the lens 21 and the obstructing object S is extremely small and essentially zero, a so-called zero distance state. As shown in (c-1), if the camera 20 is in an obstructed state, for example, if a foreign object, i.e., the obstructing object S, is attached to one or more parts of the imaging surface of the lens 21, the light is blocked by the obstructing object S. Therefore, as shown in (c-2), the parts of the captured image that correspond to the parts blocked by the obstructing object S are blacked out.

[0019] 4 shows examples of images captured when other occlusion states occur. In the example shown in (d-1), the occlusion object S is located a little distance away from the lens 21. That is, the lens 21 is covered at a position other than zero distance. In this case, the light from the projector 30 may not reach the occlusion object S, resulting in a black image as in (b-2). However, the light from the projector 30 may also be reflected by the occlusion object S and reach the image sensor 23. In this case, for example, as shown in (d-2), the image may be a washed-out color image or a gray image somewhere between a white image and a black image.

[0020] In the example shown in (e-2), the light-projecting surface of floodlight 30 is covered by a shielding object S. In this case, if sufficient brightness is available from the outside, such as inside a vehicle during the day, it is possible to capture an image as shown in (a-2). However, if sufficient brightness is not available from the outside, such as inside a vehicle at night, the captured image may be a black image as shown in (e-2).

[0021] The notification device 10 includes a vehicle control device (hereinafter, sometimes referred to as an ECU (Electronic Control Unit)) 11 and a non-volatile memory 12. The ECU 11 is responsible for controlling the entire notification device 10 and also has the function of controlling various electronic devices mounted on the vehicle. The ECU 11 is connected to a door opening / closing switch 201, an IG button 202, a camera 20, a floodlight 30, a display device group 40, and the like via an in-vehicle network such as a CAN (Controller Area Network). The non-volatile memory 12, which serves as a storage area, stores information in a manner that prevents the stored information from being erased even when power is not supplied to the notification device 10. The non-volatile memory 12 can be configured, for example, by a ROM (Read Only Memory) or a flash memory.

[0022] The vehicle is equipped with a door open / close switch 201 and an ignition button 202. The door open / close switch 201 detects whether a vehicle door is open or closed. When the ECU power supply unit 111 is in an OFF state and the vehicle door changes from a closed state to an open state, the ECU power supply unit 111 is turned ON. The ignition button 202 accepts an operation from the driver to start or stop the engine. The ECU 11 switches the vehicle ignition (not shown) on and off.

[0023] The ECU 11 includes an ECU power supply unit 111, a CAN signal acquisition unit 112, a camera control unit 113, and a floodlight control unit 114. The ECU power supply unit 111 provides the driving power for the notification device 10. The CAN signal acquisition unit 112 is connected to the door open / close switch 201, the IG button 202, the camera 20, the floodlight 30, the display device group 40, and the like via an in-vehicle network such as a CAN. The camera control unit 113 controls the operation of the camera 20. The floodlight control unit 114 controls the operation of the floodlight 30. The camera control unit 113 and the floodlight control unit 114 may detect the constantly changing luminance distribution of the subject to be captured (in this case, the driver's face) and control the imaging conditions of the camera system so that the face can be captured with a desirable degree of shading. For example, the camera control unit 113 can control the exposure time and gain. The floodlight control unit 114 can control the amount of light.

[0024] In this embodiment, the camera 20 and / or the floodlight 30 are connected to the ECU 11 through an in-vehicle network such as a CAN, but this is not limiting. For example, in other embodiments, the camera 20 and / or the floodlight 30 may be connected to the ECU 11 through wiring dedicated to the camera system.

[0025] The notification device 10 includes an image acquisition unit 13, an image recognition unit 14, a steady-state determination unit 15, a storage processing unit 16, a startup determination unit 17, and a driving suitability determination unit 18. The image acquisition unit 13, the image recognition unit 14, the steady-state determination unit 15, the storage processing unit 16, the startup determination unit 17, and the driving suitability determination unit 18 are realized in software by the ECU executing a program stored in the non-volatile memory 12. Note that any one or all of the image acquisition unit 13, the image recognition unit 14, the steady-state determination unit 15, the storage processing unit 16, the startup determination unit 17, and the driving suitability determination unit 18 may be configured by hardware, or may be realized by a combination of software and hardware.

[0026] The image acquisition unit 13 has a function of acquiring images captured by the camera 20. The image recognition unit 14 acquires images captured by the camera 20 via the image acquisition unit 13 and recognizes the face of the captured subject (in this case, the driver's face) appearing in the image. The steady-state determination unit 15 determines whether a non-temporary malfunction state occurs in the camera system when the notification device 10 is in a steady state, in which the ECU power supply unit 111 is constantly operating the notification device 10. The storage processing unit 16 stores malfunction information, which indicates whether the steady-state determination unit 15 has determined whether the camera system is in a non-temporary malfunction state, in the non-volatile memory 12 under predetermined conditions. The startup determination unit 17 determines whether the camera system is in a non-temporary malfunction state during a predetermined period from when the ECU power supply unit 111 begins supplying drive power, i.e., during the period until the steady-state determination unit 15 becomes capable of making a determination. In other words, the startup determination unit 17 determines whether the camera system is in a non-temporary malfunction state during the period from when the ECU power supply unit 111 begins supplying drive power until the steady state is reached. The driving suitability determination unit 18 has the function of recognizing the facial image of the subject captured by the camera system, in this case the driver, and detecting an unsuitable state for driving, such as the driver looking away or falling asleep while driving, or an inability to drive, such as a coma, and alerting the driver.

[0027] The driving suitability determination unit 18 includes a state estimation unit 181, an alarm determination unit 182, and an alarm output processing unit 183. The state estimation unit 181 calculates the driver's state using the driver's facial physical state calculated by the image recognition unit 14 as input information. In this specification, the facial physical state refers to the physical state of the face, including the position of the face, the direction of the face, the degree of eye opening and closing, the direction of gaze, etc. The driver state includes at least "normal" and "unsuitable." "Normal" refers to a state in which the driver's face is facing forward. "Unsuitable" refers to, for example, a state in which the driver's eyes are closed for more than two seconds, the driver is dozing, or the driver's face is out of the field of view of the camera 20 for more than 10 seconds, i.e., the driver's posture is poor. The alarm determination unit 182 determines whether to output an alarm based on the driver's state calculated by the state estimation unit 181. For example, if the driver's state is "normal," the alarm determination unit 182 determines that the driver is in a state suitable for driving and therefore no alarm is necessary. If the driver's condition is "unsuitable", the warning determination unit 182 determines that the driver is not suitable for driving and therefore a warning is required.

[0028] The alarm output processing unit 183 sends an instruction to output an alarm to the alarm output means based on the determination result of the alarm determination unit 182. The alarm output means can be realized by various means such as optical means, auditory means, tactile means, olfactory means, etc. For example, the alarm output means may be realized by the display device group 40 including the meter and IVI.

[0029] The steady-state determination unit 15 determines whether a non-temporary malfunction state has occurred in the camera system based on multiple images acquired by the image acquisition unit 13 within a time window set to a predetermined period. The predetermined period can be set, for example, within a range of 5 to 30 minutes. In other words, the steady-state determination unit 15 cannot determine whether a non-temporary malfunction state has occurred during at least the predetermined period from when the ECU power supply unit 111 starts supplying drive power and when the camera 20 starts capturing images. Therefore, during that period, the startup determination unit 17 provisionally determines whether a non-temporary malfunction state has occurred.

[0030] The memory processing unit 16 has a function of storing malfunction information, which indicates whether or not the camera system is in a non-temporary malfunction state, determined by the steady-state determination unit 15 immediately before the IG button 202 is operated to change the state from IG-ON (ignition on) to IG-OFF (ignition off), in a storage area, for example, the non-volatile memory 12. In another embodiment, the notification system 1 may store the malfunction information in a server connected to an external network system instead of the non-volatile memory 12. The memory processing unit 16 may store the malfunction state in the storage area at predetermined intervals while the ECU power supply unit 111 is ON. In this case, the memory processing unit 16 may overwrite the latest malfunction state with the previous malfunction state. That is, the non-volatile memory 12 stores at least the most recent malfunction state determined by the steady-state determination unit 15 and stored by the memory processing unit 16.

[0031] The startup determination unit 17 includes a previous information acquisition unit 171 and a provisional determination unit 172. The previous information acquisition unit 171 acquires previous information when the ECU power supply unit 111 is changed from an OFF state to an ON state by the door opening / closing switch 201. The previous information is malfunction information stored in the non-volatile memory 12 at the time when the ECU power supply unit 111 was changed from an ON state to an OFF state the previous time. In other words, the previous information is malfunction information stored in the non-volatile memory 12 at the end of the period from the ON state to the OFF state the previous time. The provisional determination unit 172 provisionally determines whether the current camera system is in a non-temporary malfunction state based on at least the previous information.

[0032] In this case, if the previous information indicates that a non-temporary malfunction state is present, the provisional decision unit 172 provisionally decides that the current non-temporary malfunction state is present. If the previous information indicates that a non-temporary malfunction state is absent, the provisional decision unit 172 provisionally decides that the current non-temporary malfunction state is absent. If the provisional decision unit 172 determines that a non-temporary malfunction state is present, the startup determination unit 17 notifies the driving suitability determination unit 18 that a non-temporary malfunction state is present. The warning output processing unit 183 notifies the driver via the display device group 40 or the like that the image captured by the camera 20 is unavailable. This notifies the driver that the image captured by the camera 20 is unavailable during the period from when the ECU power supply unit 111 is activated until the steady-state determination unit 15 becomes able to make a determination. This allows the driver to recognize the content of the notification before starting driving. In addition, in some cases, the driver can take measures to resolve the non-temporary malfunction state occurring in the camera system before starting driving.

[0033] (Flowchart of steady-state determination section) The processing of the notification device 10 in a steady state will be described with reference to the flowchart shown in Figure 5. At the start of Figure 5, the notification device 10 is supplied with driving power from the ECU power supply unit 111, and a predetermined period of time has passed since the camera 20 started capturing images. In the flowcharts of this specification, processing by each component will be described as processing by the notification device 10.

[0034] In step S11, the notification device 10 acquires an image from the camera 20. The camera 20 captures images multiple times over a predetermined period, for example, approximately 30 times per second. In step S12, the notification device 10 determines whether an obstructed state occurs in the image acquired by the image acquisition unit 13. If it is determined that an obstructed state does not exist (NO in step S12), the notification device 10 returns the process to step S11. If it is determined that an obstructed state exists (YES in step S12), the notification device 10 proceeds to step S13. In step S13, the notification device 10 determines whether the obstructed state has continued for a predetermined period, for example, five minutes or more. In this case, the steady-state determination unit 15 determines whether an obstructed state has occurred in all of the multiple images acquired by the image acquisition unit 13, i.e., all of the images captured during the predetermined period. If the obstructed state has not continued for the predetermined period or more (NO in step S13), the notification device 10 returns the process to step S11. If the blocked state continues for a predetermined period of time or more (YES in step S13), the notification device 10 proceeds to step S14. In step S14, the notification device 10 determines that a non-temporary malfunction state exists and that the camera system is unavailable. In step S15, the notification device 10 sends an instruction to, for example, the display device group 40, to notify the driver that the camera system is unavailable. In this way, the processing of the notification device 10 in the steady state is executed. Note that the notification device 10 may return the processing to step S11 again after outputting the warning.

[0035] (IG-OFF flow chart) The process performed by the notification device 10 when the IG button 202 is operated to change the IG-ON state to the IG-OFF state will be described with reference to the flowchart shown in Fig. 6. At the start of Fig. 6, the notification device 10 is in the IG-ON state.

[0036] In step S21, the notification device 10 determines whether the CAN signal acquisition unit 112 has detected that the IG button 202 has been turned off. That is, the notification device 10 determines whether the IG-ON state has been changed to the IG-OFF state. If the CAN signal acquisition unit 112 has not detected the IG-OFF operation (NO in step S21), the notification device 10 repeats the process of step S21. If the CAN signal acquisition unit 112 has detected the IG-OFF operation (YES in step S21), the notification device 10 proceeds to step S22. In step S22, the notification device 10 records in the nonvolatile memory 12 the malfunction information determined immediately before the change from the IG-ON state to the IG-OFF state. That is, the presence or absence of the blocked state determined in step S12 of FIG. 5 is recorded in the nonvolatile memory 12. As a result, the malfunction state immediately before the change from the IG-ON state to the IG-OFF state is stored in a referable manner in the nonvolatile memory 12 as a storage area.

[0037] In step S23, the notification device 10 determines whether or not an ECU-OFF signal is present. That is, the notification device 10 determines whether or not the CAN signal acquisition unit 112 has detected an ECU power OFF signal. If an ECU power OFF signal has not been detected (NO in step S23), the notification device 10 repeats the processing of step S23. If an ECU power OFF signal has been detected (YES in step S23), the notification device 10 proceeds to processing in step S24. In step S24, the ECU power supply unit 111 stops supplying drive power. In this way, the processing by the notification device 10 is executed when the IG-ON state changes to the IG-OFF state.

[0038] (Flowchart when ECU power is turned on) The processing by the notification device 10 when the ECU power supply state changes from OFF to ON will be described with reference to the flowchart shown in Fig. 7. At the start of Fig. 7, the ECU power supply is ON and power is being supplied from the ECU power supply unit 111, but the IG-OFF state is being maintained.

[0039] In step S31, the notification device 10 acquires the previous information recorded in the non-volatile memory 12 at the time of the previous IG-OFF. In step S32, the notification device 10 determines whether the previous information indicates a malfunction. If the previous information indicates no malfunction (NO in step S32), the notification device 10 proceeds to step S33. In step S33, the notification device 10 provisionally determines that there is no non-temporary malfunction in the current camera system.

[0040] If the previous information indicates a malfunction (YES in step S32), the notification device 10 proceeds to step S34. In step S34, the notification device 10 provisionally determines that the camera system is currently experiencing a non-temporary malfunction. In step S35, the notification device 10 sends a command to, for example, the display device group 40 to issue a warning to the driver that the camera system is unavailable. In this case, the display device group 40 can notify the driver that the camera system is unavailable by, for example, turning on a warning light. In this way, the processing of the notification device 10 when the ECU power is turned ON is executed.

[0041] A non-temporary malfunction state may be intentionally caused by the driver, for example, by hanging a hat or cloth over the camera 20 or the floodlight 30. In this case, if the same driver drives the vehicle continuously, the driver should be aware that the camera 20 or the floodlight 30 is being blocked. On the other hand, if the driver is changed, or if the same driver gets out of the vehicle and drives again after a while, the driver may not be aware that a non-temporary malfunction state has occurred. In preparation for such a case, it is preferable to notify the driver of the occurrence of a non-temporary malfunction state after getting in the vehicle and before the vehicle returns to a normal state. Furthermore, if a non-temporary malfunction state occurred in the camera system before the previous IG-ON state was changed to the IG-OFF state, there is a high probability that a non-temporary malfunction state will also occur in the camera system after the ECU power-off state is changed to the ECU power-on state this time.

[0042] In contrast, the notification device 10 of the present embodiment described above includes an image acquisition unit 13, a steady-state determination unit 15, a storage processing unit 16, a startup determination unit 17, and an alarm output processing unit 183. The image acquisition unit 13 acquires images from a camera 20 that captures images of the interior of the vehicle. The steady-state determination unit 15 determines whether or not a non-temporary malfunction has occurred in the camera system including the camera 20 and / or the floodlight 30 in a steady state based on the images acquired by the image acquisition unit 13. The storage processing unit 16 stores malfunction information, which is information regarding the non-temporary malfunction determined by the steady-state determination unit 15 in response to the vehicle ignition being switched from an on state to an off state, in the non-volatile memory 12 as a storage area. The startup determination unit 17 determines whether or not a non-temporary malfunction has occurred in the camera system during the period from when the ignition is switched from an off state to an on state until the steady-state determination unit 15 becomes able to make a determination. The warning output processing unit 183 outputs a warning to notify the driver when the steady-state determination unit 15 or the startup determination unit 17 determines that a non-temporary malfunction state exists. The startup determination unit 17 has a previous information acquisition unit 171 that retrieves malfunction information stored in the non-volatile memory 12 when the ECU 11, which is a vehicle control device, last changed from a power-on state to an off state, and a provisional decision unit 172 that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit 171 indicates the presence of a non-temporary malfunction state, and determines that a non-temporary malfunction state does not exist if the malfunction information acquired by the previous information acquisition unit 171 indicates the absence of a non-temporary malfunction state.

[0043] The notification system 1 of this embodiment includes an image acquisition unit 13, a steady-state determination unit 15, a non-volatile memory 12 serving as a storage area, a storage processing unit 16, a startup determination unit 17, and an alarm output processing unit 183. The image acquisition unit 13 acquires images from a camera 20 capturing images of the interior of the vehicle. The steady-state determination unit 15 determines whether a non-temporary malfunction occurs in the camera system including the camera 20 in a steady state based on the images acquired by the image acquisition unit 13. The non-volatile memory 12 stores malfunction information, which is information regarding a non-temporary malfunction occurring in the camera system determined by the steady-state determination unit 15, in a manner that is not erased even when the vehicle control device is powered off. The storage processing unit 16 stores the malfunction information in the non-volatile memory 12 when the vehicle ignition is switched from an on state to an off state. The startup determination unit 17 determines whether a non-temporary malfunction occurs in the camera system during the period from when the ignition is switched from an off state to an on state until the steady-state determination unit 15 becomes able to make a determination. The notification unit outputs a warning to notify the driver when the steady-state determination unit 15 or the startup determination unit determines that a non-temporary malfunction state exists. The startup determination unit has a previous information acquisition unit that retrieves malfunction information stored in a storage area the last time the vehicle control device was turned off from a power-on state, and a provisional decision unit that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that no non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that no non-temporary malfunction state exists.

[0044] According to the notification device 10 or notification system 1 of this embodiment, a non-temporary malfunction state is determined not by using an image captured by a camera after the vehicle's control device is started, but by using malfunction information stored the previous time the vehicle was dismounted. Therefore, the period required to determine a non-temporary malfunction state can be shortened. Therefore, even when the steady-state determination unit 15 cannot determine whether a non-temporary malfunction state exists during the period leading up to the start of driving, such as when driving is started within a short period of time after the ECU power supply unit 111 is turned on, the driver can be notified of a non-temporary malfunction state that has been provisionally determined with a high probability. This improves driver convenience and allows the driver to receive a notification before driving, thereby improving the safety of the vehicle during driving.

[0045] Although the notification device 10 of this embodiment is configured with one circuit board, it may also be configured with a combination of multiple circuit boards. The notification system 1 of this embodiment may be configured to include multiple physically separated components. Furthermore, some of the components of the notification system 1 may be provided in an external device such as a smartphone or tablet device brought in by a driver or the like from outside the vehicle, or may be provided in a server outside the vehicle connected via a telecommunications line such as the Internet.

[0046] (Second embodiment) The second embodiment will be described with reference to Figures 8 to 12. In this embodiment, the startup determination unit 17 further includes a first image acquisition processing unit 173, a second image acquisition processing unit 174, and a similarity determination unit 175.

[0047] The storage processing unit 16 stores a previous image in the nonvolatile memory 12 as a storage area before the ECU power supply changes from ON to OFF the previous time. The previous image is an image acquired by the image acquisition unit 13 during the period from the previous ECU power supply change from ON to OFF the previous time. More specifically, the previous image is an image last acquired by the image acquisition unit 13 before the previous IG-ON state changes to the IG-OFF state. In other words, the previous image is the image last acquired by the image acquisition unit 13 in the previous steady state. Note that the storage processing unit 16 may store images acquired by the image acquisition unit 13 in the storage area at predetermined intervals while the ECU power supply is in the IG-ON state. In this case, the storage processing unit 16 may overwrite the most recent image with the image information of the previous time. In other words, the nonvolatile memory 12 stores the image acquired by the image acquisition unit 13 at predetermined intervals and last stored in the nonvolatile memory 12 by the storage processing unit 16 before the ECU power supply is turned OFF.

[0048] The first image acquisition processing unit 173 acquires a first image from the nonvolatile memory 12 serving as a storage area. The second image acquisition processing unit 174 acquires a second image from the image acquisition unit 13. The second image is an image captured by the camera 20 and acquired by the image acquisition unit 13 after the ECU power supply unit 111 has changed from the current OFF state to the ON state.

[0049] The similarity determination unit 175 compares the first image and the second image and determines whether they are similar. For example, the similarity determination unit 175 compares the first image and the second image and calculates the similarity between the first image and the second image. The similarity determination unit 175 may determine that the first image and the second image are similar if the similarity between the first image and the second image is equal to or greater than a predetermined threshold, and may determine that the first image and the second image are dissimilar if the similarity is less than the predetermined threshold. In this case, if a similar occlusion state occurs when the first image and the second image are captured, similar occlusion patterns, such as those shown in FIGS. 3(b-2), (c-2), 4(d-2), and (e-2), appear in both images. The similarity determination unit 175 can determine whether the first image is similar or calculate the similarity using any known method. For example, the similarity determination or calculation can be performed using a brightness histogram.

[0050] For example, as shown in an example in FIG. 9, the similarity or dissimilarity can be determined or the degree of similarity can be calculated by comparing the brightness histograms of the first image and the second image. The similarity determination unit 175 divides the first image and the second image into a plurality of regions, in this case five regions from frame (a) to frame (e). In this case, at least one or all of the plurality of regions may be set so that they partially overlap each other. The similarity determination unit 175 calculates the brightness histograms for each of the regions (a) to (e) of the first image and the second image. For example, the example shown in FIG. 9 shows an example of the brightness histogram for frame (c).

[0051] The similarity determination unit 175 compares the luminance histograms of the first and second images in each corresponding region. If the luminance histograms of all corresponding regions of the first and second images match, the similarity determination unit 175 determines that the first and second images are similar. If the luminance histograms of the first and second images in at least one corresponding region do not match, the similarity determination unit 175 determines that the first and second images are not similar. The predetermined threshold for similarity described above can be set within a range of, for example, approximately 85% or more, approximately 90% or more, or approximately 95% or more. In this example, the predetermined threshold for similarity is 90%. That is, in this case, the similarity determination unit 175 determines that the first and second images are similar if the calculated similarity between the first and second images is equal to or greater than the predetermined threshold of 90%, and determines that the first and second images are not similar if the calculated similarity between the first and second images is less than the predetermined threshold of 90%.

[0052] For example, the luminance histogram of the (c) frame in the second image (i) shown in Figure 9 matches the luminance histogram of the (c) frame in the first image. If the luminance histograms of the remaining (a), (b), (d), and (e) frames also match, the similarity determination unit 175 determines that the first image and the second image (i) are similar. The luminance histogram of the (c) frame in the second image (ii) does not match the luminance histogram of the (c) frame in the first image. The similarity determination unit 175 determines that the first image and the second image (ii) are not similar.

[0053] Another embodiment is shown in FIG. 10. In this example, the similarity determination unit 175 calculates an absolute value difference image abs (first image - second image) between a first image and a second image (not shown). In this case, similar to the above, the similarity determination unit 175 divides the first image and the second image into a plurality of regions, in this case five regions from frame (a) to frame (e). In this case, at least one or all of the plurality of regions may be set so that they partially overlap each other. The similarity determination unit 175 calculates a luminance histogram of the absolute value difference image abs (first image - second image) for each of regions (a) to (e). For example, in the example shown in FIG. 10, an example of the luminance histogram of the absolute value difference image abs (first image - second image) for frame (c) is shown.

[0054] The similarity determination unit 175 determines that the first and second images match in a region if a predetermined range of pixel counts in the brightness histogram of the absolute difference image abs (first image - second image) for each of the regions (a) to (e) is within a predetermined range of absolute difference values; otherwise, the similarity determination unit 175 determines that the first and second images do not match in that region. If the similarity determination unit 175 determines that the first and second images match in all regions, it determines that the first and second images are similar. If the first and second images do not match in at least one corresponding region, it determines that the first and second images are not similar. The predetermined range of pixel counts can be set, for example, within a range of approximately 85% or more, approximately 90% or more, or approximately 95% or more. In this example, the predetermined range of pixel counts is set to approximately 95% or more. The predetermined range of absolute difference values ​​can be set, for example, within a range of 0 to approximately 30, 0 to approximately 25, or 0 to approximately 20. In this example, the predetermined range of the absolute difference value is set to a range of 0-20.

[0055] For example, in the brightness histogram of the (c) frame of the absolute difference image abs(first image - second image (i)) between the first image and the second image (i) shown in FIG. 10, 100% of the pixels are within the range of absolute difference values ​​of 0 to 20. In this case, the similarity determination unit 175 determines that the first image and the second image (i) are similar for the (c) frame. If the brightness histograms of the remaining (a), (b), (d), and (e) frames are also within the range of absolute difference values ​​of 0 to 20, the similarity determination unit 175 determines that the first image and the second image (i) are similar. On the other hand, in the brightness histogram of the (c) frame of the absolute difference image abs(first image - second image (ii)) between the first image and the second image (ii), 100% of the pixels are not within the range of absolute difference values ​​of 0 to 20. The similarity determination unit 175 determines that the first image and the second image (ii) are not similar for the (c) frame. The similarity determination unit 175 also determines that the first image and the second image (ii) are not similar.

[0056] When the similarity determination unit 175 determines that the first image and the second image are similar, it can be estimated that there is no change in the non-temporary malfunction state between the first image and the second image. In other words, when the first image and the second image are similar, if there was a non-temporary malfunction state when the first image was captured, that is, when the IG was turned off last time, it can be estimated that the non-temporary malfunction state will continue to exist when the second image is captured, that is, when the ECU power supply unit 111 is turned on this time.

[0057] The provisional determination unit 172 determines whether the camera system is currently in a non-temporary malfunction state based on the malfunction information at the time of the previous IF-OFF and the similarity / dissimilarity determined by the similarity determination unit 175. In this case, the startup determination unit 17 acquires the second image and determines whether the first image and the second image are similar or dissimilar when the malfunction information at the time of the previous IF-OFF indicates a non-temporary malfunction state "present."

[0058] (IG-OFF flow chart) 11, the processing by the notification device 10 when the IG button 202 is operated to change the IG-ON state to the IG-OFF state will be described. Note that the processing that differs from the processing in the first embodiment shown in FIG. 6 will be described.

[0059] After step S22, the notification device 10 advances the process to step S41. In step S41, the notification device 10 stores the first image in the non-volatile memory 12 serving as a storage area. As a result, the malfunction state immediately before the change from the IG-ON state to the IG-OFF state and the image taken by the camera 20 are stored in the storage area so that they can be referenced. After step S41, the notification device 10 advances the process to step S23.

[0060] (Flowchart when ECU power is turned on) The processing by the notification device 10 when the ECU power supply unit 111 changes from an OFF state to an ON state will be described with reference to the flowchart shown in Fig. 12. Note that only differences from the processing in the first embodiment shown in Fig. 7 will be described.

[0061] After step S31, the notification device 10 proceeds to step S51. In step S51, the notification device 10 acquires a first image from the non-volatile memory 12, which serves as a storage area. After step S51, the notification device 10 proceeds to step S32. If the previous information indicates a malfunction (YES in step S32), the notification device 10 proceeds to step S52. In step S52, the notification device 10 acquires a second image from the camera 20 via the image acquisition unit 13.

[0062] In step S53, the notification device 10 determines whether the first image and the second image are similar. If it is determined that the two are not similar (NO in step S54), the notification device 10 proceeds to step S55. In step S55, the notification device 10 determines whether or not the current camera system is in a non-temporary malfunction state as "undetermined." That is, even if the malfunction information from the previous power-off was "present," if the first image and the second image are not similar, the notification device 10 does not determine that the non-temporary malfunction state is "present," but rather determines that it is "undetermined."

[0063] If it is determined that the first image and the second image are similar (YES in step S54), the notification device 10 proceeds to step S34. That is, if the previous malfunction information is "present" and the first image and the second image are similar, there is an extremely high probability that a non-temporary malfunction state is still continuing. Therefore, the notification device 10 provisionally determines that the current non-temporary malfunction state is "present."

[0064] According to the present embodiment described above, the storage processing unit 16 stores a first image, which is an image acquired by the image acquisition unit 13 before the vehicle ignition is switched from an on state to an off state, in the non-volatile memory 12 serving as a storage area. The image acquisition unit 13 acquires a second image, which is an image captured after the ECU 11 serving as a vehicle control device is switched from a power-off state to a power-on state, from the camera 20. The startup determination unit 17 includes a first image acquisition processing unit 173 that acquires the first image from the non-volatile memory 12 serving as a storage area, a second image acquisition processing unit 174 that acquires the second image from the image acquisition unit 13, and a similarity determination unit 175 that compares the first image and the second image to determine whether they are similar. The provisional decision unit 172 determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit 171 indicates that a non-temporary malfunction state is "present" and the similarity determination unit 175 determines that the first image and the second image are similar.

[0065] As a result, the non-temporary malfunction state at the time of this ride is determined based on the similarity between the image taken the previous time the vehicle was dismounted and the image taken the current time the vehicle was boarded, so that the non-temporary malfunction state can be determined with a higher probability within a short period of time after the ECU power supply unit 111 is turned on, thereby improving convenience for the driver and improving the safety of vehicle driving.

[0066] In this embodiment, the notification device 10 does not issue a notification via the warning output unit when it determines that the non-temporary malfunction state is "undetermined," but in other embodiments, the notification device 10 may be configured to notify the driver that the non-temporary malfunction state is undetermined.

[0067] (Third embodiment) A third embodiment will be described with reference to Fig. 13 to Fig. 16. In this embodiment, as shown in Fig. 13, the notification device 10 includes a target detection unit 19. The startup determination unit 17 also includes a condition instruction unit 176. In this embodiment, the condition instruction unit 176 sets the imaging conditions of the camera system at the time the ECU power is turned ON this time, in accordance with the malfunction information stored the previous time the IG was turned OFF.

[0068] The object detection unit 19 has a function of confirming that an object can be recognized in the image acquired by the image acquisition unit 13. The detection object may be, for example, a component constituting a part of the vehicle, such as the driver, the driver's seat, the headrest of the driver's seat, or a pillar next to the driver's seat. The detection object may be a plurality of items, in which case it is sufficient that at least one of the plurality of detection objects can be recognized. The object detection unit 19 is realized in software by the ECU executing a program stored in the non-volatile memory 12. Note that the object detection unit 19 may be configured as hardware, or may be realized as a combination of software and hardware.

[0069] As described below, if the object detection unit 19 cannot detect the detection object, it is highly likely that a non-temporary malfunction state has occurred in the camera system. On the other hand, if the object detection unit 19 detects the detection object, it is highly likely that a non-temporary malfunction state has not occurred in the camera system.

[0070] In this embodiment, the startup determination unit 17 determines whether or not a non-temporary malfunction state exists at startup based on whether or not the target detection unit 19 has detected a detection target, in addition to the malfunction information at the time of the previous IG-OFF acquired by the previous information acquisition unit 171. More specifically, if the malfunction information at the time of the previous IG-OFF acquired by the previous information acquisition unit 171 indicates that a non-temporary malfunction state exists, the target detection unit 19 determines whether or not a detection target can be detected in the second image.

[0071] If the similarity determination unit 175 determines that the first image and the second image are similar and the object detection unit 19 does not detect the detection object, the provisional determination unit 172 determines that the camera system is in a non-temporary malfunction state during the period from when the EUC power is turned on this time until the normal state is reached. On the other hand, if the similarity determination unit 175 determines that the first image and the second image are not similar and the object detection unit 19 detects the detection object, the provisional determination unit 172 determines that the camera system is not in a non-temporary malfunction state during the period from when the EUC power is turned on this time until the normal state is reached. Furthermore, even if the similarity determination unit 175 determines that the first image and the second image are similar, if the object detection unit 19 detects the detection object, the provisional determination unit 172 determines that the camera system is not in a non-temporary malfunction state during the period from when the EUC power is turned on this time.

[0072] The condition instruction unit 176 instructs the imaging conditions of the camera system at the time when the ECU power supply unit 111 is turned ON this time to the camera control unit 113 and / or the projector control unit 114. That is, the condition instruction unit 176 has a function of instructing the imaging conditions of the camera system for capturing the second image.

[0073] In a steady state, the camera control unit 113 and / or the projector analyzes the brightness distribution of the captured image and sets optimal imaging conditions to capture a clear image of the target. The imaging conditions can be changed by one or more items selected from a plurality of conditions, including, but not limited to, the exposure time of the camera 20, the gain amount, and the light intensity of the projector 30. The imaging conditions can be set according to the amount of light reaching the image sensor 23 through the lens 21 from the target, for example, according to the distance between the lens 21 and the target. The amount of received light and the gain can be set smaller as the distance between the lens 21 and the target is shorter, and larger as the distance between the lens 21 and the target is longer. Increasing the amount of received light and the gain can increase the photoelectric charge, enabling a clearer image to be captured even when the vehicle interior is dark.

[0074] The imaging conditions include a plurality of modes that are set depending on the amount of light received by the imaging element 23 and / or the gain amount, i.e., the sensitivity, of the imaging element 23. That is, the camera control unit 113 and / or the projector 30 can set a plurality of modes that change the imaging conditions. For example, as shown in FIG. 14 , the imaging conditions can be set to include at least three modes: a "face detection mode," a "seat detection mode," and an "occlusion detection mode." The "face detection mode" is a mode in which a typical distance between the driver's face and the lens 21 is set in advance, and the amount of received light and the amount of gain are set so that an object at that distance can be captured with a desirable degree of shading. The "seat detection mode" is a mode in which a typical distance between the driver's seat and the lens 21 is set in advance, and the amount of received light and the amount of gain are set so that an object at that distance can be captured with a desirable degree of shading. Since it is assumed that the driver's seat is located farther from the lens 21 than the driver's face, the amount of light received in the "sheet detection mode" is set to be greater than the amount of light received in the "face detection mode", and / or the gain amount in the "sheet detection mode" is set to be greater than the gain amount in the "face detection mode". The amount of light received in the "occlusion detection mode" is set to be greater than the amount of light received in the "face detection mode" and "sheet detection mode", and / or the gain amount in the "occlusion detection mode" is set to be greater than the gain amount in the "face detection mode" and "sheet detection mode".

[0075] In this case, if the camera system is not occluded, as shown in (a-1) of Figure 3, the driver's face can be properly detected in the image captured in "face detection mode." The image captured in "seat detection mode" has more light than in "face detection mode," so the image appears whitish overall. The image captured in "occlusion detection mode" may be slightly overexposed, but the driver's face can still be detected. In this case, even if the interior of the vehicle is dark, such as at night, the driver's face and other detection targets can be detected in the image captured in "occlusion detection mode."

[0076] 3(b-1), if there is an obstruction S at zero distance that covers the entire lens 21 of the camera 20, no light reaches the image sensor 23, regardless of whether an image is captured in the "face detection mode," "sheet detection mode," or "occlusion detection mode," and the image is black. In this case, the object detection unit 19 cannot detect the detection object.

[0077] 3(c-1), if there is an obstruction S at zero distance that partially covers the lens 21 of the camera 20, no matter which imaging mode, "face detection mode," "sheet detection mode," or "occlusion detection mode," is used to capture an image, the portion corresponding to the obstructed portion will appear black because no light reaches the imaging element 23. In this case, depending on the size of the obstruction S, the object detection unit 19 may not be able to detect the detection target.

[0078] In a steady state, the camera control unit 113 and / or the projector control unit 114 analyzes the brightness distribution of the captured image, and if the image is dark, captures the image under the "occlusion detection mode" imaging condition. If the image is partially or entirely black even when captured in the "occlusion detection mode," the steady-state determination unit 15 determines that either the camera 20 or the projector 30, or both, are occluded. In other words, when the steady-state determination unit 15 determines that there is malfunction information, the captured image was captured under the "occlusion detection mode" imaging condition. Therefore, when the malfunction information indicates that there is non-temporary malfunction information, the first image simultaneously stored in the non-volatile memory 12 is the one captured under the "occlusion detection mode."

[0079] In response to an operation of the IG-ON / OFF button to change the IG-ON state to the IG-OFF state, the storage processing unit 16 stores the imaging conditions set by the camera control unit 113 and the projector control unit 114 before the IG-OFF state was entered in the non-volatile memory 12 as a storage area. That is, when the IG-ON state is changed to the IG-OFF state, the storage processing unit 16 stores the imaging conditions of the first image in addition to the first image. The imaging conditions set by the camera control unit 113 and the projector control unit 114 before the IG-OFF state is entered are referred to as first imaging conditions.

[0080] If the malfunction information acquired by the previous information acquisition unit 171 at the time of the previous IG-OFF indicates a non-temporary malfunction state, the condition designation unit 176 sets the imaging conditions of the camera system to one or more modes including at least the occlusion detection mode. That is, even when imaging is performed in at least the "occlusion detection mode," if the captured image is a black image or a partially black image, there is a high possibility that occlusion exists in the camera system. Therefore, if the previous malfunction information indicates a non-temporary malfunction state and the target detection unit 19 cannot detect the detection target even when imaging in the "occlusion detection mode," the probability that the non-temporary malfunction state is continuing becomes even higher.

[0081] In this embodiment, if the malfunction information acquired by the previous information acquisition unit 171 when the IG-OFF was last performed indicates that a non-temporary malfunction state exists, the condition instruction unit 176 sets the imaging conditions when the ECU power is turned ON this time to the same imaging conditions as the first imaging conditions. In this case, the condition instruction unit 176 sets the imaging conditions when the ECU power is turned ON this time to the "shield mode." Therefore, it is not necessary to newly capture an image using the camera system, observe the luminance distribution of the driver's face area, and adjust the imaging conditions, so that the second image can be acquired more quickly, that is, in a shorter period of time after the ECU power supply unit 111 is started up.

[0082] When capturing the second image, the condition instructing unit 176 instructs the camera control unit 113 and the projector control unit 114 to use the first capturing conditions, i.e., the same conditions as the capturing conditions when the first image was captured and stored in the nonvolatile memory 12.

[0083] The object detection unit 19 determines whether or not the detection object can be detected in the second image captured under the imaging conditions set by the condition designation unit 176.

[0084] (IG-OFF flow chart) The processing by the notification device 10 when the IG-ON / OFF button is operated to change the IG-ON state to the IG-OFF state will be described with reference to the flowchart shown in Fig. 15. Note that differences from the processing in the first embodiment shown in Fig. 6 will be described.

[0085] After step S41, the notification device 10 advances the process to step S61. In step S61, the notification device 10 stores the imaging conditions set by the camera control unit 113 and the projector control unit 114 before the IG-OFF state was entered, i.e., the first imaging conditions, in the nonvolatile memory 12 as a storage area. As a result, the malfunction state immediately before the change from the IG-ON state to the IG-OFF state, the image taken by the camera 20, and the first imaging conditions are stored in the nonvolatile memory 12 so as to be able to be referenced. After step S61, the notification device 10 advances the process to step S23.

[0086] (Flowchart when ECU power is turned on) The processing by the notification device 10 when the ECU power supply unit 111 changes from an OFF state to an ON state will be described with reference to the flowchart shown in Fig. 16. Note that only the differences from the processing in the second embodiment shown in Fig. 10 will be described.

[0087] If the previous malfunction information stored in the non-volatile memory 12 serving as a storage area when the IG-OFF occurred the previous time indicates that a non-temporary malfunction state exists (YES in step S32), the notification device 10 proceeds to step S71. In step S71, the notification device 10 acquires information on the first imaging condition from the non-volatile memory 12 serving as a storage area. In step S72, the notification device 10 sets the imaging condition of the camera system to the same imaging condition as the first imaging condition. As a result, the second image is captured in an "occlusion detection mode" in which the amount of light or gain is increased compared to the imaging condition when there is no non-temporary malfunction state. After step S72, the notification device 10 proceeds to step S52.

[0088] After step S52, the notification device 10 proceeds to step S73. In step S73, the notification device 10 determines whether a detection target has been detected in the captured image. If a detection target has been detected (YES in step S73), the notification device 10 proceeds to step S33. In other words, if a detection target has been detected, it is highly likely that the non-temporary malfunction state has been resolved, for example, by removing an obstruction, between the time when the IG was last turned OFF and the time when the ECU power was turned ON this time. Therefore, in this case, the startup determination unit 17 determines that there is no non-temporary malfunction state.

[0089] If the detection target is not detected (NO in step S73), the notification device 10 proceeds to step S54. In other words, if the detection target is not detected, it is highly likely that a non-temporary malfunction state continues, such as an obstruction not being removed, and therefore the similarity or dissimilarity between the first image and the second image is determined to provisionally determine whether a non-temporary malfunction state exists.

[0090] According to the present embodiment described above, the notification device 10 includes an object detection unit 19 that detects a detection object inside the vehicle on the second image. The provisional determination unit 172 determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit 171 indicates the presence of a non-temporary malfunction state and when it is determined that the first image and the second image are similar and the object detection unit 19 does not detect the detection object. Even if the malfunction information acquired by the previous information acquisition unit 171 indicates the presence of a non-temporary malfunction state, the provisional determination unit 172 determines that a non-temporary malfunction state does not exist if the similarity determination unit determines that the first image and the second image are not similar and the object detection unit 19 detects the detection object.

[0091] This further improves the accuracy of the startup determination unit 17 in determining that a non-temporary malfunction state exists, based on a condition that can be determined in a relatively short time, i.e., whether or not an object is detected in an image captured after the ECU power is turned on this time. Also, if the detection object is detected, it means that the non-temporary malfunction state has been resolved, and the camera system can be used, improving user convenience.

[0092] The storage processing unit 16 stores the imaging conditions of the first image in the nonvolatile memory 12 as a storage area. When the malfunction information acquired by the previous information acquisition unit 171 indicates the presence of a non-temporary malfunction state, the startup determination unit 17 sets the imaging conditions of the second image to be the same as the imaging conditions of the first image.

[0093] This allows the second image after boarding to be captured under the same imaging conditions as the first image captured before disembarking the previous time, eliminating error components due to differences in imaging conditions when comparing the first and second images. Furthermore, since there is no need to analyze the brightness distribution of the image again and set optimal imaging conditions when capturing the second image, the time required to acquire the second image can be reduced. Therefore, the start-up determination unit 17 can determine whether or not a non-temporary malfunction exists in a shorter time.

[0094] The imaging conditions for the first image when there is a non-temporary malfunction state are imaging conditions in which the amount of light received by the imaging element 23 of the camera 20 or the sensitivity is increased compared to the imaging conditions when there is no non-temporary malfunction state.

[0095] When a non-temporary malfunction state occurs, even if the amount of light received by the image sensor 23 or the sensitivity is increased, the image will be partially or entirely black. Therefore, by comparing the first image and the second image captured using the "occlusion detection mode," in which the amount of light received by the image sensor 23 or the sensitivity is increased compared to the "face detection mode" or the "sheet detection mode," it is possible to improve the accuracy with which the two images are correctly determined to be similar when a non-temporary malfunction state continues. This improves the accuracy with which the startup determination unit 17 determines whether or not a non-temporary malfunction state exists.

[0096] (Fourth embodiment) A fourth embodiment will be described with reference to Figures 17 and 18. In this embodiment, similar to the third embodiment, the notification device 10 includes an object detection unit 19. Furthermore, the startup determination unit 17 includes a condition specification unit 176.

[0097] If the malfunction information stored in the non-volatile memory 12 indicates a non-temporary malfunction state when the ECU power supply is switched from the OFF state to the ON state, the notification device 10 changes the imaging conditions of the camera system, captures images multiple times, and acquires second images captured under the multiple imaging conditions when the ECU power supply is switched from the OFF state to the ON state this time. That is, the condition specifying unit 176 specifies multiple imaging conditions to the camera control unit 113 and / or the projector control unit 114 when capturing the second images. The notification device 10 compares the multiple second images captured under the multiple imaging conditions with the first image and performs a similarity determination for each of the second images and the first image. The multiple imaging conditions are multiple conditions set by increasing or decreasing the amount of light and / or gain received by the image sensor 23. At least one of the multiple imaging conditions is the imaging condition used when capturing the first image, i.e., the "occlusion detection mode." In this embodiment, the multiple imaging conditions are three imaging conditions, including the "face detection mode," "seat detection mode," and "occlusion detection mode" described above.

[0098] As shown in Figure 14, when the occlusion state is a zero-distance occlusion state like (b-1), even if the image is captured under multiple imaging conditions by increasing or decreasing the amount of light and / or gain, the captured image will be a black image. On the other hand, if the image is black simply because the image is captured at night or the ambient light is low, the detection target may be captured by capturing the image under imaging conditions with increased light and / or gain. Therefore, by comparing the second image captured under multiple imaging conditions with the first image, the accuracy of determining whether a non-temporary malfunction state exists can be improved.

[0099] The similarity determination unit 175 compares each of the multiple second images with the first image to determine whether they are similar. If a non-temporary malfunction state occurred the last time the IG-ON state was changed to the IG-OFF state and all of the multiple second images are similar to the first image, it is determined that there is a high possibility that the non-temporary malfunction state is still continuing. On the other hand, if not all of the multiple second images are similar to the first image, there is a possibility that the non-temporary malfunction state is not currently continuing.

[0100] The provisional determination unit 172 provisionally determines whether the camera system is in a non-temporary malfunction state when the ECU power supply unit 111 is turned ON this time, based on the similarity / dissimilarity between the multiple second images and the first image determined by the similarity determination unit.

[0101] (Flowchart when ECU power is turned on) The processing by the notification device 10 when the ECU power supply is changed from an OFF state to an ON state will be described with reference to the flowcharts shown in Figures 17 and 18. Note that only the differences from the processing in the second embodiment shown in Figure 13 will be described.

[0102] If the malfunction information stored in the non-volatile memory 12 as a storage area when the IG-OFF occurred last time indicates a non-temporary malfunction state (YES in step S32), the notification device 10 proceeds to step S81. In step S81, the notification device 10 executes second image processing. The second image processing is processing for acquiring multiple second images captured under multiple imaging conditions and determining whether each of the second images is similar to the first image.

[0103] As shown in the flowchart of FIG. 18, when the second image processing starts, the notification device 10 sets imaging condition A, which is the first imaging condition, in step S91A. The notification device 10 acquires a second image A captured under imaging condition A in step S92A. The notification device 10 calculates whether the second image A and the first image are similar or not in step S93A. For example, imaging condition A is the "face detection mode." The notification device 10 sets imaging condition B, which is the second imaging condition, in step S91B. Imaging condition B is an imaging condition different from imaging condition A. The notification device 10 acquires a second image B captured under imaging condition B in step S92B. The notification device 10 calculates whether the second image B and the first image are similar or not in step S93B. For example, imaging condition B is the "sheet detection mode." The notification device 10 sets imaging condition C, which is the third imaging condition, in step S91C. Imaging condition C is an imaging condition different from imaging condition A and imaging condition B. In step S92C, the notification device 10 acquires a second image C captured under imaging condition C. In step S93C, the notification device 10 calculates whether the second image C and the first image are similar. For example, imaging condition C is the "occlusion detection mode."

[0104] In this embodiment, the number of imaging conditions in the second image processing is set to three, but is not limited to this. The imaging conditions in the second image processing may be two or more imaging conditions including at least the "occlusion detection mode," and may be two imaging conditions or four or more imaging conditions. In this case, the notification device 10 repeats the processes from steps S91X to S93X (X is any alphabet) the number of times equal to the number of imaging conditions.

[0105] In this way, in the second image processing, second images captured under a plurality of imaging conditions including at least the "occlusion detection mode" are acquired, and it is determined whether each second image is similar to the first image. When all processing in the second image processing is completed (return), the notification device 10 proceeds to step S73 in the flowchart of FIG.

[0106] If the detection target is not detected (NO in step S73), the notification device 10 proceeds to step S82. In step S82, the notification device 10 determines whether all the similarity determination results in the second image processing are similar. If all the similarity determination results are similar (YES in step S82), the notification device 10 proceeds to step S34. If not all the similarity determination results are similar, that is, if some or all of the similarity determination results are dissimilar (NO in step S82), the notification device 10 proceeds to step S55. In other words, if even one of the second images captured under multiple imaging conditions is dissimilar, there is a possibility that the state of the camera system has changed from the state before the previous disembarkation, so it is not determined that a non-temporary malfunction exists, and the non-temporary malfunction state is left undetermined.

[0107] According to the present embodiment described above, when the malfunction information acquired by the previous information acquisition unit 171 indicates the presence of a non-temporary malfunction state, the similarity determination unit 175 compares each of the multiple second images captured under multiple imaging conditions set by increasing or decreasing the amount of light received by the image sensor 23 of the camera 20 or the gain amount with the first image to determine whether they are similar or not.

[0108] In a non-temporary malfunction state, such as when the camera 20 or the projector 30 is blocked at zero distance, the second image is an all-black image regardless of the light amount or sensitivity. By utilizing this characteristic to determine the similarity between the first image and the second image captured under multiple imaging conditions, the accuracy of the similarity determination when a non-temporary malfunction state occurs can be improved. Therefore, it is possible to suppress erroneous determination of a non-temporary malfunction state at startup, improving convenience and safety for the driver.

[0109] When the similarity determining section 175 determines that the first image and the second images captured under the respective imaging conditions are all similar to each other, the provisional determination section determines that a non-temporary malfunction state exists.

[0110] If the similarity or dissimilarity between the first and second images differs depending on the imaging conditions of the second image, it is highly likely that the current startup is not in a non-temporary malfunction state due to occlusion at zero distance. Therefore, by determining that a non-temporary malfunction state exists when the images are similar under all multiple imaging conditions, it is possible to reduce erroneous determinations due to differences in imaging conditions.

[0111] Although the present disclosure has been described with reference to the embodiments, it is understood that the present disclosure is not limited to the embodiments or structures. The present disclosure also encompasses various modifications and equivalent modifications. In addition, various combinations and forms, including only one element, more than one element, or less than one element, are also within the scope and spirit of the present disclosure.

[0112] The controller and the method described herein may be implemented by a special-purpose computer configured with a processor and memory programmed to perform one or more functions embodied in a computer program. Alternatively, the controller and the method described herein may be implemented by a special-purpose computer configured with a processor comprising one or more dedicated hardware logic circuits. Alternatively, the controller and the method described herein may be implemented by one or more special-purpose computers configured with a processor and memory programmed to perform one or more functions in combination with a processor configured with one or more hardware logic circuits. Furthermore, the computer program may be stored in a computer-readable non-transitory tangible storage medium as instructions executed by a computer.

[0113] In addition to the inventions set forth in the claims, the present disclosure further includes the following inventions.

[0114] [1] an image acquisition unit (13) that acquires an image from a camera (20) that captures an image of the interior of the vehicle; a steady-state determination unit (15) that determines whether or not a non-temporary malfunction occurs in a camera system including the camera in a steady state based on the image acquired by the image acquisition unit; a storage processing unit (16) for storing in a storage area (12) malfunction information, which is information regarding a non-temporary malfunction state occurring in the camera system determined by the steady-state determination unit in response to the ignition of the vehicle being changed from an on state to an off state; a startup determination unit (17) that determines whether or not a non-temporary malfunction occurs in the camera system during a period from when the ignition is turned on from an off state until when the steady-state determination unit becomes capable of making a determination; a warning output processing unit (183) that outputs a warning to notify a driver when the steady-state determination unit or the startup determination unit determines that the camera system is in a non-temporary malfunction state, The startup determination unit includes a previous information acquisition unit (171) that retrieves the malfunction information stored in the storage area when the vehicle control device (11) last changed from a power-on state to an power-off state, and a provisional decision unit (172) that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that a non-temporary malfunction state does not exist if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state does not exist. Notification device (10).

[0115] [2] the storage processing unit stores in the storage area a first image that is an image acquired by the image acquisition unit before an ignition of the vehicle is changed from an on state to an off state; the image acquisition unit acquires from the camera a second image that is an image captured after the control device has been switched from a power-off state to a power-on state; the startup determination unit includes a first image acquisition processing unit (173) that acquires the first image from the storage area, a second image acquisition unit (174) that acquires the second image from the image acquisition unit, and a similarity determination unit (175) that compares the first image with the second image and determines whether or not they are similar; the provisional determination unit determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists and the similarity determination unit determines that the first image and the second image are similar. [1] The notification device according to [1].

[0116] [3] an object detection unit (19) that detects an object within the vehicle on the second image; the provisional determination unit determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and when the similarity determination unit determines that the first image and the second image are similar and the object detection unit does not detect the detection object, The provisional determination unit determines that there is no non-temporary malfunction state if the similarity determination unit determines that the first image and the second image are not similar and the object detection unit detects the detection object, even if the malfunction information acquired by the previous information acquisition unit indicates that there is a non-temporary malfunction state. [2] The notification device according to [2].

[0117] [4] the storage processing unit stores the imaging conditions of the first image in the storage area; When the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, the startup determination unit sets the imaging conditions of the second image to be the same as the imaging conditions of the first image. A notification device according to any one of [1] to [3].

[0118] [5] The imaging conditions for the first image when the non-temporary malfunction state exists are imaging conditions in which the amount of light received by the imaging element of the camera or the sensitivity is increased compared to the imaging conditions when the non-temporary malfunction state does not exist. [1] to [4]. A notification device according to any one of the preceding claims.

[0119] [6] When the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, the similarity determination unit compares each of the plurality of second images captured under a plurality of imaging conditions set by increasing or decreasing the amount of light received by the imaging element of the camera or the amount of gain with the first image to determine whether they are similar or not. A notification device according to any one of [1] to [3].

[0120] [7] the provisional determination unit determines that a non-temporary malfunction state exists when the similarity determination unit determines that the second images and the first image captured under the multiple imaging conditions are all similar to each other; A notification device according to any one of [1] to [3] or [6].

[0121] [8] an image acquisition unit (13) that acquires an image from a camera (20) that captures an image of the interior of the vehicle; a steady-state determination unit (15) that determines whether or not a non-temporary malfunction occurs in a camera system including the camera in a steady state based on the image acquired by the image acquisition unit; a storage area (12) for storing malfunction information, which is information relating to a non-temporary malfunction state occurring in the camera system determined by the steady-state determination unit in a manner that will not be erased even when the power supply of the vehicle control device is turned off; a storage processing unit (16) for storing the malfunction information in the storage area in response to an ignition switch of the vehicle being switched from an on state to an off state; a startup determination unit (17) that determines whether or not a non-temporary malfunction occurs in the camera system during a period from when the ignition is turned on from an off state until when the steady-state determination unit becomes capable of making a determination; a warning output processing unit (183) that notifies a driver when the steady-state determination unit or the startup determination unit determines that a non-temporary malfunction state exists; The startup determination unit includes a previous information acquisition unit (171) that retrieves the malfunction information stored in the storage area when the control device was last switched from a power-on state to an off state, and a provisional decision unit (172) that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that a non-temporary malfunction state does not exist if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state does not exist. Notification system (1). [Explanation of symbols]

[0122] 1...Notification system, 10...Notification device, 11...Vehicle control device (ECU, control device), 12...Non-volatile memory (storage area), 13...Image acquisition unit, 15...Steady-state judgment unit, 16...Storage processing unit, 17...Start-up judgment unit, 171...Previous information acquisition unit, 172...Temporary decision unit, 173...First image acquisition processing unit, 174...Second image acquisition processing unit, 175...Similarity judgment unit, 183...Alarm output processing unit, 19...Object detection unit, 20...Camera

Claims

1. an image acquisition unit (13) that acquires an image from a camera (20) that captures an image of the interior of the vehicle; a steady-state determination unit (15) that determines whether or not a non-temporary malfunction has occurred in a camera system including the camera in a steady state based on the image acquired by the image acquisition unit; a storage processing unit (16) for storing malfunction information, which is information relating to the non-temporary malfunction state determined by the steady-state determination unit in response to the ignition of the vehicle being changed from an on state to an off state, in a storage area (12); a startup determination unit (17) that determines whether or not a non-temporary malfunction state occurs in the camera system during a period from when the ignition is turned on from an off state until when the steady-state determination unit becomes capable of making a determination; a warning output processing unit (183) that outputs a warning to notify a driver when the steady-state determination unit or the startup determination unit determines that the camera system is in a non-temporary malfunction state, The startup determination unit includes a previous information acquisition unit (171) that retrieves the malfunction information stored in the storage area when the vehicle control device (11) last switched from a power-on state to an off state, and a provisional decision unit (172) that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that a non-temporary malfunction state does not exist if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state does not exist. Notification device (10).

2. the storage processing unit stores in the storage area a first image, which is an image acquired by the image acquisition unit before an ignition of the vehicle is changed from an on state to an off state; the image acquisition unit acquires from the camera a second image that is an image captured after the control device has been switched from a power-off state to a power-on state; the startup determination unit includes a first image acquisition processing unit (173) that acquires the first image from the storage area, a second image acquisition unit (174) that acquires the second image from the image acquisition unit, and a similarity determination unit (175) that compares the first image with the second image and determines whether they are similar to each other; the provisional determination unit determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists and the similarity determination unit determines that the first image and the second image are similar. The notification device of claim 1 .

3. an object detection unit (19) that detects an object to be detected inside the vehicle on the second image; the provisional determination unit determines that a non-temporary malfunction state exists when the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and when the similarity determination unit determines that the first image and the second image are similar and the object detection unit does not detect the detection object, The provisional determination unit determines that there is no non-temporary malfunction state if the similarity determination unit determines that the first image and the second image are not similar and the object detection unit detects the detection object, even if the malfunction information acquired by the previous information acquisition unit indicates that there is a non-temporary malfunction state. The notification device of claim 2 .

4. the storage processing unit stores the imaging conditions of the first image in the storage area; When the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, the startup determination unit sets the imaging conditions of the second image to be the same as the imaging conditions of the first image. The notification device of claim 2 .

5. The imaging conditions for the first image when the non-temporary malfunction state exists are imaging conditions in which the amount of light received by the imaging element of the camera or the sensitivity is increased compared to the imaging conditions when the non-temporary malfunction state does not exist. The notification device of claim 4.

6. When the malfunction information acquired by the previous information acquisition unit indicates a non-temporary malfunction state, the similarity determination unit compares each of the plurality of second images captured under a plurality of imaging conditions set by increasing or decreasing the amount of light received by the imaging element of the camera or the amount of gain with the first image to determine whether they are similar or not. The notification device of claim 2 .

7. the provisional determination unit determines that a non-temporary malfunction state exists when the similarity determination unit determines that the second images and the first image captured under the plurality of imaging conditions are all similar to each other; The notification device of claim 6.

8. an image acquisition unit (13) that acquires an image from a camera (20) that captures an image of the interior of the vehicle; a steady-state determination unit (15) that determines whether or not a non-temporary malfunction occurs in a camera system including the camera in a steady state based on the image acquired by the image acquisition unit; a storage area (12) for storing malfunction information, which is information relating to a non-temporary malfunction state occurring in the camera system determined by the steady-state determination unit in a manner that will not be erased even when the power supply of the vehicle control device is turned off; a storage processing unit (16) for storing the malfunction information in the storage area in response to an ignition of the vehicle being switched from an on state to an off state; a startup determination unit (17) that determines whether or not a non-temporary malfunction occurs in the camera system during a period from when the ignition is turned on from an off state until when the steady-state determination unit becomes capable of making a determination; a warning output processing unit (183) that notifies a driver when the steady-state determination unit or the startup determination unit determines that a non-temporary malfunction state exists; The startup determination unit includes a previous information acquisition unit (171) that retrieves the malfunction information stored in the storage area when the control device was last switched from a power-on state to an off state, and a provisional decision unit (172) that determines that a non-temporary malfunction state exists if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state exists, and determines that a non-temporary malfunction state does not exist if the malfunction information acquired by the previous information acquisition unit indicates that a non-temporary malfunction state does not exist. Notification system (1).

Images