Image processing device

Abstract

An image processing device according to the present invention comprises: an anomaly determination unit that determines an anomaly in an imaging unit on the basis of a captured image; an environment determination unit that determines an environmental change using brightness information of a captured image; an anomaly cancellation determination unit that determines whether to cancel an anomaly flag of the imaging unit determined by the anomaly determination unit to be anomalous; and an adjustment unit that is capable of adjusting a determination criterion in the anomaly cancellation determination unit on the basis of the determination result from the environment determination unit.

Description

Image processing device

[0001] The present invention relates to an image processing device for determining an abnormality in an imaging unit mounted on a vehicle.

[0002] A camera mounted on a vehicle is judged to be abnormal if a foreign object adheres to the lens surface.Regarding this abnormality judgment, if an abnormality is judged to be present when the system is stopped, the abnormality judgment must be maintained when the system is started again.

[0003] Previously, it took about five seconds for the system to run after startup before it could re-determine the state of the camera's lens, but there is now a demand to be able to determine the state of the camera's lens immediately after the system is started.

[0004] A technology for determining the state of a camera lens is disclosed in Patent Document 1. In Patent Document 1, an image taken at the time the ignition switch is turned off is stored in a nonvolatile memory, and another image is acquired immediately after the ignition switch is turned on, and the luminance of the image degradation area in both images is compared to determine the state of the camera lens.

[0005] Japanese Patent Application Laid-Open No. 2021-052236

[0006] However, the technology of Patent Document 1 has the following problem. That is, if the environment changes (e.g., from night to morning) between when the ignition switch is turned off and when it is turned on, the brightness of the entire image changes in accordance with the change in environment. For example, if an abnormality occurs in the camera and the ignition switch is turned off at night, and then the next time the ignition switch is turned on during the day, the brightness of the entire image increases because it is daytime. As a result, even though the camera abnormality has not been resolved, the brightness of the image-degraded area increases, leading to a false determination that a foreign object has been removed from the lens and the camera abnormality has been resolved.

[0007] In view of the above-mentioned problems, the present invention has an object to provide an image processing apparatus that can correctly determine the state of an imaging unit immediately after the system is started up.

[0008] In order to solve the above problem, the image processing device of the present invention comprises an abnormality judgment unit that judges abnormalities in the imaging unit based on the captured image, an environment judgment unit that judges environmental changes using brightness information of the captured image, an abnormality cancellation judgment unit that judges whether or not to cancel the abnormality flag of the imaging unit that has been judged to be abnormal by the abnormality judgment unit, and an adjustment unit that can adjust the judgment criteria in the abnormality cancellation judgment unit based on the judgment result of the environment judgment unit.

[0009] According to the present invention, it is possible to provide an image processing device that can correctly determine the state of the imaging unit immediately after the system is started. Further features related to the present invention will become apparent from the description of this specification and the accompanying drawings. In addition, the problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0010] A diagram showing an example of a hardware configuration including an image processing device according to an embodiment of the present invention. A block diagram showing the functional configuration of an image processing device according to an embodiment of the present invention. A diagram for explaining an overview of processing executed by an image processing device according to an embodiment of the present invention. A diagram for explaining brightness difference thresholds before and after adjustment. A flowchart showing processing executed by an image processing device according to an embodiment of the present invention.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a block diagram showing the hardware configuration of a vehicle including an image processing device according to one embodiment. The vehicle 1 includes at least an image processing device 100, a camera 15, and a sensor 16. The image processing device 100 includes a CPU (Central Processing Unit) 11 that calculates parameters necessary for vehicle control based on data, a ROM (Read Only Memory) 12 that is a storage device from which data can be read, a RAM (Random Access Memory) 13 that can both write and read data, and a communication module 14 that converts the calculation results into a communication protocol and transmits the converted results.

[0013] The image processing device 100 is also connected to an external server 19 via a communication path 18. The communication path 18 may physically include multiple communication buses, and the standards of each communication bus may be the same or different. These communication bus standards include CAN (registered trademark), LIN (registered trademark), FlexRay (registered trademark), Ethernet (registered trademark), etc.

[0014] The image processing device 100 receives data that has passed through the communication path 18. In response to reception, the received data is expanded in the RAM 13. The CPU 11 then performs calculations based on the data written in the RAM 13 and the data read from the ROM 12. A program is pre-installed in the image processing device 100, and the CPU 11 executes the program to perform the processing described below. Note that the image processing device 100 may be configured to operate entirely within the vehicle, without being connected to an external server 19.

[0015] The vehicle 1 also includes a camera 15 and a sensor 16. A plurality of cameras 15 and sensors 16 are provided, and they acquire information about the vehicle itself, other vehicles traveling around the vehicle, and the surrounding environment of the vehicle (roads, signs, etc.). The camera 15 may be, for example, a stereo camera or a monocular camera, and the sensor 16 may be, for example, a millimeter-wave radar or a LiDAR (Light Detection and Ranging).

[0016] The vehicle 1 also includes an ignition switch 17. The ignition switch 17 is a switch for starting the system of the vehicle 1, and the vehicle 1 is started by a physical key, an electronic key, or the like, but the form of the switch is not particularly limited. The system of the vehicle 1 refers to a system for controlling the operation of the vehicle 1, and the image processing device 100 determines whether an environmental change has occurred between the time the system is shut down and the time it is started up, and whether an abnormal state of the camera has been resolved.

[0017] 2 is a functional block diagram showing the functional configuration of the image processing device 100. As shown in FIG. 2, the image processing device 100 according to this embodiment includes an abnormality determination unit 101, a storage unit 102, an environment determination unit 103, an abnormality resolution determination unit 104, and an adjustment unit 105.

[0018] The abnormality determination unit 101 determines whether or not an abnormality has occurred in the camera 15. The abnormality determination method used by the abnormality determination unit 101 can use known technology, but specifically, for example, the unit divides the image acquired by the camera 15 into multiple regions, compares the brightness of the entire image for each region, and determines that a region with a lower average brightness than the other regions is a degraded region where foreign matter or the like has adhered to the lens. If this degraded region occupies a predetermined percentage or more of the entire image, it can be determined that an abnormality has occurred in the camera 15. This predetermined percentage can be set arbitrarily, for example, to 30% of the entire screen.

[0019] In this embodiment, a degraded area refers to an area where foreign matter or the like has adhered to the lens and the brightness is below a predetermined value, as described above, and a non-degraded area refers to an area where no or almost no foreign matter has adhered to the lens and the average brightness is above a predetermined value.

[0020] The storage unit 102 has a function of storing images acquired by the camera 15. The storage unit 102 has a non-volatile storage function, and can continue to store images even when the ignition switch is turned off and the system is shut down.

[0021] When the environment determination unit 103 receives a signal indicating system startup from the ignition switch 17, it determines whether or not a change has occurred in the environment surrounding the vehicle 1 between the time the system is stopped and the time it is started.

[0022] Specifically, when a signal indicating that the system has been started is received from the ignition switch 17, the environment determination unit 103 compares the average brightness of the non-degraded area of ​​the image stored in the storage unit 102 immediately before the system was stopped with the image captured by the camera 15 immediately after the system was started. If the difference in average brightness is equal to or greater than a predetermined value, it is determined that an environmental change has occurred between the time the system was stopped and the time it was started.

[0023] The abnormality release determination unit 104 determines whether or not the abnormality flag of the camera 15 determined to be abnormal by the abnormality determination unit 101 may be released.

[0024] Specifically, the abnormality resolution determination unit 104 compares the average brightness of the degraded area in the image stored in the storage unit 102 immediately before the system was shut down with the average brightness of the area corresponding to the degraded area in the image captured by the camera 15 immediately after the system was started. If the difference in average brightness is equal to or greater than a predetermined brightness difference threshold, it determines that the abnormality in the camera 15 has been resolved, and cancels the flag stored in the storage unit 102 indicating that there is an abnormality in the camera 15. At this time, the brightness difference threshold is adjusted by the adjustment unit 105, which will be described later.

[0025] As shown in FIG. 3 , the environment determination unit 103 and the abnormality resolution determination unit 104 cooperate to perform the following functions. First, the environment determination unit 103 compares the average luminance of the non-degraded area at system shutdown with the average luminance of the area corresponding to the non-degraded area after system restart. Similarly, the abnormality resolution determination unit 104 compares the average luminance of the degraded area in the image at system shutdown with the average luminance of the area corresponding to the degraded area at system restart. If an environmental change occurs between system shutdown and restart (e.g., if the system is shut down at night and restarted during the day), the amount of exposure in the degraded area increases, which could lead to a false determination that the camera abnormality has been resolved even when the abnormality has not been resolved. However, in this embodiment, the environment determination unit 103 simultaneously compares the average luminance of the non-degraded area, thereby determining that the amount of exposure of the entire image has increased as the environment changed from night to day, making it possible to change the criterion for determining the degraded area (luminance difference threshold).

[0026] When it is determined that an environmental change has occurred between the time the system is shut down and the time it is started up, the adjustment unit 105 adjusts the brightness difference threshold used by the abnormality release determination unit 104 according to the ratio of change in exposure (specifically, the ratio of change in average exposure for all pixels constituting the degraded area) between the degraded area of ​​the image stored in the storage unit 102 immediately before the system was shut down and the area corresponding to the degraded area of ​​the image acquired by the camera 15 immediately after the system was started up.

[0027] To summarize the image processing device of this embodiment, the image processing device 100 includes an abnormality determination unit 101 that determines whether or not there is an abnormality in the camera (imaging unit 15) based on the captured image, an environment determination unit 103 that determines environmental changes using brightness information of the captured image, an abnormality release determination unit 104 that determines whether or not to release the abnormality flag of the camera 15 that has been determined to be abnormal by the abnormality determination unit 101, and an adjustment unit 105 that can adjust the determination criteria in the abnormality release determination unit 104 based on the determination result of the environment determination unit 103.

[0028] More specifically, the environmental determination unit 103 determines whether or not there is an environmental change by comparing the brightness of the non-image-degraded area before the system is shut down with the brightness of the area corresponding to the non-image-degraded area in the image captured after the system is started up, and if it is determined that there is an environmental change, the adjustment unit 105 adjusts the brightness difference threshold used by the abnormality release determination unit 104 as a criterion for determining whether or not to release the abnormality flag of the camera 15 that has been determined to be abnormal by the abnormality determination unit 101, depending on the change ratio between the exposure of the image-degraded area at the time of system shutdown and the exposure of the area corresponding to the image-degraded area before the system is shut down in the image captured after the system is started up.

[0029] In addition, the abnormality release judgment unit 104 compares the brightness of the image degradation area before the system is shut down with the brightness of the area corresponding to the image degradation area before the system is shut down in the image captured after the system is started up, and judges whether or not to release the abnormality flag of the imaging unit that has been judged to be abnormal by the abnormality judgment unit 101.

[0030] Furthermore, if the difference in brightness between the area of ​​the image captured after the system is started that corresponds to the area of ​​image degradation before the system is shut down and the area of ​​image degradation before the system is shut down is equal to or greater than a predetermined brightness difference threshold, the abnormality release judgment unit 104 releases the abnormality flag of the imaging unit that has been judged to be abnormal by the abnormality judgment unit 101.

[0031] An example of a specific method for adjusting the brightness difference threshold value executed by the adjustment unit 105 will be described with reference to FIG. 4. FIG. 4 is a graph showing the brightness of degraded and non-degraded areas in the cases of night and day. As shown in FIG. 4, the brightness is higher during the day than at night, and the brightness of non-degraded areas is higher than that of degraded areas. Furthermore, the brightness of degraded areas during the day is higher than that of non-degraded areas at night. Note that the brightness on the vertical axis in FIG. 4 indicates the average brightness of each area, or more specifically, indicates the average brightness of all pixels constituting each area.

[0032] Suppose an environmental change occurs between the time the system is shut down and the time it is started up, i.e., it is nighttime when the system is shut down but it is daytime when the system is started up. In this case, as explained in Patent Document 1, the conventionally used luminance difference threshold (the luminance difference threshold before adjustment) does not take into account the environmental change, and therefore becomes the difference between the luminance Tn2 of the non-degraded area at night and the luminance Tn1 of the degraded area at night.

[0033] If the environment changes from night to day in this state, the brightness of the degraded area will increase even if the camera abnormality has not been resolved as described above (the difference between the brightness of the degraded area on the saved image and the brightness of the degraded area on the image acquired after the system is started will be greater than the brightness difference threshold before adjustment), and it will be erroneously determined that the camera abnormality has been resolved.

[0034] Therefore, in this embodiment, when it is determined that an environmental change has occurred, the brightness difference threshold is set to be less than the difference (Td2-Tn1) between the brightness Td2 of the non-degraded area during the day and the brightness Tn1 of the degraded area at night, and greater than or equal to the difference (Td1-Tn1) between the brightness Td1 of the degraded area during the day and the brightness Tn1 of the degraded area at night. By setting it in this way, the brightness of the degraded area is determined taking the environmental change into account, making it possible to correctly determine whether the camera abnormality has been resolved.

[0035] Next, the processing executed by the image processing apparatus 100 in this embodiment will be described with reference to the flowchart of FIG.

[0036] This process starts when the environment determination unit 103 receives a system startup signal from the ignition switch 17. First, in step S101, the environment determination unit 103 reads out the camera information at the time of system shutdown that is stored in the storage unit 102. Next, in step S102, the environment determination unit 103 determines whether the read camera information includes an abnormality flag.

[0037] If the camera information does not include a camera abnormality flag, the process proceeds to step S108, where it is determined that there is no abnormality in the camera 15 and the process ends (camera status is normal).

[0038] If the camera information includes an abnormality flag, the environment determination unit 103 performs environment determination in step S103. The method for this is as described above, and the average brightness of the non-degraded area of ​​the image saved when the system was stopped and the average brightness of the area corresponding to the non-degraded area of ​​the image acquired after the system was started are acquired.

[0039] In step S104, the environment determination unit 103 determines whether the difference in average brightness acquired in step S103 is equal to or greater than a predetermined value. If it is determined that the difference is equal to or greater than the predetermined value, it is determined that there has been an environmental change and the process proceeds to step S105. If it is determined that the difference is less than the predetermined value, it is determined that there has not been an environmental change and the process proceeds to step S106.

[0040] In step S105, the adjustment unit 105 calculates a brightness difference threshold from the change rate of the exposure of the deteriorated area of ​​the image saved when the system was stopped as described above and the area corresponding to the deteriorated area of ​​the image acquired after the system was started, and adds this to the brightness difference threshold used by the abnormality release determination unit 104. Specifically, as described with reference to Fig. 4 , the brightness difference threshold before adjustment is added to the brightness difference threshold according to the change rate of the exposure to obtain the adjusted brightness difference threshold.

[0041] In step S106, the abnormality resolution determination unit 104 determines whether the abnormality in the camera has been resolved using the brightness difference threshold (or the adjusted brightness difference threshold if adjusted by the adjustment unit 105 in step S105). That is, the difference between the brightness of the deteriorated area before the system was shut down and the brightness of the area corresponding to the deteriorated area after the system was restarted is compared with the brightness difference threshold. If the brightness difference is less than the threshold, it is determined that the abnormality in the camera 15 has not been resolved, and the process proceeds to step S107. If the brightness difference is equal to or greater than the threshold, it is determined that the abnormality in the camera 15 has been resolved, and the process proceeds to step S108.

[0042] In step S107, the camera abnormality flag is held, and the driver is notified that the camera abnormality continues, for example, by issuing a warning through the abnormality release determination unit 104. In step S108, the abnormality flag stored in the storage unit 102 is deleted, and the process ends.

[0043] In the above embodiment, the average brightness on the image is used as the feature for detecting an abnormality in the camera, but this is not limiting, and brightness variance or parallax may also be used.

[0044] The above-described embodiment of the present invention provides the following advantageous effects.

[0045] (1) The image processing device of the present invention includes an abnormality determination unit that determines an abnormality in the imaging unit based on a captured image, an environment determination unit that determines an environmental change using brightness information of the captured image, an abnormality cancellation determination unit that determines whether to cancel the abnormality flag of the imaging unit that has been determined to be abnormal by the abnormality determination unit, and an adjustment unit that can adjust the determination criteria in the abnormality cancellation determination unit based on the determination result of the environment determination unit.

[0046] With the above configuration, it is possible to change the criteria for determining whether a camera has an abnormality in accordance with changes in the environment, making it possible to provide an image processing device that can correctly determine whether a camera has an abnormality immediately after the system is started, even if a change occurs in the environment.

[0047] (2) The environmental determination unit determines whether or not there is an environmental change by comparing the brightness of the non-image degradation area before the system is shut down with the brightness of the area corresponding to the non-image degradation area before the system is shut down in the image captured after the system is started up.If it is determined that there is an environmental change, the adjustment unit adjusts the brightness difference threshold used by the abnormality release determination unit as a criterion for determining whether or not to release the abnormality flag of the imaging unit that has been determined to be abnormal by the abnormality determination unit, depending on the change ratio between the exposure of the image degradation area before the system is shut down and the exposure of the area corresponding to the image degradation area before the system is shut down in the image captured after the system is started up.

[0048] (3) The abnormality release judgment unit compares the brightness of the image degradation area before the system is shut down with the brightness of the area corresponding to the image degradation area before the system is shut down in the image captured after the system is started, and judges whether or not to release the abnormality flag of the imaging unit that has been judged to be abnormal by the abnormality judgment unit.

[0049] (4) The abnormality resolution determination unit clears the abnormality flag of the imaging unit determined to be abnormal by the abnormality determination unit when the difference between the brightness of an area corresponding to an image degradation area before system shutdown in an image captured after system startup and the brightness of the image degradation area before system shutdown is equal to or greater than a predetermined threshold. This threshold is adjusted by the adjustment unit according to the exposure difference. The adjustment unit adjusts the brightness difference threshold according to environmental changes, such as a change from night to day, even in cases where a conventional system would have erroneously determined that the camera abnormality had been resolved. The abnormality resolution determination unit can prevent erroneous determinations.

[0050] (5) The abnormality resolution determination unit uses the average luminance, luminance variance, or parallax in the image degradation area as a feature used to determine whether the abnormality in the imaging unit should be resolved. In this way, by using various numerical values, it becomes possible to determine whether the abnormality should be resolved from multiple perspectives.

[0051] (6) The imaging unit is a stereo camera or a monocular camera. Most vehicle-mounted cameras are stereo cameras, but the present invention can also be applied to monocular cameras used in security cameras and the like.

[0052] It should be noted that the present invention is not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to embodiments that include all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, or to add or replace other configurations.

[0053] 15 Camera (imaging unit), 100 Image processing device, 101 Abnormality determination unit, 103 Environment determination unit, 104 Abnormality cancellation determination unit, 105 Adjustment unit

Claims

1. An image processing apparatus comprising: an abnormality determination unit that determines an abnormality of an imaging unit based on a captured image; an environment determination unit that determines an environmental change using luminance information of the captured image; an abnormality cancellation determination unit that determines whether to cancel an abnormality flag of the imaging unit determined to be abnormal by the abnormality determination unit; and an adjustment unit that can adjust a determination criterion in the abnormality cancellation determination unit based on a determination result of the environment determination unit.

2. The image processing apparatus according to claim 1, wherein the environment determination unit determines the presence or absence of an environmental change by comparing the luminance of a non-image-degraded region before system termination with the luminance of a region corresponding to the non-image-degraded region before system termination in an image captured after system startup, and the adjustment unit, when it is determined that there is an environmental change, adjusts a luminance difference threshold value used by the abnormality cancellation determination unit as a criterion for determining whether to cancel the abnormality flag of the imaging unit determined to be abnormal by the abnormality determination unit according to a change ratio between the exposure amount of an image-degraded region before system termination and the exposure amount of a region corresponding to the image-degraded region before system termination in an image captured after system startup.

3. The image processing apparatus according to claim 2, wherein the abnormality cancellation determination unit determines whether to cancel the abnormality flag of the imaging unit determined to be abnormal by the abnormality determination unit by comparing the luminance of an image-degraded region before system termination with the luminance of a region corresponding to the image-degraded region before system termination in an image captured after system startup.

4. The image processing apparatus according to claim 3, wherein the abnormality cancellation determination unit cancels the abnormality flag of the imaging unit determined to be abnormal by the abnormality determination unit when the difference between the luminance of a region corresponding to the image-degraded region before system termination in an image captured after system startup and the luminance of the image-degraded region before system termination is equal to or greater than a predetermined luminance difference threshold value.

5. The image processing apparatus according to claim 1, wherein the abnormality cancellation determination unit uses any one of the average luminance, luminance dispersion, or parallax in an image-degraded region as a feature amount used for the determination to cancel the abnormality of the imaging unit.

6. The image processing apparatus according to claim 1, wherein the imaging unit is a stereo camera or a monocular camera. An image processing apparatus characterized by this.

Images