Camera Anomaly Detection Device

Abstract

This system allows for more accurate detection of abnormalities in cameras mounted on vehicles. [Solution] The camera abnormality detection device 100 determines abnormalities in the camera 20 mounted on the vehicle V. The camera abnormality detection device 100 includes an abnormality detection unit 13 that compares a reference image of a reference camera 21 with a target image of the camera to be judged 22 and determines that an abnormality has occurred in the camera to be judged 22. The abnormality detection unit 13 determines that an abnormality has occurred in the camera to be judged 22 if the color components of the reference image and the color components of the image to be judged deviate by a color component threshold or more.

Description

Technical Field

[0002] , ,

[0001] The present disclosure relates to a camera abnormality determination device that determines an abnormality in a camera mounted on a vehicle.

Background Art

[0002] Vehicles are equipped with a plurality of cameras for detecting surrounding obstacles and the like. In such cameras, plastic lenses may be used as camera lenses. Such plastic lenses may discolor to yellow (yellowing) over time. In this case, yellow may strongly appear in the captured image of the camera, which may affect the detection of obstacles and the visibility of the captured image (video).

[0003] For this reason, for example, Patent Document 1 describes a device for determining such an abnormality in a camera. In this device, image data of a certain period in the past captured by the camera is stored, and when the difference from the current captured image exceeds a certain value, it is determined that an abnormality has occurred in the camera.

Prior Art Documents

Patent Documents

[0007] A camera abnormality determination device according to one aspect of the present disclosure is: [1] "A camera abnormality determination device for determining abnormalities in a camera mounted on a vehicle, comprising: an image acquisition unit that acquires captured images from each of a plurality of cameras; a reference image which is the captured image of a reference camera among the plurality of cameras; and a determination target image which is the captured image of a camera among the plurality of cameras that is subject to abnormality determination, and an abnormality determination unit that determines an abnormality in the determination target camera, wherein the abnormality determination unit determines that an abnormality has occurred in the determination target camera if the color components of the reference image and the color components of the determination target image deviate by a color component threshold or more."

[0008] The above camera abnormality detection device may also be [2] "the camera abnormality detection device according to [1] above, wherein the abnormality detection unit determines that an abnormality has occurred in the target camera when the color component of the target image is stronger than the color component of the reference image."

[0009] The camera abnormality detection device described above may also be [3] "the camera abnormality detection device described in [2] above, wherein the abnormality detection unit determines that an abnormality has occurred in the target camera if the color component of the reference image is within a predetermined white area and the color component of the target image is within a predetermined yellow area."

[0010] The above camera abnormality detection device may also be the camera abnormality detection device described in [3] above, further comprising a cumulative temperature calculation unit that calculates a cumulative temperature obtained by accumulating the temperature of the camera to be determined at predetermined time intervals, wherein the abnormality determination unit determines that an abnormality has occurred in the camera to be determined if the cumulative temperature is equal to or greater than a cumulative temperature threshold, the color components of the reference image are within a predetermined white area, and the color components of the image to be determined are within a predetermined yellow area.

[0011] The above camera abnormality detection device may also be [5] "the camera abnormality detection device according to any one of [1] to [4] above, wherein the camera lens of the reference camera is made of glass and the camera lens of the camera to be determined is made of plastic." [Effects of the Invention]

[0012] According to one aspect of this disclosure, abnormalities in a camera mounted on a vehicle can be detected with greater accuracy. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing an example of a camera anomaly detection device according to an embodiment. [Figure 2] Figure 2 is a top view of the vehicle illustrating the reference camera and the camera to be judged that are mounted on the vehicle. [Figure 3] Figure 3 shows the color coordinates. [Figure 4] Figure 4 is a flowchart showing the flow of the camera anomaly detection process performed in the camera anomaly detection device. [Modes for carrying out the invention]

[0014] The following describes exemplary embodiments with reference to the drawings. In each drawing, identical or equivalent elements are denoted by the same reference numerals, and redundant explanations are omitted.

[0015] As shown in Figure 1, the camera anomaly detection device 100 is mounted on the vehicle V. The camera anomaly detection device 100 detects anomalies in the camera 20 mounted on the vehicle V. The vehicle V is equipped with multiple cameras 20. The cameras 20 include a reference camera 21 and a camera to be judged 22. The camera anomaly detection device 100 determines an anomaly in the camera to be judged 22 based on the image captured by the reference camera 21 and the image captured by the camera to be judged 22.

[0016] As shown in Figure 2, the multiple cameras 20 include a left rear camera 21A that captures the area behind the left side of the vehicle V, and a right rear camera 21B that captures the area behind the right side of the vehicle V, for the purpose of detecting obstacles around the vehicle V. In addition, the multiple cameras 20 may include a camera that captures the area in front of the vehicle V, a camera that captures the area behind the vehicle V, a camera that captures the area in front of the left side of the vehicle V, and a camera that captures the area in front of the right side of the vehicle V, for the purpose of detecting obstacles around the vehicle V. The multiple cameras 20 may also include a camera for an electronic rearview mirror that displays the image of the area behind the vehicle V on an electronic rearview mirror or the like.

[0017] Furthermore, the multiple cameras 20 may include a front camera 22A that captures images in front of the vehicle V, a left-side camera 22B that captures images to the left of the vehicle V, a right-side camera 22C that captures images to the right of the vehicle V, and a rear camera 22D that captures images behind the vehicle V, as cameras for a panoramic view monitor (PVM) that stitches together images of the area around the vehicle V and displays them on an in-vehicle display. The reference camera 21 and the camera to be judged 22 are selected from among the multiple cameras 20 mounted on the vehicle V.

[0018] Note that the imaging ranges of the reference camera 21 and the camera 22 to be determined partially overlap, and they can each image the common area. For example, in the example shown in FIG. 2, as an example, the left rear camera 21A may be used as the reference camera 21, and the left side camera 22B may be used as the camera 22 to be determined. The left rear camera 21A and the camera 22 to be determined can each image the common area R. However, the cameras 20 selected as the reference camera 21 and the camera target 22 to be determined are not limited to the left rear camera 21A and the camera 22 to be determined.

[0019] Also, as the reference camera 21, a camera with a camera lens formed of glass is selected. As the camera 22 to be determined, a camera with a camera lens formed of plastic is selected. When the camera lens is formed of glass, the glass camera lens does not yellow. On the other hand, when the camera lens is formed of plastic, the plastic camera lens yellows over time. Note that it has been confirmed in advance that the imaging image of the reference camera 21 equipped with a glass camera lens is not yellowed. Also, it is guaranteed that the glass camera lens of the reference camera 21 does not yellow when mounted on the vehicle V.

[0020] Furthermore, a temperature sensor 22a is provided in the camera 22 to be determined. The temperature sensor 22a detects the temperature of the camera 22 to be determined (the temperature inside the camera 22 to be determined). The temperature sensor 22a can detect the temperature around the camera lens provided in the camera 22 to be determined.

[0021] As shown in FIG. 1, the camera abnormality determination device 100 includes an ECU [Electronic Control Unit] 10 that comprehensively manages the device. The ECU 10 is an electronic control unit having a CPU [Central Processing Unit] and a storage unit. The storage unit is composed of, for example, a ROM [Read Only Memory], a RAM [Random Access Memory], an EEPROM [Electrically Erasable Programmable Read-Only Memory], and the like. In the ECU 10, for example, various functions are realized by executing a program stored in the storage unit with the CPU. The ECU 10 may be composed of a plurality of electronic control units.

[0022] A plurality of cameras 20 including a reference camera 21 and a determination target camera 22 are connected to the ECU 10. Functionally, the ECU 10 includes an image acquisition unit 11, an accumulated temperature calculation unit 12, and an abnormality determination unit 13. The image acquisition unit 11 acquires captured images from the plurality of cameras 20 respectively. The captured images acquired by the image acquisition unit 11 include the captured image captured by the reference camera 21 and the captured image captured by the determination target camera 22. Hereinafter, the captured image of the reference camera 21 is referred to as a reference image. The captured image of the determination target camera 22 is referred to as a determination target image. The image acquisition unit 11 may acquire only the reference image of the reference camera 21 and the determination target image of the determination target camera 22 among the captured images of the plurality of cameras 20.

[0023] The accumulated temperature calculation unit 12 calculates an accumulated temperature obtained by accumulating the temperature of the determination target camera 22 every predetermined time (for example, every hour) using the detection result of the temperature sensor 22a. The accumulated temperature may be a value obtained by adding temperatures, or may be a value obtained by adding the sensor values of the temperature sensor 22a. The accumulated temperature calculation unit 12 may calculate the accumulated temperature when power is supplied to the determination target camera 22 and imaging is possible. Or, the accumulated temperature calculation unit 12 may calculate the accumulated temperature at all times.

[0024] The abnormality detection unit 13 compares a reference image from a reference camera 21, which is a reference among the multiple cameras 20, with a target image from a target camera 22, which is the camera that is subject to abnormality detection among the multiple cameras 20, and determines that an abnormality has occurred in the target camera 22. The abnormality detection unit 13 determines that an abnormality has occurred in the target camera 22 if the color components of the reference image and the color components of the target image deviate by more than a color component threshold.

[0025] The anomaly detection unit 13 can use the average color of the reference image as the color component of the reference image. The anomaly detection unit 13 can use the average color of the image to be judged as the color component of the image to be judged. This color component can be represented using color coordinate values. For example, the color component (color coordinate value) can be represented using the color coordinates shown in Figure 3. As an example, in Figure 3, the horizontal axis represents red (R) and green (G), with red becoming stronger as you move to the right on the horizontal axis and green becoming stronger as you move to the left on the horizontal axis. Also in Figure 3, the vertical axis represents blue (B) and green (G), with blue becoming stronger as you move upwards on the vertical axis and green becoming stronger as you move downwards on the vertical axis.

[0026] Furthermore, when the abnormality detection unit 13 determines an abnormality, it may use an image of the common area (common area image) from the reference image of the reference camera 21 that overlaps with the imaging range of the camera to be judged 22. Alternatively, when the abnormality detection unit 13 determines an abnormality, it may use an image of the common area R (common area R image) from the image to be judged of the camera to be judged 22 that overlaps with the imaging range of the reference camera 21. In this case, the reference image and the image to be judged will be images of the same area (common area R) captured by each other.

[0027] Therefore, if the camera lens of the camera to be judged 22 is not yellowed and there is no abnormality in the camera to be judged 22, the reference image of the reference camera 21 and the target image of the camera to be judged 22 will have the same (or nearly the same) color components. For example, the color components of the reference image and the target image taken during the daytime will be within the white area A1 in the color coordinates shown in Figure 3. The white area A1 is an area within a predetermined range that includes white in the color coordinates. The color components of the reference image and the target image taken when the sun is setting will be within the yellow area A2 in the color coordinates shown in Figure 3. The yellow area A2 is an area within a predetermined range that includes yellow in the color coordinates. The white area A1 and the yellow area A2 do not overlap with each other. The white area A1 and the yellow area A2 are separated by a predetermined color component threshold or more.

[0028] For example, the color components of an image of a daytime blue sky fall within the blue area A3 in the color coordinates shown in Figure 3. The blue area A3 is a predetermined area in the color coordinates that includes blue. For example, the color components of an image of trees fall within the green area A4 in the color coordinates shown in Figure 3. The green area A4 is a predetermined area in the color coordinates that includes green.

[0029] The abnormality detection unit 13 determines that an abnormality has occurred in the target camera 22 if the color component of the image to be judged has a stronger yellow component than the color component of the reference image. Specifically, as shown in Figure 3, the abnormality detection unit 13 determines that an abnormality has occurred in the target camera 22 if the color coordinate P1 of the color component of the reference image is within the white area A1 and the color coordinate P2 of the color component of the image to be judged is within the yellow area A2.

[0030] In other words, normally, the color coordinates of the color components of the reference image, which is an image taken during the daytime, and the color coordinates of the color components of the image to be judged are both located within the white area A1. However, if the camera lens of the camera to be judged 22 yellows, the image to be judged will appear strongly yellow. Therefore, the abnormality detection unit 13 uses the reference image of the reference camera 21 as a reference and determines the abnormality of the camera to be judged 22 based on the difference between the color components of the reference image and the color components of the image to be judged.

[0031] Furthermore, when performing a determination, the abnormality determination unit 13 may perform the determination by assuming that the color components of the reference image are located within the white area A1, even if the color components of the reference image are within the blue area A3. In addition, the abnormality determination unit 13 can determine that an abnormality has occurred in the target camera 22 if it detects that the color coordinate P1 of the color components of the reference image is within the white area A1 and the color coordinate P2 of the color components of the image to be determined is within the yellow area A2 for a predetermined period of time or longer.

[0032] Here, the yellowing of the camera lens of the camera 22 being judged is correlated with the camera temperature and elapsed time. Therefore, the abnormality determination unit 13 can determine an abnormality in the camera 22 being judged by also considering the cumulative temperature calculated by the cumulative temperature calculation unit 12. Specifically, the abnormality determination unit 13 determines that an abnormality has occurred in the camera 22 being judged if the cumulative temperature is above the cumulative temperature threshold, the color component of the reference image is within the white area A1, and the color component of the image being judged is within the yellow area A2. In this case, the abnormality determination unit 13 can determine that an abnormality has occurred in the camera 22 being judged if the condition in which the color coordinate P1 of the color component of the reference image is within the white area A1 and the color coordinate P2 of the color component of the image being judged is within the yellow area A2 continues for a predetermined time or longer, and furthermore, the cumulative temperature is above the cumulative temperature threshold.

[0033] Next, the flow of the camera abnormality detection process performed by the camera abnormality detection device 100 will be explained. The process shown in Figure 4 starts when imaging is started by the reference camera 21 and the camera to be judged 22. As shown in Figure 4, the image acquisition unit 11 acquires a reference image from the reference camera 21 and an image to be judged from the camera to be judged 22 (S101). The abnormality detection unit 13 calculates the color components of the reference image and calculates the color components of the image to be judged (S102).

[0034] The abnormality determination unit 13 determines whether the condition in which the color coordinates of the color components of the reference image are within the white area A1 and the color coordinates of the color components of the image to be judged are within the yellow area A2 has continued for a predetermined time or longer (S103). If it has continued for a predetermined time or longer (S103: YES), the abnormality determination unit 13 determines whether the cumulative temperature calculated by the cumulative temperature calculation unit 12 is equal to or greater than the cumulative temperature threshold (S104).

[0035] If it is determined in S103 that the period has not lasted longer than the predetermined time (S103:NO), or if it is determined in S104 that the cumulative temperature is not equal to or greater than the cumulative temperature threshold (S104:NO), the abnormality determination unit 13 determines that the camera 22 to be judged is normal (S105). After the process in S105, the camera abnormality determination device 100 acquires the next captured reference image and the image to be judged in S101, and repeats the process described above.

[0036] If the cumulative temperature is determined to be equal to or greater than the cumulative temperature threshold in S104 (S104: YES), the abnormality determination unit 13 determines that an abnormality has occurred in the target camera 22 (S106). If an abnormality has been determined in the target camera 22, the camera abnormality determination device 100 can notify the occupants of the vehicle V that an abnormality has occurred in the target camera 22.

[0037] As described above, when the camera anomaly detection device 100 determines an anomaly in the target camera 22, which uses a plastic camera lens, it uses a reference image from a reference camera 21, which uses a glass camera lens, as a reference. The anomaly detection unit 13 of the camera anomaly detection device 100 compares the reference image from the reference camera 21 with the target image from the target camera 22. The anomaly detection unit 13 determines that an anomaly has occurred in the target camera 22 if the color components of the reference image and the color components of the target image deviate by more than a color component threshold. As a result, the camera anomaly detection device 100 can suppress the effects of sunlight and other factors and determine anomalies in the target camera 22 with greater accuracy.

[0038] The abnormality detection unit 13 determines that an abnormality has occurred in the target camera 22 if the color component of the image to be judged is stronger in the yellow component than the color component of the reference image. Furthermore, the abnormality detection unit 13 determines that an abnormality has occurred in the target camera 22 if the color component of the reference image is within the white area A1 and the color component of the image to be judged is within the yellow area A2. This allows the camera abnormality detection device 100 to more accurately determine if yellowing has occurred in the camera lens (plastic camera lens) of the target camera 22.

[0039] The abnormality detection unit 13 further considers the cumulative temperature of the camera 22 to be detected and determines whether the camera 22 is abnormal. As a result, the camera abnormality detection device 100 can determine whether the camera 22 is abnormal with greater accuracy by considering the correlation between the yellowing of the camera lens of the camera 22 and the temperature and elapsed time of the camera 22.

[0040] Although embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above. For example, the abnormality determination unit 13 may change the cumulative temperature threshold, which is compared with the cumulative temperature of the camera to be determined 22, according to the state of the camera to be determined 22. For example, when the rate of change of the temperature of the camera to be determined 22 is large (when there is a large fluctuation), the abnormality determination unit 13 may make the cumulative temperature threshold smaller than when the rate of change of the temperature is small. In other words, it may be made easier to determine that there is an abnormality. [Explanation of Symbols]

[0041] 11...Image acquisition unit, 12...Cumulative temperature calculation unit, 13...Anomaly detection unit, 20...Camera, 21...Reference camera, 22...Target camera for judgment, 100...Camera anomaly detection device, V...Vehicle.

Claims

1. A camera malfunction detection device that determines abnormalities in a camera mounted on a vehicle, An image acquisition unit that acquires captured images from each of the multiple cameras, An abnormality determination unit compares a reference image, which is the captured image of a reference camera among the multiple cameras, with a target image, which is the captured image of a target camera among the multiple cameras that is subject to abnormality determination, and determines an abnormality in the target camera. Equipped with, The abnormality determination unit determines that an abnormality has occurred in the target camera when the color components of the reference image and the color components of the target image deviate by a color component threshold or more.

2. The camera abnormality detection device according to claim 1, wherein the abnormality detection unit determines that an abnormality has occurred in the target camera if the color component of the target image is stronger in yellow than the color component of the reference image.

3. The camera abnormality determination device according to claim 2, wherein the abnormality determination unit determines that an abnormality has occurred in the target camera if the color components of the reference image are within a predetermined white area and the color components of the target image are within a predetermined yellow area.

4. The system further includes a cumulative temperature calculation unit that calculates a cumulative temperature by accumulating the temperature of the camera to be judged at predetermined time intervals, The camera abnormality determination device according to claim 3, wherein the abnormality determination unit determines that an abnormality has occurred in the target camera when the cumulative temperature is equal to or greater than the cumulative temperature threshold, the color component of the reference image is within the white area, and the color component of the target image is within the yellow area.

5. The camera lens of the aforementioned reference camera is made of glass. The camera abnormality detection device according to any one of claims 1 to 4, wherein the camera lens of the camera to be determined is made of plastic.

Images