Image monitoring device and image monitoring method

Abstract

To provide an image monitoring device and an image monitoring method, each enabling suppression of such a problem that contamination removal of a lens of an on-vehicle camera is erroneously detected.SOLUTION: An image monitoring device according to the present disclosure, comprises: an acquisition circuit; a detection circuit; an extraction circuit; and a determination circuit. The acquisition circuit acquires a captured image of an outside of a vehicle imaged by an imaging device. The detection circuit detects a contaminated region which is a region corresponding to contamination adhered to a lens of the imaging device from the captured image acquired by the acquisition circuit. The extraction circuit extracts an unclear region which is a region of an unclear image from the captured image. The determination circuit determines whether or not the contamination is irradiated by an external light source on the basis of a change of an illumination value of the unclear region. Then, the determination circuit determines that the contamination is irradiated by the external light source when a pixel of an illumination value of a first threshold value or more is a second threshold value or more in the unclear region.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to an image monitoring device and an image monitoring method. [Background technology]

[0002] In the existing technology, a vehicle grasps the situation around the vehicle based on an image captured by an on-board camera. Then, the vehicle executes various processes according to the grasped situation. If the lens of the on-board camera is dirty, it is difficult for the vehicle to grasp the situation around the vehicle. Therefore, a technology is known that determines whether or not the lens of the on-board camera is dirty based on an image captured by the on-board camera.

[0003] In addition, when dirt adhering to the lens of the vehicle-mounted camera is removed, the vehicle releases the state in which the dirt is attached to the lens. After detecting the attachment of dirt, the vehicle determines that the dirt has been removed (cleaned) if the brightness value of the dirt position in the image captured by the vehicle-mounted camera is high. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-052237 [Patent Document 2] JP 2016-040532 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when a light source is present, the image may have high brightness values ​​even if the lens has not been cleaned (cleaned), causing the vehicle to determine that the lens has been cleaned, even though the dirt has not been removed.

[0006] The present disclosure provides an image monitoring device and an image monitoring method that can suppress erroneous detection of removal of dirt from the lens of an in-vehicle camera. [Means for solving the problem]

[0007] The image monitoring device according to the present disclosure includes an acquisition circuit, a detection circuit, an extraction circuit, and a determination circuit. The acquisition circuit acquires an image of the outside of a vehicle captured by an imaging device. The detection circuit detects a dirt area, which is an area corresponding to dirt attached to the lens of the imaging device, from the captured image acquired by the acquisition circuit. The extraction circuit extracts an unclear area, which is an area of ​​an unclear image, from the captured image. The determination circuit determines whether the dirt is illuminated by an external light source based on a change in luminance value of the unclear area. Then, the determination circuit determines that the dirt is illuminated by an external light source if the number of pixels in the unclear area with a luminance value equal to or greater than a first threshold is equal to or greater than a second threshold. Effect of the Invention

[0008] According to the image monitoring device and image monitoring method disclosed herein, erroneous detection of removal of dirt from the lens of an in-vehicle camera can be suppressed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a vehicle equipped with an in-vehicle device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a configuration in the vicinity of a driver's seat of a vehicle according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a hardware configuration of the vehicle-mounted camera according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of a functional configuration of the image processing unit according to the first embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the first captured image. [Figure 6] FIG. 6 is a diagram showing an example of the second captured image. [Figure 7] FIG. 7 is a diagram showing an example of the third captured image. [Figure 8]FIG. 8 is a diagram showing an example of the fourth captured image. [Figure 9] FIG. 9 is a diagram showing an example of the first extracted image. [Figure 10] FIG. 10 is a diagram showing an example of the second extracted image. [Figure 11] FIG. 11 is a diagram showing an example of the third extracted image. [Figure 12] FIG. 12 is a diagram showing an example of the fourth extracted image. [Figure 13] FIG. 13 is a flowchart showing an example of the removal determination process executed by the image processing unit according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of an image monitoring device and an image monitoring method according to the present disclosure will be described with reference to the drawings.

[0011] (First embodiment) 1, the vehicle 1 includes a vehicle body 12 and two pairs of wheels 13 arranged along a predetermined direction on the vehicle body 12. The two pairs of wheels 13 include a pair of front tires 13f and a pair of rear tires 13r.

[0012] 1 includes four wheels 13, the number of wheels 13 is not limited to this. For example, the vehicle 1 may be a two-wheeled vehicle.

[0013] The vehicle body 12 is coupled to wheels 13 and can move by the wheels 13. In this case, the predetermined direction in which the two pairs of wheels 13 are arranged is the traveling direction of the vehicle 1. The vehicle 1 can move forward or backward by switching gears (not shown) or the like. The vehicle 1 can also turn right or left by steering.

[0014] The vehicle body 12 has a front end F, which is an end forward of the front tire 13f, and a rear end R, which is an end rearward of the rear tire 13r. The vehicle body 12 has a substantially rectangular shape when viewed from above, and the four corners of the substantially rectangular shape may be referred to as ends. Although not shown in FIG. 1, the vehicle 1 also includes a display device, a speaker, and an operation unit.

[0015] A pair of bumpers 14 are provided at the front and rear ends F, R of the vehicle body 12 near the lower end of the vehicle body 12. Of the pair of bumpers 14, the front bumper 14f covers the entire front surface and part of the side surface near the lower end of the vehicle body 12. Of the pair of bumpers 14, the rear bumper 14r covers the entire rear surface and part of the side surface near the lower end of the vehicle body 12.

[0016] At predetermined ends of the vehicle body 12, wave transmitting / receiving units 15f, 15r for transmitting and receiving sound waves such as ultrasonic waves are disposed. For example, one or more wave transmitting / receiving units 15f are disposed at the front bumper 14f, and one or more wave transmitting / receiving units 15r are disposed at the rear bumper 14r. Hereinafter, when the wave transmitting / receiving units 15f, 15r are not limited, they are simply referred to as the wave transmitting / receiving unit 15. In addition, the number and positions of the wave transmitting / receiving units 15 are not limited to the example shown in FIG. 1. For example, the vehicle 1 may be provided with wave transmitting / receiving units 15 on the left and right sides.

[0017] In this embodiment, a sonar using ultrasonic waves is taken as an example of the wave transmitting and receiving unit 15, but the wave transmitting and receiving unit 15 may be a radar that transmits and receives electromagnetic waves. Alternatively, the vehicle 1 may be equipped with both a sonar and a radar. Also, the wave transmitting and receiving unit 15 may simply be called a sensor.

[0018] The wave transmitting / receiving unit 15 detects obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves. In addition, the wave transmitting / receiving unit 15 measures the distance between the vehicle 1 and obstacles around the vehicle 1 based on the results of transmitting and receiving sound waves or electromagnetic waves.

[0019] Vehicle 1 also includes a first vehicle-mounted camera 16a that captures an image in front of vehicle 1, a second vehicle-mounted camera 16b that captures an image behind vehicle 1, a third vehicle-mounted camera 16c that captures an image on the left side of vehicle 1, and a fourth vehicle-mounted camera that captures an image on the right side of vehicle 1. The fourth vehicle-mounted camera is not shown in the figure.

[0020] Hereinafter, when there is no need to distinguish between the first vehicle-mounted camera 16a, the second vehicle-mounted camera 16b, the third vehicle-mounted camera 16c, and the fourth vehicle-mounted camera, they will simply be referred to as vehicle-mounted cameras 16. The positions and the number of vehicle-mounted cameras 16 are not limited to the example shown in FIG. 1. For example, the vehicle 1 may also be equipped with two vehicles, the first vehicle-mounted camera 16a and the second vehicle-mounted camera 16b. Alternatively, the vehicle 1 may have other vehicle-mounted cameras in addition to the above-mentioned examples.

[0021] The vehicle-mounted camera 16 is capable of capturing images of the surroundings of the vehicle 1, and is, for example, a camera that captures color images. Note that the image data captured by the vehicle-mounted camera 16 may be video or still images. The vehicle-mounted camera 16 may be a camera built into the vehicle 1, or may be a camera of a drive recorder that is retrofitted to the vehicle 1.

[0022] The vehicle 1 is also equipped with an in-vehicle device 100. The in-vehicle device 100 is an information processing device that can be installed in the vehicle 1, and is, for example, an ECU (Electronic Control Unit) or an OBU (On Board Unit) provided inside the vehicle 1. Alternatively, the in-vehicle device 100 may be an external device installed near the dashboard of the vehicle 1. The in-vehicle device 100 may also function as a car navigation device or the like. Image data captured by the in-vehicle camera 16 may be input to the ECU and used, for example, to control automatic braking.

[0023] Next, a configuration in the vicinity of the driver's seat of the vehicle 1 according to this embodiment will be described.

[0024] 2, the vehicle 1 includes a driver's seat 130a and a passenger seat 130b. In front of the driver's seat 130a, a windshield 180, a dashboard 190, a steering wheel 140, a display device 120, and operation buttons 141 are provided.

[0025] The display device 120 is a display provided on a dashboard 190 of the vehicle 1. As an example, the display device 120 is located in the center of the dashboard 190 as shown in FIG. 2. The display device 120 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The display device 120 may also function as a touch panel. The display device 120 is an example of a display unit in this embodiment.

[0026] The steering wheel 140 is provided in front of the driver's seat 130a and can be operated by the driver. The rotation angle of the steering wheel 140, for example, the steering angle, is electrically or mechanically linked to the change in the direction of the front tires 13f, which are the steered wheels. The steered wheels may be the rear tires 13r, or both the front tires 13f and the rear tires 13r may be steered wheels.

[0027] The operation button 141 is, for example, a button capable of accepting an operation for cleaning the lens of the vehicle-mounted camera 16 by a user. In this embodiment, the user is, for example, the driver of the vehicle 1. The position of the operation button 141 is not limited to the example shown in FIG. 2, and may be provided on the steering wheel 140, for example. The operation button 141 is an example of an operation unit in this embodiment. In addition, when the display device 120 also functions as a touch panel, the display device 120 may be an example of an operation unit. In addition, an operation terminal capable of transmitting a signal to the vehicle 1 from outside the vehicle 1, such as a tablet terminal, a smartphone, a remote controller, or an electronic key (not shown), may be an example of an operation unit.

[0028] Next, the hardware configuration of the vehicle-mounted camera 16 of this embodiment will be described.

[0029] As shown in FIG. 3, the vehicle-mounted camera 16 includes a lens 161, an imaging element 162, a cleaning unit 163, a video signal processing unit 164, an exposure control unit 165, an image processing unit 166, and an image memory 167.

[0030] The lens 161 is made of a transparent material and serves to diverge or converge the incident light.

[0031] The imaging element 162 is an image sensor such as a complementary metal-oxide semiconductor (CMOS) or a charge coupled device (CCD). The imaging element 162 receives light that has passed through the lens 161 and converts it into a video signal. The imaging element 162 and the lens 161 are collectively referred to as a photographing device.

[0032] The cleaning unit 163 is a device that cleans dirt adhering to the lens 161 by spraying water or the like onto the lens 161 .

[0033] Video signal processing unit 164 generates an image based on the video signal output from imaging element 162. Exposure control unit 165 controls the brightness of the image generated by video signal processing unit 164. For example, video signal processing unit 164 generates an image with a brightness controlled by exposure control unit 165. For example, when the image is dark, exposure control unit 165 increases the brightness of the image. On the other hand, when the image is bright, exposure control unit 165 decreases the brightness of the image.

[0034] The image processing unit 166 executes various types of image processing on the image generated by the video signal processing unit 164. The image memory 167 is a main storage device of the image processing unit 166. The image memory 167 is used as a working memory for the image processing by the image processing unit 166.

[0035] The image processing unit 166 is configured with a computer or the like, and controls image processing by cooperation between hardware and software. For example, the image processing unit 166 includes a processor 166A, a RAM (Random Access Memory) 166B, a memory 166C, and an I / O (Input / Output) interface 166D.

[0036] The processor 166A is, for example, a central processing unit (CPU) capable of executing a computer program. Note that the processor 166A is not limited to a CPU. For example, the processor 166A may be a digital signal processor (DSP) or another processor.

[0037] The RAM 166B is a volatile memory used as a cache or a buffer. The memory 166C is a non-volatile memory that stores various information including computer programs. The processor 166A realizes various functions by reading a specific computer program from the memory 166C and expanding it in the RAM 166B.

[0038] The I / O interface 166D controls input and output of the image processing unit 166. For example, the I / O interface 166D executes communication with the video signal processing unit 164, the image memory 167, and the in-vehicle device 100.

[0039] The cleaning unit 163 may be an independent device, not integrally formed with the vehicle-mounted camera 16. At least a part of the video signal processing unit 164, the exposure control unit 165, the image processing unit 166, and the image memory 167 may be included in the ECU or OBU, not limited to the vehicle-mounted camera 16, or may be provided in the vehicle-mounted device 100, or may be an independent device, or may be incorporated in another device.

[0040] Next, functions of the image processing unit 166 according to the first embodiment will be described.

[0041] 4 realizes various functions by reading out a specific computer program from memory 166C and loading it into RAM 166B. More specifically, image processing unit 166 includes image acquisition unit 1661, monitoring control unit 1662, dirt detection unit 1663, dirt notification unit 1664, determination unit 1665, change detection unit 1666, unclear region extraction unit 1667, light source determination unit 1668, and cleaning control unit 1669.

[0042] The image acquisition unit 1661 acquires captured images of the outside of the vehicle 1 captured by the in-vehicle camera 16. The image acquisition unit 1661 is an example of an acquisition circuit. The in-vehicle camera 16 is an example of an imaging device. For example, the image acquisition unit 1661 acquires a first captured image G11 (see FIG. 5), a second captured image G12 (see FIG. 6), a third captured image G13 (see FIG. 7), and a fourth captured image G14 (see FIG. 8) as the captured images.

[0043] For example, when dirt is attached to the lens 161 of the vehicle-mounted camera 16, the image acquisition unit 1661 acquires a first captured image G11 having a dirt area G113. As shown in FIG. 5, the first captured image G11 has a non-imaged area G111 and an imaged area G112. The non-imaged area G111 is a detection area of ​​the image sensor 162, but is an area where the outside of the vehicle 1 is not captured due to the housing of the vehicle-mounted camera 16 or the like. The imaged area G112 is an area where the outside of the vehicle 1 is captured by light incident through the lens 161. The imaged area G112 may have a dirt area G113 where dirt such as mud attached to the lens 161 is captured.

[0044] Here, the vehicle-mounted camera 16 is focused on the front, not on the lens 161. Therefore, when dirt such as mud adheres to the lens 161, the vehicle-mounted camera 16 captures an image in which the portion corresponding to the periphery of the dirt is blurred. As a result, the dirt area G113 becomes an area in which the difference in brightness value between each pixel is small near the center because visible light is blocked by the dirt, and becomes an unclear area near the periphery because it is not in focus. For example, the area near the periphery of the dirt area G113 becomes an area in which the gradient of the change in brightness value is gentle.

[0045] Furthermore, when dirt is attached to the lens 161 of the vehicle-mounted camera 16 and a light source is located at a position overlapping the dirt, the image acquisition unit 1661 acquires a second captured image G12 having a dirt area G113 overlapping the light source area G114. As shown in FIG. 6, the second captured image G12 has a non-captured area G111 and an imaged area G112. Furthermore, the imaged area G112 has a dirt area G113 overlapping the light source area G114. The light source area G114 is an area captured by a light source such as the sun, the moon, an electric lamp, or a light. The light source area G114 is an area where the difference in luminance value of each pixel is small near the center due to visible light emitted by the light source, and the periphery is an unclear area where the luminance value decreases as the distance from the center increases. Furthermore, when the vehicle-mounted camera 16 has a light source that irradiates visible light, the luminance value of each pixel of the dirt area G113 increases due to scattering of the visible light irradiated by the light source.

[0046] Furthermore, when dirt is attached to the lens 161 of the vehicle-mounted camera 16 and the light source is located at a position that does not overlap with the dirt, the image acquisition unit 1661 acquires a third captured image G13 having a light source region G114 and a dirt region G113. As shown in FIG. 7, the third captured image G13 has a non-captured region G111 and an imaged region G112. Furthermore, the imaged region G112 has a light source region G114 and a dirt region G113. Also in this case, since the vehicle-mounted camera 16 has a light source that irradiates visible light, the luminance value of each pixel in the dirt region G113 becomes high due to scattering of the visible light irradiated by the light source.

[0047] Furthermore, when there is no dirt on the lens 161 of the vehicle-mounted camera 16 and there is no light source for the vehicle-mounted camera 16, the image acquisition unit 1661 acquires a fourth captured image G14. As shown in Fig. 8, the fourth captured image G14 has a non-captured area G111 and an captured area G112. Furthermore, since there is no dirt on the captured area G112, there is no dirty area G113. Furthermore, since there is no light source in the captured area G112, there is no light source area G114.

[0048] The monitoring control unit 1662 monitors the outside of the vehicle 1 based on the image acquisition unit 1661. For example, the monitoring control unit 1662 detects other vehicles and pedestrians traveling outside the vehicle 1. Then, the vehicle 1 executes various processes according to the detection results.

[0049] The dirt detection unit 1663 detects a dirt area G113, which is an area corresponding to dirt adhering to the lens 161 of the vehicle-mounted camera 16, from the captured image acquired by the image acquisition unit 1661. The dirt detection unit 1663 is an example of a detection circuit.

[0050] The dirt detection unit 1663 detects the dirty area G113 by various methods. For example, the dirt detection unit 1663 detects dirt adhering to the lens 161 of the vehicle-mounted camera 16 by a process similar to that of the lens dirt discrimination device shown in FIG. 1 of Patent Document 2.

[0051] The dirt detection unit 1663 may detect dirt by other methods. Here, when dirt such as mud adheres to the lens 161 of the in-vehicle camera 16, the image sensor 162 has difficulty receiving visible light due to the dirt adhered to the lens 161. Therefore, the brightness value of the image corresponding to the part where the dirt adheres is low. For example, the image corresponding to the part where the dirt adheres becomes a flat area where the difference in the brightness value of each pixel is small. Therefore, the dirt detection unit 1663 determines that the lens 161 is dirty based on the ratio of the flat area where the difference in the brightness value of each pixel is small in the captured image. Then, the dirt detection unit 1663 detects the flat area where the difference in the brightness value of each pixel is small in the captured image as the dirty area G113.

[0052] When the dirt detection unit 1663 detects the dirt area G113, the dirt notification unit 1664 notifies that dirt is attached to the lens 161 of the vehicle-mounted camera 16. More specifically, the dirt notification unit 1664 may notify that dirt is attached to the lens 161 of the vehicle-mounted camera 16 from the time when the dirt detection unit 1663 detects the dirt area G113 until the determination unit 1665 determines that the dirt has been removed (cleaned).

[0053] For example, the dirt notification unit 1664 may notify the display device 120 or the like that dirt is attached to the lens 161 of the in-vehicle camera 16 via the ECU or OBU. The dirt notification unit 1664 may notify the display device 120 or the like by sound, by illuminating an LED or the like, or by other methods, without being limited to the display device 120. The dirt notification unit 1664 may notify the display device 120 or the like that dirt is attached to the lens 161 directly, without being via the ECU or OBU.

[0054] The determination unit 1665 determines whether or not the dirt adhering to the lens 161 of the vehicle-mounted camera 16 detected by the dirt detection unit 1663 is illuminated by the light source.

[0055] Here, if the lens 161 of the in-vehicle camera 16 is dirty, it becomes difficult for the imaging element 162 to receive visible light due to the dirt on the lens 161. Then, if the dirt on the lens 161 is removed, the imaging element 162 becomes able to receive visible light. For example, if the dirt on the lens 161 is removed, the brightness value of the captured image becomes higher in the area that was previously the dirty area G113.

[0056] Therefore, a method of determining whether the dirt has been removed based on the luminance value of the area that was the dirty area G113 is considered. However, even if the dirt has not been removed, when visible light is irradiated from a light source, the luminance value of the dirty area G113 in the captured image becomes high as shown in Figures 6 and 7. In such a case, the in-vehicle camera 16 of the existing technology erroneously determines that the dirt has been removed even though the dirt has not been removed.

[0057] Therefore, the determination unit 1665 uses the following functions to prevent the determination unit 1665 from erroneously determining that the stain has been removed when the stain has not been removed. More specifically, the determination unit 1665 includes a change detection unit 1666, an unclear region extraction unit 1667, and a light source determination unit 1668.

[0058] The change detection unit 1666 detects a change in the luminance value of the stained region G113. For example, the change detection unit 1666 detects a change in the average value of the luminance value of each pixel of the stained region G113. For example, the change detection unit 1666 compares the average value of the luminance value at a first time point, for example, when the stain is detected, with the average value of the luminance value at a second time point, for example, when the comparison is performed. Then, the change detection unit 1666 determines whether or not the average value of the luminance value of each pixel of the stained region G113 has changed. For example, the change detection unit 1666 determines whether or not the average value of the luminance value of each pixel of the stained region G113 is higher than a threshold value. Note that the change detection unit 1666 may detect a change not only by the average value, but also by a median value, a maximum luminance value, a minimum luminance value, or another value.

[0059] The target of the detection operation of the change detection unit 1666 is not limited to the region along the edge of the stain as shown in Fig. 5 etc., but may be a rectangle, or may be a region extracted in block units separated by dotted lines in Fig. 5 etc. The change detection unit 1666 also detects a change in luminance regarding the unclear region G211 output by the unclear region extraction unit 1667 described later.

[0060] The unclear region extraction unit 1667 extracts an unclear region G211, which is a region of an unclear image, from the captured image acquired by the image acquisition unit 1661. The unclear region extraction unit 1667 is an example of an extraction circuit. For example, the unclear region extraction unit 1667 extracts an unclear region G211, which is a region of a blurred image caused by a stain attached to the lens 161 being out of focus. The unclear region G211 is a region including a peripheral portion of the stain region G113. Moreover, the unclear region G211 is a region of the stain region G113 that is out of focus.

[0061] More specifically, the unclear region extraction unit 1667 executes a filter process for extracting the unclear region G211 from the captured image. For example, the filter process is a process for extracting a region in which the gradient of the change in luminance value is gentler than that of the surrounding pixels. In this way, the unclear region extraction unit 1667 generates an extracted image in which the unclear region G211 included in the captured image is extracted.

[0062] For example, the unclear region extraction unit 1667 generates a first extracted image G21 shown in Fig. 9 by performing image processing on the first captured image G11 shown in Fig. 5. As shown in Fig. 9, the unclear region extraction unit 1667 extracts an unclear region G211 from the vicinity of the periphery of the stain region G113.

[0063] For example, the unclear region extraction unit 1667 generates the second extracted image G22 shown in Fig. 10 by executing image processing on the second captured image G12 shown in Fig. 6. Fig. 6 shows an unclear region G211 and a light-source unclear region G212 in a state where the light source region G114 overlaps with the dirt region G113. Therefore, the unclear region extraction unit 1667 extracts the unclear region G211 from the vicinity of the periphery of the dirt region G113, and extracts the light-source unclear region G212, which is an unclear region derived from the light source region G114.

[0064] In addition, since the visible light irradiated from the light source is scattered by dirt, the unclear region extraction unit 1667 extracts a wider region of the unclear region G211 shown in Fig. 10 compared to the first extracted image G21 without a light source shown in Fig. 9. In addition, the image sensor 162 generates a blurred image for the light source region G114, which is a portion irradiated with visible light. Therefore, the unclear region extraction unit 1667 extracts the light source unclear region G212 from the light source region G114, but if it is difficult to separate the light source unclear region G212 from the unclear region G211 in the dirt region G113, it is not necessary to separate the light source unclear region G212 from the unclear region G211.

[0065] For example, the unclear region extraction unit 1667 generates a third extracted image G23 shown in Fig. 11 by executing image processing on the third captured image G13 shown in Fig. 7. Fig. 7 and Fig. 11 are different from Fig. 6 and Fig. 10 in that the light source region G114 is separated from the dirt region G113. Therefore, as shown in Fig. 11, the unclear region extraction unit 1667 extracts an unclear region G211 from the vicinity of the periphery of the dirt region G113, and separates and extracts the unclear region G211 from the light source unclear region G212.

[0066] For example, the unclear region extraction unit 1667 generates a fourth extracted image G24 shown in Fig. 12 by executing image processing on the fourth captured image G14 shown in Fig. 8. As shown in Fig. 12, since there is no stain region G113 or light source region G114, the unclear region extraction unit 1667 does not extract the unclear region G211 and the light source unclear region G212. Therefore, as shown in Fig. 12, the unclear region extraction unit 1667 generates a fourth extracted image G24 without the unclear region G211 and the light source unclear region G212.

[0067] The light source determination unit 1668 determines whether or not the dirt is irradiated by an external light source based on a change in the luminance value of the captured image acquired by the image acquisition unit 1661. For example, the light source determination unit 1668 determines whether or not the luminance value of the dirt attached to the lens 161 of the vehicle-mounted camera 16 has changed due to a light source. The light source determination unit 1668 is an example of a determination circuit. For example, when the luminance value of the average value of the dirt region G113 becomes high in the information on the change in the luminance value output by the change detection unit 1666, the light source determination unit 1668 determines whether or not the change in the luminance value is due to a light source.

[0068] Here, when visible light is irradiated from a light source, the luminance value of the stained region G113 becomes high even if the stain is not removed. Here, when visible light is irradiated from a light source, the luminance value of each pixel in the unclear region G211 becomes high because the visible light is scattered by the stain. Therefore, when the change detection unit 1666 detects that there are pixels in the unclear region G211 with a luminance value equal to or higher than the first threshold value equal to or higher than the second threshold value, the light source determination unit 1668 determines that the stain is irradiated by an external light source. For example, the light source determination unit 1668 determines that the stain has not been removed and that the luminance value has increased due to the light source. For example, the second threshold is 40% of the unclear region G211, taking into account light scattering in the stain region G113. For example, the light source determination unit 1668 determines that the stain has not been removed when the unclear region G211 in the stain region G113 has 40% or more high-luminance pixels.

[0069] Also, as shown in FIG. 11, the unclear region extraction unit 1667 may extract the light source unclear region G212 from the light source region G114 outside the dirt region G113. However, the dirt detection unit 1663 determines that the unclear region G211 other than the dirt region G113 is not dirt. Note that, as shown in FIG. 10, when the light source region G114 exists near the dirt region G113 or the dirt region G113 and the light source region G114 overlap, the unclear region extraction unit 1667 extracts the unclear region G211 where the dirt region G113 and the light source region G114 overlap. In this case, the unclear region extraction unit 1667 may add a condition that the unclear region G211 in the dirt region G113 is extracted and the unclear region G211 outside the dirt region G113 is discarded. For this reason, in FIG. 10, the portion of the light source unclear region G212 other than the dirty region G113 does not need to be extracted as the unclear region G211.

[0070] More specifically, the light source determination unit 1668 counts the number of pixels in the unclear region G211 in the stained region G113 whose luminance value is equal to or greater than the first threshold value based on information on the change in the luminance value of the unclear region G211 output by the change detection unit 1666. Then, when the number of pixels in the unclear region G211 in the stained region G113 whose luminance value is equal to or greater than the first threshold value is equal to or greater than the second threshold value, the light source determination unit 1668 determines that the luminance value of the stain has increased due to the light source. Also, since the luminance of the stained region G113 is in a state in which it has been increased due to the light source, the light source determination unit 1668 activates the light source flag. Note that, when the number of pixels whose luminance value is equal to or greater than the first threshold value does not satisfy the second threshold value or greater, the light source determination unit 1668 determines that the luminance value of the stain has not increased due to the light source. This allows the light source determination unit 1668 to determine whether the change in luminance of the unclear region G211 is due to the light source or not, and thus prevents erroneous determination that the stain has been removed (cleaned).

[0071] In addition, even after it is determined that visible light is being irradiated from the light source, the luminance value of the dirt may change. Therefore, based on the information on the change in the luminance value of the dirt area G113 output by the change detection unit 1666, when the difference between the average value of the luminance value of each pixel in the dirt area G113 at the time of detection of the dirt area G113 and the average value of the luminance value of each pixel in the dirt area G113 at the current time is equal to or greater than a third threshold, the light source determination unit 1668 determines whether or not the luminance value of the dirt area G113 has changed depending on whether or not the difference between the average value of the luminance value of each pixel in the dirt area G113 at the time of detection of the dirt area G113 and the average value of the luminance value of each pixel in the dirt area G113 at the current time is equal to or greater than a third threshold. Note that the current time is the time when the average value of the luminance value of each pixel in the dirt area G113 of the captured image changes after the dirt area G113 is detected.

[0072] If the difference between the luminance value when the stain detection unit 1663 detects the stained area G113 and the luminance value when the luminance value of the captured image changes after the stained area G113 is equal to or greater than a third threshold value and the light source flag is valid (luminance has changed due to the light source), the stained area G113 is determined to be continuously irradiated with visible light from the light source. Also, if the difference between the average value of the luminance value of each pixel when the stained area G113 is detected and the average value of the luminance value of each pixel of the stained area G113 at the current time is less than the third threshold value and the light source flag is valid (determination is continued), the light source determination unit 1668 determines that the stained area G113 is not continuously irradiated with visible light from the light source.

[0073] The image processing unit 166 may periodically execute the dirt detection by the dirt detection unit 1663, and may execute the dirt detection by the dirt detection unit 1663 again when the comparison of the luminance values ​​by the light source determination unit 1668 has been performed a specified number of times. This allows the image processing unit 166 to determine a change in the luminance value caused by the natural peeling off of dirt attached to the lens 161.

[0074] The cleaning control unit 1669 controls the cleaning unit 163 to clean the lens 161 of the vehicle-mounted camera 16. For example, the cleaning control unit 1669 instructs the cleaning unit 163, which removes dirt from the lens 161 of the vehicle-mounted camera 16, to perform cleaning. The cleaning control unit 1669 is an example of a cleaning control circuit. The cleaning unit 163 is an example of a cleaning device. For example, when the dirt detection unit 1663 detects that dirt is attached to the lens 161, the dirt notification unit 1664 causes the display device 120 to display that dirt is attached to the lens 161. In response to this, an operator such as a driver inputs an operation to have the cleaning unit 163 clean the lens 161 using the operation button 141 or a touch panel that also serves as the display device 120. Then, when the cleaning control unit 1669 receives an operation to have the cleaning unit 163 clean the lens 161, the cleaning control unit 1669 causes the cleaning unit 163 to clean the lens 161. After the cleaning control unit 1669 cleans the lens 161, the image processing unit 166 may cause the dirt detection unit 1663 to perform dirt detection again.

[0075] Next, the flow of the light source determination process executed by the image processing unit 166 will be described with reference to Fig. 13. The light source determination process is a process for determining whether or not the luminance value of dirt adhering to the lens 161 of the vehicle-mounted camera 16 has changed due to the light source.

[0076] The image acquisition unit 1661 acquires an image captured by the vehicle-mounted camera 16 (step S1).

[0077] The dirt detection unit 1663 detects a dirt area G113, which is an area of ​​dirt adhering to the lens 161 of the vehicle-mounted camera 16, from the captured image (step S2).

[0078] The light source determination unit 1668 initializes the light source flag (step S3).

[0079] The unclear region extraction unit 1667 executes a filter process to extract the unclear region G211 and the light source unclear region G212 (step S4).

[0080] The light source determination unit 1668 counts the number of pixels whose luminance value is equal to or greater than the first threshold in the unclear region G211 in the stain region G113 (step S5).

[0081] The light source determination unit 1668 determines whether or not visible light is being emitted from the light source based on whether or not the number of pixels in the unclear region G211 whose luminance value is equal to or greater than the first threshold value is equal to or greater than the second threshold value (step S6).

[0082] If the number of pixels whose brightness value is equal to or greater than the first threshold is less than the second threshold (step S6; No), it is determined that the unclear area G211 in the dirty area G113 is not illuminated by the light source, and the light source determination unit 1668 remains in step S6 until it becomes equal to or greater than the second threshold.

[0083] If the number of pixels whose brightness value is equal to or greater than the first threshold is equal to or greater than the second threshold (step S6; Yes), the light source determination unit 1668 determines that the unclear area G211 in the dirty area G113 is illuminated by a light source, and sets the light source flag to active (step S7).

[0084] The light source determination unit 1668 determines whether the dirty region G113 is continuously illuminated by the light source based on whether the difference between the average brightness value of each pixel at the time of detection of the dirty region G113 and the average brightness value of each pixel of the dirty region G113 at the current time is equal to or greater than a third threshold (step S8). If the difference between the average brightness values ​​is equal to or greater than the third threshold (step S8; Yes), the light source determination unit 1668 determines that the dirty region G113 is continuously illuminated by the light source, and proceeds to step S8 again.

[0085] If the difference in the average brightness values ​​is less than the third threshold value (step S8; No), the light source determining unit 1668 sets the light source flag to invalid (step S9), and the light source determining unit 1668 proceeds to step S6.

[0086] Note that the image processing unit 166 may move to step S1 and perform stain detection again after processing the judgment of step S6 and the judgment of step S8 a specified number of times. This allows the image processing unit 166 to determine that the change in the brightness value of the stained area G113 is due to natural peeling of the stain. For example, after the Yes judgment of step S6 (the brightness of the unclear area G211 in the stained area G113 is higher than the second threshold) or after the Yes judgment of step S8 (the average brightness of the stained area G113 is higher than the third threshold), the image processing unit 166 executes stain detection and, if it is determined that there is no stain, it can determine that the brightness value has increased due to natural peeling of the stain.

[0087] Furthermore, in the determination of step S6, if the light source determination unit 1668 determines whether or not visible light is being irradiated from an external light source onto dirt adhering to the lens 161 a predetermined number of times, the cleaning control unit 1669 may instruct the cleaning unit 163 to remove the dirt from the lens 161. In this way, the cleaning control unit 1669 can remove the dirt adhering to the lens 161.

[0088] Furthermore, for example, after a No judgment in step S6 (the brightness of the unclear region G211 in the dirty region G113 is not higher than the second threshold) or after a Yes judgment in step S8 (the average brightness of the dirty region G113 is not higher than the third threshold), the image processing unit 166 performs dirt detection, and if it is judged that the position of the dirty region G113 has changed, it can be determined that the brightness value is not higher than the threshold due to natural peeling of part of the dirt.

[0089] Furthermore, in the determination of step S8, if the light source determination unit 1668 determines whether or not visible light from the external light source continues to be irradiated onto the dirt adhering to the lens 161 a predetermined number of times, the cleaning control unit 1669 may instruct the cleaning unit 163 to remove the dirt adhering to the lens 161. In this way, the cleaning control unit 1669 can remove the dirt adhering to the lens 161.

[0090] With the above, the image processing unit 166 ends the light source determination process.

[0091] As described above, the image processing unit 166 according to the first embodiment detects the dirt area G113, which is an area corresponding to dirt attached to the lens 161 of the vehicle-mounted camera 16, from the captured image acquired by the image acquisition unit 1661. The image processing unit 166 also extracts the unclear area G211, which is an area of ​​an unclear image formed by dirt attached to the lens 161 of the vehicle-mounted camera 16, from the captured image acquired by the image acquisition unit 1661. Then, the image processing unit 166 judges whether or not the dirt is irradiated by a light source based on a change in the luminance value of the captured image acquired by the image acquisition unit 1661. When the number of pixels having a luminance value equal to or greater than the first threshold value is equal to or greater than the second threshold value in the unclear area G211, the image processing unit 166 judges that the luminance value has changed due to irradiation of visible light by the light source. Therefore, the image processing unit 166 can suppress an event in which the image processing unit 166 judges that the dirt has been removed due to a change in the luminance of the dirt after detecting the adhesion of the dirt to the lens 161 of the vehicle-mounted camera 16.

[0092] Although some embodiments of the present disclosure have been described, these embodiments are presented as examples and are not intended to limit the scope of the present disclosure. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the present disclosure. These embodiments and their modifications are included in the scope and gist of the present disclosure, as well as in the scope of the present disclosure and its equivalents as described in the claims.

[0093] The order in which each step is performed in the flowchart is merely an example for specifically explaining the present disclosure, and an order other than the above may be used. In addition, some of the steps may be performed simultaneously (in parallel) with other steps, or some of the steps may not be performed.

[0094] In addition, the division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to another functional block. Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

[0095] In addition, in the above-described embodiments, the notation "... part" used for each component may be replaced with other notations such as "... circuitry," "... assembly," "... device," "... unit," or "... module." [Explanation of symbols]

[0096] 1 vehicle 15 Transmitter / receiver unit 16. In-vehicle camera 120 Display device 141 Operation Button 161 Lens 162 Image sensor 163 Cleaning Department 164 Video signal processing section 165 Exposure control section 166 Image Processing Unit 167 Image Memory 166A Processor 166B RAM (Random Access Memory) 166C Memory 166D I / O (Input / Output) Interface 1661 Image Acquisition Unit 1662 Monitoring and control section 1663 Dirt detection unit 1664 Stains Notification Department 1665 Judgment section 1666 Change detection unit 1667 Obscure region extraction part 1668 Light source determination section 1669 Cleaning control section G11 First captured image G12 Second captured image G13 3rd captured image G14 4th image G111 Non-imaging area G112 Imaging area G113 Dirt area G114 Light source area G21 First extracted image G22 Second Extracted Image G23 3rd extracted image G24 4th extracted image G211 Obscure area G212 Light source unclear area

Claims

1. an acquisition circuit for acquiring an image of the outside of the vehicle captured by the imaging device; a detection circuit that detects a dirty area corresponding to dirt adhering to a lens of the imaging device from the captured image acquired by the acquisition circuit; an extraction circuit for extracting an unclear region, which is a region of an unclear image, from the stain region; a determination circuit for determining whether the stain is illuminated by an external light source based on a change in the luminance value of the unclear region; Equipped with The determination circuit determines that the stain is illuminated by the external light source when the number of pixels having a luminance value equal to or greater than a first threshold in the unclear region is equal to or greater than a second threshold. Image surveillance equipment.

2. The unclear region is a region including a peripheral portion of the stain region.

2. The image monitoring device according to claim 1.

3. The unclear area is an area of ​​the dirt area that is out of focus.

2. The image monitoring device according to claim 1.

4. The determination circuit determines that irradiation of the dirt by the external light source is continuing when a difference between a luminance value when the dirty area is detected by the detection circuit and a luminance value when the luminance value of the captured image changes after the dirty area is detected is equal to or greater than a third threshold value.

3. An image monitoring device according to claim 1 or 2.

5. A cleaning control circuit is further provided for instructing a cleaning device for removing dirt from the lens to perform cleaning. When the determination circuit determines whether dirt is irradiated by the external light source a predetermined number of times, the cleaning control circuit instructs the cleaning device to remove dirt from the lens.

3. An image monitoring device according to claim 1 or 2.

6. A cleaning control circuit is further provided for instructing a cleaning device for removing dirt from the lens to perform cleaning. When the determination circuit determines whether or not the dirt is continuously irradiated by the external light source a predetermined number of times, the cleaning control circuit instructs the cleaning device to remove the dirt from the lens.

3. An image monitoring device according to claim 1 or 2.

7. Acquire an image of the exterior of the vehicle captured by the imaging device; Detecting a dirt area corresponding to dirt adhering to a lens of the imaging device from the captured image; Extracting an unclear region, which is a region of an unclear image, from the stain region; determining whether the stain is illuminated by an external light source based on a change in luminance value of the unclear region; When the number of pixels having a luminance value equal to or greater than the first threshold in the unclear region is equal to or greater than the second threshold, it is determined that the stain is illuminated by the external light source. Image monitoring method.

Images