Imaging device

Abstract

In the imaging device, the influence of surrounding components on the forward-facing camera is determined. [Solution] The imaging device 1 comprises a forward-facing camera 2, a common area determination unit 31, and a window fogging determination unit 34. The forward-facing camera 2 photographs the area in front of the vehicle from inside the vehicle via the front windshield 5, including at least a part of the area photographed by the surrounding-facing camera 4, which photographs the area around the vehicle from outside the vehicle. The common area determination unit 31 determines a common area in which the same object is photographed in the first image taken by the forward-facing camera 2 and the second image taken by the surrounding-facing camera 4. The window fogging determination unit 34 determines that window fogging has occurred on the front windshield 5 when the amount of blur in the common area in the first image is greater than a predetermined first determination value, and the amount of blur in the common area in the second image is less than a predetermined second determination value.

Description

Technical Field

[0001] The present invention relates to an imaging device mounted on a vehicle.

Background Art

[0002] The imaging device described in Patent Document 1 includes a front monitoring camera that captures an image of the front area of a vehicle from inside the vehicle cabin, and a control unit that corrects the image captured by the front monitoring camera. The control unit performs correction to reduce the brightness of the area in the image captured by the front monitoring camera where the dashboard is reflected, according to the illuminance acquired from an illuminance sensor installed on the dashboard. The image corrected by the control unit is used for image recognition of objects existing around the vehicle.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, generally, the illuminance sensor is installed upward on the dashboard inside the vehicle cabin. Therefore, the illuminance detected by the illuminance sensor is affected by the transmission characteristics of the front windshield or window fogging, etc. The image captured by the front monitoring camera is also affected by the transmission characteristics of the front windshield or window fogging, etc. Therefore, with the configuration of the imaging device described in Patent Document 1, it is difficult to accurately determine the influence received by the image captured by the front monitoring camera from peripheral components (for example, the dashboard, hood, front windshield, etc.) around the camera.

[0005] In view of the above points, an object of the present invention is to determine the influence by peripheral components of a front monitoring camera or the like in an imaging device.

Means for Solving the Problems

[0006] To achieve the above objective, the invention according to claim 1 is an imaging device mounted on a vehicle (10) equipped with a surrounding surveillance camera (4) that photographs the area around the vehicle from outside the vehicle compartment, A forward-facing surveillance camera (2) captures the area in front of the vehicle, including at least a portion of the area captured by the surrounding surveillance camera, from inside the vehicle via the front windshield (5), A common area determination unit (31) determines a common area in which the same object is captured in the first image taken by the forward-facing camera and the second image taken by the surrounding-facing camera, The system includes a window fogging determination unit (34) that determines that window fogging has occurred on the front windshield when the amount of blur in a common area of ​​the first image is greater than a predetermined first determination value, and the amount of blur in a common area of ​​the second image is less than a predetermined second determination value.

[0007] According to this, the first image is affected by fogging of the front windshield, but the second image is not affected by the front windshield. Therefore, the window fogging detection unit can determine that window fogging has occurred when the amount of blur in the first image is large and the amount of blur in the second image is small. Note that the amount of blur in an image refers to the degree to which the outline of the subject is blurred, or the degree to which rapid fluctuations in pixel values ​​are reduced.

[0008] The reference numerals in parentheses attached to each component indicate an example of the correspondence between that component and the specific components described in the embodiments described later. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of a vehicle equipped with an imaging device according to the first reference example. [Figure 2] This is an explanatory diagram to show the condition in which white areas appear in the first image. [Figure 3] This is a block diagram showing the schematic configuration of the imaging device according to the first reference example. [Figure 4] This is a diagram showing an example of the first image. [Figure 5] This is a diagram showing an example of the second image. [Figure 6] This flowchart shows an example of the control processing performed by the control unit of the imaging device according to the first reference example. [Figure 7] This flowchart shows another example of the control processing performed by the control unit of the imaging device according to the first reference example. [Figure 8] This flowchart shows an example of the control processing performed by the control unit of the imaging device according to the second reference example. [Figure 9] This is a block diagram showing the schematic configuration of the imaging device according to the first embodiment. [Figure 10] This flowchart shows an example of the control processing performed by the control unit of the imaging device according to the first embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described below with reference to the drawings. In the following embodiments, parts that are the same or equivalent to each other are denoted by the same reference numerals, and their descriptions are omitted.

[0011] (1st reference example) The first reference example will be explained with reference to the drawings. As shown in Figure 1, the imaging device 1 of the first reference example is mounted on a vehicle 10 and includes a forward-facing camera 2, a hood 7, and a control unit 3. The arrows in Figure 1 indicate front, rear, top, and bottom, respectively, representing the front, rear, top, and bottom of the vehicle.

[0012] The vehicle 10 on which the imaging device 1 of the first reference example is installed is equipped with multiple surrounding surveillance cameras. These multiple surrounding surveillance cameras are mounted on parts outside the vehicle interior, such as the front bumper 11, rear bumper 12, and side mirrors (not shown), and capture images of the area around the vehicle. The images captured by the multiple surrounding surveillance cameras are used to generate an overhead view image of the area around the vehicle by combining these images. Figure 1 shows only the surrounding surveillance camera 4 that captures the area in front of the vehicle 10. Hereinafter, the surrounding surveillance camera 4 that captures the area in front of the vehicle 10 will be referred to as the "forward-facing surrounding surveillance camera 4". The forward-facing surrounding surveillance camera 4 is mounted facing forward, for example, on the front bumper 11 or front grille.

[0013] As shown in Figures 1 and 2, the forward-facing camera 2 is mounted near the ceiling inside the vehicle, facing forward. As indicated by arrow L1 in Figure 2, light enters the forward-facing camera 2 from the area in front of the vehicle outside through the front windshield 5. Therefore, the forward-facing camera 2 captures the area in front of the vehicle 10 through the front windshield 5. In Figure 2, a pedestrian 6 outside the vehicle is shown as an example of a subject present in the area in front of the vehicle 10. As shown in Figure 1, both the forward-facing camera 2 and the forward-facing surrounding camera 4 face forward. Therefore, the forward-facing camera 2 captures an area from inside the vehicle through the front windshield 5 that includes at least a portion of the area captured by the forward-facing surrounding camera 4.

[0014] A hood 7 is provided on the lower side of the vehicle in the vertical direction relative to the forward-facing camera 2. The hood 7 is an example of a peripheral component of the forward-facing camera 2. The hood 7 is a plate-shaped light-shielding component that extends in the vehicle width direction (i.e., the depth direction of the paper in Figure 2) and the vehicle front-rear direction. The hood 7 prevents light emitted from an unshown light source inside the vehicle (e.g., a smartphone) from being reflected by the front windshield 5 and entering the lens of the forward-facing camera 2. This prevents the light source inside the vehicle from being reflected in the image captured by the forward-facing camera 2.

[0015] However, when an external light source L2 (such as the sun) is located obliquely in front of the vehicle 10, as shown by the arrows L3, L4, and L5 in FIG. 2, the light emitted from the external light source L2 may be reflected by the hood 7 and the front windshield 5 in that order and enter the forward monitoring camera 2. As a result, white floating may occur in the image captured by the forward monitoring camera 2. Note that white floating refers to a phenomenon in which excessive brightness is incorporated into the entire image, or in other words, a phenomenon in which the pixel values of the entire image become high within the dynamic range of the image sensor.

[0016] In view of this point, the imaging device 1 of the first reference example aims to obtain an image with reduced white floating due to the hood 7.

[0017] As shown in FIG. 3, the forward monitoring camera 2 includes an optical system 21, an image sensor 22, an image generation unit 23, and the like.

[0018] The optical system 21 includes a lens 24, an aperture 25, and the like. The lens 24 forms an image of the light incident from the forward region of the vehicle 10 on the image sensor 22. The aperture 25 adjusts the amount of light passing through the lens 24 according to a predetermined aperture value.

[0019] The image sensor 22 is, for example, a CCD (Charge Coupled Devices) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image sensor 22 has, for example, a photodiode for each pixel, and when light with a value equal to or higher than the lower limit value and equal to or lower than the upper limit value is incident on the photodiode, an electrical signal corresponding to the amount of light is output. The image sensor 22 includes an electronic shutter. The electronic shutter adjusts the exposure time by electronic control within the image sensor 22 (such as switching the control of transistors).

[0020] The image generation unit 23 synthesizes, for example, a short-shutter image and a long-shutter image to generate a high-dynamic range composite image (hereinafter simply referred to as an "image"). The image is output to the control unit 3.

[0021] The control unit 3 is primarily composed of a microcomputer equipped with a processor and memory such as ROM, RAM, and flash memory. The control unit 3 implements functions such as a common area determination unit 31, a difference calculation unit 32, and a brightness adjustment unit 33 by having the processor execute programs stored in the memory.

[0022] The control unit 3 is configured to receive images captured by the forward-facing camera 2 and images captured by the forward-facing perimeter camera 4. Figure 4 shows an example of an image captured by the forward-facing camera 2, and Figure 5 shows an example of an image captured by the forward-facing perimeter camera 4 at the same time as the image in Figure 4. In the following explanation, the image captured by the forward-facing camera 2 will be referred to as the "first image," and the image captured by the forward-facing perimeter camera 4 will be referred to as the "second image." The first and second images differ in various specifications such as distortion and resolution, but there is a common region in which the same object is captured. In Figure 5, the common region in the second image is indicated by a dashed line with the code CA. In the example in Figure 4, the entirety of the first image constitutes the common region.

[0023] Returning to Figure 3, the common area determination unit 31 determines the common area where the same object is captured in the first and second images. The common area determination unit 31 determines the common area by performing image recognition when the forward-facing monitoring camera 2 and the forward-facing surrounding monitoring camera 4 are in operation. Alternatively, the common area determination unit 31 may determine the common area when the forward-facing monitoring camera 2 and the forward-facing surrounding monitoring camera 4 are installed on the vehicle 10 or during maintenance.

[0024] The brightness adjustment unit 33 calculates the difference between the brightness of the common area in the first image and the brightness of the common area in the second image (hereinafter simply referred to as the "brightness difference") when the forward-facing surveillance camera 2 is in operation. Specifically, the brightness adjustment unit 33 calculates the brightness difference as the difference between the average value of the pixel values ​​in the common area in the first image and the average value of the pixel values ​​in the common area in the second image.

[0025] The brightness adjustment unit 33 calculates the difference in brightness under normal conditions and stores it in memory. Under normal conditions refers to the time when conditions are met under which white balance does not occur, for example, when the amount of sunlight is less than a predetermined threshold, or when the external light source L2 is located in a position other than the upper diagonal front of the vehicle 10. The difference in brightness under normal conditions is caused by differences in the specifications of the forward monitoring camera 2 and the surrounding monitoring camera 4, as well as the transmission characteristics of the front windshield 5. The brightness adjustment unit 33 calculates the difference in brightness under normal conditions when the conditions are met under which white balance does not occur when the forward monitoring camera 2 is in operation. Alternatively, the brightness adjustment unit 33 may calculate the difference in brightness under normal conditions when the forward monitoring camera 2 and the forward-facing surrounding monitoring camera 4 are installed on the vehicle 10, or during maintenance.

[0026] The brightness adjustment unit 33 then performs image processing to correct the brightness of the first image so that the current brightness difference is closer to the normal brightness difference if the current brightness difference is different from the normal brightness difference. Specifically, the brightness adjustment unit 33 calculates the amount of white balance as the difference obtained by subtracting the difference in normal brightness from the current brightness difference. Then, the brightness adjustment unit 33 performs a correction on the current first image by subtracting the amount of white balance.

[0027] Alternatively, when the forward-facing camera 2 is in operation, the brightness adjustment unit 33 may perform exposure control, aperture control, or sensitivity control on the forward-facing camera 2 if the current brightness difference differs from the normal brightness difference. This exposure control, aperture control, or sensitivity control is performed to bring the current brightness difference closer to the normal brightness difference. Specifically, the brightness adjustment unit 33 sets the shutter speed (i.e., exposure time), aperture value, or gain for amplifying the signal from the image sensor 22 of the forward-facing camera 2 so that the amount of white balance is reduced from the current first image.

[0028] As a result, the imaging device 1 can obtain images with reduced overexposure. The images obtained by the imaging device 1 are transmitted to the electronic control unit 8 (hereinafter referred to as "ECU8"). The ECU8 performs image recognition using these images to provide driving assistance, such as automatic driving control. ECU stands for Electronic Control Unit.

[0029] Next, an example of the control processing performed by the control unit 3 of the imaging device 1 according to the first reference example will be explained with reference to the flowchart in Figure 6. In the following explanation and figures, steps will be denoted as "S". Also, in the flowchart, the forward monitoring camera 2 is denoted as FCM, and the forward-facing periphery monitoring camera 4 is denoted as PVM. FCM is an abbreviation for front camera module, and PVM is an abbreviation for panoramic view monitor or panoramic view monitor.

[0030] As shown in Figure 6, in S10, the control unit 3 acquires a first image from the forward-facing monitoring camera 2 and a second image from the forward-facing surrounding monitoring camera 4.

[0031] Next, in S20, the common area determination unit 31 determines the common area in which the same object was captured in the first image and the second image.

[0032] Next, in S30, the brightness adjustment unit 33 compares the first image and the second image and calculates the difference in current brightness.

[0033] Next, in S40, the brightness adjustment unit 33 determines whether the difference in the current brightness is different from the difference in the normal brightness stored in memory beforehand. If the difference in the current brightness is the same as the difference in the normal brightness, the brightness adjustment unit 33 determines that no white distortion has occurred and proceeds to S50.

[0034] In S50, the brightness adjustment unit 33 transmits the first image captured by the forward-facing camera 2 to the ECU 8 using normal control.

[0035] In response, in S40, the brightness adjustment unit 33 determines that white distortion is occurring if the difference in current brightness is different from the difference in normal brightness, and proceeds to S60.

[0036] In S60, the brightness adjustment unit 33 calculates the amount of white balance as the difference between the current brightness and the difference in normal brightness, and performs a correction (i.e., image processing) on ​​the current first image by subtracting the amount of white balance. The control unit 3 then transmits the image processed in this way to the ECU 8.

[0037] Next, another example of the control processing performed by the control unit 3 of the imaging device 1 according to the first reference example will be explained with reference to the flowchart in Figure 7.

[0038] As shown in Figure 7, the processes from S10 to S50 are the same as the example of control processing described with reference to Figure 6. In another example of the control processing, if the brightness adjustment unit 33 determines in S40 that white distortion is occurring (i.e., the determination in S40 is Yes), the process proceeds to S70.

[0039] In S70, the brightness adjustment unit 33 calculates the amount of white balance as the difference between the current brightness and the difference in normal brightness. The brightness adjustment unit 33 then sets the shutter speed (i.e., exposure time), aperture value, and gain for amplifying the signal from the image sensor 22 of the forward-facing camera 2 to reduce the amount of white balance from the current first image, and performs exposure control, aperture control, or sensitivity control on the forward-facing camera 2. The control unit 3 then transmits the resulting image to the ECU 8.

[0040] The imaging device 1 of the first reference example described above has the following effects. (1) In the first reference example, the control unit 3 of the imaging device 1 calculates the difference in brightness between the first image and the second image, and based on that difference, performs image processing to correct the brightness of the first image, or performs exposure control, aperture control, or sensitivity control on the forward monitoring camera 2. According to this, the orientation of the forward-facing peripheral surveillance camera 4 and the orientation of the forward-facing surveillance camera 2 are similar, and the second image captured by the forward-facing peripheral surveillance camera 4 is not affected by the surrounding components of the forward-facing surveillance camera 2 (for example, white highlighting due to the hood 7). Therefore, by calculating the difference in brightness between the first and second images, it is possible to determine the influence that the first image is receiving from the surrounding components of the forward-facing surveillance camera 2. Accordingly, the imaging device 1 can obtain an image with reduced influence from the surrounding components of the forward-facing surveillance camera 2 by performing image processing to correct the brightness of the first image based on that difference, or by performing exposure control, aperture control, or sensitivity control on the forward-facing surveillance camera 2.

[0041] (2) In the first reference example, the control unit 3 stores the difference in brightness under normal conditions. When the difference in current brightness differs from the difference in brightness under normal conditions, the control unit 3 performs image processing to correct the brightness of the first image so that the difference in current brightness approaches the difference in brightness under normal conditions. Alternatively, when the difference in current brightness differs from the difference in brightness under normal conditions, the control unit 3 performs exposure control, aperture control, or sensitivity control on the forward-facing camera 2 so that the difference in current brightness approaches the difference in brightness under normal conditions. According to this, the control unit 3 can determine whether or not white distortion caused by the hood 7 is occurring in the first image by comparing the difference in current brightness with the difference in normal brightness, and can also calculate the amount of white distortion. Therefore, the control unit 3 can obtain an image with reduced white distortion by performing image processing to correct the brightness of the first image so that the difference in current brightness approaches the difference in normal brightness, or by performing exposure control, aperture control, or sensitivity control on the forward monitoring camera 2.

[0042] (3) In the first reference example, the control unit 3 calculates the amount of white balance as the difference in brightness under normal conditions by subtracting the difference in brightness under normal conditions from the difference in brightness under current conditions. The control unit 3 then either performs a correction to subtract the amount of white balance from the current first image, or performs exposure control, aperture control, or sensitivity control on the forward monitoring camera 2 so that the amount of white balance is subtracted from the current first image. According to this, the control unit 3 can obtain an image with reduced white balance by performing image processing to correct the brightness of the first image using the specifically calculated amount of white balance, or by performing exposure control, aperture control, or sensitivity control on the forward-facing camera 2.

[0043] (2nd reference example) The second reference example will now be explained. The second reference example differs from the first reference example mainly in that the control processing performed by the control unit 3 of the imaging device 1 has been modified. Otherwise, it is substantially the same as the first reference example, so only the differences from the first reference example will be explained.

[0044] The imaging device 1 in the second reference example also includes a forward-facing camera 2 and a control unit 3, similar to those described in the first reference example with reference to Figures 1 to 3. Note that the imaging device 1 in the second reference example does not necessarily have the hood 7 described in the first reference example.

[0045] In the second reference example, the forward-facing camera 2 is mounted near the ceiling inside the vehicle, facing forward, and captures images of the area in front of the vehicle 10 through the front windshield 5. In the second reference example, the front windshield 5 is an example of a peripheral component of the forward-facing camera 2.

[0046] The transmission characteristics of the front windshield 5 (i.e., the relationship between wavelength and transmittance) vary depending on the vehicle model and grade. However, if the transmission characteristics of the front windshield 5 are not input to the imaging device 1 when it is mounted on the vehicle, and an image affected by these transmission characteristics is transmitted from the imaging device 1 to the ECU 8, it may become difficult for the ECU 8 to perform image recognition using that image. For example, if the ECU 8 uses the image transmitted from the imaging device 1 to recognize the taillights of the vehicle in front, it may become difficult to accurately measure the red brightness information in an image affected by the transmission characteristics of the front windshield 5.

[0047] Generally, the imaging device 1 may have an auto white balance function to correct the color tone of the image. However, when recognizing the taillights of a vehicle ahead, it is necessary to measure the red brightness information in the image without correction, so correcting the color tone of the image with auto white balance is meaningless.

[0048] In view of this, the imaging device 1 of the second reference example aims to obtain an image with reduced influence from the transmission characteristics of the front windshield 5.

[0049] An example of the control processing performed by the control unit 3 of the imaging device 1 according to the second reference example will be explained with reference to the flowchart in Figure 8.

[0050] As shown in Figure 8, in S110, the control unit 3 acquires a first image from the forward-facing monitoring camera 2 and a second image from the forward-facing surrounding monitoring camera 4.

[0051] Next, in S120, the common area determination unit 31 determines the common area in which the same object was captured in the first image and the second image.

[0052] Next, in S130, the brightness adjustment unit 33 compares the first image and the second image and calculates the difference between the pixel values ​​in the common area of ​​the first image and the pixel values ​​in the common area of ​​the second image for each of the color information (e.g., RGB).

[0053] Next, in S140, the brightness adjustment unit 33 performs image processing to correct the color information of each pixel value of the first image based on the difference of each color information calculated by the difference calculation unit 32. As a result, the imaging device 1 can obtain an image with reduced influence from the transmission characteristics of the front windshield 5. The imaging device 1 then transmits this image to the ECU 8.

[0054] The imaging device 1 of the second reference example described above has the following effects. In the second example, the difference calculation unit 32 calculates the difference between the pixel values ​​in the common area of ​​the first image and the pixel values ​​in the common area of ​​the second image for each color information. Then, the brightness adjustment unit 33 performs image processing to correct the color information of the pixel values ​​of the first image based on the difference calculated by the difference calculation unit 32. According to this, the first image is affected by the transparency characteristics of the front windshield 5, while the second image is not affected by the transparency characteristics of the front windshield 5. Therefore, by calculating the difference between the pixel values ​​in the common region of the first image and the pixel values ​​in the common region of the second image for each color information, it is possible to determine the influence that the first image receives from the front windshield 5, i.e., the transparency characteristics of the front windshield 5. Accordingly, the imaging device 1 can obtain an image with reduced influence from the transparency characteristics of the front windshield 5 by correcting the color information of the pixel values ​​of the first image based on this difference. As a result, the accuracy of image recognition can be improved in the ECU 8, which performs image recognition using the image transmitted from the imaging device 1. Specifically, for example, when the ECU 8 performs taillight recognition of a vehicle in front, the measurement accuracy of the red brightness information in the image can be improved.

[0055] (First Embodiment) A first embodiment will be described. In the configuration of the imaging device described in Patent Document 1 above, if blurring occurs in the image acquired from the forward-facing camera 2, it is difficult to determine whether the cause is fogging of the front windshield 5 or fog generated outside the vehicle. Image blurring refers to a state in which the outline of the subject is blurred, or a state in which rapid fluctuations in pixel values ​​are reduced. If the driver of the vehicle 10 or the vehicle ECU 8, etc., does not take appropriate action against the cause of blurring in the image acquired from the forward-facing camera 2 (specifically, fogging of the front windshield 5 or fog outside the vehicle), it may impair the safe driving of the vehicle 10.

[0056] In view of this, the imaging device 1 of the first embodiment is intended to determine the influence of surrounding components and the external environment on the forward-facing camera 2 (specifically, condensation on the front windshield 5 or fog outside the vehicle).

[0057] As shown in Figure 9, the imaging device 1 of the first embodiment also includes a forward-facing camera 2 and a control unit 3, etc. Note that the imaging device 1 of the first embodiment does not necessarily have the hood 7 described in the first reference example.

[0058] In the first embodiment, the forward-facing camera 2 is mounted near the ceiling inside the vehicle interior, facing forward, and captures images of the area in front of the vehicle 10 through the front windshield 5. In the first embodiment, the front windshield 5 is an example of a peripheral component of the forward-facing camera 2.

[0059] As shown in Figure 9, the control unit 3 of the imaging device 1 of the first embodiment realizes functions such as a common area determination unit 31, a window fogging determination unit 34, and a fog determination unit 35 by having the processor execute a program stored in memory.

[0060] The common area determination unit 31 determines a common area where the same object is captured in the first image and the second image. The common area determination unit 31 determines the common area by performing image recognition when the forward-facing monitoring camera 2 and the forward-facing surrounding monitoring camera 4 are in operation. Alternatively, the common area determination unit 31 may determine the common area when the forward-facing monitoring camera 2 and the forward-facing surrounding monitoring camera 4 are installed on the vehicle 10 or during maintenance.

[0061] The window fogging determination unit 34 determines whether or not fogging has occurred on the front windshield 5. Here, the first image is affected by the fogging on the front windshield 5, but the second image is not affected by the front windshield 5. Therefore, the window fogging determination unit 34 determines that fogging has occurred on the front windshield 5 when the amount of blur in the common area of ​​the first image is greater than a predetermined first determination value, and the amount of blur in the common area of ​​the second image is less than a predetermined second determination value. The first determination value is a value that can determine the amount of blur due to window fogging, which is set in advance through experiments and stored in the control unit 3. The second determination value is a value that can determine the amount of blur due to fog outside the vehicle, which is set in advance through experiments and stored in the control unit 3.

[0062] Here, the amount of blur in an image can be determined, for example, by the amount of high-frequency components with short periods of brightness and darkness in the pixel values. That is, if there are many of these high-frequency components in the image, there is no blur in the image, and if there are few of these high-frequency components, it can be said that blur occurs in the image. Specifically, the window fogging determination unit 34 can determine that window fogging has occurred on the front windshield 5 when the amount of high-frequency components in the common area of ​​the first image is less than a predetermined first threshold, and the amount of high-frequency components in the common area of ​​the second image is greater than a predetermined second threshold. The first threshold is a value that can determine the amount of blur due to window fogging and is set in advance through experiments, etc., and is stored in the control unit 3. The second threshold is a value that can determine the amount of blur due to fog outside the vehicle and is set in advance through experiments, etc., and is stored in the control unit 3.

[0063] The fog detection unit 35 determines whether or not fog is present outside the vehicle. If fog is present outside the vehicle, blurring occurs in both the first and second images. Therefore, the fog detection unit 35 determines that fog is present outside the vehicle when the amount of blurring in the common area of ​​the first image is greater than a predetermined first determination value, and the amount of blurring in the common area of ​​the second image is greater than a predetermined second determination value.

[0064] Specifically, the fog detection unit 35 can determine that fog is present outside the vehicle when the high-frequency components in a common region of the first image are less than a predetermined first threshold, and the high-frequency components in a common region of the second image are less than a predetermined second threshold.

[0065] The results determined by the window fogging detection unit 34 and the fog detection unit 35 are transmitted to the vehicle 10's ECU 8. The vehicle 10's ECU 8 informs the occupants of the determination results, for example, through an in-cabin display, lamp, or speaker. This allows the occupants to take appropriate action according to the situation, such as turning on the defroster or switching the headlights to high beam.

[0066] Next, an example of the control processing performed by the control unit 3 of the imaging device 1 according to the first embodiment will be explained with reference to the flowchart in Figure 10.

[0067] As shown in Figure 10, in S210, the control unit 3 acquires a first image from the forward-facing monitoring camera 2 and a second image from the forward-facing surrounding monitoring camera 4.

[0068] Next, in S220, the common area determination unit 31 determines the common area in which the same object was captured in the first image and the second image.

[0069] Next, in S230, the control unit 3 calculates the amount of high-frequency components in the common region of the first image, and also calculates the amount of high-frequency components in the common region of the second image. The amount of high-frequency components is calculated by frequency analysis, such as the fast Fourier transform, of the period of brightness and darkness of the pixel values.

[0070] Next, in S240, the control unit 3 determines whether the high-frequency components in the common region of the first image are less than the first threshold. If the control unit 3 determines that the high-frequency components in the common region of the first image are more than the first threshold (i.e., the amount of blur in the first image is small), it proceeds to S260.

[0071] In the S260, the processing unit determines that there is no condensation on the front windshield 5 and that there is no fog outside the vehicle.

[0072] In response to this, if the control unit 3 determines in S240 that the high-frequency components in the common region of the first image are less than the first threshold (i.e., the amount of blur in the first image is large), it proceeds to S250.

[0073] In S250, the control unit 3 determines whether the high-frequency components in the common region of the second image are less than the second threshold. If the control unit 3 determines that the high-frequency components in the common region of the second image are greater than the second threshold (i.e., the amount of blur in the second image is small), it proceeds to S270.

[0074] In S270, the processing unit determines that window fogging has occurred on the front windshield 5. The imaging device 1 transmits the determination result from S270 to the ECU 8 of the vehicle 10.

[0075] In response to this, if the control unit 3 determines in S250 that the high-frequency components in the common region of the second image are less than the second threshold (i.e., the amount of blur in the second image is large), it proceeds to S280.

[0076] In S280, the processing unit determines that fog is present outside the vehicle. The imaging device 1 transmits the determination result from S280 to the ECU 8 of the vehicle 10.

[0077] The imaging device 1 of the first embodiment described above provides the following effects. (1) In the first embodiment, the window fogging determination unit 34 determines that window fogging has occurred on the front windshield 5 when the amount of blur in the common area of ​​the first image is greater than a predetermined first determination value, and the amount of blur in the common area of ​​the second image is less than a predetermined second determination value. According to this, the first image is affected by the fogging of the front windshield 5, but the second image is not affected by the front windshield 5. Therefore, the window fogging detection unit 34 can determine that window fogging has occurred when the amount of blur in the first image is large and the amount of blur in the second image is small.

[0078] (2) In the first embodiment, the window fogging determination unit 34 determines that window fogging has occurred on the front windshield 5 when the high-frequency components in the common region of the first image are less than a predetermined first threshold, and the high-frequency components in the common region of the second image are greater than a predetermined second threshold. According to this, when the high-frequency components in the common region of the first image are less than the first threshold, the amount of blur in the first image is considered large. When the high-frequency components in the common region of the second image are greater than the second threshold, the amount of blur in the second image is considered small. Therefore, the window fogging determination unit 34 can determine that window fogging has occurred when the amount of blur in the first image is large and the amount of blur in the second image is small.

[0079] (3) In the first embodiment, the fog determination unit 35 determines that fog is present outside the vehicle when the amount of blur in the common area of ​​the first image is greater than a predetermined first determination value, and the amount of blur in the common area of ​​the second image is greater than a predetermined second determination value. According to this, the fog detection unit 35 can determine that fog is present outside the vehicle when the amount of blur in the first image is large and the amount of blur in the second image is also large.

[0080] (4) In the first embodiment, the fog determination unit 35 determines that fog is occurring outside the vehicle when the high-frequency components in the common region of the first image are less than a predetermined first threshold, and the high-frequency components in the common region of the second image are less than a predetermined second threshold. According to this, the fog detection unit 35 can determine that fog is present outside the vehicle when the amount of blur in the first image is large and the amount of blur in the second image is also large.

[0081] (Other embodiments) (1) In each of the above embodiments, the forward-facing camera 2 and the control unit 3 of the imaging device 1 have been described as separate configurations, but the invention is not limited to this, and for example, the forward-facing camera 2 and the control unit 3 may be configured as a single unit.

[0082] (2) In each of the above embodiments, the control unit 3 of the imaging device 1 and the ECU 8 of the vehicle 10 were described as having separate configurations, but the invention is not limited to this, and for example, some or all of the functions of the control unit 3 may be incorporated into the ECU 8 of the vehicle 10.

[0083] (3) In the second reference example above, the recognition of the taillights of a vehicle ahead was described as an example of image recognition performed by the ECU8, but it is not limited to that, and may also be image recognition of signals or signs, for example.

[0084] (4) In the first embodiment described above, it was determined that the amount of blur in the image was large when there were few high-frequency components with short periods of brightness and darkness in the pixel values, but it is not limited to this. For example, it may be determined that the amount of blur in the image is large when there are many low-frequency components with long periods of brightness and darkness in the pixel values, or it may be determined by various methods such as deep learning.

[0085] (5) In the first embodiment described above, the determination results from the window fogging determination unit 34 and fog determination unit 35 of the control unit 3 were transmitted to the ECU 8 of the vehicle 10. However, the embodiment is not limited to this. For example, the determination results from the control unit 3 may be transmitted directly to a display, lamp, or speaker in the vehicle interior.

[0086] The present invention is not limited to the embodiments described above, and can be modified as appropriate within the scope of the claims. Furthermore, the embodiments and parts thereof are not unrelated to each other, and can be combined as appropriate, except in cases where the combination is clearly impossible. In addition, it goes without saying that the elements constituting the embodiments are not necessarily essential, except in cases where they are explicitly stated to be particularly essential or where they are clearly considered essential in principle. Furthermore, in the embodiments, when numerical values ​​such as the number, numerical values, quantities, or ranges of the components of the embodiments are mentioned, the invention is not limited to those specific numbers, except in cases where they are explicitly stated to be particularly essential or where they are clearly limited to a specific number in principle. Furthermore, when the shapes, positional relationships, etc., of the components, etc., are mentioned in the embodiments, the invention is not limited to those shapes, positional relationships, etc., except in cases where they are explicitly stated to be particularly essential or where they are clearly limited to a specific shape, positional relationship, etc., in principle.

[0087] The control unit and method described in the present invention may be implemented by a dedicated computer provided by configuring a processor and memory programmed to perform one or more functions embodied by a computer program. Alternatively, the control unit and method described in the present invention may be implemented by a dedicated computer provided by configuring a processor by one or more dedicated hardware logic circuits. Alternatively, the control unit and method described in the present invention may be implemented by one or more dedicated computers configured by a combination of a processor and memory programmed to perform one or more functions and a processor configured by one or more hardware logic circuits. Furthermore, the computer program may be stored as instructions executed by the computer on a computer-readable non-transitional tangible recording medium. [Explanation of Symbols]

[0088] 1. Imaging device 2. Forward-facing surveillance camera 4 Surrounding surveillance cameras 5. Front windshield 10 vehicles 31 Common area determination section 34 Window fogging detection unit

Claims

1. An imaging device mounted on a vehicle (10) equipped with a surrounding surveillance camera (4) that photographs the area around the vehicle from outside the vehicle compartment, A forward-facing camera (2) captures the area in front of the vehicle, including at least a portion of the area captured by the surrounding surveillance camera, from inside the vehicle via the front windshield (5), A common area determination unit (31) determines a common area in which the same object is captured in the first image taken by the forward-facing camera and the second image taken by the surrounding-facing camera, An imaging device comprising: a window fogging determination unit (34) that determines that window fogging has occurred on the front windshield when the amount of blur in the common area of ​​the first image is greater than a predetermined first determination value, and the amount of blur in the common area of ​​the second image is less than a predetermined second determination value.

2. The imaging apparatus according to claim 1, wherein the window fogging determination unit determines that window fogging has occurred on the front windshield when the number of high-frequency components with short periods of brightness and darkness of pixel values ​​in the common region of the first image is less than a predetermined first threshold, and the number of high-frequency components with short periods of brightness and darkness of pixel values ​​in the common region of the second image is greater than a predetermined second threshold.

3. The imaging device according to claim 1 or 2, further comprising a fog determination unit (35) that determines that fog is present outside the vehicle when the amount of blur in the common region of the first image is greater than a predetermined first determination value, and the amount of blur in the common region of the second image is greater than a predetermined second determination value.

4. The imaging device according to claim 3, wherein the fog determination unit determines that fog is occurring outside the vehicle when the number of high-frequency components with short periods of brightness and darkness of pixel values ​​in the common region of the first image is less than a predetermined first threshold, and the number of high-frequency components with short periods of brightness and darkness of pixel values ​​in the common region of the second image is less than a predetermined second threshold.

Images