Rainfall estimation device
The rainfall estimation device uses a camera and deep neural network to determine rainfall images and estimate rainfall based on the time since the last wiper operation, addressing inaccuracies due to environmental changes and enabling precise wiper control.
Patent Information
- Application Number
- JP2021034548
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-04
- Publication Date
- 2024-11-13
- Estimated Expiration
- 2041-03-04
AI Technical Summary
Existing rainfall estimation methods struggle to accurately estimate rainfall due to discontinuous changes in the observation environment, such as when it starts to rain or after windshield wiper operations.
A rainfall estimation device using a camera and deep neural network to determine rainfall images and estimate rainfall based on the time from the last windshield wiper operation to the detection of raindrops, employing image determination and rain amount estimation processes.
Enables accurate rainfall estimation unaffected by discontinuous changes in the observation environment, allowing for precise wiper control based on estimated rainfall.
Smart Images

Figure 0007581973000001 
Figure 0007581973000002 
Figure 0007581973000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a rainfall estimation device, and more particularly to a rainfall estimation device that estimates the amount of rainfall based on raindrops adhering to a windshield of a vehicle. [Background technology]
[0002] Patent Document 1 discloses a raindrop detection device that actually detects raindrops on the windshield of a vehicle in order to detect (estimate) the amount of rainfall. This raindrop detection device performs frequency analysis on the photographic data captured by a CCD camera, extracts the raindrops on the windshield as high-frequency components, calculates the amount of energy of the high-frequency components, and determines the amount of raindrops on the windshield based on the change in the amount of energy over time. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2001-153969 A Summary of the Invention [Problem to be solved by the invention]
[0004] In the technology described in Patent Document 1, the amount of change in a parameter for estimating the amount of rainfall (i.e., the amount of energy of the high-frequency components) is observed. With this method, there is a concern that it may be difficult to estimate the amount of rainfall appropriately when there is a discontinuous change in the observation environment, such as when rain starts to fall or before and after the windshield wiper is wiping the windshield.
[0005] The present invention has been made in consideration of the above-mentioned problems, and has an object to make it possible to appropriately estimate rainfall without being affected by discontinuous changes in the observation environment. [Means for solving the problem]
[0006] A rainfall estimation device according to the present invention is a device that estimates the amount of rainfall based on raindrops adhering to a windshield of a vehicle, and includes a camera that captures an image of the area ahead of the vehicle through the windshield, and a processor. The processor The deep neural network is trained on images that have rainfall characteristics from among images captured in advance. captured by camera Imaging image By inputting The image determination process determines whether the image is a rainfall image indicating a rainfall state, and the rainfall amount estimation process estimates the amount of rainfall based on the time from the most recent wiping action of the vehicle's windshield wipers to the detection of the rainfall image by the image determination process. The image determination process is a process for determining whether or not captured images input in sequence to the deep neural network at a predetermined imaging period are rainfall images. The rainfall estimation process estimates that the amount of rainfall when the mth captured image to be subjected to image determination processing after the start of the most recent wiping operation is detected as a rainfall image is greater than the amount of rainfall when the m+nth captured image to be subjected to image determination processing after the start of the most recent wiping operation is detected as a rainfall image. Effect of the Invention
[0007] According to the present invention, the amount of rain is estimated based on the time from the most recent wiping action of the wipers to the detection of the rainfall image, which makes it possible to appropriately estimate the amount of rain without being affected by discontinuous changes in the observation environment. [Brief description of the drawings]
[0008] [Figure 1] 1 is a diagram illustrating an example of a configuration of a rainfall estimation device according to an embodiment. [Diagram 2] FIG. 1 is a diagram for explaining a specific method for determining a rainfall image (rainy scene image) using machine learning. [Diagram 3] 10 is a diagram for explaining the relationship between the amount of rain and the time t until it is determined that a rainfall state is occurring by the rainfall estimation process. FIG. [Figure 4] 6A and 6B are diagrams for explaining an example of a wiper control process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, the embodiments of the present invention will be described with reference to the accompanying drawings. In the embodiments shown below, when the number, quantity, amount, range, etc. of each element is mentioned, the present invention is not limited to the mentioned number, unless otherwise specified or clearly specified in principle. Furthermore, the structures, etc. described in the embodiments shown below are not necessarily essential to the present invention, unless otherwise specified or clearly specified in principle.
[0010] 1. Example of rainfall estimation device configuration 1 is a diagram illustrating an example of a configuration of a rainfall estimation device 10 according to an embodiment. The rainfall estimation device 10 includes a camera 12 and an electronic control unit (ECU) 14.
[0011] The camera 12 is an imaging element such as a CCD or CMOS, and is mounted on the vehicle so as to capture an image of the area ahead of the vehicle through the windshield of the vehicle. More specifically, the camera 12 captures an image of the area ahead of the vehicle at a predetermined angle of view and outputs the captured image to the ECU 14.
[0012] The ECU 14 includes a processor 16 and a storage device 18. The camera 12 is connected to the ECU 14 directly or via a communication network such as a Controller Area Network (CAN) installed in the vehicle. Images captured by the camera 12 are repeatedly transmitted to the ECU 14 at a predetermined cycle.
[0013] The vehicle on which the camera 12 is mounted is equipped with a wiper 20 that wipes off raindrops from the windshield. The operation of wiping the windshield by the wiper 20 is controlled by the ECU 14, for example, as described later with reference to FIG.
[0014] The processor 16 of the ECU 14 estimates the amount of rainfall based on image information through the windshield from the camera 12. More specifically, to estimate the amount of rainfall, the processor 16 executes an "image determination process" and a "rainfall amount estimation process." The storage device 18 of the ECU 14 stores programs related to these processes, and the "image determination process" and the "rainfall amount estimation process" are each realized by the processor 16 executing the programs. The processor 16 also executes a "wiper control process" that controls the operation of the wipers based on the information on the estimated amount of rainfall.
[0015] 2. Image Judgment Processing The image determination process of this embodiment is a process for determining whether an image captured by the camera 12 is a rainfall image showing a rainfall state (in other words, whether an image of a rainy scene is captured). This determination can be made, for example, by using an image recognition function using machine learning provided in the processor 16.
[0016] FIG. 2 is a diagram for explaining a specific method of determining a rainfall image (rainy scene image) using machine learning. As an example, the image determination process is performed using a deep neural network (DNN) 22 constructed for determining a rainfall image. The DNN 22 is constructed to determine a rainfall image (rainy scene image) by learning an image having characteristics of a rainfall state (rainy scene) from among images captured in advance as a correct answer image. In the image determination process, the processor 16 inputs an image of the camera 12 to the DNN 22 as an input image, thereby being able to determine whether the input image is a rainfall image or not.
[0017] 3. Rainfall estimation processing The rainfall estimation process of this embodiment is a process for estimating the amount of rainfall, more specifically, how much rain has fallen within a certain period of time (the level of rainfall within a certain period of time). Specifically, in the rainfall estimation process, the processor 16 receives operation information of the wipers 20 from the wipers 20. Then, the processor 16 estimates the amount of rainfall based on the time t from the most recent wiping operation of the wipers 20 on the windshield to the detection (determination) of a rainfall image by the image determination process.
[0018] There are no raindrops on the windshield immediately after the wiping operation of the wiper 20. Therefore, in the image determination process, the processor 16 does not determine that the image captured by the camera 12 immediately after the wiping operation is a rainfall image. On the other hand, as raindrops adhere to the windshield after the wiping operation, the number of characteristic parts in the image that indicate a rainfall state (rainy scene) increases. Therefore, the image taken at a certain point after the wiping operation is determined to be a rainfall image.
[0019] FIG. 3 is a diagram for explaining the relationship between the amount of rain and the time t until it is determined that a rainfall state is occurring by the rainfall estimation process. More specifically, FIG. 3(A) shows an example of determination when the amount of rain is large, and FIG. 3(B) shows an example of determination when the amount of rain is small compared to FIG. 3(A). "Image 1" in FIG. 3 corresponds to the image captured by camera 12 for the first time after the wiping action of wiper 20 is performed. "Image 2" corresponds to the image captured by camera 12 in the next imaging cycle after image 1. The same applies below.
[0020] In the example shown in Fig. 3(A), the mth image m taken after the wiping operation of the wiper 20 is determined to be in a rainy state (rainy scene image) by the image determination process. On the other hand, in the example shown in Fig. 3(B), which corresponds to a case where the amount of rain is less than that in Fig. 3(A), the m+nth image m+n taken after the wiping operation is determined to be in a rainy state (rainy scene image) by the image determination process. Image m+n is an image captured n times after image m.
[0021] As can be understood from the description with reference to FIG. 3, it can be said that the time t from the most recent wiping operation to the detection of the rainfall image becomes shorter when the rainfall is heavy, while it becomes longer when the rainfall is light. Therefore, according to the rainfall determination process of the present embodiment, the processor 16 can estimate the rainfall based on the length of the time t by measuring the time t.
[0022] In addition, the rainfall estimated by the rainfall estimation process is calculated, for example, so that it increases as the above-mentioned time t becomes shorter. Further, instead of such a calculation example, the rainfall may be calculated, for example, as the arithmetic mean value of a plurality of rainfall values (values corresponding to the time t) calculated by the rainfall estimation process within a predetermined period.
[0023] 4. Wiper control process The processor 16 of the ECU 14 changes the operation of the wiper 20 according to the rainfall estimated by the above-described rainfall estimation process. Note that this wiper control process may be performed by a processor of an ECU for controlling the wiper 20 that is separate from the ECU 16.
[0024] FIG. 4 is a diagram for explaining an example of the wiper control process. Here, as an example, the operation speed of the wiper 20 is changed according to the rainfall estimated by the rainfall estimation process. FIG. 4 shows the correspondence between each example of the above-mentioned time t (sec) correlated with the estimated rainfall and the optimum operation speed tw (sec / revolution) of the wiper 20 under each example of the time t.
[0025] In the setting example shown in FIG. 4, the operation speed tw of the wiper 20 is set to increase as the estimated rainfall increases (that is, as the time t becomes shorter). Specifically, when the time t is equal to or greater than the value t0 and less than the value t1, the operation speed tw takes the value tw0. When the time t is equal to or greater than the value t1 and less than the value t2, the operation speed tw takes the value tw1 (<tw0). The same applies hereinafter. The processor 16 controls the wiper 20 so as to obtain the operation speed tw corresponding to the estimated rainfall (time t) according to the relationship shown in FIG. 4.
[0026] 5.Effects As described above, the image determination process of this embodiment determines whether or not the image captured by the camera 12 capturing an image of the area ahead of the vehicle through the windshield is a rainfall image. Then, the rainfall amount estimation process estimates the amount of rainfall based on the time t from the most recent wiping action of the wipers 20 to the detection of the rainfall image. This method makes it possible to appropriately estimate the amount of rainfall without being affected by discontinuous changes in the observation environment. [Explanation of symbols]
[0027] 10 Rainfall estimation device 12 Camera 14 Electronic Control Unit (ECU) 16 processors 20 Wiper
Claims
[Claim 1] A rainfall estimation device that estimates rainfall based on raindrops on a windshield of a vehicle, A camera that captures an image of the area in front of the vehicle through the windshield; A processor; Equipped with The processor, an image determination process for inputting an image captured by the camera to a deep neural network that has been trained on images captured in advance that have characteristics of rainfall conditions as correct answers, thereby determining whether or not the captured image is a rainfall image that shows the rainfall condition; A rainfall amount estimation process that estimates the amount of rainfall based on a time from a most recent wiping operation of the windshield wiper of the vehicle to a detection of the rainfall image by the image determination process; Run The image determination process determines whether the captured images inputted in sequence to the deep neural network at a predetermined imaging period are rainfall images, The rainfall estimation device is characterized in that the rainfall estimation process estimates that the amount of rain when the captured image that is the mth image determination process after the start of the most recent wiping operation is detected as the rainfall image is greater than the amount of rain when the captured image that is the m+nth image determination process after the start of the most recent wiping operation is detected as the rainfall image.
Citation Information
Patent Citations
Rain drop sensor and rain-drop-sensitive wiper
JP1997126998A
Raindrop detecting apparatus
JP2001153969A
Raindrop detector and wiper control device
JP2007309655A
Raindrop sensor and wiper controller having the same
US20070272884A1
