Rain detection method and device

Abstract

The invention relates to a rain detection device (110) installed in a vehicle, comprising an image acquisition module (111) configured to retrieve image data from a vehicle windshield camera (101), characterized in that it further comprises: - a sound acquisition module (112), configured to retrieve sound data from a microphone (102) in the vehicle's passenger compartment, - a perception module (113) configured to use the image and sound data to determine the presence of rain. Figure 1 (for the abstract)

Description

Title of the invention: Method and device for detecting rain Technical field

[0001] The invention relates to a device for detecting rain, in particular on glazing. Technological background

[0002] This type of device is used to detect rain on the windows of motor vehicles, in particular to enable the automatic activation of a windscreen wiper.

[0003] It is known to use a dedicated detection sensor at the windshield level. But this type of sensor increases the cost of the vehicle.

[0004] More and more vehicles are equipped with a windshield camera used by vehicle driver assistance systems. It has been proposed to use this camera to detect the presence of rain without a dedicated sensor on the windshield. Such a system is notably described in document US 2021 / 0101564. These systems are not entirely satisfactory because the reliability of the detection is lower than that of a dedicated sensor.

[0005] An aim of the present invention is to overcome all or part of the drawbacks of the prior art noted above. Summary of the invention

[0006] An object of the present invention is to propose a solution for detecting rain effectively without increasing the cost of the vehicle, in particular without adding dedicated equipment.

[0007] To this end, the invention relates to a rain detection device on board a vehicle comprising an image acquisition module configured to recover image data from a windshield camera of the vehicle, characterized in that it further comprises: - a sound acquisition module, configured with sound data from a microphone in the vehicle interior, - a perception module configured to use image data and sound data to determine the presence of rain.

[0008] By using two separate data sources, the invention makes it possible to improve the quality of rain detection. In addition, by using sensors already present in the vehicle, it makes it possible to obtain an effective detector without using dedicated equipment and therefore increasing the cost of the vehicle.

[0009] According to a first embodiment, the perception module is configured to:

[0010] - Combine the image data and the sound data,

[0011] - Determine the presence of rain from the combined data by putting in implements a deep convolutional neural network with optimized coefficients during a training step.

[0012] According to a second embodiment, the perception module is configured to:

[0013] - determine a first estimate of the presence of rain from the data image, by implementing a first deep convolutional neural network comprising coefficients optimized during a learning step,

[0014] - determine a second estimate of the presence of rain from the data sound by implementing a second deep convolutional neural network with optimized coefficients during a learning step,

[0015] - Determine the presence of rain from the first and second estimates.

[0016] According to one embodiment of the invention, the perception module is configured to determine the presence of rain from only data from one of the camera and microphone, when the other of the camera and microphone is faulty.

[0017] Advantageously, the detection module (113) emits a windshield wiper actuation signal (104) to a windshield wiper control module (103), when it determines the presence of rain.

[0018] The invention also relates to a computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to the invention.

[0019] The invention also relates to a method for detecting rain by a device on board a vehicle comprising steps of:

[0020] - Acquisition of image data from a vehicle windshield camera,

[0021] - Acquisition of sound data from at least one microphone in the passenger compartment of the vehicle,

[0022] - Determination, by a perception module, of the presence of rain from the image data and sound data.

[0023] The invention also relates to a vehicle comprising a rain detection device according to the invention. Brief description of the figures

[0024] Other characteristics and advantages of the invention will emerge from the description of the non-limiting embodiments of the invention below, with reference to the appended figures, in which:

[0025] [Fig-1] functionally illustrates a device, according to an exemplary embodiment particular of the present invention;

[0026] [Fig.2] schematically illustrates a particular exemplary embodiment of the present invention;

[0027] [Fig.l] shows an exemplary embodiment of a rain detection device according to the invention.

[0028] The rain detection device 110, on board a vehicle, comprises an image acquisition module 111 configured to recover image data from a windshield camera 101 of the vehicle.

[0029] The rain detection device 110 also comprises a sound acquisition module 112, configured with sound data coming from a microphone 102 in the passenger compartment of the vehicle.

[0030] The rain detection device 110 also comprises a perception module 113 configured to use the image data and the sound data to determine the presence of rain.

[0031] When it detects the presence of rain, the perception module 113 emits a signal to a windshield wiper controller 103 to control the actuation of the vehicle's windshield wipers 104.

[0032] According to a first embodiment, called early fusion, perception module 113 is configured to:

[0033] - Combine image data and sound data,

[0034] - Determine the presence of rain from the combined data by putting in implements a deep convolutional neural network with optimized coefficients during a training step.

[0035] The perception module 113 relies, for example, on a learning approach which makes it possible to train a neural network to detect the presence of rain. During a learning step, the neural network evolves to learn to recognize a set of characteristic elements which make it possible to confirm the presence of rain.

[0036] The learning step includes capturing image data and sound data representative of the presence or absence of rain.

[0037] These acquired data are optionally processed to format and code them according to a determined format and coding. Each combination of image data and sound data is associated with the presence or absence of rain to constitute a learning base. The learning implements a deep convolutional neural network iteratively receiving as input a pair of data (image data and sound data / presence of rain) from the learning base and optimizing the coefficients of the neural network.

[0038] The coefficients optimized in the learning step are entered as parameters into a vehicle computer implementing said neural network. and which, based on the data acquired by the camera and the microphone and the optimized coefficients, provides an estimate of the type of rain presence.

[0039] According to a second embodiment, called late fusion, the perception module 113 is configured to:

[0040] - determine a first estimate of the presence of rain from the data image, by implementing a first deep convolutional neural network comprising coefficients optimized during a learning step,

[0041] - determine a second estimate of the presence of rain from the data sound by implementing a second deep convolutional neural network with optimized coefficients during a learning step,

[0042] - Determine the presence of rain from the first and second estimates.

[0043] The first estimate is obtained with a learning approach that allows the first neural network to be trained to detect the presence of rain. During a learning step, the first neural network evolves to learn to recognize a set of characteristic elements that allow the presence of rain to be confirmed from image data. The way of obtaining the learning information may differ (while being compatible) with the way of using the network on the vehicle (for example, the network may be trained with data from a 'lab' type camera, and a camera optimized for the vehicle may be used for the use of the first neural network).

[0044] The learning step comprises capturing a plurality of images representative of the presence or absence of rain on the windshield of the vehicle;

[0045] These acquired images are optionally processed to format and code them according to a determined format and coding. Each image is associated with the presence or absence of rain to constitute a learning base. The learning implements a first deep convolutional neural network iteratively receiving as input a pair of data (captured image, presence of rain) from the learning base and optimizing the coefficients of the first neural network.

[0046] The coefficients optimized in the learning step are entered as parameters into a computer of the vehicle implementing said first neural network and which, from the images acquired by the vehicle's windshield camera and the optimized coefficients, delivers an estimate of the type of presence of rain.

[0047] As for the first estimate, the second estimate is obtained with a learning approach which makes it possible to train the second neural network to detect the presence of rain. During a learning step, the second neural network evolves to learn to recognize a set of characteristic elements which make it possible to confirm the presence of rain from sound data. The way of obtaining the learning information may differ (while being compatible) with the way of using the network on the vehicle (for example, we can train the second network with data from a 'lab' type microphone, and use a microphone optimized for the vehicle for the use of the second neural network).

[0048] The learning step comprises capturing a plurality of sound samples representative of the presence or absence of rain outside the vehicle;

[0049] These acquired sound samples are optionally processed to format and encode them according to a determined format and encoding. Each sound sample is associated with the presence or absence of rain to constitute a learning base. The learning implements a second deep convolutional neural network iteratively receiving as input a pair of data (captured sound sample, presence of rain) from the learning base and optimizing the coefficients of the second neural network.

[0050] The coefficients optimized in the learning step are entered as parameters into a computer of the vehicle implementing said second neural network and which, from the sound samples acquired by the microphone in the passenger compartment of the vehicle and the optimized coefficients, delivers an estimate of the type of presence of rain.

[0051] The detection module 113 is configured to use the first and second estimates to determine the presence of rain. This module uses, for example, a data fusion method according to the known art. The first and second neural networks each produce, for example, an output vector. It is then possible to calculate the distance between the two vectors, and then, from this distance, to make a classification.

[0052] When it detects the presence of rain, the detection module 113 transmits a signal to a windshield wiper controller 103 to control the actuation of the vehicle's windshield wipers 104.

[0053] According to a characteristic of the invention, the perception module 113 is configured to determine the presence of rain from only the data from one of the camera and the microphone, when the other of the camera and the microphone is defective.

[0054] In other words, when the camera is faulty, the perception module is capable of operating in a degraded manner and determining the presence of rain only with the sound data coming from the microphone. Conversely, when the microphone is faulty, the perception module is capable of operating in a degraded manner and determining the presence of rain only with the image data coming from the camera.

[0055] The invention also relates to a method for detecting rain by a device 110 on board a vehicle comprising steps of: - Acquisition of image data from a vehicle windshield camera, - Acquisition of sound data from a microphone in the vehicle interior, - Determination, by a perception module 113, of the presence of rain from image data and sound data.

[0056] [Fig. 2] represents an example of a rain detection device 301 included in the vehicle, in a network (“cloud”) or in a server. This device 301 can be used as a centralized device in charge of at least certain steps of the method described below with reference to [Fig. 2].

[0057] This device 301 can take the form of a box comprising printed circuits, any type of computer or even a mobile telephone (“smartphone”).

[0058] The device 301 comprises a random access memory 302 for storing instructions for the implementation by a processor 303 of at least one step of the method as described above. The device also comprises a mass memory 304 for storing data intended to be retained after the implementation of the method.

[0059] The device 301 may further comprise a digital signal processor (DSP) 305. This DSP 305 receives data to format, demodulate and amplify, in a manner known per se, this data.

[0060] The device 301 also comprises an input interface 306 for receiving the data implemented by the method according to the invention and an output interface 307 for transmitting the data implemented by the method according to the invention.

Claims

Claims

1. Rain detection device (110) on board a vehicle comprising an image acquisition module (111) configured to retrieve image data from a windshield camera (101) of the vehicle, characterized in that it further comprises: - a sound acquisition module (112), configured to retrieve sound data from a microphone (102) in the passenger compartment of the vehicle, - a perception module (113) configured to use the image data and the sound data to determine the presence of rain, the perception module (113) being configured to: - determine a first estimate of the presence of rain from the image data, by implementing a first deep convolutional neural network comprising coefficients optimized during a learning step,- determine a second estimate of the presence of rain from the sound data by implementing a second deep convolutional neural network with optimized coefficients during a learning step, - Determine the presence of rain from the first and second estimates.,

2. A rain detection device according to one of the preceding claims, wherein the perception module (113) is configured to determine the presence of rain from only the data from one of the camera or the microphone, when the other of the camera or the microphone is faulty.

3. Rain detection device according to one of the preceding claims, in which the detection module (113) transmits a signal for actuating the windshield wipers (104) to a windshield wiper control module (103), when it determines the presence of rain.

4. Method for detecting rain by a device (110) on board a vehicle comprising steps of: - Acquisition of image data from a windshield camera of the vehicle, - Acquisition of sound data from at least one microphone in the passenger compartment of the vehicle,

5.

6. - Determination, by a perception module (113), of the presence of rain from the image data and the sound data, this step comprising a determination of a first estimate of the presence of rain from the image data, by implementing a first deep convolutional neural network comprising coefficients optimized during a learning step, a determination of a second estimate of the presence of rain from the sound data by implementing a second deep convolutional neural network comprising coefficients optimized during a learning step, a determination of the presence of rain from the first and second estimates. Computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to the preceding claim. Vehicle comprising a rain detection device according to one of claims 1 to 4.

Images