Reducing environmental noise in vehicle sensor data

Abstract

Disclosed is a control system, a vehicle, a method, and to a computer readable medium for reducing environmental noise present in sensor data. The sensor data includes at least one type of environment

Description

TECHNICAL FIELD The present disclosure relates to a method and apparatus for reducing environmental noise in vehicle sensor data. Aspects of the invention relate to a control system, to a vehicle, to a method, and to a computer readable medium. BACKGROUND Environmental factors may degrade the quality of sensor data provided by sensors of a vehicle which may reduce the accuracy or utility of the sensor data. Certain weather conditions may impact the ability of the sensor to provide clear sensor data, for example due to water droplets collecting on a camera lens, or reduced visibility due to fog. Examples of weather conditions that may degrade the ability a sensor to provide clear sensor data include rain, snow, sun glare, fog, hail, and / or water spray. Furthermore, other environmental conditions such as low light levels, the vehicle passing through a tunnel, presence of road furniture, screen debris, and / or general air obstructions including smoke or dust, may degrade the accuracy or clarity of vehicle sensor data. The presence of environmental noise in the sensor data may present a potential risk to the accuracy and intended operation of advanced driver assistance systems provided on a vehicle. To avoid unintended operation of ADAS features caused by sensor data including environmental noise, ADAS features may typically be disabled in the presence of significant sources of environmental noise. It is an aim of the present invention to address one or more of the disadvantages associated with the prior art. SUMMARY OF THE INVENTION Aspects and embodiments of the invention provide a control system, a method of reducing environmental noise present in sensor data, a computer program product and a vehicle as claimed in the appended claims. According to an aspect of the present invention there is provided a method of reducing environmental noise present in sensor data, the sensor data including at least one type of environmental noise from a plurality of predetermined types of environmental noise, the method comprising processing sensor data using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data, the classifier having been trained to classify sensor data according to a type of environmental noise of the plurality of predetermined types of environmental noise, based on the classification of the sensor data, selecting a de-noising module from a plurality of de-noising modules, each de-noising module comprising a machine-learning algorithm trained to reduce the effect on the sensor data of one type of environmental noise of the plurality of predetermined types of environmental noise, and processing the sensor data using the selected de-noising module to reduce an amount of environmental noise present in the sensor data. Advantageously, by classifying the type of environmental noise present in the sensor data and selectively applying a de-noising module according to the type of environmental noise present, de-noising modules that are dedicated to the removal / reduction of a particular type of environmental noise can be used resulting in improved overall signal-to-noise ratio of the processed sensor data. Optionally, the method comprises receiving the sensor data, from a sensor of a vehicle, wherein the sensor comprises one of a camera, a radar sensor, and a lidar sensor. Optionally, the method comprises, in response to the sensor data being classified as including a type of environmental noise associated with an obstruction on the sensor, providing an indication to a sensor cleaning apparatus to perform a sensor cleaning operation. Optionally, wherein the plurality of types of environmental noise comprise rain, sleet, snow, glare, fog, water droplets, hail, glare, camera flare, obstruction due to traffic furniture, low light levels, and / or dirt or dust present on a sensor. According to another aspect of the present invention there is provided a control system for reducing environmental noise present in sensor data obtained from a sensor of a vehicle, the control system comprising one or more processors collectively configured: to receive sensor data from a sensor of the vehicle; process the sensor data using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data, the classifier having been trained to classify sensor data according to a type of environmental noise of the plurality of predetermined types of environmental noise; based on the classification of the sensor data, select a de-noising module from a plurality of de-noising modules, each de-noising module comprising a machine-learning algorithm trained to reduce the effect on the sensor data of one type of environmental noise of the plurality of predetermined types of environmental noise; and process the sensor data using the selected de-noising module to generate processed sensor data having a reduced amount of environmental noise. Optionally, the one or more processors are collectively configured to output the processed sensor data to an advanced driving assistance system of the vehicle. Optionally, the sensor comprises one of a camera, a radar sensor, and a lidar sensor. Optionally, the one or more processors are collectively configured to provide an indication to a sensor cleaning apparatus to perform a sensor cleaning operation, in response to a classification of the sensor data being as including a type of environmental noise associated with an obstruction on the sensor. Optionally, the plurality of types of environmental noise comprise rain, sleet, snow, glare, fog, water droplets, hail, glare, camera flare, obstruction due to traffic furniture, low light levels, and / or dirt or dust present on a sensor. According to further aspect of the invention, there is provided a vehicle including a sensor and the control system as described above. According to a yet further aspect of the invention, there is provided a computer program product comprising computer program code that when executed on a processor of a control system of a vehicle causes the control system to perform a method as described above. Within the scope of this application it is expressly intended that the various aspects, embodiments, examples and alternatives set out in the preceding paragraphs, in the claims and / or in the following description and drawings, and in particular the individual features thereof, may be taken independently or in any combination. That is, all embodiments and / or features of any embodiment can be combined in any way and / or combination, unless such features are incompatible. The applicant reserves the right to change any originally filed claim or file any new claim accordingly, including the right to amend any originally filed claim to depend from and / or incorporate any feature of any other claim although not originally claimed in that manner. BRIEF DESCRIPTION OF THE DRAWINGS One or more embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which: Figure 1 shows a vehicle including a control system in accordance with embodiments of the invention; Figure 2 shows a method of processing sensor data according to an embodiment of the invention; Figure 3 shows a generative adversarial network suitable for implementing a de-noising module according to embodiments of the invention; Figure 4 illustrates a combined classifier / generator according to an embodiment of the invention; Figure 5 shows example image sensor data including environmental noise; Figure 6 shows example radar and lidar sensor data including environmental noise; Figure 7 shows a method of reducing environmental noise in sensor data; and Figure 8 shows a device comprising a computer-readable storage medium s. DETAILED DESCRIPTION According to embodiments of the invention, a control system may process sensor data obtained from a sensor of a vehicle to reduce an amount of environmental noise present in the sensor data. In particular, the control system processes the sensor data in a classifier algorithm trained to identify a type of environmental noise present in the sensor data as comprising at least one of a plurality of predetermined types of environmental noise. Based on the output of the classifier algorithm, a sensor data de-noising module is selected corresponding to the type of environmental noise determined to be present in the sensor data and used to process the sensor data to reduce the environmental noise present. With reference to Figure 1, there is illustrated a vehicle 100 comprising a control system 106 and at least one sensor 102 coupled to the control system. The control system 106 is communicatively coupled to the at least one sensor 102 to receive sensor data 104 output by the at least one sensor 102. The control system 106 may implement one or more autonomous driving functions (e.g. Advanced Driver Assistance Systems - ADAS), for example active emergency braking, lane keep assist, adaptive cruise control, collision warning, blind spot warning, etc. based on the sensor data 104 provided by the one or more sensors 102. This may be achieved by providing control signals (not illustrated) to one or more actuators or other systems present on the vehicle 100. The control system 106 as illustrated in Figure 1 comprises one controller, although it will be appreciated that this is merely illustrative. The controller 106 comprises processing means 108 and memory means 110. The processing means 108 may be one or more electronic processing devices which operably execute computer-readable instructions. The memory means 110 may be one or more memory devices. The memory means 110 is electrically coupled to the processing means 108. The memory means 110 is configured to store instructions, and the processing means 108 is configured to access the memory means 110 and execute the instructions stored thereon. The controller 106 comprises an input means and an output means. The input means may comprise an electrical input of the controller 106, for example to receive signal data 104. The output means may comprise an electrical output of the controller 106. During operation of the vehicle 100, one or more environmental factors may impact on the operation of the at least one sensor 102 to reduce the accuracy or utility of the sensor data 104. Certain weather conditions may impact the ability of the sensor to provide clear sensor data, for example due to water droplets collecting on a camera lens, or reduced visibility due to fog. Examples of weather conditions that may degrade the ability of the at least one sensor 102 to provide clear sensor data include rain, snow, sun glare, fog, hail, and / or water spray. Furthermore, other environmental conditions such as low light levels, the vehicle passing through a tunnel, presence of road furniture, screen debris, and / or general air obstructions including smoke or dust, may degrade the accuracy or clarity of the sensor data 104. The presence of environmental noise in the sensor data 104 may present a potential risk to the accuracy and intended operation of ADAS features provided by the control system 106. To avoid unintended operation of ADAS features caused by sensor data 104 that includes environmental noise, ADAS features may typically be disabled when environmental noise that may be expected to impact the operation of the ADAS features is detected. According to embodiments of the invention, particular types of environmental noise that are present in sensor data 104 may be detected and a corresponding de-noising module applied to the sensor data to reduce the level of noise, i.e. increase the signal-to-noise ratio present in the sensor data, to allow for normal operation of the ADAS systems of the vehicle based on the signal data 104 even in the presence of environmental noise that might previously have resulted in ADAS features being disabled. Figure 2 illustrates a method 200 of processing sensor data to reduce the level of environmental noise present in the sensor data according to an embodiment of the invention. According to embodiments, the method 200 illustrated in Figure 2 may be performed by the control system 106 of Figure 1. According to the method 200, sensor data 104 is received from the at least one sensor and provided to a classifier 202. Classifier 202 comprises a classification algorithm trained to identify a type of environmental noise present in the sensor data 104 and to output a classification result based on the identified type of environmental noise. For example, classifier 202 may be operable to process an image provided by a camera and determine the presence of water droplets obscuring portions of the image. Training of the classifier 202 may be achieved through the use of a training set comprising labelled examples of sensor data including a plurality of examples of sensor data containing a type of environmental noise of the plurality of types of environmental noise that the classifier 202 is to identify. The training set may include examples of sensor data including no environmental noise, one type of environmental noise, and / or multiple types of environmental noise, with each example annotated to reflect the presence and type of environmental noise present in the sensor data. The classifier 202 may comprise a neural network, for example a residual neural network (ResNet), including a plurality of layers of nodes having associated weights. A method of training of the classifier 202 may comprise applying an example of sensor data from the training set to obtain a classification result; characterizing an error between the classification result and the type of environmental noise according to the labelled example; and updating weights of the neural network using an optimization algorithm based on the characterized error. In some embodiments the classifier 202 may be implemented as a plurality of binary classifiers, each of the plurality of binary classifiers trained to identify the presence or absence of a single type of environmental noise present in the sensor data 104. Based on the classification result, a de-noising module is selected and the sensor data 104 is processed in the selected de-noising module to reduce an amount of environmental noise present in the sensor data. For example, in the example embodiment of Figure 2, four de-noising modules 204, 206, 208 and 210 are illustrated, however it will be recognized that different numbers of de-noising modules may be implemented according to the number of different types of environmental noise that the method 200 is to reduce. As an illustrative example, the method 200 may include de-droplet / dirt module 204; de-glare / exposure correction module 206; de-rain module 208; and de-haze module 210. Following processing of the sensor data 104 in the selected de-noising algorithm, de-noised sensor data 212 is generated and may be used to provide the one or more ADAS features present on the vehicle 100. The use of a plurality of de-noising modules each dedicated to reduction of a specific type, or similar types, of environmental noise present in the sensor data 104 has been found to provide improved performance, i.e. improved signal-to-noise in the processed sensor data, as compared to a general purpose de-noising algorithm intended to reduce multiple kinds of noise present in the sensor data. Thus, by classifying the type of noise present in the sensor data and then selectively applying the corresponding de-noise module may result in de-noised sensor data 212 having improved signal-to-noise characteristics. According to some embodiments, in the case that the sensor data 104 is classified as including a type of environmental noise associated with an obstruction on a sensor, for example the presence of water droplets or dirt on a camera lens, the control system 106 may provide a signal to a sensor cleaning apparatus of the vehicle 100, such as a wash / wipe system associated with a camera lens, to perform a sensor cleaning operation to attempt to remove the source of the environmental noise. Figure 3 illustrates a generative adversarial network (GAN) 300 that can be used to train and implement a de-noise module 204, 206, 208, 210 as illustrated in Figure 2. The GAN comprises a generator 302 coupled to a discriminator 308. As part of a training process for the GAN 300, the generator 302 receives sensor data including environmental noise of a particular type corresponding to a type of environmental noise to be reduced by a de-noise module. The generator 302 processes the sensor data and generates a predicted output 304. The discriminator 308 receives the predicted output 304 along with a ground truth input 306, i.e. sensor data not including the particular type of environmental noise and corresponding to the sensor data 104. The discriminator attempts to classify the input data 304 to the discriminator to generate output 310 indicating the received data is real or synthesized, that is, whether the generated data 304 sufficiently matches the ground truth data 306. The generator 302 and discriminator 308 may be trained together with a loss function for the generator 302 based on whether the output 304 of the generator 302 ‘fools’ the classifier 308 (that is, passes its test), while discriminator 308 is trained on a loss function based on successfully identifying which of the received inputs is real (306) or fake (304). In this way, the generator 302 is trained to generate an output 304 that falls within a realistic range of possibilities for de-noised sensor data corresponding to the sensor data 104. Once trained in this way, the generator 302 can be used as a de-noising module 204, 206, 208, 210 for the particular type of environmental noise present in the training input. By training multiple generators for each of the types of environmental noise to be reduced, a plurality of de-noise modules can be provided to address the different types of environmental noise. In an embodiment illustrated in Figure 4, a generator 400 corresponding to a combined classifier 202 and one of the de-noising modules 204, 206, 208,210, can be trained to provide an output indicating whether the particular type of environmental noise associated with the de-noising module is present in the input sensor data 104. For example, generator 400 may be trained to reduce glare and correct exposure issues and may provide an indication that glare and / or exposure issues are present / not present in the sensor data 104. Thus, a de-noising module including generator 400 as illustrated in Figure 4 may provide both de-noising of the sensor data as well as classifying the sensor data 104 as including or not including a particular type of environmental noise. In the case that the de-noising module determines that the sensor data 104 does not include a type of environmental noise associated with a particular de-noising module, that de-noising module may halt processing of the sensor data and refrain from generating a corresponding output 304. Alternatively, if its classifier function fails to detect noise of the type that its de-noising function is adapted to handle, its output 304 may simply be the sensor data input 104 unchanged. That is, multiple generators 400 arranged in parallel, each with a different de-noising function according to a particular type of noise that may be present in the sensor data, can correspond with the arrangement in Figure 2. In embodiments, the at least one sensor 102 may be a camera and the sensor data 104 may be an image frame provided by the camera. Figure 5 illustrates examples of image data including environmental noise along with corresponding de-noised images. In particular, a first image 502 provides an illustrative example of environmental noise associated with water droplets on a camera lens used to capture the image data, while a second image 504 illustrates the same scene having removed or reduced the image artifacts caused by the water droplets. Similarly a third image 506 provides an illustrative example of glare, including camera flare, due to a strong light source present within the field of view of the camera, while a fourth image 508 illustrates the same scene having removed or reduced the glare and camera flare from the image data. In some embodiments, the sensor 102 may comprise a radar sensor, a lidar sensor, etc. Figure 6 provides illustrative examples of radar sensor data 602 and lidar sensor data 606 including environmental noise associated with one or more obstructions present, and de-noised radar sensor data 604 and lidar sensor data 608. It will be recognized that a training data set can be compiled for a particular sensor type by collecting multiple examples of each type of environmental noise, such as illustrated in Figures 5 and 6. The training data set may then be used to train a plurality of de-noise modules as discussed above. Figure 7 illustrates a method 700 according to an embodiment of the invention. The method 700 is a method of reducing environmental noise in sensor data captured by a sensor of a vehicle, such as the vehicle 100 illustrated in Figure 1. According to the method 700, at block 702, sensor data received from a sensor 102 of the vehicle 100 is processed using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data. The classifier has been trained, as discussed above, to classify the sensor data as including a particular type of environmental noise of a plurality of types of environmental noise that may be present in the sensor data. At block 704, based on the output of the classifier indicating the presence of a particular type of environmental noise, a de-noising module is selected from a plurality of de-noising modules, the selected de-noising module comprising a machine-learned algorithm that has been trained to reduce the effect of the particular type of environmental noise indicated to be present in the sensor data by the classifier. The sensor data 104 is then processed using the selected de-noising module at block 706 to generate output sensor data having reduced environmental noise that can be used as input to one or more ADAS features of the vehicle 100. The method 700 may be performed by the control system 106 as illustrated in Figure 1. In particular, the memory 110 may comprise computer-readable instructions which, when executed by the processor 108, perform the method 100 according to an embodiment of the invention. In some embodiments, the sensor data 104 may be affected by more than one type of the plurality of predetermined types of environmental noise. In such embodiments, the classifier 202 may classify the sensor data 104 as including each type of environmental noise present and in response multiple de-noising modules may be selected. The sensor data 104 may then be processed by each of the selected de-noising modules. Processing may be performed in series, with the output of a first de-noising module provided as the input to a subsequent de-noising module until the data has been processed by all of the selected de-noising modules. Alternatively, the sensor data may be processed by each selected de-noising module in parallel and the resulting processed sensor data provided by each de-noising module combined to provide final output data 212. With respect to the embodiment illustrated in Figure 4, sensor data may be supplied to each of the de-noising modules, which each includes, first, the classification function to determine the presence of the type of noise it has been trained to handle, and second its de-noising function, which is only implemented if the classification function determines the presence of the type of noise the de-noising function has been trained to handle. Where the de-noising modules are in series, the sensor data 104 is input to the first de-noising module and the output of each module 400 may therefore be either a denoised data set, if the noise appropriate to that module is detected in its input, or an unchanged data set. Where the modules are in parallel, each is supplied with the original sensor data 104 and if none of them detect noise of the type which it is trained to handle, the original data set 104 is implemented by the subsequent ADAS or other system. If one or more of them detect appropriate noise, the input to the subsequent ADAS or other system is the output of the one de-noising module that detected noise it is trained to handle, ora combined output if multiple de-noising modules detect appropriate noise. Certain methods and systems as described herein may be implemented by one or more processors that process program code that is retrieved from a non-transitory storage medium. Figure 8 shows an example 800 of a device comprising a computer-readable storage medium 820 coupled to at least one processor 810. The computer-readable media 820 can be any media that can contain, store, or maintain programs and data for use by or in connection with an instruction execution system. Computer-readable media can comprise any one of many physical media such as, for example, electronic, magnetic, optical, electromagnetic, or semiconductor media. More specific examples of suitable machine-readable media include, but are not limited to, a hard drive, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory, ora portable disc. In Figure 8, the computer-readable storage medium comprises program code to perform a method corresponding to the embodiment shown in Figure 7, that is: processing 702 sensor data using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data; selecting 704, based on the classification of the sensor data, a de-noising module from a plurality of de-noising modules; and processing 706 the sensor data 104 using the selected de-noising module. It will be appreciated that various changes and modifications can be made to the present invention without departing from the scope of the present application.

Claims

1. A method of reducing environmental noise present in sensor data, the sensor data including at least one type of environmental noise from a plurality of predetermined types of environmental noise, the method comprising:processing sensor data using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data, the classifier having been trained to classify sensor data according to a type of environmental noise of the plurality of predetermined types of environmental noise;based on the classification of the sensor data, selecting a de-noising module from a plurality of de-noising modules, each de-noising module comprising a machine-learning algorithm having been trained to reduce the effect on the sensor data of one type of environmental noise of the plurality of predetermined types of environmental noise; andprocessing the sensor data using the selected de-noising module to reduce an amount of environmental noise present in the sensor data.

2. The method of claim 1, further comprising receiving the sensor data, from a sensor of a vehicle, wherein the sensor comprises one of a camera, a radar sensor, and a lidar sensor.

3. The method of claim 2, further comprising, in response to the sensor data being classified as including a type of environmental noise associated with an obstruction on the sensor, providing an indication to a sensor cleaning apparatus to perform a sensor cleaning operation.

4. The method of any preceding claim, wherein the plurality of types of environmental noise comprise rain, sleet, snow, glare, fog, water droplets, hail, glare, camera flare, obstruction due to traffic furniture, low light levels, and / or dirt or dust present on a sensor.

5. A control system for reducing environmental noise present in sensor data obtained from a sensor of a vehicle, the control system comprising one or more processors collectively configured to:receive sensor data from a sensor of the vehicle;process the sensor data using a classifier to classify the sensor data according to at least one type of environmental noise present in the sensor data, the classifier having been trained to classify sensor data according to a type of environmental noise of the plurality of predetermined types of environmental noise;based on the classification of the sensor data, select a de-noising module from a plurality of de-noising modules, each de-noising module comprising a machine-learning algorithm trained to reduce the effect on the sensor data of one type of environmental noise of the plurality of predetermined types of environmental noise; andprocess the sensor data using the selected de-noising module to generate processed sensor data having a reduced amount of environmental noise.

6. The control system of claim 5, wherein the one or more processors are collectively configured to output the processed sensor data to an advanced driving assistance system of the vehicle.

7. The control system of claim 5 or claim 6, wherein the sensor comprises one of a camera, a radar sensor, and a lidar sensor.

8. The control system of any of claims 5 to 7, wherein the one or more processors are collectively configured to provide an indication to a sensor cleaning apparatus to perform a sensor cleaning operation, in response to a classification of the sensor data being as including a type of environmental noise associated with an obstruction on the sensor.

9. The control system of any of claims 5 to 8, wherein the plurality of types of environmental noise comprise rain, sleet, snow, glare, fog, water droplets, hail, glare, camera flare, obstruction due to traffic furniture, low light levels, and / or dirt or dust present on a sensor.

10. A vehicle comprising:at least one sensor; anda control system according to any of claims 6 to 11, the control system being communicatively coupled to the sensor.

11. A computer program product comprising computer program code that when executed ona processor of a control system of a vehicle causes the control system to perform the method of any of claims 1 to 4.

Images