Camera contamination detection using attention-guided camera depth and LIDAR ranging consistency gating

Abstract

A method includes receiving a plurality of images, where a first image of the one or more images includes a ranging image and a second image includes a camera image; and filtering the first image to generate a filtered first image. The method further includes generating a plurality of depth estimates based on the second image; and generating an attention map by combining the filtered first image and the plurality of depth estimates. Additionally, the method includes generating a consistency score indicative of depth estimate consistency between the first image and the second image based on the attention map; modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate modulated one or more features; and generating a classification of the one or more contaminated regions in the second image based on the modulated one or more features.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 464,769, filed on September 11, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to cognitive neural networks. Background Technology

[0003] Automated and semi-automated vehicles can utilize artificial intelligence and machine learning, particularly deep neural networks (DNNs), to perform any number of operations for maneuvering, driving, and navigating the vehicle. For example, DNNs can be used for object detection, lane and road boundary detection, safety analysis, drivable free space analysis, control generation during vehicle maneuvers, and / or other operations. Before any automated or semi-automated vehicle can safely navigate on a road, the DNNs and other software enabling the vehicle's autonomous driving are typically tested to verify and confirm their safe performance. More specifically, DNN-driven automated and semi-automated vehicles should be able to respond appropriately to an extremely diverse range of situations, including interactions with emergency vehicles, pedestrians, animals, and virtually countless other obstacles.

[0004] For autonomous vehicles to achieve Level 3 to 5 of autonomous driving (e.g., conditional automation (Level 3), high automation (Level 4), and full automation (Level 5)), they must be able to operate safely in all environments and without human intervention in the event of potentially unsafe situations. Advanced Driver Assistance Systems (ADAS) use sensors and software to help vehicles avoid hazardous situations, ensuring safety and reliability. Cameras are an essential part of the sensor suite for achieving Level 3 autonomous driving because they provide a high-resolution view of the surrounding environment. Cameras allow autonomous vehicles to see other vehicles, pedestrians, and obstacles and make decisions about how to navigate safely. Surround-view cameras are a type of camera mounted on the exterior of autonomous vehicles. Surround-view cameras provide a 360-degree view of the vehicle's surroundings, which is crucial for autonomous driving. However, surround-view cameras are also directly exposed to the external environment, meaning they can be contaminated by rain, fog, or snow. Summary of the Invention

[0005] Generally speaking, this disclosure describes techniques for camera contamination detection. Surround-view cameras are directly exposed to the external environment, meaning they are susceptible to contamination from rain, fog, snow, dust, and mud. Contamination can significantly affect camera performance, making it difficult for autonomous vehicles to see other vehicles, pedestrians, and obstacles. For example, rain can cause water droplets to form on the camera lens, which can blur the image. Fog can also cause image blurring, and snow can completely obstruct the view. Dust and mud can also interfere with the camera's ability to see clearly.

[0006] This disclosure describes example techniques for more accurately detecting contamination on cameras. Specifically, in one or more examples, the example techniques involve using ranging images, such as those generated from a light detection and ranging (LIDAR) sensor, to facilitate contamination detection. A LIDAR sensor uses a laser to measure the distance to objects in the environment. This allows the LIDAR sensor to create a three-dimensional (3D) map of the surrounding area, which can be used for various purposes, including autonomous driving. The ranging image can be obtained from the LIDAR sensor. A ranging image is a two-dimensional (2D) image representing the distance to objects in a scene. The ranging image can be obtained by using the LIDAR sensor to measure the time it takes for a laser pulse to travel to and return from the object.

[0007] Depth estimation can be obtained from the camera's depth decoder. The encoder, part of the camera, is used to extract features from the image. These features are then used to train a model that can estimate the depth of objects in a scene. The extracted features can then be used in a feature gating module to determine which features should be used to estimate the scene's depth. The depth estimates from the camera and LiDAR sensor can then be combined to produce a single depth estimate of the scene. Next, a distance-based segmentation technique generates an attention map that highlights areas where the camera is likely to be contaminated. Advantageously, the disclosed real-time contamination detection technique improves the safety of autonomous driving systems. As another non-limiting advantage, the disclosed machine learning technique is computationally efficient.

[0008] In one example, a method includes: receiving a plurality of images, wherein a first image of one or more images includes a ranging image, and a second image includes a camera image; and filtering the first image by one or more processors to generate a filtered first image. The filtering includes filters configured to fill one or more sparse regions in the first image. The method also includes generating multiple depth estimates by one or more processors based on the second image; and generating an attention map by one or more processors by combining the filtered first image and the multiple depth estimates. Additionally, the method includes: generating a consistency score indicating consistency between the depth estimates of the first and second images by one or more processors based on the attention map; and modulating one or more features extracted from the second image using a gating mechanism based on the consistency score to generate one or more modulated features. Finally, the method includes generating a classification of one or more contaminated regions in the second image by one or more processors based on the modulated features.

[0009] In another example, an apparatus for camera contamination detection includes: a memory for storing multiple images; and processing circuitry in communication with the memory. The processing circuitry is configured to: receive multiple images, wherein a first image in one or more of the images comprises a ranging image, and a second image comprises a camera image; and filter the first image to generate a filtered first image. The filtering includes filters configured to fill one or more sparse regions in the first image. The processing circuitry is further configured to: generate multiple depth estimates based on the second image; and generate an attention map by combining the filtered first image and the multiple depth estimates. Additionally, the processing circuitry is configured to: generate a consistency score based on the attention map, indicating consistency between the depth estimates of the first and second images; and modulate one or more features extracted from the second image using a gating mechanism based on the consistency score to generate one or more modulated features. Finally, the processing circuitry is configured to generate a classification of one or more contamination regions in the second image based on the modulated features.

[0010] In yet another example, a computer-readable medium includes instructions that, when applied by processing circuitry, cause the processing circuitry to: receive a plurality of images, wherein a first image of one or more of the images comprises a ranging image and a second image comprises a camera image; and filter the first image to generate a filtered first image. The filtering includes filters configured to fill one or more sparse regions in the first image. Additionally, the instructions cause the processing circuitry to: generate a plurality of depth estimates based on the second image; generate an attention map by combining the filtered first image and the plurality of depth estimates; generate a consistency score based on the attention map indicating consistency between the depth estimates of the first and second images; modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate one or more modulated features; and generate a classification of one or more contaminated regions in the second image based on the modulated features.

[0011] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0012] Figure 1 This is an illustration of an example of an autonomous vehicle based on the technology disclosed herein.

[0013] Figure 2 This is a block diagram illustrating an example system that can perform the techniques of this disclosure.

[0014] Figure 3 This is a block diagram illustrating an example pollution detection framework that can perform the techniques of this disclosure.

[0015] Figure 4 These are illustrations illustrating various non-limiting examples of contamination detection in images according to the techniques of this disclosure.

[0016] Figure 5 This is a flowchart illustrating an example method for classifying contaminated areas in an image according to the technology of this disclosure. Detailed Implementation

[0017] For autonomous vehicles to achieve Level 3 to 5 of autonomous driving (e.g., conditional automation (Level 3), high automation (Level 4), and full automation (Level 5)), they must be able to operate safely in all environments and without human intervention in the event of potentially unsafe situations. Advanced Driver Assistance Systems (ADAS) use sensors and software to help vehicles avoid hazardous situations, ensuring safety and reliability. Cameras are an essential part of the sensor suite for achieving Level 3 autonomous driving because they provide a high-resolution view of the surrounding environment. Cameras allow autonomous vehicles to see other vehicles, pedestrians, and obstacles and make decisions about how to navigate safely. Surround-view cameras are a type of camera mounted on the exterior of autonomous vehicles. Surround-view cameras provide a 360-degree view of the vehicle's surroundings, which is crucial for autonomous driving. However, surround-view cameras are also directly exposed to the external environment, meaning they can be contaminated by rain, fog, or snow.

[0018] Dust and mud can significantly impact the performance of vision tasks because they can block light from reaching camera sensors. This obstruction can make it difficult for the camera to see objects in a scene, potentially leading to errors in vision tasks such as object detection, tracking, and segmentation. Dust and mud can also scatter light, making it difficult for the camera to focus on objects. Scattered light can also cause errors in vision tasks. Additionally, dust and mud can alter the color of objects in a scene, which can also make it difficult for the camera to identify objects. Cameras experience a much higher performance degradation due to contamination compared to other sensors because they rely on light to see. Other sensors, such as radar and LiDAR, do not rely on light to see and are therefore unaffected by contamination. Radar uses radio waves to measure distances to objects, and LiDAR sensors use lasers to measure distances to objects. Both sensors can still function even if the camera lens is contaminated. The performance degradation due to contamination can be particularly severe for cameras used in autonomous driving, as autonomous vehicles need to be able to see clearly for safe navigation. For higher levels of autonomous driving, such as Level 4 and Level 5, autonomous vehicles are expected to operate without any human input. In other words, even in challenging conditions such as rain, fog, or snow, the sensors in autonomous vehicles should be able to accurately detect and avoid obstacles. If cameras are contaminated, they may not be able to see obstacles clearly, which could lead to accidents.

[0019] This disclosure describes example techniques for more accurately detecting contamination on cameras. In one or more examples, the example techniques include using ranging images from a LIDAR sensor to facilitate contamination detection.

[0020] A ranging image is a 2D image where each pixel represents the distance between the sensor and the nearest object in the scene. A ranging image can be created using a LiDAR sensor that emits laser pulses and measures the time it takes for the pulses to return. The distance to the nearest object can then be calculated based on the time-of-flight of the laser pulses. In autonomous driving, ranging images can be used to create a 3D map of the surrounding environment, which can help autonomous vehicles navigate safely. Advantageously, ranging images can be generated in real time. A ranging image provides a denser representation of the 3D scene than a sparse point cloud representation. A sparse point cloud representation is a collection of points in 3D space, where each point represents the location of an object in the scene. These points are typically unconnected, meaning there is no information about the relationships between these points. On the other hand, a ranging image is a complete 2D representation of the 3D scene. In other words, a ranging image includes information about the distance between the sensor and every object in the scene. This information can be used to reconstruct the 3D scene, which can be useful for autonomous driving. A morphological filter is a type of image processing filter that can be used to fill sparse regions in a ranging image. A dilation filter is a morphological filter that expands high-intensity regions in an image. Dilation filters can be used to fill sparse regions in a ranging image by expanding high-intensity regions, which are typically areas containing objects.

[0021] On one hand, once ranging images are obtained from the LiDAR sensor and camera depth estimates are obtained from the camera, the machine learning system can use distance-based segmentation techniques to generate an attention map that highlights areas where the camera may be contaminated. The attention map captures one or more regions where the depth estimates differ significantly, indicating that these regions are more likely to be affected by contamination. On another hand, the machine learning system can generate a Structural Similarity Index (SSIM) score to create an attention-guided depth consistency metric. On yet another hand, the machine learning system can use a feature gating mechanism to modulate the extracted camera features based on the attention-guided depth consistency metric, and then pass the results to the contamination decoder. Finally, the contamination decoder can output multi-class probabilities using a softmax function.

[0022] Figure 1An example of an autonomous vehicle 102 is shown. The autonomous vehicle 102 in the illustrated example may include a passenger vehicle, such as a sedan or truck, capable of accommodating a human driver and / or human passengers. In one aspect, the autonomous vehicle 102 may include an ADAS system. The autonomous vehicle 102 may include a body 104 suspended on a chassis, in this example including four wheels and associated axles. A propulsion system 108 (such as an internal combustion engine, a hybrid power plant, or even an all-electric motor) may be connected via a drivetrain to drive some or all of the wheels, which may include a gearbox (not shown). A steering wheel 110 may be used to steer some or all of the wheels to guide the autonomous vehicle 102 along a desired path while the propulsion system 108 is operating and engaged to propel the autonomous vehicle 102. For a Level 5 implementation, the steering wheel 110, etc., may be optional. One or more controllers 114A-114C (controller 114) may provide autonomous driving capabilities in response to signals continuously provided in real time from a sensor array, as described more fully below.

[0023] Each controller 114 can essentially be one or more onboard computers configured to perform deep learning and artificial intelligence functions and output automated operation commands to autonomously drive the autonomous vehicle 102 and / or assist a human driver in driving the vehicle. Each vehicle can have any number of unique controllers for functional safety and additional features. For example, controller 114A can serve as the main computer for autonomous driving functions, controller 114B can serve as an auxiliary computer for functional safety functions, controller 114C can provide artificial intelligence functions for in-camera sensors, and controller 114(D) (not shown) can provide infotainment functions and additional redundancy for emergency situations.

[0024] The controller 114 can transmit command signals to operate the vehicle's brakes 116 via one or more brake actuators 118, operate the steering mechanism via steering actuators, and operate the propulsion system 108, which also receives accelerator / throttle actuation signals 122. Actuation can be performed by methods known to those skilled in the art, typically transmitting signals via a controller area network data interface (“CAN bus”)—a network used within modern automobiles to control brakes, acceleration, steering, windshield wipers, etc. The CAN bus can be configured with dozens of nodes, each with its own unique identifier (CANID). The bus can be read to find steering wheel angle, ground speed, engine RPM, button positions, and other vehicle status indicators. The functional safety level of the CAN bus interface is typically Automotive Safety Integrity Level (ASIL) B. Other protocols can also be used for communication within the vehicle, including FlexRay and Ethernet.

[0025] On one hand, actuation controllers can be obtained through dedicated hardware and software, allowing control of throttle, braking, steering, and gear shifting. The hardware can provide a bridge between the vehicle's CAN bus and controller 114, forwarding vehicle data to controller 114, including steering signals, wheel speeds, acceleration, pitch, roll, yaw, GPS data, tire pressure, fuel level, sonar, braking torque, and more. Similar actuation controllers can be configured for any other brand and type of vehicle, including dedicated patrol and security vehicles, autonomous taxis, long-haul trucks including towing trailer configurations, ladder fire trucks, agricultural vehicles, industrial vehicles, and buses.

[0026] The controller 114 can provide autonomous driving outputs in response to a sensor input array, which includes, for example: one or more ultrasonic sensors 124, one or more radar sensors 126, one or more LiDAR sensors 128, one or more surround cameras 130 (typically located at various positions on the vehicle body 104 to image all areas around the vehicle body), one or more stereo cameras 132 (in one aspect, at least one such stereo camera may be forward-facing to provide object recognition in the vehicle's path), one or more infrared cameras 134, a GPS unit 136 providing position coordinates, a steering sensor 138 detecting steering angles, speed sensors 140 (one for each wheel), an inertial sensor or inertial measurement unit (“IMU”) 142 monitoring the movement of the vehicle body 104 (this sensor may be, for example, an accelerometer and / or gyroscope sensor and / or magnetic compass), tire vibration sensors 144, and microphones 146 placed around and inside the vehicle. Other sensors may be used as known to those skilled in the art.

[0027] The controller 114 can also receive input from the dashboard 148 and provide human-perceptible output to a human operator via a human-machine interface (“HMI”) display 150, auditory signalers, speakers, and / or other components. In addition to conventional information such as speed, time, and other well-known information, the HMI display 150 can also provide vehicle occupants with information about the map and the vehicle's location, the location of other vehicles (including occupancy grids), and even the controller's identification of objects and states. For example, when the controller identifies a contaminated area, a stop sign, a warning sign, or a traffic light change and is taking appropriate action, the HMI display 150 can alert passengers, reassuring them that the controller 114 is operating as intended.

[0028] On one hand, the dashboard 148 may include a separate controller / processor configured to perform deep learning and artificial intelligence functionalities.

[0029] The autonomous vehicle 102 may collect data preferably used to aid in training and optimizing a neural network for autonomous driving. The autonomous vehicle 102 may include a modem 152, preferably a system-on-a-chip that provides modulation and demodulation functions and allows the controller 114 to communicate via a wireless network 154. As known in the art, the modem 152 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be implemented via known direct conversion processes (directly from baseband to RF and vice versa) or via a superheterodyne process, as known in the art. Alternatively, such RF front-end functionality may be provided by a separate chip. The modem 152 preferably includes wireless functionality substantially compliant with one or more wireless protocols, such as, but not limited to, LTE, WCDMA, UMTS, GSM, CDMA2000, or other known and widely used wireless protocols.

[0030] It should be noted that cameras 130-134 are capable of generating a richer set of features at a much lower cost compared to sonar and radar sensors 126. Therefore, the autonomous vehicle 102 may include multiple cameras 130-134 to capture images of the entire perimeter surrounding the autonomous vehicle 102. The camera type and lens selection depend on the nature and type of the function. The autonomous vehicle 102 may have a mix of camera types and lenses to provide complete coverage around the autonomous vehicle 102; typically, narrow lenses have a narrow field of view but can see further. All camera locations on the autonomous vehicle 102 can support interfaces such as Gigabit Multimedia Serial Link (GMSL) and Gigabit Ethernet.

[0031] On one hand, controller 114 may receive multiple images acquired by multiple cameras 130 to 134. At least one of the multiple images may include a LiDAR sensor ranging image 302 obtained from LiDAR sensor 128. At least one other image may include a multi-camera input image 304 obtained from one or more cameras 130 to 134. Next, controller 114 may process the first image ( Figure 3 The LIDAR sensor ranging image 302 shown is filtered to fill one or more sparse regions in the first image. Then, the controller 114 can filter the second image (…). Figure 3The camera input image 304 shown is used to generate multiple depth estimates. Additionally, the controller 114 can generate an attention map by combining the filtered first image and the multiple depth estimates. Furthermore, the controller 114 can generate a consistency score based on the attention map, indicating the consistency of depth estimates between the first and second images. The controller 114 can use a gating mechanism to modulate one or more features extracted from the second image based on the consistency score. Next, the controller 114 can generate a classification of one or more contamination regions in the second image based on the modulated features.

[0032] Figure 2 This is a block diagram illustrating an example computing system 200. As shown, the computing system 200 includes processing circuitry 243 and a memory 202 for executing a machine learning system 204, which may represent an example instance of any controller 114 described in this disclosure, such as... Figure 1 The controller 114. In one aspect, the machine learning system 204 may include one or more neural networks, such as a feature extractor 206A, a shared encoder 206B, and a deep decoder 206C (collectively, “layers 208”) comprising corresponding layers 208A, 208B, 208C. Each neural network in neural network 206 may include various types of neural networks, such as, but not limited to, recurrent neural networks (RNNs), convolutional neural networks (CNNs), and deep neural networks (DNNs).

[0033] The computing system 200 may also be implemented as any suitable external computing system accessible to the controller 114, such as one or more server computers, workstations, laptop computers, mainframes, special-purpose devices, cloud computing systems, high-performance computing (HPC) systems (i.e., supercomputing), and / or other computing systems capable of performing the operations and / or functions described in one or more aspects of this disclosure. In some examples, the computing system 200 may represent a cloud computing system, server farm, and / or server cluster (or a portion thereof) that provides services to client devices and other devices or systems. In other examples, the computing system 200 may represent or be implemented through one or more virtualized computing instances (e.g., virtual machines, containers, etc.) of a data center, cloud computing system, server farm, and / or server cluster.

[0034] The techniques described in this disclosure can be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, aspects of the described techniques can be implemented within processing circuitry 243 of computing system 200, which may include one or more microprocessors, controllers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or equivalent discrete or integrated logic circuits, or other types of processing circuitry. The terms "processor" or "processing circuitry" can generally refer to any of the aforementioned logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit, including hardware, may also perform one or more of the techniques of this disclosure.

[0035] In another example, computing system 200 includes any suitable computing system having one or more computing devices, such as desktop computers, laptops, game consoles, smart TVs, handheld devices, tablets, mobile phones, smartphones, etc. In some examples, at least a portion of computing system 200 is distributed within a cloud computing system, data center, or network (such as the Internet, other public or private communication networks, such as broadband, cellular, Wi-Fi, ZigBee, Bluetooth). ® (or other personal area networks—PANs, near field communications (NFC), ultra-wideband, satellite, enterprise, service provider, and / or other types of communication networks) for sending data between computing systems, servers, and computing devices.

[0036] Memory 202 may include one or more storage devices. One or more components of computing system 200 (e.g., processing circuitry 243, memory 202, neural network 206, contamination decoder 252, etc.) may be interconnected to enable inter-component communication (physical, communicative, and / or operational). In some examples, such connectivity may be provided by a system bus, network connection, inter-process communication data structure, local area network, wide area network, or any other method of data transmission. Processing circuitry 243 of computing system 200 may implement functions associated with computing system 200 and / or execute instructions associated with computing system 200. Examples of processing circuitry 243 include a microprocessor, application processor, display controller, auxiliary processor, one or more sensor hubs, and any other hardware configured to function as a processor, processing unit, or processing device. Computing system 200 may use processing circuitry 243 to perform operations according to one or more aspects of this disclosure by using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and / or executing on computing system 200. The one or more storage devices of memory 202 may be distributed among multiple devices.

[0037] Memory 202 may store information for processing during operation of computing system 200. In some examples, memory 202 includes temporary memory, meaning that the primary purpose of one or more storage devices of memory 202 is not long-term storage. Memory 202 may be configured to store information short-term as volatile memory, so that the stored contents are not retained if it is deactivated. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, memory 202 may also include one or more computer-readable storage media. Memory 202 may be configured to store a larger amount of information than volatile memory. Memory 202 may also be configured to store information long-term as non-volatile storage space and retain the information after an activation / deactivation cycle. Examples of non-volatile memory include magnetic hard disks, optical disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM). Memory 202 may store program instructions and / or data associated with one or more modules described according to one or more aspects of this disclosure.

[0038] Processing circuitry 243 and memory 202 can provide an operating environment or platform for one or more modules or units (e.g., neural network 206, contamination decoder 252, etc.), which can be implemented as software, but in some examples may also include any combination of hardware, firmware, and software. Processing circuitry 243 can execute instructions, and one or more storage devices (e.g., memory 202) can store instructions and / or data of one or more modules. The combination of processing circuitry 243 and memory 202 can retrieve, store, and / or execute instructions and / or data of one or more application programs, modules, or software. Processing circuitry 243 and / or memory 202 can also be operatively coupled to one or more other software and / or hardware components, including but not limited to... Figure 2 One or more components as shown.

[0039] Processing circuitry 243 may use virtualization modules, such as virtual machines or containers running on the underlying hardware, to execute the machine learning system 204. One or more such modules may be executed as one or more services of an operating system or computing platform. Various aspects of the machine learning system 204 may be executed as one or more executable programs at the application layer of the computing platform.

[0040] One or more input devices 244 of the computing system 200 can generate, receive, or process input. Such input may include input from a keyboard, pointing device, voice response system, camera, biometric detection / response system, button, sensor, mobile device, control panel, microphone, pressure-sensitive screen, network, or any other type of device used to detect human or machine input.

[0041] One or more output devices 246 may generate, send, or process output. Examples of output are haptic, audio, visual, and / or video output. Output device 246 may include a display, sound card, video graphics adapter card, speaker, pressure-sensitive screen, one or more USB interfaces, video and / or audio output interfaces, or any other device capable of generating haptic, audio, video, or other output. Output device 246 may include a display device that can operate as an output device using technologies including liquid crystal displays (LCDs), quantum dot displays, dot matrix displays, light-emitting diode (LED) displays, organic light-emitting diode (OLED) displays, cathode ray tube (CRT) displays, electronic ink, or monochrome, color, or any other display type capable of generating haptic, audio, and / or visual output. In some examples, computing system 200 may include a pressure-sensitive display that can be used as a user interface device that acts as both one or more input devices 244 and one or more output devices 246.

[0042] One or more communication units 245 of computing system 200 can communicate with devices outside computing system 200 (or between separate computing devices of computing system 200) by sending and / or receiving data, and in some respects can function as both an input device and an output device. In some examples, communication unit 245 can communicate with other devices via a network. In other examples, communication unit 245 can send and / or receive radio signals on a radio network such as a cellular radio network. Examples of communication unit 245 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, GPS receivers, or any other type of device capable of sending and / or receiving information. Other examples of communication unit 245 may include Bluetooth, found in mobile devices. ® GPS, 3G, 4G and Wi-Fi ® Radio and Universal Serial Bus (USB) controllers, etc.

[0043] exist Figure 2In the example, neural network 206 can receive input data 210 and generate output data 212. The output data 212 processed by different neural networks 206 can be used as input data for contamination decoder 252. Input data 210 and output data 212 can contain various types of information. For example, input data 210 can include multimodal data. As used herein, the terms "multimodal data" or "multimodal information" refer to information that can consist of multiple media or data types, such as, but not limited to, image data, video data, audio data, source text data, numerical data, speech data, etc. Output data 212 can include various features, depth estimates, etc.

[0044] Each layer group 208 may include a corresponding set of artificial neurons. For example, layer 208A may include an input layer, a feature layer, an output layer, and one or more hidden layers. Layer 208 may include fully connected layers, convolutional layers, pooling layers, and / or other types of layers. In a fully connected layer, the output of each neuron in the previous layer forms the input of each neuron in the fully connected layer. In a convolutional layer, each neuron in the convolutional layer processes the input from neurons associated with that neuron's receptive field. A pooling layer combines the outputs of a cluster of neurons in one layer into a single neuron in the next layer.

[0045] Each input of each artificial neuron in each layer of layer group 208 is associated with a corresponding weight in weight group 216. The output of the k-th artificial neuron in neural network 206 can be defined as:

[0046]

[0047] In equation (1), y k It is the output of the k-th artificial neuron. It is the activation function, W k X is the weight vector of the k-th artificial neuron (e.g., the weights in weight 216), and X k This is the input vector of the k-th artificial neuron. In some examples, one or more inputs to the k-th artificial neuron are not the output values ​​of another artificial neuron or bias terms not based on the source data. Various activation functions are known in the art, such as Rectified Linear Unit (ReLU), Hyperbolic Tangent, Sigmoid, etc.

[0048] According to the techniques described herein, the machine learning system 204 may include a pre-trained model trained using training data 213 and one or more pre-trained neural networks 206. In one aspect, a LiDAR sensor ranging image feature extractor 206A may be configured to implement a computer vision algorithm for extracting features from LiDAR sensor ranging images. The extracted features can then be used for various tasks, such as object detection, 3D reconstruction, and scene understanding. Various different feature extractors exist that can be used for LiDAR sensor ranging images. Some of the most common include, but are not limited to, intensity feature extractors, specific feature extractors, and geometric feature extractors. A multi-camera input image shared encoder 206B may have a neural network architecture that can be used to extract features from multiple camera images. The shared encoder 206B can be shared across all cameras. Therefore, the shared encoder 206B can extract features in a consistent manner. The shared encoder 206B can aid tasks such as object detection and scene understanding because it helps ensure that features are extracted correctly. The combined features can then be used by the machine learning system 204 for downstream tasks, such as, but not limited to, depth estimation. A depth decoder 206C may have a neural network architecture that can be used to estimate the depth of objects in a scene from a single image. The depth decoder 206C can take the output of the shared encoder 206B as input and output a depth map, which is a 2D image representing the distance to each pixel in the scene.

[0049] In one aspect, the machine learning system 204 may also include a contamination decoder 252. The contamination decoder 252 may have a neural network architecture that can be used to detect and segment contamination on camera images. The contamination decoder 252 may be configured to output a mask, which may be a 2D image indicating the contaminated regions of the image. The contamination decoder 252 can be implemented using various neural network architectures, such as, but not limited to, convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The choice of architecture may depend on the specific task at hand. In addition to detecting and segmenting contamination on camera images, the contamination decoder 252 can also be used to improve the performance of other tasks, such as, but not limited to, object detection and scene understanding.

[0050] As noted above, even under challenging conditions such as rain, fog, or snow, the sensors of autonomous vehicles should be able to accurately detect and avoid obstacles. If cameras are contaminated, they may be unable to clearly see obstacles, potentially leading to accidents. Various techniques, including but not limited to deep learning, the computation of attention-guided depth consistency metrics, and feature gating, can be used to detect and segment contamination in a scene. Attention-guided depth consistency metrics are methods used to measure the consistency of depth estimates between two images. This method uses an attention mechanism to focus on the most salient features in an image and then compares the depth estimates of those features. The deep learning techniques described in this paper involve using deep neural networks to learn features of contaminated regions. Feature gating is a technique in machine learning used to select which features are most important for a specific task. The goal of feature gating is to improve the performance of machine learning models by reducing the number of features it needs to consider.

[0051] Figure 3 This is a block diagram illustrating an example pollution detection framework that can perform the techniques of this disclosure. Figure 3 This disclosure is provided for illustrative purposes and should not be construed as a limitation on the techniques extensively illustrated and described herein. For illustrative purposes, this disclosure describes... Figure 3 The illustrated framework 300, Figure 3 This is a block diagram illustrating an example pollution detection framework that can perform the techniques of this disclosure. Figure 3In the example, neural network 206 may receive input data 210. On one hand, feature extractor 206A may receive one or more LIDAR sensor ranging images 302 acquired by one or more LIDAR sensors 128, and shared encoder 206B may receive one or more multi-camera input images 304 acquired by multiple cameras 130 to 134. At least some of the aforementioned images may include one or more contaminated areas. On one hand, LIDAR sensor ranging images 302 may be created using LIDAR sensor 128 rotating about a vertical axis. As LIDAR sensor 128 rotates, it emits laser pulses and measures the time it takes for the laser pulses to return. The distance to the nearest object can then be calculated by LIDAR sensor 128 for each pixel in the image. As noted above, in autonomous driving, LIDAR sensor ranging images 302 can be used to create a 3D map of the surrounding environment, which can be used to assist in the safe navigation of autonomous vehicle 102. LIDAR sensor ranging images 302 may be a complete 2D representation of a 3D scene. In other words, the LIDAR sensor ranging image 302 can contain information about the distance between the LIDAR sensor 128 and each object in the scene. In the autonomous vehicle 102, the multi-camera input image 304 can be used to provide a more complete and accurate view of the environment surrounding the autonomous vehicle 102. This accurate view of the environment can aid in various tasks, such as, but not limited to, object detection, scene understanding, and obstacle avoidance. There are several benefits to using the multi-camera input image 304 in the autonomous vehicle 102. First, the multi-camera input image 304 can help improve the accuracy of object detection because multiple cameras 130 to 134 can provide different perspectives on the same object, which can help reduce occlusion and improve the accuracy of object bounding boxes. Second, the multi-camera input image 304 can help improve the accuracy of scene understanding because multiple cameras 130 to 134 can provide different views of the same scene. Different views of the same scene can help identify objects that might be difficult to see from a single perspective.

[0052] On one hand, the LIDAR sensor ranging image feature extractor 206A can be configured to implement a computer vision algorithm for extracting features from the LIDAR sensor ranging image 302. The extracted features can then be used for various tasks, such as object detection, 3D reconstruction, and scene understanding. Various different feature extractors exist that can be used for the LIDAR sensor ranging image 302. Some of the most common include, but are not limited to: intensity feature extractors, specific feature extractors, and geometric feature extractors. Intensity features can be based on the intensity of LIDAR sensor echoes. For example, the average intensity of a region can be used to indicate the presence of an object in that region. Spatial features can be based on the spatial distribution of LIDAR sensor echoes. For example, the number of LIDAR sensor echoes in a region can be used to indicate the size of an object. Geometric features can be based on the geometric properties of LIDAR sensor echoes. For example, the shape of a region can be used to indicate the type of object. Once the feature extractor 206A extracts features from the LIDAR sensor ranging image 302, the machine learning system 204 can further process one or more LIDAR sensor ranging images 302 by using a morphological filter 306 (e.g., a dilation filter) to fill any sparse regions and make the extracted features denser.

[0053] The morphological filter 306 is a type of image processing filter that expands high-intensity regions in an image. The morphological filter 306 can be used to fill sparse regions in an image, such as those that may be caused by noise or occlusion. Dilation is a mathematical operation that takes an image as input and outputs a new image. A new image can be created by expanding high-intensity regions in an input image. This expansion can be accomplished using structured elements, which are small shapes that can be used to probe the input image. Structured elements are typically squares or circles, but in general, they can be any shape. The size of the structured element determines how much the high-intensity region is expanded.

[0054] For illustrative purposes only, assume R i The LIDAR sensor ranging image 302 is acquired by LIDAR sensor 128, and it is assumed that R' i The morphological filter 306 is applied to R through the machine learning system 204. i The obtained densely ranged image. The morphological filter 306 can be represented as a binary mask, where if each pixel in the mask is in R... i Within a specific radius r of the effective pixels in the image, the value is 1; otherwise, it is 0. The morphological filter 306 can be a dilation filter for expanding high-intensity regions in the image. The morphological filter 306 can also be used to fill sparse regions in the image, such as those that may be caused by noise or occlusion. In one aspect, the machine learning system 204 can use a binary mask to determine the ranging image R. iWhich pixels in the ranging image R are eligible for expansion? i If a pixel in the range image R has a value of 1 in the binary mask, then it will be expanded. i If a pixel in the binary mask has a value of 0, it will not be expanded. The radius r of the binary mask determines the morphological filter 306, which will then measure the ranging image R. i How far does the high-intensity region extend in the image? A larger radius will result in a denser ranging image, while a smaller radius will result in a less dense ranging image. Dense ranging image R' i The following formula (2) can be used to calculate:

[0055] R' i =Ri⊕B(2)

[0056] in:

[0057] Ri is the LIDAR sensor ranging image 302 acquired by LIDAR sensor 128.

[0058] B is a binary mask.

[0059] ⊕ is the dilation operator.

[0060] The dilation operator is a mathematical operation that expands high-intensity regions in an image. One approach is to implement the dilation operator using convolution operations. The resulting denser ranging image R' i It can be further expressed using formula (3):

[0061] R'ij=max{Rik|k∈B(i, r)}(3)

[0062] in:

[0063] R ij It is a densely distributed ranging image R' i The intensity of pixel (i, j) in the image.

[0064] R ik It is the ranging image R i The intensity of pixel (i, k) in the image.

[0065] B(i, r) is the set of pixels in a binary mask centered at i with a radius of r.

[0066] `max` is the maximum value operator.

[0067] On one hand, the max operation can select the maximum range value within the neighborhood defined by the binary mask. In other words, the max operation effectively fills R. i The gaps between effective pixels in the image are reduced to produce a denser ranging image R. iFor example, if a binary mask has a radius of 2 pixels, the neighborhood defined by the binary mask can include the pixel at (i, j), and the pixels at (i-1, j), (i+1, j), (i, j-1), and (i, j+1). The max operation then selects the maximum ranging value from these 5 pixels. If the maximum ranging value is 10, the ranging image R' is densified. i The intensity of pixel (i, j) in the image will be 10. If the maximum ranging value is 5, then the ranging image R' will be densified. i The intensity of pixel (i, j) in the matrix will be 5. The max operation can efficiently fill R by assigning the maximum range value to all pixels in the neighborhood defined by the binary mask. i The gap between effective pixels in the image. In other words, the max operation produces a larger image than the original ranging image R. i More accurate and denser ranging images R' i .

[0068] On one hand, the shared encoder 206B may have a neural network architecture that can be used to extract features from multiple camera input images 304. The shared encoder 206B can be shared across all cameras 130 to 134. Therefore, the shared encoder 206B can extract features in a consistent manner. Figure 3 As shown, the input to the shared encoder 206B can be a set of multiple camera input images 304. The shared encoder 206B can be configured to extract features from each of the camera input images 304 and combine them. In one aspect, the machine learning system 204 can use the combined features as input to the deep decoder 206C. The shared encoder 206B can be implemented using various different neural network architectures. For example, the shared encoder 206B can be implemented as a CNN or an RNN.

[0069] The depth decoder 206C may have a neural network architecture that can be used to estimate the depth of objects in a scene. The depth decoder 206C may take the output (combined features) of the shared encoder 206B as input and may output one or more depth estimates 308. In one aspect, the depth decoder 206C may generate depth estimates 308 by learning the relationship between the depth of the camera input image 304 and the depth of objects in the scene. For example, the depth decoder 206C may be trained on a dataset of paired images and a depth map. The images in the dataset can be used to train the depth decoder 206C to identify objects in the scene, and the depth map can be used to train the depth decoder 206C to estimate the depth of the objects. Once the depth decoder 206C is trained, the machine learning system 204 may use the pre-trained depth decoder 206C to generate depth estimates for new images based on the combined features provided by the shared encoder 206B. For example, the depth decoder 206C may first identify objects in a new image based on the combined features, and then the depth decoder 206C may estimate the depth of the objects based on the relationships it has learned from the training dataset.

[0070] On one hand, once the machine learning system 204 obtains the LIDAR sensor ranging depth features 309 from the morphological filter 306 and the depth estimate 308 from the depth decoder 206C, the machine learning system 204 can use distance-based segmentation techniques to generate an attention map for the contamination detection problem. Distance-based segmentation is a type of image segmentation technique that uses the distance between pixels to segment an image. Typically, various methods (such as, but not limited to, Euclidean distance, Manhattan distance, and Chebyshev distance) can be used to calculate the distance between pixels. In the context of contamination detection, the machine learning system 204 can use distance-based segmentation techniques to generate an attention map by calculating the distance between each pixel in the image and the nearest contaminated pixel. The pixel closest to the contaminated pixel will have a high value in the attention map, while the pixel furthest from the contaminated pixel will have a low value. The attention map can then be used to focus the attention of the machine learning system 204 on the pixels most likely to be contaminated. The distance-based segmentation techniques described below help the machine learning system 204 improve its accuracy in detecting contamination.

[0071] In one aspect, the machine learning system 204 can implement distance-based segmentation by dividing the camera image into multiple layers based on the distance from the camera. More specifically, the machine learning system 204 can calculate the distance between each pixel in the camera image and the camera. The pixels closest to the camera can be in the lowest layer, while the pixels farthest from the camera can be in the highest layer. Instead of directly projecting the points from the LIDAR sensor onto the image plane of the camera, the machine learning system 204 can project each LIDAR sensor point onto the layer closest to it. This projection ensures that points that are close together in 3D space are mapped to adjacent pixels in the camera image.

[0072] In one aspect, the machine learning system 204 can use the following formula (4) to perform the projection:

[0073] x = K[R|t]pi (4)

[0074] Where:

[0075] P = {p i} is the LIDAR sensor ranging image, where p i is the ranging value of pixel i,

[0076] K is the camera intrinsic matrix, which describes the focal length and principal point of the camera,

[0077] R is the rotation matrix, which describes the rotation of the camera relative to the LIDAR sensor,

[0078] t is the translation matrix, which describes the translation of the camera relative to the LIDAR sensor,

[0079] x is the projected point in the image plane.

[0080] The projection formula (4) indicates that the projected point x is equal to the product of the concatenation of the camera intrinsic matrix K, the rotation matrix R, and the translation matrix t and the LIDAR sensor ranging point p i . In other words, the projection formula (4) takes the LIDAR sensor ranging point p i and projects it onto the camera plane using the intrinsic and extrinsic parameters of the camera. The result is the projected point x, which is a 2D point in the image plane.

[0081] In one aspect, next, for each projected point x, the machine learning system 204 can calculate the distance between x and the camera center. Additionally, the machine learning system 204 can find the layer i such that d_layer_i - 1 < distance < d_layer_i, and can assign the projected point x to layer i. The projected points in layer i can be represented as X i . The purpose of this process is to group these projected points based on the distance of the projected points from the camera.

[0082] On the one hand, in order to calculate the distance between the pixel and the center of the camera, the machine learning system 204 can use the following formula (5).

[0083] (5)

[0084] in:

[0085] K is the camera's intrinsic matrix, which describes the camera's focal length and principal point.

[0086] u i and v i These are pixel coordinates, which are the (x, y) coordinates of a pixel in the image plane.

[0087] d ij It is a depth value, which is the distance of a pixel from the camera.

[0088] Z ij It is the distance between a pixel and the center of the camera in the camera coordinate system.

[0089] The distance formula (5) shows that the distance between a pixel and the camera center is equal to the inverse of the camera intrinsic matrix K multiplied by the vector [u]. i , v i [1] and then multiply by the depth value d ij In other words, formula (5) takes the pixel coordinates and depth values ​​and uses the camera's intrinsic matrix to calculate the distance between the pixel and the camera center in the camera coordinate system.

[0090] On the one hand, in order to assign each pixel to a layer, the machine learning system 204 can use the following formula (6):

[0091] layer(i, j)=min{i|Z ij <d_layer_i}(6),

[0092] in:

[0093] layer(i, j) is the layer to which pixel (i, j) is assigned.

[0094] Z ij It is the distance between pixel (i, j) and the camera center in the camera coordinate system.

[0095] d_layer_i is the distance of layer i from the camera.

[0096] min is the minimum value function.

[0097] Assignment formula (6) shows that pixel (i, j) is assigned to the layer with minimum value i, such that Z ij<d_layer_i. In other words, formula (6) takes the distance between the pixel and the camera center and uses the layer distance to find the layer to which the pixel is closest. For example, if the camera view has 3 layers at distances of 100m, 200m, and 300m, a pixel at a distance of 150m will be assigned to layer 2. The purpose of this process is to improve the grouping of these pixels based on their distance from the camera.

[0098] In one aspect, the machine learning system 204 may use the layers to create an attention map A using the following formula (7):

[0099] If, for some k, layer(i, j) < layer(i + k, j + l), l ∈ B(0, r), then A ij = 1, otherwise A ij = 0 (7),

[0100] where:

[0101] A ij is the attention weight of pixel (i, j),

[0102] layer(i, j) is the layer to which pixel (i, j) is assigned,

[0103] k and l are pixel offsets,

[0104] B(0, r) is a 2D ball of radius r centered around the origin.

[0105] The attention map formula (7) indicates that if pixel (i, j) is closer to the camera than its neighbors within a radius r, the attention weight of that pixel is 1. Otherwise, the attention weight is 0. In other words, formula (7) takes the layer assignments of each pixel and uses them to create an attention map that highlights the pixels closest to the camera.

[0106] On one hand, attention map A can capture regions where depth estimates differ significantly. As noted above, machine learning system 204 creates the attention map by comparing the layer assignment of each pixel with the layer assignments of its neighbors. If a pixel and its neighbors have different layer assignments, the pixel is likely affected by contamination. On another hand, machine learning system 204 can compute an SSIM score. The SSIM score can be a measure of similarity between two images. The SSIM score can be used to measure depth estimation consistency. On another hand, machine learning system 204 can create an attention-guided depth consistency metric 310 by combining attention map A with the SSIM score. The attention map can be used by machine learning system 204 to weight the SSIM score, so that regions with high attention weights have a greater impact on the overall metric. The purpose of the attention-guided depth consistency metric 310 is to identify regions where depth estimates are inconsistent. These regions are more likely to be affected by contamination, and therefore they can be used to focus attention on depth refinement algorithms.

[0107] On one hand, attention maps and SSIM scores are two distinct measures of depth consistency. An attention map measures how much attention the machine learning system 204 gives to different parts of an image, while an SSIM score measures how similar two images are. To create an attention-guided depth consistency measure 310, the machine learning system 204 can combine these two measures into a weighted sum. The weight of the attention map is typically higher than that of the SSIM score because the attention map provides more information about which parts of the image are important for depth consistency. The resulting attention-guided depth consistency measure 310 can be a more robust measure of depth consistency than either the attention map or the SSIM score alone, as it considers both the overall similarity of the images and the focus of the model's attention. On the other hand, the machine learning system 204 can compute the attention-guided depth consistency measure 310 by first calculating the weighted SSIM score using formula (8).

[0108] weighted_SSIM_score=attention_map SSIM_score(8).

[0109] in

[0110] attention_map is an attention map.

[0111] SSIM_score is the SSIM score.

[0112] weighted_SSIM_score is the weighted SSIM score.

[0113] An attention map is a measure of how much attention the machine learning system 204 gives to different parts of an image. Therefore, an attention map can be high in regions where the machine learning system 204 has high confidence in its depth estimates, or in regions where contamination is more likely, and low in regions where the network has low confidence. An SSIM score is a measure of how similar two images are. Therefore, an SSIM score may be high when two images are very similar, and low when two images are very different. By element-wise multiplying the attention map by the SSIM score, the machine learning system 204 can give more importance to regions where the depth estimates are significantly different, since the attention map may be high in these regions and the SSIM score may be low. In other words, regions where the depth estimates are significantly different are more important for depth consistency, and therefore, they receive more weight in the attention-guided depth consistency measure 310. In one aspect, the machine learning system 204 can use a weighted SSIM score as a depth consistency measure.

[0114] On one hand, the machine learning system 204 can also compute an attention-guided depth consistency score. The attention-guided depth consistency score can be a measure of how consistent the depth estimates from the camera sensor and the LiDAR sensor are in the attention map. On one hand, the machine learning system 204 can first take a weighted average of the SSIM score and the attention map. This weighted average can be computed by multiplying the SSIM score by the attention map and then dividing by the sum of the attention maps. The attention-guided depth consistency score is a single score representing overall depth consistency. The attention-guided depth consistency score can be a more robust measure of depth consistency than the SSIM score alone because it takes into account both the overall similarity of the images and the focus of the attention of the machine learning system 204.

[0115] On one hand, the machine learning system 204 can use a feature gating mechanism 312 to modulate the extracted camera features based on an attention-guided depth consistency score. For example, the machine learning system 204 can multiply the camera features by weights based on the attention-guided depth consistency score. These weights can be calculated by the feature gating mechanism 312, which maps the attention-guided depth consistency score to values ​​between 0 and 1. A value of 0 means the camera features will be ignored, while a value of 1 means the camera features will be passed to the contamination decoder 252 unchanged. On the other hand, the feature gating mechanism 312 can be implemented as a sigmoid function or a softplus function. The sigmoid function outputs values ​​between 0 and 1, while the softplus function outputs values ​​between 0 and infinity. The gated features can then be used by the machine learning system 204 as input to the contamination decoder 252, which can be configured to predict the presence and type of contamination. The feature gating mechanism 312 allows the machine learning system 204 to focus on regions where depth estimates are more consistent. This is important because contamination is more likely to occur in regions where depth estimates are inconsistent. The feature gating mechanism 312 also allows the machine learning system 204 to learn to ignore regions where the depth estimation is unreliable, since the machine learning system 204 may sometimes make mistakes in its depth estimation, and the feature gating mechanism 312 can help prevent these mistakes from affecting the contaminated prediction.

[0116] The advantage of using feature gating mechanism 312 before passing features to contamination decoder 252 is that, since feature gating mechanism 312 uses attention-guided depth consistency scores to weight camera features, it allows machine learning system 204 to selectively focus on features important to the contamination task while suppressing irrelevant or noisy features. Attention-guided depth consistency scores indicate regions where depth estimates are more consistent and are more likely to be important for the contamination task. By weighting camera features with attention-guided depth consistency scores, machine learning system 204 can focus on these regions and ignore regions where depth estimates are less consistent, thereby improving the accuracy of contamination predictions. Additionally, feature gating mechanism 312 allows machine learning system 204 to learn to ignore regions where depth estimates are unreliable.

[0117] On one hand, the feature gating mechanism 312 can lead to more accurate and robust contamination detection performance in at least two ways. First, by selectively focusing on important features, the machine learning system 204 can better distinguish between different types of contaminated and non-contaminated instances in an image, as it can focus on the features most relevant to the contamination task and ignore irrelevant features. Second, by discarding redundant or unimportant features during inference, the machine learning system 204 can reduce computational costs and memory usage, as it does not have to process all features in the image, only those most relevant to the contamination task. In addition to the above benefits, the feature gating mechanism 312 can also help improve the interpretability of the machine learning system 204. For example, the gating function can be used to visualize which features are most important for the contamination task. This visualization helps understand how the machine learning system 204 makes its predictions and can help identify potential areas for improvement. On the other hand, the contamination decoder 252 can be a neural network that takes the output of the feature gating mechanism 312 (gated features F') as input and outputs a probability distribution on different types of contamination. A softmax function can be used to normalize the probability distribution so that the sum of the probabilities is 1. The softmax function is a commonly used activation function in neural networks for classification tasks. It takes a real-valued vector as input and outputs a probability vector, where each probability represents the likelihood that the input belongs to a specific class. In the case of a contaminated decoder 252, the softmax function outputs... Figure 4 The probability distributions for different types of contamination are shown. This distribution can be used to determine the most likely type of contamination in an image.

[0118] On one hand, the distance-based segmentation technique disclosed herein provides a way to create attention maps that highlight areas where the camera is likely to be contaminated. The disclosed technique works by first calculating the distance between the camera and the LIDAR sensor 128. The distance map can then be thresholded to create a binary map, where 1 indicates a pixel is close to the LIDAR sensor 128 and 0 indicates a pixel is far from the LIDAR sensor 128. On the other hand, the machine learning system 204 can use the binary map to create the attention map, where pixels with a value of 1 are given high attention weights and pixels with a value of 0 are given low attention weights. The machine learning system 204 can use the attention map to weight the SSIM score, thereby giving more importance to areas where the depth estimates are significantly different. By incorporating the attention map into the attention-guided depth consistency metric 310, the machine learning system 204 can improve the accuracy of contamination detection because the attention map allows the machine learning system 204 to focus on areas where the depth estimates are more consistent, which are more likely to be contaminated. Additionally, the attention map allows the machine learning system 204 to learn to ignore areas where the depth estimates are unreliable. Ignoring unreliable regions can be important because machine learning systems can sometimes make mistakes in their depth estimates, and attention maps can help prevent these errors from contaminating predictions.

[0119] In summary, contamination of camera sensors can significantly degrade the performance of the autonomous vehicle 102 in several ways. For example, contamination may: 1) reduce the amount of light reaching the sensor, making it difficult to see objects in low-light conditions; 2) blur the image, making it difficult to identify objects; 3) introduce noise into the image, making it difficult to distinguish objects from the background; and 4) alter the color of the image, making it difficult to identify objects. All of the aforementioned effects can make it difficult for the autonomous vehicle 102 to accurately perceive its surroundings, potentially leading to accidents. By detecting and correcting contamination in real time, the disclosed technology can help improve the safety of the autonomous vehicle 102 by: 1) reducing the amount of light blocked by contamination, thereby allowing the corresponding sensor to see more objects; 2) removing blur from the image, making it easier to identify objects; 3) reducing noise in the image, making it easier to distinguish objects from the background; and 4) correcting the color of the image, making it easier to identify objects. All of these effects can help the autonomous vehicle 102 perceive its surroundings more accurately, which can help prevent accidents. In addition to improving safety, the disclosed technology can also help improve the efficiency of the autonomous vehicle 102 by: 1) reducing the number of positive misidentifications that may lead to unnecessary braking or steering; 2) reducing the number of negative misidentifications that may lead to accidents; and 3) improving the overall performance of the machine learning system 204, making it more reliable and able to operate under a wider range of conditions.

[0120] Figure 4 These are illustrations of various examples of contamination detection in images. More specifically, Figure 4 Examples of mud contamination 402B, grass contamination 402D, sand contamination 402F, and water contamination 402H (collectively, “contamination” 402) detected in the corresponding input images 402A, 402C, 402E, and 402G are illustrated. As used herein, the term “mud contamination” refers to a type of contamination that may occur when mud is deposited on a camera sensor. As used herein, the term “grass contamination” refers to a type of contamination that occurs when grass is deposited on a camera sensor. The term “sand contamination” refers to a type of contamination that occurs when sand is deposited on a camera sensor. As used herein, the term “water contamination” refers to a type of contamination that occurs when water is deposited on a camera sensor. Mud / grass / sand / water can be deposited on camera sensors in various ways, such as, but not limited to: 1) mud / grass / sand / water being splashed onto camera sensors by rain, snow or other water sources; 2) mud / grass / sand / water being blown onto camera sensors by wind; 3) mud / grass / sand / water being transferred to camera sensors through contact with contaminated / wet objects (such as tires, feet or other vehicles).

[0121] Pollution 402 may cause positive or negative misidentifications in object detection and classification, which could lead to incorrect decisions by the autonomous vehicle 102. For example, pollution may cause one or more cameras 130 to 134 to misidentify a pedestrian as a traffic cone, resulting in unnecessary braking by the autonomous vehicle 102. As yet another example, pollution may cause radar sensors to misidentify a car as a pedestrian, which could cause the autonomous vehicle 102 to swerve suddenly to avoid the pedestrian and collide with another object. Therefore, some additional beneficial effects of accurate detection and classification of polluted areas may include, but are not limited to: improving the efficiency of the autonomous vehicle 102 by reducing the number of unnecessary braking or steering events; improving passenger comfort by reducing the number of sudden braking or steering events; and reducing the environmental impact of the autonomous vehicle 102 by reducing the amount of fuel wasted due to unnecessary braking or steering events.

[0122] Conventional methods for cleaning camera sensors 130 to 134 in an autonomous vehicle 102 may require manual intervention, which can be time-consuming and result in downtime. For example, a human operator may need to physically clean the sensors, or they may need to use cleaning solutions that may take time to dry. This downtime can be problematic in situations where the autonomous vehicle 102 needs to operate in a timely manner or where the environment is unfavorable for manual cleaning. By providing real-time contamination detection and correction, the disclosed solution can improve the efficiency of the autonomous vehicle 102 by eliminating the need for manual intervention, reducing the amount of time the autonomous vehicle 102 is not in operation, and reducing the risk of accidents caused by contaminated sensors.

[0123] On one hand, the proposed solution is based on a multimodal approach that combines information from LiDAR sensor 128 and camera sensors 130-134. Since LiDAR sensor 128 and camera sensors 130-134 have different strengths and weaknesses, combining their information allows the autonomous vehicle 102 to obtain a more complete and accurate picture of the environment. LiDAR sensor 128 excels at providing depth information but is not good at detecting small objects or objects close to the ground. Camera sensors 130-134 excel at detecting small objects and objects close to the ground but are not good at providing depth information. By combining information from LiDAR sensor 128 and camera sensors 130-134, the autonomous vehicle 102 can better understand the environment, including distance to objects, object size, and object type. This environmental information can be used to detect pollution and correct its effects. The proposed solution is also based on deep learning methods. In other words, the autonomous vehicle 102 is able to learn from data and improve its performance over time, which is important because the environment in which the autonomous vehicle 102 operates is constantly changing, and the solution needs to be able to adapt to these changes. The proposed solution is based on multimodal and deep learning methods, making it a general solution applicable to a wide range of autonomous driving scenarios and environments. Below are some examples of how the proposed solution can be applied to different autonomous driving scenarios. In urban environments, the proposed technique can be used to detect pollution 402 on camera sensors 130 to 134 mounted on the front of an autonomous vehicle 102. The detected pollution information can be used to operate the autonomous vehicle 102 (e.g., an ADAS system). For example, the classification of polluted areas can be used to improve the performance of object detection and classification algorithms of the autonomous vehicle 102. In highway environments, the proposed technique can be used to detect pollution 402 on LIDAR sensors 128 mounted on the front and rear of the autonomous vehicle 102. The pollution 402 information detected on the LIDAR sensors 128 can be used to correct the effects of pollution and improve the performance of obstacle avoidance algorithms of the autonomous vehicle 102. In rural environments, the proposed technique can be used to detect pollution 402 on camera sensors 130 to 134 mounted on the sides of the autonomous vehicle 102. This pollution information 402 can be used to correct the effects of pollution and improve the performance of lane keeping and lane departure warning algorithms of autonomous vehicles 102.

[0124] Figure 5 This is a flowchart illustrating an example method for detecting contaminated areas in an image according to the technology of this disclosure. Although it is aimed at a computing system 200 ( Figure 2 This is described in the context of [the original text], but it should be understood that other devices can also be configured to perform similar [actions]. Figure 5The method.

[0125] In this example, the machine learning system 204 may initially receive one or more images (502). At least one of the images may include a LiDAR sensor ranging image 302 obtained from LiDAR sensor 128. At least one other image may include a multi-camera input image 304 obtained from one or more cameras 130 to 134. A morphological filter 306 may filter the first image (LIDAR sensor ranging image 302) to fill one or more sparse regions (504) in the first image. For example, a depth morphological filter 306 may expand high-intensity regions in the first image. The morphological filter 306 may generate a denser ranging image. The depth decoder 206C may then generate multiple depth estimates 308 (506) based on a second image (camera input image 304). In one aspect, the depth decoder 206C may generate depth estimates 308 by learning the relationship between the camera input image 304 and the depth of objects in the scene. For example, the depth decoder 206C may be trained on a dataset of paired images and depth maps. Next, the machine learning system 204 can generate an attention map (508) by combining the filtered first image and multiple depth estimates 308. Next, the machine learning system 204 can generate a consistency score (510) based on the attention map, indicating the consistency of depth estimates between the first and second images. The machine learning system 204 can use a gating mechanism to modulate one or more features extracted from the second image based on the consistency score (512). The contamination decoder 252 can generate a classification of one or more contaminated regions in the second image based on the modulated one or more features (514).

[0126] The following numbered clauses illustrate one or more aspects of the devices and technologies described in this disclosure.

[0127] Clause 1. A method comprising: receiving a plurality of images, wherein a first image of one or more images comprises a ranging image and a second image comprises a camera image; filtering the first image by one or more processors to generate a filtered first image, wherein the filtering includes filters configured to fill one or more sparse regions in the first image; generating a plurality of depth estimates by the one or more processors based on the second image; generating an attention map by the one or more processors by combining the filtered first image and the plurality of depth estimates; generating a consistency score by the one or more processors based on the attention map, indicating consistency of depth estimates between the first image and the second image; modulating one or more features extracted from the second image by the one or more processors using a gating mechanism based on the consistency score to generate one or more modulated features; and generating a classification of one or more contaminated regions in the second image by the one or more processors based on the modulated one or more features.

[0128] Clause 2 - The method described in Clause 1, wherein the first image is generated in real time by a light detection and ranging (LIDAR) sensor.

[0129] Clause 3 - The method according to Clause 1 further includes extracting one or more features from the first image prior to performing the filtering on the first image.

[0130] Clause 4 - The method according to Clause 1, wherein the filter includes a morphological filter having a pixel-based binary mask that extends one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter extends the one or more regions of the first image.

[0131] Clause 5 - The method according to any one of Clauses 2 to 3, the method further comprising using a depth decoder to generate the plurality of depth estimates.

[0132] Clause 6 - The method according to any one of Clauses 2 to 3, wherein generating the attention map further includes dividing the second image into multiple layers based on the distance from the camera used to acquire the second image.

[0133] Clause 7 - The method according to Clause 6, wherein dividing the second image into the plurality of layers further includes grouping one or more projection points based on the distance from the camera used to acquire the second image.

[0134] Clause 8 - The method according to Clause 7 further includes calculating the distance from the camera based on a plurality of pixel coordinates in the plane of the second image and the center of the camera in the camera coordinate system.

[0135] Clause 9 - The method according to any one of Clauses 7 to 8, wherein generating the consistency score includes combining the attention map with the generated structural similarity index matrix (SSIM) score.

[0136] Clause 10 - The method according to Clause 9, wherein the SSIM score indicates the consistency of the plurality of depth estimates.

[0137] Clause 11 - The method according to Clause 9 further includes identifying the one or more contaminated areas based on the consistency score.

[0138] Clause 12 - The method according to Clause 1, wherein generating the classification further includes using a softmax function configured to normalize a probability distribution to generate the classification of the one or more contaminated areas.

[0139] Clause 13 - The method described in Clause 1 further includes operating the Advanced Driver Assistance System (ADAS) based on the classification of the one or more contaminated areas.

[0140] Item 14 - An apparatus for camera contamination detection, the apparatus comprising: a memory for storing a plurality of images; and processing circuitry in communication with the memory, wherein the processing circuitry is configured to: receive the plurality of images, wherein a first image of one or more of the images comprises a ranging image and a second image comprises a camera image; filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; generate a plurality of depth estimates based on the second image; generate an attention map by combining the filtered first image and the plurality of depth estimates; generate a consistency score based on the attention map indicating consistency of depth estimates between the first image and the second image; modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate one or more modulated features; and generate a classification of one or more contamination regions in the second image based on the modulated features.

[0141] Clause 15 - The apparatus according to Clause 14, wherein the first image is generated in real time by a light detection and ranging (LIDAR) sensor.

[0142] Clause 16 - The apparatus according to Clause 14, wherein the processing circuitry is further configured to extract one or more features from the first image prior to performing the filtering on the first image.

[0143] Clause 17 - The apparatus according to Clause 14, wherein the filter includes a morphological filter having a pixel-based binary mask that extends one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter extends the one or more regions of the first image.

[0144] Clause 18 - An apparatus according to any one of Clauses 15 to 16, wherein the processing circuitry is further configured to use a depth decoder to generate the plurality of depth estimates.

[0145] Clause 19 - An apparatus according to any one of Clauses 15 to 16, wherein the processing circuitry configured to generate the attention map is further configured to divide the second image into multiple layers based on the distance from the camera used to acquire the second image.

[0146] Clause 20 - The apparatus according to Clause 19, wherein the processing circuitry configured to divide the second image into the plurality of layers is further configured to group one or more projection points based on the distance from the camera used to acquire the second image.

[0147] Clause 21 - The apparatus according to Clause 20, wherein the processing circuitry is further configured to calculate the distance from the camera based on a plurality of pixel coordinates in the plane of the second image and the center of the camera in the camera coordinate system.

[0148] Clause 22 - An apparatus according to any one of Clauses 20 to 21, wherein the processing circuitry configured to generate the consistency score is further configured to combine the attention map with the generated structural similarity index matrix (SSIM) score.

[0149] Clause 23 - The apparatus described in Clause 22, wherein the SSIM score indicates the consistency of the plurality of depth estimates.

[0150] Clause 24 - The apparatus according to Clause 22, wherein the processing circuitry is further configured to identify the one or more contaminated areas based on the consistency score.

[0151] Clause 25 - The apparatus according to Clause 14, wherein the processing circuitry configured to generate the classification is further configured to generate the classification of the one or more contaminated areas using a softmax function configured to normalize the probability distribution.

[0152] Clause 26 - The apparatus according to Clause 14, wherein the processing circuitry is further configured to operate the Advanced Driver Assistance System (ADAS) based on the classification of the one or more contaminated areas.

[0153] Clause 27 - A computer-readable medium storing instructions that, when applied by processing circuitry, cause the processing circuitry to: receive a plurality of images, wherein a first image of one or more of the images comprises a ranging image and a second image comprises a camera image; filter the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; generate a plurality of depth estimates based on the second image; generate an attention map by combining the filtered first image and the plurality of depth estimates; generate a consistency score based on the attention map indicating consistency between the depth estimates of the first image and the second image; modulate one or more features extracted from the second image based on the consistency score using a gating mechanism to generate one or more modulated features; and generate a classification of one or more contaminated regions in the second image based on the modulated features.

[0154] Item 28 - An apparatus for camera contamination detection, the apparatus comprising one or more components for performing the following steps: receiving a plurality of images, wherein a first image of the one or more images comprises a ranging image and a second image comprises a camera image; filtering the first image to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; generating a plurality of depth estimates based on the second image; generating an attention map by combining the filtered first image and the plurality of depth estimates; generating a consistency score based on the attention map indicating consistency between the depth estimates of the first image and the second image; modulating one or more features extracted from the second image based on the consistency score using a gating mechanism to generate one or more modulated features; and generating a classification of one or more contamination regions in the second image based on the modulated features.

[0155] Clause 29 - The device pursuant to Clause 28, wherein the one or more components include one or more processors implemented in a circuit.

[0156] Clause 30 - The device according to Clause 29 further includes a memory for storing the plurality of images.

[0157] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0158] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0159] By way of example, and not limitation, such computer-readable storage media may include one or more of RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (such as infrared, radio, and microwave) is included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs use lasers to reproduce data optically. Combinations of these should also be included within the scope of computer-readable media.

[0160] Instructions can be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0161] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Specifically, as described above, various units may be combined in a codec hardware unit, or various units may be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0162] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method, the method comprising: Receive multiple images, wherein a first image in one or more of the images includes a ranging image, and a second image includes a camera image; The first image is filtered by one or more processors to generate a filtered first image, wherein the filtering includes filters configured to fill one or more sparse regions in the first image; The one or more processors generate multiple depth estimates based on the second image; An attention map is generated by the one or more processors by combining the filtered first image and the plurality of depth estimates; The one or more processors generate a consistency score based on the attention map, indicating the consistency of depth estimation between the first image and the second image; The one or more processors use a gating mechanism to modulate one or more features extracted from the second image based on the consistency score to generate the modulated one or more features; as well as The one or more processors generate a classification of one or more contaminated areas in the second image based on one or more modulated features.

2. The method according to claim 1, wherein the first image is generated in real time by a light detection and ranging (LIDAR) sensor.

3. The method of claim 1, further comprising extracting one or more features from the first image before performing the filtering on the first image.

4. The method of claim 1, wherein the filter comprises a morphological filter having a pixel-based binary mask that extends one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter extends the one or more regions of the first image.

5. The method of claim 1, further comprising using a depth decoder to generate the plurality of depth estimates.

6. The method of claim 1, wherein generating the attention map further comprises dividing the second image into multiple layers based on the distance from the camera used to acquire the second image.

7. The method of claim 6, wherein dividing the second image into the plurality of layers further comprises grouping one or more projection points based on the distance from the camera used to acquire the second image.

8. The method of claim 7, further comprising calculating the distance from the camera based on a plurality of pixel coordinates in the plane of the second image and the center of the camera in the camera coordinate system.

9. The method of claim 1, wherein generating the consistency score comprises combining the attention map with the generated structural similarity index matrix (SSIM) score.

10. The method of claim 9, wherein the SSIM score indicates the consistency of the plurality of depth estimates.

11. The method of claim 9, further comprising identifying the one or more contaminated areas based on the consistency score.

12. The method of claim 1, wherein generating the classification further comprises using a softmax function configured to normalize a probability distribution to generate the classification of the one or more contaminated areas.

13. The method of claim 1, further comprising operating the advanced driver assistance system (ADAS) based on the classification of the one or more contaminated areas.

14. An apparatus for detecting camera contamination, the apparatus comprising: A memory, wherein the memory is used to store multiple images; and A processing circuit that communicates with the memory, wherein the processing circuit is configured to: Receive multiple images, wherein a first image in one or more of the images includes a ranging image, and a second image includes a camera image; The first image is filtered to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; Multiple depth estimates are generated based on the second image; An attention map is generated by combining the filtered first image and the multiple depth estimates; A consistency score indicating the consistency of depth estimation between the first image and the second image is generated based on the attention map; A gating mechanism is used to modulate one or more features extracted from the second image based on the consistency score to generate one or more modulated features; as well as The classification of one or more contaminated areas in the second image is generated based on one or more modulated features.

15. The apparatus of claim 14, wherein the first image is generated in real time by a light detection and ranging (LIDAR) sensor.

16. The apparatus of claim 14, wherein the processing circuitry is further configured to extract one or more features from the first image prior to performing the filtering on the first image.

17. The apparatus of claim 14, wherein the filter comprises a morphological filter having a pixel-based binary mask that extends one or more regions of the first image, wherein the binary mask is associated with a radius, and wherein the radius of the binary mask determines how far the morphological filter extends the one or more regions of the first image.

18. The apparatus of claim 14, wherein the processing circuitry is further configured to use a depth decoder to generate the plurality of depth estimates.

19. The apparatus of claim 14, wherein the processing circuitry configured to generate the attention map is further configured to divide the second image into multiple layers based on the distance from the camera used to acquire the second image.

20. The apparatus of claim 19, wherein the processing circuitry configured to divide the second image into the plurality of layers is further configured to group one or more projection points based on the distance from the camera used to acquire the second image.

21. The apparatus of claim 20, wherein the processing circuitry is further configured to calculate the distance from the camera based on a plurality of pixel coordinates in the plane of the second image and the center of the camera in the camera coordinate system.

22. The apparatus of claim 14, wherein the processing circuitry configured to generate the consistency score is further configured to combine the attention map with the generated structural similarity index matrix (SSIM) score.

23. The apparatus of claim 22, wherein the SSIM score indicates the consistency of the plurality of depth estimates.

24. The apparatus of claim 22, wherein the processing circuitry is further configured to identify the one or more contaminated areas based on the consistency score.

25. The apparatus of claim 14, wherein the processing circuitry configured to generate the classification is further configured to generate the classification of the one or more contaminated areas using a softmax function configured to normalize the probability distribution.

26. The apparatus of claim 14, wherein the processing circuitry is further configured to operate the advanced driver assistance system (ADAS) based on the classification of the one or more contaminated areas.

27. A computer-readable medium storing instructions that, when applied by processing circuitry, cause the processing circuitry to: Receive multiple images, wherein a first image in one or more of the images includes a ranging image, and a second image includes a camera image; The first image is filtered to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; Multiple depth estimates are generated based on the second image; An attention map is generated by combining the filtered first image and the multiple depth estimates; A consistency score indicating the consistency of depth estimation between the first image and the second image is generated based on the attention map; A gating mechanism is used to modulate one or more features extracted from the second image based on the consistency score to generate one or more modulated features; as well as The classification of one or more contaminated areas in the second image is generated based on one or more modulated features.

28. An apparatus for detecting camera contamination, the apparatus comprising one or more components for performing the following steps: Receive multiple images, wherein a first image in one or more of the images includes a ranging image, and a second image includes a camera image; The first image is filtered to generate a filtered first image, wherein the filtering includes a filter configured to fill one or more sparse regions in the first image; Multiple depth estimates are generated based on the second image; An attention map is generated by combining the filtered first image and the multiple depth estimates; A consistency score indicating the consistency of depth estimation between the first image and the second image is generated based on the attention map; A gating mechanism is used to modulate one or more features extracted from the second image based on the consistency score to generate one or more modulated features; as well as The classification of one or more contaminated areas in the second image is generated based on one or more modulated features.

29. The device of claim 28, wherein the one or more components include one or more processors implemented in a circuit.

30. The device of claim 29, further comprising a memory for storing the plurality of images.

Images