Detecting and improving sensor deterioration

Abstract

A sensor degradation detection and repair system includes one or more sensors configured to collect image data from an environment. A combination of techniques may be used to detect degradation within regions of image data captured by the sensors, including one or more of determining a level of visual consistency between related image regions captured by different sensors, determining a level of opacity of an image region, and / or measuring temporal behavior of an image region captured by a sensor over a period of time. Operation of a vehicle or other system may be controlled based at least in part on the detection of degradation in the image data captured by the sensors, including automatic cleaning of the sensor surface, reducing the level of reliance on image data received from the sensor, and / or changing the direction of travel of the vehicle.

Description

[Technical field]

[0001] The present invention relates to sensor degradation detection and improvement. [Background technology]

[0002] This application claims priority to U.S. patent application Ser. No. 16 / 728,532, filed Dec. 27, 2019, entitled “SENSOR DEGRADATION DETECTION AND REMEDIATION,” the entire contents of which are incorporated herein by reference.

[0003] Data captured by vehicle sensors in the environment can be used to assist the vehicle in navigation and obstacle avoidance as the vehicle moves through the environment. For example, cameras and other vehicle sensors can collect image data that the vehicle can analyze and use in real time for navigation, obstacle detection, and avoidance of roadway obstacles. However, the quality of the data collected by the vehicle sensors can degrade in certain circumstances, including based on environmental factors such as weather, traffic, or road conditions, as well as based on internal errors or failures that may occur within the sensors themselves. In such cases, the data collected by the vehicle sensors may be suboptimal or even unusable, potentially affecting vehicle navigation, obstacle detection and avoidance, and other vehicle functions that rely on the sensor data. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears, and the same reference numbers in different drawings indicate similar or identical items.

[0005] [Figure 1] FIG. 1 illustrates an example vehicle system that collects and analyzes image data within an environment to detect corruption in the image data. [Diagram 2] 4 is a flowchart illustrating an example process for detecting degradation in data captured by a variety of different vehicle sensors by capturing and analyzing data from the vehicle sensors. [Diagram 3] 4 is a flowchart illustrating an example process for detecting degradation in data captured by a vehicle sensor by analyzing pixel intensities in the image data captured by the vehicle sensor. [Figure 4] 4 is a flowchart illustrating an example process for detecting corruption in data captured by a vehicle sensor by analyzing the relative temporal behavior of image regions from the vehicle sensor over a period of time. [Diagram 5] 1 is a flowchart illustrating an example process for detecting corruption in data captured by vehicle sensors by performing a combination of techniques on image data received from a variety of different vehicle sensors. [Figure 6] 10 is a flowchart illustrating another example process for detecting corruption in data captured by vehicle sensors by performing a combination of techniques on image data received from a variety of different vehicle sensors over a period of time. [Figure 7A] 1A-1C show example images captured by vehicle sensors along with example images generated during the impairment detection techniques described herein. [Figure 7B] 1A-1C show example images captured by vehicle sensors along with example images generated during the impairment detection techniques described herein. [Figure 7C] 1A-1C show example images captured by vehicle sensors along with example images generated during the impairment detection techniques described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] As noted above, the quality of data collected by vehicle sensors may degrade in certain circumstances, including based on environmental factors such as weather, traffic, or road conditions. The present disclosure is directed to systems and techniques for detecting degradation of sensor data collected by sensors and controlling various systems based on the detection of the degradation. For example, the quality of image data captured by cameras and other sensors may degrade due to sensor obstructions (e.g., dirt, mud, raindrops, snow, or other materials on the sensor's lens) and / or environmental conditions (e.g., optical flare, fog, rain, snow, exhaust, etc.), and / or errors or failures within the sensor itself (e.g., focus errors, damage to the camera lens, mount, or other sensor components, errors in image capture or processing software, etc.). Systems and techniques according to the present disclosure may enable degradation of image data captured by sensors to be detected and identified using a combination of image capture and analysis techniques. In certain examples, one or more systems may be controlled to eliminate or repair the degradation of the image data. Certain techniques are described in the context of sensors in autonomous vehicles. However, the techniques described herein may be used in connection with non-autonomous vehicles, as well as other robotic systems.

[0007] For example, the techniques described herein may be applied to manufacturing, location monitoring and security systems, augmented reality, and the like. In one example, stereo matching techniques may be implemented, where image data may be captured by a variety of different sensors (e.g., cameras) on a vehicle and analyzed for visual consistency to identify degradation in the image data from one or both of the sensors. Although described throughout with respect to stereo images, the invention is not meant to be so limited as any multi-view geometry is contemplated, so long as there is at least partial overlap of fields of view. Overlapping images from different vehicle sensors may be captured, restored, and analyzed to match corresponding regions in the image data. The level of visual consistency between image regions may be compared to detect degradation of one of the vehicle sensors.

[0008] Other techniques may be used in place of or in conjunction with the image analysis for stereo matching. For example, a dark channel technique may be implemented, where a series of images may be captured from a vehicle sensor and pixel intensity analysis may be performed for relevant image regions. A minimum dark channel value may be identified based on pixel intensities from an image region in the series of images, and the dark channel value may be used to detect degradation of the image data. For example, a dark channel value may be determined for an image based on intensity values ​​associated with different image channels of data. An average dark channel image value may also be generated for an image based on intensity values ​​for the dark channel values ​​of various relevant image frames captured over time. A dark channel intensity threshold may be generated, for example, based on an average intensity over different regions of the average dark channel image, and a particular image region (e.g., pixel) may be compared to the dark channel intensity threshold. Image regions having an intensity higher than the dark channel intensity threshold may have a higher probability of being degraded by material occlusions such as optical flare, haze, fog, or raindrops, which may cause a higher dark channel intensity.

[0009] An additional or alternative technique for detecting degradation of image data captured by a vehicle sensor may include measuring the temporal motion of an image region relative to other surrounding image regions. For example, a series of images may be captured from a vehicle sensor over a period of time when the vehicle is moving. The temporal motion of an image region may be analyzed and compared to the temporal motion of adjacent image regions, and degradation of the image data may be detected based on differences in the relative temporal motion of the surrounding image regions.

[0010] Additionally or alternatively, one or more machine learning models may be used to detect degradation of image data captured by vehicle sensors. In such cases, the machine learning models and / or training data repository may operate on the vehicle and / or on an external computer system to train the machine learning models to accurately identify various types of degradation (e.g., dirt, mud, raindrops, optical flare, fog, lens focus errors, etc.). The image analysis component may analyze and classify the training data, and the machine learning engine may generate and train the machine learning models based on the classified training data. In certain examples, ground truth training data may be obtained automatically or semi-automatically from log data captured and stored by the vehicle traversing the environment. A variety of different machine learning techniques and algorithms may be used, and in some cases, a variety of different trained models may be used in combination with further degradation detection and repair techniques described herein. In certain examples, log data previously collected by the vehicle may be used to label the training data. For example, a vehicle occupant or operator may use a control to input the current weather (e.g., "rain") and a corresponding hashtag or other metadata label may be added to the data. Additionally or alternatively, other vehicle sensors (e.g., rain sensors) may be used, beginning with capturing and labeling sensor data to be used as training data for the machine learning model. In another example, if a repair technique is used to remove or repair an obstruction (e.g., cleaning the sensor to remove dirt or mud), failure to detect an obstruction after the repair technique is performed can be used to confirm the obstruction, and sensor data collected prior to the repair technique can be captured and labeled and used as training data for the machine learning model.

[0011] Using these and other techniques described herein, degradation of image data captured by a vehicle sensor may be detected and the source or type of degradation may be identified. Such degradation may be caused, for example, by the material on the surface of the vehicle sensor, optical flare (e.g., solar flare or lens flare caused by headlights, street lights, or other lighting phenomena within the detection area of ​​the vehicle sensor), fog affecting visibility, or other environmental factors such as focus errors or other malfunctions of the vehicle sensor. In other examples, degradation may be detected based on miscalibration of the vehicle sensor, which may be caused by hitting the sensor by an object (e.g., a pedestrian), or normal vibration modes, etc.

[0012] When degradation of image data from a vehicle sensor is detected, the systems and techniques described herein may include controlling operation of the vehicle to eliminate, mitigate, and / or remedy the effects of the degradation. For example, an automatic cleaning operation may be initiated to clean the surface of the sensor to eliminate the detected degradation. In other cases, the navigation and control system of the autonomous vehicle may reduce the confidence level of image data received from a sensor where degradation is detected, and / or the vehicle's heading may be altered to remedy the effects of the degradation. Thus, the techniques and embodiments described herein may provide technical advantages that improve the performance of autonomous vehicles and other computer systems that rely on sensor data, including faster and more accurate detection of sensor obstructions and errors, improved remediation techniques such as remediation based on the type (or source) and / or severity of sensor degradation, more efficient processing of sensor data based on the detected degradation of the sensor data, improved safety of the autonomous vehicle, or / and improved overall performance of computer systems that rely on the sensor data.

[0013] (Example Architecture) FIG. 1 illustrates an example autonomous vehicle system 100 that detects degradation of image data captured by vehicle sensors and controls vehicle operation based thereon. In one example, the autonomous vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the U.S. National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey without expecting full-time vehicle control from a driver (or passenger). However, in other examples, the autonomous vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. Furthermore, in some examples, the techniques for detecting degradation of vehicle sensor data and the associated vehicle control actions described herein may be usable by non-autonomous vehicles. Also, while there are examples in which the vehicle is a land vehicle, the techniques described herein are also applicable to aircraft, watercraft, and other vehicles. It is contemplated that the techniques described herein may be applied to more than just robotic control, such as autonomous vehicles. For example, the techniques described herein may be applied to manufacturing, location monitoring and security systems, augmented reality, and the like.

[0014] In accordance with the techniques described herein, the autonomous vehicle 102 may receive sensor data from sensors 104 of the autonomous vehicle 102. For example, the sensors 104 may include cameras configured to capture image data of the external environment surrounding the vehicle. As shown in this example, a variety of different camera sensors 104 may be mounted on the autonomous vehicle 102 at different positions relative to the vehicle 102 and / or integrated into the autonomous vehicle 102. Such sensors 104 may also be of different types or qualities, may be oriented at different angles, and may be configured with different image capture characteristics (e.g., different focal lengths, capture rates, focus, field of view, color capabilities, etc.) to capture a variety of different images 106 of the environment surrounding the autonomous vehicle 102. Thus, the sensors 104 may include any number of cameras associated with the autonomous vehicle 102, including common optical or light-based cameras, as well as infrared cameras, thermal imaging cameras, and night vision cameras, each of which may be configured to capture a different image 106 from the environment of the autonomous vehicle 102. Thus, the images 106 captured by the sensor 104 may include, for example, night vision images in which lower light levels are amplified allowing adjacent objects to be distinguished, or thermographic images captured by an infrared or thermal imaging camera.

[0015] The sensors 104 of the autonomous vehicle 102 may additionally or alternatively include one or more light detection and ranging (lidar) systems configured to transmit a pulsed laser to measure distances to adjacent objects, radio detection and ranging (radar) systems configured to detect and determine distances to adjacent objects using radio waves, sonar sensors configured to use sound pulses to measure the distance or depth of objects, time-of-flight sensors configured to measure the distance of objects based on the time difference between the emission of signals and their return to the sensors, or other sensors configured to capture other information about the environment. Additional sensors 104 of the autonomous vehicle 102 may include ultrasonic transducers, sonar sensors, a global positioning system (GPS) that receives location signals (e.g., GPS signals), as well as motion sensors (e.g., speedometers, compasses, accelerometers, and / or gyroscopes) configured to detect the current location, motion, and orientation of the autonomous vehicle 102. The additional sensors 104 may also include magnetometers, wheel encoder sensors, microphones, and other audio sensors, as well as environmental and climate sensors (eg, temperature sensors, light sensors, pressure sensors, rain and precipitation sensors, wind sensors, etc.).

[0016] As illustrated in FIG. 1 , the sensors 104 may include multiple instantiations of each of these or other types of sensors. For example, the lidar sensors may include individual multiple lidar sensors located at corners, front, rear, sides, and / or top of the vehicle 102. As another example, the camera sensors may include multiple cameras located at various locations about the exterior and / or interior of the vehicle 102. In one example, different sensors 104 of the same type (e.g., multiple cameras) and / or different sensors of different types (e.g., one camera and one lidar system) may have at least partially overlapping fields of view. The sensors 104 may provide input to the vehicle computing system 108 and / or transmit sensor data over one or more networks 130 to various external computing devices and systems (e.g., computing device 140) at a determined frequency, such as after a predetermined period of time, in near real-time, or the like.

[0017] Using the data captured by the various sensors 104, the autonomous vehicle 102 can receive images 106A(1)-106A(N) (collectively "images 106A") from a first sensor 104 (e.g., camera sensor 104A), images 106B(1)-106B(N) (collectively "images 106B") from a second sensor 104 (e.g., camera sensor 104B), images 106C(1)-106C(N) (collectively "images 106C") from a third sensor 104 (e.g., camera sensor 104C), etc. In this illustration, a small obstacle 103B(1) is illustrated in image 106B(1) to indicate that an obstruction (e.g., raindrop) or other obstacle may be attached to the surface or lens of the sensor 104 generating image 106B. Additional sets of image data may be received from sensors 104, such as thermographic and night vision cameras, or lidar and / or radar systems, which may be collectively referred to as images 106. Additionally, while this disclosure may primarily describe images 106 with reference to visual images, for ease and to better explain the techniques and concepts described herein, it is contemplated that the techniques and concepts may be applied to any sensor data capable of identifying and representing objects (e.g., a lidar point cloud including points representing objects) within the environment of autonomous vehicle 102.

[0018] In one example, the autonomous vehicle 102 may include one or more computing systems 108, which may include an impairment detection engine 110 and a vehicle control system 120. As shown in this example, the impairment detection engine 110 executing on the computing system 108 of the autonomous vehicle 102 may include one or more components and / or subsystems configured to detect impairments in image data 106 captured by the vehicle sensors 104. Different subsystems or components within the impairment detection engine 110 may be implemented to perform different impairment detection techniques, including an image comparison component 112, a pixel intensity analysis component 114, and an image motion analysis component 116. These techniques, each of which will be described in more detail below, may be used individually or in combination by the impairment detection engine 110 to analyze the image data 106 captured by the vehicle sensors 104 to detect and identify impairments in the image data 106.

[0019] Further, in some embodiments, the computing system 108 of the autonomous vehicle 108 may include one or more machine learning models 118 configured to receive, store, and execute machine learning models trained to detect degradation of the sensor data. In various examples, the machine learning models 118 (and / or the machine learning engine 148 described below) may be implemented as neural networks and / or other trained machine learning models. For example, the neural networks and / or any other machine learning techniques using the machine learning models 118 may be trained to receive the image data 106 and / or various other sensor data captured by the sensors 104 of the autonomous vehicle 102, and analyze and detect degradation in the sensor data caused by opaque substances or raindrops on the surface of the sensors 104, optical flare and other visual phenomena affecting the image data, and / or focus errors or other failures of the sensors 104.

[0020] As described above, the machine learning model 118 may include one or more artificial neural networks. A neural network is a biologically inspired technology in which input data can be transmitted through a series of connected layers to generate an output. Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network may utilize machine learning, which may refer to a broad class of such technologies in which an output is generated based on learned parameters.

[0021] In other embodiments, the machine learning model 118 (and the machine learning engine 148 described below) may include any other type of machine learning technique and algorithm may be used consistent with this disclosure.For example, machine learning techniques include regression techniques (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scatterplot smoothing (LOESS)), instance-based techniques (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), Elastic Net, least-angle regression (LARS)), decision tree techniques (e.g., classification and regression tree (CART), iterative dichotomiser 3 (ID3), chi-squared automated interaction detection (CHAID), decision strains, conditional decision trees), Bayesian techniques (e.g., naive Bayes, Gaussian naive Bayes, multinomial naive Bayes, average one-dependence (AODE)), and others). estimators), Bayesian confidence networks (BNNs), Bayesian networks), clustering techniques (e.g., k-means, k-medians, expectation maximization (EM), hierarchical clustering), association rule learning techniques (e.g., perceptrons, backpropagation, Hopfield networks, RBFNs (Radial Basis Function Network), deep learning techniques (e.g., deep Boltzmann machine (DBM), deep confidence network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction techniques (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixed discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble techniques (e.g., boosting, bootstrap aggregation (bagging), adaboost, hierarchical generalization (blending), gradient boosting machine (GBM), gradient boosted regression tree (GBRT), random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc., but are not limited to these.Further example architectures include neural networks such as ResNet70, ResNet101, VGG, DenseNet, and PointNet.

[0022] The computing system 108 may also include a vehicle control system 120 that initiates one or more corrective actions for the autonomous vehicle 102 in response to degradation detected in the image data 106 by the degradation detection engine 110. The vehicle control system 120 may include one or more system controllers that may be configured to control any or all systems of the autonomous vehicle 102, including perception, planning, steering, propulsion, braking, safety systems, emitters, communications, etc. As one example, described in more detail below, in response to detection of opaque substances (e.g., dirt, mud, raindrops) on a surface of the vehicle sensor 104, the vehicle control system 120 may initiate a process of sensor cleaning that attempts to remove the opaque substances from the surface. In another example, the vehicle control system 120 may change the orientation and / or heading of the autonomous vehicle 102 in response to detection of degradation in the image data 106 received from one or more sensors 104 for one end of the vehicle 102. In yet another example, the vehicle control system 120 can change configuration settings within the navigation and / or safety systems of the autonomous vehicle 102 to reduce reliance (e.g., reduce the reliability weighting) on ​​image data 106 associated with any sensor 104 in which degradation is detected.

[0023] 1, but so as not to obscure other components illustrated therein, the computing system 108 in the autonomous vehicle 102 may include a processing unit having one or more processors and memory communicatively coupled to the one or more processors. The impairment detection engine 110, the vehicle control system 120, and other subsystems and components within the computing system 108 may be implemented in hardware, software, or a combination of hardware and software components. In an embodiment, the subsystems or components within the impairment detection engine 110 and / or the vehicle control system 120 may be implemented as computer-executable instructions or other software code components stored on a non-transitory computer-readable medium within the computing system 108, which may be executed by a processor of the computing system 108 to perform the functions described herein. Additionally, while depicted in FIG. 1 as residing on the internal computing system 108 of the autonomous vehicle 102 for illustrative purposes, it is contemplated that one or more of the vehicle sensors 104, the degradation engine 110, and the vehicle control system 120 may be accessible to the vehicle 102 (e.g., may be stored in or otherwise accessible by memory remote from the vehicle 102, such as memory 144 of a remote computing device 140).

[0024] Also, while the components described herein (e.g., the degradation detection engine 110, the image comparison component 112, the pixel intensity analysis component 114, the image motion analysis component 116, the machine learning model 118, the vehicle control system 120) are described as separated for illustrative purposes, the operations performed by the various components may be combined or performed in any other component of the vehicle computing system 108.

[0025] The autonomous vehicle 102 may also include one or more wireless transceivers and / or other network devices to enable network connections and communications between the vehicle 102 and one or more other local or remote computing devices. For example, one or more wireless network interfaces in the computing system 108 may facilitate communications for the same autonomous vehicle 102 with other local computing systems 108, other computing systems in other autonomous vehicles, and / or various remote computing devices and systems (e.g., computing device 140). Such network interfaces and associated communication systems may also enable the vehicle 102 to communicate with teleoperated computing devices or other remote services.

[0026] Within the autonomous vehicle 102, wireless transceivers, physical and / or logical network interfaces, and / or other network components may enable the vehicle computer system 108 to access and connect to other computing devices or networks, such as the network 130. The communication system of the vehicle 102 may enable Wi-Fi-based communications, such as frequencies defined by the IEEE 1402.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables the vehicle's respective computing systems to interface with other computing devices or networks.

[0027] In an embodiment, one or more autonomous vehicles 102 can receive and / or transmit data over network 130 to one or more backend computing devices 140. Computing device 140 may include a processor 142 and memory 144 that store various components configured to interact with computing systems 108 of one or more autonomous vehicles 102. For example, computing device 140 may include an impairment detection component 145 configured to communicate with impairment detection engine 110 of one or more autonomous vehicles 102, a vehicle control component 146 configured to communicate with vehicle control system 120 of one or more autonomous vehicles 102, etc.

[0028] In one embodiment, the autonomous vehicle 102 may transmit any or all sensor data captured via the sensors 104, including image data 106, to the computing device 140. The sensor data may include raw sensor data and / or processed sensor data and / or representations of the sensor data. In one example, the sensor data (raw or processed) may be stored and / or transmitted as one or more log files.

[0029] Additionally or alternatively, the autonomous vehicle 102 may transmit to the computing device 140 data associated with the detected degradation, such as data indicating that degradation of the image data 106 was detected, the particular sensor 104 from which the degradation was detected, the type or source of the degradation (e.g., dirt, mud, rain, optical flare, or sensor error, etc.), the time and geographic location associated with the degradation, any remedial action taken (e.g., sensor cleaning, reducing reliance on sensor data, changing the vehicle's orientation or heading, etc.), and / or data indicating whether the remedial action was successful.

[0030] 1 may operate based on image data 106 captured directly by sensors 104 of the autonomous vehicle 102, in some embodiments, the vehicle 102 may receive some or all of its image data 106 from the computing device 140 and / or other remote data sources. For example, the autonomous vehicle 102 may communicate over the network 130 and receive image data 106 and / or other sensor data from other autonomous vehicles located near the vehicle 102, traffic cameras or traffic sensors near the vehicle 102, security cameras in buildings or parking lots, and / or satellite imaging systems in communication with the computing device 140.

[0031] In one embodiment, the impairment detection component 145 in the computing device 140 can analyze impairment detection data received from one or more autonomous vehicles 102. Based on the data received from the vehicles 102 and / or based on impairment detection policies received from a system administrator, the impairment detection component 145 can generate and update the impairment detection techniques used by the autonomous vehicle 102. For example, the impairment detection component 145 can select different combinations of impairment detection techniques (e.g., the image comparison component 112, the pixel intensity analysis component 114, and / or the image motion analysis component 116) for use by the autonomous vehicle 102. The impairment detection component 145 can also select and modify algorithms, matching thresholds, and other configuration settings used by the various impairment detection techniques. These updated impairment detection techniques and algorithms can be transmitted to the autonomous vehicle 102 for deployment in the impairment detection engine 110, thereby enabling the back-end computing device 140 to remotely and dynamically modify / optimize the impairment detection behavior of the autonomous vehicle 102 and / or other autonomous vehicles in the fleet of autonomous vehicles.

[0032] Further, in an embodiment, the computing device 140 can communicate with a variety of different autonomous vehicles 102 and can transmit different sets of deterioration detection instructions to different vehicles 102. For example, the computing device 140 may select different deterioration detection techniques and algorithms for different vehicles based on the vehicle's capabilities (e.g., the number and location of sensors 104, the vehicle's 102 cruising speed, acceleration, and braking force, remedial actions supported by the vehicle 102 due to degradation of the sensor data addresses), the current driving behavior of the vehicle 102 (e.g., current vehicle speed, road surface, traffic level, number and profile of occupants, etc.), the current geographic location in which the vehicle 102 is operating (e.g., county, state, country, or other jurisdiction), and / or the current environmental conditions around the vehicle (e.g., current weather conditions, road conditions, time of day, lighting conditions, etc.).

[0033] A vehicle control component 146 in the computing device 140 may be configured to receive data from and / or transmit to the autonomous vehicle 102 that may control operation of the vehicle control system 120 at the vehicle 102. For example, the vehicle control component 146 may determine and transmit to the vehicle 102 instructions that control remedial behaviors to be implemented by the vehicle 102 when degradation of the image data 106 is detected. Examples of such remedial behaviors may include which remedial techniques (e.g., automated cleaning of sensor surfaces, changing the vehicle's heading or orientation, reducing reliance on sensor data, etc.) are executed in response to different types of sensor data and severity of degradation. Similar to the degradation detection component 145, the vehicle control component 146 may determine a set of preferred or optimized remedial behaviors based on feedback data received from the autonomous vehicle 102 and / or degradation remediation policies received from a system administrator. Additionally, the vehicle control component 146 may select and transmit different sets of degradation repair instructions to different autonomous vehicles 102 based on any of the vehicle-specific factors listed above (e.g., vehicle capabilities, current driving behavior, geographic location, environment, etc.).

[0034] As noted above, in some embodiments, the computing device 140 may be configured to generate and train a machine learning model for use in detecting degradation of the image data 106 and / or selecting a remedial action to be performed by the autonomous vehicle 102 in response to the detected degradation. In such a case, the computing device 140 may include an image analysis component 147, a machine learning engine 148, and / or a training data repository 150. The training data 150 may include a set of image data 106 received from the autonomous vehicle 102 and / or other external data sources that includes examples of images 106 captured by the vehicle sensors 104 that exhibit various types of degradation (e.g., dirt, mud, rain drops, optical flare, lens focus error, etc.), as well as other images 106 that are free of any degradation. The image analysis component 147 may analyze and classify the training data 150, and the machine learning engine 148 may generate and train a machine learning model based on the classified training data 150. Any of the various machine learning techniques and algorithms described herein may be used, and in some cases, various different trained models may be used in combination. In one embodiment, a trained machine learning model can be generated by the computing device 140 and transmitted to one or more autonomous vehicles 102, where the trained model can be executed in real time to aid in detecting degradation of image data 106 captured by the vehicle sensors 104.

[0035] In some cases, the training data 150 may include or be derived from image data 106 captured by the autonomous vehicle 102. For example, the image data 106 captured by the vehicle 102 may be labeled (e.g., in the image metadata or with separate associated data) with corresponding degradation data indicating whether the image data 106 includes degradation (e.g., visual impairment or occlusion), as well as the type or source of the degradation, the severity of the degradation, the size and / or location of the degradation within the image data, etc. In some embodiments, the degradation data may be determined by analyzing additional sensor data and / or log data from the vehicle 102. For example, if image data 106 captured and analyzed by the autonomous vehicle 102 is determined to include impairments using one or more of the techniques described herein (e.g., stereo matching, dark channel, optical flow, etc.), that determination can then be validated against vehicle log data of captured sensor data to ascertain whether a remedial action (e.g., sensor cleaning, change in vehicle direction or orientation, etc.) was performed in response to the detected impairments in the image data 106, and if so, whether the remedial action was successful. In some cases, if a remedial action was performed and determined to be successful, the image data 106 may be labeled to indicate that impairments were present in the image data 106 and used as ground truth for training a machine learning model. For example, analyzing additional image data captured by the same sensor 104 after sensor cleaning can confirm (or refute) that the impairments detected before the sensor cleaning were in fact sensor occlusion. Similarly, analyzing additional image data captured after a change in vehicle direction or orientation can confirm (or refute) that the impairments detected before the change in vehicle direction or orientation were optical flare.In one example, the annotation service may be implemented to include a user interface that outputs the logged image data to a user and receives input from the user indicating whether the image data is degraded. The annotation service may also receive input via the user interface indicating which portions or regions of the image are degraded and the source and / or type of degradation. In another example, the logged images may be used in non-critical applications. Failures of such applications may be hypothesized to be caused by degraded images of the set and may be flagged to be annotated as degraded. Furthermore, if the vehicle is not able to continue to navigate autonomously due to degraded sensor data, one or more remedial actions may be taken (e.g., control of the vehicle may be taken over by a human driver, a request for assistance may be sent to a teleoperator, the vehicle may perform a safe stopping maneuver, etc.), and the corresponding image data may be collected and added to a training dataset for the machine learning model.

[0036] In one embodiment, the vehicle 102 and / or computing device 140 can generate synthetic training data 150 (both purely synthetic, as well as augmented data) that includes image data that has been degraded based on a library of undegraded image data (either real or synthetic). For example, one or more undegraded images captured from the vehicle 102 may be modified by using a transform or filter to overlay a separate image layer that includes one or more synthetically generated image degraded regions. Thus, real image data captured from the vehicle 102 may be overlaid with one or more layers of synthetically degraded data that includes representations of raindrops, dirt or mud spots, optical flares, etc.

[0037] The computing device 140 may implement one or more processes to generate synthetic degradation data based on a set of predefined features associated with different types or sources of degradation. Because the size, density, and distribution of raindrops that may accumulate against the vehicle sensor 104 during a rain shower are not random, the computing device 140 may implement the process using preprogrammed raindrop size, shape, and distribution patterns configured to model different types of rainstorms. For example, a first pattern for drizzle, a second pattern for heavy rain, a third pattern for heavy rain with heavy traffic, a fourth pattern for slow-moving heavy rain, a fifth pattern for high-velocity heavy rain, etc. may be stored. In one implementation, the computing device 140 may perform the process of synthetic data generation for a specific raindrop type / pattern using a set of preconfigured raindrop parameters for droplet generation rate, average size, size variation, mortality size, and / or droplet lifespan. Using a pre-programmed set of parameters, along with randomization functions and distributions applied to the parameters, as well as other conditions (e.g., lighting, traffic, vehicle speeds), the synthetic degradation data generation process can generate a synthetic image of raindrop patterns corresponding to a particular type and / or severity of rainstorm. Additionally, although this illustration relates to generating synthetic raindrop images, the synthetic degradation data generation process may also be used with sets of parameters for the size, shape, pattern, and distribution of other types of degradation of image data, such as dirt or mud smearing or sputtering on the sensor surface, optical flare caused by the sun or oncoming headlights, etc.

[0038] The processor of computing system 108 and processor 142 of computing device 140 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. In one example, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement encoded instructions.

[0039] The memory of the computing system 108 and the memory 144 of the computing device 140 are examples of non-transitory computer-readable media. The memory can store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, the memory can be implemented using any suitable memory technology, such as static RAM (SRAM), synchronous DRAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which those shown in the accompanying drawings are merely examples relevant to the description herein.

[0040] In one example, the memory of the computing system 108 and the memory 144 of the computing device 140 may include at least a working memory and a storage memory. For example, the working memory may be a limited capacity of high-speed memory (e.g., cache memory) used to store data operated on by an associated processor. In one example, the memory of the computing system 108 and / or the computing device 140 may include a storage memory, which may be a relatively large capacity of slower memory used for long-term storage of data. In some cases, the processor of the computing system 108 and / or the computing device 140 may not be able to directly operate on data stored in the storage memory, and the data may need to be loaded into the working memory to perform operations based on the data, as described herein.

[0041] 1 is illustrated as a distributed system, it should be noted that in alternative illustrations, components of the vehicle 102 may be associated with the computing device 140 and / or components of the computing device 140 may be associated with the vehicle 102. That is, the vehicle 102 may perform one or more of the functions associated with the computing device 140, and vice versa.

[0042] (Example Process) 2 illustrates an example process for detecting degradation in data captured by a variety of different vehicle sensors using stereo matching techniques in which data from the sensors is captured and analyzed. In one example, the process 200 may be accomplished by one or more components of the vehicle system 100, such as the image comparison component 112 of the autonomous vehicle 102, which may operate independently or in cooperation with various other components within the vehicle 102 or the external computing device 140. While FIG. 2 is described in the particular context of detecting degradation in image data 106 based on images captured by two different vehicle cameras, it should be understood that an embodiment may perform similar or identical operations based on images 106 captured from more than two cameras, images 106 captured by other types of vehicle sensors 104, and / or images 106 received from other data sources external to the autonomous vehicle 102.

[0043] At 202, image data 106 can be captured by various sensors 104 (e.g., cameras) of the autonomous vehicle 102. As illustrated in the illustrated portion of FIG. 2 associated with block 202, this description can refer to two visual images captured by the cameras of the autonomous vehicle 102, a first image 106A captured by a first camera sensor 106A, and a second image 106B captured by a second camera sensor 104A. In an embodiment, the image comparison component 112 can be configured to receive images from two or more sensors 104 (e.g., cameras) that have overlapping fields of view or detections. For example, the image comparison component 112 can retrieve previously captured images 106A and 106B and / or instruct the sensors 104A and 104B to capture new images. Additionally, in one example, the image comparison component 112 can require that the images 106A and 106B be captured either simultaneously or within a small time window. As shown in this example, images 106A and 106B include overlapping fields of view captured at or near the same point in time. Additionally, small opaque matter 203A (e.g., dirt, mud, etc.) may adhere to the surface or lens of sensor 104A, causing matter 203A to be visible in image 106A.

[0044] At 204, an image rectification process may be performed on the images 106A and 106B captured by the sensors 104A and 104B, respectively. The image rectification process may transform the images 106A and 106B into a common image plane. In one embodiment, a stereo rectification process may be performed, where the image comparison component 112 may identify an image transformation that achieves stereo alignment of horizontal epipolar lines in the common image plane. As above, this example describes image rectification and region matching for two images 106A and 106B, but it is understood that image data 106 from more than two vehicle sensors 104 (and / or image data 106 from other sources) may be rectified and region matched. In one example, the rectification process at 204 may be optional and / or other rectification techniques may be used. For example, complete calibration data of each sensor relative to other sensors (e.g., intrinsic and external) can be determined based on pre-stored information, and the image comparison component 112 and / or computing device 140 can pre-calculate which pixels in different images map to each other. Thus, 204 may include lookups or routines for comparison, as opposed to requiring image correction.

[0045] At 206, one or more regions in the first image 106A captured by the first sensor 104A can be matched to related (or corresponding) image regions in the second image 106B captured by the second sensor 104B. The region size can be determined by the image comparison component 112 and / or received as a configuration setting by the computing device 140. A larger region size may provide the distinct advantage of faster image processing, but in certain embodiments a smaller region size may be selected and may provide the technical advantage of identifying degraded regions with greater accuracy. Furthermore, because the sensors 104A and 104B may capture their respective images from different angles, there may be a certain level of visual inconsistency between the images 106A and 106B, and therefore a larger region size may provide better results.

[0046] In one embodiment, a process of naive stereo matching may be performed by the image comparison component 112. During the process of naive stereo matching, a minimum distance within a certain range along a common epipolar line may be determined and output to identify related or corresponding regions between the images 106A and 106B. To determine the minimum distance, a sum of squared distances (SSD) may be used, which may better preserve color, brightness, and exposure differences between the images in one embodiment. However, in one embodiment, other techniques may be used to determine the minimum distance. In some cases, the sensors 104A and 104B may also be configured to have the same exposure / gain.

[0047] Further, in one example, the naive stereo matching process can calculate a second image disparity in addition to determining the minimum distance. The second image disparity may be referred to as an infinite matching disparity. To calculate the infinite matching disparity between the two images, the patch distance between the sensors 104A and 104B is directly used at the same location and does not require searching along the epipolar line. The calculation of the infinite matching disparity for the images 106A and 106B can enable the process to efficiently remove sky pixels that may have less or no value to the navigation and control system of the autonomous vehicle 102. The result of the naive stereo matching process can be a final posterior sigmoid, which may be calculated as the product of two distances (e.g., the minimum distance disparity and the infinite matching disparity).

[0048] In one example, when performing the stereo matching process at 206, an exponentially increasing search range may be used from the top of the images 106A and 106B to the bottom of the images. This may provide an advantage over other stereo matching techniques by compensating for the fact that for an image 106 captured by a sensor 104 of an autonomous vehicle 102, the bottom portions of the images 106A and 106B may be assumed to be closer to the camera than the top portions of the images. Furthermore, in one example, the minimum distance used by the stereo matching technique to identify related regions between images 106A and 106B may be determined as a running average of a variety of different distance readings to remove any noise present in the distance readings.

[0049] After determining at 206 that a first image region from image 106A is associated (e.g., matches or corresponds) with a second image region from image 106B, at 208 the first image region from image 106A and the second image region from image 106B may be compared for visual consistency. As noted above, images 106A and 106B may be captured by different sensors 104A and 104B at different positions relative to the vehicle, so that even associated matching image regions within any degradation will have slightly different perspectives. However, in the absence of degradation (e.g., dirt, raindrops, optical flare, etc.) affecting either image 106A or 106B, the associated regions of images 106A and 106B may have a much higher level of visual consistency. In contrast, if degradation is affecting the associated regions of either image 106A or 106B, the level of visual consistency may be much lower. In other examples, additional image characteristics may be identified within the relevant regions of images 106A or 106B and additional image matching techniques such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Oriented FAST and Rotated Brief (ORB), etc. may be performed based on the external calibration. Additionally, corners or edges may be detected within an image region as a measure of blur and compared to the relative counts from other relevant image regions.

[0050] Thus, a visual consistency threshold may be defined and applied at 208 by the image comparison component 112 for each set of related regions in images 106A and 106B. If the level of visual consistency for each set of related image regions is below the threshold (YES at 208), the image comparison component 112 may determine at 210 that degradation has affected the image data captured by one or both of sensors 104A and 104B. As illustrated in this example, the presence of opaque material 203A (e.g., dirt, mud) in image 106A, which is not present in image 106B, may cause a lower level of visual consistency between image regions 106A and 106B. In contrast, if the level of visual consistency is at or above the threshold for the set of related image regions (NO at 208), the image comparison component 112 may determine at 212 that degradation has not affected the image data captured by sensor 104A or sensor 104B.

[0051] FIG. 3 illustrates another exemplary process for detecting degradation in data captured by a vehicle sensor using a dark channel technique in which pixel intensities are analyzed for image data captured by the vehicle sensor. For example, the dark channel prior art can be performed by analyzing intensity values ​​of pixels in an image to detect haze and other degradation in the image. For a particular image or a series of related image frames captured at different times, the dark channel may be identified as the image channel having the least intensity. When degradation is present in an image (e.g., optical flare, raindrops, fog, or haze), the pixel intensity of the dark channel for the degraded image region may be greater than other image regions that are not degraded. Thus, as described below, measuring and comparing the dark channel intensities of specific image regions (e.g., pixels or groups of pixels) may be used to detect degradation in the specific image regions. In one example, the process 300 may be accomplished by one or more components of the vehicle system 100, such as the pixel intensity analysis component 114 of the autonomous vehicle 102, which may operate independently or in cooperation with various other components within the vehicle 102 or the external computing device 140. While FIG. 3 is described in the particular context of detecting degradation of image data 106 based on images captured by two different vehicle cameras, it should be understood that an embodiment may perform similar or identical operations based on images 106 captured from more than two cameras, images 106 captured by other types of vehicle sensors 104, and / or images 106 received from other data sources external to the autonomous vehicle 102.

[0052] At 302, one or more images 106 may be captured by a sensor 104 (e.g., a camera) of an autonomous vehicle 102. As shown in the illustrated portion of FIG. 3 associated with block 302, this description begins with a first image 106A(t0) captured at time t0 and continues with subsequent images 106A(t0) captured at a later time t n Image 106A captured at (tn ) may refer to a series of various visual images captured by one sensor 104A of the autonomous vehicle 102 at various different times, ending with . In other examples, however, the pixel intensity analysis of process 300 may be performed using only one pixel.

[0053] At 304, the pixel intensity analysis component 114 may determine one or more image regions within the image data received at 302. As noted above, the region size may be determined by the pixel intensity analysis component 114 and / or may be received as a configuration setting by the computing device 140. Furthermore, while a larger region size may provide the distinct advantage of faster image processing, in certain embodiments, a smaller region size may be selected and may provide the technical advantage of identifying degraded regions with greater accuracy. In one example, a region size as small as one pixel may be selected.

[0054] At 306, the pixel intensity analysis component 114 may perform pixel intensity analysis for each of the determined image regions at 304. In an embodiment, the pixel intensity analysis at 306 may be a dark channel (or dark channel first) technique, where a dark channel may be determined for an image or image region based on intensity values ​​associated with different image channels of data. In the dark channel technique, an image region may be split into separate red-green-blue (RGB) channels, and a minimum pixel intensity from the three channels may be selected. If an image region is not affected by degradation, empirical analysis of captured images has demonstrated that at least one of the channel intensities may be zero or close to zero (e.g., the dark channel for an image region). However, if the image region is affected by a particular type or source of degradation, such as raindrops, optical flare, or camera focus error, the degradation causes the image region to have higher pixel intensity values ​​in the dark channel.

[0055] As described above, the pixel intensity analysis at 306 may be performed for one image 106A or for a series of images captured by the vehicle sensor 104A over a period of time (e.g., 106A(t0) to 106A(t n For an image, the dark channel intensity for an image region may correspond to the minimum pixel intensity of the distinct RGB channels in that image region. n ) are captured and analyzed, pixel intensity analysis component 114 may perform the same pixel intensity analysis (e.g., dark channel) for each of the series of images and select the minimum dark channel value for the image region of the series of images. Further, while color channel RGB is used in this example, it is understood that any other color channel model or encoding system (e.g., YUV, CMYK) may be used in other examples. An average dark channel image value may be generated for the image data based on the intensity values ​​for the dark channel of various related image frames captured over time.

[0056] After determining pixel intensity (e.g., dark channel) values ​​for one or more image regions of the image data 106 at 306, the pixel intensity values ​​for the image regions at 308 may be compared to an intensity threshold. For example, a dark channel intensity threshold may be generated based on an average intensity across different regions of an average dark channel image, and a particular image region (e.g., pixel) may be compared to the dark channel intensity threshold. Image regions having an intensity higher than the dark channel intensity threshold may have a higher probability of being degraded by a particular occlusion, such as optical flare, haze, fog, or raindrops, which may cause a higher dark channel intensity. As described above, when using dark channel techniques, image regions affected by a particular degradation may have a higher minimum pixel intensity, while image regions not affected by the degradation may have a lower minimum pixel intensity. Thus, a pixel intensity threshold (e.g., dark channel threshold) may be defined and applied at 308 by the pixel intensity analysis component 114 to each region in the received image 106. For each image region, if the pixel intensity value (e.g., dark channel value) is greater than the threshold (YES at 308), the pixel intensity analysis component 114 can determine, at 310, that an impairment affects the region of the image 106 captured by the sensor 104. In contrast, if the pixel intensity value (e.g., dark channel value) does not meet the threshold (NO at 308), the pixel intensity analysis component 114 can determine, at 312, that there is no impairment affecting the region of the image 106 captured by the sensor 104A.

[0057] 4 illustrates another example process for detecting degradation in data captured by a vehicle sensor using optical flow techniques in which the temporal motion of image regions from the vehicle sensor is analyzed over a period of time. In one example, the process 400 may be accomplished by one or more components of the vehicle system 108, such as the image motion analysis component 116 of the autonomous vehicle 102, which may operate independently or in cooperation with various other components within the vehicle 102 or external computing device 140. FIG. 4 illustrates a sequence of images 106A (e.g., 106A(t0)-106A(t n Although described in the specific context of detecting degradation of image data 106A based on a signal from a vehicle sensor 104, it is understood that an embodiment may perform similar or identical operations based on images 106 captured by other types of vehicle sensors 104 and / or images 106 received from other data sources external to the autonomous vehicle 102.

[0058] At 402, a series of images 106 can be captured by a sensor 104 (e.g., a camera) of an autonomous vehicle 102. As shown in the illustrated portion of FIG. 4 associated with block 402, this description begins with a first image 106A(t0) captured at time t0, and continues with a second image 106A(t0) captured at time t0. n Image 106A captured at (t n ) may refer to a series of various visual images captured by one sensor 104A of the autonomous vehicle 102 during a section within a time interval ending with .

[0059] At 404, the image motion analysis component 116 may determine one or more image regions within the image data received at 402. As noted above, the region size may be determined by the image motion analysis component 116 and / or may be received as a configuration setting by the computing device 140. Furthermore, while a larger region size may provide the distinct advantage of faster image processing, in certain embodiments, a smaller region size may be selected and may provide the technical advantage of identifying degraded regions with greater accuracy.

[0060] At 406, the image motion analysis component 116 can determine the relative temporal motion of each of the image regions determined at 404. The temporal motion of the image regions can be calculated based on the time course of the sequence of images 106A(t0)-106A(t n) of the image region. For example, when the autonomous vehicle 102 is operating, the image data 106 captured by the vehicle sensors 104 (e.g., cameras) may be expected to continually change as the visual environment around the vehicle 102 changes (i.e., as the vehicle's position / perspective changes over time, the view captured by the sensors 104 changes as well). The rate of temporal movement may be based on the speed and direction of the vehicle 102, the type of environment surrounding the vehicle (e.g., highway vs. city), and other factors. Thus, the relative temporal behavior of an image region may refer to the temporal behavior of adjacent image regions across a series of images compared to the temporal behavior of adjacent image regions across the same series of images. In other examples, the measurement of the temporal behavior of an image region need not be relative compared to the temporal behavior of adjacent image regions. For example, the temporal behavior can be measured at 406 and compared to a threshold at 408 based on the determined temporal behavior within the image region (e.g., without relying on the temporal behavior of other regions in the same series of images). In such cases, if the temporal motion of an image region is below a threshold, image motion analysis component 116 can determine a higher likelihood of occlusion in the image region preventing detection of the temporal motion.

[0061] A high level of relative temporal motion in an image region may indicate that degradation (e.g., dirt, raindrops, optical flare, etc.) affects the image data 106 at the location of the image region or in adjacent image regions. In contrast, a low level of relative temporal motion in an image region may indicate that degradation is not affecting the image data 106 at the location of the image region or in adjacent image regions. Thus, a threshold for relative temporal motion may be defined and applied at 408 by the image motion analysis component 116 for each region in the received image 106. If, for the respective image region, the level of relative temporal motion is greater than the threshold (YES at 408), the image motion analysis component 116 may determine at 410 that degradation affects the region of the image 106 captured by the sensor 104 and / or adjacent image regions. In contrast, if the level of relative temporal motion does not meet the threshold (NO at 408), the image motion analysis component 116 may determine at 412 that the degradation does not affect areas of the image 106 or adjacent image regions.

[0062] The above three separate techniques for detecting degradation in image data 106 captured by vehicle sensors 104, visual consistency analysis between overlapping images captured by separate sensors 104 as described in FIG. 2, pixel intensity analysis of image regions captured by sensors 104 as described in FIG. 3, and relative temporal motion analysis of image regions captured by sensors 104 as described in FIG. 4, may be performed individually or in any combination in various embodiments. In further illustration, any combination of these three degradation detection techniques may also be performed in conjunction with other techniques, such as using one or more trained machine learning models to analyze image data 106 captured by vehicle sensors 104. Thus, each of these techniques, individually or in combination, may provide technical advantages in autonomous vehicles or other sensor data processing computer systems by improving the speed and accuracy of detection of sensor obstructions and errors.

[0063] 5 and 6 illustrate two alternative processes for detecting corruption in image 106 data captured by a vehicle sensor 104, based on different combinations of the above techniques.

[0064] FIG. 5 illustrates a first exemplary process in which degradation is detected by performing a visual consistency analysis between overlapping images captured by separate sensors 104 described in FIG. 2, by combining with any of pixel intensity analysis of image regions captured by sensors 104 described in FIG. 3, relative temporal motion analysis of image regions captured by sensors 104 described in FIG. 4, and / or by accessing a machine learning model trained to identify degradation of image data. At 502, a first image data 106 may be captured by a vehicle sensor 104. At 504, a second image data 106 having an overlapping field of view with the first image data 106 may be captured by the second vehicle sensor 104. At 506, the first image data and the second image data may be rectified and matched to determine one or more associated (or corresponding) image regions in the first image data and the second image data. At 508, the associated image regions in the first image data and the second image data may be compared to determine a level of visual consistency between the image regions. Hence, 502-508 of FIG. 5 may be similar or identical to 202-208 and may be performed as described above by the image comparison component 112 operating within the degradation detection engine 110.

[0065] Continuing with FIG. 5, an exemplary process may perform one or more of performing a pixel intensity analysis of the first and / or second image regions at 510, accessing one or more machine learning models trained to recognize impairments in the first and / or second image regions at 512, or performing a relative temporal motion of the first and / or second image regions at 514. As noted above, the relative temporal motion analysis performed at 514 may be similar or identical to the relative temporal motion analysis described above at 402-408 and may be performed by the relative temporal motion analysis component 116, while the pixel intensity analysis performed at 510 may be similar or identical to the pixel intensity analysis described above at 302-308 and may be performed by the pixel intensity analysis component 114. Additionally, accessing a machine learning model at 512 to recognize impairments in the image data may include techniques similar or identical to those described above performed by the machine learning model 118. In various examples, any combination of the individual techniques at 510 , 512 , and 514 may be performed by the impairment detection engine 110 .

[0066] At 516, the impairment detection engine 110 of the autonomous vehicle 102 may determine whether impairment affects the image data 106 captured by one or more vehicle sensors 104. The determination at 516 may be performed based on a combination of one or more of the determination of a level of visual consistency between image regions performed at 508, the pixel intensity analysis performed at 510, the machine learning model accessed at 512, or the temporal motion analysis performed at 514. In one example, the impairment detection engine 110 may determine and apply different thresholds for the different determinations at 508, 510, 512, and / or 514, such that impairment is detected only if the individual techniques detect impairment. In other examples, the degradation detection engine 110 can determine and apply a combined threshold where a first data metric representing the level of visual consistency determined at 508 is combined (e.g., summed or multiplied) with either a second data metric representing the level of pixel intensity determined at 510, a third data metric representing the confidence of the machine learning model that degradation was detected at 512, and / or a fourth data metric representing the level of relative temporal activity determined at 514.

[0067] If the impairment detection engine 110 detects degradation in the image data 106 captured by any of the vehicle sensors 104 (YES at 516), then one or more operations of the autonomous vehicle 102 can be controlled in response to the detection of the degradation at 518. In contrast, if the impairment detection engine 110 does not detect degradation in the image data 106 captured by the vehicle sensors 104 (NO at 516), then in this example, the process can return to 502 to await capture of further image data 106 from the vehicle sensors 104.

[0068] 6 illustrates a second exemplary process in which degradation is detected by performing visual consistency analysis between overlapping images captured by separate sensors 104 described in FIG. 2 in combination with pixel intensity analysis of image regions captured by sensors 104 described in FIG. 3 and by relative temporal motion analysis of image regions captured by sensors 104 described in FIG. 4. At 602, first image data 106 can be captured from a vehicle sensor 104. At 604, related image data 106 can be retrieved and includes image data 106 captured from one or more different sensors 104 having an overlapping field of view with the first image data 106, and / or a series of images 106 captured by the same or other vehicle sensors 104 over a period of time. At 606, the first image data received at 502 and the related image data received at 504 can be analyzed (e.g., corrected and / or matched) to determine one or more related (or corresponding) image regions between the sets of image data. As described above, the related sets of image data may represent images 106 captured by different vehicle sensors 104 at the same or similar time points and / or may represent a series of images 106 captured by the same or different vehicle sensors 104 over a period of time. At 608, related image regions in the image data 106 captured by the different sensors 104 may be compared to determine a level of visual consistency between the image regions, which may be similar or identical to 202-208 and may be performed as described above by an image comparison component 112 operating within the degradation detection engine 110. At 610, a pixel intensity analysis of the first image region and / or the second image region may be performed, which may be similar or identical to the pixel intensity analysis described above at 302-308 and may be performed by a pixel intensity analysis component 114.At 612, a relative temporal motion analysis of the first image region and / or the second image region can be performed, which may be similar or identical to the relative temporal motion analysis described above at 402-408 and may be performed by the relative temporal motion analysis component 116. At 614, the degradation detection engine 110 can access one or more machine learning models trained to recognize degradation in the first image region and / or the second image region. As described above, the machine learning models can be trained to detect degradation in the image data and / or to identify specific types or sources of degradation.

[0069] At 616, the degradation detection engine 110 of the autonomous vehicle 102 may determine whether degradation affects the image data 106 captured by one or more vehicle sensors 104. The determination at 616 may be performed based on a combination of the determination of the level of visual consistency between image regions performed at 608, the pixel intensity analysis performed at 610, the temporal motion analysis performed at 612, and the machine learning model accessed at 614. As noted above, in various different examples, the determination at 616 may be based solely on any of the individual techniques at 608, 610, 612, or 614, or on any combination of these or other techniques. Examples of other techniques that may be used to identify degradation in the sensor data include analyzing the saturation level, exposure, contrast, or other image data characteristics of the image data. Further, the degradation detection engine 110 can determine and apply different thresholds for the different decisions at 608, 610, 612, and 614, such that degradation is detected only if one or more of the techniques detect the degradation with a predetermined threshold level of confidence, degradation size threshold, and / or degradation severity threshold. In other examples, the degradation detection engine 110 can determine and apply one or more combined thresholds, where a first data metric representing a level of visual consistency determined at 608 is combined (e.g., summed or multiplied) with a second data metric representing a level of pixel intensity determined at 610, and / or a third data metric representing a level of relative temporal activity determined at 612, and / or a fourth data metric representing a confidence in the machine learning model that the degradation was detected.

[0070] If the impairment detection engine 110 detects degradation in the image data 106 captured by any of the vehicle sensors 104 (YES at 616), then one or more operations of the autonomous vehicle 102 can be controlled in response to the detection of the degradation at 618. In contrast, if the impairment detection engine 110 does not detect degradation in the image data 106 captured by the vehicle sensors 104 (NO at 616), then in this example, the process can return to 602 to await capture of further image data 106 from the vehicle sensors 104.

[0071] The following equations describe an example embodiment that may be used in combination with the degradation detection techniques described above to determine the probability that a particular image region (e.g., a pixel or group of pixels in an image) is degraded by the presence of optical flare. In this example, the techniques described above may be used to determine the probability that an image region (SC ob Observed stereo consistency of the image region (OF ob ) and the observed optical flow in the image region (DC ob We determine a quantifiable metric for the observed dark channel of the stereo image (denoted as SC). th ), optical flow (OF th ), and the dark channel (DC th An associated threshold for each of the techniques (denoted as ##EQU1##) can be determined based on an analysis of minimum and maximum values ​​for each technique within a plurality of samples of the optical flare image region.

[0072] In this example, the probability that an image region will be degraded by optical flare may be defined as follows: P (isFlare | SC ob , OF ob , DC ob ) = = P (SC ob , OF ob , DC ob | isFlare) * P (isFlare) / (P (SC ob , OF ob , DC ob | isFlare) * P (isFlare) + P (SC ob , OF ob , DC ob | NotFlare) * P (NotFlare)) = P (SC ob | isFlare) * P (OF ob | isFlare) * P (DC ob | isFlare) / (P (SC ob | isFlare) * P (OF ob | isFlare) * P (DC ob | isFlare) + P (SC ob , OF ob , DC ob | NotFlare) * P (NotFlare) / P (isFlare)) = sigmoid (SC ob - SC th ) * sigmoid (OF ob - OF th ) * sigmoid (DC ob - DC th ) / (sigmoid (SC ob - SC th ) * sigmoid (OF ob - OF th ) * sigmoid (DC ob - DC th ) + f (vehicle speed) * g (sun angle)))

[0073] As shown in the above example, a sigmoid function may be used to estimate the conditional probability of optical flare and no optical flare, given observed stereo consistency data, optical flow data, and dark channel data of an image region. ob , OF ob , DC ob |NotFlare) may be a function of the current vehicle speed and / or sun angle relative to the vehicle. In one example, the f (vehicle speed) parameter may be used to implement a policy where detection is stopped if the average observed optical flow falls below a certain threshold. The g (sun angle) parameter need not be used when sun angle data is not available or relevant, such as driving at night or on cloudy days, and may be represented as a constant value.

[0074] The output of the above formula is P(isFlare|SC ob , OF ob , DC ob ) is greater than a predetermined threshold, the degradation detection engine 110 may determine that the image region is degraded by optical flare, while if the output of the equation is less than the threshold, the degradation detection engine 110 may determine that the image region is not degraded by optical flare. Additionally, while the exemplary equations above are specific to detecting optical flare, similar equations and techniques may be used to detect other types or sources of degradation (e.g., dirt, mud, fog, raindrops, sensor errors, etc.).

[0075] Referring again to 518 and 618, when the operation of the autonomous vehicle 102 is controlled in response to detection of degradation of the image data 106, one or a combination of remedial actions can be performed. In some cases, an automatic cleaning process may be initiated to clean the surface of the sensor 104 (e.g., camera lens) on which the degraded image data 106 was captured. In some cases, the navigation system and / or other vehicle control systems of the autonomous vehicle 102 may be reconfigured to reduce the level of reliance on image data received from the sensor 104 on which the degraded image data 106 was received. Additionally or alternatively, the heading and / or orientation of the vehicle 102 may be altered to remediate the effects of the detected degradation. Other examples of remedial actions that may be performed at 518 and / or 618 may include adjusting one or more operational driving parameters of the vehicle. For example, a semi-autonomous or fully autonomous vehicle may be configured with operational driving parameters that control the vehicle speed (e.g., maximum and / or cruising speed), as well as driving parameters such as which roads / surfaces the vehicle may select, whether the vehicle will turn right on a red light, whether the vehicle will perform an unprotected turn, etc. Any of these operational driving parameters may be adjusted or overridden at 518 or 618 in response to detection of degradation of the image data.

[0076] In some examples, some repair actions may be performed based on detection of image data degradation using any one of the image degradation techniques, while other repair actions may be performed based on degradation of the image data using various image degradation techniques. For example, the vehicle may initiate sensor cleaning in response to detection of image data degradation using any one of the stereo matching, dark channel, optical flow, or machine learning techniques, and may bring the vehicle to a stop in response to detection of image data degradation using various techniques described herein. In some examples, the output of one or more of the techniques described herein may be input to a further machine learning model trained to determine potential degradation based on the output of various degradation detection techniques (e.g., stereo matching, dark channel, optical flow, machine learning techniques, etc.).

[0077] In one embodiment, the remedial actions performed at 518 and 618 may depend on the type or source of the impairment detected, along with various other factors that may be determined in real time by the computing system 108 of the autonomous vehicle 102. For example, when detecting an impairment, the impairment detection engine 110 may also determine the type or source of the impairment, such as an opaque material (e.g., dirt, mud, etc.) on the surface of the sensor 104, a translucent or light distorting material (e.g., water droplets) on the surface of the sensor 104, optical flare or other optical phenomena, or focus error or other anomaly of the sensor 104. The determination of the type or source of the impairment may be based on the execution of various impairment detection techniques described above. For example, the impairment detection technique that analyzes visual consistency between overlapping image regions captured by separate sensors 104 described above with reference to FIG. 2 may be effective at identifying opaque or translucent material on the surface of the sensor 104, and may be relatively ineffective at identifying optical flare or other visual phenomena. In contrast, the pixel intensity analysis technique described above with reference to FIG. 3 may be effective at identifying translucent materials on the surface of the sensor 104, and at identifying optical flare and other visual phenomena, but may be relatively ineffective at identifying opaque materials on the surface of the sensor 104. Finally, the relative temporal motion analysis technique described above with reference to FIG. 4 may be effective at identifying opaque materials on the surface of the sensor 104, and at identifying optical flare or other visual phenomena, but may be relatively ineffective at identifying translucent materials on the surface of the sensor 104. Thus, the techniques used to detect degradation in the image data 106, and the results of those techniques, may provide a further technical advantage, in that the degradation detection engine 110 may also be able to determine a specific type or source of degradation (e.g., dirt, mud, raindrops, optical flare, fog, lens focus error, etc.), thereby enabling appropriate remedial action to be performed based on the type or source of degradation.

[0078] Additionally or alternatively, other techniques may be used to determine the type or source of the detected impairment of the image data 106 captured by the vehicle sensors 104. For example, as described above, visual analysis (e.g., rule-based analysis and / or trained machine learning models) of the image data 106 may be performed by the impairment detection engine 110, alone or in combination with any of the above techniques, to determine the type or source of the impairment. Furthermore, data from any of the other sensors 104 described herein and / or data received from one or more external data sources may also be used to identify the type or source of the detected impairment. For example, the current time, the orientation of the vehicle 102, the sun angle relative to the vehicle 102, and the current weather conditions may be received and analyzed by the impairment detection engine 110 to determine that optical flare may cause the detected impairment. As another example, the current weather conditions, road surface and conditions, traffic conditions, and vehicle speed may be analyzed in conjunction with the image data 106 to determine that dirt, mud, and / or water droplets may cause the detected impairment.

[0079] In some embodiments, the vehicle control system 120 may select a remediation option based on the source or type of degradation. For example, if the degradation detection engine 110 determines that the degradation is due to opaque or translucent materials (e.g., dirt, mud, raindrops, etc.) on the surface of the sensor 104, the vehicle control system 120 may initiate a process of automatic cleaning of the degraded surface. In contrast, if the degradation detection engine 110 determines that the degradation is due to optical flare or other light phenomena, a process of automatic cleaning of the sensor may not remediate the degradation and a different remedial action (e.g., reducing reliance on image data 106) may be initiated.

[0080] In one example, vehicle control system 120 may use additional data factors and additional analysis to determine whether and when a remedial action will be taken in response to the detected degradation, as well as which remedial action will be taken. For example, degradation detection engine 110 and / or vehicle control system 120 may determine and analyze the severity of the degradation (e.g., which portion of image data 106 is degraded), the location of the degradation within image data 106 (e.g., diagonal to the center of the captured image 106), how much the image data 106 from sensor 104 has degraded (e.g., a time threshold may be used before a remedial action is taken), and / or whether one or more other vehicle sensors 104 are available to capture the degraded portion of image data 106 (e.g., cameras or other sensors 104 with overlapping fields of view). Based on any of these factors, alone or in combination, vehicle control system 120 may determine whether the detected degradation will trigger a remedial action, when the remedial action will be taken, and which remedial action will be taken to address the degradation.

[0081] 7A-7C depict three exemplary sets of image data captured by vehicle sensor 104 along with an exemplary modified set of images that may be generated during the degradation detection techniques described herein. Specifically, each set of images illustrated in FIGS. 7A, 7B, and 7C includes a first exemplary image captured by vehicle sensor 104, a second image illustrating one or more of the degradation detection techniques described herein, and a third image illustrating output from one or more of the degradation detection techniques described herein. With respect to images 701a, 701b, and 701c (collectively "images 701"), image 701a is an exemplary image captured by vehicle sensor 104 (e.g., a camera) where the image is partially degraded by raindrops on the lens surface of the sensor, while exemplary images 701b and 701c are partially degraded by optical flare.

[0082] Intermediate images 702a, 702b, and 702c (collectively, "images 702") illustrate an example of the above techniques for detecting degradation in image data, where a potential degradation region 704 is indicated by a contour shape. The potential degradation region 704 may be associated with a probability that degradation exists in the particular region. The probability may be based on the output of one or more degradation detection techniques applied. The probability may be represented as a probability distribution or a heat map. In the example of FIG. 7C, four regions of potential degradation are illustrated, including a first region of potential degradation 704a having a first probability, a second region of potential degradation 704b having a second probability, a third region of potential degradation 704c having a third probability, and a fourth region of potential degradation 704d having a fourth probability. The probability may be used in determining whether the potential degradation region represents an occlusion. For example, if the probability of there being degradation at a particular location is at or above a threshold probability, the region may be determined to be degraded. If the probability is below the threshold, the region will be determined to be not degraded.

[0083] Finally, images 703a, 703b, and 703c (collectively, "images 703") may represent the output of the degradation detection techniques described herein. In image 703, non-degraded portions of each image are depicted in black, while portions of each image depicted in white represent regions determined to be degraded. In the example of FIG. 7C, a first region of potential degradation 704a, a second region of potential degradation 704b, and a third region of potential degradation 704c have a probability above a threshold probability and are therefore represented as degraded regions depicted in white in output image 703c. Potentially degraded region 704d in this example has a probability below a threshold (e.g., preventing degradation) and is therefore not represented in output image 703c.

[0084] As described above, the vehicle control system 120 can determine whether a detected impairment will trigger a remedial action, when the remedial action will be performed, and / or which remedial action will be performed to address the detected impairment. In some embodiments, some or all of these determinations may be based on the image 703. For example, a representation of the degraded and non-degraded portions of the image 703 may be used to determine the severity of the impairment (e.g., how much of the image 703 is degraded and how much is usable), the location of the impairment within the image data 703 (e.g., center diagonal, bottom vs. top, etc.), and / or whether any other sensors 104 are available to capture the degraded portion of the image 703 (e.g., a camera or other sensor 104 with a field of view that covers the degraded portion). Thus, the impairment detection engine 110 and / or the vehicle control system 120 can analyze the output image 703 and determine when and which remedial action may be performed in response to the impairment represented by the image 703.

[0085] (Example clauses) Any of the example clauses in this section may be used with any of the example clauses or any of the other examples or embodiments described herein.

[0086] A. The vehicle comprises one or more processors, a second sensor that captures sensor data of the vehicle's environment, the second sensor having a field of view that at least partially overlaps with the field of view of the first sensor, and a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the vehicle to perform operations including receiving first image data captured by the first sensor at a first time, identifying a first image region within the first image data, reading second image data captured by the second sensor, identifying a second image region within the second image data associated with the first image region, determining a level of visual consistency between the first image region and the second image region, determining the level of visual consistency based on a combination of the determined level of visual consistency and at least one of an intensity associated with the first image region, a temporal behavior measurement associated with the first image region based on third image data captured at a second time, different from the first time, or output from a machine learning model trained to detect degradation based on the image data, and controlling operation of the vehicle based at least in part on the detection of the degradation.

[0087] B. The vehicle of paragraph A, wherein detecting degradation of the first image data includes identifying a source or type of degradation, wherein the identified source or type of degradation includes at least one of an opaque substance on a surface of the first sensor, water droplets on the surface of the first sensor, optical flare within a detection field of the first sensor, or a visual focus error of the first sensor.

[0088] C. The vehicle of paragraph A or B, wherein controlling operation of the vehicle based at least in part on detecting degradation of the first image data includes at least one of initiating cleaning of a surface of the first sensor, reducing reliance on data received from the first sensor, changing a direction of travel of the vehicle, or adjusting one or more operational driving parameters of the vehicle.

[0089] D. A vehicle as described in any one of paragraphs A-C, wherein detecting degradation of the first image data is based on measuring a determined level of visual consistency between the first image region and the second image region, an intensity associated with the first image region, and a temporal motion associated with the first image region.

[0090] E. A vehicle as described in any one of paragraphs A-D, wherein detecting the deterioration further includes inputting the first image data into a machine learning model and receiving an output from the machine learning model, the output being based at least in part on the first image data, the output being indicative of deterioration of the first image data captured by the first sensor.

[0091] F. The method comprises receiving first image data captured by a first sensor of the vehicle at a first time; identifying a first image region within the first image data; detecting degradation of the first image data captured by the first sensor, where the detecting is based on at least two of a determined level of visual consistency between the first image region and a corresponding second image region in second image data captured by a second sensor, an intensity associated with the first image region, a measure of temporal behavior associated with the first image region based on third image data captured at a second time, different from the first time, output from a machine learning model trained to detect degradation based on the image data; and controlling operation of the vehicle based at least in part on the detection of the degradation.

[0092] G. The method of paragraph F, wherein detecting degradation of the first image data includes determining a first probability of degradation of the first image data, where the first probability is based on a level of visual consistency; determining a second probability of degradation of the first image data, where the second probability is based on at least one of an intensity associated with the first image region, a measurement of temporal motion associated with the first image region, or output from a machine learning model; calculating a third probability of degradation of the first image data based on at least the first probability and the second probability; and comparing the third probability of degradation to a probability threshold.

[0093] H. The method of paragraph F or G, wherein detecting degradation of the first image data is based at least in part on a determined level of visual consistency between the first image area and the second image area, the method further comprising determining that the second sensor has an overlapping detection field with the first sensor, reading out second image data captured by the second sensor, identifying a second image area in the second image data associated with the first image area, and determining a level of visual consistency between the first image area and the second image area.

[0094] I. The method of any one of paragraphs F-H, wherein detecting degradation of the first image data is based at least in part on an intensity associated with the first image region, the method further comprising: determining a dark channel value associated with the first image data based at least in part on intensity values ​​associated with various image channels of the first image data, generating an average dark channel image value based at least in part on intensity values ​​of various image frames captured over time, determining a threshold value based on the average dark channel image value, comparing the intensity associated with the first image region to the threshold value, and detecting degradation of the first image data based at least in part on determining that the intensity associated with the first image region is greater than the threshold value.

[0095] J. The method of any one of paragraphs F-I, wherein detecting degradation of the first image data is based at least in part on measuring temporal motion associated with the first image region, the method further comprising receiving third image data captured by the first sensor at a second time different from the first time, and determining temporal motion of the first image region by comparing the first image region in the first image data to the first image region in the third image data using the first image data and the third image data captured by the first sensor.

[0096] K. The method of any one of paragraphs F-J, wherein detecting degradation further includes inputting at least the first image data into a machine learning model and receiving output from the machine learning model, the output based at least in part on the first image data, the output indicative of degradation of the first image data captured by the first sensor.

[0097] L. The method of paragraphs F-K, wherein detecting degradation of the first image data includes identifying a source or type of degradation, wherein the identified source or type of degradation includes at least one of an opaque substance on a surface of the first sensor, water droplets on the surface of the first sensor, optical flare within a detection field of the first sensor, or a visual focus error of the first sensor.

[0098] M. The method of paragraphs F-L, wherein controlling operation of the vehicle based at least in part on detecting degradation of the first image data includes at least one of initiating cleaning of a surface of the first sensor, reducing reliance on data received from the first sensor, changing a direction of travel of the vehicle, or adjusting one or more operational driving parameters of the vehicle.

[0099] N. A non-transitory computer-readable medium storing processor executable instructions, which when executed by one or more processors, causes the one or more processors to perform operations including receiving first image data captured by a first sensor of the vehicle at a first time; identifying a first image region within the first image data; detecting degradation of the first image data captured by the first sensor, where the detecting is based on at least two of a determined level of visual consistency between the first image region and an associated second image region in second image data captured by a second sensor, an intensity associated with the first image region, a temporal movement measurement associated with the first image region based on third image data captured at a second time, different from the first time, and output from a machine learning model trained to detect degradation based on the image data; and controlling operation of the vehicle based at least in part on the detection of the degradation.

[0100] O. The non-transitory computer readable medium of paragraph N, detecting degradation of the first image data includes determining a first probability of degradation of the first image data, where the first probability is based on a level of visual consistency; determining a second probability of degradation of the first image data, where the second probability is based on at least one of an intensity associated with the first image region, a measurement of temporal motion associated with the first image region, or output from a machine learning model; calculating a third probability of degradation of the first image data based on at least the first probability and the second probability; and comparing the third probability of degradation to a probability threshold.

[0101] P. A non-transitory computer readable medium as described in paragraphs N or O, wherein detecting degradation of the first image data is based at least in part on a determined level of visual consistency between the first image region and the second image region, and the operations further include determining that the second sensor has an overlapping detection field with the first sensor, reading out second image data captured by the second sensor, and determining a level of visual consistency between the first image region and the second image region.

[0102] Q. The non-transitory computer readable medium of any one of paragraphs N-P, wherein detecting degradation of the first image data is based at least in part on an intensity associated with the first image region, and the operations further include determining a dark channel value associated with the first image data based at least in part on intensity values ​​associated with various image channels of the first image data, generating an average dark channel image value based at least in part on intensity values ​​of the various image frames captured over time, determining a threshold value based on the average dark channel image value, comparing the intensity associated with the first image region to the threshold value, and detecting degradation of the first image data based at least in part on determining that the intensity associated with the first image region is greater than the threshold value.

[0103] R. A non-transitory computer readable medium as described in any one of paragraphs N-Q, wherein detecting degradation of the first image data is based at least in part on measuring temporal motion associated with the first image region, the operation further including receiving third image data captured by the first sensor at a second time different from the first time, and using the first image data and the third image data captured by the first sensor to determine the temporal motion of the first image region by comparing the first image region in the first image data to the first image region in the third image data.

[0104] S. The non-transitory computer readable medium of any one of paragraphs N-R, wherein detecting degradation of the first image data captured by the first sensor further includes inputting at least the first image data into a machine learning model and receiving output from the machine learning model, the output being based at least in part on the first image data, the output being indicative of degradation of the first image data captured by the first sensor.

[0105] T. The non-transitory computer readable medium of any one of paragraphs N-S, wherein controlling operation of the vehicle based at least in part on detecting degradation of the first image data includes at least one of initiating cleaning of a surface of the first sensor, reducing reliance on data received from the first sensor, changing a direction of travel of the vehicle, or adjusting one or more operational driving parameters of the vehicle.

[0106] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0107] The components described herein represent instructions that may be stored on any type of computer-readable medium and may be implemented in software and / or hardware. All of the above methods and processes may be embodied through software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof, and may be fully automated. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

[0108] Unless expressly stated to the contrary, conditional terms such as "can," "may," "can," or "may" are understood within the context of, among other things, an example including or presenting a certain feature, element, and / or step that another example does not include. Thus, such conditional terms are not intended to imply that a certain feature, element, and / or step is required in any way in one or more examples in general, or that one or more examples necessarily include logic for determining whether a certain feature, element, and / or step is included in or should be performed in any particular example, with or without user input or prompting.

[0109] Conjunctions such as the phrase "at least one of X, Y, or Z," unless specifically stated otherwise, should be understood to mean that the item, term, etc. may be either X, Y, or Z, or any combination thereof, including collections of each of the elements. Unless expressly described as singular, "a" means singular as well as plural.

[0110] Any routine illustrations, elements, or block diagrams in the flow diagrams described herein and / or shown in the accompanying figures should be understood as potentially representing modules, segments, or portions of code that comprise one or more computer-executable instructions for implementing a particular logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, in which elements or functions may be omitted or performed out of order from that shown or described, including substantially simultaneously or in reverse order, depending on the functionality involved, as will be understood by those skilled in the art.

[0111] It should be emphasized that many variations and modifications can be made to the above examples, and those elements should be understood to be within the scope of other acceptable examples, and all such variations and modifications are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims

1. 1. A system comprising: one or more processors; When executed by one or more of the processors, the system comprises: Receiving first image data captured by a first sensor of the vehicle at a first time; Identifying a first image region within the first image data; determining a level of visual consistency between the first image region and a corresponding second image region in second image data captured by a second sensor; determining a first probability of degradation of the first image data based on the level of visual consistency; A second probability of degradation of the first image data, an intensity associated with the first image region; a measurement of a temporal motion associated with the first image region based on third image data captured at a second time different from the first time; or output from a machine learning model trained to detect degradation based on image data; and determining based on at least one of detecting the degradation of the first image data captured by the first sensor based at least in part on the first probability and the second probability; and controlling operation of the vehicle based at least in part on the detection of the impairment. A system that provides.

2. Detecting the degradation of the first image data is based at least in part on the determined level of visual consistency between the first image region and the second image region, and the operation includes: determining that the second sensor has an overlapping detection field with the first sensor; reading out the second image data captured by the second sensor; identifying a second image region in the second image data associated with the first image region; determining the level of visual consistency between the first image region and the second image region. The system of claim 1 .

3. Detecting the degradation of the first image data is based at least in part on the intensity associated with the first image region, and the operation includes: determining a dark channel value associated with the first image data based at least in part on intensity values ​​associated with various image channels of the first image data; generating an average dark channel image value based at least in part on intensity values ​​of various image frames captured over time; determining a threshold value based on the average dark channel image value; comparing the intensity associated with the first image region to a threshold; and detecting the degradation of the first image data based at least in part on determining that the intensity associated with the first image region is greater than the threshold.

3. A system according to claim 1 or 2.

4. Detecting the degradation of the first image data is based at least in part on the measurement of a temporal motion associated with the first image region, the motion comprising: receiving the third image data captured by the first sensor at the second time, the second time being different from the first time; and determining a temporal motion of the first image area using the first image data and the third image data captured by the first sensor by comparing the first image area in the first image data to the first image area in the third image data. A system according to any one of claims 1 to 3.

5. Detecting the degradation of the first image data includes: inputting at least the first image data into the machine learning model; receiving the output from the machine learning model, the output based at least in part on the first image data, the output indicative of the degradation of the first image data captured by the first sensor. A system according to any one of claims 1 to 4.

6. Detecting the degradation of the first image data includes identifying a source or type of the degradation, the identified source or type of the degradation comprising: an opaque material on a surface of the first sensor; a drop of water on the surface of the first sensor; optical flare within the detection field of the first sensor; or at least one of the visual focus error of the first sensor; A system according to any one of claims 1 to 5.

7. Controlling the operation of the vehicle based at least in part on the detection of the degradation of the first image data includes: commencing cleaning of a surface of the first sensor; reducing reliance on data received from the first sensor; Changing the direction of travel of the vehicle; or adjusting one or more operational drive parameters of the vehicle. A system according to any one of claims 1 to 6.

8. receiving first image data captured by a first sensor of the vehicle at a first time; identifying a first image region within the first image data; determining a level of visual consistency between the first image region and a corresponding second image region in second image data captured by a second sensor; determining a first probability of degradation of the first image data based on the level of visual consistency; A second probability of degradation of the first image data, an intensity associated with the first image region; a measurement of a temporal motion associated with the first image region based on third image data captured at a second time different from the first time; or output from a machine learning model trained to detect degradation based on image data; determining based on at least one of: detecting the degradation of the first image data captured by the first sensor based at least in part on the first probability and the second probability; controlling operation of the vehicle based at least in part on the detection of the impairment. How to prepare.

9. The step of detecting the degradation of the first image data is based at least in part on the determined level of visual consistency between the first image region and the second image region, and the operation comprises: determining that the second sensor has an overlapping detection field with the first sensor; reading out the second image data captured by the second sensor; identifying a second image region in the second image data that is associated with the first image region; and determining the level of visual consistency between the first image region and the second image region. The method according to claim 8.

10. The step of detecting the degradation of the first image data is based at least in part on the intensity associated with the first image region, the method comprising: determining a dark channel value associated with the first image data based at least in part on intensity values ​​associated with various image channels of the first image data; generating an average dark channel image value based at least in part on intensity values ​​of various image frames captured over time; determining a threshold value based on the average dark channel image value; comparing the intensity associated with the first image region to a threshold; and detecting the degradation of the first image data based at least in part on determining that the intensity associated with the first image region is greater than the threshold.

10. The method according to claim 8 or 9.

11. The step of detecting the degradation of the first image data is based at least in part on the measurement of temporal motion associated with the first image region, the method comprising: receiving the third image data captured by the first sensor at the second time, the second time being different from the first time; and using the first image data and the third image data captured by the first sensor to determine a temporal movement of the first image area by comparing the first image area in the first image data with the first image area in the third image data. A method according to any one of claims 8 to 10.

12. Detecting the degradation of the first image data includes: inputting at least the first image data into the machine learning model; receiving the output from the machine learning model, the output based at least in part on the first image data, the output indicative of the degradation of the first image data captured by the first sensor.

12. A method according to any one of claims 8 to 11.

13. 13. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 8 to 12.

Images