Optical interference detection during vehicle navigation

Abstract

In some examples, the processor may receive images from a camera mounted on the vehicle. The processor may generate a disparity image based on features in at least one of the images. In addition, the processor may determine at least one region in a first image of the received images having a luminance above a luminance threshold. Further, the processor may determine at least one region in the disparity image having a disparity information level below a disparity information threshold. The processor may determine an optical interference region based on an overlap between the at least one region in the first image and the at least one region in the disparity image, and may perform at least one action based on the optical interference region.

Description

[Technical field]

[0001] Some implementations herein are directed to techniques and configurations for determining the presence of optical interference (e.g., bright spots) in a camera's field of view (FOV), for example due to direct bright light impingement or reflected bright light impingement. [Background technology]

[0002] Advanced driver assistance systems (ADAS), as well as semi-autonomous, fully autonomous, or other automated driving (AD) systems, are systems that automate or enhance vehicle control for improved safety, automated navigation, and the like. These systems typically use many types of sensors to recognize the roadway and to recognize and avoid other vehicles, obstacles, pedestrians, and the like. In particular, cameras are reliable sensors for capturing visual representations of the environment around the vehicle, for example, for roadway recognition, sign recognition, obstacle avoidance, and the like. Direct or reflected bright light impingement on the camera, for example from sunlight, oncoming headlights, emergency vehicle lights, or other bright light sources, can cause optical interference, such as bright spots, that can partially or completely dazzle the camera by obscuring features in the captured image and making image recognition unreliable. Such conditions can degrade the vehicle's ability to detect obstacles, other vehicles, pedestrians, lane markers, street signs, and the like. Summary of the Invention

[0003] In some implementations, the processor may receive images from a camera mounted on the vehicle. The processor may generate a disparity image based on features in at least one of the images. In addition, the processor may determine at least one region in a first image of the received images having a luminance above a luminance threshold. Further, the processor may determine at least one region in the disparity image having a disparity information level below a disparity information threshold. The processor may determine an optical interference region based on determining an overlap between the at least one region in the first image and the at least one region in the disparity image, and may perform at least one action based on the optical interference region. [Brief description of the drawings]

[0004] The detailed description will now be described with reference to the accompanying drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items or features. [Figure 1] FIG. 1 illustrates an example system configured to detect optical interference, according to some implementations. [Diagram 2] FIG. 2 shows an example architecture of a perception and control system that may be included in a vehicle, according to some implementations. [Diagram 3] FIG. 3 shows an example architecture of a perception and vehicle control system 300 that may be included in a vehicle, according to some implementations. [Figure 4] FIG. 4 illustrates an example of image capture performed by a camera system, according to some implementations. [Diagram 5] FIG. 5 illustrates an example of comparing a standard image and a corresponding disparity image to detect optical interference, according to some implementations. [Figure 6] FIG. 6 is a flow diagram illustrating an example process for detecting optical interference, according to some implementations. [Figure 7]FIG. 7 is a flow diagram illustrating an example process for determining whether to run a light interference detection program, according to some implementations. [Figure 8] FIG. 8 is a flow diagram illustrating an example process for responding to optical interference for autonomy levels 1-3, according to some implementations. [Figure 9] FIG. 9 is a flow diagram illustrating an example process for responding to light interference for autonomy level 4, according to some implementations. [Figure 10] FIG. 10 is a flow diagram illustrating an example process for responding to light interference for autonomy level 5, according to some implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Some implementations herein are directed to techniques and configurations for identifying the presence of optical interference (e.g., bright spots) in a camera's field of view (FOV) due to direct bright light impingement or reflected bright light impingement. For example, the detection techniques may be used in a vehicle-mounted stereo or mono camera by inspecting both a standard image captured by the camera and a corresponding disparity image generated for the same scene and field of view (FOV). As an example, one or more processors mounted on the vehicle may execute a recognition program including an optical interference detection program that configures the processor to recognize optical interference that may affect the accuracy of the recognition results of the image received from the camera while the optical interference is affecting the capabilities of the camera's image sensor. As an example, implementations herein may compare a first image captured by the camera image sensor with a disparity image generated based at least in part on the parallax determined from the first image to identify the detected bright spots or other optical interference by comparing the images.

[0006] In some cases, the recognition program may follow various rules that control the timing of the execution of the light interference detection program to minimize drain on the processing power of the onboard processor. For example, the recognition program may receive signals from an onboard digital compass, brightness sensor, and / or clock to determine whether the vehicle is above a danger threshold for being affected by light interference. For example, if the vehicle is operating at night, such as when the camera is more likely to be facing a low angle of the sun (the most common cause of light interference), such as within a threshold time after sunrise or before sunset, or when other vehicle headlights or flashing lights of emergency vehicles are more likely to affect the camera image, the recognition program may execute the light interference detection program along with other image processing and recognition functions already performed by the recognition program. In addition, the recognition program may receive local weather data, local sunrise and sunset times, etc., and may consider this additional information when determining whether to execute the light interference detection program. As an example, the onboard TCU (telematics control unit) may be used to communicate over a cellular network to access various weather data and other local data.

[0007] In some examples, if optical interference is detected that may corrupt an area of ​​interest in an image captured by the camera, such as an area ahead of the vehicle in the direction of travel, the system may be configured to respond with one or more actions, depending, for example, on the autonomy level currently in effect for the vehicle. For example, in a lower level system (e.g., where autonomy levels 1-3 are in effect), the optical interference detection program may cause a notification, such as an audible and / or visual warning, to be provided to the vehicle operator to inform the operator that operation of some ADAS systems, such as adaptive cruise control or automatic emergency braking, may be temporarily suspended until light conditions improve. Additionally, the notification may include a suggestion to clean the windshield (e.g., a dirty windshield may cause light reflections and associated bright spots on the captured image) or may perform one or more other actions in response to the detection of optical interference.

[0008] If the vehicle system is operating at a higher level of autonomy (e.g., autonomy level 4 or 5) and light interference is identified in a safety-critical area, a visual and / or audio notification may inform the driver to assume control of the vehicle within a threshold time, such as within 15 seconds, 30 seconds, etc., due to poor light conditions. The notification may further recommend that the vehicle driver clean the windshield or take one or more additional actions. If the driver does not assume control of the vehicle within the threshold time, the vehicle may be configured to slow down, pull over, or perform other safety actions. Additionally, if the vehicle is not capable of driver control, such as in the case of a level 5 fully autonomous vehicle, the vehicle may be configured to slow down, pull over, or perform other safety actions automatically in response to the detection of light interference. Additionally, in any of the above-mentioned scenarios, the vehicle may be configured to issue a notification of poor light conditions to nearby vehicles, for example, via a DSRC (Dedicated Short Range Communications) service or similar communication system.

[0009] As defined by the Society of Automotive Engineers (SAE), there are six driving automation levels, from Level 0 to Level 5. Notably, at "Level 0" (no driving automation), the driver performs all operational tasks such as steering, braking, accelerating, etc. At "Level 1" (driver assistance), the vehicle may assist with some functions (e.g., cruise control), but the driver is still responsible for all acceleration, braking, and monitoring the surrounding environment. At "Level 2" (partial driving automation), the vehicle may assist with steering or acceleration functions, relieving the driver from some of those tasks. Adaptive cruise control is one example of autonomy level 2.

[0010] The concept of autonomous driving primarily starts with "Level 3" (conditional driving automation), which means the vehicle itself may monitor its surroundings and may exercise some control over the vehicle (e.g., automatic emergency braking). "Level 4" (high driving automation) allows the vehicle to drive independently most of the time, but still requires that it will not operate unless all conditions are met and that a driver is available to take over. "Level 5" (full driving automation) allows the vehicle to drive anywhere in all conditions without a driver.

[0011] For purposes of illustration, some example implementations are described in the context of detecting and recognizing optical interference, such as bright spots in an image, that may reduce the effectiveness of a vehicle camera during navigation of the vehicle. However, implementations herein are not limited to the specific examples provided and may be extended to other types of cameras, other techniques for determining disparity images, other types of vehicles, other types of roads and obstacles, other weather conditions, etc., as would be apparent to one of ordinary skill in the art in light of the disclosure herein.

[0012] 1 illustrates an example system 100 configured to detect optical interference, according to some implementations. In this example, assume that a vehicle 102 is traveling on a road or other travel path 104 in a direction toward a sun 106, which in this example is relatively low in the sky. The vehicle 102 may include at least one camera system 108, which may be mounted on a roof 110 of the vehicle 102. In the illustrated example, the camera system 108 may include a mono camera or a stereo camera, such as a visible light camera and / or a near infrared (IR) camera. Additionally, although the camera system 108 is shown mounted on the vehicle's roof 110 in this example, in other examples, the camera system 108, or one or more other cameras, may be located in any of a variety of different locations on the vehicle 102.

[0013] In some cases, the camera system 108 may continuously capture images corresponding to the camera's field of view (FOV) while the vehicle 102 is in motion, for example, at 10 frames per second, 15 frames per second, 30 frames per second, 60 frames per second, or any other desired frequency that provides images at a rate great enough to permit recognition of the travel path 104 and recognition of any obstacles in the travel path in time to take evasive action or otherwise respond to the recognition information. As an example, the image capture frequency (sampling frequency) of the camera system 108 may increase as the vehicle speed increases.

[0014] The vehicle 102 may include one or more vehicle computing devices 116, as described further below. In some examples, the vehicle computing device 116 may execute a recognition program 118 and a vehicle control program 120. For example, the recognition program 118 may execute to receive images captured by the cameras of the camera system 108 and process the images to generate recognition about the current travel path 104 of the vehicle 102. The recognition program 118 may provide recognition information about detected and recognized features and obstacles to the vehicle control program 120, which may initiate one or more actions based on the recognition information, such as issuing a notification to alert a vehicle occupant, braking the vehicle 102, steering one or more wheels of the vehicle 102, etc. The recognition program 118 may include or execute a light interference detection program 122 to detect when a concentrated light or other bright light is affecting the ability of the camera to capture an image usable for image recognition.

[0015] In some cases, the camera system 108 itself may include at least one vehicle computing device 116 that executes the recognition program 118. In other cases, the vehicle computing device 116 may be separate from the camera system 108 and located elsewhere on the vehicle 102 to execute the recognition program 118. In still other cases, a portion of the recognition program and / or the light interference detection program 122 may execute within the camera and another portion may execute on another computing device at another location on the vehicle. In any of these cases, the vehicle computing device 116 may receive images from the camera system 108 and process the images to detect roads, road features, signs, obstacles, other vehicles, etc. The light interference detection program 122 may also execute to recognize light interference that may affect at least a portion of the camera image sensor.

[0016] In some examples, the recognition program 118 may generate a parallax-based disparity image from the received images, for example using a stereo camera image pair or continuously captured mono camera images. When using a mono camera, as an example, the disparity map may be calculated using a trained machine learning model (not shown in FIG. 1 ) that may determine changes in distance of features, for example, by considering the current vehicle speed and changes in continuously captured images. For example, initially, a set of monocular images and their corresponding ground truth parallax maps may be captured at various different speeds and used to train the machine learning model. The trained machine learning model may then be run on the vehicle to predict an approximation of the parallax map in response to newly captured images.

[0017] Alternatively, in the case of a stereo camera or multiple mono cameras usable as a stereo pair, the images may be captured by two cameras as a stereo pair. The difference between the captured images may be used to calculate the parallax using a block matching technique such as semi-global block matching or any other suitable technique. The parallax information may be used to generate a disparity image. In some examples herein, a stereo camera system is used as an example system to illustrate some example implementations, but one skilled in the art will also understand that similar configurations and techniques may be applied using a system with a single mono camera or multiple mono cameras.

[0018] The disparity map may assist in the detection and recognition of features within the camera FOV, such as roads, obstacles, lane markers, and may provide information for estimating distances to recognized features. The quality of the generated disparity map may correlate with the quality of the recognition and distance estimation performance. A high quality or "dense" disparity map may provide more accurate disparity information. On the other hand, a low quality or "sparse" disparity map may provide information that is sufficiently accurate for most applications herein. For example, the creation of a dense disparity map may be relatively computationally expensive compared to the creation of a sparse disparity map, requiring more expensive processing equipment and memory and / or longer computation time. Thus, in some examples herein, a sparse disparity map may be used to reduce computing requirements, even though a dense disparity map may provide slightly more accurate results, and may be used instead if sufficient processing power is available.

[0019] In the illustrated example, it is assumed that the vehicle 102 is traveling in the general direction of the sun 106 and that the sun 106 is low enough in the sky (e.g., below a threshold angle above the horizon) that sunlight impinges on the image sensor of the camera system 108, e.g., by direct and / or reflected light. For example, direct or reflected sunlight can cause optical interference, such as bright spots on images captured by the camera system 108, that can make it difficult for the recognition program 118 to recognize surfaces such as the road 104, lane markers 124, road signs 126, etc. The recognition program 118 may cause the optical interference detection program 122 to detect that optical interference is affecting the captured images based on the time of day and the vehicle travel direction.

[0020] As described further below, the optical interference detection program 122 may compare the images captured by the camera system 108 with a disparity image that may be generated at least in part from the captured image or one or more related images, such as from a stereo image pair that may also be captured by the camera system 108. For example, if the camera system 108 is capturing images at 60 frames per second, it may not be necessary to use the same images used to generate the disparity image for comparison with the disparity map. Rather, a pair of images captured at a time close to the image used for comparison may be used to generate the disparity map and still provide satisfactory results in detecting bright spots or other optical interference. This flexibility may provide additional time to calculate the disparity image before making the comparison.

[0021] Of course, in other examples, one or both of the same images used to generate the disparity map may be compared to the disparity map to detect the presence of any optical interference. Upon detecting optical interference in an area of ​​interest (e.g., an area including a travel path), such as through multiple comparisons over a threshold period of time, the optical interference detection program 122 may transmit at least one signal that may cause at least one action to be taken by the vehicle and / or the vehicle driver or other occupant. Additional features, variations, and advantages of implementations herein are described below.

[0022] 2 illustrates an example architecture of a perception and control system 200 that may be included in a vehicle 102, according to some implementations. Each vehicle computing device 116 may include one or more processors 202, one or more computer-readable media 204, and one or more communication interfaces 206. In some examples, the vehicle computing device 116 may include one or more ECUs (electronic control units) or any other various types of computing devices. For example, the computing device 116 may include one or more ADAS / AD ECUs for controlling critical vehicle systems and perform ADAS and / or AD tasks such as navigation, braking, steering, acceleration, deceleration, etc. The computing device 116 may also include other ECUs for controlling other vehicle systems.

[0023] An ECU is a general term for any embedded system that controls one or more of the systems, subsystems, or components in a vehicle. Software such as the recognition program 118 and the vehicle control program 120 may be executed by one or more ECUs and may be stored in a portion of the computer-readable medium 204 (e.g., program ROM) associated with each ECU to enable the ECU to operate as an embedded system. The ECUs may communicate with each other via a vehicle bus 208, typically according to a vehicle bus protocol. As an example, the Controller Area Network Bus (CAN bus) protocol is a vehicle bus protocol that allows ECUs and other vehicle devices and systems to communicate with each other without a host computer. CAN buses may include at least two different types. For example, high-speed CAN may be used in applications where a bus is laid from one end of an environment to the other, while fault-tolerant CAN may be used when a group of nodes are connected together.

[0024] Each ECU or other vehicle computing device 116 may include one or more processors 202, which may include one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcomputer, a microcontroller, a digital signal processor, a state machine, a logic circuit, and / or any device that manipulates signals based on operational instructions. By way of example, the processor 202 may include one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to execute the algorithms and other processes described herein. The processor 202 may be configured to fetch and execute computer-readable instructions stored on a computer-readable medium 204, which may program the processor 202 to perform the functions described herein.

[0025] The computer readable medium 204 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer readable instructions, data structures, programs, program modules, and other code or data. For example, the computer readable medium 204 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, magnetic disks, cloud storage, or any other medium that can be used to store the desired information and accessible by a computing device. Depending on the configuration of the vehicle computing device 116, the computer readable medium 204 may be a tangible non-transitory medium to the extent that, when referred to, non-transitory computer readable medium excludes media such as energy, carrier signals, electromagnetic waves, and / or signals themselves. In some cases, the computer readable medium 204 may be co-located with the vehicle computing device 116, while in other examples, the computer readable medium 204 may be partially remote from the vehicle computing device 116, accessible, for example, via a wireless network or the like.

[0026] The computer readable medium 204 may be used to store any number of functional components executable by the processor 202. In many implementations, these functional components include instructions or programs executable by the processor 202 that, when executed, specifically program the processor 202 to perform operations attributed to the vehicle computing device 116 herein. The functional components stored on the computer readable medium 204 may include a recognition program 118 and a vehicle control program 120, each of which may include one or more computer programs, applications, executable code, or portions thereof. For example, the recognition program 118 may include the optical interference detection program 122 described above. Additionally, although these programs are illustrated together in this example, during use, some or all of these programs may be executed on separate vehicle computing devices 116.

[0027] Additionally, the computer-readable medium 204 may store data, data structures, and other information used to perform the functions and services described herein. For example, the computer-readable medium 204 may store image data 210, recognition information 212, vehicle data 214, other sensor data 216, and in some examples, such as when a mono camera is used, one or more machine learning models (MLMs) 218. Additionally, although these data and data structures are shown together in this example, during use, some or all of these data and / or data structures may be stored by or with a separate computing device 116. The computing device 116 may also include or maintain other functional components and data, which may include programs, drivers, etc., and data used or generated by the functional components. Additionally, the computing device 116 may include many other logical, programmatic, and physical components, of which the above-mentioned are merely examples relevant to the discussion herein.

[0028] The one or more communications interfaces 206 may include one or more software and hardware components for enabling communication with various other devices, for example, via the vehicle bus 208 and / or via one or more networks (not shown in FIG. 2 ). For example, the communications interface 206 may enable communication via one or more of a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi) and a wired network (e.g., CAN, Fibre Channel, Fiber Optic, Ethernet), a direct connection, and short-range communications such as BLUETOOTH®, as further enumerated elsewhere herein.

[0029] The computing device 116 may be capable of communicating with the camera system 108 via a vehicle bus 208, a direct connection, or any other type of connection to receive image data 210 from the camera system 108. For example, as described in more detail below, the recognition program 118 may receive the image data 210 from the camera system 108 and perform recognition of features within the images. In the illustrated example, the camera system 108 includes a stereo camera 119 including a left camera 220 and a right camera 221. However, in other examples, the camera system 108 may include a mono camera instead of the stereo camera 219, and the machine learning model 218 may be used with multiple images captured using the mono camera to determine a disparity image.

[0030] 2, the received image data 210 may include captured images 222, which may include a left and right image pair if a stereo camera 219 is used. For example, some or all of the image data 210 may be received as raw image data without substantial processing. Alternatively, in other examples, the camera system 108 may perform at least some of the recognition on the images rather than transmitting the raw image data to a separate vehicle computing device 116, for example as described further below in connection with FIG.

[0031] Additionally, the computing device 116 may receive vehicle data 214 from other systems and / or other sensors within the vehicle. For example, the vehicle may include a number of other sensors 225 in addition to the camera system 108 that may provide sensor information used by the vehicle control program 120. Some non-exhaustive examples of the other sensors 225 may include radar, LiDAR, ultrasonic, a global positioning system (GPS) receiver, thermal cameras, other cameras pointing in other directions, etc. Additionally, the vehicle data 214 used by the vehicle control program 120 may include information received from or related to various vehicle systems, such as, for example, a suspension controller 224 associated with a suspension system, a steering controller 226 associated with a steering system, a vehicle speed controller 228 associated with a braking and acceleration system, etc.

[0032] As an example, for example, in the case of autonomy levels 3-5, the recognition program 118 may continuously receive captured images 222 from the camera system 108 as part of the image data 210, for example, as the camera system 108 captures images of the vehicle's travel path or other surroundings while the vehicle is traveling. Additionally, the recognition program 118 may process the received image data 210 to recognize road features, obstacles, and the like. The recognition program 118 may provide recognition information 212 regarding any detected obstacles, road features, and the like to the vehicle control program 120, which in the case of autonomy levels 3-5 may take one or more actions in response to the recognition information. In some examples, the vehicle control program 120 and / or the recognition program 118 may fuse or combine the recognition information 212 determined from the image data 210 with other sensor data 216 to provide additional available information to the vehicle control program 120 for controlling the vehicle.

[0033] For example, the vehicle control program 120 may use rule-based and / or artificial intelligence-based control algorithms to determine parameters for vehicle control. For example, the vehicle control program 120 may apply one or more additional machine learning models (not shown in FIG. 2 ) to determine appropriate actions, such as braking, steering, deceleration, acceleration, etc. Additionally, the vehicle control program 120 may send one or more control signals 238 to one or more vehicle systems in response to the recognition information 212. For example, the vehicle control program 120 may send the control signals 238 to the suspension controller 224, the steering controller 226, and / or the vehicle speed controller 228. For example, the control signals 238 may include specified spring rate and / or damping control information sent to the suspension controller 224, a specified steering angle sent to the steering controller 226 to steer one or more wheels, and / or specified braking or acceleration control information sent to the vehicle speed controller 228.

[0034] Additionally or alternatively, for example when the vehicle is under the control of a human driver, the vehicle control program 120 may send a control signal 238 to a display 240 to present a warning, and / or may send a signal to one or more warning devices 242, such as an audible or visual warning device. Examples of warning devices 242 include speakers that may generate an audible warning, haptic devices that may generate a vibration or other type of tactile warning (e.g., in the seat or steering wheel), and / or visual signaling devices that may generate a visual warning.

[0035] For example, as described above, the optical interference detection program 122 may run continuously or periodically (e.g., based on detected light conditions) and may generate disparity images from the captured images 222 or may use already-generated disparity images generated by the recognition program 118 from the captured images 222 to detect optical interference that may affect the accuracy of the recognition performed by the recognition program 118. For example, when performing recognition to determine the distance of a recognized obstacle and other features in the captured images 222, the recognition program 118 may already be configured to generate disparity images as part of the image data 210 and may use the disparity images to determine the distance to various features recognized in the captured images. In such a system, the optical interference detection program 122 need not generate disparity images itself, but may use already-generated disparity images to perform the optical interference detection herein.

[0036] When optical interference is detected, the optical interference detection program 122 or the recognition program 118 may cause the vehicle control program 120 to perform one or more actions, such as, for example, sending one or more control signals 238 to the display 240 and / or the warning device 242 to notify a vehicle occupant and / or sending control signals 238 to slow down the vehicle, stop the vehicle, cause the vehicle to communicate with other vehicles, etc., as described further below. In other examples, the optical interference detection program 122 and / or the recognition program 118 may send notifications directly to the display 240 and / or the warning device 242 and / or send control signals 238 to the vehicle systems 224-228.

[0037] Additionally, in some examples, such as for autonomy level 4 and / or autonomy level 5, the vehicle control program 120 may include a sensor fusion function that may receive the perception information 212 and may also receive other sensor data 216. The sensor fusion may take into account all sensor detection results (e.g., lidar, radar, other cameras), may compare the results of the various sensors with the results of other sensors, and may use the combined results to effect vehicle control. If the sensors disagree on the presence or location of one or more potential obstacles, the sensor fusion function may determine a confidence value for each sensor, may determine how many sensors agree, may determine whether current environmental conditions may be adversely affecting sensor performance, etc., may determine a conclusion regarding the likelihood that a particular object is actually present.

[0038] For example, the fusion function may compare and match the perception information 212 with the other sensor data 216 to provide the vehicle control program 120 with a more complete view of the vehicle's surroundings. In some examples, the output of the light interference detection program 122 (e.g., the presence of light interference in an area of ​​safety concern) may be associated with a confidence value. For example, the confidence value may indicate to the sensor fusion function that the stereo camera is operating at a reduced capacity and that more weight should be given to the other sensor data 216 from the other sensors 225. Based on all this information, the vehicle control program 120 may determine whether to continue driving the vehicle when the perception information from the camera system 108 may be unreliable due to light interference.

[0039] Additionally, the light interference detection program 122 and / or the recognition program 118 may determine whether to execute the light interference detection program 122. For example, to minimize the amount of computational resources used by the light interference detection program 122, signals received from various on-board sensors may be used to determine whether to execute the light interference detection program 122. In particular, sunrise and sunset are times of high risk for bright light interference because the sun is at a low angle and may shine directly on the camera aperture. Thus, in some examples, the vehicle may include a light sensor 244, a clock 246, and a digital compass 248. Based on information received from these devices, the system may determine whether the light interference detection program 122 is likely to be beneficial, for example, by executing the recognition program 118. For example, information from the light sensor 244, the clock 246, and the compass 248 may be used by the recognition program 118 to determine the time of day, the expected angle of the sun in the sky, and the relative orientation of the sun to the vehicle. For example, if the recognition program determines that the vehicle is traveling and using the camera system 108 near sunrise or sunset (e.g., when the sun is likely to be within a threshold angle of the horizon) and the vehicle is facing toward the sun, the recognition program may invoke or activate the light interference detection program 122.

[0040] Additionally, in some cases, the recognition program 118 may use information received via the TCU (telematics control unit) 250 or other cellular or wireless connection to at least partially determine whether to execute the light interference detection program 122. For example, the recognition program 118 may receive weather information as well as local sunrise and sunset times, allowing the recognition program 118 to more accurately determine whether to execute the light interference detection program 122.

[0041] FIG. 3 illustrates an example architecture of a perception and vehicle control system 300 that may be included in a vehicle 102, according to some implementations. In this example, the camera system 108 may include processing capabilities for determining perception information 212 independent of other vehicle computing devices 116. Thus, the camera system 180 includes one or more processors 302, one or more computer-readable media 304, and one or more communication interfaces 306. The one or more processors 302 may be or include any of the processors 202 described above in connection with FIG. 2, or other suitable processors for performing the processing described herein. Additionally, the one or more computer-readable media 302 may be or include any of the computer-readable media 204 described above in connection with FIG. 2, or other suitable computer-readable media. Similarly, the communication interface 306 may be or include any of the communication interfaces 206 described above in connection with FIG. 2, or other suitable communication interfaces.

[0042] Additionally, camera system 108 includes a stereo camera 219, which may include one or more lenses, one or more focusing systems, and one or more image sensors, as known in the art (not shown in FIG. 3). In this example, camera system 108 may execute recognition program 118 and optical interference detection program 122 on one or more processors 302. Thus, stereo camera 219 may capture images within its field of view, and the captured images may be stored in computer readable medium 304 as part of image data 210.

[0043] In some examples, the recognition program 118 receives image data 210 from the camera system 108. For example, the recognition program 118 may continuously receive image data 210 from the camera system 108 to a memory buffer (not shown in FIG. 3 ) in the computer-readable medium 304 as captured images 122 are captured. The recognition program 118 may also perform recognition processing on the received images, as described above. The recognition program 118 may transmit recognition information 212 to the vehicle computing device 116 in real time. The vehicle control program 120 may process the recognition information 212 to control the vehicle, as described above in connection with FIG. 2 and further described below.

[0044] As mentioned above, vehicle control program 120 may include a sensor fusion function that may receive perception information 212 and may also receive other sensor data 216. For example, the fusion function may compare and match perception information 212 with other sensor data 216 to provide vehicle control program 120 with a more complete view of the vehicle's surroundings. Based on the confidence associated with perception information 212 and / or light interference information 310, as well as the confidence associated with the other sensor data 216, the vehicle control program may determine whether to continue operating the vehicle when perception information from camera system 108 may be unreliable due to light interference.

[0045] In addition, similar to the example described with respect to FIG. 2, the recognition program 118 may continuously or periodically launch the optical interference detection program 122 based on determined light conditions that may affect, for example, the stereo camera 219. The optical interference detection program 122 may generate a disparity image from the received image or may use a disparity image that has already been generated to perform the optical interference detection herein. For example, the optical interference detection program 122 may determine areas of the received image that exceed a brightness threshold and compare these areas with areas of the disparity image to determine whether the disparity information in these areas is below a threshold for detecting optical interference. Additionally, the optical interference detection program 122 may determine whether the detected optical interference is affecting the camera image sensor in areas of interest, such as areas in the direction of travel, in the road, or other areas that may be necessary for safe navigation.

[0046] When light interference is detected, the light interference detection program 122 or the recognition program 118 may send light interference information 310 to the vehicle control program to cause the vehicle control program 120 to perform one or more actions, such as, for example, to send one or more control signals 238 to the display 240 and / or warning device 242 to notify a vehicle occupant and / or to send control signals 238 to slow down the vehicle, stop the vehicle, or communicate with other vehicles, as described further below. Similar to FIG. 2 described above, the vehicle control program 120 may send one or more control signals 238 to one or more vehicle systems 224, 226, 228, 240, and / or 242 based on the received recognition information 212 and other sensor data 216. Additionally, similar to the example described above with respect to FIG. 2, the camera system 108 may receive inputs from the light sensor 244, the clock 246, the compass 248, and / or the TCU 250 to determine when to run the light interference detection program 122.

[0047] 4 illustrates an example 400 of image capture performed by the camera system 108 according to some implementations. In this example, the camera system 108 includes a stereo camera 219. For example, the stereo camera 219 may include a right lens 402 and a left lens 404. The right lens 402 may capture a right image 406 within a right field of view (FOV) 407, and the left lens 404 may capture a left image 408 within a left FOV 409. In the illustrated example, the stereo camera 219 includes a right image sensor 410 for capturing images through the right lens 402 and a left image sensor 412 for capturing images through the left lens 404. In other examples, the stereo camera 219 may include a single image sensor (not shown in FIG. 4) that alternately captures images through the right lens 402 and the left lens 404.

[0048] A system (e.g., systems 100, 200, and / or 300) can use the right image 406 and the left image 408, respectively, to determine a parallax image, called a disparity image. For example, the system can calculate the disparity image using the stereo right image 406 and the left image 408, e.g., based on a block matching method. As an example, a point P L If u1, v1, then the corresponding point P R = (u2,v2) is the sum of the two values ​​of v1 = v2 measured from a common baseline. L Thus, a measurement of the parallax can be determined using simple stereo camera theory, where parallax can be defined as:

number

[0049] Based on the determined parallax, in the implementation herein, a parallax image can be determined by determining depth information of a 3D point from the parallax, since the parallax is inversely proportional to the corresponding parallax. Thus, the depth can be calculated using the left and right images and the actual parallax, for example, as follows:

number

[0050] FIG. 5 illustrates an example 500 of comparing a standard captured image 502 with a corresponding disparity image 504 to detect optical interference, according to some implementations. The captured image 502 may be a standard vision image and may correspond to the disparity image 504 by being of the same or nearly the same scene. For example, the captured image 502 may be one of the images (left or right image) captured by a stereo camera and used to generate the disparity image 504. Alternatively, the captured image may be an image captured close in time (e.g., within the same second, half second, etc., depending on the speed and motion of the vehicle) as the captured image used to generate the disparity image 504. The optical interference detection program 122 may be executed to detect when optical interference is affecting the captured image, for example, in environments with harsh sun exposure or other bright light conditions.

[0051] As mentioned above, a stereo camera can obtain two types of data: standard visual data captured by the left and right cameras, and data regarding distances to objects and other points detected in the surrounding environment, which can be generated by comparing the left and right images and used to generate a disparity image 504. High light reflective areas and other bright light interference areas can appear on the stereo camera's standard visual captured image 502 as bright spots 505 and 506. This indicates that bright light is impinging on the camera's image sensor, interfering with the recognition program's ability to recognize objects.

[0052] Optical interference may also hinder the recognition program 118's ability to determine distance. For example, low-parallax information spots 507 and 508 may be present in the corresponding parallax image 504, which may correspond to bright spots 505 and 506, respectively. On the parallax image 504, bright light reflections or other bright areas of optical interference may appear as a lack of parallax data (e.g., black areas in this example). However, even good parallax images that are not affected by optical interference may naturally tend to have areas of low parallax data due to differences in materials and lighting conditions. Thus, the lack of parallax data alone is not enough to confirm optical interference. For example, the parallax image 504 has other black areas that do not correspond to bright spots, indicating insufficient parallax data. For example, the black areas of low parallax data may be due to surface characteristics and environmental conditions.

[0053] In view of the above, implementations herein identify bright spots and other light interference by comparing both the standard visual imaging image 502 and the parallax image 504. For example, light interference may correspond to bright spots as white pixels or locally bright areas on the standard visual imaging image 502 and as areas with low or no disparity data (e.g., below a threshold level compared to the rest of the parallax image 504) on the parallax image 504. By inspecting both of these images 502 and 504 together, areas affected by light interference may be identified and thus determined to be areas of low reliability. For example, squares 509 and 510 in the imaging image 502 and squares 511 and 512 in the parallax image 504 may indicate areas of low reliability for object and distance recognition.

[0054] A common technique in automotive applications of image processing is object tracking. Object recognition based on captured images may not be smooth and continuous. For example, the recognition program 118 may intermittently detect and then lose sight of an object as the vehicle moves. As an example, a detected object may be temporarily "lost" by the object recognition program for one or two camera frames, e.g., a fraction of a second, and then detected again. To improve the usefulness of the detection results, the recognition program 118 may track the object over multiple frames, based in part on the speed and direction of the vehicle. Thus, if a previously detected object is momentarily lost, its expected location may be tracked until the object is detected again.

[0055] By using this method, an object that is momentarily lost to object detection due to occlusion by optical interference can be tracked and redetected if the occlusion is only for a short period of time, whereas continuous occlusion of the object by optical interference may cause the object to be invisible to the camera and recognition program 118. If the optical reflection is in a safety-critical area, such as in front of the vehicle, any vehicles, pedestrians, or other obstacles moving or stationary in that area will not be imaged by the camera, potentially causing a collision.

[0056] By extending conventional object detection and recognition functions with the optical interference detection techniques herein, the recognition program 118 can identify areas of an image that may cause interruptions in the object detection function, thereby informing the tracking algorithm of the recognition program 118 that an object may be lost due to optical reflections, and indicating that the recognition program 118 should continue tracking the object until it emerges from the area affected by the optical interference.

[0057] If the light interference persists for a threshold time in an area of ​​interest (e.g., an area ahead of the vehicle, an area in the direction the vehicle is turning, etc.), the recognition program 118 may be configured to send a communication to the vehicle control program to take the light interference into account and take one or more actions based on the autonomy level of the vehicle's driving conditions. In lower level systems (e.g., autonomy levels 1-3), the driver is always in control, so ADAS features that rely on stereo cameras, such as automatic emergency braking (AEB) and / or adaptive cruise control (ACC), can provide visual and / or audible warnings to the vehicle driver before safely bringing the vehicle to a halt. Additionally, vehicle systems at all autonomy levels may attempt to warn surrounding vehicles of poor light conditions if equipped with such communication.

[0058] Additionally, if the vehicle is operating at autonomy level 4, the system may alert the vehicle operator to regain control based on detection of sustained optical interference. In this case, if optical interference is identified in the area of ​​safety concern for a threshold time, the optical interference detection program may send a communication to cause the vehicle to alert the operator with visual and audio warnings. The operator may be instructed to take control of the vehicle within a specified time. If the driver does not take control of the vehicle, the vehicle control program may be configured to slow the vehicle and / or pull over to the side of the road.

[0059] Also, if the vehicle is operating at level 5 autonomy, the vehicle occupant may not be able to take over operation of the vehicle. Thus, if optical interference is detected in an area of ​​safety concern and the vehicle system cannot compensate using other sensors on board the vehicle, the vehicle may alert the occupant with a visual and / or audio warning. Additionally, the vehicle control program may safely slow the vehicle and / or pull over to the side of the road. Again, this response may depend on whether the vehicle system is ready for human interaction.

[0060] 6-10 include flow diagrams illustrating exemplary algorithms or other processes according to some implementations. The processes are illustrated as a collection of blocks in a logical flow diagram that represent a sequence of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, programs the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the blocks are described should not be construed as limiting. Any number of the described blocks may be combined in any order and / or in parallel to perform a process or alternative processes, and not all blocks need to be executed. For purposes of illustration, the processes are described with reference to the environments, frameworks, and systems described in the examples herein, although the processes may be implemented in a wide variety of other environments, frameworks, and systems.

[0061] 6 is a flow diagram illustrating an example process 600 for detecting light interference, according to some implementations. In some examples, the process 600 may be performed by the above-mentioned systems 100, 200, and / or 300, for example, by execution of the recognition program 118, the light interference detection program 122, and / or the vehicle control program 120.

[0062] At 602, the system may execute an optical interference detection program. For example, the system may determine whether to execute the optical interference detection program 122 based on process 700, described below with respect to FIG.

[0063] At 604, the system may determine whether the camera being used is a stereo or mono camera. For example, a first procedure may be used to generate disparity images in the case of a stereo camera, and a second procedure may be used to generate disparity images in the case of a mono camera.

[0064] At 606, in the case of a stereo camera, the system may receive left and right stereo images captured by the stereo camera.

[0065] At 608, the system may generate a disparity image from the left and right stereo images. Examples of generating a disparity image are described above, for example, with respect to FIGS.

[0066] At 610, in the case of a mono camera, the system may receive successive mono images captured by the mono camera.

[0067] At 612, the system may generate a disparity image from multiple consecutive mono camera images. As one example, the system may use a machine learning model that takes into account the speed of the vehicle and how often frames are captured by the mono camera to determine distances to various features in the mono image. Also, while a machine learning model is provided as an example for performing this function, in other examples, the disparity image may be generated heuristically without the use of a machine learning model.

[0068] At 614, the system may process at least one of the received images using the operations of blocks 616-620. As described above, the images may be standard vision images captured by either the left or right side of the stereo camera. Alternatively, in the case of a mono image, the image processed may be one of the images captured in succession by the mono camera. For example, the image processed at 614 may be one of the same images used to generate the disparity image, or may be an image captured at a time close to the image used to generate the disparity image, such that the received image processed at 614 corresponds to the disparity image generated at 608 or 612 by being of approximately the same scene as the disparity image.

[0069] At 616, the system may convert the received image corresponding to the disparity image to a grayscale image to allow for easier distinction between light and dark areas.

[0070] At 618, the system may reduce noise in the grayscale image, for example using any of a number of known noise reduction algorithms.

[0071] At 620, the system may identify one or more regions of relatively high brightness in the grayscale noise reduced image. For example, the system may examine individual pixels of the image for comparison to a threshold level of brightness and perform clustering on pixels that exceed the brightness threshold to determine any bright spots in the image. Additionally or alternatively, in some examples, template matching may be used to detect bright spots in the image. For example, template matching may include searching the entire image for features of interest using one or more feature templates. Template matching involves creating a template of a target feature to search for in the image. For example, the template may include one or more examples of bright areas such as starbursts, lens flares, blooms, etc. The template may be "dragged" across the image such that all areas of the image are compared to the template. The template matching process may output a list of potential locations in the image that may at least partially match the template. For example, the list may be organized according to the closeness of the match.

[0072] As another example, one or more machine learning models, such as neural networks, classifiers, etc., may be trained on a training set of images, at least some of which may contain known optical interference. Once the machine learning model is trained to an acceptable accuracy, the machine learning model may be used to detect bright spots in the standard image. As described below in block 624, a similar machine learning model may be trained to recognize low disparity regions in the disparity image. Thus, the machine learning model may be used to detect only standard images or only disparity images. As another example, a separate machine learning model trained for standard images may operate in parallel with a separate machine learning model trained for disparity images. And as yet another example, a single machine learning model may be trained to receive both standard and disparity images (or at least matching regions of these images) to simultaneously use information from both the standard and disparity images to identify optical interference regions. Numerous other variations will be apparent to those skilled in the art having the benefit of the disclosure herein.

[0073] At 622, the system may process the disparity image using the operations of blocks 624 and 626.

[0074] At 624, the system may determine one or more regions in the disparity image that have low or no disparity results. For example, low disparity or negligible disparity regions may be determined based on identifying and clustering pixels that are black below a threshold level compared to the remainder of the disparity image. Additionally or alternatively, in some examples, low disparity or negligible disparity regions may be determined based in part on a comparison with any bright regions identified at 620. For example, since low disparity regions may be relatively common in some images, it may be more efficient to first determine bright regions in the standard image and then confirm that corresponding regions in the disparity image have low or no disparity information. Thus, in some cases, block 624 may be combined with block 628 below. Additionally, as another example, as described above, a machine learning model may be trained on a number of images known to have low disparity regions due to optical interference. The trained machine learning model may then be used to recognize low disparity or negligible disparity regions in the disparity image.

[0075] At 626, the system may ignore any identified regions with valid disparity results. For example, if the search of the standard image returns candidate regions for optical interference (e.g., meets the threshold requirement for likelihood of optical interference), the system may check the disparity results in those regions. If the disparity image at that location shows a normal disparity result, there is no optical interference and the location is discarded as a potential case of optical interference. If all locations have been discarded, there is currently no optical interference and the vehicle continues to operate as is (or returns to normal operation from optical reflection mode). In some examples, block 626 may be executed after block 628.

[0076] At 628, the system may compare the grayscale reduced noise image and the disparity image to identify overlapping regions that are both relatively high brightness and relatively low disparity or negligible disparity to identify the overlapping regions as regions of optical interference. For example, the system may use block matching or other various techniques to match bright regions in the received image with regions of low disparity or negligible disparity in the disparity image to determine the overlapping regions. Additionally, the overlapping optical interference regions may be required to exceed a threshold size, such as exceeding a threshold number of pixels or a threshold percentage of the image size.

[0077] At 630, the system may determine whether one or more regions of optical interference were identified at 628. If so, the process proceeds to 632. If not, the process proceeds to 602 to process the next image set.

[0078] At 632, the system may determine whether the identified light interference region overlaps with a safety region of concern, such as an area in front of the vehicle, an area in the direction the vehicle is turning, etc. If so, the process proceeds to 634. If not, the process proceeds to 602 to process the next set of images.

[0079] At 634, the system may determine whether the optical interference was detected for a threshold period of time. For example, to avoid a situation where a brief flash may be interpreted as optical interference, the system may require that optical interference be detected continuously for a threshold period of time, such as ½ second, 1 second, 1½ seconds, etc., rather than a brief, transient period. Additionally, additional information may assist in this determination. For example, if a flashing light from an emergency vehicle causes transient optical interference, this may be determined in part by a microphone simultaneously detecting the siren. If the system determines that optical interference was detected for the threshold period of time, the process proceeds to 634. If not, the process proceeds to 602 to process the next set of images.

[0080] At 636, the system may send a communication to cause at least one action to be taken. For example, a processor or processing thread executing the light interference detection program 122 may send a notification to a processor or processing thread executing the vehicle control program 120 to indicate that light interference may be affecting the reliability of images being processed by the image recognition program 118. In response, the vehicle control program 120 may perform one or more actions, such as those described below with respect to FIGS. 8-10. Alternatively, in other examples, the light interference detection program 122 may be configured to perform some or all of the actions described with respect to FIGS. 8-10. The light interference detection program 122 may repeatedly monitor for any light interference on the received camera images, such as continuously at set intervals or continuously until a stop command is received or a condition is met.

[0081] 7 is a flow diagram illustrating an example process 700 for determining whether to execute the light interference detection program 122, according to some implementations. In some examples, the process 700 may be performed by the above-mentioned systems 100, 200, and / or 300, for example, by execution of the recognition program 118, the light interference detection program 122, and / or the vehicle control program 120.

[0082] At 702, the system may receive information from an onboard clock. In some examples, the onboard clock may be associated with one of the processors 202, 302, while in other examples, the onboard clock may be independent of the processor.

[0083] At 704, the system may receive information from a light sensor. For example, the light sensor may be associated with the camera system 108 or may be separate from the camera system 108.

[0084] At 706, the system may determine whether the current time is daytime. For example, the light sensor and clock inputs may both indicate whether the current time is daytime. If it is daytime, the process proceeds to 708. If not, the process proceeds to 714, causing the light interference detection program 122 to run. In some examples, if the current time is nighttime, the light interference detection program 122 may run continuously, such as when the vehicle encounters oncoming headlights of another vehicle.

[0085] After the system determines that the current time is daytime at 708, the system may determine whether the angle of the sun is likely to be within a threshold sun height angle of the horizon. For example, if the sun is forecasted to be less than 30 degrees, less than 25 degrees, etc., from the horizon, the sun may cause optical interference to the camera system. If the angle of the sun is likely to be within the threshold sun height angle, the process proceeds to 710. If not, the process proceeds to 716.

[0086] At 710, the system may receive information from a compass to determine if the vehicle is currently traveling in the direction of the sun. In some cases, the compass may be associated with the vehicle's navigation system.

[0087] At 712, the system may determine whether the vehicle is facing the sun within a threshold heading angle that is likely to result in optical interference. For example, if the vehicle is heading in a direction that exceeds an angle, e.g., 40 degrees, 45 degrees, etc., to either side of the sun, optical interference may be unlikely. If the vehicle is likely facing within the threshold angle of the sun, the process proceeds to 714. If not, the process proceeds to 716.

[0088] At 714, the system may execute the optical interference detection program 122, as described with respect to FIG. 6 above.

[0089] At 716, the system may not execute the light interference detection program 122. For example, if the light interference detection program 122 is not currently running, execution is not initiated. Additionally, if the light interference detection program 122 is currently running, conditions may have changed and execution may no longer be necessary. For example, the sun may have risen above a threshold sun elevation angle, or the vehicle may no longer be pointing toward the sun within a threshold heading angle.

[0090] 8 is a flow diagram illustrating an example process 800 for responding to light interference for autonomy levels 1-3, according to some implementations. In some examples, process 800 may be performed by systems 100, 200, and / or 300 described above, for example, by execution of recognition program 118, light interference detection program 122, and / or vehicle control program 120.

[0091] At 802, the system may receive an indication of detected light interference in a safety region of concern, as described above in block 636 of Figure 6. For example, a processor or processing thread executing light interference detection program 122 may send a notification to a processor or processing thread executing vehicle control program 120 indicating that light interference may be affecting the reliability of images being processed by image recognition program 118.

[0092] At 804, the system may send a visual and / or audio notification to the vehicle operator. For example, the vehicle may display a notification to the operator and / or sound an alarm. Additionally, in some examples, the notification may recommend that the operator wash the windshield to reduce the amount of light that may be reflected toward the camera.

[0093] The system may wait for a threshold time at 806. For example, rather than immediately disabling ADAS features such as automatic emergency braking (AEB) and / or adaptive cruise control (ACC) features, the system may first provide a warning to the vehicle operator.

[0094] At 808, the system may disable ADAS systems, such as AEB and / or ACC functions.

[0095] At 810, the system may send a notification to nearby vehicles to provide a warning of the poor light condition.

[0096] At 812, the system may wait a threshold amount of time.

[0097] At 814, the system may determine whether optical interference is still detected. If so, the process returns to 812 to wait for an additional threshold time. If not, the process proceeds to 816.

[0098] At 816, based on the optical interference no longer being detected, the system may restore ADAS functions such as AEB and ACC functions and may notify the vehicle operator thereof.

[0099] 9 is a flow diagram illustrating an example process 900 for responding to light interference for autonomy level 4, according to some implementations. In some examples, the process 900 may be performed by the systems 100, 200, and / or 300 described above, for example, by execution of the recognition program 118, the light interference detection program 122, and / or the vehicle control program 120.

[0100] At 902, the system may receive an indication of detected light interference in a safety region of concern, as described above in block 636 of Figure 6. For example, a processor or processing thread executing light interference detection program 122 may send a notification to a processor or processing thread executing vehicle control program 120 indicating that light interference may be affecting the reliability of images being processed by image recognition program 118.

[0101] At 904, the system may send a visual and / or audio notification to the vehicle operator to take over control of the vehicle. Additionally, in some examples, the notification may recommend that the operator wash the windshield to reduce the amount of light that may be reflected toward the camera.

[0102] At 906, the system may send a notification to nearby vehicles to provide a warning of the poor light condition.

[0103] At 908, the system may wait a threshold amount of time for the driver to take over control of the vehicle.

[0104] At 910, the system may determine whether the driver has taken control of the vehicle, for example, by detecting one or more driver actions. If the driver has taken control of the vehicle, the process continues to 914. If not, the process continues to 912.

[0105] At 912, if the driver does not take control of the vehicle, the system may slow and / or stop the vehicle. For example, the system may check to determine if the vehicle can be stopped safely, and if so, may pull the vehicle over to a curb and stop it. If not, the system may simply slow the vehicle down, etc.

[0106] At 914, the system may wait a threshold time for the optical interference to cease.

[0107] At 916, the system may determine whether optical interference is still detected. If so, the process returns to 914 to wait for an additional threshold time. If not, the process proceeds to 918.

[0108] At 918, if the optical interference is no longer detected, the system may continue normal operation or may notify the vehicle operator.

[0109] 10 is a flow diagram illustrating an example process 1000 for responding to light interference for autonomy level 5, according to some implementations. In some examples, the process 1000 may be performed by the systems 100, 200, and / or 300 described above, for example, by execution of the recognition program 118, the light interference detection program 122, and / or the vehicle control program 120.

[0110] At 1002, the system may receive an indication of detected light interference in a safety region of concern, as described above in block 636 of Figure 6. For example, a processor or processing thread executing the light interference detection program 122 may send a notification to a processor or processing thread executing the vehicle control program 120 indicating that light interference may be affecting the reliability of images being processed by the image recognition program 118.

[0111] At 1004, the system may send a visual and / or audio notification to a vehicle occupant. For example, the vehicle may display a notification to the occupant and / or sound an alarm.

[0112] At 1006, the system may send a notification to nearby vehicles to provide a warning of the poor light condition.

[0113] At 1008, based on fusion with other sensor information, the system may determine whether to slow down and / or stop, or alternatively, continue the current drive until the light interference ends and rely on input from other sensors for navigation.

[0114] At 1010, the system may determine whether to slow down or stop based on fusion. If so, the process proceeds to 1012. If not, the process proceeds to 1018.

[0115] At 1012, the system may slow down and / or stop the vehicle. For example, the system may check to determine if it is safe to stop the vehicle, and if so, may pull over and stop the vehicle. If not, the system may simply slow down the vehicle, etc. Additionally, in some examples, the vehicle may spray the windshield and activate the windshield wipers to attempt to reduce the amount of light reflected back towards the camera.

[0116] At 1014, the system may wait a threshold time for the optical interference to cease.

[0117] At 1016, the system may determine whether optical interference is still detected. If so, the process returns to 1012 to wait for an additional threshold time. If not, the process proceeds to 1016.

[0118] At 1018, if the optical interference is no longer detected, the system may continue normal operation or may notify the vehicle driver / occupant.

[0119] The exemplary processes described herein are merely example processes provided for discussion purposes. In light of the disclosure herein, numerous other variations will be apparent to those skilled in the art. Furthermore, although the disclosure herein describes several examples of suitable frameworks, architectures, and environments for carrying out the processes, the implementations herein are not limited to the specific examples shown and described. Furthermore, the disclosure provides various exemplary implementations as described and illustrated in the drawings. However, the disclosure is not limited to the implementations described and illustrated herein, and may extend to other implementations as known or to become known to those skilled in the art.

[0120] Various instructions, processes, and techniques described herein may be considered in the general context of computer-executable instructions, such as computer programs and applications stored on a computer-readable medium and executed by a processor herein. In general, the terms programs and applications may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, and the like, for performing particular tasks or implementing particular data types. These programs, applications, and the like may be executed as native code, or may be downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of the programs and applications may be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques may be stored in computer storage media or transmitted over some form of communication medium.

[0121] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. one or more processors configured to perform processing according to executable instructions; The process comprises: receiving, from a stereo camera mounted on a vehicle, a left image and a right image captured by the stereo camera; generating a disparity image indicative of distances from the left and right images to features in the left and right images; determining at least one region in a first of the left or right images having a luminance above a luminance threshold; determining at least one region in the disparity image having a disparity information level below a disparity information threshold; comparing the first image and the disparity image to determine an overlap between at least one of the regions in the first image and at least one of the regions in the disparity image; determining an optical interference region based at least on said overlap; and A system included in a vehicle that includes transmitting a communication for controlling at least one vehicle function based on the region of optical interference.

2. The system of claim 1 , wherein the process of determining the region of optical interference based at least on the overlap further comprises determining that the overlap exceeds a minimum threshold size.

3. The system of claim 1 , wherein the process of determining the region of optical interference based at least on the overlap further comprises determining that the region of optical interference overlaps with a region corresponding to a direction of travel of the vehicle.

4. The process further includes determining to execute a process for detecting the optical interference region; The process comprises: determining the at least one region within the first image; determining said at least one region in said disparity image; and comparing the first image and the disparity image to determine the overlap; The determining to perform the process for detecting the optical interference region comprises at least determining that a current time is daytime based on receiving information from at least one of a light sensor or a clock; determining that the expected position of the sun is within a threshold angle of the horizon; receiving heading information from a compass; and The system of claim 1 , wherein the decision to execute the process for detecting optical interference is based on determining that at least a heading is within a heading threshold of the expected position of the sun.

5. 2. The system of claim 1, wherein the at least one vehicle function comprises a first vehicle function when the vehicle is operating under a first level of autonomy or a second, different vehicle function when the vehicle is operating under a second, different level of autonomy.

6. The process of determining at least one region having a luminance above a luminance threshold comprises: applying a threshold to pixels of the first image; and The system of claim 1 , further comprising: performing clustering on the pixels above the threshold to determine the at least one region having a luminance above the luminance threshold.

7. receiving, by one or more processors, images from a camera mounted on the vehicle; generating a disparity image based on features in at least one of the images; determining at least one region in a first one of the received images having a luminance above a luminance threshold; determining at least one region in the disparity image having a disparity information level below a disparity information threshold; determining an optical interference region based on an overlap between at least the at least one region in the first image and the at least one region in the parallax image; and performing at least one action based on the region of optical interference.

8. The method of claim 7 , wherein determining the region of optical interference based at least on the overlap further comprises determining that the overlap exceeds a minimum threshold size.

9. The method of claim 7 , wherein determining the region of light interference based at least on the overlap further comprises determining that the region of light interference overlaps with a region corresponding to a direction of travel of the vehicle.

10. The method of claim 7 , wherein determining the region of optical interference based at least on the overlap further comprises determining that the region of optical interference is detected for a threshold time.

11. The method of claim 7 , wherein performing the at least one action includes sending a communication to control at least one vehicle function.

12. 12. The method of claim 11, wherein the at least one vehicle function comprises a first vehicle function when the vehicle is operating under a first level of autonomy or a second, different vehicle function when the vehicle is operating under a second, different level of autonomy.

13. Determining at least one region having a luminance above a luminance threshold includes: applying a threshold to pixels of the first image; and The method of claim 7 , further comprising performing clustering on the pixels above the threshold to determine the at least one region having a luminance above the luminance threshold.

14. one or more processors configured to perform processing according to executable instructions; The process comprises: receiving images from a camera mounted on the vehicle; generating a disparity image based on features in at least one of the images; determining at least one region in a first one of the received images having a luminance above a luminance threshold; determining at least one region in the disparity image having a disparity information level below a disparity information threshold; determining an optical interference region based on an overlap between at least the at least one region in the first image and the at least one region in the parallax image; and performing at least one action based on the region of optical interference.

15. The system of claim 14 , wherein the process of determining the region of optical interference based at least on the overlap further comprises determining that the overlap exceeds a minimum threshold size.

16. The system of claim 14 , wherein the process of determining the region of optical interference based at least on the overlap further comprises determining that the region of optical interference overlaps with a region corresponding to a direction of travel of the vehicle.

17. The process further includes determining to execute a process for detecting the optical interference region; The process comprises: determining the at least one region within the first image; determining said at least one region in said disparity image; and comparing the first image and the disparity image to determine the overlap; The determining to perform the process for detecting the optical interference region comprises at least determining that a current time is daytime based on receiving information from at least one of a light sensor or a clock; determining that the expected position of the sun is within a threshold angle of the horizon; receiving heading information from a compass; and The system of claim 14 , wherein the decision to execute the process for detecting optical interference is based on determining that at least a heading is within a heading threshold of the expected position of the sun.

18. The system of claim 14 , wherein the process of performing the at least one action includes sending a communication to control at least one vehicle function.

19. 20. The system of claim 18, wherein the at least one vehicle function comprises a first vehicle function when the vehicle is operating under a first level of autonomy or a second, different vehicle function when the vehicle is operating under a second, different level of autonomy.

20. The process of determining at least one region having a luminance above a luminance threshold comprises: applying a threshold to pixels of the first image; and The system of claim 14 , further comprising: performing clustering on the pixels above the threshold to determine the at least one region having a luminance above the luminance threshold.

Images