Method and system for mitigating the effects of weather-related attenuation on radar images

Abstract

The technology disclosed relates to methods and systems for mitigating the effects of weather on radar images.SOLUTION: The computing system may be configured to receive radar images corresponding to an environment of a vehicle and estimate weather conditions for the environment using a weather forecasting model. The computing system may then be configured to detect weather-related attenuation effects on the radar image based on the estimated weather conditions of the environment, and correct the radar image by removing the weather-related attenuation effects. The corrected radar image may be used for object detection and tracking, which may assist a control system of the vehicle. In some cases, the system may be configured to determine that the operating range of the radar system is temporarily reduced and responsively reduce the speed of the vehicle until the weather conditions improve.SELECTED DRAWING: None

Description

[Background technology]

[0001] Advances in computing, sensors, and other technologies have enabled some vehicles to safely move between locations autonomously, i.e., without requiring input from a human driver. By processing sensor measurements of the surrounding environment in real time, semi-autonomous or autonomous vehicles can transport passengers or objects (e.g., cargo) between locations while avoiding obstacles, following traffic requirements, predicting the movements of nearby agents (e.g., other vehicles), and performing other actions typically performed by a driver. By shifting both vehicle decision-making and control to the vehicle system, vehicle occupants can devote their attention to tasks other than driving.

[0002] Automotive radar is a type of sensor technology used by advanced driver assistance systems (ADAS) and autonomous driving systems to detect and monitor the surrounding environment. In some cases, radar is used to detect and identify other vehicles, pedestrians, and various obstacles, as well as to ascertain their distance, speed, and trajectory.

[0003] Radar typically includes a radiating element with an antenna that transmits radio waves. These waves reflect off objects in the environment and are then captured by the antenna. The returned signals are then processed to evaluate the attributes of the reflecting objects, including a radar cross section (RCS) measurement that indicates how easily each object can be detected by the radar. A larger RCS indicates that the object is more easily detected, while a smaller RCS suggests that the object is more difficult to detect. An object's RCS depends on several factors, including the object's size, shape, material, and surface characteristics, as well as the frequency and polarization of the radar signal. Different types of objects, such as cars, motorcycles, pedestrians, and other objects, typically have varying RCS values ​​that can help identify the type of object in a radar image. Summary of the Invention

[0004] Exemplary embodiments relate to methods and systems for mitigating the effects of weather-related attenuation on radar imagery and other types of sensor data to improve object detection and tracking. The disclosed solutions can reduce the effects of weather-related attenuation on radar and other types of sensor data, allowing vehicle systems to continue using radar and other sensors under adverse weather conditions, such as rain, fog, or snow. In some cases, vehicle systems can implement the disclosed solutions to detect potential losses in sensor data caused by ambient environmental conditions and enhance the performance of driver assistance or vehicle autonomous systems while traveling in dynamic weather conditions. For example, a vehicle system can implement the disclosed techniques to identify when environmental conditions affect the operating range of radar and / or other sensors, allowing the system to adjust how sensor data from these sensors is used until weather conditions improve. In some cases, a vehicle may adjust its speed to prevent overdriving the vehicle's environmental visibility when the maximum sensing range of a radar or another sensor is limited by environmental conditions.

[0005] In one aspect, an exemplary method is described. The method includes receiving, at a computing system coupled to a vehicle, a radar image corresponding to an environment of the vehicle and estimating, by the computing system, weather conditions of the environment using a weather estimation model. The method also includes detecting weather-related attenuation effects on the radar image based on the estimated weather conditions of the environment and removing the weather-related attenuation effects to correct the radar image. The method also includes providing the corrected radar image to a control system of the vehicle.

[0006] In another aspect, a system is described. The system includes a vehicle radar system and a computing device coupled to the vehicle. The computing device is configured to receive a radar image corresponding to an environment of the vehicle and estimate weather conditions of the environment using a weather estimation model. The computing device is also configured to detect weather-related attenuation effects on the radar image based on the estimated weather conditions of the environment and remove the weather-related attenuation effects to correct the radar image. The computing device is also configured to provide the corrected radar image to a control system of the vehicle.

[0007] In yet another aspect, a non-transitory computer-readable medium is described. The non-transitory computer-readable medium is configured to store instructions that, when executed by a computing system having one or more processors, cause the computing system to perform operations. The operations include receiving a radar image corresponding to an environment of a vehicle and estimating, by the computing system, weather conditions of the environment using a weather estimation model. The operations also include detecting weather-related attenuation effects on the radar image based on the estimated weather conditions of the environment and removing the weather-related attenuation effects to correct the radar image. The operations further include providing the corrected radar image to a control system of the vehicle.

[0008] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art from a reading of the following detailed description, where appropriate with reference to the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram illustrating a vehicle, according to an exemplary embodiment. [Figure 2A] FIG. 2A is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2B]FIG. 2B is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2C] FIG. 2C is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2D] FIG. 2D is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2E] FIG. 2E is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2F] FIG. 2F is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2G] FIG. 2G is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2H] FIG. 2H is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2I] FIG. 2I is an illustration of a physical configuration of a vehicle, according to an exemplary embodiment. [Figure 2J] FIG. 2J is an illustration of the fields of view of various sensors, according to an example embodiment. [Figure 2K] FIG. 2K is an illustration of beam steering relative to a sensor, according to an example embodiment. [Figure 3] FIG. 3 is a conceptual illustration of wireless communication between various computing systems associated with an autonomous or semi-autonomous vehicle, in accordance with an example embodiment. [Figure 4] FIG. 4 is a block diagram of a system including a radar unit in accordance with an exemplary embodiment. [Figure 5] FIG. 5 is a conceptual diagram of a system for mitigating the effects of weather-related attenuation on vehicle radar data, in accordance with an exemplary embodiment. [Figure 6] FIG. 6 is a flowchart of a method for mitigating the effects of weather-related attenuation on vehicle radar data, according to an exemplary embodiment. [Figure 7] FIG. 7 is a flowchart of another method for mitigating the effects of weather-related attenuation on vehicle radar data, according to an exemplary embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Exemplary methods and systems are described in this disclosure. Any exemplary embodiment or configuration described in this disclosure should not necessarily be construed as preferred or advantageous over other embodiments or configurations. Moreover, the exemplary embodiments described in this disclosure are not meant to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated in this disclosure. Additionally, the specific arrangements illustrated in the figures should not be considered limiting. It should be understood that other embodiments can include more or fewer of each element shown in a given figure. Additionally, some of the illustrated elements can be combined or omitted. Still further, exemplary embodiments may include elements not illustrated in the figures.

[0011] Automotive radar systems may experience rain, snow, cloudiness, and other adverse weather conditions that can affect radar signal propagation and weaken the overall signal. When sufficient attenuation occurs, the overall quality of the radar data decreases, potentially reducing the radar system's ability to accurately detect and track objects in the surrounding environment. In some cases, attenuation caused by the vehicle's environment can reduce the maximum sensing capability of the radar (and potentially other sensors) used by the vehicle. This suboptimal performance by the vehicle's radar system (and potentially other sensor systems) may lead to overly confident driving behavior by the vehicle in adverse weather conditions. In particular, the vehicle system may fail to implement corrective action, such as reducing the vehicle's speed, if the system is unable to determine that environmental conditions have temporarily reduced the maximum sensing range of the vehicle's radar system and / or other sensor systems.

[0012] Weather conditions can cause attenuation of radar and other sensor signals in different ways. Falling rain, snow flakes, and other types of precipitation can scatter radar signals and reduce the amount of energy reflected back to the radar system. The specific level of attenuation may depend on the intensity, size, and fall velocity of the precipitation. For example, heavy rain with a higher rainfall rate and / or larger raindrops may weaken the return signal received by the radar system and cause greater attenuation than light rain. Similarly, wet, heavy snow can similarly produce a more severe attenuation effect on radar signals compared to the effect produced by dry, powdery snow. In addition, water droplets or ice particles located on the sensor's transmit aperture and / or radome can also adversely affect sensor performance by absorbing and / or scattering the signal, further reducing the signal strength of transmitted and received signals. In some cases, atmospheric gases such as oxygen and water vapor can also contribute to signal strength loss by absorbing electromagnetic energy at certain frequencies. Drafty conditions experienced during vehicle navigation can also increase the attenuation effect by causing precipitation or other particle movement in the air to an extent that affects the scattering and absorption properties of the air.

[0013] By affecting the propagation and reception of sensor signals, some environmental conditions can degrade the performance of vehicle sensors, such as limiting the sensor's current maximum sensing range and other operating parameters. Degraded radar imagery and other sensor data may appear dimmer than similar sensor data generated during favorable weather conditions due to signal loss caused by environmental conditions. Losses associated with radar imagery or other types of sensor data can reduce the accuracy of algorithms used to detect and track objects in the radar imagery or sensor data. As such, vehicle systems may not be able to rely on radar imagery or sensor data without accounting for the reduced intensity caused by weather conditions. These issues may be more severe in conditions of heavy precipitation, which causes radar imagery and / or other sensor data to experience greater attenuation, leading to greater losses.

[0014] The present disclosure relates to methods and systems for detecting and mitigating the effects of weather-related attenuation on vehicle sensor data, including radar imagery. In some aspects, the disclosed methods and systems address the problem of radar beam attenuation that can occur due to adverse weather conditions and can help prevent overreliability or undesirable autonomous vehicle driving behavior under adverse weather conditions. A vehicle system may implement the disclosed techniques to determine when environmental conditions have reduced the maximum range and / or other operating parameters of the vehicle radar system and other vehicle sensor systems. The vehicle system may then temporarily adjust the vehicle's speed or other control parameters (e.g., braking distance) to prevent the vehicle from overdriving its visibility. If the vehicle system determines that the operating range of the radar system and / or other sensor systems has improved due to changes in environmental conditions, the vehicle system can then further adjust the vehicle control strategy.

[0015] The disclosed solutions generally involve estimating and factoring the impact of weather-related attenuation on radar imagery and / or other sensor data to enable real-time adjustment of vehicle operation and control strategies to respond when the environment impairs sensor performance. For example, a vehicle computing system may be configured to use the disclosed techniques to restore radar imagery to what would appear under favorable weather conditions. The radar image correction process may include real-time estimation of radar sensitivity degradation, which can be used by the vehicle system to implement one or more corrective actions that can improve the safety and reliability of general autonomous vehicle operation. In some cases, the disclosed solutions are used by the vehicle system to estimate and account for the impact of weather conditions on radar data and other types of sensor data, such as camera imagery or point cloud data from lidar.

[0016] First, an exemplary method for mitigating the effects of weather attenuation on radar imagery may include using a weather estimation model as an intermediary to estimate current weather conditions experienced in the vehicle's environment. For example, a vehicle system or another computing system may be configured to estimate rainfall in the vehicle's environment using a weather estimation model configured with a rainfall estimator. The rainfall estimator may analyze newly acquired radar imagery to output rainfall measured in millimeters per hour (mm / hr) or another unit of measure, which can then be mapped to aperture and atmospheric losses in return signal strength for one or more radars operating on the vehicle. These losses in return signal strength may be expressed in decibels (dB), enabling the vehicle system to apply corrective measures to maintain safe navigation despite adverse environmental conditions.

[0017] To estimate losses caused by environmental conditions, the computing system may be configured to use a weather loss model. The weather loss model may be trained to map precipitation to losses in sensor data caused by environmental conditions. The losses may represent decreases in RCS values ​​caused by precipitation (e.g., rain) and other environmental conditions. The computing system may be configured to use the estimated losses to determine whether to implement corrective actions. In some cases, the losses may not be substantial enough to trigger the vehicle to implement one or more corrective actions. For example, the computing system may be configured to utilize a loss threshold or multiple loss thresholds to determine when to implement a particular corrective action. In some examples, each corrective action may be associated with a different loss threshold.

[0018] Corrective actions represent different strategies the vehicle system may employ to mitigate effects caused by weather and / or other environmental conditions. In some cases, the control system may be configured to adjust vehicle control strategies based on estimated losses associated with one or more sensors. For example, if environmental conditions temporarily reduce the maximum sensing range of a radar system, the control system may be configured to reduce the vehicle's speed to prevent the vehicle from operating beyond reduced visibility. In this manner, the vehicle system may implement the disclosed techniques to determine when environmental conditions reduce the operating range of one or more sensors.

[0019] In some cases, a corrective action may be applied to the radar image, such as applying an offset. For example, the corrective action may include offsetting a decrease in return signal strength caused by weather conditions. The size of the offset depends on the size of the estimated loss. For example, if the computing system determines that the radar image has an approximate loss of negative 20 decibels (dB), the computing system may be configured to apply a positive 20 dB offset. The weather loss model may be trained to output an appropriate offset or offsets to be applied to convey how the radar image, or other form of sensor data, will appear in clear weather conditions.

[0020] In some embodiments, the computing system may be configured to use multiple models, with the weather estimation model and the weather loss model existing as separate models. In other embodiments, the weather estimation model and the weather loss model may be part of the same model. Additionally, the models may be updated in real time on the vehicle and / or periodically by a remote computing system. For example, updated models may be received by the vehicle system via over-the-air updates generated by the remote computing system.

[0021] The computing system may be configured to estimate weather and other conditions in the vehicle's environment using various techniques and information sources. For example, vehicle sensors, information provided by external sources (e.g., other vehicles or computing systems), and weather forecasts may be used by the vehicle system to detect precipitation and other environmental conditions that may affect sensor performance. Some exemplary vehicle sensors that may be used to detect external conditions include radar, camera, radar, temperature sensors, humidity sensors, and / or wind sensors, among others. For example, a weather estimation model may be trained to map raw backscatter values ​​in a radar image to an estimated rainfall amount. The estimated rainfall amount may indicate the rate at which rain is falling in the vehicle's environment. The weather estimation model may similarly be trained to estimate the rate at which other forms of precipitation (snow, hail, sleet) fall. The computing system may then use a weather loss model trained to map the estimated rainfall amount (or the estimated rate at which other types of precipitation are falling) to losses associated with radar images or other types of sensor data acquired by the vehicle sensors during navigation in those environmental conditions. The weather loss model may be configured to detect an estimated decrease in observed RCS values ​​in radar images. During vehicle navigation, the computing system may be configured to use the weather loss model to analyze estimated weather conditions to detect potential adverse effects of the weather conditions on radar images and / or other types of sensor data.

[0022] Another exemplary method may bypass estimating current weather conditions in the vehicle's environment and proceed directly to comparing the observed sensor radar data to a baseline model or set of baseline models. The results of the comparison may enable a computing system to estimate losses associated with radar imagery or other types of sensor data. This method may be performed periodically or continuously by the vehicle system to understand how the dynamic environment experienced during travel affects sensor performance. For example, the computing system may be configured to periodically or continuously compare incoming radar data to one or more baseline models to assess the maximum sensing range and other operating parameters of the radar system. In some cases, if the loss difference between the incoming radar data and the baseline models exceeds a loss threshold, the computing system may be configured to trigger the execution of one or more corrective actions, such as adjusting the vehicle's current control strategy and / or applying an offset to the radar data to correct the radar data.

[0023] The computing system may be configured to use one or more baseline models to detect when environmental conditions degrade radar imagery or other types of sensor data. The baseline model may represent what the sensor data would likely look like if the vehicle were operating in clear weather conditions. In some cases, the baseline model may represent expected values ​​if the sensor data were generated in clear weather conditions. For example, a baseline model used to analyze radar imagery may convey expected RCS values ​​for objects in the environment. The baseline model may represent information useful for evaluating the impact of environmental conditions on sensor performance using various benchmarks. The format and information conveyed by the baseline model may depend on the type of sensor being analyzed. For example, a radar baseline model may take the form of a radar map, information conveying expected RCS values ​​for different object types, or an expected clutter-to-noise profile.

[0024] By comparing observed approach data to baseline expectations, the vehicle system may be configured to detect and estimate potential losses caused by environmental conditions on the radar image and, in response, implement one or more corrective actions. In some cases, corrective actions may include adjusting the vehicle's speed to accommodate a reduction in the detection range or visibility of a radar system or other type of sensor system, modifying the use or reliance on particular vehicle sensors, or pulling the vehicle over until environmental conditions change. Other corrective actions may be taken by the vehicle system.

[0025] The disclosed methods can be implemented using radar data in various forms. For example, a computing system may perform the techniques described in this disclosure using radar data in voxel format. In other instances, the observed radar data may be represented as a range-Doppler map (RDM). The RDM may provide a two-dimensional representation of the radar data, with one dimension representing the range (or distance) to a target and the other representing the Doppler shift (or relative velocity) of the target. In other cases, the observed radar data may be represented as a range-angle image. The range-angle image may provide a two-dimensional representation of the radar data, with one dimension representing the range to a target and the other representing the angle of arrival of the radar signal. Furthermore, the computing system implementing the disclosed methods may vary within embodiments and may be configured to include any combination of computing devices located on or remotely from the vehicle. In some cases, a field-programmable gate array (FPGA) and / or application-specific integrated circuit located at the radar may be configured to perform one or more aspects of the disclosed methods.

[0026] In some aspects, correcting a radar image may include removing weather-related attenuation effects based on a comparison with one or more baseline models. This process may include identifying differences between observed radar data and expected values ​​represented by one or more baselines. These differences may represent the effects of weather-related attenuation on the radar image, and these differences may be removed from the observed radar data to correct the radar image. Thus, the corrected radar image may represent what the radar image would look like under favorable weather conditions, providing a more accurate representation of the environment.

[0027] In some cases, the computing system may be configured to use a baseline model including a radar map that represents how radar images appear at different locations during favorable weather conditions. Favorable weather conditions may represent conditions in which the environment lacks precipitation or other factors (e.g., high winds or humidity) that can cause radar signal attenuation, reducing the accuracy of radar images generated by the radar system. In some examples, the computing system may be configured to create or update the radar map based on radar images captured during favorable weather conditions. The computing system may be configured to use other sensor data to verify that current weather conditions are fair when collecting radar images to create or update the radar map. In other examples, the computing system may be configured to receive the model with the radar map via an over-the-air update received from a remote computing system. For example, the remote computing system may be configured to generate the radar map based on radar data generated during known test conditions and / or acquired from multiple vehicles traveling during favorable weather conditions, and distribute the radar map to a fleet of vehicles as part of the update.

[0028] In some embodiments, the computing system may be configured to compare RCS values ​​represented by the current radar image with previously determined RCS values ​​for different object types during favorable weather conditions. For example, the computing system may be configured to use a baseline model that indicates these predetermined RCS values ​​for various objects frequently encountered during vehicle navigation, such as other vehicles, buildings, signs, lane markers, traffic signals, pedestrians, and bicyclists, among others. If the RCS values ​​of objects in the environment derived from the current radar data differ from the expected RCS values ​​for the same object types specified in the model, the computing system may perform correction operations to calibrate the current radar data. In particular, the computing system may be configured to apply an offset to the RCS values ​​derived from the current radar image as a way of accounting for the effect of current weather conditions on the performance of the current radar system.

[0029] In some examples, the computing system may be configured to detect effects on radar imagery or other types of sensor data by using an expected clutter-to-noise profile for weather estimation, which may provide a baseline reference for expected clutter and noise levels in radar imagery under favorable weather conditions. One or more types of baseline references may be used to compare with observed radar data to detect levels of weather-related attenuation effects on radar imagery. For example, one or more models may be used, with each model representing one or more baseline references that can be used to assess the quality of the radar imagery.

[0030] In some cases, the computing system may instruct downstream vehicle systems to rely less on or refrain from using information sensed based on radar data until the computing system determines that environmental weather conditions have improved. The vehicle control system may also be configured to adjust vehicle control strategies when radar data is corrupted by weather conditions. For example, the vehicle control system may be configured to increase following distance, apply braking earlier, and / or adjust speed and steering strategies, among other potential changes.

[0031] In some embodiments, the selection of a baseline model used to analyze the quality of radar data received from one or more vehicle radars may be configured depending on criteria such as estimated weather conditions for the vehicle's environment, the vehicle's location, and / or the availability of a baseline model. The baseline model may be customized for each vehicle or may be a common model used by multiple vehicles that may operate as part of a fleet of vehicles. For example, the radar map used for weather compensation may be configured to vary depending on the particular implementation. In some cases, a common radar map may be used for all vehicles in the fleet. For example, the common radar map may be a standardized map used across the fleet of vehicles, which may depend on vehicle type, location, and / or radar system parameters. Vehicles using radar systems operating within a particular frequency band may use one common radar map, while vehicles using radar systems operating in other frequency bands may use another common radar map.

[0032] In another example, each vehicle may be configured to develop its own radar map. This individual radar map may be specific to the vehicle and may be developed based on the vehicle's own radar data and experience. The accuracy of the radar map may affect the accuracy of the weather compensation. In some cases, the vehicles may be configured to customize the common map based on their own radar data and particular experience. The vehicles may also be configured to provide radar data and other information to a central system, which can update the common map and provide the updated common map back to the fleet of vehicles. This feedback loop allows the vehicles to enhance the radar map or other types of baseline models used to detect the effects of severe weather on radar data and other types of sensors.

[0033] In addition to the above method, the present disclosure also describes a system that can mitigate the adverse effects of weather conditions on radar data and other types of sensor data. By mitigating the effects of weather on sensor data, a vehicle may be able to rely on the sensor data to navigate safely during various weather conditions. The system may include a vehicle radar system and a computing device coupled to the vehicle. The computing device may be configured to receive radar images corresponding to the vehicle's environment, estimate weather conditions of the environment using a weather estimation model, detect weather-related attenuation effects on the radar image based on the estimated weather conditions, remove the weather-related attenuation effects to correct the radar image, and provide the corrected radar image to a control system of the vehicle. This system may enhance the safety and reliability of autonomous vehicle operation by providing more accurate radar data regardless of weather conditions. The system may also be used to enhance a driver's understanding of their surroundings by providing information (e.g., an audio-visual representation) about the environment derived from the corrected radar data.

[0034] In some aspects, the methods and systems may be configured to provide real-time estimates of radar sensitivity degradation. This real-time estimate may enable immediate adjustments to the radar system or the driving behavior of the autonomous vehicle, potentially enhancing the safety and reliability of autonomous vehicle operation. For example, if radar sensitivity is determined to be severely degraded due to heavy rain, the autonomous vehicle may be instructed to slow down or take other preventative measures.

[0035] In some cases, the methods and systems may include comparing current radar data products or summary metrics at a particular location with historical data that ranges from unresolved to highly resolved. This comparison may allow for an estimation of a level of resolution that can be used to correct the radar image. The corrected radar image may then be provided to a vehicle control system that can use the image to control the operation of the vehicle.

[0036] The disclosed methods and systems may provide several technical benefits that may enhance vehicle navigation. Reducing the effects of weather-related attenuation on radar images may enhance the performance of object detection and tracking systems used by vehicles under various weather conditions. The disclosed methods and systems enable improved recognition in adverse weather conditions. By reducing the effects of weather-related attenuation, radar systems can maintain high-quality perception capabilities even under adverse weather conditions such as rain, fog, or snow. More accurate radar data may enable better performance by vehicle control systems, leading to safer driving behavior overall. For example, vehicle systems may use radar data to reliably detect obstacles, pedestrians, or other vehicles and take appropriate action to avoid collisions.

[0037] In some cases, real-time estimation of radar sensitivity degradation may enable immediate adjustments to the radar system or vehicle's driving behavior. This dynamic response may be useful for improving vehicle performance and safety in rapidly changing weather conditions. Corrected radar imagery may also improve the detection and classification of objects in the surrounding environment. This may lead to more accurate predictions regarding the behavior of other road users and better strategic planning for autonomous navigation systems. The disclosed methods and systems may also enable radar systems to provide increased operating range and better object detection and classification. The ability to compensate for weather effects potentially increases the operating range of ADAS and autonomous systems, allowing them to function effectively in a wider range of weather conditions. In some examples, a vehicle may use the corrected radar imagery to display a representation of detected targets to the vehicle, which can enhance driver awareness. Furthermore, systems for mitigating weather-related attenuation may be tailored to a specific geographic location by considering local weather patterns and conditions, which can further improve the accuracy of weather compensation and radar data interpretation.

[0038] The following description and the accompanying drawings highlight features of various exemplary embodiments. The embodiments provided are by way of example and are not intended to be limiting. Accordingly, dimensions of the drawings are not necessarily to scale.

[0039] Exemplary systems within the scope of the present disclosure will now be described in more detail. The exemplary system may be implemented in or take the form of an automobile. Additionally, the exemplary system may also be implemented in or take the form of a variety of vehicles, such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor-trailers), motorcycles, buses, airplanes, helicopters, drones, lawn mowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment or vehicles, construction machinery or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, golf carts, trains, trolleys, walkway transport vehicles, and robotic devices. Other vehicles are possible as well. Furthermore, in some embodiments, the exemplary system may not include a vehicle.

[0040] Referring now to the figures, FIG. 1 is a functional block diagram illustrating an example vehicle 100 that may be configured to operate fully or partially in an autonomous mode. More specifically, vehicle 100 may operate in the autonomous mode without human interaction through receiving control instructions from a computing system. As part of its operation in the autonomous mode, vehicle 100 may use sensors to detect, and in some cases detect, objects in the surrounding environment to enable safe navigation. Additionally, vehicle 100 may operate in a partially autonomous (i.e., semi-autonomous) mode in which some functions of vehicle 100 are controlled by a human driver of vehicle 100 and some functions of vehicle 100 are controlled by a computing system. For example, vehicle 100 may also include subsystems that enable the driver to control the operation of vehicle 100, such as steering, acceleration, and braking, while the computing system performs assistance functions, such as lane departure warning / lane keeping assist or adaptive cruise control, based on other objects (e.g., vehicles) in the surrounding environment.

[0041] As described herein, in a partially autonomous driving mode, the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), and emergency braking), but the human driver is expected to maintain situational awareness of the vehicle's surroundings and supervise the assisted driving operations. Here, the vehicle may perform all driving tasks in a particular situation, but the human driver is expected to assume responsibility for taking control as needed.

[0042] For simplicity and brevity, various systems and methods are described below in conjunction with autonomous vehicles; however, these or similar systems and methods may be used in various driver assistance systems that fall short of a fully autonomous driving system (i.e., a partially autonomous driving system). In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving behavior to indicate how much or how little control the vehicle has over the driving; however, different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used in SAE Level 2 driver assistance systems, which implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver support. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems, which are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems, which operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such systems, accurate lane estimation is performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, and overall safety of autonomous, semi-autonomous, and other driver assistance systems. As noted above, in addition to the way the SAE classifies levels of autonomous driving performance, other organizations in the United States or other countries may classify levels of autonomous driving performance differently. Without limitation, the systems and methods disclosed herein may be used with driver assistance systems defined by these other organizations' levels of autonomous driving performance.

[0043] As shown in FIG. 1 , vehicle 100 may include various subsystems, such as propulsion system 102, sensor system 104, control system 106, one or more peripheral devices 108, power source 110, computer system 112 (which may also be referred to as a computing system) having data storage 114, and user interface 116. In other examples, vehicle 100 may include more or fewer subsystems, each of which may include multiple elements. The subsystems and components of vehicle 100 may be interconnected in various ways. Additionally, the functionality of vehicle 100 described herein may be divided into additional functional or physical components or combined into fewer functional or physical components within an embodiment. For example, control system 106 and computer system 112 may be combined into a single system that operates vehicle 100 according to various operations.

[0044] Propulsion system 102 may include one or more components operable to provide powered motion for vehicle 100, such as an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121, among other possible components. For example, engine / motor 118 may be configured to convert energy source 119 into mechanical energy and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, among other possible options. For example, in some embodiments, propulsion system 102 may include multiple types of engines and / or motors, such as gasoline engines and electric motors.

[0045] Energy source 119 represents an energy source that may fully or partially power one or more systems (e.g., engine / motor 118) of vehicle 100. For example, energy source 119 may correspond to gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, energy source 119 may include a combination of a fuel tank, a battery, a capacitor, and / or a flywheel.

[0046] The transmission 120 may be configured to transfer mechanical power from the engine / motor 118 to the wheels / tires 121 and / or other possible systems of the vehicle 100. As such, the transmission 120 may include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft may include an axle that connects to one or more wheels / tires 121.

[0047] The wheels / tires 121 of the vehicle 100 may be configured in a variety of configurations within the exemplary embodiment. For example, the vehicle 100 may exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or four wheels of a car / truck, among other possible configurations. Thus, the wheels / tires 121 may connect to the vehicle 100 in a variety of ways and may exist in different materials, such as metal and rubber.

[0048] The sensor system 104 may include various types of sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, radar 126, lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, among other possible sensors. In some embodiments, the sensor system 104 may also include sensors configured to monitor internal systems of the vehicle 100 (e.g., an O monitor, a fuel gauge, engine oil temperature, and brake wear).

[0049] The GPS 122 may include a transceiver operable to provide information regarding the position of the vehicle 100 relative to the Earth. The IMU 124 may include one or more accelerometers and / or gyroscopes and may sense changes in the position and orientation of the vehicle 100 based on inertial acceleration. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or moving.

[0050] Radar 126 may represent one or more systems configured to sense objects in the environment surrounding vehicle 100, including the object's speed and orientation, using radio signals. Thus, radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, radar 126 may correspond to a mountable radar configured to obtain measurements of the environment surrounding vehicle 100.

[0051] The LIDAR 128 may include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may be configured to operate in a coherent mode (e.g., using heterodyne detection) or an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, the one or more detectors of the LIDAR 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged (e.g., through serial electrical connections) in an array (e.g., as with silicon photomultiplier tubes (SiPMs)). In some examples, the one or more photodetectors are devices operating in Geiger mode, and the LIDAR includes subcomponents designed for such Geiger mode operation.

[0052] Camera 130 may include one or more devices (e.g., a still camera, a video camera, a thermal imaging camera, a stereo camera, and a night vision camera) configured to capture images of the environment surrounding vehicle 100.

[0053] The steering sensor 123 may be configured to sense a steering angle of the vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representative of the angle of the steering wheel. In some embodiments, the steering sensor 123 may be configured to measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the forward axle of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or subset) of the steering wheel angle, the electrical signal representative of the steering wheel angle, and the angle of the wheels of the vehicle 100.

[0054] The throttle / brake sensor 125 may be configured to detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angle of both the accelerator pedal (throttle) and the brake pedal, or may measure an electrical signal representative of, for example, the accelerator pedal (throttle) angle and / or the brake pedal angle. The throttle / brake sensor 125 may also measure the angle of a throttle body of the vehicle 100, which may include part of the physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., a butterfly valve and a carburetor). Additionally, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on a rotor of the vehicle 100, or a combination (or subset) of the accelerator pedal (throttle) and the brake pedal angle, an electrical signal representative of the accelerator pedal (throttle) and the brake pedal angle, the throttle body angle, and the pressure applied by at least one brake pad to a rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure pressure applied to a pedal of the vehicle, such as a throttle or brake pedal.

[0055] The control system 106 may include components configured to assist in navigating the vehicle 100, such as a steering unit 132, a throttle 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / pathfinding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the heading of the vehicle 100, and the throttle 134 may be configured to control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 may slow the vehicle 100, which may include slowing the wheels / tires 121 using friction. In some embodiments, the braking unit 136 may convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by a system or systems of the vehicle 100.

[0056] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithm capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide an assessment based on the received sensor data, such as an assessment of individual objects and / or features, an assessment of a particular situation, and / or an assessment of possible effects within a given situation.

[0057] Computer vision system 140 may include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a special-purpose processor such as a graphics processing unit (GPU) or a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) operable to process and analyze images to determine moving objects (e.g., other vehicles, pedestrians, bicyclists, or animals) and non-moving objects (e.g., road lights, roadway boundaries, speed bumps, or potholes). Thus, computer vision system 140 may employ object recognition, structure-from-motion (SFM), video tracking, and other algorithms used in computer vision, for example, to recognize objects, map environments, track objects, estimate object speed, etc.

[0058] Navigation / routing system 142 may determine a driving route for vehicle 100, which may include dynamically adjusting navigation during operation. Thus, navigation / routing system 142 may use data from sensor fusion algorithms 138, GPS 122, and maps, among other sources, to navigate vehicle 100. Obstacle avoidance system 144 may assess potential obstacles based on sensor data and cause systems of vehicle 100 to avoid or otherwise navigate around the potential obstacles.

[0059] 1 , vehicle 100 may also include peripherals 108, such as a wireless communication system 146, a touchscreen 148, a microphone 150 (e.g., one or more internal and / or external microphones), and / or a speaker 152. Peripherals 108 may provide controls or other elements for a user to interact with a user interface 116. For example, touchscreen 148 may provide information to a user of vehicle 100. User interface 116 may also accept input from a user via touchscreen 148. Peripherals 108 may also enable vehicle 100 to communicate with devices, such as devices in other vehicles.

[0060] The wireless communication system 146 may be configured to communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communications such as Code Division Multiple Access (CDMA), Evolutionary Data Optimized (EVDO), Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or 4G Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may communicate with a wireless local area network (WLAN) using Wi-Fi or other possible connections. The wireless communication system 146 may also be configured to communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, are possible within the context of this disclosure. For example, wireless communication system 146 may include one or more dedicated short range communication (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0061] Vehicle 100 may include a power source 110 for powering the components. Power source 110, in some embodiments, may include a rechargeable lithium-ion or lead-acid battery. For example, power source 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power sources. In an exemplary embodiment, power source 110 and energy source 119 may be integrated into a single energy source.

[0062] Vehicle 100 may also include a computer system 112 for performing operations such as those described herein. Accordingly, computer system 112 may include a processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory computer-readable medium, such as data storage 114. As such, processor 113 may represent one or more processors. In some embodiments, computer system 112 may represent multiple computing devices that may function to control individual components or subsystems of vehicle 100 in a distributed manner.

[0063] In some embodiments, data storage 114 may include instructions 115 (e.g., program logic) executable by processor 113 to perform various functions of vehicle 100, including those described above in connection with Figure 1. Data storage 114 may also include additional instructions, including instructions to transmit data to, receive data from, interact with, and / or control one or more of propulsion system 102, sensor system 104, control system 106, and peripherals 108.

[0064] In addition to instructions 115, data storage 114 may store data such as road maps, route information, etc., among other information. Such information may be used by vehicle 100 and computer system 112 during operation of vehicle 100 in autonomous, semi-autonomous, and / or manual modes.

[0065] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. User interface 116 may control or allow control of content and / or the layout of interactive images that may be displayed on touchscreen 148. Additionally, user interface 116 may include one or more input / output devices in a set of peripherals 108, such as wireless communication system 146, touchscreen 148, microphone 150, and speaker 152.

[0066] Computer system 112 may be configured to control functions of vehicle 100 based on inputs received from various subsystems (e.g., propulsion system 102, sensor system 104, or control system 106) and from user interface 116. For example, computer system 112 may be configured to use inputs from sensor system 104 to estimate outputs generated by propulsion system 102 and control system 106. Depending on the embodiment, computer system 112 may be operable to monitor many aspects of vehicle 100 and its subsystems. In some embodiments, computer system 112 may be configured to disable some or all functions of vehicle 100 based on signals received from sensor system 104.

[0067] Components of vehicle 100 may be configured to function in an interconnected manner with other components, either within or external to their respective systems. For example, in an exemplary embodiment, camera 130 may be configured to capture multiple images that may represent information about the state of the environment surrounding vehicle 100 while it is operating in an autonomous or semi-autonomous mode. The state of the environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be capable of recognizing slopes (gradients) or other features based on multiple images of the road. Additionally, the combination of GPS 122 and the features recognized by computer vision system 140 may be used in conjunction with map data stored in data storage 114 to determine specific road parameters. Furthermore, radar 126, lidar 128, and / or some other environmental mapping, range, and / or positioning sensor system may also provide information about the vehicle's surroundings.

[0068] In other words, the combination of various sensors (which may be referred to as input indicator sensors and output indicator sensors) and computer system 112 may interact to provide an indication of the inputs provided to control the vehicle or an indication of the vehicle's surroundings.

[0069] In some embodiments, computer system 112 may be configured to detect various objects based on data provided by systems other than a wireless system. For example, vehicle 100 may have laser or other optical sensors configured to sense objects within the vehicle's field of view. Computer system 112 may be configured to use output from the various sensors to detect information about objects within the vehicle's field of view and to measure distance and direction information to the various objects. Computer system 112 may also be configured to determine whether an object is desirable or undesirable based on output from the various sensors.

[0070] 1 depicts various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) as being integrated into vehicle 100, one or more of these components may be attached to or provided separately from vehicle 100. For example, data storage 114 may be partially or completely separate from vehicle 100. Thus, vehicle 100 may be provided with device elements in separate locations or with device elements co-located at the same location. The device elements comprising vehicle 100 may be communicatively coupled together in a wired and / or wireless manner.

[0071] 2A-2E illustrate an exemplary vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle) that may include some or all of the functionality described in connection with vehicle 100 with reference to FIG. 1. Vehicle 200 is illustrated in FIGS. 2A-2E as a van with side mirrors for illustrative purposes, but the present disclosure is not so limited. For example, vehicle 200 may represent a truck, a car, a semi-trailer truck, a motorcycle, a golf cart, an off-road vehicle, an agricultural vehicle, or any other vehicle described elsewhere herein (e.g., a bus, a boat, an airplane, a helicopter, a drone, a lawn mower, a bulldozer, a submarine, an all-terrain vehicle, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, farm equipment, construction machinery or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, a tram, a train, a trolley, a walkway transport vehicle, and a robotic device).

[0072] Vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidars, one or more radars, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding vehicle 200. In other words, any sensor system now known or hereafter created may be coupled to vehicle 200 and / or utilized in conjunction with various operations of vehicle 200. As an example, lidar may be utilized for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of vehicle 200. Additionally, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent a combination of the sensors described herein (e.g., one or more lidars and radars, one or more lidars and cameras, one or more cameras and radars, or one or more lidars, cameras, and radars).

[0073] 2A-E are intended as non-limiting examples of the locations, numbers, and types of such sensor systems in an autonomous or semi-autonomous vehicle. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to accommodate vehicle size, shape, aerodynamics, fuel economy, aesthetics, or other considerations to reduce cost or suit particular environmental or application situations). For example, sensor systems (e.g., 202 and 204) may be disposed in various other locations on the vehicle (e.g., at location 216) and configured to provide a corresponding field of view of the interior and / or surrounding environment of vehicle 200.

[0074] Sensor system 202 may be mounted on top of vehicle 200 and may include one or more sensors configured to detect information about the environment surrounding vehicle 200 and output an indication of that information. For example, sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). Sensor system 202 may include one or more movable mounts that may be operable to adjust the orientation of one or more sensors in sensor system 202. In one embodiment, the movable mount may include a rotating platform that may scan the sensors to obtain information from each direction around vehicle 200. In another embodiment, the movable mount of sensor system 202 may be movable in a scanning manner within a specified range of angles, and / or azimuth and / or elevation angles. Sensor system 202 may be mounted on the roof of a vehicle, although other mounting locations are also possible.

[0075] Additionally, the sensors of sensor system 202 may be distributed across different locations and need not be co-located at a single location. Furthermore, each sensor of sensor system 202 may be configured to be moved or scanned independently of the other sensors in sensor system 202. Additionally or alternatively, multiple sensors may be mounted in one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two lidar devices mounted at a sensor location and / or there may be one lidar device and one radar mounted at a sensor location.

[0076] One or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more lidar devices. For example, a lidar device may include multiple light emitter devices arranged across a range of angles relative to a given plane (e.g., the x-y plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis perpendicular to the given plane (e.g., the z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment may be sensed based on detecting various aspects of the reflected light pulses (e.g., the elapsed time of flight, polarization, and intensity).

[0077] In an exemplary embodiment, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide respective point cloud information that may be associated with physical objects within the environment surrounding vehicle 200. While vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are illustrated as including certain features, it will be understood that other types of sensor systems are contemplated within the scope of the present disclosure. Furthermore, vehicle 200 may be configured to include any of the components described in connection with vehicle 100 of FIG. 1 .

[0078] In an exemplary configuration, one or more radars may be disposed on vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves having frequencies between 30 Hz and 300 GHz). Such radio waves may be used to measure the distance and / or speed of one or more objects in the environment surrounding vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some examples, one or more radars may be disposed near the rear of vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of vehicle 200 for the presence of radio wave-reflecting objects. Similarly, one or more radars may be disposed near the front of vehicle 200 (e.g., sensor systems 212 or 214) to actively scan the environment near the front of vehicle 200. The radar may be located, for example, in a location suitable for illuminating an area including the forward travel path of vehicle 200 without being obstructed by other features of vehicle 200. For example, the radar may be embedded in and / or mounted on or near the front bumper, front headlights, cowl, and / or hood, etc. Furthermore, one or more additional radars may be positioned to actively scan the sides and / or rear of vehicle 200 for the presence of radio wave reflective objects, such as by including such devices in or near the rear bumper, side panels, rocker panels, and / or chassis, etc.

[0079] Vehicle 200 may include one or more cameras. For example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The cameras may be light-sensitive devices, such as still cameras, video cameras, thermal imaging cameras, stereo cameras, or night vision cameras, configured to capture multiple images of the environment surrounding vehicle 200. To this end, the cameras may be configured to detect visible light or, additionally or alternatively, to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The cameras may be two-dimensional detectors, optionally with a three-dimensional spatial sensitivity range. In some embodiments, the cameras may include range detectors configured to generate two-dimensional images, for example, indicating the distance from the camera to points in the environment. To this end, the cameras may use one or more range detection techniques. For example, the camera may provide range information by using structured light techniques, in which the vehicle 200 illuminates objects in its surrounding environment with a predetermined light pattern, such as a glider or checkerboard pattern, and uses the camera to detect reflections of the predetermined light pattern from the surrounding environment. Based on distortion aberrations in the reflected light pattern, the vehicle 200 may measure the distance to a point on the object. The predetermined light pattern may include infrared light or other wavelengths of radiation suitable for such measurements. In some embodiments, the camera may be mounted on the inside of the windshield of the vehicle 200. Specifically, the camera may be positioned to capture images from a forward-looking perspective relative to the orientation of the vehicle 200. Other camera mounting locations and viewing angles may also be used, either inside or outside the vehicle 200. Additionally, the camera may have associated optics operable to provide an adjustable field of view. Furthermore, the camera may be mounted on the vehicle 200 using a movable mount to change the camera's pointing angle, such as via a pan / tilt mechanism.

[0080] Vehicle 200 may also include one or more acoustic sensors used to sense the vehicle's environment (e.g., one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, or microelectromechanical systems (MEMS) microphones) used to sense acoustic waves (i.e., pressure differentials) in a fluid (e.g., air) in the environment surrounding vehicle 200. Such acoustic sensors may be used to identify sounds in the environment (e.g., sirens, human speech, animal sounds, or alarms) on which a control strategy for vehicle 200 may be based. For example, if an acoustic sensor detects a siren (e.g., an ambulance siren or a fire engine siren), vehicle 200 may slow down and / or navigate to the curb.

[0081] Although not shown in FIGS. 2A-2E , vehicle 200 may include a wireless communication system (e.g., similar to and / or in addition to wireless communication system 146 of FIG. 1 ). The wireless communication system may include a wireless transmitter and a wireless receiver that may be configured to communicate with devices external or internal to vehicle 200. Specifically, the wireless communication system may include a transceiver configured to communicate with other vehicles and / or computing devices, for example, in a vehicle communication system or road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transportation systems.

[0082] Vehicle 200 may include one or more other components in addition to or instead of those shown. The additional components may include electrical or mechanical functionality.

[0083] A control system of vehicle 200 may be configured to control vehicle 200 according to a control strategy from among a plurality of possible control strategies. The control system may be configured to receive information from sensors (on or off vehicle 200) coupled to vehicle 200, modify the control strategy (and associated driving behavior) based on the information, and control vehicle 200 according to the modified control strategy. The control system may be further configured to monitor the information received from the sensors and continuously evaluate driving conditions, and may be configured to modify the control strategy and driving behavior based on changing driving conditions. For example, the path taken by the vehicle from one destination to another may be modified based on driving conditions. Additionally or alternatively, speed, acceleration, turning angle, following distance (i.e., the distance of the current vehicle to the vehicle ahead), lane selection, etc. may all be modified in response to changing driving conditions.

[0084] As noted above, in some embodiments, vehicle 200 may take the form of a van, although alternative configurations are possible and contemplated in this disclosure. Accordingly, FIGS. 2F-2I illustrate an embodiment in which vehicle 250 takes the form of a semi-truck. For example, FIG. 2F illustrates a front view of vehicle 250, and FIG. 2G illustrates an isometric view of vehicle 250. In embodiments in which vehicle 250 is a semi-truck, vehicle 250 may be configured to include tractor portion 260 and trailer portion 270 (illustrated in FIG. 2G). FIGS. 2H and 2I provide side and top views, respectively, of tractor portion 260. Similar to vehicle 200 illustrated above, vehicle 250 illustrated in FIGS. 2F-2I may also be configured to include various sensor systems (e.g., similar to sensor systems 202, 206, 208, 210, 212, and 214 shown and described with reference to FIGS. 2A-2E). In some embodiments, vehicle 200 of Figures 2A-2E may be configured to include only a single copy of some sensor systems (e.g., sensor system 204), while vehicle 250 illustrated in Figures 2F-2I may be configured to include multiple sensor systems (e.g., sensor systems 204A and 204B as illustrated).

[0085] While the figures and general description may refer to a given vehicle form (e.g., semi-truck vehicle 250 or vehicle 200 shown as a van), it will be understood that the embodiments described in this disclosure are equally applicable in various vehicle contexts (e.g., with modifications employed to account for the vehicle form factor). For example, sensors and / or other components described or illustrated as being part of vehicle 200 may also be used in semi-truck vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0086] FIG. 2J illustrates various sensor fields of view (e.g., associated with vehicle 250, described above). As described above, vehicle 250 may include multiple sensors / sensor units. The various sensor locations may correspond to the sensor locations disclosed in FIGS. 2F-2I, for example. However, in some examples, sensors may be located elsewhere. To simplify the illustration, sensor location reference numbers are omitted from FIG. 2J. For each sensor unit of vehicle 250, FIG. 2J illustrates a representative field of view (e.g., fields of view labeled 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The field of view of a sensor may include an angular region (e.g., azimuth region and / or elevation region) in which the sensor may detect objects.

[0087] FIG. 2K illustrates beam steering for a sensor of a vehicle (e.g., vehicle 250 shown and described with reference to FIGS. 2F-2J) according to an example embodiment. In various embodiments, the sensor unit of vehicle 250 may be radar, lidar, sonar, or the like. Additionally, in some embodiments, the sensor may be configured to perform a scan within the sensor's field of view during sensor operation. Various different scan angles for the example sensor are shown as regions 272, each representing an angular region in which the sensor is operating. The sensor may be configured to periodically or repeatedly change the region in which it is operating. In some embodiments, multiple sensors may be used by vehicle 250 to measure region 272. In other examples, other regions may be included. For example, one or more sensors may measure aspects of vehicle 250's trailer 270 and / or the region ahead of vehicle 250.

[0088] At some angles, the sensor's operating area 275 may include the rear wheels 276A, 276B of the trailer 270. Thus, the sensor may measure the rear wheels 276A and / or 276B during operation. For example, the rear wheels 276A, 276B may reflect lidar or radar signals transmitted by the sensor. The sensor may receive signals reflected from the rear wheels 276A, 276B. Thus, the data collected by the sensor may include data from reflections from the wheels.

[0089] In some instances, such as when the sensor is a radar, reflections from the rear wheels 276A, 276B may appear as noise in the received radar signal. As a result, the radar may operate with an enhanced signal-to-noise ratio in instances where the rear wheels 276A, 276B direct the radar signal away from the sensor.

[0090] 3 is a conceptual illustration of wireless communication between various computing systems associated with an autonomous or semi-autonomous vehicle, according to an example embodiment. In particular, wireless communication may occur between a remote computing system 302 and the vehicle 200 over a network 304. Wireless communication may also occur between a server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0091] Vehicle 200 may correspond to various types of vehicles capable of transporting passengers or objects between locations and may take the form of any one or more of the vehicles discussed above. In some examples, vehicle 200 may be configured to operate in an autonomous or semi-autonomous mode, where a control system uses sensor measurements to enable vehicle 200 to safely navigate between destinations. When operating in an autonomous or semi-autonomous mode, vehicle 200 may be configured to navigate with or without a passenger. As a result, vehicle 200 may pick up and drop off passengers between desired destinations.

[0092] Remote computing system 302 may represent any type of device associated with remote assistance technologies, including but not limited to those described in this disclosure. In examples, remote computing system 302 may represent any type of device configured to (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can then observe the information and input a response related to the information, and (iii) transmit the response to vehicle 200 or to another device. Remote computing system 302 may take various forms, such as a workstation, a desktop computer, a laptop, a tablet, a mobile phone (e.g., a smartphone), and / or a server. In some examples, remote computing system 302 may be a configuration including multiple computing devices operating together in a network configuration.

[0093] The remote computing system 302 may include one or more subsystems and components similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described in this disclosure. In some embodiments, the remote computing system 302 may also include a user interface, such as input / output devices, such as a touchscreen and speakers. Other implementations are possible.

[0094] Network 304 represents an infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0095] The location of remote computing system 302 can vary within the scope of the embodiments. For example, remote computing system 302 may be at a location remote from vehicle 200 with wireless communication over network 304. In another embodiment, remote computing system 302 may correspond to a computing device within vehicle 200 that is separate from vehicle 200 but that allows a human operator to interact with a passenger or driver of vehicle 200. In some embodiments, remote computing system 302 may be a computing device with a touchscreen that can be operated by a passenger of vehicle 200.

[0096] In some embodiments, operations described in this disclosure as being performed by remote computing system 302 may additionally or alternatively be performed by vehicle 200 (i.e., by any system or subsystem of vehicle 200). In other words, vehicle 200 may be configured to provide remote assistance mechanisms with which a driver or passengers of the vehicle can interact.

[0097] Server computing system 306 may be configured to wirelessly communicate with remote computing system 302 and vehicle 200 (or, in some cases, directly with remote computing system 302 and / or vehicle 200) over network 304. Server computing system 306 may represent any computing device configured to receive, store, sense, and / or transmit information related to vehicle 200 and its remote assistance. As such, server computing system 306 may be configured to perform any operations or portions of such operations described in this disclosure as being performed by remote computing system 302 and / or vehicle 200. Some embodiments of wireless communication related to remote assistance may utilize server computing system 306, while other embodiments may not.

[0098] The server computing system 306 may include one or more subsystems and components similar to or identical to those of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform the various operations described in this disclosure, and a wireless communication interface for receiving information from and providing information to the remote computing system 302 and the vehicle 200.

[0099] The various systems described above may be configured to perform various operations, and these operations and associated features will now be described.

[0100] In keeping with the above discussion, computing systems (e.g., remote computing system 302, server computing system 306, or a computing system local to vehicle 200) may be configured to operate to capture images of the autonomous or semi-autonomous vehicle's surrounding environment using a camera. Generally, at least one computing system can analyze the images and, if possible, control the autonomous or semi-autonomous vehicle.

[0101] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data indicative of objects in an environment surrounding the vehicle (also referred to in this disclosure as "environmental data") in various ways. A sensor system on the vehicle may be configured to provide the environmental data indicative of objects in the surrounding environment. For example, the vehicle may be configured with various sensors, such as cameras, radar, lidar, microphones, radio units, and other sensors. Each of these sensors may be configured to communicate environmental data to a processor within the vehicle regarding information detected by each respective sensor.

[0102] In one example, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned at different orientations. Also, in some embodiments, the camera may be capable of moving to capture images and / or video in different directions. The camera may be configured to store captured images and video in memory for later processing by the vehicle's processing system. The captured images and / or video may be environmental data. Additionally, the camera may be configured to include an image sensor, as described herein.

[0103] In another example, a radar may be configured to transmit electromagnetic signals that are reflected by various objects in the vicinity of the vehicle and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals may enable the radar (or a processing system) to perform various detections of the objects that reflected the electromagnetic signals. For example, the radar may be configured to measure the distance and location of the various reflecting objects. In some embodiments, the vehicle may be configured with two or more radars, with different orientations. The radar may be configured to store the captured information in a memory for later processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0104] In another example, a LIDAR may be configured to transmit electromagnetic signals (e.g., infrared light, such as from a gas or diode laser, or other possible light source) that are reflected by target objects near the vehicle. The LIDAR may be capable of capturing the reflected electromagnetic (e.g., infrared light) signals. The captured reflected electromagnetic signals may enable a ranging system (or processing system) to measure distances to various objects. The LIDAR may also determine the velocity or speed of target objects and store that as environmental data.

[0105] Additionally, in one embodiment, a microphone may be configured to capture audio of the vehicle's surrounding environment. Sounds captured by the microphone may include emergency vehicle sirens and sounds of other vehicles. For example, the microphone may be configured to capture the sound of an ambulance, fire engine, or police siren. The processing system may be capable of detecting that the captured audio signal is indicative of an emergency vehicle. In another embodiment, the microphone may be configured to capture the sound of an exhaust from another vehicle, such as an exhaust from a motorcycle. The processing system may be capable of detecting that the captured audio signal is indicative of a motorcycle. Data captured by the microphone may form part of the environmental data.

[0106] In yet another example, the radio unit may be configured to transmit an electromagnetic signal, which may take the form of a Bluetooth signal, an 802.11 signal, and / or other wireless technology signal. The first electromagnetic radiation signal may be configured to be transmitted via one or more antennas located on the radio unit. Furthermore, the first electromagnetic radiation signal may be configured to be transmitted in one of many different wireless signal modes. However, in some embodiments, it may be desirable to transmit the first electromagnetic radiation signal in a signal mode that solicits responses from devices located in the vicinity of the autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on responses transmitted back to the radio unit and use this communicated information as part of the environmental data.

[0107] In some embodiments, the processing system may be able to combine information from various sensors to further sense the vehicle's surroundings. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to make decisions about the surrounding environment.

[0108] While operating in an autonomous mode (or semi-autonomous mode), the vehicle may be configured to control its operation with little or no human input. For example, if a human operator inputs an address into the vehicle, the vehicle may be able to drive to the specified destination without further human input (e.g., without the human having to steer or touch the brake / accelerator pedals). Furthermore, while the vehicle is operating autonomously or semi-autonomously, the sensor system may be configured to receive environmental data. The vehicle's processing system may be configured to modify control of the vehicle based on the environmental data received from the various sensors. In some examples, the vehicle may be configured to modify the vehicle's speed in response to the environmental data from the various sensors. The vehicle may be configured to modify its speed to avoid obstacles, obey traffic laws, etc. The vehicle may be able to modify its speed or otherwise alter its movement when the processing system on the vehicle detects an object near the vehicle.

[0109] If the vehicle detects an object but does not have sufficient confidence in the object detection, the vehicle may request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object is actually present in the surrounding environment (e.g., whether there is an actual stop sign or whether there is an actual stop sign), (ii) verifying whether the vehicle's detection of the object is correct, (iii) correcting the detection if it is incorrect, and / or (iv) providing supplemental instructions (or modifying current instructions) for an autonomous or semi-autonomous vehicle. Remote assistance tasks also include the human operator providing instructions to control the vehicle's operation (e.g., if the human operator detects that the object is a stop sign, commanding the vehicle to stop at the stop sign), although in some scenarios the vehicle itself may control its own operation based on the human operator's feedback related to the object detection.

[0110] To facilitate this, the vehicle may be configured to analyze environmental data representative of objects in the surrounding environment to detect at least one object having a detection confidence below a threshold. A processor of the vehicle may be configured to detect various objects in the surrounding environment based on the environmental data from the various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, bicyclists, street signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0111] The detection confidence may indicate the likelihood that a detected object is correctly detected or present in the surrounding environment. For example, the processor may be configured to perform object detection of objects in image data in the received environmental data and determine that at least one object has a detection confidence below a threshold based on the object not being detected with a detection confidence above a threshold. If the object detection or object recognition results are inconclusive, the detection confidence may be low or below a set threshold.

[0112] The vehicle may detect objects in the surrounding environment in a variety of ways, depending on the source of the environmental data. In some embodiments, the environmental data may be image or video data coming from a camera. In other embodiments, the environmental data may come from a lidar. The vehicle may be configured to analyze captured image or video data to detect objects in the image or video data. Methods and apparatus may be configured to monitor the image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the surrounding environment based on the radar, audio, or other data.

[0113] In some embodiments, the technique used by the vehicle to detect objects may be based on a known set of data. For example, data related to environmental objects may be stored in a memory located in the vehicle. The vehicle may compare received data with the stored data to make a determination about the object. In other embodiments, the vehicle may make a determination about the object based on the context of the data. For example, construction-related street signs may generally have an orange color. Thus, the vehicle may be configured to detect orange objects located near the side of the road as construction-related street signs. Additionally, as the vehicle's processing system detects objects in the captured data, it may also calculate a confidence score for each object.

[0114] Additionally, the vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object detected. For example, the confidence threshold may be lower for an object that may require a rapid response action from the vehicle, such as the brake lights of another vehicle. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with a detected object is higher than the confidence threshold, the vehicle may be configured to assume that the object was correctly identified and responsively adjust vehicle control based on that assumption.

[0115] If the confidence associated with the detected object is lower than a confidence threshold, the action taken by the vehicle may vary. In some embodiments, the vehicle may be configured to react as if the detected object is present despite the low confidence level. In other embodiments, the vehicle may be configured to react as if the detected object is not present.

[0116] Upon detecting an object in the surrounding environment, the vehicle may also calculate a confidence level associated with the particular detected object. The confidence level may be calculated in various ways, depending on the embodiment. In one example, upon detecting an object in the surrounding environment, the vehicle may be configured to compare environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle may be configured to use a mathematical analysis of the environmental data to calculate a confidence level associated with the object.

[0117] In response to detecting an object with a detection confidence below a threshold, the vehicle may be configured to transmit a request for remote assistance along with the detection of the object to a remote computing system. As discussed above, the remote computing system may take various forms. For example, the remote computing system may be a computing device separate from the vehicle, but with a touchscreen interface or the like for displaying remote assistance information through which a human operator can interact with a passenger or driver of the vehicle. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located outside the vicinity of the vehicle.

[0118] The request for remote assistance may include environmental data, including the object, such as image data, audio data, etc. The vehicle may be configured to transmit the environmental data over a network (e.g., network 304), and in some embodiments, via a server (e.g., server computing system 306), to a remote computing system. A human operator of the remote computing system may then use the environmental data as a basis for responding to the request.

[0119] In some embodiments, when an object is detected as having a confidence below a confidence threshold, the object may be configured to be given a preliminary identification, and the vehicle may be configured to adjust the operation of the vehicle in response to the preliminary identification. Such adjustment of operation may take the form of stopping the vehicle, switching the vehicle to a human-controlled mode, changing the vehicle's velocity (e.g., speed and / or direction), among other possible adjustments.

[0120] In other embodiments, if the vehicle detects an object with a confidence level that meets or exceeds a threshold, the vehicle may still act on the detected object (e.g., stop if the object is detected with high confidence as a stop sign), but may be configured to request remote assistance at the same time (or after) the vehicle acts on the detected object.

[0121] Figure 4 is a block diagram of a system according to an example embodiment. In particular, Figure 4 shows a system 400 including a system controller 402, a radar system 410, a sensor 412, and a controllable component 414. The system controller 402 includes a processor 404, a memory 406, and instructions 408 stored on the memory 406 and executable by the processor 404 to perform functions such as the operations disclosed herein.

[0122] The processor 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., having a single core or multiple cores) and / or one or more special-purpose microprocessors. The one or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also contemplated in this disclosure.

[0123] The memory 406 may be configured to include a computer-readable medium, such as a non-transitory computer-readable medium, which may include, but is not limited to, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid state drive (SSD), hard disk drive (HDD), compact disc (CD), digital motion picture disc (DVD), digital tape, read / write (R / W) CD, R / W DVD, etc.

[0124] The radar system 410 can be used in autonomous or semi-autonomous vehicles for navigation and object detection by using radio waves to detect and measure the distance, speed, and direction of objects in the surrounding environment. The radar system 410 can include one or more radar units, each consisting of a radar transmitter that emits radio waves and a radar receiver that captures the waves reflected from objects. By analyzing the time it takes for the waves to return and their frequency shift (Doppler effect), the radar system 410 can determine the presence, location, and movement of objects.

[0125] In the context of an autonomous or semi-autonomous vehicle, the radar system 410 provides measurements that can assist in navigation and collision avoidance. Radar units are typically mounted on the exterior of the vehicle, such as on the front, rear, and sides. During navigation, the radar system 410 may be configured to continuously emit radio waves in various directions to scan the environment around the vehicle. When the waves encounter an object, they bounce back to the radar receiver, allowing the radar system 410 to analyze the reflected waves and calculate the object's distance, relative speed, and angle. This information can be used by the vehicle's control system to detect and react to obstacles, pedestrians, vehicles, and other potential hazards in its path, and adjust the vehicle's trajectory accordingly. By providing real-time data about the surrounding environment, the radar system 410 can enhance the vehicle's perception capabilities and contribute to safer and more reliable navigation.

[0126] The radar system 410 offers operational advantages over other types of sensors, such as cameras and lidar, in some aspects. Radar can perform well in adverse weather conditions, such as rain, fog, or dust, where other sensors have limitations. In particular, the radio waves emitted by the radar system 410 can penetrate such adverse conditions and provide reliable object detection. This makes radar particularly useful for improving the robustness and safety of autonomous or semi-autonomous vehicles in various weather scenarios. In addition, radar is also excellent at detecting the velocity and relative speed of nearby objects, which is useful for assessing the movement of surrounding vehicles, pedestrians, and other obstacles. By providing accurate speed information, the radar system 410 enables the vehicle (or the vehicle's driver) to make informed decisions about potential collision risks and adjust its behavior accordingly. In some cases, the radar system 410 can also provide a longer measurement range and a wider field of view compared to other sensors coupled to the vehicle.

[0127] Similarly, system controller 402 can sense characteristics of system 400 and / or the surrounding environment using output from radar system 410 and sensors 412. For example, sensors 412 may include one or more of a GPS, an IMU, an image capture device (e.g., a camera), a light sensor, a thermal sensor, one or more lidar devices, and other sensors that indicate parameters related to system 400 and / or the surrounding environment. Radar system 410 is depicted as separate from sensors 412 for illustrative purposes, and in some examples may be considered part of or as sensors 412.

[0128] Based on characteristics of the system 400 and / or the surrounding environment sensed by the system controller 402 based on outputs from the radar system 410 and the sensors 412, the system controller 402 may be configured to control the controllable components 414 to perform one or more actions. For example, the system 400 may correspond to a vehicle, in which case the controllable components 414 may include the vehicle's braking system, turning system, and / or acceleration system, and the system controller 402 may be configured to modify aspects of these controllable components based on characteristics sensed from the radar system 410 and / or the sensors 412 (e.g., when the system controller 402 controls the vehicle in an autonomous or semi-autonomous mode). In some embodiments, the radar system 410 and the sensors 412 are also controllable by the system controller 402.

[0129] The radar system 410 and sensors 412 can be used to detect and interpret the surrounding environment during vehicle navigation. However, in some cases, the performance of the radar system 410 and / or sensors 412 can be adversely affected by weather conditions such as rain, fog, or snow. Dynamic weather conditions can affect radar and other types of sensors in a variety of ways. For example, some weather conditions can cause attenuation of the radar signal, leading to a degradation in the quality of the radar image produced by the radar system 410. This degradation, in turn, can lead to overly reliable driving behavior by an autonomous or semi-autonomous vehicle using the radar system 410, as the vehicle system may not accurately perceive the environment due to the degraded radar image.

[0130] To address potential effects on radar data and / or other types of sensor data that may be caused by adverse weather conditions, the disclosed solutions may be implemented to optimize the operation of the radar system 410 and sensor 412 while moving under adverse weather conditions. In some cases, the disclosed solutions may be used to estimate and remove the effects of weather-related attenuation on radar images to restore radar images to what they would appear like in good weather conditions, thereby providing a more accurate representation of the environment for object detection, localization, and identification. In other examples, the disclosed solutions may also be used to estimate and remove incremental weather effects on other types of sensor data, such as lidar data and imagery.

[0131] Additionally, a system may be configured to implement the disclosed techniques to monitor and adjust vehicle control strategies based on changes in the operational range of a radar system and / or other sensor systems. For example, if environmental conditions reduce the sensing range of a radar system and / or another sensor system, the vehicle system may be configured to reduce the vehicle's speed to prevent the vehicle from driving beyond its sensor coverage. In other examples, the vehicle system may be configured to rely less (or not at all) on a sensor system or particular sensor after determining that the sensor's operational range or performance is degraded in the current environment.

[0132] FIG. 5 is a conceptual diagram of a system for mitigating the effects of weather-related attenuation on vehicle radar data and other types of sensor data. In some embodiments, the system may be configured to enhance the safety and reliability of autonomous and / or ADAS operation by evaluating the performance of radar and / or other sensor systems during navigation under adverse environmental conditions. The system may enable the vehicle system to detect changes in the quality of radar and other types of sensor data and determine whether these detected changes should trigger some type of corrective action by the vehicle system. In some cases, the system may be configured to detect that current weather conditions temporarily reduce the maximum sensing range of one or more sensors. In response, the system may be configured to reduce the vehicle's speed or take other corrective action due to the reduced sensor performance.

[0133] In some examples, the vehicle system may be configured to estimate and remove the effects of weather-related attenuation on the radar image to restore the radar image to how it would appear in good weather conditions. The system may perform similar techniques to remove the effects of weather conditions from other types of sensor data. In this way, the system may be configured to generate output that provides a more accurate representation of the vehicle's environment, which the vehicle system may use for object detection and tracking, as well as to make more informed navigation decisions. During adverse weather conditions, the corrected sensor data generated by the system may be used to help prevent overreliable autonomous vehicle driving behavior and potentially reduce the risk of accidents.

[0134] In the example shown in FIG. 5 , the system may be implemented in a vehicle 500 and may include a computing device 502, a radar system 504, sensors 506, a control system 508, and memory 510. Memory 510 includes a weather estimation model 512 and a baseline model 513. In the example, baseline model 513 may be used as a weather loss model and includes a radar map 514, a predicted RCS value 516, and a predicted clutter-to-noise profile 518. As part of the system's operation, vehicle 500 may communicate with a remote computing system 520 and a vehicle 522 via wireless communication 524. The elements of the system are shown for illustrative purposes, and other examples may include more or fewer elements. For example, remote computing system 520 and vehicle 522 may represent any number of computing systems and vehicles, respectively.

[0135] Vehicle 500 represents any type of mobile machine designed to transport passengers from one location to another. For example, vehicle 500 may be a car, truck, bus, semi-truck, construction vehicle, or another type of mobile machine. In some cases, vehicle 500 may be a robotic device or another specialized vehicle. In some examples, vehicle 500 is an autonomous or semi-autonomous vehicle that uses sensor data from radar system 504 and / or sensors 506 to understand its surroundings and safely navigate between locations.

[0136] Computing device 502 may represent one or more computing devices configured to perform operations of vehicle 500, including methods described herein. In some aspects, computing device 502 includes a processor and memory that stores instructions that, when executed by the processor, cause computing device 502 to perform various operations. The memory may correspond to memory 510 or another memory specific to computing device 502. The operations performed by computing device 502 may include estimating weather conditions in the environment of vehicle 500 using weather estimation model 512, detecting weather-related attenuation effects on a radar image based on the estimated weather conditions, removing the weather-related attenuation effects to correct the radar image, and providing the corrected radar image to control system 508 of vehicle 500. Computing device 502 may, in some cases, perform variations of these operations.

[0137] During navigation by the vehicle 500, the computing device 502 may be configured to receive radar data generated by the radar system 504, which may use one or more radars to generate radar data representative of the vehicle 500's environment. The radar data may be processed in various formats, such as a radar image. In some embodiments, the radar system 504 may output radar data in a range-doppler map (RDM), which is a two-dimensional representation that plots the distance of objects from a radar sensor against their relative velocity. The range dimension indicates how far an object is, while the Doppler dimension indicates whether the object is moving toward or away from the radar sensor and at what speed. The RDM may be useful for detecting and tracking moving objects, such as other vehicles (e.g., vehicle 522). Additionally, the radar system 504 may output radar data in a range-angle image, which provides a two-dimensional view in which one axis represents the range to the target and the other represents the angle of arrival of the radar signal. Range angle images can be useful in determining the azimuth or bearing of objects relative to the vehicle, which can help create a spatial map of the environment around the vehicle.

[0138] Computing device 502 may also be configured to receive sensor data from one or more sensors 506. In some embodiments, sensors 506 may include various types of sensors, such as cameras, lidar, temperature sensors, wind sensors, humidity sensors, and / or ultrasonic sensors. Sensors 506 may be configured to detect various environmental conditions, such as temperature, humidity, wind speed, or visibility. Sensors 506 may also be used to detect weather conditions, such as rain, fog, or snow. Sensor data output by sensors 506 may be used to supplement radar images from radar system 504 and provide additional information about the environment of vehicle 500.

[0139] The vehicle 500 also includes a control system 508 that can manage and regulate the operation of the vehicle 500. In some embodiments, the control system 508 and the computing device 502 may be part of a computing system used to implement various operations of the vehicle 500. The control system 508 may be configured to perform a wide range of functions, from basic vehicle operation to complex decision-making processes for autonomous or semi-autonomous navigation. In some embodiments, the control system 508 may be configured to perform functions such as steering control, acceleration and braking, stability and traction control, navigation and path planning, and obstacle detection and avoidance. Components within the control system 508 may include sensors, actuators, an electronic control unit (ECU), and a communication network. In some embodiments, the control system 508, in cooperation with the computing device 502, may process data from various sensors to perceive the environment, make complex decisions based on the data, and control the movement of the vehicle 500 accordingly. The control system 508 may also be configured to continuously monitor performance and safety parameters to ensure the vehicle 500 operates within safe limits. The control system 508 may be configured to rely on accurate sensor data to make informed decisions. In this manner, corrected radar imagery compensated for weather-related effects may be a useful input to the control system 508.

[0140] Computing device 502 may be configured to use memory 510, which may be used to store data and models. For example, memory 510 may store weather estimation model 512 and baseline model 513. In an embodiment, baseline model 513 includes radar map 514, predicted RCS values ​​516, and predicted clutter-to-noise profile 518. Weather estimation model 512 may be configured to be used by computing device 502 to estimate weather conditions in the environment of vehicle 500. Radar map 514, predicted RCS values ​​516, and predicted clutter-to-noise profile 518 may be used by computing device 502 to estimate the impact weather effects on radar data, thereby enabling computing device 502 to correct the radar data by removing weather-related attenuation effects.

[0141] The computing device 502 may be configured to implement the disclosed methods to calculate a real-time estimate of radar sensitivity degradation. The real-time estimate may enable immediate adjustments to the radar system 504 or the control strategy of the vehicle 500 to increase the safety and reliability of autonomous vehicle operation. For example, if radar sensitivity is determined to be severely degraded due to heavy rain, the control system 508 may be configured to slow the vehicle 500 or implement other preventative measures. Reducing the speed of the vehicle 500 may help prevent the vehicle 500 from driving too fast for the sensor coverage of the surrounding environment. In some cases, weather conditions may reduce the maximum sensing range of the radar system 504 and / or the sensors 506, which may trigger the computing device 502 and / or the control system 508 to reduce the speed of the vehicle 500 to accommodate the reduced driving capabilities.

[0142] The vehicle 500 may be configured to communicate wirelessly 524 with the remote computing system 520 and the vehicle 522. The wireless communication 524 may enable data exchange or collaborative processing between the vehicle 500, the remote computing system 520, and the vehicle 522. For example, the vehicle 500 may be configured to share radar imagery, sensor data, or corrected radar imagery with the remote computing system 520 and / or the vehicle 522. Similarly, the vehicle 500 may be configured to receive radar imagery, sensor data, or corrected radar imagery from the remote computing system 520 or the vehicle 522. This collaborative processing may enhance the accuracy and reliability of the radar image correction process.

[0143] Weather estimation model 512 represents a computational tool or algorithm designed to estimate current weather conditions in the environment of vehicle 500 based on various input data sources. In some embodiments, vehicle 500 may utilize weather estimation model 512 in conjunction with onboard sensors (e.g., radar system 504 and sensors 506) to accurately estimate weather conditions in the environment. For example, weather estimation model 512 may be trained to map raw backscatter values ​​in a radar image to estimated precipitation rates. Weather estimation model 512 may similarly map raw backscatter values ​​in a radar image to estimated precipitation fall rates associated with snow, hail, or another type of weather.

[0144] In some examples, the weather estimation model 512 may integrate sensor data into predictive modeling to ensure the vehicle 500 understands its surroundings in real time. Sensors 506, which may include LIDAR, optical cameras, temperature sensors, and other chemical sensors such as hygrometers and thermometers, may be used to continuously collect environmental data providing information regarding humidity and visibility. The weather estimation model 512 may be configured to analyze the sensor data to identify patterns or signatures corresponding to specific weather conditions, such as rain, fog, or snow. In some cases, the weather estimation model 512 may be configured to use machine learning techniques, statistical analysis, and / or physical principles to interpret sensor signal characteristics and estimate weather conditions.

[0145] As an example, the weather estimation model 512 may be configured to estimate the amount of rainfall in the environment by examining the attenuation and scattering patterns of the radar signal caused by rainfall. In some instances, the weather estimation model 512 may also incorporate data from additional sensors 506 measuring visibility, humidity, and / or temperature to refine its estimate. The output of the weather estimation model 512 may then be used to detect weather-related attenuation effects on the radar image, which can be corrected to provide a more accurate representation of the environment for the control system 508.

[0146] In another example, the weather estimation model 512 may be configured to estimate rainfall amount or raindrop size by analyzing the reflectivity of a radar signal. The reflectivity of the signal may be affected by the size and concentration of the raindrops, with larger, denser droplets reflecting more radar energy and producing a stronger return signal. In some instances, the radar system 504 may be configured to transmit and receive radar pulses at different polarizations and detect variations in reflectivity between these polarizations, which can provide insight into the shape and size of the droplets, as raindrops tend to flatten as they increase in size. Additionally, the weather estimation model 512 may factor in Doppler shift in the radar signal, which provides information about the velocity of the raindrops, which may further aid in estimating the rainfall rate.

[0147] In some cases, the weather estimation model 512 may also be configured to factor the attenuation and scattering patterns of the radar signal as affected by interactions with raindrops. These interactions vary with droplet size, which may allow the weather estimation model 512 to estimate size distributions within the rain. For increased accuracy, the weather estimation model 512 may incorporate environmental data (e.g., ambient temperature and humidity) that may affect droplet formation and size. By integrating these analytical techniques and environmental data, the weather estimation model 512 may provide detailed estimates of rainfall amount and droplet size, which may then be used to calibrate the sensitivity of the radar system 504 or to correct radar imagery to enable more accurate object detection and range under adverse weather conditions.

[0148] In some embodiments, the weather estimation model 512 may also be configured to analyze the effects of snow or high winds on radar signal characteristics. For snow, the weather estimation model 512 may be configured to analyze the volume scattering characteristics of the radar signal because snow flakes may have different scattering properties compared to rain due to their shape, size, and low density. The weather estimation model 512 may be configured to distinguish between snow types (e.g., light, fluffy snow and wet, heavy snow) by examining the amplitude and phase shift of the received radar signal. In some cases, the weather estimation model 512 may be configured to detect wind speed and direction by evaluating Doppler shift variations in the radar signal caused by moving debris or vegetation agitated by the wind.

[0149] The generation and calibration of the weather estimation model 512 can help ensure that predictions made by the weather estimation model 512 accurately reflect real-world weather conditions. The initial stage of calibration can occur in a controlled environment, such as a laboratory, where specific weather conditions can be artificially created and standardized. To calibrate the rainfall estimator aspects of the weather estimation model 512, various intensities and types of rainfall can be simulated to observe signal attenuation and reflectivity. The process can include generating conditions ranging from light rain to heavy rain, allowing the radar to capture a wide range of data on raindrop size, distribution, and resulting radar signal characteristics. The goal of this process is to establish a comprehensive data set representing the radar's performance under various precipitation scenarios.

[0150] In some cases, the calibration process may consist of preparing the weather estimation model 512 using known rainfall amounts and known target RCS values. This calibration process may help ensure that the rainfall estimation model accurately estimates rainfall amounts and RCS values ​​from radar imagery under various weather conditions. In some cases, the calibration process may be repeated periodically or as needed to maintain the accuracy of the rainfall estimation model.

[0151] In addition to rain, the calibration process may prepare the weather estimation model 512 for other weather conditions, such as snow and cloudy weather. For snow, the calibration process may include simulating how different snow densities and types affect radar signals. For example, the process may test a range from fine, powdery snow to wet, heavy flakes. Calibrating the weather estimation model 512 for cloudy weather may include analyzing the attenuation and scattering effects caused by water droplets suspended in the air, which may vary in density and distribution. Testing may allow for the collection of data showing how these conditions change the return strength and clarity of received radar signals. To calibrate the weather estimation model 512 to analyze wind conditions, the calibration setup may include moving objects and vegetation to replicate the dynamic environment often experienced by a vehicle.

[0152] Following controlled calibration, the weather estimation model 512 may be further refined through real-world operational testing. For example, as a vehicle 500 equipped with the radar system 504 navigates different environments, the radar system 504 may be configured to continuously collect radar imagery along with actual weather data. This real-time data acquisition by the vehicle 500 and other vehicles (e.g., vehicle 522) may allow the weather estimation model 512 to adapt and fine-tune its algorithms, enhancing its predictive capabilities for a wide range of weather conditions. The weather estimation model 512 may be configured to use advanced machine learning techniques to analyze this data and adjust its parameters to more accurately reflect the effects of weather on the radar signal.

[0153] Additionally, to enhance the calibration process, the weather estimation model 512 may also be configured to incorporate external data sources, such as weather stations or collaborative data sharing from a network of vehicles. For example, the vehicle 500 may be configured to receive additional data from a remote computing system 520 and / or the vehicle 522. The additional data may provide broader context for updating the weather estimation model 512, which may enable the weather estimation model 512 to be modified to understand local weather patterns and their particular effect on radar performance.

[0154] In some examples, the weather estimation model 512 may be generated onboard the onboard vehicle 500 through the use of the radar system 504, sensors 506, and computing power. For example, the radar system 504 and sensors 506 may be used to collect data regarding local weather conditions such as precipitation, temperature, humidity, and visibility. The collected data may then be processed by the computing device 502, which may use machine learning algorithms to analyze the collected information and generate (or update) the weather estimation model 512 based on the environment of the vehicle 500. The weather estimation model 512 may be continuously updated as new data is collected, which may allow for real-time adjustments.

[0155] In other examples, the weather estimation model 512 may be received from a remote computing system 520, which may be a centralized server or cloud-based platform that aggregates and processes large amounts of environmental and weather data from various sources, including satellite imagery, weather stations, and a network of vehicles. The remote computing system 520 may use the data from the various sources to develop a comprehensive and updated weather estimation model 512 that may account for broader weather patterns and regional variations. The weather estimation model 512 may then be transmitted to individual vehicles via wireless communication 524 over a wireless communications network. This approach may enable the vehicle 500 and other vehicles (e.g., vehicle 522) to receive an improved model based on local sensor data and extensive meteorological insights, providing more accurate and reliable assessments of weather conditions.

[0156] In some cases, the weather estimation model 512 may be received by the vehicle 500 based on the geographic location or the route of the vehicle 500. For example, the computing device 502 may receive a version of the weather estimation model 512 that is specific to a particular geographic location or a route the vehicle 500 is expected to traverse. The location- or route-specific weather estimation model may take into account specific weather patterns or conditions associated with the geographic location or route and may provide a more accurate weather estimation for the environment of the vehicle 500. In some cases, the geographic location may be based on a Global Positioning System (GPS) signal received by the vehicle 500. In other cases, the vehicle's route may be determined based on the vehicle's 500 navigation system. Using the geographic location or route to receive the weather estimation model may increase the accuracy of the weather estimation and improve correction of radar images.

[0157] To correct the sensor data, computing device 502 may be configured to detect weather-related attenuation effects on the sensor data. For example, computing device 502 may be configured to use estimated weather conditions to determine how to correct incoming radar images generated by radar system 504. Predicting weather-related attenuation effects on the sensor data may include the use of one or more baseline models 513. In the example shown in FIG. 5 , baseline models 513 include radar map 514, expected RCS values ​​516, and expected clutter-to-noise profile 518. In other embodiments, baseline models 513 may include other data that can be used to detect any weather-related effects on radar data or other types of sensor data.

[0158] Generally, a baseline model is a reference framework or set of data that represents normal or expected conditions for sensor data (e.g., radar data) when weather is evident, described in this disclosure as favorable weather conditions. Each baseline model can serve as a standard against which current sensor data can be compared to identify anomalies or deviations caused by weather-related factors. The baseline model 513 can be used to detect weather-related attenuation by providing a benchmark for how radar signals interact with the environment and various objects under ideal conditions.

[0159] The radar map 514 corresponds to a model that shows expected radar returns in favorable weather conditions. In particular, the radar map 514 may provide a pre-established baseline for how a radar image of the surrounding environment, including roads, buildings, and other static objects, typically appears without any weather interference. In some cases, a common radar map may be used for all vehicles. This common radar map may be a standardized map used across a fleet of vehicles. In other instances, each vehicle may develop its own radar map. This individual radar map may be specific to the vehicle and developed based on the vehicle's own radar data and experience.

[0160] The expected RCS values ​​516 may provide a reference for the RCS of objects in the vehicle's 500 environment and may be organized based on object type, such as vehicles, buildings, traffic signs, pedestrians, etc. RCS is a measure of how detectable an object is by radar. Thus, the expected RCS values ​​516 may provide a reference for the radar signatures of different objects in the absence of weather effects, which may be used to detect the impact of adverse weather on radar images.

[0161] The expected clutter-to-noise profile 518 may represent the expected levels of clutter (unwanted radar sounds from stationary objects) and noise (random and background electromagnetic interference) in the radar signal under clear weather conditions. In some cases, the expected clutter-to-noise profile 518 may represent the expected levels of clutter and noise under other weather conditions, which may be used for comparison with radar images recently generated by the radar system 504.

[0162] By comparing real-time radar data to one or more baseline models 513, discrepancies due to weather, such as increased noise or signal attenuation due to rain, fog, or snow, can be identified. The system can then apply corrections to the radar imagery to account for these weather-related effects, thereby maintaining the accuracy and reliability of the radar system's 504 perception of the environment. In some instances, the corrected radar imagery can be provided to a control system 508 that controls the operation of the vehicle 500 using the corrected radar imagery. This can increase the safety and reliability of autonomous vehicle operation by providing more accurate radar data regardless of weather conditions.

[0163] In some embodiments, computing device 502 may be configured to compare current radar data products or summary metrics at a particular location with historical data that varies from unresolved to heavily resolved. The comparison may allow computing device 502 to estimate a level of degradation of radar data from one or more radars from radar system 504, which may be used to correct the radar image. The corrected radar image may then be provided to control system 508 of control vehicle 500 using the corrected radar image.

[0164] Removing weather-related attenuation effects may include adjusting the observed radar data based on the identified differences. For example, if the observed radar data shows lower signal strength than one or more baseline models 513 due to rain attenuation, the signal strength may be increased to match the expected baseline. Similarly, if the observed radar data shows higher noise levels than one or more baseline models 513 due to snow, the noise levels may be decreased to match the baseline expectations. These adjustments may help restore the radar image to how it appears under favorable weather conditions.

[0165] In some configurations, removing weather-related attenuation effects may include applying a correction factor to the observed radar data. The correction factor may be determined based on the identified discrepancies and may be used to adjust the observed radar data to match prior expectations. The correction factor may be applied to various aspects of the observed radar data, such as signal strength, noise level, RCS value, or clutter-to-noise profile. Applying the correction factor may help to correct the radar image, which may increase the accuracy of the representation of the vehicle 500's environment.

[0166] In some embodiments, the corrected radar image may be provided to a control system 508 of the vehicle 500, which uses the corrected radar image to control operations such as steering, acceleration, braking, or obstacle avoidance. By using the corrected radar image, the control system 508 may make more accurate decisions, potentially enhancing the safety and reliability of autonomous vehicle operation.

[0167] In some cases, a representation of the detected object may be displayed relative to the vehicle 500. The display may be provided on a display interface of the vehicle 500, such as a dashboard display, a head-up display, or a mobile device connected to the vehicle 500. The representation of the object may include various information about the object, such as the type of object, the distance of the object from the vehicle, the velocity of the object relative to the vehicle, or the direction of the object relative to the vehicle 500. The representation of the object may be updated in real time based on the corrected radar imagery, providing the driver and passengers of the vehicle 500 or the control system 508 with up-to-date information about the environment of the vehicle 500. This display of the representation of the object relative to the vehicle may increase the situational awareness of the driver or control system 508 and improve the decision-making process for driving the vehicle.

[0168] In some examples, the weather estimation model 512 and the baseline model 513 may be combined into a single model for use by the vehicle 500 as well as other vehicles (e.g., vehicle 522). In this way, machine learning techniques within the model can make predictions based on historical data and current observations and adjust vehicle system operations to the detected conditions. For example, if the model predicts heavy rain, the vehicle 500 may use the information to adjust its radar system 504 to account for signal attenuation caused by rain and to adapt its driving strategy. This may include maintaining a safe stopping distance in wet conditions and slowing down to avoid driving beyond the vehicle's ability to see its surroundings. In some cases, the vehicle 500 may be configured to alter its route to avoid areas of severe weather. Additionally, the computing system 502 may use the weather data to adjust sensor parameters for better performance, such as changing radar sensitivity to improve object detection in cloudy conditions or increasing or decreasing the power used by a sensor or sensor system, among other operations.

[0169] 6 is a flowchart of a method 600 for mitigating weather-related attenuation effects on radar imagery. Method 600 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 602, 604, 606, 608, and 610. While the blocks are illustrated in sequential order, these blocks may, in some cases, be performed in parallel and / or in a different order than described in this disclosure. Additionally, various blocks may be combined into fewer blocks, split into additional blocks, and / or eliminated based on a desired implementation.

[0170] Additionally, for method 600 and other processes and methods disclosed in this disclosure, a flowchart illustrates the functionality and operation of one possible implementation of the present embodiment. In this regard, each block may represent a module, segment, or portion of program code, which includes one or more instructions executable by a processor to perform specific logical functions or steps in the process. The program code may be stored on any type of computer-readable medium or memory, such as, for example, a storage device including a disk or hard drive. Various types of computing systems can be used to implement method 1100 and other methods described in this disclosure.

[0171] In some embodiments, a computing system implementing the disclosed techniques may include one or more processors, which may be part of the same or different computing devices. The computing system may be located on a vehicle and may be configured to communicate with one or more remotely located computing devices. In some embodiments, a specialized type of processor may perform method 600. For example, one or more application specific integrated circuits (ASICs) and / or field programmable gate arrays (FPGAs) may perform method 600. As an example, an ASIC or FPGA may perform one or more blocks of method 600. In some cases, the computing system may be configured to perform method 600 for different radars to compare the relative performance of the radars.

[0172] At block 602, the method 600 includes receiving a radar image corresponding to an environment of the vehicle. The radar image may be generated by a radar system, such as radar system 504 shown in Figure 5. The radar image may convey information indicative of the location and movement of various objects in the vehicle's environment, as well as weather conditions.

[0173] At block 604, the method 600 includes estimating weather conditions of the environment using a weather estimation model. The weather estimation model may be configured to estimate weather conditions in the environment based on information from one or more sources. For example, the weather estimation model may be used to analyze radar imagery to estimate environmental conditions, such as the rate at which precipitation falls. In some cases, the computing system may be configured to incorporate sensor data from other sensors to sense conditions of the environment. For example, the computing system may be configured to estimate weather conditions of the environment by using sensor data from one or more non-radar sensors, such as lidar, cameras, and other types of sensors. The computing system may then estimate weather conditions of the environment based at least in part on the sensor data.

[0174] The weather estimation model used by the computing system may include a rainfall estimator. The rainfall estimation may be configured to map raw backscatter values ​​in a radar image to a rainfall estimate for the vehicle's environment. In some cases, the rainfall estimation model may be trained in an enclosed radar chamber with synthetic rain generated with known rainfall amounts falling from a ceiling.

[0175] At block 606, the method 600 includes detecting weather-related attenuation effects on the radar image. The weather-related attenuation effects on the radar image may be detected based on estimated weather conditions. The attenuation effects may represent degradation of the quality of the radar image due to weather or other environmental conditions, such as rain, fog, or snow.

[0176] At block 608, the method 600 includes removing weather-related attenuation effects to correct the radar image. Removing the attenuation effects may involve adjusting the radar data based on the detected attenuation effects, thereby restoring the radar image to how it would appear under favorable weather conditions.

[0177] In some examples, removing weather-related attenuation to correct the radar image includes using a weather loss model to map the rainfall estimate to an aperture and atmospheric pressure loss in the return signal strength corresponding to the radar image, and performing one or more corrective actions based on the aperture and atmospheric pressure loss in the return signal. For example, the computing system may be configured to provide a signal to a control system to reduce the speed of the vehicle. In some cases, the computing system may be configured to determine an offset based on the aperture and atmospheric pressure loss in the return signal strength corresponding to the radar image, and apply the offset to correct the radar image.

[0178] In some examples, removing weather-related attenuation effects to correct the radar image includes comparing estimated weather conditions of the environment to a baseline model corresponding to favorable weather conditions and removing weather-related attenuation effects based on the comparison. The baseline model may include a radar map representing radar images generated during favorable weather conditions. In some instances, the computing system may be configured to receive a set of radar images representing the vehicle's environment, determine that the vehicle's environment has favorable weather conditions, and modify the baseline model based on the set of radar images. The computing system may be configured to receive the baseline model from a remote computing system based on the vehicle's position or route.

[0179] In some cases, the baseline model includes an expected clutter-to-noise profile having a set of expected RCS values ​​for multiple object types and / or a clutter profile based on a noise profile that is the median output of stationary pixels as a function of range and the median output of non-stationary pixels as a function of range.

[0180] In some cases, removing weather-related attenuation effects may include adjusting the observed radar data based on identified differences relative to one or more baseline models. For example, if the observed radar data indicates a lower signal strength than expected as represented by the baseline model due to rain attenuation, the signal strength may be increased to match the expected strength. Similarly, if the observed radar data indicates a higher noise level than previously expected due to snow, the noise level may be reduced to match the previous expectation. These adjustments may help restore the radar image to how it appears under favorable weather conditions.

[0181] In some configurations, removing weather-related attenuation effects may include applying a correction factor to the observed radar data. The correction factor may be determined based on the identified discrepancies and may be used to adjust the observed radar data to match prior expectations. The correction factor may be applied to various aspects of the observed radar data, such as signal strength, noise level, RCS value, or clutter-to-noise profile. Applying the correction factor may help to correct the radar image and provide a more accurate representation of the environment for the autonomous vehicle.

[0182] At block 610, the method 600 includes providing the corrected radar image to a control system of the vehicle. The control system may be configured to use the corrected radar image to control vehicle operation, such as steering, acceleration, braking, or obstacle avoidance. By providing the corrected radar image to the control system, the method may increase the safety and reliability of autonomous vehicle operation by providing more accurate radar data regardless of weather conditions.

[0183] In some embodiments, method 600 may also include detecting a reduction in range of a radar coupled to the vehicle. The computing system may then provide an output to one or more vehicle systems communicating the detected reduction in range to the radar. Additionally, method 600 may include using a weather loss model to decompose radar images received during favorable weather conditions. The decomposed radar images may be used for model calibration testing and / or future training.

[0184] In some examples, the computing system may be configured to compare RCS values ​​in a radar image to a baseline model that represents RCS values ​​of objects in favorable weather conditions. In some cases, the computing system may determine that a difference between the RCS values ​​in the radar image and the RCS values ​​of the baseline model exceeds a threshold difference. The computing system may then be configured to perform one or more corrective actions based on determining that the difference between the RCS values ​​in the radar image and the RCS values ​​of the baseline model exceeds the threshold difference.

[0185] As an example, the computing system may be configured to use a baseline model that includes a radar map representing radar images generated during favorable weather conditions. The computing system may then be configured to compare RCS values ​​in the radar image with RCS values ​​of one or more objects specified by the radar map based on the vehicle's location. In some cases, the computing system may be configured to receive a set of radar images representing the vehicle's environment, determine that the vehicle's environment has favorable weather conditions, and modify the baseline model based on the set of radar images. In some examples, the computing system may be configured to receive the baseline model from a remote computing system and based on the vehicle's location.

[0186] In some examples, a computing system may be configured to use a neural network to remove weather-related attenuation effects based on estimated weather conditions of the environment. The neural network may be trained to mitigate weather-related attenuation effects on radar imagery or other sensor data by utilizing a comprehensive data set that pairs affected sensor readings with their corresponding clean, unattenuated states. During the training phase, the neural network may adjust its internal parameters to minimize discrepancies between its output and the clean data, effectively learning a transformation that corrects for weather-induced distortions. The process may include feature extraction to identify patterns in the data that indicate weather effects, such as reduced signal strength or increased noise due to precipitation. Once trained, the neural network may be configured to apply its learned transformation to new ingress sensor data (e.g., radar imagery) by first estimating current weather conditions and then estimating their impact on signal quality. The neural network may then correct the data by compensating for these estimated effects, thus restoring the data to a representation that more accurately reflects the actual environment without weather interference. The output generated by the neural network may be a corrected set of sensor data that provides a more reliable basis for the vehicle's control systems and other applications, which may enhance the vehicle's ability to navigate and respond to its surroundings under various weather conditions.

[0187] In some aspects, the corrected radar image may be used to detect objects in the vehicle's environment. A computing device coupled to the vehicle may analyze the corrected radar image to identify the presence of an object. This detection may include identifying a radar signature in the corrected radar image that corresponds to the object. The radar signature may be based on various factors, such as the RCS of the object, the distance of the object from the vehicle, the relative velocity of the object with respect to the vehicle, or the angle of arrival of a radar signal reflected by the object. By detecting objects in the corrected radar image, the computing device may provide a more accurate perception of the environment for the autonomous vehicle and may enhance the safety and reliability of autonomous vehicle operation.

[0188] In some cases, a representation of the detected object may be displayed to the vehicle. This display may be provided on a display interface of the vehicle, such as a dashboard display, a head-up display, or a mobile device connected to the vehicle. The representation of the object may include various information about the object, such as the type of object, the distance of the object from the vehicle, the velocity of the object relative to the vehicle, or the direction of the object relative to the vehicle. The representation of the object may be updated in real time based on the corrected radar imagery to provide the driver or the vehicle's control system with up-to-date information about the vehicle's environment. This display of the representation of the object to the vehicle may increase the situational awareness of the driver or the control system and improve the decision-making process for driving the vehicle.

[0189] In some aspects, the weather estimation model may be received based on the geographic location or route of the vehicle. For example, a computing device coupled to the vehicle may receive a weather estimation model specific to a particular geographic location or a route the vehicle is expected to traverse. This location- or route-specific weather estimation model may take into account specific weather patterns or conditions associated with the geographic location or route and may therefore provide a more accurate weather estimation for the vehicle's environment. In some cases, the geographic location may be determined based on a Global Positioning System (GPS) signal received by the vehicle. In other cases, the vehicle's route may be determined based on the vehicle's navigation system. Use of the geographic location or route to receive the weather estimation model may enhance the accuracy of the weather estimation and, as a result, the correction of the radar image.

[0190] In some aspects, the radar image may be used to estimate the size of the raindrops. This estimation may be based on characteristics of the radar signal reflected by the raindrops. For example, larger raindrops may reflect a stronger radar signal compared to smaller raindrops. By analyzing the intensity of the reflected radar signal, the computing device may estimate the size of the raindrops. This information may then be used to further refine the weather estimation model, as the size of the raindrops may affect the level of radar beam attenuation. For example, larger raindrops may cause greater attenuation of the radar beam compared to smaller raindrops. By taking into account the estimated size of the raindrops, the weather estimation model may provide a more accurate estimate of the weather-related attenuation effect on the radar image.

[0191] 7 is a flowchart of a method 700 for mitigating weather-related attenuation effects on radar imagery. Method 700 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 702, 704, 706, and 708. While the blocks are illustrated in sequential order, these blocks may, in some cases, be performed in parallel and / or in a different order than described in this disclosure. Additionally, various blocks may be combined into fewer blocks, split into additional blocks, and / or eliminated based on a desired implementation.

[0192] At block 702, method 700 includes receiving a radar image corresponding to an environment of the vehicle. Alternatively, the computing system may be configured to receive radar data in other formats and / or other types of sensor data in other embodiments.

[0193] At block 704, method 700 includes comparing the radar image to the baseline representation. For example, the computing system may be configured to compare the RCS values ​​of the radar image to the expected RCS values ​​represented by the baseline model. The computing system may be configured to periodically or continuously compare and estimate the RCS values ​​to the baseline model when environmental conditions potentially degrade radar performance. The baseline model may have different forms in embodiments, such as a radar map conveying expected RCS values ​​for stationary objects at various locations. The baseline model may also convey expected RCS values ​​for non-stationary objects, such as other vehicles, motorcycles, and pedestrians. If the comparison indicates that the RCS values ​​in the newly generated radar data differ from the expected RCS values ​​articulated by the baseline model, the computing system may proceed to estimate losses caused by the environmental conditions.

[0194] At block 706, the method 700 includes estimating a loss associated with the radar image. The computing system may be configured to estimate the loss associated with the radar image using the comparison. For example, the estimated loss may be calculated based on the difference between the RCS values ​​in the radar data and the corresponding RCS values ​​for the matching object type represented by the baseline model. The estimated loss may be compared to estimated losses derived for other radars to assess individual radar performance. By comparing the detected losses for different radars over the same or multiple time periods, the computing system may further distinguish between aperture losses experienced by each radar and atmospheric losses that affect the entire radar system. Similarly, comparing the detected results for multiple radars may also help identify situations in which a particular radar degrades more significantly than other radars. For example, debris may affect radar performance and trigger the computing system to potentially provide an alert for radar maintenance if suboptimal performance of the radar continues during further testing and after cleaning operations.

[0195] At block 708, the method 700 includes generating an output based on the estimated loss. The output may be a signal for a control system to adjust the vehicle's speed to prevent drive current sensor coverage. The output may also provide information that a vehicle system can use to adjust vehicle control strategies, modify the use of or reliance on particular sensors or sensor systems, and / or take other action (e.g., provide an alert to a passenger or a remote computing device).

[0196] In some cases, the computing system may be configured to perform one or more corrective actions. Corrective actions may include adjusting a vehicle navigation strategy or implementing other actions, such as informing the vehicle system to temporarily rely less (or not at all) on radar data until the computing system determines that weather conditions in the vehicle's environment have changed to more favorable conditions for capturing useful radar data. Similarly, the vehicle may be configured to reduce its speed or adjust other control parameters (e.g., braking distance) in response to determining that environmental conditions have degraded the maximum sensing range of the radar system or another sensor system. Reducing vehicle speed may help prevent the vehicle from driving beyond current visibility.

[0197] In some cases, the output may be configured to indicate that a particular sensor in the sensor system is degrading. For example, debris may cause one radar to perform poorly relative to other radars on the vehicle. As such, the computing system may be configured to avoid use of the radar and / or provide a warning that the radar needs service. The computing system may be configured to periodically check whether the quality of data from the radar is improving before resuming use of the radar.

[0198] In some embodiments, radar is used with control electronics, which may include one or more field programmable gate arrays (FPGAs), ASICs, CPUs, GPUs, and / or TPUs. For example, radar units can generate and receive complex signals that require significant processing. One or more control electronics may be programmed to implement various signal processing algorithms, such as filtering, modulation / demodulation, noise reduction, and digital beamforming. These operations help extract relevant information from received radar signals, enhance signal quality, and improve target detection and tracking. Radar systems often also include converting analog signals to digital form for further processing. The control electronics may include analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) to facilitate these conversions. The control electronics may receive analog signals from the radar sensor, digitize them, and process the digital data for analysis and interpretation.

[0199] The control electronics may also provide capabilities for real-time control and coordination of various radar system components. For example, the control electronics may handle synchronization, timing generation, and system control to ensure proper timing and sequencing of operations within the radar system. This real-time control is important for accurate, synchronized signal transmission and reception. The control electronics may be capable of efficiently handling the large amounts of data generated by the radar system. The control electronics may be capable of implementing data storage, buffering, and data flow management techniques to enable efficient data handling during signal transmission. This may include tasks such as data compression, data packetization, and data routing to ensure smooth and reliable data transmission within the radar system. The control electronics may also integrate various interfaces and protocols required for radar signal transmission, such as processors, memory modules, communications modules, and display units. The control electronics may provide the necessary interface logic to facilitate seamless data exchange between these components, enabling efficient data flow and system integration. The control electronics may also be reconfigurable and customizable to meet specific radar system requirements and adapt to changing operational needs. This allows radar system designers to implement and optimize algorithms and functions specific to their application, improving performance and efficiency.

[0200] The present disclosure is not limited with respect to the specific embodiments described in this application, which are intended as illustrations of various aspects. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from the spirit and scope of the present disclosure. In addition to the methods and devices recited in this disclosure, functionally equivalent methods and devices within the scope of the present disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be within the scope of the appended claims.

[0201] The above detailed description sets forth various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the figures, like symbols typically refer to like components identically, unless the context dictates otherwise. The exemplary embodiments described herein and in the figures are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the disclosure as generally described herein and illustrated in the figures may be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated.

[0202] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the figures and discussed in this disclosure, each step, block, operation, and / or communication may represent processing of information and / or transmission of information according to the example embodiments. Alternative embodiments are included within the scope of these example embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be executed in an order different from that shown or discussed, such as substantially concurrently or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used in any of the message flow diagrams, scenarios, and flowcharts discussed in this disclosure, and these message flow diagrams, scenarios, and flowcharts may be combined with each other, either in part or in whole.

[0203] A step, block, or operation corresponding to the processing of information may correspond to circuitry that may be configured to perform specific logical functions of a method or technique described herein. Alternatively or additionally, a step or block corresponding to the processing of information may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more instructions executable by a processor to perform specific logical operations or actions in a method or technique. The program code and / or associated data may be stored in any type of computer-readable medium, such as a storage device including RAM, a disk drive, a solid-state drive, or another storage medium.

[0204] Additionally, steps, blocks, or acts representing one or more information transmissions may correspond to information transmissions between software and / or hardware modules in the same physical device, however, other information transmissions may be information transmissions between software and / or hardware modules in different physical devices.

[0205] The particular arrangement shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Also, some of the illustrated elements may be combined or omitted. Also, example embodiments may include elements not illustrated in the figures.

[0206] While various aspects and embodiments are disclosed in this disclosure, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed in this disclosure are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims.

Claims

1. It is a method, In a computing system coupled to a vehicle, the system receives radar images corresponding to the vehicle's environment, generated by the vehicle's radar system. The meteorological conditions of the environment are estimated by the computing system using a weather estimation model, Based on the estimated meteorological conditions of the environment, the weather-related attenuation effect on the radar image is detected. In order to correct the radar image, the weather-related attenuation effect is removed by comparing the RCS value in the radar image with a baseline model representing the RCS values ​​of multiple objects under favorable weather conditions, and by determining whether the difference between the RCS value in the radar image and the RCS value in the baseline model exceeds a threshold difference. A method comprising providing the corrected radar image to the vehicle's control system.

2. The weather estimation model includes a rainfall estimator, The method according to claim 1, wherein the rainfall estimator is configured to map the raw backscatter values ​​in the radar image to the vehicle's rainfall estimation for the environment.

3. In order to correct the radar image, the weather-related attenuation effect is removed. Using a weather loss model, the rainfall estimate is mapped to the aperture loss and atmospheric pressure loss in the return signal intensity corresponding to the radar image, The method according to claim 2, comprising performing one or more corrective measures based on the aperture loss and atmospheric pressure loss in the return signal.

4. Implementing one or more of the aforementioned corrective measures is The method according to claim 3, comprising providing a signal to the control system to reduce the speed of the vehicle.

5. Implementing one or more of the aforementioned corrective measures is The offset is determined based on the aperture loss and atmospheric pressure loss in the return signal intensity corresponding to the radar image, The method according to claim 3, comprising correcting the radar image by applying the offset.

6. To estimate the meteorological conditions of the aforementioned environment, Receiving sensor data from one or more non-radar sensors, The method according to claim 1, comprising estimating the weather conditions of the environment based at least in part on the sensor data.

7. The baseline model includes a radar map representing radar images generated during favorable weather conditions, Comparing the RCS values ​​in the radar image with the baseline model representing the RCS values ​​of the multiple objects under favorable weather conditions, The method according to claim 1, comprising comparing the RCS values ​​in the radar image with the RCS values ​​of one or more objects specified by the radar map based on the position of the vehicle.

8. Receiving a set of radar images representing the environment of the vehicle, Determining that the environment of the vehicle has favorable weather conditions, The method according to claim 1, further comprising modifying the baseline model based on the set of radar images.

9. The method according to claim 1, further comprising receiving the baseline model from a remote computing system based on the location or route of the vehicle.

10. The method according to claim 1, wherein the baseline model includes an expected clutter-versus-noise profile having a clutter profile based on the median output of stationary pixels as a function of distance and a noise profile which is the median output of non-stationary pixels as a function of distance.

11. To detect a decrease in the effective detection range of the radar of the vehicle radar system, The method according to claim 1, further comprising providing an output to one or more vehicle systems that transmits the detected decrease in the sensing effective range of the radar.

12. The process involves detecting an object using the corrected radar image, The method according to claim 1, further comprising displaying a representation of the object for the vehicle on the display interface of the vehicle.

13. The method according to claim 1, wherein the control system is configured to control the vehicle based on the corrected radar image.

14. It is a system, A vehicle radar system comprising a first radar and a second radar, A computing device attached to a vehicle, Receiving radar images corresponding to the vehicle's environment, generated by the vehicle radar system, Using a weather estimation model, estimate the weather conditions of the aforementioned environment, The method includes detecting a weather-related attenuation effect on the radar image based on the estimated meteorological conditions of the environment, by detecting a first weather-related attenuation effect on a first radar image received from the first radar and a second weather-related attenuation effect on a second radar image received from the second radar, and by comparing the first weather-related attenuation effect on the first radar image with the second weather-related attenuation effect on the second radar image to distinguish between atmospheric loss and aperture loss. The process involves removing the weather-related attenuation effect and correcting the radar image, A system including a computing device configured to provide the corrected radar image to the vehicle's control system.

15. The computing device, The system according to claim 14, further configured to use a neural network to remove the weather-related attenuation effect based on the estimated weather conditions of the environment.

16. The system according to claim 15, wherein the computing device is The system according to claim 14, further configured to degrade radar images received during good weather conditions using a weather loss model.

17. The computing device, Based on the weather-related attenuation effect, a decrease in the effective detection range of the vehicle radar system's radar is detected. The system according to claim 14, further configured to provide an output to one or more vehicle systems that transmits the detected decrease in the sensing effective range of the radar.

18. A non-temporary computer-readable medium configured to store instructions, wherein when the instructions are executed by a computing system comprising one or more processors, the computing system... Receiving radar images generated by the vehicle's radar system that correspond to the vehicle's environment, The meteorological conditions of the environment are estimated by the computing system using a weather estimation model, Based on the estimated meteorological conditions of the environment, the weather-related attenuation effect on the radar image is detected. In order to correct the radar image, the weather-related attenuation effect is removed by comparing the RCS value in the radar image with a baseline model representing the RCS values ​​of multiple objects under favorable weather conditions, and by determining whether the difference between the RCS value in the radar image and the RCS value in the baseline model exceeds a threshold difference. A non-temporary computer-readable medium that causes an operation to be performed, including providing the corrected radar image to the vehicle's control system.

19. The instruction is provided to the computing system The non-temporary computer-readable medium according to claim 18, further comprising causing a remote computing system to receive the baseline model based on the location or route of the vehicle.

20. The non-temporal computer-readable medium according to claim 18, wherein the baseline model includes an expected clutter-versus-noise profile having a clutter profile based on the median output of stationary pixels as a function of distance and a noise profile which is the median output of non-stationary pixels as a function of distance.

Images