Method and system for estimating sensitivity of a vehicle sensor

Abstract

The example relates to a method and system for estimating radar sensitivity. The computational system uses radar data to detect objects in the environment of a vehicle. The system can then filter the radar data corresponding to objects based on a predetermined set of criteria to identify specific objects corresponding to a first type of object. The system can perform a comparison between radar parameters determined based on the radar data corresponding to the specific object and expected radar parameters represented by a data model. The data model is generated based on aggregated radar data for multiple objects matching the first type of object. The system can then estimate a radar sensitivity loss based on the comparison and adjust the radar operation or the vehicle's control strategy when the estimated radar sensitivity loss exceeds a threshold loss.

Description

Technical Field

[0001] The example embodiments relate to methods and systems for estimating the sensitivity of radar and other types of vehicle sensors. Background Technology

[0002] Advances in computing, sensors, and other technologies have enabled some vehicles to navigate safely between locations autonomously, i.e., without requiring input from a human driver. By processing sensor measurements of the surrounding environment in real time, semi-autonomous or autonomous vehicles can transport passengers or objects (e.g., cargo) between locations while avoiding obstacles, complying with traffic rules, anticipating the movement of nearby agents (e.g., other vehicles), and performing other actions typically performed by a driver. Delegating both decision-making and control of the vehicle to the vehicle system allows passengers to focus their attention on tasks other than driving.

[0003] Automotive radar is a type of sensor technology used by advanced driver-assistance systems (ADAS) and autonomous driving systems to detect and monitor the surrounding environment. In some cases, radar is used to detect and identify other vehicles, pedestrians, and various obstacles, as well as to determine their distance, speed, and trajectory.

[0004] Radar typically comprises radiating elements equipped with antennas that transmit radio waves. These waves are reflected from objects in the environment and subsequently captured by the antenna. The returned signals are then processed to assess the properties of the reflecting objects, including a radar cross-section (RCS) measurement that indicates the degree to which each object is detectable by radar. A larger RCS indicates that the object is easier to detect, while a smaller RCS indicates that the object is more difficult to detect. An object's RCS depends on a variety of factors, including the object's size, shape, material, and surface properties, as well as the frequency and polarization of the radar signal. Different types of objects, such as cars, motorcycles, pedestrians, and others, typically have different RCS values, which can be used to help identify the type of object in radar data. Summary of the Invention

[0005] The disclosed solution involves estimating radar sensitivity loss in a vehicle radar system by comparing filtered radar data with a data model of the expected signal-to-noise ratio (SNR) and / or radar cross-section (RCS) values ​​of similar objects. Radar data can be filtered based on criteria such as range, line-of-sight level, object visibility relative to the radar (e.g., unobstructed visibility), and weather conditions. The computational system compares this filtered data with the data model to estimate the sensitivity loss, which serves as a real-time indicator of radar degradation. When the loss exceeds certain thresholds, the vehicle system can use this information to identify potential hardware failures, adjust operations, and / or trigger maintenance. This approach enables continuous monitoring and optimization of radar performance, thereby enhancing overall vehicle safety and reliability.

[0006] In one aspect, an example method is described. This method involves receiving radar data representing the environment of a vehicle from a radar coupled to a vehicle at a computing system, and detecting one or more objects located in the environment based on the radar data. The method also involves filtering the radar data corresponding to the one or more objects based on a set of predetermined criteria to identify specific objects corresponding to objects of a first type, and performing a comparison between radar parameters determined based on the radar data corresponding to the specific objects and expected radar parameters represented by a data model. The data model is generated based on radar data aggregated for multiple objects matching the first type of objects. The method also involves estimating radar sensitivity loss based on this comparison.

[0007] In another aspect, an example system is described. This system includes a vehicle equipped with radar and a computing device. The computing device is configured to receive radar data representing the environment of the vehicle from radar coupled to the vehicle, and to detect one or more objects located in the environment based on the radar data. The computing device is also configured to filter the radar data corresponding to the one or more objects based on a set of predetermined criteria to identify specific objects corresponding to a first type of object, and to perform a comparison between radar parameters determined based on the radar data corresponding to the specific object and expected radar parameters represented by a data model. The data model is generated based on radar data aggregated for multiple objects matching the first type of object. The computing device is also configured to estimate radar sensitivity loss based on the comparison.

[0008] In another aspect, a non-transitory computer-readable medium is described. The non-transitory computer-readable medium is configured to store instructions that, when executed by a computing system comprising one or more processors, cause the computing system to perform operations. The operations involve receiving radar data representing the environment of a vehicle from a radar coupled to a vehicle, and detecting one or more objects located in the environment based on the radar data. The operations also involve filtering the radar data corresponding to the one or more objects based on a set of predetermined criteria to identify specific objects corresponding to objects of a first type, and performing a comparison between radar parameters determined based on the radar data corresponding to the specific objects and expected radar parameters represented by a data model. The data model is generated based on radar data aggregated for multiple objects matching the first type of objects. The operations also involve estimating radar sensitivity loss of the radar based on this comparison.

[0009] These and other aspects, advantages, and alternatives will become apparent to those skilled in the art upon reading the following detailed description and, where appropriate, referring to the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a functional block diagram of a vehicle according to an example embodiment.

[0011] Figure 2A This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0012] Figure 2B This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0013] Figure 2C This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0014] Figure 2D This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0015] Figure 2E This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0016] Figure 2F This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0017] Figure 2G This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0018] Figure 2H This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0019] Figure 2I This is a diagram illustrating the physical configuration of a vehicle according to an example embodiment.

[0020] Figure 2J This is a diagram illustrating the field of view of various sensors according to an example embodiment.

[0021] Figure 2K This is an illustration of the beam steering of a sensor according to an example embodiment.

[0022] Figure 3 This is a conceptual illustration of wireless communication between various computing systems associated with autonomous or semi-autonomous carriers, according to example embodiments.

[0023] Figure 4 This is a block diagram of a system including a radar system according to an example embodiment.

[0024] Figure 5 This is a conceptual illustration of a system for estimating and evaluating the sensitivity and performance of vehicle radar and other types of vehicle sensors, based on an example embodiment.

[0025] Figure 6 This is a flowchart of a method for estimating and evaluating the sensitivity of a vehicle radar according to an example embodiment. Detailed Implementation

[0026] This document contemplates exemplary methods and systems. Any exemplary embodiments or features described herein are not necessarily to be construed as preferred or advantageous over other embodiments or features. Furthermore, the exemplary embodiments described herein are not intended to be limiting. It will be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein. Additionally, the specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Moreover, exemplary embodiments may include elements not shown in the figures.

[0027] This disclosure presents a method and system for estimating the sensitivity and performance of automotive radar and other vehicle sensors. The method utilizes radar-sensing associated objects filtered based on specific criteria to establish a common set of objects with similar signal-to-noise ratios (SNR) or radar cross-sections (RCS). By comparing SNR and / or RCS data from these filtered objects with data models of similar filtered objects, the computational system can estimate the radar sensitivity loss. The vehicle system can then use this estimated loss to trigger cleaning processes, adjust sensor operation, request maintenance, modify sensor processing thresholds, or perform other optimizations to enhance sensor and vehicle performance.

[0028] For example, a vehicle computing system can compare SNR and / or RCS data from radar echoes reflected from a passenger vehicle with a data model generated for this purpose. This model represents expected radar measurements (such as SNR and RCS range) for radar echoes from a passenger vehicle, collected by a calibrated radar with a clean radome under optimal operating conditions. The model can be based on aggregated measurements from many vehicle radar systems in various environments, thus providing a robust set of expected values. While this example uses a passenger vehicle, this technique can be applied to a variety of objects typically encountered during navigation, such as traffic signs, buildings, trucks, or bridges.

[0029] These technologies enable vehicle systems to identify rapid degradation in radar sensitivity in real time, creating a loss estimator that allows for immediate adjustments to vehicle operation and sensor parameters as needed. They also function as sensitivity monitors to identify radars with potential hardware failures, detect performance effects from debris, or determine when weather conditions negatively impact radar performance. By implementing these technologies, vehicle systems can optimize radar performance, such as by adjusting operation when radar signal loss exceeds a threshold. This approach enhances the overall reliability and effectiveness of vehicle sensor systems, contributing to improved safety and performance across a wide range of driving conditions.

[0030] Data models used to evaluate sensor performance are typically developed off-vehicle, incorporating various parameters such as radar installation location, waveform characteristics, range, and gain settings. This model serves as a reference for comparison with real-time detection data collected during vehicle navigation. For example, the model could be based on radar measurements from similarly sized vehicles at different ranges. Once developed, the model is distributed to vehicles for use during navigation. When the vehicle's radar system detects an object, the parameters of these detections (e.g., SNR and RCS profiles) are compared to the model, enabling the vehicle system to estimate performance losses in real time and adjust operations as needed.

[0031] In some instances, the vehicle can generate its own data model or refine a model received from a remote computing system. The vehicle can use radar measurements taken under ideal conditions to establish baselines for SNR, RCS, and other radar parameters. These baselines are then used as a reference for evaluating radar sensitivity and performance during subsequent navigation.

[0032] The disclosed techniques also incorporate weather-specific filtering processes, recognizing that weather conditions can significantly impact radar sensitivity. This weather-specific filtering of the perceived object allows for more accurate estimation of radar sensitivity loss under diverse environmental conditions. By addressing the challenges of monitoring radar performance in complex automotive environments, these techniques provide a comprehensive approach to radar sensitivity estimation, enabling rapid relative estimation and validation. While primarily discussed in the context of radar, these techniques are applicable to other types of vehicle sensors, providing a universal solution for maintaining optimal sensor performance across a wide range of automotive applications.

[0033] During navigation, the computing system may receive radar data representing objects in the vehicle's environment, generated by one or more onboard radar sensors. These objects may include other vehicles, pedestrians, animals, buildings, trees, or any other entities present in the environment. The computing system then filters this radar data based on predetermined criteria. These criteria may include the range of objects relative to the radar and / or the vehicle, weather conditions in the environment, and whether the objects are within the radar's direct line of sight. For example, the system may filter out objects that are beyond a certain range, objects that are not within the radar's line of sight to a specific degree, or objects when weather conditions are unfavorable for radar detection (e.g., heavy rain or snow). The computing system may adjust these criteria based on the vehicle's environment and other relevant factors.

[0034] The computational system then compares the filtered object's SNR and / or RCS data with a data model. This model represents the expected SNR and RCS data aggregated from measurements of similar object types under various conditions and scenarios. This comparison may involve calculating the difference between the actual and expected SNR / RCS data, or using other suitable comparison methods. Based on this comparison, the computational system estimates the radar sensitivity loss, thereby indicating a degradation in the radar's ability to detect objects in its environment. This estimation can be used in real time to identify rapid sensitivity degradation or monitor sensitivity over time, potentially identifying hardware faults leading to reduced sensitivity. When the estimated loss exceeds a certain threshold, the vehicle system can adjust radar operation or usage. Additional actions may include triggering sensor cleaning operations, providing alerts to passengers and / or remote systems, and adjusting detection thresholds. This approach enables continuous monitoring and optimization of radar performance, thereby enhancing the overall vehicle safety and reliability.

[0035] In some examples, the system can perform the disclosed techniques to monitor and identify sensitivity loss of individual radars on a vehicle. Specifically, this technique can be performed individually for each radar on the vehicle. The computing system can receive radar data from multiple radar sensors, each mounted at a different location on the vehicle. For each radar sensor, the computing system can filter the radar data based on predetermined criteria, including range, weather conditions, and direct line-of-sight specific to that radar's location and field of view. The system can then perform a comparison between measured radar parameters (e.g., SNR data and / or RCS data) detected by the specific radar from the filtered object and a data model representing expected SNR or RCS data for similar objects. Based on this comparison, the computing system can estimate the radar sensitivity loss of that individual radar. By applying this process independently to each radar, the system can identify when a particular radar suffers sensitivity loss compared to other radars.

[0036] In some cases, the data models used for comparison can be customized for the specific characteristics of each radar, such as its installation location, waveform parameters, range, and gain settings. This allows for more accurate sensitivity loss estimates that take into account the unique characteristics and expected performance of each radar. Additionally, the computational system can track the estimated sensitivity loss of each radar over time. This historical data can be used to identify trends or sudden changes in the performance of individual radars. For example, if a radar consistently shows a higher sensitivity loss compared to other radars under similar conditions, it may indicate a hardware problem specific to that radar or that the radar may require cleaning.

[0037] In some respects, the system can apply different thresholds based on the importance of different radars to various vehicle functions to identify significant sensitivity losses for different radars. For example, a forward-facing radar can have a lower threshold for triggering an alarm compared to a side-facing radar. In this way, the real-time loss estimator can be applied individually to each radar, allowing the system to quickly identify situations where the sensitivity of a specific radar rapidly degrades. This is particularly useful for detecting sudden problems such as physical obstructions or damage to a single radar sensor.

[0038] In some cases, the system can compare sensitivity loss estimates across different radars on a vehicle. Significant differences between radars covering similar areas or operating under similar conditions may indicate that one of the radars is experiencing an unusual sensitivity loss. Furthermore, the sensitivity monitor function can be used to individually track the long-term performance trends of each radar. This can help identify radars that may gradually degrade over time due to environmental factors or hardware failures specific to their installation location.

[0039] In some aspects, the system can adjust the frequency of sensitivity loss estimates for individual radars based on their performance history. Radars showing signs of degradation can be monitored more frequently to track their status more closely. When sensitivity loss exceeds a predetermined threshold, the system can generate a separate alarm or notification for each radar. This allows for target maintenance or troubleshooting of specific radar units with poor performance. In some cases, the vehicle's sensor fusion algorithm can be adjusted based on individual radar sensitivity loss estimates. The system can reduce the weighting of data from radars showing significant sensitivity loss during overall perception and decision-making processes. By individually applying the disclosed techniques to each radar, the system can provide a comprehensive and fine-grained evaluation of the radar suite's performance, enabling timely identification and resolution of sensitivity issues in specific radar units on the vehicle.

[0040] In response to the detection of sensitivity loss at the radar, the vehicle system can take several actions. The system can adjust the sensor fusion algorithm to reduce reliance on data from the affected radar, thereby giving greater weight to other normally functioning sensors or radars. The frequency of sensitivity monitoring of the affected radar can be increased to more closely track its performance. The vehicle system can attempt to modify radar operating parameters, such as transmit power, gain settings, waveform characteristics, radar timeline, the number of active transmitting elements, and / or beam steering parameters, to compensate for the detected loss where possible.

[0041] In some cases, the system can fine-tune the radar's signal processing algorithm, adjust detection thresholds, modify tracking filters, or update clutter suppression techniques based on observed performance metrics. The radar's operating frequency can be shifted to avoid interference or improve penetration under adverse weather conditions. In multi-sensor systems, the fusion algorithm can be adjusted based on evaluated radar performance. The system can generate alerts regarding reduced sensor capabilities and adjust vehicle control strategies, potentially reducing the maximum permissible rate or increasing the following distance. Self-diagnostic routines can be initiated, and, if equipped, an automatic cleaning system can be triggered.

[0042] Vehicle systems can record detailed performance data for later analysis and potential over-the-air updates. They can also notify telematics systems to schedule maintenance or inspections. In some cases, the system can dynamically reconfigure functional radars to compensate for lost coverage, activate backup sensors (if available), or limit certain advanced pilot assistance features that heavily rely on the affected radar. These actions are designed to maintain overall sensing capability and ensure safe vehicle operation, despite reduced radar performance.

[0043] The following description and accompanying drawings will illustrate the features of various exemplary embodiments. The embodiments are provided by way of example and are not intended to be limiting. Therefore, the dimensions of the drawings are not necessarily drawn to scale.

[0044] Example systems within the scope of this disclosure will now be described in more detail. The example systems may be implemented in or take the form of automobiles. Additionally, the example systems may be implemented in or take the form of various vehicles, such as cars, trucks (e.g., pickup trucks, vans, tractors, and tractor trailers), motorcycles, buses, airplanes, helicopters, drones, lawnmowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, farm equipment or vehicles, construction equipment or vehicles, warehouse equipment or vehicles, factory equipment or vehicles, trams, golf carts, trains, handcarts, sidewalk transport vehicles, and robotic equipment. Other vehicles are also possible. Furthermore, in some embodiments, the example systems may not include a vehicle.

[0045] Now refer to the diagram, Figure 1 This is a functional block diagram illustrating an example vehicle 100, which can be configured to operate fully or partially in an autonomous mode. More specifically, vehicle 100 can operate in autonomous mode without human interaction by receiving control commands from a computing system. As part of operating in autonomous mode, vehicle 100 can use sensors to detect and, possibly, identify objects in the surrounding environment to enable safe navigation. Furthermore, vehicle 100 can operate in a partially autonomous (i.e., semi-autonomous) mode, in which some functions of vehicle 100 are controlled by a human driver, while others are controlled by the computing system. For example, vehicle 100 may also include subsystems that enable the driver to control the operation of vehicle 100 (such as steering, acceleration, and braking), while the computing system performs assistance functions, such as lane departure warning / lane keeping assist or adaptive cruise control, based on other objects in the surrounding environment (e.g., the vehicle).

[0046] As described in this article, in partially autonomous driving modes, even when the vehicle assists with one or more driving operations (e.g., steering, braking, and / or acceleration to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), and emergency braking), the human driver is expected to be situationally aware of the vehicle's surroundings and supervise the assisted driving operations. Here, even if the vehicle may perform all driving tasks in certain situations, the human driver is expected to take over control as needed.

[0047] Although various systems and methods are described below in conjunction with autonomous vehicles for the sake of brevity and simplicity, these or similar systems and methods can be used in various driver assistance systems that do not rise to the level of fully autonomous driving systems (i.e., partially autonomous driving systems). In the United States, the Society of Automotive Engineers (SAE) has defined different levels of automated driving operation to indicate how high or low the degree of vehicle control is, although different organizations in the United States or other countries may classify levels differently. More specifically, the disclosed systems and methods can be used in SAE Level 2 driver assistance systems, which implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver support. The disclosed systems and methods can be used in SAE Level 3 driver assistance systems capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods can be used in vehicles using SAE Level 4 automated driving systems, which operate autonomously in most normal driving situations and require only occasional human operator attention. In all such systems, accurate lane estimation can be performed automatically without driver input or control (e.g., when the vehicle is in motion), leading to improved reliability of vehicle positioning and navigation, as well as overall safety improvements for autonomous, semi-autonomous, and other driver assistance systems. As previously mentioned, other organizations in the U.S. or other countries may classify levels of automated driving operations differently than the SAE does. Without limitation, the systems and methods disclosed herein can be used in driver assistance systems defined by the automated driving operation levels of these other organizations.

[0048] like Figure 1 As shown, vehicle 100 may include various subsystems, such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power supply 110, a computer system 112 (which may also be referred to as a computing system) with a data storage device 114, and a user interface 116. In other examples, vehicle 100 may include more or fewer subsystems, each subsystem including multiple elements. The subsystems and components of vehicle 100 may be interconnected in various ways. Additionally, within embodiments, the functionality of vehicle 100 described herein may be divided into additional functions or physical components, or combined into fewer functions or physical components. For example, control system 106 and computer system 112 may be combined into a single system to operate vehicle 100 according to various operations.

[0049] The propulsion system 102 may include one or more components operable to provide powered motion to the vehicle 100, and may include an engine / motor 118, an energy source 119, a transmission 120, and wheels / tires 121, as well as other possible components. For example, the engine / motor 118 may be configured to convert the energy source 119 into mechanical energy, and may correspond to one or a combination of an internal combustion engine, an electric motor, a steam engine, or a Stirling engine, as well as other possible options. For example, in some embodiments, the propulsion system 102 may include multiple types of engines and / or motors, such as gasoline engines and electric motors.

[0050] Energy source 119 refers to a source of energy that can provide power, in whole or in part, to one or more systems of vehicle 100 (e.g., engine / motor 118). For example, energy source 119 may correspond to a source of gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other electrical energy. In some embodiments, energy source 119 may include a combination of a fuel tank, battery, capacitor, and / or flywheel.

[0051] The transmission 120 can send mechanical power from the engine / motor 118 to the wheels / tires 121 of the vehicle 100 and / or other possible systems. Thus, the transmission 120 may include a gearbox, clutch, differential, and drive shaft, as well as other possible components. The drive shaft may include an axle connected to one or more wheels / tires 121.

[0052] In the example embodiment, the wheels / tires 121 of the vehicle 100 can have various configurations. For example, the vehicle 100 can exist as a unicycle, bicycle / motorcycle, tricycle, or four-wheeled car / truck, among other possible configurations. Thus, the wheels / tires 121 can be attached to the vehicle 100 in various ways and can be made of different materials, such as metal and rubber.

[0053] Sensor system 104 may include various types of sensors, such as a Global Positioning System (GPS) 122, an inertial measurement unit (IMU) 124, radar 126, lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, as well as other possible sensors. In some embodiments, sensor system 104 may also include sensors configured to monitor the internal systems of vehicle 100 (e.g., O2 monitor, fuel gauge, engine oil temperature, and brake wear).

[0054] GPS 122 may include a transceiver operable to provide information about the positioning of vehicle 100 relative to the Earth. IMU 124 may be configured to use one or more accelerometers and / or gyroscopes and can sense changes in the positioning and orientation of vehicle 100 based on inertial acceleration. For example, IMU 124 can detect the pitch and yaw of vehicle 100 when vehicle 100 is stationary or in motion.

[0055] Radar 126 may represent one or more systems configured to use radio signals to sense objects (including the object's speed and direction of travel) within the surrounding environment of vehicle 100. Thus, radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, radar 126 may correspond to an mountable radar configured to obtain measurements of the surrounding environment of vehicle 100.

[0056] The lidar 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components, and may operate in a coherent mode (e.g., using heterodyne detection) or an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of the lidar 128 may include one or more photodetectors, which may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single-photon avalanche diodes (SPADs)). Furthermore, such photodetectors may be arranged (e.g., via series electrical connections) in an array (e.g., as in a silicon photomultiplier (SiPM)). In some examples, one or more photodetectors are Geiger-mode operating devices, and the lidar includes sub-components designed for such Geiger-mode operation.

[0057] Camera 130 may include one or more devices (e.g., still camera, video camera, thermal imaging camera, stereo camera, and night vision camera) configured to capture images of the surrounding environment of vehicle 100.

[0058] The steering sensor 123 can sense the steering angle of the vehicle 100, which may involve measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 can measure the angle of the wheels of the vehicle 100, such as detecting the angle of the wheels relative to the forward axle of the vehicle 100. The steering sensor 123 can also be configured to measure a combination (or subset) of the steering wheel angle, an electrical signal representing the steering wheel angle, and the angles of the wheels of the vehicle 100.

[0059] The throttle / brake sensor 125 can detect the positioning of the throttle valve or brake of the vehicle 100. For example, the throttle / brake sensor 125 can measure the angle of both the accelerator pedal (throttle valve) and the brake pedal, or it can measure an electrical signal that can represent, for example, the angle of the accelerator pedal (throttle valve) and / or the angle of the brake pedal. The throttle / brake sensor 125 can also measure the angle of the throttle body of the vehicle 100, which may include part of a modulated physical mechanism (e.g., a butterfly valve and a carburetor) that provides energy to the engine / motor 118 via the power source 119. Additionally, the throttle / brake sensor 125 can measure the pressure of one or more brake blocks on the rotor of the vehicle 100, or a combination (or subset) of the angle of the accelerator pedal (throttle valve) and the brake pedal, an electrical signal representing the angle of the accelerator pedal (throttle valve) and the brake pedal, the angle of the throttle body, and the pressure exerted by at least one brake block on the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal (such as a throttle or brake pedal).

[0060] The control system 106 may include components configured to assist the navigation of the vehicle 100, such as a steering unit 132, a throttle valve 134, a braking unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / path system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the forward direction of the vehicle 100, and the throttle valve 134 may control the operating rate of the engine / motor 118 to control the acceleration of the vehicle 100. The braking unit 136 may decelerate the vehicle 100, which may involve using friction to slow down the wheels / tires 121. In some embodiments, the braking unit 136 may convert the kinetic energy of the wheels / tires 121 into electrical current for subsequent use by one or more systems of the vehicle 100.

[0061] Sensor fusion algorithm 138 may include Kalman filters, Bayesian networks, or other algorithms capable of processing data from sensor system 104. In some embodiments, sensor fusion algorithm 138 may provide evaluations based on incoming sensor data, such as assessments of individual objects and / or features, assessments of specific situations, and / or assessments of potential impacts within a given situation.

[0062] Computer vision system 140 may include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a dedicated processor such as a graphics processing unit (GPU) or tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) operable to process and analyze images in an effort to determine moving objects (e.g., other vehicles, pedestrians, cyclists, or animals) and stationary objects (e.g., traffic lights, road boundaries, speed bumps, or potholes). Thus, computer vision system 140 may use object recognition, structure from motion (SFM), video tracking, and other algorithms used in computer vision, such as to identify objects, map the environment, track objects, estimate object rates, etc.

[0063] The navigation / path system 142 can determine the driving path of the vehicle 100, which may involve dynamically adjusting the navigation during operation. Thus, the navigation / path system 142 can navigate the vehicle 100 using data from sensor fusion algorithm 138, GPS 122, maps, and other sources. The obstacle avoidance system 144 can assess potential obstacles based on sensor data and enable the vehicle 100's systems to avoid or otherwise traverse potential obstacles.

[0064] like Figure 1 As shown, vehicle 100 may also include peripheral devices 108, such as wireless communication system 146, touchscreen 148, microphone 150 (e.g., one or more internal and / or external microphones) and / or speaker 152. Peripheral devices 108 may provide controls or other elements for a user to interact with user interface 116. For example, touchscreen 148 may provide information to the user of vehicle 100. User interface 116 may also accept input from the user via touchscreen 148. Peripheral devices 108 may also enable vehicle 100 to communicate with devices such as other vehicle equipment.

[0065] The wireless communication system 146 can communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 can use 3G cellular communication (such as code-division multiple access (CDMA), evolution-data optimized (EVDO), global system for mobile communications (GSM) / general packet radio service (GPRS)) or cellular communication (such as 4G worldwide interoperability for microwave access (WiMAX) or long-term evolution (LTE) or 5G). Alternatively, the wireless communication system 146 can communicate with a wireless local area network (WLAN) using Wi-Fi® or other possible connections. The wireless communication system 146 can also communicate directly with devices using, for example, an infrared link, Bluetooth, or ZigBee. Within the context of this disclosure, other wireless protocols, such as various vehicle communication systems, are possible. For example, wireless communication system 146 may include one or more dedicated short-range communications (DSRC) devices, which may include public and / or private data communications between vehicles and / or roadside stations.

[0066] The carrier 100 may include a power supply 110 for powering components. In some embodiments, the power supply 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power supply 110 may include one or more batteries configured to provide power. The carrier 100 may also use other types of power supplies. In an example embodiment, the power supply 110 and the energy source 119 may be integrated into a single energy source.

[0067] The vehicle 100 may also include a computer system 112 to perform operations, such as those described herein. Thus, the computer system 112 may include a processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored in a non-transitory computer-readable medium, such as a data storage device 114. In this way, the processor 113 may represent one or more processors. In some embodiments, the computer system 112 may represent multiple computing devices that can be used to control individual components or subsystems of the vehicle 100 in a distributed manner.

[0068] In some embodiments, the data storage device 114 may include instructions 115 (e.g., program logic) executable by the processor 113 to perform various functions of the vehicle 100, including those described above. Figure 1 The functions described. The data storage device 114 may also contain additional instructions, including instructions for sending data to, receiving data from, interacting with, and / or controlling one or more of the propulsion system 102, sensor system 104, control system 106, and peripheral devices 108.

[0069] In addition to command 115, data storage device 114 can store data such as road maps, route information, and other information. Such information can be used by vehicle 100 and computer system 112 during autonomous, semi-autonomous, and / or manual operation of vehicle 100.

[0070] Vehicle 100 may include a user interface 116 for providing information to or receiving input from a user of vehicle 100. User interface 116 may control or enable control over the content and / or layout of interactive images that may be displayed on touchscreen 148. Furthermore, user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as wireless communication system 146, touchscreen 148, microphone 150, and speaker 152.

[0071] Computer system 112 can control the functions of vehicle 100 based on input received from various subsystems (e.g., propulsion system 102, sensor system 104, or control system 106) and from user interface 116. For example, computer system 112 can utilize input from sensor system 104 to estimate the outputs generated by propulsion system 102 and control system 106. Depending on the embodiment, computer system 112 can be operable to monitor many aspects of vehicle 100 and its subsystems. In some embodiments, computer system 112 can disable some or all functions of vehicle 100 based on signals received from sensor system 104.

[0072] The components of vehicle 100 can be configured to operate in a manner interconnected with other components, either internally or externally to their respective systems. For example, in an example embodiment, camera 130 can capture multiple images that may represent information about the state of the surrounding environment of vehicle 100 operating in autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be able to identify slope (gradient) or other features based on multiple images of the road. Furthermore, a combination of GPS 122 and features identified by computer vision system 140 can be used with map data stored in data storage device 114 to determine specific road parameters. Additionally, radar 126 and / or lidar 128 and / or some other environmental mapping, ranging, and / or positioning sensor systems may also provide information about the vehicle's surrounding environment.

[0073] In other words, the combination of various sensors (which may be referred to as input indication and output indication sensors) and computer system 112 can interact to provide indications of inputs provided to control the vehicle or indications of the vehicle's surrounding environment.

[0074] In some embodiments, computer system 112 can make determinations about various objects based on data provided by systems other than radio systems. For example, vehicle 100 may have lasers or other optical sensors configured to sense objects in the vehicle's field of view. Computer system 112 can use the outputs from various sensors to determine information about objects in the vehicle's field of view, and can determine distance and orientation information to various objects. Computer system 112 can also determine whether an object is desired or undesirable based on the outputs from various sensors.

[0075] although Figure 1 Various components of the vehicle 100 (i.e., the wireless communication system 146, the computer system 112, the data storage device 114, and the user interface 116) are shown as integrated into the vehicle 100; however, one or more of these components may be installed or associated separately from the vehicle 100. For example, the data storage device 114 may exist partially or wholly separate from the vehicle 100. Therefore, the vehicle 100 can be provided in the form of device elements that can be positioned separately or together. The device elements constituting the vehicle 100 can be communicatively coupled together in a wired and / or wireless manner.

[0076] Figures 2A to 2E An example vehicle 200 (e.g., a fully autonomous vehicle or a semi-autonomous vehicle) is shown, which may include a reference vehicle. Figure 1 The vehicle 100 describes some or all of its functions. Although for illustrative purposes, the vehicle 200... Figures 2A to 2EThe vehicle is illustrated as a van with side mirrors, but this disclosure is not limited thereto. For example, vehicle 200 may represent a truck, car, semi-trailer truck, motorcycle, golf cart, off-road vehicle, agricultural vehicle, or any other vehicle described elsewhere herein (e.g., bus, boat, aircraft, helicopter, drone, lawnmower, bulldozer, submarine, all-terrain vehicle, snowmobile, aircraft, recreational vehicle, amusement park vehicle, farm equipment, construction equipment or vehicle, warehouse equipment or vehicle, factory equipment or vehicle, tram, train, handcart, sidewalk transport vehicle, and robotic equipment).

[0077] Vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidar systems, one or more radar systems, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding vehicle 200. In other words, any sensor system now known or created later may be coupled to vehicle 200 and / or may be utilized in conjunction with various operations of vehicle 200. As an example, lidar may be utilized in autonomous driving or other types of navigation, planning, perception, and / or mapping operations of vehicle 200. Additionally, sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may represent combinations of sensors described herein (e.g., one or more lidar and radar; one or more lidar and camera; one or more camera and radar; or one or more lidar, camera and radar).

[0078] Notice, Figures 2A to 2E The number, location, and type of sensor systems (e.g., 202 and 204) depicted are intended as non-limiting examples of the location, number, and type of such sensor systems for autonomous or semi-autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., consistent with vehicle size, shape, aerodynamics, fuel economy, aesthetics, or other conditions to reduce costs or adapt to specific environmental or application situations). For example, sensor systems (e.g., 202 and 204) can be positioned at various other locations on the vehicle and can have a field of view corresponding to the interior and / or surrounding environment of the vehicle 200.

[0079] Sensor system 202 may be mounted on top of vehicle 200 and may include one or more sensors configured to detect information about the environment surrounding vehicle 200 and output indications of that information. For example, sensor system 202 may include any combination of cameras, radar, lidar, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). Sensor system 202 may include one or more movable mounts operable to adjust the orientation of one or more sensors in sensor system 202. In one embodiment, the movable mount may include a rotating platform that can scan the sensors to obtain information from every direction around vehicle 200. In another embodiment, the movable mount of sensor system 202 may be movable in a scanning manner within a specific angular and / or azimuth and / or elevation range. Sensor system 202 may be mounted on the top of the roof of a car, but other mounting locations are possible.

[0080] Furthermore, the sensors of sensor system 202 can be distributed at different locations and do not need to be juxtaposed at a single location. Additionally, each sensor of sensor system 202 can be configured to move or scan independently of other sensors in sensor system 202. Alternatively or additionally, multiple sensors can be installed at one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there may be two lidar devices installed at the sensor locations and / or one lidar device and one radar device installed at the sensor locations.

[0081] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more lidar devices. For example, a lidar device may include multiple light emitter devices arranged within an angular range relative to a given plane (e.g., the xy-plane). For example, one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis perpendicular to the given plane (e.g., the z-axis) to illuminate the environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment can be determined based on various aspects of the detected reflected light pulses (e.g., elapsed time of flight, polarization, and intensity).

[0082] In example embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide corresponding point cloud information that can be related to physical objects within the surrounding environment of vehicle 200. While vehicle 200 and sensor systems 202, 204, 206, 208, 210, 212, 214, and 218 are illustrated to include certain features, it should be understood that other types of sensor systems are contemplated within the scope of this disclosure. Furthermore, vehicle 200 may include combinations of… Figure 1 Any component described in vehicle 100.

[0083] In the example configuration, one or more radars may be located on vehicle 200. Similar to radar 126 described above, one or more radars may include antennas configured to transmit and receive radio waves (e.g., electromagnetic waves with frequencies between 30 Hz and 300 GHz). Such radio waves can be used to determine the distance to one or more objects in the surrounding environment of vehicle 200 and / or the velocity of one or more objects in the surrounding environment of vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may include one or more radars. In some examples, one or more radars may be located near the rear of vehicle 200 (e.g., sensor systems 208 and 210) to actively scan the environment near the rear of vehicle 200 in response to the presence of radio-reflecting objects. Similarly, one or more radars may be located near the front of vehicle 200 (e.g., sensor systems 212 or 214) to actively scan the environment near the front of vehicle 200. For example, the radar can be located in an area suitable for illuminating the path of the vehicle 200 without being obstructed by other features of the vehicle 200. For example, the radar can be embedded in or near the front bumper, headlights, front bulkhead cover, and / or hood. Furthermore, one or more additional radars can be positioned to actively scan the sides and / or rear of the vehicle 200 in response to the presence of radio-reflecting objects, such as by including such devices in or near the rear bumper, side panels, sill beams, and / or chassis.

[0084] Vehicle 200 may include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214 and / or 218 may include one or more cameras. The cameras may be photosensitizing instruments, such as still cameras, video cameras, thermal imaging cameras, stereo cameras, night vision cameras, etc., configured to capture multiple images of the surrounding environment of vehicle 200. For this purpose, the cameras may be configured to detect visible light and may additionally or alternatively be configured to detect light from other parts of the spectrum, such as infrared or ultraviolet light. The cameras may be two-dimensional detectors and may optionally have a three-dimensional spatial sensitivity range. In some embodiments, the camera may include, for example, a range detector configured to generate two-dimensional images indicating distances from the camera to multiple points in the surrounding environment. For this purpose, the camera may use one or more range detection techniques. For example, the camera may provide range information by using structured light technology, in which vehicle 200 illuminates objects in the surrounding environment with a predetermined light pattern (such as a grid or checkerboard pattern), and the camera is used to detect reflections of the predetermined light pattern from the surrounding environment. Based on the distortion in the reflected light pattern, vehicle 200 can determine the distance to a point on an object. The predetermined light pattern may include infrared light or radiation of other suitable wavelengths for such measurement. In some examples, the camera may be mounted inside the windshield of vehicle 200. Specifically, the camera may be positioned to capture images from a forward-facing view relative to vehicle 200. Other mounting positions and viewing angles of the camera may also be used, inside or outside vehicle 200. Furthermore, the camera may have associated optics operable to provide an adjustable field of view. Further still, the camera may be mounted to vehicle 200 with a movable mount to change the camera's pointing angle, such as via a translation / tilt mechanism.

[0085] Vehicle 200 may also include one or more acoustic sensors for sensing the surrounding environment of vehicle 200 (e.g., one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, or microelectromechanical systems (MEMS) microphones) for sensing sound waves (i.e., pressure differences) in a fluid (e.g., air) surrounding vehicle 200. Such acoustic sensors can be used to identify sounds in the surrounding environment (e.g., sirens, human voices, animal sounds, or alarms), and the control strategy of vehicle 200 can be based on these sounds. For example, if the acoustic sensors detect a sirens (e.g., a medical transport vehicle siren or a fire truck siren), vehicle 200 may decelerate and / or navigate to the edge of a road.

[0086] Despite Figures 2A to 2E Not shown, but vehicle 200 may include a wireless communication system (e.g., similar to...). Figure 1 Wireless communication systems 146 and / or other than Figure 1 (Outside of the wireless communication system 146). The wireless communication system may include a wireless transmitter and receiver, which may be configured to communicate with devices external to or internal to the vehicle 200. Specifically, the wireless communication system may include transceivers configured to communicate with other vehicles and / or computing devices (e.g., other vehicles and / or computing devices in a vehicle communication system or road station). Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transportation systems.

[0087] The vehicle 200 may include one or more other components besides those shown or in place of those shown. Additional components may include electrical or mechanical functions.

[0088] The control system of vehicle 200 can be configured to control vehicle 200 according to a control strategy among a plurality of possible control strategies. The control system can be configured to receive information from sensors coupled to vehicle 200 (on or outside vehicle 200), modify the control strategy (and associated driving behavior) based on that information, and control vehicle 200 according to the modified control strategy. The control system can also be configured to monitor information received from sensors and continuously evaluate driving conditions; and can also be configured to modify the control strategy and driving behavior based on changes in driving conditions. For example, the route taken by the vehicle from one destination to another can be modified based on driving conditions. Additionally or alternatively, speed, acceleration, turning angle, following distance (i.e., distance to the vehicle in front of the current vehicle), lane selection, etc., can all be modified in response to changes in driving conditions.

[0089] As described above, in some embodiments, the vehicle 200 may be in the form of a van, but alternative forms are also possible and are envisioned herein. Thus, Figures 2F to 2I An embodiment of vehicle 250 in semi-truck form is shown. For example, Figure 2F A front view of vehicle 250 is shown, and Figure 2G An isometric view of vehicle 250 is shown. In an embodiment where vehicle 250 is a semi-truck, vehicle 250 may include a tractor unit 260 and a trailer 270 (in... Figure 2G (As shown in the middle diagram). Figure 2H and Figure 2I Side and top views of the tractor unit 260 are provided. It is similar to the vehicle 200 illustrated above. Figures 2F to 2I The vehicle 250 shown in the diagram may also include various sensor systems (e.g., similar to those in the reference citation). Figures 2A to 2E Sensor systems 202, 206, 208, 210, 212, 214 are shown and described. In some embodiments, although Figures 2A to 2E The vehicle 200 may include only a single copy of some sensor systems (e.g., sensor system 204), but Figures 2F to 2I The vehicle 250 illustrated may include multiple copies of the sensor system (e.g., sensor systems 204A and 204B, as illustrated).

[0090] While the entire drawing and description may refer to a vehicle of a given form (e.g., a semi-truck vehicle 250 or a vehicle 200 shown as a van), it should be understood that the embodiments described herein can be equally applied in various vehicle contexts (e.g., where modifications are employed to take into account the form factor of the vehicle). For example, sensors and / or other components described or illustrated as part of vehicle 200 may also be used in semi-truck vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0091] Figure 2J Various sensor fields of view are shown (e.g., associated with the aforementioned vehicle 250). As described above, vehicle 250 may contain multiple sensors / sensor units. For example, the positions of various sensors may be related to... Figures 2F to 2I The locations of the sensors disclosed in the figures correspond to those in the diagram. However, in some instances, the sensors may have other locations. To simplify the figures, [the diagram is shown below]. Figure 2J Sensor location labels are omitted from the attached diagram. For each sensor unit of vehicle 250, Figure 2J Representative fields of view are shown (e.g., fields of view labeled 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The field of view of a sensor may include angular regions on which the sensor can detect objects (e.g., azimuth and / or elevation regions).

[0092] Figure 2K An example embodiment for a vehicle (e.g., reference) is shown. Figures 2F to 2JThe beam steering of the sensors of the vehicle 250 is shown and described. In various embodiments, the sensor unit of the vehicle 250 may be radar, lidar, sonar, etc. Furthermore, in some embodiments, the sensor may be scanned within its field of view during operation. Various different scanning angles of the example sensor are shown as regions 272, each indicating the angular region on which the sensor is operating. The sensor may periodically or iteratively change the region on which it is operating. In some embodiments, the vehicle 250 may use multiple sensors to measure region 272. Additionally, other regions may be included in other examples. For example, one or more sensors may measure aspects of the trailer 270 of the vehicle 250 and / or the area directly in front of the vehicle 250.

[0093] At some angles, the sensor's operating area 275 may include the rear wheels 276A and 276B of the trailer 270. Therefore, the sensor can measure the rear wheels 276A and / or 276B during operation. For example, the rear wheels 276A and 276B may reflect lidar or radar signals transmitted by the sensor. The sensor can receive the reflected signals from the rear wheels 276A and 276B. Therefore, the data collected by the sensor may include data from reflections from the wheels.

[0094] In some instances, such as when the sensor is radar, reflections from the rear wheels 276A and 276B may appear as noise in the received radar signal. Therefore, in instances where the rear wheels 276A and 276B guide the radar signal away from the sensor, the radar can operate with an enhanced signal-to-noise ratio.

[0095] Figure 3 This is a conceptual illustration of wireless communication between various computing systems associated with an autonomous or semi-autonomous vehicle, according to an example embodiment. Specifically, wireless communication can occur between the remote computing system 302 and the vehicle 200 via network 304. Wireless communication can also occur between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0096] Vehicle 200 can correspond to various types of vehicles capable of transporting passengers or objects between locations, and can take any one or more forms of vehicles discussed above. In some instances, vehicle 200 can operate in autonomous or semi-autonomous mode, which enables the control system to use sensor measurements to safely navigate vehicle 200 between destinations. When operating in autonomous or semi-autonomous mode, vehicle 200 can navigate with or without passengers. As a result, vehicle 200 can pick up and drop off passengers between desired destinations.

[0097] The remote computing system 302 can represent any type of device related to remote assistance technology, including but not limited to those described herein. Within the example, the remote computing system 302 can represent any type of device configured to (i) receive information relating to vehicle 200, (ii) provide an interface through which a human operator can sequentially perceive the information and input a response relating to the information, and (iii) send the response to vehicle 200 or other devices. The remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop computer, tablet computer, mobile phone (e.g., smartphone), and / or server. In some examples, the remote computing system 302 may include multiple computing devices operating together in a network configuration.

[0098] The remote computing system 302 may include one or more subsystems and components that are similar to or identical to those of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface that includes input / output devices such as a touchscreen and speakers. Other examples are also possible.

[0099] Network 304 represents the infrastructure that enables wireless communication between remote computing system 302 and vehicle 200. Network 304 also enables wireless communication between server computing system 306 and remote computing system 302, and between server computing system 306 and vehicle 200.

[0100] The location of the remote computing system 302 can vary within the examples. For instance, the remote computing system 302 may have a remote location relative to the vehicle 200, with wireless communication via network 304. In another example, the remote computing system 302 may correspond to a computing device within the vehicle 200, which is detached from the vehicle 200, but a human operator can interact with the computing device while simultaneously acting as a passenger or driver of the vehicle 200. In some examples, the remote computing system 302 may be a computing device with a touchscreen operable by passengers of the vehicle 200.

[0101] In some embodiments, the operations performed by the remote computing system 302 described herein may additionally or alternatively be performed by the vehicle 200 (i.e., by any of the vehicle 200's systems or subsystems). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism that the vehicle's driver or passengers can interact with.

[0102] Server computing system 306 can be configured to wirelessly communicate with remote computing system 302 and vehicle 200 (or possibly directly with remote computing system 302 and / or vehicle 200) via network 304. Server computing system 306 can represent any computing device configured to receive, store, determine, and / or transmit information relating to vehicle 200 and its remote assistance. Thus, server computing system 306 can be configured to perform any of the operations(s) described herein as being performed by remote computing system 302 and / or vehicle 200, or portions thereof. Some embodiments of wireless communication relating to remote assistance may utilize server computing system 306, while others may not.

[0103] Server computing system 306 may include one or more subsystems and components that are similar to or identical to the subsystems and components of remote computing system 302 and / or vehicle 200, such as processors configured to perform the various operations described herein, and wireless communication interfaces for receiving information from and providing information to remote computing system 302 and vehicle 200.

[0104] The various systems described above can perform a variety of operations. These operations and related characteristics will now be described.

[0105] Based on the above discussion, a computing system (e.g., a remote computing system 302, a server computing system 306, or a computing system local to the vehicle 200) can operate to use a camera to capture images of the surrounding environment of the autonomous or semi-autonomous vehicle. Typically, at least one computing system will be able to analyze the images and may control the autonomous or semi-autonomous vehicle.

[0106] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data representing objects in its surrounding environment (also referred to herein as “environmental data”) in various ways. Sensor systems on the vehicle can provide environmental data representing objects in the surrounding environment. For example, the vehicle may have various sensors, including cameras, radar, lidar, microphones, radio units, and other sensors. Each of these sensors can transmit environmental data about the information received by each respective sensor to a processor within the vehicle.

[0107] In one example, the camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have more than one camera positioned in different orientations. Moreover, in some embodiments, the camera may be able to move to capture images and / or video in different directions. The camera may be configured to store the captured images and video in memory for later processing by the vehicle's processing system. The captured images and / or video may be environmental data. Furthermore, the camera may include an image sensor as described herein.

[0108] In another example, the radar can be configured to transmit electromagnetic signals reflected by various objects near the vehicle, and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals allow the radar (or processing system) to make various determinations about the objects reflecting the electromagnetic signals. For example, the distance to the various reflecting objects and the location of the various reflecting objects can be determined. In some embodiments, the vehicle may have more than one radar in different orientations. The radar can be configured to store the captured information in a memory for later processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0109] In another example, a lidar can be configured to transmit electromagnetic signals (e.g., infrared light, such as infrared light from a gas or diode laser or other possible light source) reflected by a target object near the vehicle. The lidar can be able to capture the reflected electromagnetic (e.g., infrared light) signals. The captured reflected electromagnetic signals can enable a ranging system (or processing system) to determine the distance to various objects. The lidar can also be able to determine the rate or velocity of the target object and store it as environmental data.

[0110] Additionally, in this example, the microphone can be configured to capture audio of the environment surrounding the vehicle. The sounds captured by the microphone can include emergency vehicle sirens and other vehicle sounds. For example, the microphone could capture the sirens of ambulances, fire trucks, or police cars. The processing system can then identify the captured audio signals to indicate an emergency vehicle. In another example, the microphone could capture the exhaust sound of another vehicle, such as a motorcycle. The processing system can then identify the captured audio signals to indicate a motorcycle. The data captured by the microphone can form part of the environmental data.

[0111] In another example, the radio unit can be configured to transmit electromagnetic signals, which may take the form of Bluetooth signals, 802.11 signals, and / or other radio technology signals. The first electromagnetic radiation signal can be transmitted via one or more antennas located in the radio unit. Furthermore, the first electromagnetic radiation signal can be transmitted using one of many different radio signaling modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signaling mode that requests a response from a device located near the autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the responses transmitted back to the radio unit and use this transmitted information as part of environmental data.

[0112] In some embodiments, the processing system may be able to combine information from various sensors to further determine the vehicle's surroundings. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, the processing system may use other combinations of sensor data to make determinations about the surrounding environment.

[0113] When operating in autonomous (or semi-autonomous) mode, a vehicle can control its operation with minimal human input. For example, a human operator can input an address into the vehicle, and the vehicle can then be driven to the designated destination without further human input (e.g., the human does not need to steer or touch the brake / accelerator pedal). Furthermore, when the vehicle operates autonomously or semi-autonomously, the sensor system can receive environmental data. The vehicle's processing system can modify the vehicle's control based on the environmental data received from various sensors. In some examples, the vehicle can change its speed in response to environmental data from various sensors. The vehicle can change its speed to avoid obstacles, comply with traffic regulations, etc. When the processing system in the vehicle identifies an object near the vehicle, the vehicle may be able to change its speed or otherwise alter its movement.

[0114] When a vehicle detects an object but lacks high confidence in the detection, it may request a human operator (or a more powerful computer) to perform one or more remotely assisted tasks, such as (i) confirming whether the object actually exists in the surrounding environment (e.g., whether a stop sign is actually present or not), (ii) confirming whether the vehicle's identification of the object is correct, (iii) correcting the identification if it is incorrect, and / or (iv) providing supplementary instructions (or modifying current instructions) to an autonomous or semi-autonomous vehicle. Remotely assisted tasks may also include instructions from the human operator to control the vehicle's operation (e.g., instructing the vehicle to stop at the stop sign if the human operator determines the object is one), although in some scenarios, the vehicle itself may control its own operation based on feedback from the human operator regarding the object's identification.

[0115] To facilitate this, the vehicle can analyze environmental data representing objects in the surrounding environment to identify at least one object with a detection confidence level below a threshold. A processor within the vehicle can be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor can be configured to detect objects that may be important for the vehicle to identify. Such objects may include pedestrians, cyclists, street signs, other vehicles, indicator signals on other vehicles, and various other objects detected in the captured environmental data.

[0116] Detection confidence indicates the likelihood that a identified object is correctly identified or exists in the surrounding environment. For example, a processor can perform object detection on objects within image data in received environmental data and determine that an object has a detection confidence below a threshold based on the fact that at least one object cannot be identified with a detection confidence above a threshold. If the result of object detection or object identification is uncertain, then the detection confidence can be low or below a set threshold.

[0117] The vehicle can detect objects in the surrounding environment in various ways, depending on the source of the environmental data. In some embodiments, the environmental data may come from a camera and is image or video data. In other embodiments, the environmental data may come from LiDAR. The vehicle can analyze the captured image or video data to identify objects in the image or video data. The method and apparatus can be configured to monitor image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle can be configured to identify objects in the surrounding environment based on radar, audio, or other data.

[0118] In some embodiments, the technology used by the vehicle to detect objects can be based on a set of known data. For example, data related to environmental objects can be stored in a memory located within the vehicle. The vehicle can compare the received data with the stored data to determine objects. In other embodiments, the vehicle can be configured to determine objects based on the context of the data. For example, street signs related to construction may typically be orange. Therefore, the vehicle can be configured to detect orange objects located near one side of the road as street signs related to construction. Additionally, when the vehicle's processing system detects objects in the captured data, it can also calculate a confidence level for each object.

[0119] Furthermore, the vehicle may also have a confidence threshold. The confidence threshold can vary depending on the type of object being detected. For example, a lower confidence threshold might be used for objects that may require a rapid response from the vehicle (such as brake lights on another vehicle). However, in other embodiments, the confidence threshold may be the same for all detected objects. When the confidence associated with a detected object is greater than the confidence threshold, the vehicle can assume that the object has been correctly identified and adjust the vehicle's control accordingly based on that assumption.

[0120] The vehicle's actions can vary when the confidence level associated with a detected object is less than a confidence threshold. In some embodiments, the vehicle may react as if the detected object were present, even with a low confidence level. In other embodiments, the vehicle may react as if the detected object were not present.

[0121] When the vehicle detects an object in the surrounding environment, it can also calculate a confidence level associated with the specifically detected object. Depending on the embodiment, the confidence level can be calculated in various ways. In one example, when detecting an object in the surrounding environment, the vehicle can compare environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle can use mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0122] In response to determining that an object has a detection confidence level below a threshold, the vehicle may send a request to a remote computing system for remote assistance in object identification. As discussed above, the remote computing system can take various forms. For example, the remote computing system may be a computing device located within the vehicle, separate from the vehicle, but which a human operator can interact with (such as a touchscreen interface for displaying remote assistance information) while simultaneously acting as a passenger or driver. Additionally or alternatively, as another example, the remote computing system may be a remote computer terminal or other device located at a location not near the vehicle.

[0123] Requests for remote assistance may include environmental data containing the object, such as image data, audio data, etc. The vehicle may transmit the environmental data to a remote computing system via a network (e.g., network 304) and, in some embodiments, via a server (e.g., server computing system 306). A human operator of the remote computing system can then use the environmental data as the basis for responding to the request.

[0124] In some embodiments, when an object is detected as having a confidence level below a confidence threshold, an initial identification of the object may be given, and the vehicle may be configured to adjust its operation in response to the initial identification. Such adjustment may take the form of stopping the vehicle, switching the vehicle to human control mode, changing the vehicle's speed (e.g., rate and / or direction), and other possible adjustments.

[0125] In other embodiments, even if the vehicle detects an object with a confidence level that reaches or exceeds a threshold, the vehicle may operate in accordance with the detected object (e.g., stop if the object is identified as a stop flag with high confidence), but may be configured to request remote assistance while the vehicle is operating in accordance with the detected object (or from a later time).

[0126] Figure 4 This is a block diagram of a system according to an example embodiment. Specifically, Figure 4 System 400 is shown, which includes system controller 402, radar system 410, sensor 412, and controllable component 414. System controller 402 includes processor(s) 404, memory 406, and instructions 408, which are stored in memory 406 and can be executed by processor(s) 404 to perform functions such as those disclosed herein.

[0127] The (multiple) processors 404 may include one or more processors, such as one or more general-purpose microprocessors (e.g., having a single core or multiple cores) and / or one or more special-purpose microprocessors. One or more processors may include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also envisioned herein.

[0128] The memory 406 may include computer-readable media, such as non-transitory computer-readable media, which may include, but is not limited to, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid-state drive (SSD), hard disk drive (HDD), compact disc (CD), digital video disc (DVD), digital magnetic tape, read / write (R / W) CD, R / W DVD, etc.

[0129] Radar system 410 can be used in autonomous or semi-autonomous vehicles for navigation and object detection by detecting and measuring the distance, velocity, and orientation of objects in the surrounding environment using radio waves. Radar system 410 may include one or more radar units, each comprising a radar transmitter that emits radio waves and a radar receiver that captures reflected waves from objects. By analyzing the time taken for the waves to return and their frequency shift (Doppler effect), radar system 410 can determine the presence, location, and movement of objects.

[0130] In the context of autonomous or semi-autonomous vehicles, radar system 410 provides measurements that can assist navigation and collision avoidance. Radar units are typically mounted externally to the vehicle, such as at the front, rear, and sides. During navigation, radar system 410 can continuously emit radio waves in all directions, scanning the environment around the vehicle. When the waves encounter an object, they bounce back to the radar receiver, allowing radar system 410 to analyze the reflected waves to calculate the object's distance, relative velocity, and angle. The vehicle's control system can use this information to make decisions and adjust the vehicle's trajectory accordingly, enabling it to detect and react to obstacles, pedestrians, vehicles, and other potential hazards in its path. By providing real-time data about the surrounding environment, radar system 410 can enhance the vehicle's perception capabilities and contribute to safer and more reliable navigation.

[0131] In some respects, radar system 410 offers operational benefits superior to other types of sensors, such as cameras and lidar. Radar performs well in adverse weather conditions, such as rain, fog, or dust, where other sensors may be limited. In particular, the radio waves emitted by radar system 410 can penetrate these adverse conditions and provide reliable object detection. This makes radar particularly useful for enhancing the robustness and safety of autonomous or semi-autonomous vehicles in a variety of weather scenarios. Furthermore, radar excels at detecting the speed and relative velocity of nearby objects, which is useful for assessing the movement of surrounding vehicles, pedestrians, and other obstacles. By providing accurate velocity information, radar system 410 enables the vehicle (or its driver) to make informed decisions about potential collision risks and adjust its behavior accordingly. In some cases, radar system 410 can also provide a longer measurement range and a wider field of view compared to other sensors coupled to the vehicle.

[0132] Similarly, system controller 402 can use the outputs from radar system 410 and sensor 412 to determine the characteristics of system 400 and / or the characteristics of the surrounding environment. For example, sensor 412 may include one or more of a GPS, IMU, image capture device (e.g., camera), light sensor, thermal sensor, one or more lidar devices, and other sensors indicating parameters related to system 400 and / or the surrounding environment. For illustrative purposes, radar system 410 is depicted as separate from sensor 412, and in some examples may be considered part of or considered as sensor 412.

[0133] Based on characteristics of system 400 and / or the surrounding environment determined by system controller 402 from outputs from radar system 410 and sensor 412, system controller 402 can control controllable components 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case controllable components 414 may include the vehicle's braking system, turning system, and / or acceleration system, and system controller 402 may modify aspects of these controllable components based on characteristics determined from radar system 410 and / or sensor 412 (e.g., when system controller 402 controls the vehicle in autonomous or semi-autonomous mode). In this example, radar system 410 and sensor 412 may also be controlled by system controller 402.

[0134] Radar system 410 and sensor 412 can be used to detect and interpret the surrounding environment during vehicle navigation. However, in some cases, the performance of radar system 410 and / or sensor 412 may experience data loss for at least a threshold duration, which may trigger action of system 400. In some cases, sensor degradation may lead to abnormal driving behavior of autonomous or semi-autonomous vehicles using radar system 410, as the vehicle system may be unable to accurately perceive the environment due to degraded radar performance.

[0135] To address the potential impact on radar data and / or other types of sensor data, the disclosed solution can be executed by one or more vehicle systems to estimate the sensitivity and performance of vehicle sensors, including radar. For example, system 400 can receive radar data representing the vehicle's environment from radar coupled to the vehicle and detect one or more objects located in the environment based on the radar data. System 400 can also filter the radar data corresponding to one or more objects based on a predetermined set of criteria to identify specific objects corresponding to a first type of object. System 400 can then perform a comparison between radar parameters determined based on the radar data corresponding to the specific object and expected radar parameters represented by a data model. The data model can be generated based on radar data aggregated for multiple objects matching the first type of object. System 400 can then estimate a radar sensitivity loss based on this comparison and perform one or more actions when the loss exceeds one or more thresholds.

[0136] Figure 5 This is a conceptual diagram of a system for estimating the sensitivity and performance of vehicle radar and other types of vehicle sensors. In an example embodiment, the system includes a vehicle 500, a remote computing system 520, and a vehicle 522, and may include... Figure 5 Additional entities not shown. Vehicle 500 includes computing device 502, radar system 504, sensor 506, control system 508, and memory 510. Memory 510 is illustrated as storing data model 512 and loss threshold 518, and may also be used to store additional information, such as instructions for performing the operations disclosed herein. Figure 5 The elements of the system depicted are shown for illustrative purposes and may include more or fewer elements in other examples. For example, remote computing system 520 and vehicle 522 may each represent any number of computing systems and vehicles, respectively.

[0137] In some respects, vehicle 500 can enhance the safety and reliability of its autonomous operation and / or ADAS operation by evaluating the performance of the radar within radar system 504 in real time during navigation. For example, computing device 502 can perform the operations described herein to detect changes in the quality of radar data (or another type of sensor data) and determine whether these detected changes should trigger some type of corrective action in the vehicle system. In some cases, computing device 502 can detect that signal loss has temporarily reduced the maximum sensing range of one or more sensors used by vehicle 500. In response, computing device 502 and / or control system 508 reduce the rate and processing of vehicle 500, or perform other corrective actions that take the reduced sensor performance as a factor. In some instances, computing device 502 can provide one or more alerts via wireless communication 524, such as to remote computing system 520 or vehicle 522. For example, vehicle 500 can use the disclosed techniques to determine when one or more radars within radar system 504 are operating in a degraded state, which may be due to various factors such as weather conditions, radome conditions, and / or hardware conditions. Vehicle 500 can then provide alerts, adjust the constant false alarm rate (CFAR) detection threshold, trigger cleaning operations at sensors, and / or perform other operational modifications, such as deceleration, increasing the following distance relative to other vehicles (e.g., vehicle 522), and applying braking earlier in certain situations.

[0138] exist Figure 5 In the examples shown, vehicle 500 and vehicle 522 represent any type of mobile machine designed to transport passengers or goods from one location to another. For example, vehicle 500 could be a car, truck, bus, semi-truck, construction vehicle, or another type of mobile machine. In some cases, vehicle 500 could be a robotic device or another specialized vehicle. In some examples, vehicle 500 is an autonomous or semi-autonomous vehicle that uses sensor data from radar system 504 and sensor 506 to understand its surroundings in order to navigate safely between locations. Vehicles 500 and 522 can be capable of autonomous navigation, manual navigation, or a combination of autonomous and manual navigation. For example, a driver can switch vehicle 500 between an autonomous operating mode and a manual mode, in which the vehicle 500 safely navigates to its destination while receiving little or no input from the driver, and in a manual mode, the driver is able to control vehicle 500.

[0139] Computing device 502 controls one or more operational aspects of vehicle 500. Computing device 502 may represent one or more computing devices configured to perform operations of vehicle 500, including the methods described herein. In some aspects, computing device 502 includes one or more processors and a memory. The memory may store instructions that, when executed by one or more processors, cause computing device 502 to perform various operations, including operations related to estimating and evaluating the sensitivity and performance of radar system 504 and sensor 506. The memory may be memory 510, another memory specific to computing device 502, and / or a memory located remotely from vehicle 500 (e.g., a memory located at a remote computing system 520). Operations performed by computing device 502 may include receiving radar data representing the environment of vehicle 500 from radar coupled to vehicle 500, and identifying one or more objects in the environment corresponding to specific types of objects based on the radar data. Then, computing device 502 can filter radar data corresponding to one or more objects based on a predetermined set of criteria (e.g., range, line of sight, and / or size), and perform a comparison between one or more radar parameters determined based on the filtered radar data corresponding to one or more objects and expected radar parameters represented by a data model (e.g., data model 512). Data model 512 is generated based on radar data aggregated for multiple objects matching a specific type of object. Computing device 502 can then estimate the radar sensitivity loss based on this comparison.

[0140] Computing device 502 can receive radar data generated by radar system 504, which measures the environment surrounding vehicle 500. The radar data can be provided by one or more radars that are part of radar system 504 and can be processed in various formats, such as radar images. For example, radar system 504 can output radar data in the form of range-Doppler maps (RDMs), which are two-dimensional representations of the distance of an object from a radar sensor and its relative velocity. The distance dimension indicates how far away the object is, while the Doppler dimension shows whether the object is moving toward or away from the radar and at what rate. RDMs can be used to detect and track objects moving in the environment, such as other vehicles (e.g., vehicle 522), pedestrians, and cyclists. In some cases, radar data can be taken in the form of range-angle images, which provide a two-dimensional view where one axis represents the distance to the target and the other axis represents the angle of arrival of the radar signal. Range-angle images can be useful for determining the azimuth or orientation of an object relative to vehicle 500, which can help create a spatial map of the environment surrounding vehicle 500. Radar data can also be received in other formats, such as voxels.

[0141] The computing device 502 can also receive and use sensor data provided by sensor 506, which may include cameras, lidar, temperature sensors, wind sensors, humidity sensors, inertial measurement units (IMUs) and / or ultrasonic sensors, as well as other possible types. Sensor 506 can output data measuring various environmental and weather conditions, such as temperature, humidity, wind speed, visibility, rain, fog, or snow. Thus, the computing device 502 and the control system 508 can use sensor data from radar system 504 and / or sensor 506 to navigate safely and effectively in dynamic environments.

[0142] Control system 508 can be used to manage and regulate the operation of vehicle 500. In some examples, control system 508 and computing device 502 can be part of a computing system for performing various operations of vehicle 500, including determining and executing control strategies based on sensor measurements of the surrounding environment. Thus, control system 508 can perform a wide range of functions, from basic vehicle maneuvering to complex decision-making processes for autonomous or semi-autonomous navigation. In some aspects, control system 508 can perform functions such as steering control, acceleration and braking, stability and traction control, navigation and route planning, and obstacle detection and avoidance. Components within control system 508 may include additional sensors, actuators, electronic control units (ECUs), and communication networks. In some examples, control system 508 can work with computing device 502 to process data from various sensors to perceive the environment, make complex decisions based on the data, and control the movement of vehicle 500 accordingly. Control system 508 can also continuously monitor performance and safety parameters to ensure that vehicle 500 operates within safe limits.

[0143] Computing device 502 and other vehicle systems may use memory 510, which can be used to store data and models. In this example, memory 510 is shown storing data model 512 and loss threshold 518, as well as other information. Data model 512 can convey expected radar parameters arranged according to the type of object, including expected SNR data 514 and RCS data 516 for each type of object. In some examples, data model 512 represents expected SNR values ​​(shown as SNR data 514) and RCS values ​​(shown as RCS data 516), which are organized by object type and range when acquired by an optimally operating radar in clear weather conditions. In some examples, expected values ​​may represent range and / or specific values.

[0144] Data model 512 can be trained using aggregated radar data from radars on a large number of vehicles. For example, data model 512 can be developed using a large dataset of radar measurements collected from multiple vehicles equipped with optimally functioning radars positioned similarly on each vehicle. This dataset of radar measurements can include SNR and RCS values ​​for various common object types, such as passenger vehicles, trucks, motorcycles, pedestrians, and static objects (such as road signs or obstacles). Radar measurements can be collected under controlled conditions, particularly during clear weather, to establish a baseline for optimal radar performance. Data collection can occur across different times of day and in various geographical locations to account for potential environmental variations. In some cases, the dataset of radar measurements can be updated over time, enabling the generation and distribution of updated models to vehicles. For example, vehicles can continue to supply updated versions of radar measurements that can be used to further train data model 512. The dataset of radar measurements can also be supplemented and / or validated using radar data generated by offline radars that produce measurements in a radar room.

[0145] For each object type, data model 512 can aggregate SNR and RCS values ​​as a function of range. This may involve collecting multiple measurements for each object type at different distances from the radar, ranging from very close proximity to the radar's maximum effective range. Data model 512 can be trained on data collected from radars at different installation locations on a vehicle, such as radars mounted at the front, rear, and sides. This allows data model 512 to account for variations in radar performance based on the radar's position and orientation on the vehicle. In some examples, statistical techniques can be applied to the aggregated data to derive expected SNR and RCS values ​​for each object type at various ranges. This may include calculating the mean, standard deviation, and confidence intervals to capture the typical measurement range for each object type and distance. Data model 512 can also consider specific characteristics of the radar, such as its operating frequency, beamwidth, and other relevant parameters. This allows data model 512 to be tailored to the performance expectations of a specific radar hardware configuration.

[0146] Machine learning algorithms, such as regression models or neural networks, can be used to learn the relationships between object types, ranges, and expected SNR and RCS values. These algorithms can help capture complex patterns and dependencies in radar measurement datasets. Additionally, a separate test dataset can be used to validate the data model 512 to ensure its accuracy and generalizability across different scenarios and radar configurations. This validation process can involve comparing the predictions of the data model 512 with actual measurements from radars known to be operating at their best. As new data becomes available, periodic updates to the data model 512 can be performed, keeping it in sync with the latest radar technologies and performance characteristics. This can involve retraining the data model 512 with newly collected data or fine-tuning existing model parameters. By training the data model 512 in this way, a remote computing system 520 or another computing system can establish a robust reference to the expected radar performance under optimal conditions, which can then be used to identify deviations that may indicate radar sensitivity loss or other problems. In some examples, the remote computing system 520 can train the data model 512 and distribute it to vehicles 500 and 522, which may involve the use of wireless communication 524.

[0147] The loss threshold 518 can represent one or more thresholds that can be used to evaluate radar classification, including estimated losses associated with radar system 504 and individual radars within radar system 504. For example, computing device 502 can use one or more thresholds to evaluate estimated radar losses to determine whether corrective action is needed.

[0148] In some cases, computing device 502 can compare the estimated radar loss to a single overall threshold. This threshold can represent the maximum acceptable level of sensitivity loss before vehicle 500 requests action. If the estimated loss exceeds this threshold, computing device 502 can trigger an alarm or initiate corrective actions, such as adjusting the operation of vehicle 500, triggering sensor cleaning actions, and / or reducing how the vehicle system utilizes radar data from radar system 504. In other cases, computing device 502 can use multiple thresholds to assess the estimated radar loss. These thresholds can correspond to different levels of sensitivity loss severity, such as low, medium, and high. Each threshold can be associated with a specific action or response triggered by computing device 502.

[0149] The loss threshold 518 may include one or more dynamic thresholds that can be adjusted based on factors such as vehicle speed, weather conditions, and / or the specific radar's position on vehicle 500. For example, when the vehicle is traveling at a high speed, the threshold for a forward-facing radar may be lower, reflecting the increased value of long-range detection in such a scenario. Thus, the computing device 502 can apply different thresholds for different object types or ranges. For example, there may be a more stringent threshold for the sensitivity loss in detecting pedestrians at close range compared to detecting other vehicles at greater distances.

[0150] In some examples, computing device 502 may use time-based thresholds. For example, a lower level of sensitivity loss may be acceptable for a short duration, but if the lower level of sensitivity persists beyond a certain time threshold, it may trigger a more severe response. Computing device 502 may use statistical thresholds based on the radar's historical performance. For example, computing device 502 may flag sensitivity losses that significantly deviate from the radar's typical performance range, which may involve using measurements such as the standard deviation from the mean. In some cases, computing device 502 may use relative thresholds that compare the performance of one radar to other radars on vehicle 500. For example, this may trigger an alarm when a radar shows a significantly higher sensitivity loss compared to other radars under similar conditions, even if the radar's performance does not exceed an absolute threshold.

[0151] Computing device 502 can implement adaptive thresholds based on radar performance history adjusted over time. This approach helps identify gradual degradation that might not trigger a fixed threshold. Furthermore, multiple thresholds can be combined to create a decision matrix when assessing radar loss. For example, computing device 502 can consider both the magnitude and duration of sensitivity loss when determining an appropriate response. Thresholds can also be linked to specific vehicle functions or safety features. Exceeding certain thresholds may result in gradual degradation of some features, rather than immediate disabling by computing device 502, which can smooth out the overall degradation of vehicle 500 functionality. By using one or more thresholds in these ways, computing device 502 can provide a flexible and context-aware assessment of estimated radar loss, enabling it to trigger appropriate responses to various levels of sensitivity degradation.

[0152] In some cases, computing device 502 can determine a real-time estimate of the radar sensitivity degradation of the entire radar system 504. Computing device 502 can also assess the radar sensitivity degradation of individual radars operating as part of radar system 504. Real-time estimates can indicate the loss experienced by one or more radars, enabling immediate adjustments to control strategies for radar system 504, individual radars within radar system 504, and / or vehicle 500. For example, if it is determined that the radar sensitivity is greater than a loss threshold 518, control system 508 can cause vehicle 500 to decelerate and / or implement other preventative measures. Reducing the speed of vehicle 500 can help prevent vehicle 500 from driving too fast relative to the sensor coverage of its surroundings. In some cases, weather or other environmental conditions and / or hardware problems can reduce the maximum sensing range of radar system 504 and / or sensors 506, which can trigger computing device 502 and / or control system 508 to reduce the speed of vehicle 500 to accommodate the reduced operational capability.

[0153] Vehicle 500 can participate in wireless communication 524 with remote computing system 520 and vehicle 522. Wireless communication 524 can allow data exchange or collaborative processing between vehicle 500, remote computing system 520, and vehicle 522. For example, vehicle 500 can provide information and sensor data to remote computing system 520 and / or vehicle 522. Similarly, vehicle 500 can receive information and sensor data from remote computing system 520 or vehicle 522. As an example, remote computing system 520 can aggregate radar data from a large number of vehicles and use the radar data to generate (or update) a data model 512 for distribution to vehicles, including vehicle 500 and vehicle 522. This collaborative processing can enhance the accuracy and reliability of radar sensitivity detection and assessment by leveraging the scale of radar, vehicle, and computing resources.

[0154] As vehicle 500 navigates its environment, computing device 502 can receive radar data in real time. For example, computing device 502 can continuously receive new radar data reflecting the dynamic nature of the environment, including the movement of vehicle 500 and various objects within the environment. In other cases, radar data can be received by computing device 502 at predetermined intervals or in response to specific events or conditions. Objects represented in the radar data can include various entities present in the surrounding environment. For example, objects can include other vehicles (e.g., vehicle 522), pedestrians, animals, buildings, trees, signs and traffic signals, and / or any other physical entities that can be detected by radar system 504. The radar data can provide computing device 502 with a detailed representation of the environment surrounding vehicle 500, thereby providing useful information to perception and control systems 508 for use in controlling vehicle 500.

[0155] In some cases, computing device 502 may receive sensor data in raw or unprocessed form and may require further processing to extract useful information. For example, processing of radar data can involve various signal processing techniques such as filtering, amplification, demodulation, or decoding. The processed radar data can then be used for various purposes, such as object detection, object tracking, collision avoidance, or radar sensitivity estimation. In some cases, computing device 502 may perform processing techniques. In some examples, a field-programmable gate array (FPGA), a dedicated processor, or other processor may be locally located at an individual radar and configured to perform processing techniques (or part of processing techniques to assist computing device 502). Radar system 504 may include one or more processors that can process radar data and perform other operations described herein.

[0156] Radar data, including detection or trajectory data, can be fed into machine learning algorithms within a vehicle perception system. These algorithms combine radar information with data from other sensors, such as cameras, lidar, and ultrasonic sensors, to create sensor-fused perceived objects. These objects represent high-probability entities in the vehicle environment and contain inferred information such as object type classification (e.g., bicycle, motorcycle, sedan, smaller car, or truck), location, speed, size, dimensions, orientation, and predicted trajectory. In some cases, machine learning algorithms provide traceability, indicating which specific radar data segments contributed to each perceived object. For example, this can be represented as an index into a radar detection or tracking list. In some cases where the algorithm fails to explicitly provide this information, heuristics can be used to re-associate radar data with perceived objects. For example, heuristics could include proximity matching, speed correlation, and range / Doppler range comparison.

[0157] The correlation between raw radar data and fused sensing objects serves useful purposes, including enabling performance evaluation and tuning of sensor fusion systems, allowing for the refinement of radar processing algorithms based on higher-level perception results, and facilitating the implementation of radar-specific object tracking or prediction methods. Furthermore, this link helps evaluate the contribution of radar data to overall perception accuracy and enables radar-based backoff or redundancy in the event of failures in other sensor modes. By maintaining the connection between raw radar data and fused sensing objects, the perception system can leverage the advantages of radar, such as direct velocity measurement and all-weather performance. Simultaneously, the perception system can utilize the complementary capabilities of other sensors, resulting in a more robust and accurate understanding of the surrounding environment. This integrated approach enhances the vehicle's ability to navigate complex scenarios and make informed decisions based on comprehensive perception of its surroundings.

[0158] Computing device 502 (or another processor) can filter radar data representing one or more objects based on a predetermined set of criteria. For example, computing device 502 can apply one or more filters to the entire radar image generated by radar system 504, or focus on specific aspects of the radar image, such as bins of radar data corresponding to objects identified by computing device 502. The set of predetermined criteria can include one or more factors, such as the range of the object from the vehicle, the azimuth of the object relative to the vehicle, the object type, and / or weather conditions. For example, computing device 502 or another processor can filter out objects that are beyond a certain range from vehicle 500. This range-based filtering can help focus radar sensitivity estimation on objects within a relevant (or selected) distance from vehicle 500, which can improve the accuracy and applicability of the estimation. In some cases, the predetermined criteria may also include factors that consider the direct line of sight between the radar sensor and each object. Computing device 502 can filter out objects that are not within a certain degree of the radar's direct line of sight. In this way, the computing device 502 can use line-of-sight filtering to exclude objects that are otherwise obscured by other objects or structures, which could otherwise interfere with radar signals and affect the accuracy of radar sensitivity estimation.

[0159] In some examples, computing device 502 can also perform weather-specific filtering of radar data. Weather conditions in the vehicle environment can affect radar performance, including the sensitivity of radar system 504. For example, heavy rain or snow can attenuate radar signals and reduce radar sensitivity. Similarly, road spray kicked up by another vehicle or the tires of vehicle 500 can degrade radar performance. Thus, computing device 502 can adjust filtering criteria based on current weather conditions to account for their impact on radar sensitivity. In some cases, weather-specific filtering may involve excluding or giving less weight to radar data acquired under adverse weather conditions, or adjusting the expected SNR or RCS values ​​from data model 512 based on weather conditions. Weather-specific filtering enables computing device 502 to accurately estimate radar sensitivity loss under various weather conditions.

[0160] To assess the loss, computing device 502 can perform a comparison between radar parameters (e.g., SNR data or RCS data) corresponding to the filtered objects and data model 512, which represents the expected radar parameters (e.g., SNR data and RCS data) aggregated for objects of the same type as the filtered objects. Data model 512 can be generated based on aggregated SNR and RCS data for a large number of objects of the same type as the filtered objects. Data model 512 can be based on aggregated radar data collected under various conditions and scenarios, enabling the development of expected trends and their use as a baseline for comparison. For example, the comparison may involve calculating the differences between the actual SNR data, RCS data, and / or other radar parameters determined based on radar data corresponding to vehicle 522 and the expected SNR data, RCS data, or other expected parameters specified by data model 512 for vehicles of similar size to vehicle 522. In some cases, the differences can be calculated individually for each object, and the results can be averaged to obtain an overall estimate of the radar sensitivity loss. In other cases, the differences can be calculated collectively for a group of objects, and the results can be used as an estimate of the radar sensitivity loss. In other examples, other suitable comparison methods can be used.

[0161] In some cases, comparisons may involve correcting the detected SNR, RCS, and / or other radar parameters of an object based on its relative position. The relative position of an object may refer to its location relative to a radar or vehicle 500. Relative position can affect the strength of the radar signal reflected from the object and received by the radar, and therefore has an impact on the SNR and RCS data measured against the object. For example, an object directly in front of the radar may reflect a stronger radar signal than an object located to its side. Therefore, computing device 502 may adjust the detection of SNR, RCS, or other radar parameters associated with the object based on its relative position to account for this effect. Thus, the correction of detected radar parameters (e.g., SNR, RCS data) may involve applying a correction factor determined based on the object's relative position. This correction factor can be derived from the radar's directivity profile, which describes how the strength of the radar signal varies with the angle of incidence. The directivity profile can be a known characteristic of the radar or can be estimated based on empirical data. By applying a correction factor to the detection parameters (e.g., SNR and RCS data), the computing device 502 can obtain a corrected SNR or RCS that provides a more accurate representation of radar sensitivity.

[0162] Corrected radar parameters (e.g., SNR data, RCS data) measured from one or more objects in the environment of vehicle 500 can be used by computing device 502 in a comparison with data model 512 to estimate radar sensitivity loss. In some cases, the estimated radar sensitivity loss can represent a measure of the overall degradation of radar system 504. In other cases, the estimated radar sensitivity loss can represent the ability of an individual radar to detect objects in its environment. The estimated radar loss experienced by radar system 504 and / or individual radars can be used by the vehicle system in real time to identify situations where one or more radars rapidly experience sensitivity degradation in a scene. The vehicle system can also use the estimated loss to monitor the sensitivity of one or more radars over time to identify potential hardware failures or when radome conditions lead to reduced sensitivity. In some cases, mud, insect splashes, or other debris may cover portions of the radar radome and affect the radar's transmission and reception of radar signals until the radome is cleaned or replaced.

[0163] In some cases, the estimated radar sensitivity loss can be compared to a threshold to determine whether the radar sensitivity has significantly degraded. If the estimated radar sensitivity loss exceeds the threshold, the computing device 502 can generate an alarm or take corrective actions, such as adjusting the radar's operation or initiating diagnostic procedures to identify potential hardware faults. In some situations, the threshold can be set based on radar specifications, vehicle 500 requirements, or environmental conditions. The threshold can be adjustable to accommodate changes in radar operating conditions or performance requirements.

[0164] In some respects, the estimated radar sensitivity loss can be used by computing device 502 or another processor to adjust the operation of one or more radars and / or vehicle 500. For example, if the estimated radar sensitivity loss indicates that the radar's ability to detect objects at a certain range has been degraded, computing device 502 can adjust the radar's operating parameters, such as its power output or gain settings, to improve its detection performance at that range. Alternatively, computing device 502 can adjust the control strategy of vehicle 500, such as its rate or route, to mitigate the impact of degraded radar sensitivity on the safety and efficiency of vehicle 500.

[0165] In some cases, a data model 512 can be generated based on aggregated SNR and RCS data for a large number of objects of the same type as the filtered objects, for comparison with the SNR or RCS data of the filtered objects. Aggregated data can be collected under various conditions and scenarios, and can be processed to derive the expected SNR or RCS values ​​of the objects. Data aggregation can involve combining the SNR and RCS data of the objects into a function of range, which can create a comprehensive data model reflecting the expected radar performance within the range. Aggregated data can be collected from unobstructed localization correction detection for a specific type of sensed object. Localization correction can involve adjusting the detection data based on the relative positioning of the sensed object and the radar sensor or vehicle from which the radar measurements are obtained. This localization correction may help to explain changes in radar signal strength due to the relative positioning of the objects, thereby improving the accuracy of the aggregated data and the resulting data model.

[0166] In some cases, aggregated data can be collected for specific object types, such as cars or trucks. The choice of specific object type helps ensure that the aggregated data and the resulting data model represent typical objects that the radar expects to detect in its environment. This can help improve the relevance and accuracy of radar sensitivity estimates. In some situations, aggregated data can be collected over many radar runs or operational cycles, providing a large and diverse dataset for generating data model 512. Large datasets can help capture variations in radar performance under different conditions and scenarios, which enhances the robustness and reliability of data model 512.

[0167] In some respects, the estimated radar sensitivity loss can be applied in real time by the computing device 502 as a loss estimator. This real-time sensitivity monitoring can provide timely feedback on radar performance, enabling the computing device 502 to take rapid adjustments or corrections when necessary. In some cases, real-time sensitivity monitoring may involve calculating the range-based loss for each detected object. The range-based loss can be determined by subtracting the detected SNR or RCS from the corresponding value in the data model 512. If the range-based loss exceeds a certain threshold, the computing device 502 can conclude that the radar sensitivity has been significantly degraded. The threshold can be set based on radar specifications, the requirements of the vehicle 500's navigation system, and / or environmental conditions.

[0168] Figure 6This is a flowchart of a method 600 for estimating radar sensitivity. Method 600 may include one or more operations, functions, or actions as illustrated by one or more of blocks 602, 604, 606, 608, and 610. Although the blocks are illustrated in sequential order, in some instances these blocks may be executed in parallel, and / or in an order different from that described herein. Furthermore, depending on the desired implementation, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed.

[0169] Additionally, for method 600 and other processes and methods disclosed herein, the flowchart illustrates the functionality and operation of one possible implementation of this embodiment. In this regard, each block may represent a module, segment, or portion of program code, which includes one or more instructions executable by a processor for use in implementing a specific logical function or step in the process. The program code may be stored on any type of computer-readable medium or memory, such as a storage device including a disk or hard disk drive. Various types of computing systems can be used to perform method 600 and the other methods described herein.

[0170] In some examples, the computing system performing the disclosed techniques may include one or more processors, which may be part of the same or different computing devices. The computing system may be located on a vehicle and may communicate with one or more remotely located computing devices. In some examples, a dedicated type of processor may perform method 600. For example, one or more application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs) may perform method 600. As an example, an ASIC or FPGA may perform one or more blocks of method 600. In some cases, the computing system may perform method 600 for different radars to compare the relative performance of the radars. Method 600 may also be performed to evaluate the performance of other types of sensors.

[0171] At block 602, method 600 relates to receiving radar data from a radar coupled to a vehicle. The radar data represents the environment outside the vehicle. A computing system can receive radar data from one or more radars in various ways. For example, the computing system can be connected to the radar via the vehicle's internal communication network, such as a Controller Area Network (CAN) bus or Ethernet. This allows radar data to be transmitted in real time from multiple sensors to a central computing system. In some cases, radar sensors can wirelessly transmit data to the computing system using protocols such as Wi-Fi, Bluetooth, or Dedicated Short Range Communication (DSRC).

[0172] The computing system can directly receive raw radar data from the sensors, which may include information such as the signal strength, phase, and time of flight of each detected object. This raw data can then be processed by the computing system to extract relevant information. Alternatively, the radar can perform some initial processing and send pre-processed data to the computing system. This data may include a list of detected objects with information such as the range, velocity, and angle of each detected object. The computing system can receive radar data at regular intervals (e.g., every 20 milliseconds) to maintain an up-to-date representation of the vehicle's surrounding environment. The data transmission rate can be adjusted based on the vehicle's speed or the complexity of the environment. In some examples, the computing system can proactively request data from the radar as needed, rather than receiving a continuous data stream. This approach can be used to save bandwidth or processing power.

[0173] In some cases, computing systems can use data fusion techniques to combine information from multiple radars to create a comprehensive view of the vehicle's surroundings. This can involve aligning data from different sensors based on their positioning and orientation on the vehicle. To process large amounts of data from multiple high-resolution radars, computing systems can use high-speed data buses or dedicated hardware accelerators for efficient data transfer and processing. The computing system can also receive radar data metadata, such as timestamps, sensor health status, and calibration information. This additional information can be used to accurately interpret and utilize the radar data.

[0174] In some examples, the computing system can dynamically adjust data reception parameters based on the vehicle's current needs. For instance, when navigating complex environments or performing specific maneuvers, the computing system may request more frequent updates from certain radars.

[0175] In some examples, radar can utilize advanced signal processing techniques to generate 2D images from received radar echoes. For instance, a radar image can represent a view of the environment, where range is on one axis and azimuth (or Doppler) is on another. In some cases, radar can use multiple receiver antennas or phased arrays to achieve 2D imaging capabilities. By processing the phase difference between signals received at different antennas, the system can create a 2D image of the surrounding environment. For 3D imaging, radar can use techniques such as multiple-input multiple-output (MIMO) configurations or frequency-modulated continuous-wave (FMCW) waveforms to generate voxel data. Voxels represent 3D volumetric pixels, thus providing information about range, azimuth, and elevation. Radar systems can combine digital beamforming techniques to create high-resolution 2D images or 3D voxel maps. This approach allows for flexible and adaptive beam steering, enabling the creation of detailed spatial representations of the environment.

[0176] In some implementations, radar can combine data from multiple sweeps or scans to construct a 2D image or 3D voxel representation over time. This can help improve resolution and reduce noise in the resulting image. Radar can use machine learning algorithms to process raw radar data and generate 2D images or voxel representations. These algorithms can help extract meaningful features and patterns from radar echoes, thereby enhancing the quality of the resulting image. In some cases, the computing system can fuse data from multiple radars and / or other sensor types, such as lidar or cameras, to create a more comprehensive 2D image or voxel map of the environment. This sensor fusion approach can help overcome the limitations of individual sensors and provide a more complete picture of the surrounding environment. Vehicle systems can use dedicated hardware accelerators or graphics processing units (GPUs) to efficiently process the large amounts of data required for real-time generation of 2D images or voxel representations. By receiving and processing radar data in these ways to generate 2D images or voxels, the system can provide spatial information about the vehicle's environment, enabling more advanced object detection, tracking, and scene understanding capabilities.

[0177] At block 604, method 600 relates to detecting one or more objects in an environment based on radar data. The computing system can receive raw radar data from one or more radar sensors mounted on a vehicle. This data typically consists of reflected radar signals from objects in the environment. In some cases, the system can preprocess the raw radar data to reduce noise and improve signal quality. This can involve techniques such as filtering, amplification, or signal averaging.

[0178] The computational system can then apply signal processing algorithms to the preprocessed radar data to extract relevant features. These features can include the range, velocity, angle, and radar cross-section of potential objects. The system can use detection algorithms to identify clusters or patterns in the processed radar data that may correspond to physical objects. These algorithms can use techniques such as constant false alarm rate (CFAR) detection or adaptive thresholding processes. In some examples, the computational system can apply tracking algorithms to correlate detections across multiple radar scans, allowing objects to be tracked over time and their trajectories estimated.

[0179] At box 606, method 600 involves filtering radar data corresponding to one or more objects to identify specific objects corresponding to objects of a first type. The computing system may perform the filtering process based on a predetermined set of criteria. In some examples, the set of predetermined criteria includes a first predetermined criterion and a second predetermined criterion, the first predetermined criterion conveying a range filter for distinguishing one or more objects from additional objects in the environment based on the range associated with each object, and the second predetermined criterion selecting one or more objects based on each of the one or more objects having a direct line of sight relative to the radar.

[0180] In some examples, the computational system can apply object recognition and / or classification algorithms to processed radar data. These algorithms can analyze features such as size, shape, speed, and RCS to differentiate based on object type. For example, the computational system can distinguish passenger vehicles from other object types. In some cases, the computational system can use machine learning models, such as neural networks, trained on large datasets of radar signatures from various object types. These models can be able to identify the unique radar echoes associated with passenger vehicles. Furthermore, the computational system can use tracking algorithms to observe the behavior of objects over time. For example, passenger vehicles typically exhibit specific motion patterns and speeds, which can help distinguish them from static objects or other road users, such as pedestrians or cyclists.

[0181] In some examples, the computational system can apply feature extraction techniques to radar data to identify key characteristics of objects (e.g., passenger vehicles), such as their typical length, width, and height ranges. These characteristics can be compared to predefined templates or models of the object type. The computational system can utilize Doppler information from the radar data to analyze the velocity distribution of detected objects. For example, the computational system can leverage the fact that passenger vehicles typically have different velocity characteristics, which can be used to help distinguish them from other object types.

[0182] In some implementations, the computing system can combine radar data with information from other sensors, such as cameras or lidar, to improve object classification accuracy. This sensor fusion approach can aid in confirming passenger vehicle identification by cross-referencing radar detection with visual or point cloud data. The computing system can apply contextual analysis, taking into account factors such as the object's position on the road, its movement relative to lane markings, and its interactions with other objects. This contextual information can help refine the classification of the object as a passenger vehicle. In some examples, the radar system can use micro-Doppler analysis to detect specific features of passenger vehicles or other desired objects, such as wheel rotation or windshield wiper movement, which can provide additional cues for identification. The computing system can use probabilistic methods to assign confidence scores to object classifications. Objects with high confidence scores as passenger vehicles can be treated differently in subsequent processing or decision-making steps.

[0183] The computing system can filter radar data corresponding to one or more objects in various ways based on a predetermined set of criteria. For example, the computing system can apply a range threshold to exclude objects that are more than a certain distance from the vehicle. Range filters can help focus on objects within the relevant operating range and reduce computational load. In some cases, the range threshold can be adjusted based on factors such as vehicle speed or environmental conditions.

[0184] The computational system can use sensor data or environmental models to determine if an unobstructed path exists between the radar and the detected object. Objects obscured by other vehicles, buildings, or terrain features can be filtered out to improve the accuracy of sensitivity loss estimation. Furthermore, the computational system can adjust filtering parameters based on current weather conditions. For example, during heavy rain or snow, the filter can account for increased signal attenuation and potential false detections. Weather filters can also consider how different weather conditions affect various object types differently.

[0185] The computational system can also filter out objects based on their relative velocity. For example, predefined criteria can include velocity-based criteria that can help exclude stationary objects or objects moving at rates inconsistent with those of typical road users, thus focusing the analysis on relevant dynamic objects. In some cases, the computational system can use velocity filters to identify stationary objects (e.g., parked vehicles) for estimating radar sensitivity and performance.

[0186] In some examples, predefined criteria can filter objects with very weak echo signals to reduce noise and focus on more reliable detections. The threshold of this filter can be dynamically adjusted based on estimated radar sensitivity. The computational system can also apply different filtering criteria based on the type of object detected. For example, the system can use different range thresholds for large vehicles and pedestrians. In some cases, objects detected at extreme angles relative to the main radar beam can be filtered out because these detections are generally less reliable and more susceptible to distortion. The computational system can also filter out objects that are not consistently detected across multiple radar scans, helping to eliminate transient false detections. In some examples, the system can exclude objects that are not consistently detected by multiple sensors, thereby improving overall detection reliability.

[0187] At box 608, method 600 involves performing a comparison between radar parameters determined based on radar data corresponding to a specific object and expected radar parameters represented by a data model. In some examples, the data model is generated based on radar data aggregated for objects matching the first type of object.

[0188] The data model can convey multiple radar parameters as a function of range, where the radar parameters are based on radar data aggregated for objects matching the first type of object. The computing system can receive the data model from a remote computing device via wireless communication.

[0189] In some examples, the computing system can determine SNR data based on filtered radar data corresponding to one or more objects, and perform a comparison between the SNR data and the expected SNR data represented by the data model. The expected SNR data is based on the SNR data aggregated for objects matching a first type of object. In some examples, the computing system can determine radar cross-section (RCS) data based on filtered radar data corresponding to one or more objects. The computing system can then perform a comparison between the RCS data and the expected RCS data represented by the data model. The expected RCS data is based on the RCS data aggregated for objects matching a first type of object.

[0190] In some examples, the computational system can compare radar parameters determined based on filtered radar data corresponding to one or more objects with expected radar parameters represented by a data model in different ways. For example, the computational system can calculate the difference between observed SNR or RCS values ​​from the filtered radar data and expected values ​​from a data model used for objects of similar type and range. In some cases, the computational system can normalize the observed and expected parameters before performing the comparison to account for variations in radar configuration or environmental conditions. The computational system can use statistical methods to compare the distribution of observed radar parameters with the expected distribution from the data model. This can involve calculating measures such as mean squared error or Kullback-Leibler divergence.

[0191] In some examples, for multiple objects, the computational system can perform point-by-point comparisons of radar parameters and aggregate the results to obtain an overall measurement of the deviation from expected values. Furthermore, the computational system can apply weighting factors to different radar parameters based on their relative importance or reliability, allowing for more nuanced comparisons between observed and expected values. In some implementations, the computational system can use machine learning techniques, such as anomaly detection algorithms, to identify significant deviations between observed radar parameters and those predicted by the data model.

[0192] In some examples, the computational system can perform time-series analysis to compare how observed radar parameters change over time relative to the expected trends represented in the data model. Furthermore, for radar systems with multiple channels or frequencies, the computational system can compare parameters across different channels to identify any frequency-related deviations from the expected values.

[0193] In some cases, computational systems can use probabilistic methods to estimate the likelihood that observed radar parameters match a predicted distribution from a data model, thus providing a confidence metric for comparison. By using one or more comparison techniques, computational systems can effectively quantify the differences between observed and predicted radar parameters, enabling accurate estimation of radar sensitivity loss and identification of potential sensor degradation or failure.

[0194] At box 610, method 600 relates to estimating radar sensitivity loss based on comparison. In some examples, the computational system may compare the estimated radar sensitivity loss with a threshold loss and adjust radar operation based on whether the estimated radar sensitivity loss exceeds the threshold loss. For example, the computational system may adjust the power level used by the radar during signal transmission. In some cases, the computational system may adjust one or more of the power level, waveform, and timeline used by the radar during signal transmission. The timeline may change the transmission mode or timing used by the radar.

[0195] Computational systems can estimate radar sensitivity and other performance parameters in various ways based on comparisons (or multiple comparisons). For example, a computational system can calculate a sensitivity loss factor by determining the average difference between observed and expected signal-to-noise ratio (SNR) values ​​across multiple detected objects. This factor can provide a direct estimate of the overall radar sensitivity degradation. In some cases, computational systems can use regression analysis to model the relationship between observed and expected radar parameters. The slope of the regression line can indicate a change in sensitivity, with values ​​less than 1 indicating a sensitivity loss. Additionally, computational systems can estimate detection range performance by comparing the maximum range at which objects are reliably detected with the expected detection range from a data model. A reduction in the actual detection range can indicate a sensitivity loss. Furthermore, by analyzing discrepancies in RCS measurements across different object types and ranges, computational systems can estimate changes in the radar's dynamic range and gain characteristics.

[0196] In some cases, the computational system can use statistical methods to estimate the false alarm rate by comparing the number of detections that do not correspond to real objects with the expected false alarm rate from the data model. Additionally, in some implementations, the computational system can use machine learning algorithms to predict radar performance parameters based on the deviations between observed and expected values. These algorithms can learn the complex relationships between various radar parameters and overall system performance. The computational system can also estimate angular resolution by comparing the angular separation of observed closely spaced objects with the expected resolution capability represented in the data model. By analyzing Doppler measurements of objects with known velocities, the computational system can estimate any degradation in velocity measurement accuracy and radar velocity resolution. The computational system can also use information-theoretic concepts such as mutual information to quantify how much information the radar is capturing about the environment compared to what is expected based on the data model.

[0197] In some cases, computational systems can perform sensitivity analyses to determine which radar parameters are most affected by observed deviations, thus helping to identify specific aspects of radar performance that may be degrading. By applying these estimation techniques, computational systems can provide comprehensive insights into radar sensitivity and other performance parameters, enabling effective monitoring and maintenance of vehicle-mounted radar systems.

[0198] In some examples, the computing system can perform corrective actions, alarms, and other responses based on estimated sensitivity and other performance parameters. For instance, the computing system can generate visual or auditory alarms to notify the driver or vehicle occupants of a degraded radar performance, potentially suggesting increased attention or manual driving intervention. In some cases, the computing system can adjust the radar's operating parameters, such as increasing transmit power or modifying waveform characteristics, to compensate for detected sensitivity loss.

[0199] In some examples, adjusting radar operation in response to detected sensitivity loss can involve several strategies. The system can modify the radar aperture, for example, by increasing the number of active transmitting elements to enhance radar directivity. Radar waveform tuning can include changing the range swath to focus more on the near-range when sensitivity is compromised. The radar timeline can be altered, such as increasing waveform duration, to improve the signal-to-noise ratio (SNR). Additionally, the system can adjust the radar illumination position by modifying the transmit beam pointing to increase road duty cycle. These adaptive techniques allow the radar system to optimize its performance in response to changing conditions or degraded sensitivity, thereby maintaining effective object detection and tracking capabilities.

[0200] If sensitivity loss is suspected to be due to environmental contamination, the computing system can initiate self-cleaning procedures for the radar sensors, such as activating wipers, blowers, heaters, or cleaning systems. In some cases, the computing system can reallocate sensing tasks to prioritize data from better-performing sensors and reduce reliance on degraded units. Additionally, when radar performance is compromised, the computing system can adjust sensor fusion algorithms to place more weight on data from other sensor types, such as cameras or lidar. The computing system can also modify vehicle behavior, such as reducing speed or increasing following distance, to account for degraded radar performance. The computing system can record detailed performance data and trigger in-flight updates to refine the radar's signal processing algorithms or update its calibration parameters. For severe performance degradation, the computing system can initiate a safe stop procedure or transfer control to a human driver to ensure vehicle safety. In some cases, the computing system can schedule maintenance alerts to notify vehicle owners or fleet operators that radar sensors need inspection or replacement. The computing system can also share information about localized radar performance issues with other nearby vehicles or infrastructure, potentially enhancing overall traffic safety. By implementing one or more responses, the computing system can effectively manage radar performance issues, thereby ensuring the continued safe operation of the vehicle and the timely maintenance of the sensor systems.

[0201] In some examples, the computational system can compare the estimated radar sensitivity loss with a threshold loss and adjust the vehicle's control strategy based on whether the estimated radar sensitivity loss exceeds the threshold loss. For instance, when another vehicle is traveling in front of the vehicle, the computational system can reduce the vehicle's speed and increase the following distance used by the vehicle.

[0202] In some examples, the computational system can compare the estimated radar sensitivity loss with a threshold loss and trigger a cleaning operation at the radome corresponding to the radar based on the estimated radar sensitivity loss exceeding the threshold loss. In some cases, the computational system can further classify specific objects based on image data and LiDAR data corresponding to one or more objects, and use this classification for various actions.

[0203] In some examples, method 600 may also involve receiving second radar data representing the environment from a second radar coupled to a vehicle. Method 600 may also involve performing a second comparison between second radar parameters determined based on second radar data corresponding to a specific object and expected radar parameters represented by a data model, and estimating a second radar sensitivity loss of the second radar based on this comparison. The computing system may then perform a third comparison between the estimated radar sensitivity loss of the radar and the estimated second radar sensitivity loss of the second radar, and adjust the operation of the radar or the second radar based on the third comparison.

[0204] In some cases, Method 600 can be performed by vehicles operating in complex urban environments, where vehicle radar can be used to detect a wide variety of objects, such as other vehicles, pedestrians, cyclists, or infrastructure components. Filtering radar data based on predetermined criteria, such as range and weather conditions, can help focus radar sensitivity estimates on the most relevant objects in these environments. In other cases, the method can be used by vehicles operating on highways or in rural areas, where radar can be used to detect other vehicles at various distances. Developing data models of the expected SNR and RCS distributions based on various parameters, such as radar installation location, waveform parameters, range, and gain settings, can be particularly useful in these scenarios, as it can provide a reference for expected radar performance under various conditions.

[0205] In some respects, method 600 can be performed by a vehicle operating in adverse weather conditions such as rain, snow, or fog. Weather-specific filtering of the sensed object can allow for more accurate estimation of radar sensitivity loss under these conditions. The vehicle system can adjust the filtering criteria based on current weather conditions to account for their impact on radar sensitivity.

[0206] Furthermore, method 600 can be used in vehicles equipped with multiple radar sensors, each with different installation locations, waveform parameters, range settings, and gain settings. The system can individually estimate the radar sensitivity loss of each radar sensor, thereby allowing comprehensive monitoring of the vehicle's overall radar performance. In some cases, this method can be combined with other sensor technologies (non-radar sensors) such as lidar or camera sensors to provide multimodal perception of the vehicle's environment. The estimated radar sensitivity loss can be used to adjust the reliance on radar data in multimodal perception, thereby improving the robustness and reliability of the perception system.

[0207] In some examples, method 600 can be used in vehicles equipped with advanced radar technologies, such as phased array radar or frequency modulated continuous wave (FMCW) radar. Method 600 can be adapted to the specific characteristics of these radar technologies, such as their unique waveform parameters or directional distributions, to provide accurate and relevant estimates of radar sensitivity loss.

[0208] In some examples, individual vehicles as part of a convoy can continuously monitor their radar performance, including range-related losses. In some cases, these losses indicate various weather conditions affecting radar propagation, such as rain, snow, fog, or other atmospheric phenomena. Thus, by comparing the observed losses to a pre-established model table, each vehicle can infer the current weather conditions in its immediate environment. This information can be shared with a central convoy management system, effectively creating a distributed network of weather sensors across the entire operational area of ​​the convoy. For example, the central system can aggregate and analyze data from all vehicles to create and maintain a real-time map of weather conditions and their impact on radar performance. This map can be far more detailed and up-to-date than traditional weather forecasting methods, especially for localized weather phenomena.

[0209] Based on aggregated data, fleet management systems can implement various strategies and actions to optimize vehicle operation and enhance safety. In some examples, these actions may include rerouting vehicle routes to avoid areas with severe weather conditions, dynamically adjusting fleet-wide rate limits, switching vehicles to alternative sensor modes, and / or increasing reliance on other sensor types in areas where radar performance is significantly degraded. In some cases, the system may make decisions to halt navigation of certain vehicles or restrict operation in specific areas where weather conditions make safe operation impossible. The system can also use this data for long-term operational improvements. For example, radar degradation patterns can be used to schedule preventative maintenance or cleaning of radar sensors. Historical data on weather-related radar performance can guide route planning, vehicle deployment strategies, and even future vehicle design considerations. In manually operated vehicles, the system can provide alerts and recommendations to drivers based on observed weather conditions and their impact on sensor performance. Typically, this approach creates a feedback loop where individual vehicle experience contributes to fleet-wide operation, which in turn helps optimize the performance and safety of each vehicle. By leveraging collective awareness of weather conditions and their impact on radar performance, fleets can operate more effectively and safely across a wide range of environmental conditions. The system demonstrates how shared data and adaptive strategies can significantly enhance a vehicle fleet's ability to cope with challenging weather conditions.

[0210] In some examples, radar is used in conjunction with control electronics, which may include one or more field-programmable gate arrays (FPGAs), ASICs, CPUs, GPUs, and / or TPUs. For instance, a radar unit may generate and receive complex signals requiring extensive processing. One or more control electronics can be programmed to implement various signal processing algorithms, such as filtering, modulation / demodulation, noise reduction, and digital beamforming. These operations help extract relevant information from the received radar signals, enhance signal quality, and improve target detection and tracking. Furthermore, radar systems typically involve converting analog signals into digital formats for further processing. Control electronics may include analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) to facilitate these conversions. Control electronics can receive analog signals from radar sensors, digitize them, and process the digital data for analysis and interpretation.

[0211] Furthermore, control electronics can provide the ability to control and coordinate various radar system components in real time. For example, control electronics can handle synchronization, timing generation, and system control, ensuring the correct timing and sequencing of operations within the radar system. This real-time control is crucial for accurate and synchronized signal transmission and reception. Control electronics can efficiently process the large amounts of data generated by the radar system. They can implement data storage, buffering, and data flow management techniques to achieve efficient data processing during signal transmission. This includes tasks such as data compression, data packetization, and data routing, ensuring smooth and reliable data transmission within the radar system. Control electronics can also integrate various interfaces and protocols required for radar signal transmission, such as processors, memory modules, communication modules, and display units. They can provide the necessary interface logic to facilitate seamless data exchange between these components, enabling efficient data flow and system integration. Control electronics can also be reconfigured and customized to meet specific radar system requirements and adapt to changing operational needs. This allows radar system designers to implement and optimize application-specific algorithms and functions, resulting in enhanced performance and efficiency.

[0212] This disclosure is not limiting with respect to the specific embodiments described in this application, which are intended as illustrative of various aspects. It will be apparent to those skilled in the art that many modifications and variations can be made without departing from its spirit and scope. In addition to those listed herein, functionally equivalent methods and apparatus within the scope of this disclosure will be apparent to those skilled in the art based on the foregoing description. These modifications and variations are intended to fall within the scope of the appended claims.

[0213] The above detailed description, with reference to the accompanying drawings, illustrates various features and functions of the disclosed systems, devices, and methods. In the drawings, like symbols typically identify like components unless the context otherwise requires. The exemplary embodiments described herein and in the drawings are not intended to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the scope of the subject matter set forth herein. It will be readily understood that aspects of this disclosure, as generally described herein and illustrated in the figures, can be arranged, replaced, combined, separated, and designed in a wide variety of different configurations, all of which are expressly contemplated herein.

[0214] With respect to any or all of the message flow diagrams, scenarios, and flowcharts in the accompanying drawings and as discussed herein, each step, block, operation, and / or communication may represent the processing and / or transmission of information according to exemplary embodiments. Alternative embodiments are included within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages may be performed out of order shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved. Furthermore, more or fewer blocks and / or operations may be used with any of the message flow diagrams, scenarios, and flowcharts discussed herein, and these message flow diagrams, scenarios, and flowcharts may be combined with each other in part or in whole.

[0215] The steps, blocks, or operations representing information processing may correspond to circuitry that can be configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, the steps or blocks representing information processing may correspond to modules, segments, or portions of program code (including associated data). The program code may include one or more instructions executable by a processor to implement specific logical operations or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including RAM, a disk drive, a solid-state drive, or another storage medium.

[0216] Furthermore, a step, frame, or operation representing one or more information transfers can correspond to information transfers between software and / or hardware modules within the same physical device. However, other information transfers can occur between software and / or hardware modules in different physical devices.

[0217] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in the given figures. Furthermore, some of the illustrated elements may be combined or omitted. Additionally, example embodiments may include elements not shown in the figures.

[0218] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes and are not intended to be limiting, wherein the true scope is indicated by the appended claims.

Claims

1. A method for estimating radar sensitivity, comprising: At the computing system, radar data is received from a radar coupled to the vehicle, wherein the radar data represents the environment outside the vehicle; Detect one or more objects located in the environment based on the radar data; The radar data corresponding to the one or more objects is filtered based on predetermined criteria to identify a specific object corresponding to a first type of object; Perform a comparison between radar parameters determined based on radar data corresponding to the specific object and expected radar parameters represented by a data model, wherein the data model is generated based on aggregated radar data for multiple objects matching the first type of object; and The radar sensitivity loss of the radar is estimated based on the comparison.

2. The method according to claim 1, further comprising: The signal-to-noise ratio (SNR) data is determined based on the radar data corresponding to the specific object. as well as The comparison includes: Perform a comparison between the SNR data and the expected SNR data represented by the data model, wherein the expected SNR data is based on SNR data aggregated for the plurality of objects matching the first type of object.

3. The method according to claim 1, further comprising: The radar cross section (RCS) data is determined based on the radar data corresponding to the specific object. as well as The comparison includes: Perform a comparison between the RCS data and the expected RCS data represented by the data model, wherein the expected RCS data is based on RCS data aggregated for the plurality of objects matching the first type of object.

4. The method according to claim 1, wherein, The set of predetermined criteria includes: A first predetermined criterion, which conveys a scope filter, is used to distinguish a particular object from one or more objects in the environment based on a scope associated with each object; and The second predetermined criterion is based on the fact that the specific object has a certain degree of line of sight relative to the radar to identify the specific object.

5. The method according to claim 4, wherein, The set of predetermined criteria also includes: A third predetermined criterion is used to filter the radar data based on multiple detections within the radar data.

6. The method according to claim 1, wherein, The data model conveys multiple radar parameters as a function of range, wherein the multiple radar parameters are based on radar data aggregated for the multiple objects matching the first type of object.

7. The method according to claim 1, further comprising: The estimated radar sensitivity loss is compared with the threshold loss; as well as Based on the estimated radar sensitivity loss exceeding the threshold loss, the operation of the radar is adjusted, wherein the adjustment of the radar operation includes: Adjust one or more of the power level, waveform, and timeline used by the radar during signal transmission.

8. The method according to claim 1, further comprising: The specific objects are further classified based on image data and LiDAR data corresponding to the one or more objects.

9. The method according to claim 1, further comprising: The estimated radar sensitivity loss is compared with the threshold loss; as well as Based on the estimated radar sensitivity loss exceeding the threshold loss, the control strategy for the vehicle is adjusted.

10. The method according to claim 9, wherein, Adjusting the control strategy for the vehicle includes: When another vehicle is traveling in front of the vehicle, reduce the speed of the vehicle and increase the following distance used by the vehicle.

11. The method according to claim 1, further comprising: The estimated radar sensitivity loss is compared with the threshold loss; as well as Based on the estimated radar sensitivity loss exceeding the threshold loss, a cleaning operation is triggered at the radome corresponding to the radar.

12. The method according to claim 1, further comprising: Receive second radar data representing the environment from a second radar coupled to the vehicle; Perform a second comparison between the second radar parameters determined based on the portion of the second radar data corresponding to a specific object and the expected radar parameters represented by the data model; as well as The second radar sensitivity loss of the second radar is estimated based on the second comparison.

13. The method of claim 12, further comprising: The system receives information from a remote computing system that modifies at least the route or control strategy used by the vehicle, wherein the remote computing system is configured to aggregate radar sensitivity loss information from multiple vehicles and distribute the modifications to the multiple vehicles based on the aggregated radar sensitivity loss information.

14. The method according to claim 1, further comprising: The data model is received from a remote computing device via wireless communication, wherein the remote computing device is configured to generate the data model based on radar data aggregated for the plurality of objects.

15. A system for estimating radar sensitivity, comprising: A vehicle equipped with radar; as well as Computing device, the computing device being configured to: Receive radar data representing the external environment of the vehicle; Detect one or more objects located in the environment based on the radar data; The radar data corresponding to the one or more objects is filtered based on predetermined criteria to identify a specific object corresponding to a first type of object; Perform a comparison between radar parameters determined based on radar data corresponding to the specific object and expected radar parameters represented by a data model, wherein the data model is generated based on radar data aggregated for multiple objects matching the first type of object; as well as The radar sensitivity loss of the radar is estimated based on the comparison.

16. The system according to claim 15, wherein, The computing device is coupled to the vehicle, and the first type of object is a passenger vehicle.

17. The system according to claim 15, wherein, The radar is a first radar coupled to the vehicle, and a second radar is coupled to the vehicle. The computing device is further configured as follows: The estimated radar sensitivity loss of the first radar is compared with the second estimated radar sensitivity loss of the second radar; and Based on comparing the estimated radar sensitivity loss of the first radar with the second estimated radar sensitivity loss of the second radar, an operational degradation of either the first radar or the second radar is determined.

18. The system according to claim 15, wherein, The computing device is also configured to: The specific object is classified by using a heuristic method that uses radar data corresponding to the specific object.

19. The system according to claim 15, wherein, The computing device is also configured to: Receive sensor data from one or more non-radar sensors coupled to the vehicle; and The specific object is identified based on sensor data received from the one or more non-radar sensors.

20. A non-transitory computer-readable medium configured to store instructions that, when executed by a computing system comprising one or more processors, cause the computing system to perform operations, the operations including: Radar data is received from a radar coupled to the vehicle, wherein the radar data represents the environment outside the vehicle; Detect one or more objects located in the environment based on the radar data; The radar data corresponding to the one or more objects is filtered based on predetermined criteria to identify a specific object corresponding to a first type of object; Perform a comparison between radar parameters determined based on radar data corresponding to the specific object and expected radar parameters represented by a data model, wherein the data model is generated based on aggregated radar data for multiple objects matching the first type of object; and The radar sensitivity loss of the radar is estimated based on the comparison.

Images