Universal spectrum utilization exchange for radio frequency interference mitigation and avoidance

Abstract

To provide techniques and systems for exchanging spectrum utilization data for radio frequency interference (RFI) mitigation and avoidance.SOLUTION: A system can receive RFI data and location data from vehicles or infrastructure locations, the RFI data representing RFI parameters estimated by each vehicle during navigation or by each infrastructure location. The system can aggregate the RFI data into a representation based on the location data, the representation conveying a power level and frequency sub-band for RFI detected at multiple locations. The system can then obtain subsequent locations from vehicles, and provide data from the representation to the vehicles based on the locations. In some cases, the system can also provide mitigation instructions to the vehicles, such as waveform parameters, frequency bands, polarization, and other adjustments that help avoid RFI. The system can also reroute the vehicles to avoid areas having particularly high RFI.SELECTED DRAWING: Figure 6

Description

Background Art

[0001] Automotive radar involves using radio frequency (RF) to detect the presence, distance, direction, and speed of objects in the surrounding environment of a vehicle. A vehicle radar system radiates a radio signal from a transmitter, which then bounces off nearby objects and returns to the receiver. By analyzing the characteristics of the returned signal, the vehicle radar system can determine the location, speed, and direction of objects located in the environment, such as other vehicles, pedestrians, road boundaries, and obstacles. In some cases, the radar data is used by a vehicle's advanced driver assistance system (ADAS) or autonomous driving system (ADS) to provide warnings to the driver or take autonomous actions to avoid collisions. In other instances, the vehicle control system uses the radar data when determining control strategies for autonomous navigation by the vehicle.

Summary of the Invention

[0002] Exemplary embodiments relate to techniques and systems for exchanging spectrum utilization data to enable the aggregation of radio frequency interference (RFI) data from different sources and locations. Vehicles and other systems measure the RFI experienced at various locations and estimate RFI information that describes the signal characteristics of the RFI. Vehicles and other systems (e.g., infrastructure locations) provide the RFI information and location data to a remote central system, which aggregates the RFI information and generates an RFI map or another representation that conveys the RFI data at multiple locations in substantially real-time. The central system can distribute the RFI map or representation for use by vehicle radar systems and other emitters to reduce and avoid RFI during signal transmission. In some embodiments, a vehicle can act similarly to the remote central system and aggregate RFI information from other sources to determine mitigation strategies during navigation. The vehicle can aggregate RFI information in a distributed manner from other nearby sources, which can be used on-board by the vehicle system and can also contribute to the central system for further processing.

[0003] In one aspect, a method is described. The method involves receiving, in a computing system, radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, where the RFI data represents RFI parameters estimated by each vehicle during navigation or by each infrastructure location. The method also involves aggregating the RFI data based on the location data to generate a representation indicating the power levels and frequency sub-bands of the RFI detected at one or more locations. The method also involves, in the computing system, obtaining a location from a vehicle and providing, by the computing system and to the vehicle, at least a portion of the representation based on the location of the vehicle.

[0004] In another aspect, a system is described. The system also includes a computing device. The computing device is configured to receive radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, where the RFI data represents RFI parameters estimated by each vehicle during navigation or by each infrastructure location. The computing device is further configured to aggregate the RFI data based on the location data to generate a representation indicating the power levels and frequency sub-bands of the RFI detected at one or more locations. The computing device is also configured to obtain a location from a vehicle and provide at least a portion of the representation to the vehicle based on the location of the vehicle.

[0005] In yet another aspect, a non-transitory computer-readable medium is described. The non-transitory computer-readable medium is configured to store instructions that, when executed by a computing system comprising one or more processors, cause the computing system to perform operations. The operations include receiving radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, where the RFI data represents RFI parameters estimated by each vehicle during navigation or by each infrastructure location. The operations also include aggregating the RFI data based on the location data to generate a representation indicating the power levels and frequency sub-bands of the RFI detected at one or more locations. The operations further include obtaining a location from a vehicle and providing at least a portion of the representation to the vehicle based on the location of the vehicle.

[0006] These and other aspects, advantages, and alternatives will become apparent to those of ordinary skill in the art by reading the following detailed description with appropriate reference to the accompanying drawings.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 2G

Figure 2H

Figure 2I

Figure 2J

Figure 2K

Figure 3

Figure 4

Figure 5

Figure 6

[0008] Exemplary methods and systems are contemplated herein. Any exemplary embodiment or feature described herein should not necessarily be construed as more preferable or advantageous than other embodiments or features. Further, the exemplary embodiments described herein are not meant to be limiting. Certain aspects of the disclosed systems and methods can be arranged and combined in a variety of different configurations, and it will be readily understood that all of these configurations are contemplated herein. Additionally, the specific arrangements shown in the figures should not be regarded as limiting. It should be understood that other embodiments can include more or fewer of each element shown in a given figure. Additionally, some of the illustrated elements can be combined or omitted. Still further, the exemplary embodiments can include elements not illustrated in the figures.

[0009] Vehicles are increasingly incorporating radar systems for detecting and avoiding obstacles and for measuring road conditions and other aspects of the dynamic environment encountered during navigation. Many vehicle radar systems are designed to operate within the automotive RF band, which is within 5 gigahertz (GHz) of a spectral region that spans the range of 76 GHz to 81 GHz. The spectral region can provide sufficient bandwidth to accommodate a single vehicle radar system (i.e., 5 GHz), but RF interference (RFI) can occur and cause problems when multiple vehicle radar systems operate at the same or similar frequencies in the same general location and / or when using wide bandwidth (high range resolution) modulation. For each vehicle radar system, RFI can make it difficult to distinguish the desired reflections of signals transmitted by the vehicle radar system from other RF signals originating from other nearby emitters (e.g., other vehicle radar systems), which can reduce the ability of the vehicle radar system to accurately measure aspects of the surrounding environment. As the number of radar-equipped vehicles increases, vehicle radar systems are increasingly likely to encounter RFI during navigation, particularly in urban areas and other areas (e.g., intersections) where more vehicles typically approach and navigate in multiple directions.

[0010] For example, consider a scenario where a vehicle equipped with radar is driving in an urban area surrounded by a large number of electronic devices, a Wi-Fi network, and cell towers. These sources emit a wide range of radio frequency signals, some of which overlap with the frequencies used by the vehicle's radar system, causing the radar on the vehicle to receive mixed reflections that can cause the radar system to misperceive obstacles or vehicles that do not actually exist in the environment. As a result of receiving mixed reflections, the radar system may trigger undesirable actions by the vehicle control system, such as inappropriate braking, acceleration, or steering responses. In other cases, the radar system receiving mixed reflections may cause the vehicle control system to determine that it does not have the ability to continue navigation, potentially leading to the vehicle stopping or being put in a stationary state. Generally, RFI encountered during navigation can degrade signal quality, reduce the object detection ability of the vehicle radar system, and cause the vehicle's perception and behavior systems to misidentify or even fail to identify objects.

[0011] To address the adverse effects of RFI, vehicle radar systems can use mitigation techniques. For example, shielding materials can be used to isolate vehicle radar components from externally radiated electromagnetic signals, and the vehicle radar system can use filters to selectively process signals within its desired frequency range while rejecting unwanted frequencies. In addition, the vehicle radar system can also use signal processing algorithms that help distinguish genuine radar reflections from interference. However, these techniques may not be sufficient for the vehicle radar system to overcome the adverse effects of RFI, especially in situations where the vehicle is driving in a high-density RF environment or an area with buildings or other fixed sources that can significantly increase the impact of RFI.

[0012] The embodiments described herein relate to techniques and systems for exchanging spectrum utilization data, which enable the aggregation, management, and distribution of RFI data detected and provided by different sources at various locations, and assist vehicle radar systems and other emitters in reducing and avoiding the adverse effects of RFI. As an example, a network-based computing system may aggregate RFI data from a vehicle radar system in navigation and other sources in substantially real-time to generate an RFI map or another type of representation indicating the power levels and frequency sub-bands of RFI detected at various locations. The system can distribute the RFI representation to vehicle radar systems and other emitters to provide assistance in reducing and avoiding RFI during subsequent operations.

[0013] In addition, the RFI representation can be maintained, updated, and monitored by the system. In some embodiments, the system may determine a threshold level for the RFI and associate it with different locations. For example, the system may assign a power level threshold for the RFI at a particular location based on the power level of the RFI detected by the vehicle at that location over time. Trends and other data can be used to assign a threshold level for the RFI at that location. If one or more vehicle radar systems (or other devices) provide RFI data indicating that the currently measured RFI intensity (e.g., power level) at a particular location exceeds the threshold RFI level associated with that location, the system can be programmed to perform one or more actions in response. For example, the system may provide an alert to the vehicle at the particular location if the RFI exceeds the assigned threshold level. In some cases, the system can provide a mitigation instruction, such as a suggestion of waveforms and / or frequency parameters that can be used to mitigate the effects of the RFI at that location, to vehicles having a route passing through that location. The mitigation instruction can be determined based on input RFI data received from other vehicles and emitters at that location. For example, the RFI data can specify polarization, waveform, frequency band, and other transmission parameters used by the vehicle in that area. Consequently, the system can identify available parameters that one or more radar systems can use to switch to in order to mitigate the effects of the RFI at that location. In some cases, the system may provide an instruction to the vehicle to trigger a route change and avoid a particular location experiencing a large amount of RFI.

[0014] In some embodiments, the system can aggregate and use RFI and location information obtained from various sources to generate a real-time RFI "heat map" that represents the spectral power within a set of sub-bands arranged according to location. The RFI heat map can then be distributed by the system to vehicles and third-party sources, and updates to the RFI heat map can then be provided to the same recipients. For example, the system can split and provide data representing a portion of the RFI heat map selected by the system based on the locations of the vehicles and third-party sources. In some cases, the system provides a portion of the RFI heat map (or data from the RFI heat map) based on a route provided by the vehicle. In some cases, the RFI heat map associates numerical RFI intensity levels and waveform parameters with location coordinates.

[0015] The RFI heat map can be visually represented using different colors similar to common heat maps at various locations and can represent the spectral power detected at different locations. For example, the RFI heat map can be a graphical representation of data where individual values are represented as colors. The RFI heat map can be useful for visualizing the distribution and density of data points within a two-dimensional (2D) space such as a grid or map. By using the RFI heat map and assigning different colors to different values, patterns, trends, and variations in the data can be displayed. For example, a color scale can be used to represent various shades of colors in between, from low RFI values detected at a location (e.g., blue or green) to high RFI values detected at a location (e.g., red or yellow). In some cases, the RFI data can be split into smaller segments (cells), and each cell represents a specific area. The value associated with each cell determines the color that the system assigns to the RFI heat map. When represented in a visual format, the RFI heat map can include a legend so that third parties can understand its content, such as which areas currently have high levels of RFI that could potentially interfere with the performance of a radar.

[0016] The disclosed techniques involve vehicles that provide RFI and location information to a system as the vehicles navigate various routes. In some embodiments, the RFI and location information includes data associated with the vehicle's own radar signal transmissions and external RFI detected and experienced by the vehicle's radar system. In other embodiments, the RFI and location information may include only data that specifies transmission parameters and location data. The radar system can provide information that specifies frequency, signal strength, modulation, bandwidth, spectral distribution, polarization, and temporal characteristics for the RFI observed at a given location. In addition, the vehicle can provide data indicating interference type (e.g., continuous or intermittent), an image of potential RFI sources, the number of emitters, frequency / phase modulation patterns, and other information.

[0017] In some embodiments, the system can use the aggregated RFI data and location information to provide data that can assist vehicles and other emitters in reducing RFI. For example, each vehicle can download a data summary of the emitters within its area, which can reveal location, intensity, and waveform parameters that can assist the vehicle's radar system in adjusting its operation to improve the accuracy of the area. In some cases, the system can provide the vehicle with data notifying which sub-bands are available in an area, as well as other electronic support means for the vehicle to use to reduce the impact of RFI, such as clarifying the selection and / or sequence of waveform parameters. In other cases, the information can be converted to a reduced-sensitivity estimate that can affect the ability to detect various objects on a per-radar per-object basis and can be applied to other downstream pipelines.

[0018] The system can identify static and persistent elevated RFI sources (e.g., from non-radar RFI or stationary vehicles such as in a parking lot) if there is an increase in spectral power density over time at any number of fixed locations using RFI data received from vehicle radar systems, infrastructure, and / or other sources. For example, the system can detect these static and persistent elevated RFI sources based on RFI data supplied by multiple vehicles over a period such as a threshold period. The threshold period can vary based on location. In such cases, for the purpose of transferring information to a vehicle or a third-party source, the temporal evolution of RFI in a specific area can be observed or learned. For a vehicle, this can result in replanning its route to avoid static RFI sources that can otherwise affect its ability to maintain a certain level of vehicle autonomy. In some cases, a third party or a regulatory agency may be interested in a heat map or its derivatives. Specifically, a third party or a regulatory agency may want to understand the relationship with adverse effects on a vehicle such as a vehicle strand in relation to the radar power spectral density at the time of an accident.

[0019] In some embodiments, the system can coordinate the operation among multiple vehicle radar systems. For example, a central system can use the trend of RFI data to provide an instruction to allow individual radar systems to operate even though their paths overlap in an area that may have a large amount of RFI. For example, the system can adjust and synchronize vehicle radars to all transmit in a specific geographic direction (e.g., geographic north) at a given time to reduce RFI between vehicle radars. Similarly, different polarization, waveform parameters, frequency bands, and / or other mitigation instructions may be provided by the system to allow vehicle radar systems to operate accurately while allowing vehicles to move in close proximity to each other. The distribution of mitigation instructions by the system can depend on the route, geographic location, and / or type of vehicle radar system in some embodiments.

[0020] In some embodiments, individual vehicles can implement the disclosed techniques. For example, a vehicle can aggregate RFI data from other vehicles and infrastructure located within a threshold distance from the vehicle when the vehicle is navigating. The vehicle system can use the aggregated RFI data to understand the RF in the surrounding environment and implement mitigation actions such as changes in waveform, polarization, frequency, timing, or other dimensions of radar transmission. In some cases, the vehicle can reduce its reliance on radar when RFI is particularly high in the area during navigation (e.g., when above a threshold RFI level). For example, the vehicle system may rely on other sensors until RFI detection falls below the threshold RFI level again. Additionally, the vehicle system can communicate to other nearby vehicles that the RFI level is high at a location. Multiple vehicles can share the aggregated RFI data with each other and, in some embodiments, construct representations in a distributed fashion. Vehicles and infrastructure can share RFI data based on location or other factors.

[0021] The following description, and the accompanying drawings, disclose the features of various exemplary embodiments. The provided embodiments are by way of example and are not intended to be limiting. Accordingly, the dimensions of the drawings are not necessarily to scale.

[0022] In some embodiments, the system can facilitate vehicle - to - vehicle sharing that enables vehicles in the same general location to directly share RFI data. For example, multiple vehicles traveling within the same vicinity or along similar routes can be connected via the system in a way that allows the vehicles to share RF transmission metadata and RFI metadata.

[0023] Here, an exemplary system within the scope of the present disclosure will be described in more detail. The exemplary system can be implemented in a motor vehicle or can take the form of a motor vehicle. Additionally, the exemplary system can also be implemented in or take the form of various vehicles such as automobiles, trucks (e.g., pickup trucks, vans, tractors, and tractor trailers), motorcycles, buses, airplanes, helicopters, drones, lawn mowers, bulldozers, boats, submarines, all-terrain vehicles, snowmobiles, aircraft, recreational vehicles, amusement park vehicles, agricultural implements or agricultural vehicles, construction machinery or construction vehicles, warehouse equipment or warehouse vehicles, factory equipment or factory vehicles, trams, golf carts, trains, trolleys, pedestrian conveyors, and robotic devices. Other vehicles are similarly possible. Further, in some embodiments, the exemplary system may not include a vehicle.

[0024] Referring now to the figures, FIG. 1 is a functional block diagram illustrating an exemplary vehicle 100 that may be configured to operate fully or partially in an autonomous mode. More specifically, vehicle 100 can operate in an autonomous mode without human interaction by receiving control instructions from a computing system. As part of its operation in the autonomous mode, vehicle 100 can use sensors to detect objects in the surrounding environment and, in some cases, detect and enable safe navigation. Additionally, vehicle 100 can operate in a partially autonomous (i.e., semi-autonomous) mode where some functions of vehicle 100 are controlled by a human driver of vehicle 100 and some functions of vehicle 100 are controlled by a computing system. For example, vehicle 100 may also include a subsystem that enables a driver to control operations of vehicle 100 such as steering, acceleration, and braking, while on the other hand, the computing system implements assistive functions such as lane departure warning / lane keeping assist or adaptive cruise control based on other objects (e.g., vehicles) in the surrounding environment.

[0025] As described herein, in the partial autonomous driving mode, the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), and emergency braking), but the human driver is expected to situationally recognize the surroundings of the vehicle and monitor the assisted driving operations. Here, although the vehicle may perform all driving tasks in a particular situation, the human driver is expected to take responsibility for control as needed.

[0026] For purposes of simplification and brevity, various systems and methods are described below in conjunction with autonomous vehicles, although these or similar systems and methods may be used in various driving assistance systems that do not reach the level of a fully autonomous driving system (i.e., a partial autonomous driving system). In the United States, the Society of Automotive Engineers (SAE) has defined different levels of automated driving operations to indicate how much or how little a vehicle controls the driving, although different organizations in the United States or other countries may classify the levels differently. More specifically, the systems and methods of the present disclosure may be used in SAE Level 2 driving assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, etc., and other driver support. The disclosed systems and methods may be used in SAE Level 3 driving assistance systems that can drive autonomously under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles that use an SAE Level 4 automated driving system that operates autonomously in most normal driving situations and requires only occasional attention from a human operator. In all such systems, accurate lane estimation is automatically performed without driver input or control (e.g., while the vehicle is in motion), resulting in improved reliability of vehicle positioning and navigation, as well as overall safety of autonomous driving, semi-autonomous driving, and other driving assistance systems. As noted above, in addition to the way SAE classifies the levels of automated driving operations, other organizations in the United States or other countries may classify the levels of automated driving operations differently. Without limitation, the systems and methods disclosed herein may be used in driving assistance systems defined by the levels of automated driving operations of these other organizations.

[0027] As shown in FIG. 1, vehicle 100 may include various subsystems such as a propulsion system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, a power source 110, a computer system 112 (which may also be referred to as a computing system) having a data storage 114, and a user interface 116. In other examples, vehicle 100 may include more, fewer, or different subsystems, each including multiple elements. The subsystems and components of vehicle 100 may be interconnected in various ways. Additionally, the functions of vehicle 100 described herein may be divided among additional functional or physical components, or combined into fewer functional or physical components within an embodiment. For example, control system 106 and computer system 112 may be combined into a single system that operates vehicle 100 according to various operations.

[0028] Propulsion system 102 may include one or more components operable to provide powered movement for vehicle 100 and may include, among other possible components, engine / motor 118, energy source 119, transmission 120, and wheels / tires 121. For example, engine / motor 118 may be configured to convert energy source 119 into mechanical energy and may correspond to one or a combination of, among other possible options, an internal combustion engine, an electric motor, a steam engine, or a Stirling engine. For example, in some embodiments, propulsion system 102 may include multiple types of engines and / or motors, such as a gasoline engine and an electric motor.

[0029] Energy source 119 represents an energy source that can fully or partially power one or more systems of vehicle 100 (e.g., engine / motor 118). For example, energy source 119 can correspond to gasoline, diesel, other oil-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and / or other power sources. In some embodiments, energy source 119 may include a combination of a fuel tank, a battery, a capacitor, and / or a flywheel.

[0030] Transmission 120 may transmit mechanical power from engine / motor 118 to wheels / tires 121 and / or other possible systems of vehicle 100. Thus, transmission 120 may include, among other possible components, a gearbox, a clutch, a differential, and a drive shaft. The drive shaft may include an axle that connects to one or more wheels / tires 121.

[0031] Wheels / tires 121 of vehicle 100 may have various configurations within the exemplary embodiments. For example, vehicle 100 may exist in the form of a unicycle, a bicycle / motorcycle, a tricycle, or a four-wheel form of an automobile / truck, among other possible configurations. Thus, wheels / tires 121 may be connected to vehicle 100 in various ways and may exist in different materials such as metal and rubber.

[0032] Sensor system 104 may include various types of sensors, such as a global positioning system (GPS) 122, an inertial measurement unit (IMU) 124, a radar 126, a lidar 128, a camera 130, a steering sensor 123, and a throttle / brake sensor 125, among other possible sensors. In some embodiments, sensor system 104 may also include sensors (e.g., an 2 on-board monitor, a fuel gauge, an engine oil temperature, and a brake wear) configured to monitor the internal systems of vehicle 100.

[0033] The GPS 122 may include a transceiver operable to provide information regarding the position of the vehicle 100 relative to the earth. The IMU 124 may have a configuration using one or more accelerometers and / or gyroscopes, and based on inertial acceleration, may sense changes in the position and orientation of the vehicle 100. For example, the IMU 124 may detect the pitch and yaw of the vehicle 100 while the vehicle 100 is stationary or moving.

[0034] The radar 126 may represent one or more systems configured to sense objects in the surrounding environment of the vehicle 100, including the speed and azimuth of the objects, using radio signals. Accordingly, the radar 126 may include an antenna configured to transmit and receive radio signals. In some embodiments, the radar 126 may correspond to an attachable radar configured to obtain measurements of the surrounding environment of the vehicle 100.

[0035] The lidar 128 may include, among other system components, one or more laser sources, a laser scanner, and one or more detectors, and may operate in a coherent mode (e.g., using heterodyne detection) or an incoherent detection mode (i.e., time-of-flight mode). In some embodiments, one or more detectors of the lidar 128 may include one or more photodetectors that may be particularly sensitive detectors (e.g., avalanche photodiodes). In some examples, such photodetectors may be capable of detecting single photons (e.g., single photon avalanche diodes (SPADs)). Further, such photodetectors may be arranged within an array (e.g., like a silicon photomultiplier (SiPM)) (e.g., through series electrical connections). In some examples, one or more photodetectors are devices that operate in Geiger mode, and the lidar includes sub-components designed for such Geiger mode operation.

[0036] The camera 130 may include one or more devices (e.g., a stationary camera, a video camera, a thermal imaging camera, a stereo camera, and a night vision camera) configured to capture an image of the surrounding environment of the vehicle 100.

[0037] The steering sensor 123 may sense the steering angle of the vehicle 100, which may include measuring the angle of the steering wheel or measuring an electrical signal representing the angle of the steering wheel. In some embodiments, the steering sensor 123 may measure the angle of the wheel of the vehicle 100, such as detecting the angle of the wheel relative to the front axle of the vehicle 100. The steering sensor 123 may also be configured to measure a combination (or subset) of the angle of the steering wheel, the electrical signal representing the angle of the steering wheel, and the angle of the wheel of the vehicle 100.

[0038] The throttle / brake sensor 125 may detect either the throttle position or the brake position of the vehicle 100. For example, the throttle / brake sensor 125 may measure the angles of both the accelerator pedal (throttle) and the brake pedal, or may measure, for example, an electrical signal representing the angle of the accelerator pedal (throttle) and / or the angle of the brake pedal. The throttle / brake sensor 125 may also measure the angle of the throttle body of the vehicle 100, which may include a part of a physical mechanism that provides modulation of the energy source 119 to the engine / motor 118 (e.g., butterfly valve and carburetor). Additionally, the throttle / brake sensor 125 may measure the pressure of one or more brake pads on the rotor of the vehicle 100, or a combination (or subset) of the angles of the accelerator pedal (throttle) and the brake pedal, an electrical signal representing the angles of the accelerator pedal (throttle) and the brake pedal, the angle of the throttle body, and the pressure applied by at least one brake pad to the rotor of the vehicle 100. In other embodiments, the throttle / brake sensor 125 may be configured to measure the pressure applied to a vehicle pedal, such as the throttle or the brake pedal.

[0039] The control system 106 may include components configured to assist in the navigation of the vehicle 100, such as a steering unit 132, a throttle 134, a brake unit 136, a sensor fusion algorithm 138, a computer vision system 140, a navigation / route finding system 142, and an obstacle avoidance system 144. More specifically, the steering unit 132 may be operable to adjust the orientation of the vehicle 100, and the throttle 134 may control the operating speed of the engine / motor 118 to control the acceleration of the vehicle 100. The brake unit 136 may be able to decelerate the vehicle 100, which may involve using friction to decelerate the wheels / tires 121. In some embodiments, the brake unit 136 may convert the kinetic energy of the wheels / tires 121 into an electric current for subsequent use by a system or systems of the vehicle 100.

[0040] The sensor fusion algorithm 138 may include a Kalman filter, a Bayesian network, or other algorithms capable of processing data from the sensor system 104. In some embodiments, the sensor fusion algorithm 138 may provide an evaluation based on the received sensor data, such as an evaluation of individual objects and / or features, an evaluation of a particular situation, and / or an evaluation of possible effects within a given situation.

[0041] The computer vision system 140 may be operable to process and analyze images to determine objects that are in motion (e.g., other vehicles, pedestrians, cyclists, or animals) and objects that are not in motion (e.g., road lighting fixtures, lane boundaries, speed bumps, or depressions). Thus, the computer vision system 140 may include hardware and software (e.g., a general-purpose processor such as a central processing unit (CPU), a dedicated processor such as a graphics processing unit (GPU) or a tensor processing unit (TPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), volatile memory, non-volatile memory, or one or more machine learning models) operable to perform object recognition, structure from motion (SFM), video tracking, and other algorithms used in computer vision, such as, for example, to recognize objects, map the environment, track objects, and estimate the speed of objects.

[0042] The navigation / route finding system 142 may be able to determine the driving route of the vehicle 100, which may involve dynamically adjusting the navigation during operation. Thus, the navigation / route finding system 142 may use data from, among other sources, the sensor fusion algorithm 138, the GPS 122, and the map to navigate the vehicle 100. The obstacle avoidance system 144 may evaluate potential obstacles based on sensor data and cause the vehicle 100's system to avoid or otherwise maneuver around the potential obstacles.

[0043] As shown in FIG. 1, vehicle 100 may also include peripheral devices 108 such as a wireless communication system 146, a touch screen 148, a microphone 150 (e.g., one or more internal and / or external microphones), and / or a speaker 152. The peripheral devices 108 may provide controls or other elements for a user to interact with the user interface 116. For example, the touch screen 148 may provide information to a user of vehicle 100. The user interface 116 may also receive input from the user via the touch screen 148. The peripheral devices 108 may also enable vehicle 100 to communicate with devices such as devices of other vehicles.

[0044] The wireless communication system 146 may communicate wirelessly with one or more devices, either directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication such as Code Division Multiple Access (CDMA), Evolution-Data Optimized (EVDO), Global System for Mobile Communications (GSM) / General Packet Radio Service (GPRS), or cellular communication such as 4G Worldwide Interoperability for Microwave Access (WiMAX) or Long Term Evolution (LTE), or 5G. Alternatively, the wireless communication system 146 may communicate with a Wireless Local Area Network (WLAN) using Wi-Fi (registered trademark) or other possible connections. The wireless communication system 146 may also communicate directly with devices using, for example, infrared links, Bluetooth, or ZigBee. Other wireless protocols such as various vehicle communication systems are possible within the context of the present disclosure. For example, the wireless communication system 146 may include one or more dedicated short range communication (DSRC) devices that may include public and / or private data communication between the vehicle and / or roadside stations.

[0045] Vehicle 100 may include a power source 110 for supplying power to components. In some embodiments, the power source 110 may include a rechargeable lithium-ion or lead-acid battery. For example, the power source 110 may include one or more batteries configured to provide power. Vehicle 100 may also use other types of power sources. In an exemplary embodiment, the power source 110 and the energy source 119 may be integrated to form a single energy source.

[0046] Vehicle 100 may also include a computer system 112 for performing operations such as those described therein. Thus, the computer system 112 may include a processor 113 (which may include at least one microprocessor) operable to execute instructions 115 stored on a non-transitory computer-readable medium such as data storage 114. As such, the processor 113 can represent one or more processors. In some embodiments, the computer system 112 may represent multiple computing devices that can function to control the individual components or subsystems of vehicle 100 in a distributed manner.

[0047] In some embodiments, the data storage 114 may include instructions 115 (e.g., program logic) executable by the processor 113 for performing various functions of the vehicle 100, including those described above in connection with FIG. 1. The data storage 114 may also include additional instructions that include instructions for sending, receiving, interacting with, and / or controlling data to one or more of the propulsion system 102, the sensor system 104, the control system 106, and the peripheral devices 108.

[0048] In addition to the instructions 115, the data storage 114 may store data such as road maps, route information, among other information. Such information may be used by the vehicle 100 and the computer system 112 during operation of the vehicle 100 in autonomous mode, semi-autonomous mode, and / or manual mode.

[0049] Vehicle 100 may include a user interface 116 for providing information to a user of vehicle 100 or receiving input from a user of vehicle 100. The user interface 116 may control or enable the layout of content and / or interactive images that may be displayed on the touch screen 148. Further, the user interface 116 may include one or more input / output devices within a set of peripheral devices 108, such as the wireless communication system 146, the touch screen 148, the microphone 150, and the speaker 152.

[0050] The computer system 112 may control the functions of the vehicle 100 based on inputs received from various subsystems (e.g., the propulsion system 102, the sensor system 104, or the control system 106) as well as from the user interface 116. For example, the computer system 112 may utilize inputs from the sensor system 104 to estimate the outputs generated by the propulsion system 102 and the control system 106. Depending on the embodiment, the computer system 112 may be operable to monitor many aspects of the vehicle 100 and its subsystems. In some embodiments, the computer system 112 may disable some or all of the functions of the vehicle 100 based on signals received from the sensor system 104.

[0051] The components of vehicle 100 may be configured to function in a manner that interconnects with other components, either within or external to their respective systems. For example, in an exemplary embodiment, camera 130 can capture multiple images that can represent information regarding the state of the surrounding environment of vehicle 100 operating in autonomous or semi-autonomous mode. The state of the surrounding environment may include parameters of the road on which the vehicle is operating. For example, computer vision system 140 may be able to recognize inclinations (gradients), or other features, based on multiple images of the road. Additionally, the combination of GPS 122 and the features recognized by computer vision system 140 can be used with the map data stored in data storage 114 to determine specific road parameters. Further, radar 126, and / or lidar 128, and / or some other environmental mapping, ranging, and / or positioning sensor systems can also provide information about the surroundings of the vehicle.

[0052] In other words, the combination of various sensors (which can be referred to as input indicator sensors and output indicator sensors) and computer system 112 can interact to provide an indicator of the input provided to control the vehicle or an indicator of the surroundings of the vehicle.

[0053] In some embodiments, computer system 112 can make decisions regarding various objects based on data provided by systems other than wireless systems. For example, vehicle 100 may have a laser or other optical sensor configured to sense objects within the field of view of the vehicle. Computer system 112 can use the outputs from various sensors to determine information regarding objects within the field of view of the vehicle, and may determine the distance to and direction information of various objects. Computer system 112 may also determine whether an object is desirable or undesirable based on the outputs from various sensors.

[0054] FIG. 1 shows various components of vehicle 100 (i.e., wireless communication system 146, computer system 112, data storage 114, and user interface 116) as integrated into vehicle 100, although one or more of these components may be attached or associated separately from vehicle 100. For example, data storage 114 can exist partially or completely separately from vehicle 100. Thus, vehicle 100 may be provided in the form of device elements that can be located separately or together. The device elements that make up vehicle 100 can be communicatively coupled together in a wired and / or wireless manner.

[0055] FIGS. 2A - 2E show an exemplary vehicle 200 (e.g., a fully autonomous vehicle, or a semi - autonomous vehicle) that may include some or all of the functions described in relation to vehicle 100 with reference to FIG. 1. Vehicle 200 is illustrated in FIGS. 2A - 2E as a van with side mirrors for illustrative purposes, but the present disclosure is not so limited. For example, vehicle 200 can represent a truck, a passenger car, a semi - trailer truck, a motorcycle, a golf cart, an off - road vehicle, an agricultural vehicle, or any other vehicle described elsewhere in this specification (e.g., a bus, a boat, an airplane, a helicopter, a drone, a lawn mower, a bulldozer, a submarine, an all - terrain vehicle, a snowmobile, an aircraft, a recreational vehicle, an amusement park vehicle, an agricultural implement, a construction machine or construction vehicle, a warehouse facility or warehouse vehicle, a factory facility or factory vehicle, a tram, a train, a trolley, a pedestrian transporter, and a robotic device).

[0056] Vehicle 200 may include one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and 218. In some embodiments, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent one or more optical systems (e.g., cameras), one or more lidars, one or more radars, one or more inertial sensors, one or more humidity sensors, one or more acoustic sensors (e.g., microphones and sonar devices), or one or more other sensors configured to sense information about the environment surrounding vehicle 200. In other words, any sensor system now known or hereafter created may be coupled to vehicle 200 and / or utilized in conjunction with the various operations of vehicle 200. As an example, a lidar may be utilized for autonomous driving or other types of navigation, planning, perception, and / or mapping operations of vehicle 200. Additionally, sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may represent combinations of sensors described herein (e.g., one or more lidars and radars, one or more lidars and cameras, one or more cameras and radars, or one or more lidars, cameras, and radars).

[0057] Note that the number, location, and type of sensor systems (e.g., 202 and 204) depicted in FIGS. 2A - E are intended as non - limiting examples of the location, number, and type of such sensor systems for autonomous or semi - autonomous vehicles. Alternative numbers, locations, types, and configurations of such sensors are possible (e.g., to reduce vehicle size, shape, aerodynamics, fuel economy, aesthetics, or cost, or to conform to other conditions for special environments or application scenarios). For example, sensor systems (e.g., 202 and 204) may be disposed at various other locations on the vehicle (e.g., at location 216) and may have a field of view corresponding to the interior and / or surrounding environment of vehicle 200.

[0058] The sensor system 202 can be attached to the upper part of the vehicle 200 and may include one or more sensors configured to detect information about the environment surrounding the vehicle 200 and output an indication of that information. For example, the sensor system 202 can include any combination of cameras, radars, lidars, inertial sensors, humidity sensors, and acoustic sensors (e.g., microphones and sonar devices). The sensor system 202 can include one or more movable mounts that may be operable to adjust the orientation of one or more sensors within the sensor system 202. In one embodiment, the movable mount can include a rotating platform that can scan the sensors so as to acquire information from each direction around the vehicle 200. In another embodiment, the movable mount of the sensor system 202 can be movable in a scanning manner within a specific range of angles, and / or azimuth angles, and / or elevation angles. Although other attachment locations are possible, the sensor system 202 can be attached on the roof of the vehicle.

[0059] Additionally, the sensors of the sensor system 202 can be dispersed in different locations and do not need to be collocated in a single location. Further, each sensor of the sensor system 202 can be configured to move or scan independently of the other sensors of the sensor system 202. Additionally or alternatively, multiple sensors can be attached in one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218. For example, there can be two lidar devices attached to the sensor location, and / or there can be one lidar device and one radar attached to the sensor location.

[0060] One or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more lidar devices. For example, a lidar device may include a plurality of light emitter devices arranged over an angular range with respect to a given plane (e.g., the x-y plane). For example, one or more of the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to rotate or pivot about an axis (e.g., the z-axis) perpendicular to a given plane so as to illuminate an environment surrounding the vehicle 200 with light pulses. Information about the surrounding environment may be determined based on detecting various aspects of the reflected light pulses (e.g., elapsed time of flight, polarization, and intensity).

[0061] In an exemplary embodiment, the sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may be configured to provide respective point cloud information related to physical objects within the surrounding environment of the vehicle 200. The vehicle 200, as well as the sensor systems 202, 204, 206, 208, 210, 212, 214, and 218, are illustrated as including certain features, but it will be understood that other types of sensor systems are contemplated within the scope of the present disclosure. Further, the vehicle 200 can include any of the components described in connection with the vehicle 100 of FIG. 1.

[0062] In an exemplary configuration, one or more radars may be located on vehicle 200. Similar to the radar 126 described above, one or more radars may include an antenna configured to transmit and receive radio waves (e.g., electromagnetic waves having a frequency in the range of 30 Hz to 300 GHz). Such radio waves can be used to determine the distance and / or speed to one or more objects in the surrounding environment of vehicle 200. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more radars. In some embodiments, one or more radars are located near the rear of vehicle 200 (e.g., sensor systems 208 and 210) and can actively scan the environment near the rear of vehicle 200 for the presence of radio wave reflecting objects. Similarly, one or more radars are located near the front of vehicle 200 (e.g., sensor system 212 or 214) and can actively scan the environment near the front of vehicle 200. The radar can be installed in a location suitable for illuminating an area including the forward movement path of vehicle 200 without being blocked, for example, by other features of vehicle 200. For example, the radar can be embedded in and / or attached to or near the front bumper, front headlights, cowl, and / or hood. Additionally, one or more additional radars can be positioned to actively scan the sides and / or rear of vehicle 200 to confirm the presence of radio wave reflecting objects, such as by including such devices in or near the rear bumper, side panels, rocker panels, and / or underbody of vehicle 200.

[0063] Vehicle 200 can include one or more cameras. For example, one or more sensor systems 202, 204, 206, 208, 210, 212, 214, and / or 218 may include one or more cameras. The camera can be a photosensitive device such as a stationary camera, a video camera, a thermal imaging camera, a stereo camera, a night vision camera, etc., configured to capture multiple images of the surrounding environment of vehicle 200. For this purpose, the camera can be configured to detect visible light and, additionally or alternatively, can be configured to detect light from other parts of the spectrum such as infrared or ultraviolet light. The camera can be a two-dimensional detector and optionally can have a sensitivity range in three-dimensional space. In some embodiments, the camera can include a range detector configured to generate a two-dimensional image indicating, for example, the distance from the camera to several points within the surrounding environment. For this purpose, the camera can use one or more range detection techniques. For example, the camera can provide range information by using a structured light technique in which vehicle 200 illuminates an object within the surrounding environment with a predetermined light pattern such as a grid or checkerboard pattern and uses the camera to detect the reflection of the predetermined light pattern from the surrounding environment. Based on the distortion of the reflected light pattern, vehicle 200 can determine the distance to a point on the object. The predetermined light pattern may include infrared light or radiation of other wavelengths suitable for such measurements. In some examples, the camera can be mounted inside the front windshield of vehicle 200. Specifically, the camera can be installed to capture images from a forward view with respect to the orientation of vehicle 200. Other mounting locations of the camera and the viewing angle can also be used either inside or outside vehicle 200. Further, the camera can have associated optical elements operable to provide an adjustable field of view. Furthermore, the camera can be mounted to vehicle 200 using a movable mount to change the pointing angle of the camera, for example, via a pan / tilt mechanism.

[0064] Vehicle 200 may also include one or more acoustic sensors for sensing the surrounding environment of vehicle 200 (e.g., one or more of sensor systems 202, 204, 206, 208, 210, 212, 214, 216, 218 may include one or more acoustic sensors). The acoustic sensors may include microphones (e.g., piezoelectric microphones, condenser microphones, ribbon microphones, or microelectromechanical system (MEMS) microphones) used to sense acoustic waves (i.e., pressure differences) in the fluid (e.g., air) of the environment surrounding vehicle 200. Such acoustic sensors may be used to identify sounds in the surrounding environment (e.g., sirens, human speech, animal sounds, or alarms) on which the control strategy of vehicle 200 may be based. For example, if the acoustic sensors detect a siren (e.g., an ambulance siren or a fire truck siren), vehicle 200 may decelerate and / or navigate to the edge of the road.

[0065] Although not shown in FIGS. 2A-2E, vehicle 200 can include a wireless communication system (e.g., similar to and / or in addition to the wireless communication system 146 of FIG. 1). The wireless communication system may include a wireless transmitter and a wireless receiver configured to communicate with devices external or internal to vehicle 200. Specifically, the wireless communication system may include, for example, a transceiver configured to communicate with other vehicles and / or computing devices in a vehicle communication system or a road station. Examples of such vehicle communication systems include DSRC, radio frequency identification (RFID), and other communication standards proposed for intelligent transportation systems.

[0066] In addition to or instead of those shown, vehicle 200 may include one or more other components. The additional components may include electrical or mechanical functions.

[0067] The control system of vehicle 200 may be configured to control vehicle 200 according to a control strategy from among a plurality of possible control strategies. The control system may receive information from sensors (on or outside vehicle 200) coupled to vehicle 200, modify the control strategy (and associated driving behavior) based on that information, and be configured to control vehicle 200 according to the modified control strategy. The control system may be further configured to monitor the information received from the sensors and continuously evaluate the driving conditions, and also be configured to modify the control strategy and driving behavior based on changes in the driving conditions. For example, the route taken by the vehicle from one destination to another may be modified based on the driving conditions. Additionally or alternatively, speed, acceleration, turning angle, following distance (i.e., the distance to the vehicle ahead of the current vehicle), lane selection, etc. may all be modified in response to changes in the driving conditions.

[0068] As described above, in some embodiments, vehicle 200 may take the form of a van, although alternative forms are also possible and contemplated herein. Thus, FIGS. 2F-2I illustrate embodiments in which vehicle 250 takes the form of a semi-truck. For example, FIG. 2F illustrates a front view of vehicle 250, and FIG. 2G illustrates an isometric view of vehicle 250. In embodiments where vehicle 250 is a semi-truck, vehicle 250 may include a tractor portion 260 and a trailer portion 270 (illustrated in FIG. 2G). FIGS. 2H and 2I provide a side view and a top view of the tractor portion 260, respectively. Similar to vehicle 200 illustrated above, vehicle 250 illustrated in FIGS. 2F-2I may also include various sensor systems (e.g., similar to sensor systems 202, 206, 208, 210, 212, 214 shown and described with reference to FIGS. 2A-2E). In some embodiments, vehicle 200 of FIGS. 2A-2E may include only a single copy of some sensor systems (e.g., sensor system 204), while vehicle 250 illustrated in FIGS. 2F-2I may include multiple copies of its sensor systems (e.g., sensor systems 204A and 204B as illustrated).

[0069] The drawings and the overall description may refer to a given vehicle configuration (e.g., vehicle 200 shown as a semi - tractor vehicle 250 or a van), but it is understood that the embodiments described herein are equally applicable in the context of various vehicles (e.g., using modifications employed to account for the vehicle's form factor). For example, sensors and / or other components described or illustrated as part of vehicle 200 may also be used in semi - tractor vehicle 250 (e.g., for navigation and / or obstacle detection and avoidance).

[0070] FIG. 2J illustrates various sensor fields of view (e.g., associated with vehicle 250 as described above). As noted above, vehicle 250 may contain a plurality of sensors / sensor units. The locations of the various sensors may correspond, for example, to the sensor locations disclosed in FIGS. 2F - 2I. However, in some instances, the sensors may have other locations. For the sake of simplicity of the drawings, sensor location reference numbers are omitted from FIG. 2J. For each sensor unit of vehicle 250, FIG. 2J illustrates representative fields of view (e.g., fields of view labeled as 252A, 252B, 252C, 252D, 254A, 254B, 256, 258A, 258B, and 258C). The field of view of a sensor may include the angular region (e.g., azimuthal region and / or elevation region) within which the sensor can detect an object.

[0071] Figure 2K illustrates beam steering for sensors of a vehicle (e.g., vehicle 250 shown and described with reference to FIGS. 2F-2J) according to an exemplary embodiment. In various embodiments, the sensor unit of vehicle 250 can be a radar, lidar, sonar, etc. Further, in some embodiments, during operation of the sensor, the sensor can be scanned within the field of view of the sensor. Various different scanning angles for an exemplary sensor are shown as region 272, each indicating the angular region in which the sensor is operating. The sensor can change the region in which it is operating periodically or iteratively. In some embodiments, multiple sensors can be used by vehicle 250 to measure region 272. Additionally, other regions can be included in other examples. For example, one or more sensors can measure aspects of trailer 270 of vehicle 250 and / or regions in front of vehicle 250.

[0072] At some angles, the operating region 275 of the sensor can include the rear wheels 276A, 276B of trailer 270. Therefore, the sensor can measure rear wheels 276A and / or rear wheels 276B during operation. For example, rear wheels 276A, 276B can reflect a lidar signal or a radar signal transmitted by the sensor. The sensor can receive the signal reflected from rear wheels 276A, 276. Thus, the data collected by the sensor can include data from reflections from the wheels.

[0073] In some cases, such as when the sensor is a radar, the reflections from rear wheels 276A, 276B can appear as noise in the received radar signal. As a result, the radar can operate with an enhanced signal-to-noise ratio in cases where rear wheels 276A, 276B direct the radar signal away from the sensor.

[0074] Figure 3 is a conceptual illustration of wireless communication between various computing systems related to autonomous or semi-autonomous vehicles, according to an exemplary embodiment. In particular, wireless communication can occur between a remote computing system 302 and a vehicle 200 via a network 304. Wireless communication may also occur between a server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0075] Vehicle 200 can correspond to various types of vehicles capable of transporting passengers or objects between locations, and can take any one or more forms of the vehicles discussed above. In some cases, vehicle 200 can operate in an autonomous or semi-autonomous mode that enables a control system to use sensor measurements to safely navigate vehicle 200 between destinations. When operating in an autonomous or semi-autonomous mode, vehicle 200 can navigate regardless of the presence of passengers. As a result, vehicle 200 can pick up and drop off passengers between desired destinations.

[0076] Remote computing system 302 may represent any type of device related to remote assistance technology, including but not limited to those described herein. Among the examples, remote computing system 302 may represent any type of device configured to (i) receive information related to vehicle 200, (ii) provide an interface through which a human operator can then become aware of the information and enter a response related to the information, and (iii) transmit the response to vehicle 200 or to another device. Remote computing system 302 can take various forms, such as a workstation, desktop computer, laptop, tablet, mobile phone (e.g., smartphone), and / or server. In some examples, remote computing system 302 may include multiple computing devices that operate together in a network configuration.

[0077] The remote computing system 302 may include one or more subsystems and components that are the same as, or identical to, the subsystems and components of the vehicle 200. At a minimum, the remote computing system 302 may include a processor configured to perform the various operations described herein. In some embodiments, the remote computing system 302 may also include a user interface including input / output devices such as a touch screen and speakers. Other embodiments are also possible.

[0078] The network 304 represents an infrastructure that enables wireless communication between the remote computing system 302 and the vehicle 200. The network 304 also enables wireless communication between the server computing system 306 and the remote computing system 302, and between the server computing system 306 and the vehicle 200.

[0079] The location of the remote computing system 302 can vary within the scope of the examples. For example, the remote computing system 302 may be in a remote location from the vehicle 200 having wireless communication via the network 304. In another example, the remote computing system 302 may correspond to a computing device within the vehicle 200 that is separate from the vehicle 200 but through which a human operator can interact with a passenger or driver of the vehicle 200. In some embodiments, the remote computing system 302 may be a computing device having a touch screen operable by a passenger of the vehicle 200.

[0080] In some embodiments, the operations described herein that are performed by the remote computing system 302 may alternatively or additionally be performed by the vehicle 200 (i.e., by any system or subsystem of the vehicle 200). In other words, the vehicle 200 may be configured to provide a remote assistance mechanism with which a driver or passenger of the vehicle can interact.

[0081] The server computing system 306 may be configured to wirelessly communicate with the remote computing system 302 and the vehicle 200 (or, in some cases, directly with the remote computing system 302 and / or the vehicle 200) via the network 304. The server computing system 306 may represent any computing device configured to receive, store, determine, and / or transmit information regarding the vehicle 200 and its remote assistance. As such, the server computing system 306 may be configured to perform any operation or portion of such an operation described herein as being performed by the remote computing system 302 and / or the vehicle 200. In some embodiments of the wireless communication related to remote assistance, the server computing system 306 can be utilized, while in other embodiments it cannot.

[0082] The server computing system 306 may include one or more subsystems and components similar or identical to those of the remote computing system 302 and / or the vehicle 200, such as a processor configured to perform the various operations described herein, and a wireless communication interface for receiving information from and providing information to the remote computing system 302 and the vehicle 200.

[0083] The various systems described above may perform various operations. These operations and related features are described herein.

[0084] In accordance with the above considerations, a computing system (e.g., a remote computing system 302, a server computing system 306, or a computing system local to vehicle 200) may operate to capture an image of the surrounding environment of an autonomous or semi-autonomous vehicle using a camera. Generally, at least one computing system can analyze the image and, if possible, control the autonomous or semi-autonomous vehicle.

[0085] In some embodiments, to facilitate autonomous or semi-autonomous operation, a vehicle (e.g., vehicle 200) may receive data representing objects in the environment surrounding the vehicle (also referred to herein as "environmental data") in various ways. The vehicle's sensor system may provide environmental data representing objects in the surrounding environment. For example, the vehicle may have various sensors including cameras, radars, lidars, microphones, wireless units, and other sensors. Each of these sensors may communicate environmental data to a processor within the vehicle regarding the information received by each respective sensor.

[0086] In one example, a camera may be configured to capture still images and / or video. In some embodiments, the vehicle may have two or more cameras positioned in different orientations. Also, in some embodiments, the camera may be movable to capture images and / or video in different directions. The camera may be configured to store the captured images and video in memory for later processing by the vehicle's processing system. The captured images and / or video may be environmental data. Further, the camera may include an image sensor as described herein.

[0087] In another embodiment, the radar may be configured to transmit electromagnetic signals reflected by various objects near the vehicle and then capture the electromagnetic signals reflected from the objects. The captured reflected electromagnetic signals may enable the radar (or the processing system) to make various determinations about the objects that reflected the electromagnetic signals. For example, the distances and positions to various reflecting objects can be determined. In some embodiments, the vehicle may have two or more radars in different orientations. The radar may be configured to store the captured information in memory for later processing by the vehicle's processing system. The information captured by the radar may be environmental data.

[0088] In another embodiment, the lidar may be configured to transmit electromagnetic signals (e.g., infrared light from a gas or diode laser, or from other possible light sources) reflected by target objects near the vehicle. The lidar may be capable of capturing the reflected electromagnetic (e.g., infrared light) signals. The captured reflected electromagnetic signals may enable the ranging system (or the processing system) to determine the distances to various objects. The lidar can also determine the speed or velocity of the target object and store it as environmental data.

[0089] Additionally, in one embodiment, a microphone may be configured to capture the audio of the vehicle's surrounding environment. The sound captured by the microphone may include the sirens of emergency vehicles and the sounds of other vehicles. For example, the microphone may capture the sound of the siren of an ambulance, a fire truck, or a police vehicle. The processing system may be able to detect that the captured audio signal indicates an emergency vehicle. In another embodiment, the microphone may capture the sound of the exhaust of another vehicle, such as that from a motorcycle. The processing system may be able to detect that the captured audio signal indicates a motorcycle. The data captured by the microphone may form part of the environmental data.

[0090] In yet another embodiment, the wireless unit may be configured to transmit an electromagnetic signal that may take the form of a Bluetooth signal, an 802.11 signal, and / or other wireless technology signals. The first electromagnetic radiation signal may be transmitted via one or more antennas located at the wireless unit. Further, the first electromagnetic radiation signal may be transmitted in one of many different wireless signal modes. However, in some embodiments, it is desirable to transmit the first electromagnetic radiation signal in a signal mode that requests a response from a device located in the vicinity of the autonomous or semi-autonomous vehicle. The processing system may be able to detect nearby devices based on the responses returned to the radio unit and use this communicated information as part of the environmental data.

[0091] In some embodiments, the processing system may be able to combine information from various sensors to further determine the vehicle's surrounding environment. For example, the processing system may combine data from both radar information and captured images to determine whether another vehicle or a pedestrian is in front of the autonomous or semi-autonomous vehicle. In other embodiments, other combinations of sensor data may be used by the processing system to make determinations about the surrounding environment.

[0092] While operating in autonomous mode (or semi-autonomous mode), the vehicle can control its operation with little or no human input. For example, if a human operator inputs an address into the vehicle, the vehicle may be able to drive to the specified destination without further input from the human (e.g., without the human having to operate or touch the brake / accelerator pedals). Further, while the vehicle is operating autonomously or semi-autonomously, the sensor system can receive environmental data. The vehicle's processing system can change the control of the vehicle based on the environmental data received from various sensors. In some embodiments, the vehicle may change its speed in response to environmental data from various sensors. The vehicle can change its speed to avoid obstacles and comply with traffic laws. When the processing system in the vehicle detects an object near the vehicle, the vehicle may be able to change its speed or move in another way.

[0093] If the vehicle detects an object but does not have sufficient confidence in the detection of the object, the vehicle can request a human operator (or a more powerful computer) to perform one or more remote assistance tasks, such as (i) verifying whether the object actually exists in the surrounding environment (e.g., whether there is actually a stop sign or not), (ii) verifying whether the vehicle's detection of the object is correct, (iii) correcting the detection if it is incorrect, and / or (iv) providing supplementary instructions (or modifying current instructions) for the autonomous or semi-autonomous vehicle. The remote assistance tasks can also include the human operator providing instructions for controlling the operation of the vehicle (e.g., if the human operator determines that the object is a stop sign, instructing the vehicle to stop at the stop sign), but in some scenarios, the vehicle itself can control its own operation based on the human operator's feedback related to the detection of the object.

[0094] To facilitate this, the vehicle may analyze environmental data representing objects in the surrounding environment to determine at least one object having a detection reliability below a threshold. The vehicle's processor may be configured to detect various objects in the surrounding environment based on environmental data from various sensors. For example, in one embodiment, the processor may be configured to detect objects that may be important for the vehicle to recognize. Such objects may include pedestrians, cyclists, road signs, other vehicles, indicator signals of other vehicles, and various other objects detected in the captured environmental data.

[0095] The detection reliability may indicate the likelihood that the determined object is correctly detected or exists within the surrounding environment. For example, the processor may perform object detection of an object in the image data in the received environmental data and, based on the inability to detect that at least one object has a detection reliability exceeding a threshold, may determine that the object has a detection reliability below the threshold. When the result of object detection or object recognition of an object is not conclusive, the detection reliability may be low or below a set threshold.

[0096] The vehicle may detect objects in the surrounding environment in various ways depending on the source of the environmental data. In some embodiments, the environmental data may be image or video data coming from a camera. In other embodiments, the environmental data may come from a lidar. The vehicle may analyze the captured image or video data to detect objects in the image or video data. The method and apparatus may be configured to monitor the image and / or video data for the presence of objects in the surrounding environment. In other embodiments, the environmental data may be radar, audio, or other data. The vehicle may be configured to identify objects in the surrounding environment based on the radar, audio, or other data.

[0097] In some embodiments, the techniques used by a vehicle to detect an object may be based on a set of known data. For example, data related to environmental objects may be stored in a memory located in the vehicle. The vehicle may determine an object by comparing the received data with the stored data. In other embodiments, the vehicle may be configured to determine an object based on the context of the data. For example, a street sign related to construction may generally have an orange color. Thus, the vehicle may be configured to detect an orange object located near the roadside as a construction-related street sign. Additionally, when the vehicle's processing system detects an object in the captured data, it can also calculate the confidence level of each object.

[0098] Furthermore, the vehicle may also have a confidence threshold. The confidence threshold may vary depending on the type of object being detected. For example, for an object that may require a quick response action from the vehicle, such as the brake light of another vehicle, the confidence threshold may be low. However, in other embodiments, the confidence threshold may be the same for all detected objects. If the confidence associated with the detected object is higher than the confidence threshold, the vehicle may assume that the object is correctly recognized and, based on that assumption, adjust the vehicle's control responsively.

[0099] If the confidence associated with the detected object is lower than the confidence threshold, the action taken by the vehicle may change. In some embodiments, the vehicle may react as if the detected object exists despite the low confidence level. In other embodiments, the vehicle may react as if the detected object does not exist.

[0100] When the vehicle detects an object in the surrounding environment, it can also calculate a confidence level that is associated with the specific detected object. The confidence level can be calculated in various ways depending on the embodiment. In one example, when detecting an object in the surrounding environment, the vehicle may compare the environmental data with predetermined data associated with known objects. The closer the match between the environmental data and the predetermined data, the higher the confidence level. In other embodiments, the vehicle may use a mathematical analysis of the environmental data to determine the confidence level associated with the object.

[0101] In response to a determination that an object has a detection confidence level below a threshold, the vehicle may send a request for remote assistance to a remote computing system, along with the detection of the object. As discussed above, the remote computing system can take various forms. For example, the remote computing system can be a computing device within a vehicle separate from the vehicle, with a touch screen interface or the like for displaying remote assistance information by which a human operator can interact with the passengers or driver of the vehicle. Additionally or alternatively, as another example, the remote computing system can be a remote computer terminal or other device located at a location not near the vehicle.

[0102] The request for remote assistance may include environmental data including the object, such as image data, audio data, etc. The vehicle may send the environmental data to the remote computing system over a network (e.g., network 304), in some embodiments via a server (e.g., server computing system 306). The human operator of the remote computing system may then use the environmental data as a basis for responding to the request.

[0103] In some embodiments, when an object is detected as having a confidence level below a confidence threshold, a preliminary identification may be given to the object, and the vehicle may be configured to adjust the operation of the vehicle in response to the preliminary identification. Such adjustment of the operation may take the form of, among other possible adjustments, stopping the vehicle, switching the vehicle to a manual control mode, changing the speed (e.g., speed and / or direction) of the vehicle.

[0104] In other embodiments, even when the vehicle detects an object having a confidence level that meets or exceeds the threshold, the vehicle may operate according to the detected object (e.g., stop when the object is detected with a high confidence level as a stop sign), but the vehicle may be configured to request remote assistance either simultaneously with (or after) operating according to the detected object.

[0105] FIG. 4 is a block diagram of a system according to an exemplary embodiment. In particular, FIG. 4 shows a system 400 including a system controller 402, a radar system 410, a sensor 412, and a controllable component 414. The system controller 402 includes a processor 404, a memory 406, and instructions 408 stored on the memory 406 and executable by the processor 404 to implement functions such as the operations disclosed herein.

[0106] The processor 404 can include one or more processors such as one or more general-purpose microprocessors (e.g., having a single core or multiple cores) and / or one or more dedicated microprocessors. The one or more processors can include, for example, one or more central processing units (CPUs), one or more microcontrollers, one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), one or more ASICs, and / or one or more field-programmable gate arrays (FPGAs). Other types of processors, computers, or devices configured to execute software instructions are also contemplated herein.

[0107] The memory 406 may include computer-readable media such as, without limitation, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), non-volatile random access memory (e.g., flash memory), solid state drive (SSD), hard disk drive (HDD), compact disc (CD), digital video disc (DVD), digital tape, read / write (R / W) CD, R / W DVD, etc., and may include non-transitory computer-readable media.

[0108] The radar system 410 can be used in an autonomous or semi-autonomous vehicle for navigation and object detection by using radio waves to detect and measure the distance, speed, and direction of objects in the surrounding environment. The radar system 410 can include one or more radar units, each consisting of a radar transmitter that emits radio waves and a radar receiver that captures the waves reflected from the objects. By analyzing the time it takes for the waves to return and their frequency shift (Doppler effect), the radar system 410 can determine the presence, location, and movement of the objects.

[0109] In the context of autonomous or semi-autonomous vehicles, the radar system 410 provides measurements that can assist with navigation and collision avoidance. The radar unit is typically mounted outside the vehicle, such as at the front, rear, and sides. During navigation, the radar system 410 may continuously emit radio waves in various directions to scan the environment around the vehicle. When the waves encounter an object, they bounce back to the radar receiver, enabling the radar system 410 to analyze the reflected waves and calculate the distance, relative speed, and angle of the object. This information is used by the vehicle's control system to make decisions and accordingly adjust the vehicle's trajectory, detecting and reacting to obstacles, pedestrians, vehicles, and other potential hazards within its path. By providing real-time data about the surrounding environment, the radar system 410 can enhance the vehicle's perception capabilities and contribute to safer and more reliable navigation.

[0110] In some aspects, such as cameras and lidars, the radar system 410 offers operational benefits over other types of sensors. Radar can function well in adverse weather conditions such as rain, fog, or dust, where other sensors may have limitations. In particular, the radio waves emitted by the radar system 410 can penetrate such adverse conditions and enable reliable object detection. For this reason, radar is particularly useful for improving the robustness and safety of autonomous or semi-autonomous vehicles in various weather scenarios. Additionally, radar is excellent at detecting the speed and relative speed of nearby objects, which is useful for evaluating the movement of surrounding vehicles, pedestrians, and other obstacles. By providing accurate speed information, the radar system 410 enables the vehicle (or the vehicle's driver) to make information-based decisions about potential collision risks and adjust its behavior accordingly. In some cases, the radar system 410 may also be able to provide a longer measurement range and a wider field of view compared to other sensors attached to the vehicle.

[0111] Similarly, system controller 402 can use the outputs from radar system 410 and sensor 412 to determine the characteristics of system 400 and / or the characteristics of the surrounding environment. For example, sensor 412 may include one or more of GPS, IMU, an image capture device (e.g., a camera), a light sensor, a thermal sensor, one or more lidar devices, and other sensors that indicate parameters related to system 400 and / or the surrounding environment. Radar system 410 is depicted as separate from sensor 412 for illustrative purposes, and in some embodiments, may be part of sensor 412 or considered as sensor 412.

[0112] Based on the characteristics of system 400 determined by system controller 402 based on the outputs from radar system 410 and sensor 412 and / or the surrounding environment, system controller 402 can control controllable component 414 to perform one or more actions. For example, system 400 may correspond to a vehicle, in which case controllable component 414 may include the vehicle's braking system, steering system, and / or acceleration system, and system controller 402 can change the modes of these controllable components based on the characteristics determined from radar device 410 and / or sensor 412 (e.g., when system controller 402 controls the vehicle in an autonomous or semi-autonomous mode). In some embodiments, radar device 410 and sensor 412 are also controllable by system controller 402.

[0113] As the number of automotive radars deployed increases, the total radiated amount of RF signals in the automotive band also increases, increasing the likelihood of experiencing the adverse effects of RFI. Generally, RFI occurs when external electromagnetic signals interfere with the ability of a radar system or other emitter to radiate and receive radio frequency waves, which can lead to distortion, degradation, or incorrect measurements. RFI can originate from various sources such as other electronic devices, communication signals, and environmental factors.

[0114] FIG. 5 is a conceptual diagram of a system 500 that enables universal exchange of spectrum utilization for RFI mitigation and avoidance. In an exemplary embodiment, system 500 is shown with remote computing system 502 in wireless communication with vehicle 504, vehicle 506, and signal receiver 508 via network 510. The conceptual diagram of system 500 is intended to show a simplified arrangement in which vehicles and other devices exchange spectrum utilization and other RFI data and permit RFI mitigation and avoidance. In a real-world implementation, any number of devices, vehicles, emitters, and other sources that can contribute to spectrum utilization exchange for the purpose of RFI mitigation and avoidance may be involved.

[0115] Generally, the RF spectrum represents the range of electromagnetic frequencies over which radio waves and wireless communication signals operate. The RF spectrum is typically divided into different bands, each with specific uses and regulations. For example, different bands define the frequencies at which various technologies such as radio, television, cellular networks, Wi-Fi, Bluetooth, radar, and satellite communication operate. Thus, efficient utilization of the RF spectrum enables modern communication systems to operate effectively.

[0116] System 500 is designed to enable efficient use of RF bands typically occupied by vehicle radar and other similar emitters. In particular, System 500 aggregates RF data from vehicle radar systems and other devices and then allows it to be used to better understand RFI in real time. Within System 500, remote computing system 502 can periodically or continuously aggregate RFI data and location data from various types of devices such as vehicles (e.g., vehicle 504 and vehicle 506) and other types of signal receivers (e.g., signal receiver 508). The remote computing system 502 can then provide and return the aggregated RFI data to the vehicles and other devices, thereby potentially allowing the vehicles and devices to predict RFI and adjust their operations to mitigate or avoid the adverse effects of RFI. The way the remote computing system 502 aggregates RFI data and deploys it to the devices can vary within the embodiments.

[0117] Remote computing system 502 represents one or more computing systems that are remotely located from vehicles and other signal emitters positioned at various locations. Generally, remote computing system 502 is a network setup where computing resources (e.g., processing power, storage, and applications) are located on one or more servers installed at a location separate from the vehicles and other RF devices. As shown in FIG. 5, remote computing system 502 can communicate with vehicles and other devices via network 510.

[0118] In some embodiments, the remote computing system 502 is located on a vehicle and can implement the techniques disclosed during navigation by the vehicle. The vehicle system can use the aggregated RFI data to better understand and adapt to RFI during route navigation. In some cases, the vehicle system can change the route or implement other mitigation strategies to reduce the impact of RFI detected at a particular location.

[0119] In an embodiment, the remote computing system 502 can aggregate RFI data from vehicle radar systems and other sources using various techniques. For example, the remote computing system 502 can periodically send queries to the vehicle and other devices for data updates. Once the data is collected, the remote computing system 502 can then process and aggregate the data into batches. Additionally, vehicles 504-506, signal receivers 508, and other devices can also have the ability to send data updates including location and RFI information to the remote computing system 502 each time the device obtains new RFI information. For example, vehicles 504 and 506 can communicate RFI and location data in real time when navigating a route.

[0120] In the embodiment shown in FIG. 5, vehicles 504 and 506 are included to represent vehicles equipped with a radar system for measuring aspects of the environment during navigation. Thus, vehicles 504 and 506 can measure RFI data and provide it, along with location data, to remote computing system 502 to enable aggregation of RFI data associated with the locations where vehicles 504-506 have traveled. The embodiment illustrated in FIG. 5 shows two vehicles (vehicles 504-506), but any number of vehicles can communicate with remote computing system 502. In particular, as the number of vehicles and other devices involved in system 500 increases, the capabilities of system 500 can increase exponentially due to the network effect. In a larger vehicle base, remote computing system 502 can acquire more RFI data and thus develop RFI representations based on a more diverse and comprehensive RFI data set. With a constant supply of new RFI data from vehicles and / or other devices, the generated RFI representations can be updated to provide a more accurate depiction of potential RFI at various locations. Additionally, in some cases, multiple vehicles can provide RFI data for the same location, thereby allowing remote computing system 502 to maintain a more accurate representation of RFI at that location. Remote computing system 502 can monitor the trends in RFI data acquired over time for various locations and detect periods of several days or weeks when RFI is likely to be higher than normal.

[0121] In addition to vehicles 504 and 506, system 500 is also shown using signal receiver 508, which is included to represent any non-vehicle device designed to capture and process electromagnetic signals within the RF spectrum that are monitored by remote computing system 502. For example, signal receiver 508 can be a fixed radar receiver positioned along a road or another type of infrastructure. Some exemplary infrastructure can include spectrum analyzers and RF meters that can measure RF signals over different frequencies and provide information about the strength of signals and interference in the environment. Additionally, the infrastructure can also include directional antennas, traffic monitoring systems, and traffic light and control systems. In some cases, the traffic system can include sensors that enable the measurement of RFI at that location. Other types of signal receivers and infrastructure can be used within the examples to monitor the RF infrastructure and diagnose signal interference. Similar to vehicles 504 - 506, signal receiver 508 represents other types of infrastructure that can contribute to system 500 by measuring RFI and location data and providing it directly to remote computing system 502 or to vehicles 504 - 506.

[0122] As shown in FIG. 5, the remote computing system 502 can communicate with vehicles 504-506, signal receivers 508, and other potential devices via a network 510. Similar to the network 304 shown in FIG. 3, the network 510 represents an infrastructure that enables wireless communication between the remote computing system 502 and various devices that can share RFI data to assist in RFI mitigation and avoidance. For example, the network 510 may involve one or more cellular networks that provide mobile communication services over a wide geographic area. Technologies such as 4G LTE and 5G can allow for high-speed data transfer between the remote computing system 502, the vehicles 504-506, and the signal receivers 508. In some embodiments, the network 510 can include one or more satellite networks that can provide coverage in various locations. Similarly, the network 510 can also include one or more WiFi networks that allow for communication between different computing devices.

[0123] As further shown in FIG. 5, the remote computing system 502 includes a communication interface 512, an RFI map generation module 514, an RFI category database 516, and an RFI threshold module 518. The communication interface 512 can communicate with other devices, including aggregating RFI data from vehicles 504, vehicle 506, signal receivers 508, and other sources, used by the remote computing system 502. Generally, the communication interface 512 can be used to enable two-way communication and data transfer with vehicles and other types of devices. In some embodiments, the communication interface 512 uses encrypted communication techniques when transferring data between the remote computing system 502 and other devices.

[0124] The RFI map generation module 514 represents a module designed to acquire and format RFI data aggregated from vehicles 504-506 and other devices. Although the RFI map generation module 514 is shown as part of the remote computing system 502, vehicles 504-506 may similarly include the RFI map generation module 514 and may implement the techniques described herein using RFI data obtained from various sources. The formatting and compilation of RFI data aggregated from various sources may vary within the examples and may depend on the delays present during communication with various devices. For example, the RFI map generation module 514 may be used by the remote computing system 502 to generate one or more representations based on RFI data obtained from vehicles 504-506, signal receivers 508, and other sources. The RFI map generation module 514 may use map data and other information stored locally in the remote computing system 502 and obtained from external sources (e.g., a map database).

[0125] In some examples, vehicles 504-506 and other devices may perform operations similar to those of the RFI map generation module 514. For example, vehicles 504-506 may format and compile RFI data obtained from each other and from other sources (e.g., signal receivers 508 and the remote computing system 502).

[0126] In some embodiments, the RFI map generation module 514 can generate one or more RFI heat maps that illustrate the spectral power within a set of sub-bands arranged according to location. The RFI heat maps can then be distributed to the vehicle and third-party sources by providing updates to the RFI heat maps. For example, the remote computing system 502 can divide and provide data representing a portion of the RFI heat map selected by the system based on the locations of the vehicle and third-party sources. In some cases, the system provides a portion of the heat map based on the route provided by the vehicle. The RFI heat map can, in some embodiments, be a database and can associate numerical RFI intensity levels and waveform parameters with different location coordinates.

[0127] In some embodiments, the RFI heat map can be visually represented using different colors similar to common heat maps at different locations to represent the spectral power detected at those locations. For example, the RFI heat map can be a graphical representation of data where individual values are represented as colors. The RFI heat map can be useful for visualizing the distribution and density of data points within a two-dimensional (2D) space such as a grid or map. By using the RFI heat map and assigning different colors to different values, patterns, trends, and variations in the data can be displayed. For example, a color scale can be used to represent various shades of color between low RFI values detected at a location (e.g., blue or green) to high RFI values detected at a location (e.g., red or yellow). In some cases, the RFI data can be divided into smaller segments (cells), and each cell represents a specific area. The value associated with each cell determines the color that the system assigns to the RFI heat map. The RFI heat map can currently include a legend to enable a third party to understand its content, such as an area with a high level of RFI that can potentially interfere with radar performance.

[0128] Using the RFI category database 516, details related to RFI that a vehicle may encounter in various locations can be further remembered. For example, using the RFI category database 516, information such as the location and image about the type of the source of RFI, such as a fixed source of RFI, can be remembered. The remote computing system 502 can compare the waveforms and other reported RFI data received from the vehicle system with the waveforms or other parameters associated with that location when the vehicle system is preparing to move to a certain location. Thereby, the remote computing system 502 can determine a mitigation command for the vehicle radar system, such as an adjustment of the frequency or waveform parameters used by the vehicle radar system.

[0129] The RFI threshold module 518 can use the RFI data and trends to determine RFI thresholds for various locations represented by the RFI map generated by the RFI map generation module 514. The threshold refers to a predetermined value or level that, when exceeded, triggers the remote computing system 502 to perform a specific action or response. In this way, the RFI threshold module 518 can monitor the RFI state, detect anomalies in the RFI data, and generate a threshold to enable the remote computing system 502 to make a decision based on the observed values.

[0130] Generally, the RFI threshold module 518 can set thresholds for different locations based on the input RFI data. The value of the threshold assigned to a location can, in some cases, be determined based on the level of RFI that affects radar performance. The remote computing system 502 can then use the RFI threshold module 518 to monitor the RFI data assigned to different locations when new RFI data is provided from the vehicle and other systems positioned at various locations. When new data enters, the RFI threshold module 518 can compare the aggregated RFI current values for different locations with a predetermined threshold. If the current RFI value exceeds the predetermined threshold for a location, a violation is detected, which can trigger the remote computing system 502 to provide a warning or notification to the vehicle radar systems within the area, log and record the data associated with the RFI violation, and / or perform actions such as other automatic responses to address or mitigate the problem.

[0131] The remote computing system 502 can provide mitigation techniques to the vehicles 504 - 506 and other sources (e.g., the signal receiver 508) to reduce or avoid RFI at different locations. For example, the mitigation technique may involve instructing the vehicle radar system to perform frequency hopping, where the radar system periodically changes its operating frequency. Additionally, the remote computing system 502 can also assign specific frequency bands to individual transmitters. For example, each vehicle 504, 506 may receive different frequency channels to operate based on the remote computing system 502 determining that the vehicles 504 - 506 are in close proximity to each other. By switching frequencies, the vehicles 504 - 506 can avoid interference from other radar systems or electromagnetic interference sources that may be present in a particular frequency range. In some embodiments, the vehicles 504 - 506 and other sources can locally determine the mitigation technique based on information received from other sources such as other vehicles, the remote computing system 502, and other infrastructure.

[0132] In some cases, the remote computing system 502 may provide instructions for adjusting waveform parameters such as pulse width, repetition interval, modulation scheme, and / or amplitude based on the environment and interference conditions specified by the RFI map generation module 514. Similarly, the remote computing system 502 may provide instructions to the vehicles 504, 506, and / or the signal receiver 508 to perform side-lobe blanking when positioned at some locations with high-density RFI specified by the RFI map generation module 514. Side-lobe blanking can help prevent interference from affecting the main detection area of the radar. The mitigation instructions can also specify pulse compression techniques, which involve transmitting long-time waveforms and then compressing them in the receiver to improve target resolution. This allows the radar to distinguish between closely spaced targets and tolerate the adverse effects of RFI. The in-vehicle system can also determine mitigation instructions for the vehicle radar system to utilize to reduce the adverse effects of RFI.

[0133] In some embodiments, the remote computing system 502 may provide information to the vehicles 504 and 506 informing them that the vehicles 504 and 506 are positioned in close proximity to each other, thereby enabling the vehicles 504-506 and other nearby radar systems to create a temporary cooperative radar network. Similarly, the vehicles 504-506, the signal receiver 508, and other sources can share information indicating their positions and RFI usage. In situations where multiple radars are operating in close proximity (e.g., an urban environment), the radar systems can share information about their operating frequencies and waveforms to help avoid interference and improve performance through cooperation. Thus, the vehicles 504 and 506 can participate in communication to determine strategies for reducing interference. For example, the vehicles 504-506 can adjust transmission parameters via wireless communication to reduce interference.

[0134] Vehicles 504-506 may also implement the disclosed techniques, which may involve using information provided by remote computing system 502. For example, each vehicle may communicate with other vehicles and infrastructure located along the route the vehicle is navigating. The communication enables each vehicle to collect and aggregate RFI data to better understand and adapt to the surrounding environment. Vehicles 504-506 can determine mitigation strategies to implement based on RFI data aggregated from various sources. Mitigation strategies can involve changes to waveform, transmission pattern, frequency, ramp speed, polarization, timing, or other parameters. Other mitigation strategies can also include reducing reliance on radar in high RFI environments, particularly modifying the route to avoid areas with high RFI, adjusting the transmission pattern with other nearby vehicles to reduce interference for cooperative radar systems, or combinations of mitigation techniques.

[0135] FIG. 6 is a flowchart of a method for universal spectrum utilization exchange for RFI mitigation and avoidance, according to an exemplary embodiment. Method 600 may include one or more operations, functions, or actions, as illustrated by one or more of blocks 602, 604, 606, and 608. The blocks are illustrated in a sequential order, but these blocks may, in some cases, be performed in parallel and / or in an order different from the order described herein. Also, various blocks may be combined into fewer blocks, divided into additional blocks, and / or removed based on the desired implementation.

[0136] In addition, for the method 600 and other processes and methods disclosed herein, the flowchart illustrates the functionality and operation of one possible implementation of the present embodiment. In this regard, each block may represent a module, segment, or portion of program code that includes one or more instructions executable by a processor to implement a specific logical function or step in the process. The program code may be stored on any type of computer-readable medium or memory, such as a storage device including, for example, a disk or hard drive. In some examples, a remote computing system implements the method 600. In other examples, an in-vehicle vehicle system can implement the method 600. Similarly, other types of computing devices and systems can be programmed to implement the method 600.

[0137] In block 602, the method 600 involves receiving RFI data and location data from one or more vehicles or infrastructure locations. The RFI data represents RFI parameters estimated by each vehicle during navigation or by each infrastructure. A vehicle can potentially detect and estimate RFI parameters experienced along various roads during navigation. Similarly, an infrastructure location can include a spectrum analyzer, RF meter, traffic monitoring system, signal lights and control systems, and other types of devices capable of determining RFI data at the location.

[0138] The RFI parameters may include, but are not limited to, the power level, frequency sub-band, bearing line, location, and waveform characteristics estimated for the RFI detected during movement. The vehicle can communicate the RFI parameters, along with location data, to a computing device system in substantially real-time. For example, the remote computing system 502 shown in FIG. 5 can communicate with the vehicle and other systems to receive RFI data estimated by the vehicle radar system and other emitters positioned within the environment. In some cases, the RFI data can include data generated by passive receive-only devices positioned at various locations.

[0139] In some embodiments, the computing system may receive, from each vehicle, first RFI data representing the parameters used by the vehicle radar system to transmit a radar signal and location data representing the direction of travel of the vehicle. For example, the first RFI data represents transmission waveform metadata including the waveform and frequency sub-band parameters used by the vehicle radar system to transmit the radar signal. The computing system may also receive second RFI data corresponding to external RFI signals detected by the vehicle radar system on each vehicle, where the external RFI signals originated from emitters positioned remotely from the vehicle. The second RFI data can convey waveform metadata representing the waveform and frequency sub-band parameters estimated for the external RFI signals, as well as location data representing the estimated location of the emitter positioned remotely from the vehicle.

[0140] In block 604, method 600 involves aggregating the RFI data, based on the location data, to generate a representation indicating the power level and frequency sub-bands of the RFI detected at one or more locations. In some embodiments, the representation can convey various information aggregated from the RFI data, such as the bearing line and location of the RFI source detected during navigation.

[0141] In some embodiments, a computing system generates an RFI heat map. The RFI heat map matches RFI data with roads used by a vehicle. In some cases, the computing system may generate the RFI heat map based on trends in RFI data received from the vehicle, estimate the positions of fixed RFI sources, and cause the RFI heat map to convey the estimated positions of the fixed RFI sources. For example, the system can obtain map data representing the positions of roads and buildings at various locations. The computing system can then assign the RFI data to the map data such that the representation conveys the RFI intensity at the positions of roads and buildings at multiple locations. The representation can visually display the RFI intensity at various locations using colors similar to a heat map.

[0142] In block 606, method 600 involves obtaining a location from a vehicle. For example, a computing system may receive, from the vehicle, data specifying a planned route for the vehicle. Based on the planned route for the vehicle, the computing system may identify a portion of the representation that corresponds to the vehicle's route and provide the portion of the representation to the vehicle.

[0143] In some cases, the computing system receives, from the vehicle, data specifying a planned route for the vehicle and then provides data representing respective portions of the RFI heat map based on the planned route for the vehicle. The vehicle is configured to adjust radar operation or a route based on the data representing respective portions of the RFI heat map.

[0144] In block 608, method 600 involves providing at least a portion of the representation to the vehicle based on the vehicle's location. In some embodiments, the computing system may determine a mitigation command for use by the vehicle during navigation of a route based on a portion of the representation. The computing system can then provide the mitigation command to the vehicle in addition to the portion of the representation.

[0145] In some cases, the representation can convey RFI information estimated from a road-level perspective in accordance with how the vehicle travels along a route. In particular, the representation can indicate the location of RFI sources with respect to the road (when available at this level of detail) and can include representing bearing lines for each RFI source. In this way, the vehicle system can understand the location and arrangement of RFI sources with respect to the road as the vehicle moves. In other cases, specific location and bearing information may not be available when distributing the representation data to the vehicle and other systems.

[0146] In some embodiments, method 600 further involves receiving additional RFI data and location data from one or more vehicles. The computing system can then modify the representation based on the additional RFI data and location data and provide at least a portion of the modified representation to one or more vehicles. In other cases, participating radar systems may choose to upload their waveform parameters, which may be bookkept and shared in a similar manner.

[0147] In some embodiments, method 600 involves receiving from the vehicle data specifying the vehicle's route and identifying a portion of the representation that matches the vehicle's route. The computing system can then provide the portion of the representation to the vehicle.

[0148] In some embodiments, the computing system receives a planned route from a vehicle and provides one or more portions of an RFI heat map based on the planned route. The vehicle is configured to adjust radar operation or the planned route based on one or more portions of the RFI heat map. Optionally, the computing system may adjust the RFI heat map over time as the vehicle supplies updated RFI data from various locations. The computing system can monitor the adjustment of RFI data over the locations represented by the RFI heat map and determine trends. The computing system may also receive additional RFI data and location data from multiple vehicles and modify the RFI heat map based on the additional RFI data. The computing system may identify one or more trends for RFI data in one or more radar directions at one or more locations based on modifying the RFI heat map. For example, a trend may indicate a change in RFI at a given location based on a particular time. The computing system may then provide a mitigation command to one or more vehicles traveling near one or more locations based on one or more trends for RFI data at one or more locations.

[0149] In some embodiments, the generated representation is an RF-dependent (center frequency-based) heat map that uses multiple colors to visually convey the total received power divided into several RFI sub-bands (e.g., 76 - 77, 77 - 78, 78 - 79, 79 - 80, 80 - 81 GHz) corrected for the respective locations and directions of the individual radars of the receiving vehicle. Optionally, the computing system may monitor RFI data against an RFI threshold for a given location. In response to subsequent RFI data for a given location that exceeds the RFI threshold for that location, the computing system can trigger an alert for the given location. For example, the computing system can provide a mitigation command to the vehicle based on the location of the vehicle corresponding to the given location. In other cases, knowledge of the excess power within a sub-band can be used to correct the detected power level to reduce false detections or increase the threshold associated with background noise that may be related to an increase in RFI. Similar to the RFI heat map, any number of RFI-related attributes may be shared. These attributes (which may be used by the radar system for the purpose of reducing the impact from RFI) typically include, but are not limited to, pulse width, PRI (pulse repetition interval), center frequency, and waveform bandwidth (or ramp rate), as well as dwell time (total CPI duration), which are the main parameters of interest.

[0150] The computing system can also receive from the vehicle a route for navigation by the vehicle and, based on the route and the representation, determine a mitigation command for the vehicle. The computing system can then provide the mitigation command to the vehicle and enable the vehicle radar system on the vehicle to operate in accordance with the mitigation instructions during navigation of the route by the vehicle. For example, the computing system can determine a first frequency band for a first portion of the route for use by the vehicle radar system on the vehicle during navigation of the first portion of the route and a second frequency band for a second portion of the route for use by the vehicle radar system on the vehicle during navigation of the second portion of the route. In some cases, the computing system can determine one or more modifications to the route and provide the one or more modifications to the route to the vehicle.

[0151] In some embodiments, the vehicle radar can be used with control electronics that can include one or more field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs). For example, the radar unit can generate and receive complex signals that require significant processing. The one or more control electronics can be programmed to implement various signal processing algorithms such as filtering, modulation / demodulation, noise reduction, and digital beamforming. These operations help to extract relevant information from the received radar signals, improve signal quality, and enhance target detection and tracking. Additionally, the radar system often involves the conversion of analog signals to digital form for further processing. The control electronics can include analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) to facilitate these conversions. The control electronics can receive analog signals from the radar sensors, digitize them, and process the digital data for analysis and interpretation.

[0152] In addition, the control electronics can also provide the ability for real-time control and adjustment of various radar system components. For example, the control electronics can handle synchronization, timing generation, and system control, ensuring proper timing and sequencing of operations within the radar system. This real-time control is important for the transmission and reception of accurate and synchronized signals. The control electronics can efficiently handle the large amount of data generated by the radar system. The control electronics can implement data storage, buffering, and data flow management techniques, enabling efficient data handling during signal transmission. This includes tasks such as data compression, data packetization, and data routing, ensuring smooth and reliable data transmission within the radar system. The control electronics can also integrate various interfaces and protocols required for radar signal transmission, such as processors, memory modules, communication modules, and display units. The control electronics can provide the interface logic necessary to facilitate seamless data exchange between these components, enabling efficient data flow and system integration. The control electronics can also be reconfigured and customized to meet specific radar system requirements and adapt to changing operating needs. This allows radar system designers to implement and optimize algorithms and functions specific to their applications, improving performance and efficiency.

[0153] This disclosure is not limited to the specific embodiments described in this application, and the specific embodiments are intended as examples of various aspects. As will be apparent to those skilled in the art, many modifications and variations can be made without departing from the spirit and scope of this disclosure. In addition to the methods and apparatuses listed herein, functionally equivalent methods and apparatuses within the scope of this disclosure will be apparent to those skilled in the art from the foregoing description. Such modifications and variations are intended to be within the scope of the appended claims.

[0154] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the figures, like symbols typically refer to like components unless the context indicates otherwise. The exemplary embodiments described herein and in the figures are not intended to be limiting. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure generally described herein and illustrated in the figures can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are explicitly contemplated.

[0155] With respect to any or all of the message flow diagrams, scenarios, and flowcharts considered in the figures and herein, each step, block, operation, and / or communication can represent the processing of information and / or the transmission of information according to an exemplary embodiment. Alternative embodiments are within the scope of these exemplary embodiments. In these alternative embodiments, for example, operations described as steps, blocks, transmissions, communications, requests, responses, and / or messages can be executed in an order different from that shown or discussed, such as substantially simultaneously or in the reverse order, depending on the relevant functions. Further, more or fewer blocks and / or operations can be used in any of the message flow diagrams, scenarios, and flowcharts considered herein, and these message flow diagrams, scenarios, and flowcharts can be combined with each other, in part or in whole.

[0156] Steps, blocks, or operations corresponding to the processing of information may correspond to circuitry configured to perform specific logical functions of the methods or techniques described herein. Alternatively or additionally, steps or blocks corresponding to the processing of information may correspond to a portion of a module, segment, or program code (including associated data). The program code may include one or more instructions executable by a processor to perform specific logical operations or actions in a method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, such as a storage device including RAM, disk drive, solid state drive, or another storage medium.

[0157] Furthermore, steps, blocks, or operations corresponding to one or more information transmissions may correspond to information transmissions between software modules and / or hardware modules in the same physical device. However, other information transmissions may be between software modules and / or hardware modules in different physical devices.

[0158] The specific arrangements shown in the figures should not be regarded as limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Still further, exemplary embodiments may include elements not illustrated in the figures.

[0159] Although various aspects and embodiments are disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for illustrative purposes and not intended to be limiting, and the true scope is indicated by the following claims.

Claims

1. 1. A method comprising: receiving, in a computing system, radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, the RFI data representing RFI parameters estimated by each navigating vehicle or by each infrastructure location; aggregating the RFI data based on the location data to generate a representation indicative of power levels and frequency subbands of RFI detected at one or more locations; obtaining a location from a vehicle in the computing system; providing, by the computing system and to the vehicle, at least a portion of the representation based on the location of the vehicle; A method comprising:

2. determining the power level, the frequency sub-bands, one or more bearing lines, and waveform parameters of RFI detected at the one or more locations based on RFI data received from one or more vehicles navigating at the one or more locations; aggregating the RFI data to generate the representation, The method of claim 1 , further comprising generating the representation to further show the waveform parameters and one or more bearing lines of RFI detected at the one or more locations.

3. receiving additional RFI data and location data from one or more vehicles or infrastructure locations; modifying the representation based on the additional RFI data and location data; The method of claim 1 , further comprising: providing at least a portion of the modified representation to one or more vehicles.

4. obtaining the location from the vehicle; receiving data from the vehicle specifying a planned route for the vehicle; identifying a portion of the representation that matches the route of the vehicle based on the route planned for the vehicle; providing said portion of said representation to said vehicle.

5. determining mitigation instructions for use by the vehicle during navigation of the route based on the portion of the representation, the mitigation instructions being dependent on RFI parameters estimated by one or more vehicles positioned along the route; The method of claim 4 , further comprising: providing the mitigation command to the vehicle in addition to the portion of the representation.

6. receiving RFI data and location data from one or more vehicles or infrastructure locations; 2. The method of claim 1, comprising receiving, from each vehicle, first RFI data representative of parameters used by a vehicle radar system to transmit radar signals and location data representative of the direction of travel of the vehicle.

7. the first RFI data representative of parameters used by the vehicle radar system to transmit radar signals; The method of claim 6 , including transmit waveform metadata representing waveforms and frequency sub-band parameters used by the vehicle radar system to transmit radar signals.

8. receiving RFI data and location data from a plurality of said vehicles; 8. The method of claim 7, further comprising receiving second RFI data corresponding to external RFI detected by the vehicle radar system on each vehicle, the external RFI arising from an emitter positioned remotely from the vehicle.

9. the second RFI data corresponding to an external RFI signal; 10. The method of claim 8, further comprising receiving waveform metadata representative of estimated waveform and frequency sub-band parameters for the external RFI signal and location data representative of an estimated position of the emitter located remotely from the vehicle.

10. aggregating the RFI data into the representation, generating an RFI heat map, the RFI heat map matching the RFI data to roads used by the one or more vehicles; and estimating locations of fixed RFI sources based on trends in the RFI data; generating the RFI heat map, The method of claim 1 , comprising generating the RFI heat map such that the RFI heat map conveys the estimated locations of the fixed RFI sources.

11. obtaining the location from the vehicle; receiving data from the vehicle specifying a planned route for the vehicle; Providing at least the portion of the representation 11. The method of claim 10, comprising providing data representing respective portions of the RFI heatmap based on the route planned for the vehicle, the vehicle being configured to adjust radar operation or the route based on the data representing the respective portions of the RFI heatmap.

12. receiving additional RFI data and location data from one or more vehicles or infrastructure locations; Modifying the RFI heatmap; and 12. The method of claim 11, further comprising: identifying one or more trends in RFI data at one or more locations based on modifying the RFI heat map, the trends indicating changes in RFI at a given location based on a particular time of day.

13. The method of claim 12 , further comprising providing mitigation instructions to one or more vehicles traveling near the one or more locations based on the one or more trends in RFI data at the one or more locations.

14. obtaining map data representative of road and building locations; aggregating the RFI data into a representation based on the location data; The method of claim 1 , comprising associating the RFI data with the map data such that the representation conveys RFI strengths for roads and buildings.

15. receiving data from the vehicle specifying a planned route for the vehicle; determining, based on the route planned for the vehicle and the representation, a first frequency band for use by the vehicle while navigating a first portion of the route and a second frequency band for use by the vehicle while navigating a second portion of the route; 2. The method of claim 1, further comprising: providing mitigation instructions to the vehicle, the mitigation instructions specifying the first frequency band for use by the vehicle while navigating the first portion of the route and the second frequency band for use by the vehicle while navigating the second portion of the route.

16. 1. A system comprising: a computing device, the computing device comprising: receiving radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, the RFI data representing RFI parameters estimated by each navigating vehicle or by each infrastructure location; aggregating the RFI data based on the location data to generate a representation indicative of power levels and frequency subbands of RFI detected at one or more locations; Obtaining a location from a vehicle; and providing, to the vehicle, at least a portion of the representation based on the location of the vehicle; A system configured to:

17. 17. The system of claim 16, wherein the representation is an RFI heat map that uses multiple colors to visually convey the power levels and frequency sub-bands at each location.

18. the computing device comprising: monitoring the RFI data against an RFI threshold for a given location; 17. The system of claim 16, further configured to trigger an alert for the given location in response to subsequent RFI data for the given location exceeding the RFI threshold for the given location.

19. the computing device comprising:

20. The system of claim 18, further configured to provide abatement instructions to the vehicle based on the location of the vehicle relative to the given location.

20. A non-transitory computer-readable medium configured to store instructions that, when executed by a computing system having one or more processors, cause the computing system to: receiving radio frequency interference (RFI) data and location data from one or more vehicles or infrastructure locations, the RFI data representing RFI parameters estimated by each navigating vehicle or by each infrastructure location; aggregating the RFI data based on the location data to generate a representation indicative of power levels and frequency subbands of RFI detected at one or more locations; Obtaining a location from a vehicle; and providing, to the vehicle, at least a portion of the representation based on the location of the vehicle; A non-transitory computer-readable medium for performing operations including:

Images