Multi-frequency radar array system and sensor fusion for viewing the area around corners during autonomous driving.

Abstract

Disclosed herein are systems, apparatus, and methods that can be used for autonomous driving and / or in autonomous vehicles. Some embodiments use an integrated large-aperture, multi-band radar subsystem to leverage the unique propagation characteristics of multi-band and / or multi-sensor technologies to significantly improve scene detection and understanding, particularly for viewing oncoming surroundings and identifying over-the-horizon targets. In some embodiments, at least one processor in the system is capable of jointly processing return signals from multiple bands to provide high accuracy in a variety of conditions (e.g., weather). The disclosed radar subsystems can be used alone or in conjunction with other sensing technologies, such as LiDAR and / or cameras.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 058,471, titled "Next Generation Sensor Technology For Autonomous Driving" (Agent Reference Number NPS005P), filed on 29 July 2020, and to U.S. Provisional Patent Application No. 63 / 092,336, titled "Next Generation Sensor Technology for Autonomous Driving" (Agent Reference Number NPS007P), filed on 15 October 2020. Both of the above applications are incorporated herein by reference in their entirety for all purposes. [Background technology]

[0002] background Today, many companies are creating or using technologies for autonomous driving (AD) applications (for example, in self-driving cars). A key consideration for AD systems is safety while operating in a nearly infinite combination of complex and dynamic scenarios. The goal of AD is to reduce the probability of accidents in all scenarios that an autonomous vehicle (AV) may encounter to near zero, thereby providing a much higher level of safety than a human driver can achieve.

[0003] Line-of-sight (LOS) and non-line-of-sight (NLOS) targets (e.g., other vehicles, people, animals, buildings) can pose a hazard to vehicles equipped with an AD system. The ability to detect NLOS targets would be particularly useful as it could help avoid hidden targets that could cause collisions. For example, in an urban environment, a car accelerating towards an intersection may initially be hidden by a building. Detecting the approaching car just before it can be perceived, and avoiding an accident if the car fails to stop at a red light at the intersection, would be desirable for an AD system on a vehicle approaching the intersection from another direction, which could potentially collide with the approaching car. Adverse weather and other conditions such as rain, snow, fog, bright sunlight, and dust can also present challenges to AD systems. [Overview of the project] [Means for solving the problem]

[0004] As a result of these challenges, even under ideal conditions, current AD solutions offer only partial L4 (highly automated driving) and / or partial L5 (fully automated driving) autonomy. Therefore, it is desirable to provide AD systems, methods, and devices that address the shortcomings of current approaches.

[0005] Brief explanation of the drawing The purposes, features, and advantages of this disclosure will be readily apparent from the following description of specific embodiments relating to the accompanying drawings. [Brief explanation of the drawing]

[0006] [Figure 1] Several embodiments of the system are shown. [Figure 2] This provides a conceptual diagram of the relationships between various signals transmitted and received by a system according to several embodiments. [Figure 3] The following shows specific components of the system transmitter according to several embodiments. [Figure 4]The following shows specific components of the transmitter array of the system according to several embodiments. [Figure 5] This is a high-level block diagram of a receiver according to several embodiments. [Figure 6] An exemplary embodiment is shown, comprising multiple sensors included in a sensor array. [Figure 7] This plot shows signal attenuation as a function of rainfall rate. [Figure 8] This indicates that electromagnetic waves tend to diffract at sharp angles or propagate as "creeping" waves on curved surfaces. [Figure 9] Several embodiments of bistatic radar configurations are shown. [Figure 10] The radar cross-sections for a car and an average-sized person are shown. [Figure 11] The incident electric field, reflected electric field, and transmitted electric field relative to the wall are shown. [Figure 12A] The transmission coefficient T for three frequency bands is shown using the exemplary wall shown in Figure 11. [Figure 12B] The reflection coefficient Γ for three frequency bands is shown using the exemplary wall shown in Figure 11. [Figure 13] This indicates a rural intersection. [Figure 14] This demonstrates how electromagnetic waves can propagate through areas with dense trees. [Figure 15] For exemplary systems using sparse arrays according to several embodiments and two segmented array systems, the false detection percentage is compared as a function of the target angular position. [Figure 16A] The detection percentage is plotted as a function of the target distance in several embodiments. [Figure 16B] The detection percentage is shown as a function of the average distance to 20 random targets according to several embodiments. [Figure 17] The detection percentage is plotted as a function of the distribution of radar cross-sections of 20 randomly placed targets according to several embodiments. [Figure 18] Shows the results when the system co - processes distance and angle data according to some embodiments. [Figure 19A] Shows a simulated AD scene. [Figure 19B] Shows a simulated AD scene. [Figure 19C] Shows ray - tracing for the scenes shown in FIGS. 19A and 19B. [Figure 20A] Shows the magnitude of the channel impulse response for each of three radar bands. [Figure 20B] Shows the magnitude of the channel impulse response for each of three radar bands. [Figure 21A] Provides further visualization of simulation results according to some embodiments. [Figure 21B] Provides further visualization of simulation results according to some embodiments. [Figure 21C] Provides further visualization of simulation results according to some embodiments. [Figure 21D] Provides further visualization of simulation results according to some embodiments. ]> [Figure 21E] Provides further visualization of simulation results according to some embodiments. [Figure 21F] Provides further visualization of simulation results according to some embodiments. [Figure 21G] Provides further visualization of simulation results according to some embodiments. [Figure 21H] Provides further visualization of simulation results according to some embodiments. [Figure 21I] Provides further visualization of simulation results according to some embodiments. [Figure 22] Is a block diagram of an exemplary system according to some embodiments. [Figure 23] Shows another exemplary system according to some embodiments. [Figure 24] This shows a portion of a system including an exemplary sparse array according to several embodiments. [Figure 25] This shows a portion of a system including an exemplary optical array of a LiDAR subsystem according to several embodiments. [Figure 26A] This is a flowchart illustrating an exemplary method for identifying the location of a target in a given scenario, according to several embodiments. [Figure 26B] A flowchart of exemplary procedures that can be performed to make a decision, according to several embodiments, and the projected position of the target. [Modes for carrying out the invention]

[0007] For ease of understanding, the same reference numeral is used, where possible, to indicate the same element common to the figures. Some of the drawings show multiple examples of a particular element (e.g., signals, targets, transmitters, receivers, array elements, etc.). The convention used herein is to indicate a particular example of an element shown in the drawings by a reference numeral followed by a letter (e.g., A, B, C, etc.). When the specification usually refers to an element, only the reference numeral is used. Thus, as one example, the specification refers to a particular target 130A, 130B, etc., and the drawings show a particular target 130A, 130B, etc., and the specification also refers to simply one target 130 or more targets 130. An element disclosed in one embodiment is intended to be advantageously utilized in other embodiments without specific detail. Furthermore, a description of an element in the context of one drawing is applicable to other drawings showing that element.

[0008] Detailed explanation Multiple types of sensors can be used in AD systems. For example, cameras are a well-understood 20th-century two-dimensional sensor technology that is inexpensive and easily integrated in convenient locations inside or on a vehicle. Conventional cameras in stereo mode can be used to detect several objects and their respective speeds, but cameras have limitations in estimating distance and depth. Thus, the accuracy of cameras is often below what is required for a secure AD system, and cameras cannot detect NLOS targets. Furthermore, cameras do not function equally well at night, in fog, in direct sunlight, and under other conditions, and are also susceptible to optical illusions. Standalone cameras are insufficient for AD. As a result, other sensor technologies have been developed to increase and / or enhance the performance of cameras in detecting and / or tracking targets.

[0009] Two technologies capable of providing more accurate distance estimation and more accurate depth information are radio sensing rangefinding (radar) and optical sensing rangefinding (LiDAR). A radar system transmits electromagnetic waves (for example, at radio or microwave frequencies) and receives the reflection of the waves from a target. The position and velocity of the target can be determined from the reflection.

[0010] The performance of a radar system can be characterized by its distance resolution and angular resolution (in this specification, resolution represents how close two objects must be (in distance or angular separation) before they appear as one unified object, indistinguishable from each other). Distance resolution is the minimum distinguishable distance difference between two targets at different distances but in the same direction. Angular resolution is the minimum distance between two targets of the same size and at the same distance that can be distinguished. The distance resolution of a radar system depends on the bandwidth of the modulated waveform, and the angular resolution (both azimuth and elevation) depends on the physical aperture of the radar array itself. The accuracy of the radar, i.e., how accurately it can identify targets with respect to distance and angle, depends particularly on the received signal-to-noise ratio (SNR). Current AD systems using radar typically operate at 77 GHz and use linear frequency modulation (FM). These AD systems reportedly have sub-meter distance resolution and sub-degree angular resolution.

[0011] The radar system can operate quickly and over long distances, and does not require any mechanically moving parts. However, this can be inaccurate. For example, a radar system may miss images from smaller targets when displaying a very large target. Furthermore, high-frequency radar bands are negatively affected by bad weather (e.g., rain, fog) and other obstacles (e.g., dust), while low-frequency radar bands are less accurate and use antennas with larger apertures.

[0012] LiDAR systems work by illuminating a target area or scene with pulsed light and measuring how long it takes for the reflected pulse to return to a light detector. Many LiDAR systems use lasers to transmit the light pulses and measure the time of flight of the reflection from the target to the corresponding receiver (e.g., a photodiode). There are several variations of LiDAR, including spinning LiDAR and solid-state LiDAR. As the name suggests, spinning LiDAR systems have moving parts and physically scan the field of view. Solid-state LiDAR has no moving parts. Another class of LiDAR is flash LiDAR, which has a single high-power laser that illuminates the entire field of view of the target and a high-density array of detectors, each detector (pixel) corresponding to a specific azimuth and elevation angle. Flash LiDAR is similar to a digital camera, except that it can determine the time of flight corresponding to the image observed at a given pixel. Yet another class of LiDAR, called frequency-modulated continuous wave (FMCW) LiDAR, uses direct down-conversion of the optical signal.

[0013] LiDAR systems often outperform cameras in adverse weather conditions because they supply their own photons. Furthermore, LiDAR systems can offer finer resolution than other types of systems, including radar, thereby providing good effective range, accuracy, and resolution even for small targets. While LiDAR systems can generally detect and track LOS targets with a high probability of detection, this can be hindered by occlusion, and their accuracy can be reduced by adverse weather conditions. For LOS targets, LiDAR can achieve higher resolution than radar, but like cameras, the performance of a LiDAR system can deteriorate if its field of view (FOV) is affected by fog, rain, or bright sunlight. LiDAR systems typically have an effective range of 200-300 meters, but a greater effective range is often desirable for AD (Auditory Detection). Additionally, LiDAR systems can be fragile. For example, spinning LiDARs are prone to mechanical failure, especially if struck by foreign objects, and flash LiDARs rely on the reliable operation of a single high-power laser and highly precise optical components.

[0014] Depending on the frequency band used, radar may be less affected by weather conditions than cameras, but it typically has inferior distance and angular resolution, as well as lower accuracy, compared to LiDAR.

[0015] Therefore, with respect to AD applications, cameras, radar, and LiDAR have different functions and characteristics, and each has at least some drawbacks. The amount of information that can be obtained from each type of sensor is limited by its physical properties, which can create a dilemma for AD systems attempting to avoid accidents and / or fatalities. It is about filling the gaps in information and design between sensors that can complement each other.

[0016] Systems, apparatus, and methods that may be used for autonomous driving and / or in autonomous vehicles are disclosed herein. Several embodiments utilize an integrated large-aperture multiband radar subsystem and leverage the unique propagation characteristics of multiband and / or multiplexer technology to significantly improve landscape detection and understanding, particularly for viewing around corners and identifying NLOS targets. Several embodiments include or use a radar subsystem having a sparse array of transmit / receive elements (antennas). In some embodiments, the radar subsystem is capable of simultaneously transmitting and receiving multiple-band radar signals, in which case it may be referred to herein as a “sparse large-aperture multiband” (SWAM) radar. In some embodiments, at least one processor of the system is capable of jointly processing multiple-band return (reflection) signals to provide high accuracy under a variety of conditions (e.g., weather). The disclosed radar subsystems can be used alone or in conjunction with other sensing technologies, such as LiDAR and / or cameras.

[0017] The disclosed systems, apparatus, and methods can be used in AD systems to provide higher performance than alternative approaches. For example, LiDAR systems may fail to detect targets under certain conditions, including fog or rain. The automotive industry has addressed the fog problem by introducing 77GHz millimeter-wave radar, but this also suffers from significant loss problems due to rain. Furthermore, alternative systems may be plagued by a high probability of false positives. As a result, in the United States, AVs equipped only with LiDAR and 77GHz radar cannot consistently practice autonomy, except in desert states such as Arizona where fog and heavy rain are very rare. This limitation prevents fully autonomous driving using these types of systems in much of Europe, Asia, and other rainy or foggy regions. The disclosed systems, in particular several embodiments of the disclosed SWAM radar, can solve these problems.

[0018] Furthermore, from a safety perspective, such as the ability to detect moving and stationary targets around corners, embodiments of the disclosed system can utilize additional radar observations in multiple (e.g., low) frequency bands. The system may utilize the full width of the vehicle and sparse array to achieve higher resolution than conventional AV radar systems. The disclosed embodiments also offer advantages in crowded EM environments where multiple vehicles operate their radars simultaneously. The presence of multiple bands adds an additional dimension, for example, by further orthogonalizing the transmitted waveforms through frequency hopping or time sharing, which can reduce interference from other radars and thereby be beneficial for tailoring AD radar to urban traffic environments. Some embodiments are capable of providing an effective range of up to 360° under adverse conditions, thereby overcoming at least some of the shortcomings of conventional AD systems. In contrast to conventional radar systems or other approaches to which AD is considered, some embodiments can provide near 100% observability and improved safety.

[0019] In some embodiments, the radar subsystem augments and synchronizes with the LiDAR subsystem, camera subsystem, and / or other information sources (e.g., GPS, maps, etc.). Some embodiments access at least two information sources (e.g., radar, LiDAR, camera, GPS coordinates, etc.), and information from multiple sources is fused to improve performance. The combination or merging of scene information from different sensing technologies is referred to herein as “sensor fusion.”

[0020] Some embodiments of the disclosed systems, apparatus, and methods utilize multiple sensing techniques and sensor fusion to identify and track both LOS and NLOS targets. Some embodiments utilize SWAM radar along with advanced signal processing and probabilistic sensor fusion algorithms. In some embodiments, information obtained from different sensor types is combined to obtain a consistent understanding of the landscape. In some embodiments, the disclosed systems, apparatus, and methods utilize the unique propagation characteristics of electromagnetic (EM) waves in different frequency bands to improve landscape understanding under various weather conditions. In particular, in some embodiments, the disclosed systems can see through and around corners that are not possible with conventional systems, thereby improving the safety of autonomous driving.

[0021] The disclosed methods, systems, and apparatus can provide numerical advantages. For example, some embodiments can survey a crossroads from a distance and search for targets hidden by buildings as a vehicle approaches an urban intersection (it should be understood that, as used herein, the words “see” and “search” generally refer to detecting the presence of an object or target that may be hidden by, for example, an obstacle). Some embodiments have the ability to see around a corner as a vehicle approaches a rural intersection that is obscured by trees, wooded areas, or other vegetation. Some embodiments enable adaptive, intelligent tracking by being able to see much greater distances (e.g., 1 kilometer) compared to conventional systems (e.g., only about 300 meters at most). Some embodiments offer dynamic performance by being able to track a large number of targets (e.g., 100 or more targets) simultaneously at a high resolution rate (e.g., 50 frames per second (FPS)).

[0022] The methods, systems, and apparatus disclosed herein can provide high resolution and high-resolution / accuracy detection and tracking of numerous targets in highly complex scenarios encountered by AD systems. Furthermore, the use of multiband radar offers advantages in congested environments where multiple vehicle radar systems are operating simultaneously. Embodiments of the disclosed systems can "orthogonalize" their waveforms and transmissions for both interference reduction and distinguishing their return signals from the return signals of other vehicle radar systems. The presence of multiple bands adds an additional dimension to achieve such orthogonalization (e.g., by frequency hopping or time-sharing), which allows AD radar to be adapted to urban traffic environments.

[0023] The term “array element” may be used herein to refer to an antenna included in an antenna array. Array elements can be used for transmitting signals, receiving signals, or sending and receiving signals. A “transmitting element” is an array element capable of transmitting, and a “receiving element” is an array element capable of receiving. A single array element may also be capable of both transmitting and receiving, as further described below. The terms “antenna” and “antenna element” are used herein mostly interchangeably. An antenna is one example of a sensor, and some of the following descriptions refer to antennas and antenna elements, while others use the term “sensor.” However, as will be understood by those skilled in the art, the word “antenna” is often interchangeable with “sensor.”

[0024] Figure 1 shows System 100 in several embodiments. System 100, which may be an AD system, comprises at least one transmitter 105 (or, as further described below, a transmitter array 111) and at least one receiver 120 (or, as further described below, a sensor array 121). For simplicity, the description of Figure 1 refers to a single transmitter 105 and a single receiver 120, but it should be understood that System 100 can include a transmitter array 111 and multiple receivers 120, as will be discussed in more detail below. Similarly, Figure 1 shows a transmitter 105 and a receiver 120 in the same location, but the transmitter 105 and the receiver 120 do not have to be in the same location. In some embodiments described below, the transmitter 105 and the receiver 120 are located on the body of a vehicle, such as an automobile. The positions of the transmitting and receiving elements can be arbitrary. In other words, they can have any coordinates in three-dimensional space. The transmitting and receiving elements do not have to be a linear array or a planar array. In particular, when antenna elements are positioned on the vehicle body, they may be arranged in a curved or curved configuration.

[0025] Figure 1 shows four targets 130A, 130B, 130C, and 13D near system 100. Target 130A is at a distance of 150A from system 100, target 130B is at a distance of 150B from system 100, target 130 is at a distance of 150C from system 100, and target 130D is at a distance of 150D from system 100. According to some embodiments, one objective of system 100 is to estimate the distances 150A, 150B, 150C, and 150D. In some embodiments, another objective of system 100 is to estimate the angular position of target 130 (for example, where target 130A is on a circle with a radius equal to distance 150A, where target 130B is on a circle with a radius equal to distance 150B, etc.), where the angular position is called (or can be determined from) the angle of arrival.

[0026] During operation, the transmitter 105 transmits each waveform 140. The waveform 140 may be a baseband signal modulated to a carrier signal having a specific frequency and phase. For ease of explanation, the details of the modulation of the carrier signal in the transmitter 105 and the demodulation of the passband signal to the baseband or intermediate frequency in the receiver 120 are not described in detail herein. These techniques are common and well known in the art.

[0027] The transmitted waveform 140 propagates through a medium that causes attenuation and potentially distortion (e.g., free space, air, fog, rain, buildings, etc.) and is reflected from target 130. The reflected signals 148A (reflected at target 130A), 148B (reflected at target 130B), 148C (reflected at target 130C), and 148D (reflected at target 130D) propagate to return to receiver 120. The reflected signals 148A, 148B, 148C, and 148D are attenuated in the medium and arrive at receiver 120 at some point after transmission, where the time depends on the speed at which the signal propagates through the medium, as well as whether the reflected signals 148A, 148B, 148C, and / or 148D are direct returns from line-of-sight (LOS) target 130, returns through buildings, or multipath returns. The medium and / or receiver 120 may add noise to the reflected signals 148A, 148B, 148C, and 148D.

[0028] Figure 2 provides a conceptual diagram of the relationships between various signals transmitted and received by System 100 according to several embodiments. In the conceptual diagram of Figure 2, the transmitted waveform 140 is shown as a simple pulse. As shown in Figure 2, the received noisy waveform, referred to herein as the echo signal 142, is a superposition of reflected signals (e.g., reflected signals 148A, 148B, 148C, and 148D in Figure 1), and includes contributions from interference (e.g., multipath), noise, and other impurities. It should be understood that when the transmitted waveform 140 is modulated to the carrier signal for transmission, the received echo signal 142 has both amplitude and phase, and when converted to baseband, it has both in-phase (I) and perpendicular (Q) components. (Figure 2 does not show the phase of any contribution of the echo signal 142 or the reflected signals 148A, 148B, 148C, and 148D.) The reflected signals 148 (e.g., 148A, 148B, 148C, and 148D in Figure 2) may be constructively added at some distances between the target 130 and the system 100, and destructively added at other distances. In the high-level conceptual diagram of Figure 2, the echo signal 142 has three distinct attenuated and distorted reflections, which are obscured by noise. Each of these reflections corresponds to at least one target 130 at some distance from the system 100. One objective of the system 100 is to process the noisy echo signal 142 to identify the target 130 and their locations relative to the system 100. In some embodiments, the system 100 applies a sensor fusion procedure to identify the locations of the target 130.

[0029] For target 130 within the line of sight of system 100, since the signal travels at a known speed (for example, the speed of light if the medium is air), the distance 150 can be directly calculated from the time between when the transmitted waveform 140 is activated and when the echo signal 142 is received. The distance 150 between target 130 and system 100, which can be calculated from the measured propagation time, provides a circle centered on the position of system 100, on which target 130 lies.

[0030] According to some embodiments disclosed herein, the receiver 120 processes the echo signal 142 using an optimization procedure to obtain a denoised signal 144. The receiver 120 then uses the denoised signal 144 to estimate the distance 150 from the system 100 where the target 130 is located. The use of the denoised signal 144 to estimate the distance to the target 130 can substantially improve the performance of the system 100 compared to a conventional system (for example, by improving the SNR by 10-12 dB or more).

[0031] Figure 2 shows at a high level a denoising procedure used according to several embodiments. The receiver 120 can use the echo signal 142 to perform an optimization procedure, which is described in more detail below, in order to obtain a denoised signal 144. The optimization procedure takes advantage of the knowledge that an ideal echo signal 142 (noise-free) is a structured signal that is a superposition of a relatively small number of time-shifted and attenuated copies of the transmitted waveform 140, although the time shift and attenuation are unknown. The optimization removes noise from the signal such that the resulting denoised signal 144 looks like a linear superposition of several time-shifted and attenuated transmitted waveforms 140 and remains "close" to the received echo signal 142. The receiver 120 can then use the resulting denoised signal 144 to estimate the distance 150 from the system 100 where the target 130 resides. In some embodiments, to estimate the distance 150, the receiver 120 performs a correlation between the transmitted waveform 140 and the denoised signal 144, and then uses the peak position of the correlation result to estimate the distance 150. The correlation may be performed in the time domain or by performing an equivalent procedure in the frequency domain. Figure 2 provides a conceptual diagram of the post-correlated signal 146, which is shown to have peaks at times t1, t2, and t3. Using the speed at which the transmitted signal 140 and the reflected signal 148 propagate through the medium, the distance 150 to the LOS target can be estimated from the location of the peaks in the post-correlated signal 146.

[0032] As shown in Figure 2, the number of peaks in the post-correlation signal does not have to be the same as the number of targets 130. For example, if multiple targets 130 are approximately equidistant from system 100, their reflected signals 148 reach receiver 120 at approximately the same time. Referring again to Figure 1, targets 130A and 130B are at approximately the same distance from system 100 (for example, distance 150A is approximately the same as distance 150B). Therefore, their reflected signals 148A and 148B reach receiver 120 at approximately the same time. In Figure 2, reflected signals 148A and 148B correspond to the first "bump" (obscured by noise) of the echo signal 142. As this example shows, receiver 120 can identify from the post-correlation signal 146 that at least one target 130 corresponding to the peak at time t1 is at distance 150, but it does not have to be able to identify how many targets are at that distance 150 from the post-correlation signal 146 alone. In some embodiments, the system 100 includes a plurality of receiver sensors (e.g., an antenna array), each of which receives its own echo signal 142. In some such embodiments, in addition to (or instead of) using the echo signal 142 to estimate the distance 150 of the target 130 from the system 100, the receiver 120 processes the plurality of echo signals 142 to determine the angle of arrival of the target 130.

[0033] Figure 3 shows specific components of the transmitter 105 of system 100 according to several embodiments. The transmitter 105 comprises a waveform generator 110 and other components for transmitting a transmit waveform 140. The transmit waveform 140 may be any preferred waveform. For example, the transmit waveform 140 may comprise a pulse train, where each pulse has a specific duration, or each pulse has its own duration. In some embodiments, the transmit waveform 140 has good autocorrelation characteristics, meaning that the autocorrelation is small except near the origin. The better the autocorrelation characteristics, the better the performance of the correlation receiver, which will be discussed below. To search for targets within an area, the transmit waveform 140 may include one or more pulses with a first short duration and one or more pulses with a second longer duration to search for targets further away. Although the embodiments herein are shown in the context of pulse array processing, it should be understood that the disclosed principles are similarly applicable to continuous wave (CW) type systems (e.g., radar systems).

[0034] The transmitter 105 may modulate the transmitted waveform 140 into one or more carrier signals. For example, in the exemplary embodiment shown in Figure 3, the waveform generator 110 is coupled to the mixer 118. The mixer 118 modulates the transmitted waveform 140 into carrier signals.

[0035] The frequency of the carrier signal may always remain the same, or it may vary at different times. Similarly, the phase of the carrier signal may be constant, or it may change. The carrier signal may be generated by one or more oscillators 112, and its phase may be generated by one or more phase shifters 116. At least one processor 114 may control the oscillators 112 and / or the phase shifters 116.

[0036] In some embodiments, the system 100 is capable of transmitting in multiple frequency bands (for example, two or more of the L, S, X, C, K, ka, Ku, W, or UHF bands). In such embodiments, the same transmit waveform 140 may be transmitted in multiple frequency bands (for example, dissimilar, i.e., non-overlapping bands), or different transmit waveforms 140 may be transmitted in different frequency bands (for example, a first transmit waveform 140 may be transmitted in a first frequency band, and a second transmit waveform 140 may be transmitted in a second frequency band). The presence of multiple bands adds an additional dimension to further isolate the transmit waveforms 140 through, for example, frequency hopping or time sharing (a concept called "orthogonalization"), which can reduce interference from other radar systems that may be operating nearby and / or enable the system 100 to be used in urban traffic environments.

[0037] In some embodiments, the transmitting unit of system 100 comprises a transmitter array 111. Figure 4 shows specific components of the transmitter array 111 of system 100 according to some embodiments. The transmitter array 111 may be part of an antenna array, or may include an antenna array. The transmitter array 111 comprises waveform generators 110A, 110B, ..., 110x and other components for transmitting instances of transmit waveforms 140A, 140B, ..., 140x, where "x" represents the number of waveform generators 110 and waveforms 140. In some embodiments, at all moments in time, the transmit waveforms 140A, 140B, ..., 140x are identical to each other. Therefore, in these embodiments, when multiple waveform generators 110A, 110B, ..., 110x are active, they generate the same physical transmission waveform 140, but each waveform generator 110A, 110B, ..., 110x may generate the transmission waveform 140 independently of the other waveform generators 110. In other embodiments, at least some of the transmission waveforms 140A, 140B, ..., 140x are different. Therefore, in these embodiments, when multiple waveform generators 110A, 110B, ..., 110x are active, at least some of them generate different physical transmission waveforms 140.

[0038] The transmitted waveform 140 may be any suitable waveform. For example, the transmitted waveform 140 may comprise a pulse train, where each pulse has a specific duration, or each pulse has its own duration. As described above, in some embodiments, the transmitted waveform 140 has good autocorrelation characteristics.

[0039] The transmitter array 111 may modulate each instance of the transmit waveform 140 to one or more carrier signals. For example, in the exemplary embodiment shown in Figure 4, each of the waveform generators 110 shown is coupled to its respective mixer 118 (i.e., waveform generator 110A is coupled to mixer 118A, waveform generator 110B is coupled to mixer 118B, and so on). The mixer 118 modulates the instances of the transmit waveform 140 to carrier signals. At any given time, each carrier signal has the same frequency, but each instance of the transmit waveform 140 corresponding to a particular waveform generator 110 is modulated to a carrier signal that has a different phase from the carrier signal to which all other instances of the transmit waveform 140 are modulated. For example, as shown in Figure 4, an instance of the transmitted waveform 140A is modulated with a carrier signal having frequency "f_1" and phase "phase_1", and an instance of the transmitted waveform 140B is modulated with a carrier signal having the same frequency "f_1" but a different phase "phase_2", and so on.

[0040] The carrier frequency of each carrier signal may remain the same or may differ at different times. Similarly, the phase of each carrier signal may be constant or may vary. As shown in Figure 4, the carrier signals may be generated by one or more oscillators 112, and the various phases of the carrier signals of the waveform generator 110 may be generated by one or more phase shifters 116. At least one processor 114 may control the oscillators 112 and / or the phase shifters 116.

[0041] In some embodiments, the system 100 is capable of simultaneously transmitting in multiple frequency bands (for example, two or more of the L, S, X, C, K, Ka, Ku, W, or UHF bands). In such embodiments, the same transmit waveform 140 can be transmitted simultaneously in multiple frequency bands, or different transmit waveforms 140 can be transmitted in different frequency bands (for example, a first transmit waveform 140 may be transmitted in a first frequency band, and a second transmit waveform 140 may be transmitted in a second frequency band). In some embodiments, at any given time, each carrier signal in a particular frequency band in use has the same frequency, but each instance of the transmit waveform 140 corresponding to a particular waveform generator 110 transmitting in that band is modulated to a carrier signal having a different phase from the carrier signals modulated by all other instances of the transmit waveform 140 transmitted in that band by other waveform generators 110. As described above, it should be understood that different transmit waveforms 140 can be transmitted simultaneously in the same frequency band.

[0042] Without loss of generality, it can be assumed that the transmitter array 111 contains P-waveform generators 110. All P-waveforms of the waveform generators 110 of the transmitter array 111 can be transmitted simultaneously, but in some embodiments having the transmitter array 111, fewer P-waveform generators 110 than all of the transmitter array 111 transmit simultaneously. Furthermore, the number of waveform generators 110 may be more, less, or equal to the number of receiver sensors 122 (described later).

[0043] In some embodiments, the transmitter array 111 transmits (up to) P-dimensional sensing vectors. The sensing vectors are the result of each waveform generator 110 generating a transmit waveform 140 (e.g., pulses, pulse trains, etc.) modulated to a carrier signal having a specific carrier frequency, and are the same for all waveform generators 110 in the array. Preferably, the phases of the carrier signals transmitted by (up to) P waveform generators 110 are different from each other. In some embodiments, the phases are randomly selected. For example, they are completely random and may change over time, or they are completely random and may be permanently fixed. Alternatively, they may be deterministic and selected according to some algorithm or criterion, potentially in conjunction with at least one processor 114. The purpose of transmitting modulated carrier signals with different phases is to transmit energy in many directions at once. The different phases of the transmitted modulated carrier signals affect the amplitude and phase of the echo signal 142 received by the receiver sensor 122 (discussed below). In embodiments where system 100 includes multiple receivers (e.g., multiple receiver sensors 122), the difference in amplitude and phase of each received echo signal 142 may be used to determine the angle of arrival of the target 130.

[0044] In embodiments using random or multiple carrier signal phases, it should be understood that the randomness or variability of the carrier signal phase is spatially apparent. Each waveform generator 110 transmits the same average energy when the transmitted waveform 140 modulated by the carrier signal is the same.

[0045] In some embodiments, the system 100 comprises an array of P waveform generators 110, where different subsets of the P waveform generators 110 transmit at different times. In some embodiments, the active waveform generators 110 transmit the same amount of energy. The process of changing the subset of active waveform generators 110 over time creates a randomized antenna gain pattern suitable for exploring a three-dimensional environment.

[0046] Referring again to Figure 4, the transmitter array 111 includes at least a first waveform generator 110A and a second waveform generator 110B. The first waveform generator 110A generates a first instance (140A) of the transmit waveform 140 for a certain time interval, which is modulated into a first carrier signal having a specific carrier frequency and a first phase. The second waveform generator 110B generates a second instance (140B) of the transmit waveform 140 for the same time interval, substantially synchronized with the first waveform generator 110A, which is modulated into a second carrier signal having the same specific carrier frequency but a second phase different from the first phase. The first and second phases may be randomly selected, or they may be deterministic. For example, the first and second phases may be selected as a result of a randomization procedure (e.g., using a pseudo-random sequence generator to determine the randomized first and second phases). As another example, the first and second phases may be deterministic, meaning they are selected by an algorithm known to the first and second transmitters (for example, a deterministic sequence in which the first and second phases are selected and / or changed when the system is in operation, for example by selecting a predefined phase from a reference table).

[0047] The first and second waveform generators 110A and 110B may include, or be coupled to, at least one processor 114 configured to execute at least one machine-executable instruction. As a result of executing at least one machine-executable instruction, the at least one processor 114 may determine or select first and second phases (which may be implemented using oscillators 112 and phase shifters 116 as described above).

[0048] In some embodiments in which the first and second waveform generators 110A, 110B include or are coupled to at least one processor 114, the at least one processor 114 controls one or more characteristics of the signals transmitted by the first and second waveform generators 110A, 110B. For example, the at least one processor 114 can control the shape and / or timing and / or duration of the transmitted waveform 140 modulated to the carrier signal by each of the first and second waveform generators 110A, 110B. The at least one processor 114 may similarly or alternatively control several aspects of the carrier signal, such as carrier frequency, amplitude, and / or phase as described above, (either directly or in conjunction with other waveform generator 110 components well known to those skilled in the art).

[0049] System 100 may include other waveform generators 110 in addition to the first and second waveform generators 110A and 110B, as shown in Figure 4. In such embodiments, each further waveform generator 110x transmits each instance of the transmit waveform 140, modulated to each carrier signal having the same carrier frequency as the first and second carrier signals but with different respective phases, substantially synchronized with the first and second waveform generators 110A and 110B for the same time interval. Each waveform generator 110 transmits a modulated carrier signal having a phase different from the phase of all other modulated carrier signals. In some embodiments, each instance of the transmit waveform 140 is substantially identical to the first and second instances of the transmit waveform 140.

[0050] Referring again to Figure 1, the system 100 also includes at least one receiver 120. Figure 5 is a high-level block diagram of the receiver 120 according to several embodiments. The receiver 120 includes at least one sensor 122, at least one radio frequency (RF) / analog circuit 124 (for example, performing down-conversion), at least one analog-to-digital converter (ADC) 126, and at least one processor 128. It should be understood that the at least one receiver 120 may include components not shown in Figure 5. As just one example, the at least one receiver 120 may include memory, which may be coupled to at least one processor 128 and / or at least one ADC 126. It should also be understood that the at least one processor 128 may be exactly the same as the at least one processor 114 of the transmitter 105.

[0051] Each sensor 122 may, for example, be equipped with an antenna. In some embodiments, the sensor 122 is capable of receiving signals in at least two frequency bands (for example, two or more of the L, S, X, C, K, ka, Ku, W, and UHF bands). Such a sensor 122 may be capable of simultaneously receiving signals in more than one frequency band.

[0052] Each RF / analog circuit 124 may be any conventional RF / analog circuit 124, which is familiar and well known to those skilled in the art. In some embodiments, each RF / analog circuit 124 downconverts each received echo signal 142 to baseband. In such embodiments, at least one ADC 126 is configured to generate in-phase (I) and quadrature (Q) samples of the echo signal 142 for further processing by the receiver 120. In other embodiments, each RF / analog circuit 124 is configured to downconvert each echo signal 142 to an intermediate frequency, and the ADC is configured to sample the echo signal 142 while it is present at the intermediate frequency. Yet another embodiment does not include any RF / analog circuit 124, in which case the ADC directly samples the echo signal 142 without any downconversion.

[0053] The term “analog-to-digital converter (ADC)” is used broadly to mean any component that converts a continuous-time, continuous-amplitude (analog) received echo signal 142 into a discrete-time, discrete-amplitude (digital) signal (e.g., a sample). Such components are well known to those skilled in the art and are not further discussed herein.

[0054] Figure 6 shows an exemplary embodiment comprising a plurality of sensors 122 included in a sensor array 121 (for example, an antenna array). In the embodiment shown in Figure 6, a first sensor 122 labeled "sensor 1" is coupled to a first RF / analog circuit 124 labeled "RF / analog 1", the first RF / analog circuit 124 is coupled to a first ADC 126 labeled "ADC 1", and the first ADC 126 is coupled to at least one processor 128. Similarly, a second sensor 122 labeled "sensor 2" is coupled to a second RF / analog circuit 124 labeled "RF / analog 2", the second RF / analog circuit 124 is coupled to a second ADC 126 labeled "ADC 2", and the second ADC 126 is coupled to at least one processor 128. As shown in Figure 6, the receiver 120 may also include an additional sensor 122, an RF / analog circuit 124, and an ADC 126, the ADC 126 of which may be coupled to at least one processor 128. As described above, the RF / analog circuit 124 and ADC 126 are well known in the art. Similarly, as described above, some embodiments do not include the RF / analog circuit 124.

[0055] In embodiments including multiple sensors 122, as described above, the sensors 122 do not need to be in the same location (but can be). Furthermore, in embodiments including three or more sensors 122, the sensors 122 do not need to be collinear (but can be). Moreover, the sensors 122 do not need to be arranged in any regular manner or with any specific spacing between them. For example, unlike conventional systems, the distance between adjacent sensors 122 in the sensor array 121 does not need to be the same (but can be). As a result, the system 100 can be incorporated into vehicles with limited space and curved surfaces, as will be further described below.

[0056] Sensors 122 (for example, shown in Figures 5 and 6) may be located in the same place as the waveform generator 110 (for example, shown in Figure 4). In some embodiments, the waveform generator 110 uses sensors 122 to transmit signals. For example, the waveform generator 110 and the receiver 120 may share some or all of the sensors 122 of the sensor array 121 (for example, an antenna array). In embodiments where at least some of the sensors 122 are located in the same place as at least some of the waveform generator 110, at least one processor 128 (for example, shown in Figures 5 and 6) and at least one processor 114 (for example, shown in Figure 4) may be the same at least one processor. In other words, at least one processor 114 / 128 may be configured to coordinate and manage the transmit and receive operations of the system 100. At least one processor 114 / 128 may also perform additional functions, such as sensor fusion, which will be discussed further below.

[0057] In some embodiments, after the transmission of a transmit waveform 140 (e.g., a wide beam) by a waveform generator 110, there is a listening period during which at least one receiver 120 listens for an echo signal 142. As described above, in some embodiments, one or more waveform generators 110 share an antenna (e.g., one of the sensors 122) with at least one receiver 120. In some such embodiments, if fewer waveform generators 110 are transmitting than all of the waveform generators 110 combined, these shared antennas may be used by at least one receiver 120 to detect the echo signal 142 while other antennas are being used by one or more waveform generators 110. It should be understood that the number of antennas used by the waveform generators 110 may be the same as or different from the number of sensors 122 used by the receiver 120.

[0058] As described above, distance and accuracy (e.g., distance resolution and angular resolution) are common metrics used to characterize the performance of an AD system. In addition to distance and accuracy, other aspects of the AD system are of interest, such as the number of targets 130 that can be identified / detected and tracked (generally, a larger number is considered better), the size of the point clouds (each point cloud being a collection of points representing a three-dimensional shape or feature, from which distance, angle, and velocity information can be determined) that can be handled by the perception engine (e.g., at least one processor 114 and / or at least one processor 128), and the refresh rate (how many times per second the entire field of view is explored and the corresponding target 130 is identified).

[0059] As explained above, the basic principle of radar operation is the transmission of EM waves (e.g., transmitted waveform 140) and subsequent processing of the received scattered signal (e.g., reflected signal 148) for target detection. The accuracy of the detection procedure that processes the received scattered signal depends on the understanding of how the EM waves propagate and scatter. For AD applications, one objective of using radar instead of or in addition to LiDAR is to compensate for the shortcomings of LiDAR technology regarding the physical nature of propagation and to solve several complex detection problems. Some of the challenges involve scenes involving a large number of targets 130 at once, highly complex and dynamic environments, targets 130 positioned around corners (e.g., buildings), and targets 130 hidden by areas of dense vegetation and trees. The goal of the radar system is to meet all of these challenges under various weather conditions while providing an angular resolution close to that of a LiDAR system.

[0060] In the automotive industry, millimeter-wave radar systems in the 77-81 GHz range have recently been introduced to enhance LiDAR. One drawback of LiDAR systems and cameras is high signal loss in dense fog with poor visibility. For example, dense fog can cause signal loss of approximately 0.5 g / m³. 3The water density is approximately 0.05 g / m³, and visibility is less than 50 m, meaning that light intensity is attenuated by 10-15 dB in the range above 50 m. In moderate fog, the water density is approximately 0.05 g / m³. 3 This means that the field of view is less than 200m, and light intensity is attenuated by 10-15dB above 200m. At 77GHz (and low-frequency bands), signal loss in dense and moderate fog is only 1dB per kilometer. Therefore, in practice, millimeter-wave and low-frequency radars are unaffected by fog and can be used to compensate for the significant performance degradation of LiDAR caused by fog.

[0061] However, rain may actually be the challenge for radar systems. There is a lot of literature on the propagation of electromagnetic waves under various meteorological conditions, including rain. For example, propagation loss under heavy rain conditions (150 mm / h) can be measured and, furthermore, can be theoretically calculated for various frequency bands. Electromagnetic waves are attenuated by 0.01 dB / km, 0.08 dB / km, 1.0 dB / km, 2.5 dB / km, and 12 dB / km in the UHF, C, X, Ku, and Ka bands, respectively. Figure 7 is a plot showing signal attenuation (dB / km) as a function of rainfall rate (mm / hour) for a 1550 nm LiDAR, as well as radars at 77 GHz, 24 GHz, 5.8 GHz, and 1 GHz. At heavy rainfall of approximately 50 mm / hour, 10 mm / hour, and 2 mm / hour, respectively, the signal attenuation for LiDAR and radar operating at 77 GHz is substantially the same, at approximately 20 dB / km, 10 dB / km, and 5 dB / km. Therefore, radar systems operating at 77 GHz do not appear to offer any advantage over LiDAR systems in rain, and both systems are susceptible to the effects of both light and heavy rain. Lower frequency radar systems perform better. In particular, radar systems operating at 5.8 GHz and 1 GHz do not appear to be significantly affected by rain.

[0062] Despite being susceptible to performance degradation due to rain, radar systems operating in the 77-81 GHz frequency band, for example, can address some of the shortcomings of radar systems operating at lower frequencies, including lower distance and angular resolution. At these millimeter-wave frequencies, the EM waves emitted from the antenna array can have a narrow beamwidth and behave similarly to a laser beam. Thus, millimeter-wave radar systems can offer some of the advantages of LiDAR systems, but like LiDAR systems, they are limited to LOS detection (no choice around corners), as well as shadowing and occlusion effects.

[0063] Therefore, today, radar is used in several AD systems to enable operation in diverse environments and to detect a variety of stationary and moving targets at various distances. However, conventional radar systems used in AD applications do not detect high detail at long ranges and do not properly reference from the side or through obstacles. For example, high-frequency bands such as K and W can provide high resolution and can accurately estimate the location and speed of a target. However, these high-frequency radar signals do not penetrate building walls or can not see around corners, and as explained above, they are susceptible to rain, fog, and dust. Low-frequency bands such as UHF and C are less susceptible to these problems, but they use larger antenna elements and have a smaller available bandwidth, which reduces distance resolution (the ability to distinguish between two objects at similar directions but different distances). They also require a large physical aperture size to provide the angular resolution desired for AD applications.

[0064] Therefore, a single radar band does not appear to provide the desired accuracy, distance, and angular resolution, and to operate under all anticipated weather conditions, allowing for viewing around corners through buildings. The inventors anticipated that a system 100 operating simultaneously on multiple frequency bands would be a significant improvement over conventional systems. By using multiple different bands, the weaknesses of one band can be combined with the strengths of others.

[0065] The angular resolution of a radar system, which is the minimum distance at which two targets of the same size and distance can be identified, depends on the aperture of the radar system and is inversely proportional to the frequency at which it is used. The distance resolution, which is the minimum difference in distance at which two targets of the same direction but different distances can be identified, is inversely proportional to the bandwidth. Radar systems in the UHF (0.3-1GHz) or C-band (4-8GHz) that achieve the same angular resolution as radar systems in the K (18-27GHz) or W (75-110GHz) bands have considerably larger apertures.

[0066] An advantage of low-frequency radar systems (UHF and C-band) over millimeter-wave radar systems is that low-frequency EM waves have superior reflection, diffraction, and transmission properties, which can be important for NLOS detection. Figure 8 shows that EM waves tend to diffract at sharp angles or propagate as "creeping" waves on curved surfaces. These effects are weaker at higher frequencies (e.g., K and W bands), but they can be sufficient in the UHF and C-bands. In addition to EM wave diffraction, low transmission loss can enable radar systems operating in the UHF band to detect targets around corners.

[0067] High-frequency bands (e.g., 24 GHz and 77 GHz) can provide high angular and distance resolution, allowing radar systems to accurately estimate the location and velocity of targets in a scene, because more bandwidth is available at higher frequencies. However, as explained above, high-frequency bands generally do not penetrate building walls and are susceptible to certain obstacles (e.g., rain, fog, dust). Low-frequency bands are less affected by these obstacles, but they accommodate larger antenna elements. Furthermore, the available bandwidth may be smaller in low-frequency bands, which negatively impacts distance resolution and may require larger physical apertures to form phased arrays that provide high angular resolution.

[0068] Generally, radio frequency (RF) channels exist in rich multipath environments, and in AD environments, this can be highly congested due to the coexistence of multiple active automotive radar systems. Furthermore, rain, fog, and dust can limit radar operating distance, particularly at millimeter-wave frequencies. This extensive list of challenges led the inventors to the conclusion that no single radar band can provide high performance under all anticipated conditions. The inventors envisioned that by enabling system 100 to operate simultaneously across multiple frequency bands, as further described herein, it would be possible to view complex RF channels from multiple perspectives, potentially providing different modalities of potentially influencing information and improving the performance of AD (and potentially other) applications.

[0069] Radar performance for AD applications depends on the radar cross-section (RCS) of a typical target, path loss under various weather conditions, reflection / transmission loss due to diffraction coefficients of typical obstructing buildings / walls at sharp angles, the power required to accurately detect the target, and the aperture size for the desired angular resolution. Each of these factors / characteristics will be discussed further below.

[0070] The radar cross-section (RCS) is a measure of the ability of a target 130 to reflect a radar signal (e.g., a transmitted waveform 140) toward a radar receiver (e.g., receiver 120). In other words, RCS is a measure of the ratio of the backscatter density toward the radar (from target 130) to the power density blocked by target 130. Figure 9 shows a bistatic radar configuration according to several embodiments. In Figure 9, the transmitting antenna 106 is shown as being separated from the receiving antenna 107, but it should be understood that a single antenna may be used for both transmission and reception. A simplified form of the free-space radar equation relating the transmitted signal at transmitter 105 antenna port a(s) to the received signal at receiver 120 antenna port b(s) is given in the Laplace region:

number

number

number

number

number

number

number

number

number

number

[0071] Figure 10 shows the RCS (dBsm) calculated using the Physical Theory of Diffraction (PTD) and Unified Theory of Diffraction (UTD) high-frequency methods for two potential targets 130 that System 100 may encounter: a car and an average-sized person, in the K (left), C (center), and UHF (right) bands (note that the PTD and UTD methods may not be accurate in the UHF band for small objects). The RCS is calculated using full-wave analysis of the human body (solid line) and compared to the high-frequency method (dashed line). The RCS of the shown objects in the shown bands share some commonalities, but they are significantly different. For example, the UHF RCS of the front view of a car is considerably smaller than that of the C and K bands. This result suggests that an approaching car is more easily detected in the C and K bands, but also suggests that it may be more difficult to see ahead of the nearest car (e.g., detecting a target 130 behind the nearest car) while operating in the K or C band. Furthermore, the RCS for people in the UHF band appears considerably smoother than the RCS in the C and K bands. This result suggests that the UHF RCS will be more robust to human footsteps and position compared to the C and K bands, and from there, it can be inferred that detecting people using UHF radar will be easier.

[0072] When water is present on the surface of a scattering material, the target RCS decreases, which reduces radar reflectivity measured in the C and K bands, but this phenomenon does not particularly affect radar systems using the UHF band.

[0073] The safety of the AD system would be greatly improved if such a system could use NLOS propagation to detect targets 130 (e.g., cars, motorcycles, bicycles, pedestrians, animals, etc.) around corners or behind other targets 130. To evaluate the feasibility of a system 100 using NLOS propagation for AD, the reflectance / transmission coefficients of different objects (e.g., walls in the case of urban driving, and vegetation in the case of rural driving) can be evaluated. The transmission coefficient of an object is a measure of how much electromagnetic wave passes through the object, and the reflectance coefficient is a measure of how much the wave is reflected by impedance discontinuities in the transmitting medium.

[0074] Figure 11 shows the incident, reflected, and transmitted electric fields for a wall of thickness d 190, as an example of a method for calculating the reflection and transmission losses of a plane wave against a typical wall. In general cases, the reflection and transmission coefficients are determined by the polarization of the incident field and the angle of incidence θ. i It is a function of . In transverse electric (TE) mode, all transmitted fields are polarized in the vertical direction.

[0075] Figures 12A and 12B show the transmission coefficient T and reflection coefficient Γ for three frequency bands, respectively, using an exemplary wall 190 shown in Figure 11 with d set to 30 cm, and assuming that wall 190 is made of concrete. As shown in Figure 12A, in the UHF band, the transmission coefficient is approximately -6.5 dB over a wide range of incident angles. The transmission coefficients for the C and K bands at the plotted incident angles are not shown in Figure 12A because they are less than -35 dB and -150 dB, respectively, and therefore are considered negligible. In Figure 12B, the reflection coefficients for the C and K bands coincide. Thus, Figures 12A and 12B show that low-frequency band signals can be transmitted through the wall and pass through the building, while high frequencies cannot do so in any meaningful way.

[0076] In addition to reflection and transmission, diffraction caused by sharp angles also affects the propagation of EM waves in NLOS propagation. Based on the geometric theory of diffraction (GTD), the power of a signal diffracted by the edges of a metallic object (e.g., buildings, windows, billboards, etc.) is inversely proportional to frequency. Therefore, diffraction may not be an essential phenomenon of NLOS propagation in the high-frequency band, while it can affect low frequencies such as UHF.

[0077] Signal obstruction due to propagation loss in areas with dense trees and vegetation is important for AD applications. Various scenarios exist in rural (and urban) areas where obstruction by vegetation can increase the risk of false positives. Figure 13 shows a rural intersection. As shown, due to vegetation, vehicle 300A may not be able to see vehicle 300B when preventing an accident.

[0078] Commander's Critical Information Requirements (CCIR) Report 236-2, which discusses the influence of topography on propagation, characterizes leaf-induced loss as follows: L(dB) = 0.2f α R β Here, f is the frequency in megahertz, and R is the distance the wave extends through the leaf. The parameters α and β are functions of leaf density, plant type, humidity, and temperature. Typical values ​​for α and β are 0.3 and 0.6, respectively. The EM wave does not travel linearly in the layered medium, where the transmitter and receiver are in the loss layer due to exponential attenuation. It was shown that the EM wave can leave the loss layer, pass through the lossless layer, and then re-enter the loss layer to reach the receiving antenna. The radar signal follows the same rules for round-trip propagation in a wooded area, such as the path 305 shown in Figure 14.

[0079] The angular resolution of a radar system depends on its aperture size. In particular, for radar systems with transmit / receive elements (antennas) arranged in a linear array, the angular resolution is proportional to the wavelength and inversely proportional to the array's aperture (or physical length). Therefore, the shorter the wavelength and / or the larger the aperture, the better the angular resolution. This is one reason why millimeter waves have been proposed for AD. As explained above, angular resolution is the minimum distance at which two targets 130 of the same size and distance can be identified. In other words, angular resolution is the minimum distance at which the angles of arrival from two (or more) distinct targets 130 can be separated from each other. For example, consider a radar system with transmit / receive elements arranged in a linear array. The angular resolution (in radians) of such an arrangement is given as follows:

number

[0080] Today, a typical radar aperture is 15 cm, which at 77 GHz results in an angular resolution of approximately 1°, which is insufficient for fully autonomous vehicles to operate in crowded and / or complex situations and meet desired safety objectives. The typical angular resolution of LiDAR systems is often around 0.1° to 0.2°, which is up to an order of magnitude smaller than, for example, a radar operating at 77 GHz. It is desirable for radar systems to provide angular resolution on the order of that offered by LiDAR systems, so that such radar systems can avoid the significant performance loss that LiDAR systems suffer in certain scenarios (e.g., fog, rain, etc.). However, to achieve an angular resolution of 0.1° at 77 GHz, the aperture must be relatively large. For example, an aperture of more than 1 meter (e.g., 1.2 m) may be sufficient to obtain the desired angular resolution. Such an aperture can fit within the width of a vehicle.

[0081] In addition to measures such as angular and distance resolution, another measure of a radar system's performance is the maximum number of targets it can simultaneously detect.130 The term “range bin” is used herein to refer to objects that are inseparable from one another in terms of distance. These objects belong to the same range bin. As will be discussed further below, distance resolution can be approximated as c / 2W, where c is the speed of light and W is the bandwidth of the transmitted waveform. Thus, for example, if the bandwidth of the transmitted waveform is 100 MHz, the distance resolution is approximately 1.5 m, and the size of each range bin is also 1.5 m.

[0082] For a uniform linear array (ULA) having M transmit elements and N receive elements arranged at equal intervals (uniformly), the maximum number of targets 130 that can be simultaneously detected for any given range bin is M+N. AD environments can be very complex due to the many objects occupying the scene. Therefore, the number of targets 130 (or point clouds) that must be detectable is very large, possibly tens or hundreds per range bin. Thus, AD systems using ULAs may need to use a large number of transmit and receive elements to detect a sufficient number of targets 130 in the scene. Furthermore, the preference for large apertures further increases the number of transmit and receive antennas (elements) in the ULA. As an example, a radar with a 1.2m aperture operating at 24GHz would require 192 antennas for a ULA with (conventional) half-wavelength spacing. For most AD applications, this is an extremely large number of transceivers (transmit and receive elements). Moreover, such systems would be disadvantageous and expensive in terms of RF and digital hardware, power consumption, and weight. Furthermore, due to the geometric constraints of the automobile (for example, the curvature of its surface), it is usually not possible to position the ULA inside or on the automobile.

[0083] One alternative approach to improving the angular resolution of a radar system without employing ULA is to place multiple smaller aperture ULA radar units inside or on the vehicle body along a virtual line. This solution, referred to herein as a segmented array, has been considered by several AD system providers. While some such smaller aperture ULA units may theoretically perform equivalently to a single radar unit with the desired large aperture, the data collected by each radar unit would likely need to be processed collaboratively, which then requires all units to be perfectly synchronized with each other in both the RF and digital (sampled) domains. It also requires that the individual smaller aperture ULA units be located in precisely defined positions known to the processing block. Achieving perfect synchronization is particularly difficult, as it may require very careful calibrated clock and frequency synchronization across the entire large physical aperture so that the RF and digital chains for each transmit and receive element are perfectly synchronized.

[0084] Therefore, as will be further described below, some embodiments of system 100 use a sparse array of transmit and receive elements. For example, in some embodiments, the transmit and receive elements are placed only in a scattered subset of the locations occupied by the elements of a standard-sized ULA. A sparse array can behave as a non-sparse array having elements at locations given by distinct pairwise differences between the locations of each transmit element and each receive element. If each pairwise difference is appropriately designed to be unique, a sparse array can be made to behave as a ULA of the same aperture dimensions, but its number of array elements is the product of the number of receive and transmit elements of the sparse array. In other words, if each pairwise difference is unique, a sparse array having M transmit elements and N receive elements will behave as if it had M × N elements.

[0085] The use of a sparse array in system 100 increases the maximum number of targets 130 that can be simultaneously detected in each range bin from M+N to M×N, which is a significant increase. In the example of a 1.2m aperture radar operating in the K band (24 GHz is allocated to radar applications), a sparse array with 30 transmit elements and 30 receive elements can provide the same performance as if it had 900 elements, a performance far exceeding that of a half-wavelength-spaced ULA. Similarly, a sparse array with 12 transmit elements and 16 receive elements can behave as a ULA with 192 elements (far exceeding the performance of a half-wavelength-spaced ULA). The use of a sparse array makes it possible to embed the array in the body of a vehicle (e.g., an automobile) (e.g., distributed above, in the middle, and below the body), as further described below.

[0086] As explained above, the segmented array approach, which can be complex to implement, has been considered by several AD system manufacturers. Even if all (which can be expensive and / or cumbersome) full synchronization of the units in the segmented array approach can be achieved and the data is processed jointly, the resulting performance of the segmented array radar system will not match that of system 100 using a sparse array. To illustrate the difference, consider two large aperture radar systems, each 1.2m long, operating at 77GHz in the W bandwidth, with 12 transmitting elements and 16 receiving elements. The first system (system 100) uses a carefully designed sparse array, resulting in an equal distribution of 192 geometrically arranged virtual antenna elements. The second system uses two 14-element ULAs on either side of the 1.2m aperture. The latter configuration is used to mimic a segmented array combining two segmented individual 14-antenna ULA systems, which is an approach considered by several AD manufacturers. Although both systems have the same aperture and the same number of antenna elements, the integrated sparse array system (system 100) exhibits superior performance. Figure 15 compares the false detection percentage as a function of the target's angular position for the following three systems: system 100 with a sparse array, a segmented array system with full synchronization in the RF and digital domains (the result is a "synchronous ULA" curve), and a segmented array system in which two arrays operate independently and then their detected targets 130 are fused (the result is an "asynchronous ULA" plot).

[0087] As shown in Figure 15, system 100 with a sparse array exhibits near-zero false positives across all angles, while both versions of the segmented array system exhibit poor behavior at small angles representing the front of the array (as a result of the segmented array system's antenna elements on both sides of the array). The center of the array behaves almost as a "blind spot" for the segmented array, with false positives exceeding 45% in the synchronous case and nearly 60% in the asynchronous case. The "synchronous ULA" scenario is unrealistic (as complete simultaneity between two ULA subsets may not be possible) and therefore represents the best performance achievable by the segmented array system. Thus, the observed performance difference between system 100 using a sparse array and the segmented array approach may be entirely attributable to the careful placement of transmit and receive elements in the sparse array of system 100.

[0088] As another example demonstrating the advantages of the sparse array approach, consider two further large aperture radar systems, each 1.2 m long, both having 20 transmitting elements and 20 receiving elements. The first system is System 100 with a carefully designed sparse array, resulting in a unique pairwise distance of approximately 400 between the antenna elements. The second system has two 10-element ULAs on either side of a 1.2 m aperture, which mimics combining two divided individual 10-antenna ULA systems to form a divided array. The first and second systems have the same aperture and the same number of antenna elements.

[0089] Figures 16A and 16B show the results of simulations comparing the performance of the first system (system 100 with a sparse array) and the second system (a segmented array) described above. To provide a fair comparison, both systems used the same transmitted waveform, the same type of beamforming, and the same joint distance and angle estimation algorithm (further described below). For the second system, the 2ULA subset of the segmented array was fully synchronous (in both the RF and digital domains). Figure 16A plots the detection percentage, which is the percentage of accurately detected targets 130, as a function of the distance 150 of target 130 from the radar system (range), and Figure 16B shows the detection percentage as a function of the average distance of 20 random targets 130. As Figure 16A shows, the performance of both systems decreases as the distance increases, because the received signal (e.g., reflected signal 148) becomes weaker due to path loss. Nevertheless, the superiority of system 100 using a sparse array is evident from Figures 16A and 16B. Figure 16B shows that system 100 with a sparse array is capable of accurately detecting all targets 130, while the segmented array system detects less than 90 percent of the targets 130. As shown in Figure 16B, when there are more than 30 targets 130, the performance of the segmented array system (the second system) degrades considerably. This is because a segmented array with 20 transmit and receive antennas can detect at most 20 targets 130. In other words, the segmented system has a total of 40 antennas, i.e., M=20 transmit antennas and N=20 receive antennas (meaning there are two 10-antenna ULAs, each ULA having 10 transmit antennas and 10 receive antennas). Theoretically, the segmented system should be able to detect up to 40 targets, but in practice, the performance degradation begins sooner, namely at 30 targets in this case. Since all aspects of the two simulated systems were identical except for the array of transmit and receive elements, the superior performance of system 100 using a sparse array may be entirely attributable to the arrangement of transmit and receive elements in the sparse array, in contrast to the segmented array used in the second system.

[0090] For environments with many targets, the performance of system 100 with a sparse array and the second system (a segmented array) can be characterized by the percentage of targets 130 that are accurately detected (meaning that targets 130 are detected, estimated to be within a 50 cm radius from the true target, and both systems have a distance resolution of 75 cm). Figures 16A and 16B discussed above compare the detection percentage relative to the distance of targets 130 and the number of targets 130. Figure 17 plots the detection percentage as a function of the distribution of RCS for 20 randomly placed targets 130, allowing for a comparison of the radar cross-section distribution of targets 130. As expected, the performance of both systems improves as the RCS improves. Nevertheless, Figure 17 shows that the performance of system 100 with a sparse array is superior to that of the second system with a segmented array. In all cases, system 100 with a sparse array is considerably better than the system with a segmented array.

[0091] The performance of system 100 can be improved and / or maximized by jointly processing acquired data across the distance and angular domains. In conventional radar systems, the distance to target 130 is determined by identifying the peaks in the output of a correlation receiver (correlating the received signal (e.g., reflected signal 148) with respect to the transmitted pulse or linear FM waveform (e.g., transmitted waveform 140)). The peaks are then sampled and fed into a spatial array processing algorithm such as MUSIC or ESPRIT to determine the angle of arrival (AoA) of the target. Thus, distance and AoA are determined separately. Furthermore, distance resolution is inversely related to the bandwidth of the transmitted pulse. A good approximation for distance resolution is as follows:

number

[0092] The inventors had the prospect that multidimensional joint processing of distance and angular domains could lead to improvements in both the distance and angular resolution of a radar system. For example, consider a radar system with eight transmitting and eight receiving elements, operating at 24 GHz, with a bandwidth of 200 MHz and an aperture of 46.9 cm (corresponding to 75 wavelengths). Such a system has a distance resolution of 0.75 m and an angular resolution of 0.8° when the distance and angular domains are processed separately. Therefore, if the distance and angular domains are processed separately, targets 130 at intervals of less than 0.75 m would not be able to be analyzed by this system.

[0093] Figure 18 shows the results when system 100 jointly processes distance and angular data. Ten targets 130 are in the scene, with their true positions represented by solar-shaped symbols and their estimated positions (based on radar data) represented by smooth oval shapes. The two leftmost targets 130 are at distances of 5.0m and 5.4m, respectively (therefore they are separated by a distance resolution smaller than 0.75m), and angles of 45.0° and 45.5°, respectively (therefore they are separated by an angular resolution smaller than 0.8°). As shown in Figure 18, system 100 can identify and analyze the two targets 130 by jointly processing distance and angular data.

[0094] As explained above, even if the performance of a segmented array system is sufficient for AD applications, i.e., the deployment environment inside / on a vehicle presents challenges not only due to the limited physical space but also due to the shape of the vehicle (e.g., curvature). The sparse array approach disclosed above overcomes both of these challenges. In particular, the radar elements of system 100 (e.g., transmit / receive antennas, possibly transmitting / receiving at different frequencies) can be embedded in various locations on / inside the vehicle body (e.g., roof, hood, fenders, front, rear, sides, around the windshield, inside the windshield, etc.). The resulting array is probably not linear, but probably a two-dimensional curve or a three-dimensional curved surface. As mentioned above, the antenna element arrangement must be such that there are multiple distinct pairwise differences in the x, y, and z coordinates between the antenna elements (in other words, the distribution of array elements is non-uniform). The greater the pairwise differences (x, y, and z coordinates), the better the performance will be. Since the sparse array positioned in this manner is two-dimensional or three-dimensional, the system 100 (for example, at least one processor 114 and / or at least one processor 128) would be able to estimate the elevation angle of the target 130 in addition to the azimuth angle. The distance, azimuth angle, and elevation angle of the target 130 can be jointly estimated, but many AoA estimation methods such as ESPRIT do not work because the array is linear rather than uniform. Instead, whenever the transmit and receive elements are in any position in space, atomic norm minimization can be used to jointly estimate the distance, azimuth angle, and elevation angle of the target 130.

[0095] In atomic norm frameworks used according to several embodiments, there is a set of atoms.

number

Number

[0096] The atomic norm of the vector x is defined as follows with respect to the set of atoms

Number

Number

[0097] The processors of system 100 (for example, at least one processor 114 and / or at least one processor 128) can use an atomic norm denoiser (tuned to the corresponding transmitted waveform 140), followed by a correlation receiver. As another example, at least one processor 114 and / or at least one processor 128 can use an atomic norm denoiser, followed by any sinusoid-in-noise estimator such as MUSIC, ESPRIT, Hankel norm approximation, Prony, Burg, or others. As yet another example, at least one processor 114 and / or at least one processor 128 can use an atomic norm denoiser that involves searching a finite set of atoms. A system and method for using the atomic norm to determine the distance, angle of descent, and velocity of a target 130 is described in detail in U.S. Patent No. 10,866,304, which is incorporated herein by reference in its entirety for all purposes.

[0098] In addition to embedding / positioning radar elements in the vehicle body, system 100 may also include LiDAR elements (e.g., light sources (e.g., lasers) and photodetectors (e.g., photodiodes)), which can also be embedded in the vehicle body or positioned on the vehicle body. Various types of LiDAR elements, such as solid-state LiDAR, flash LiDAR, and single-photon avalanche detector (SPAD) elements, can be positioned as described herein, i.e., they can be positioned at any location on the vehicle or embedded within the vehicle.

[0099] As mentioned above, one desirable characteristic of an AD system would be its ability to "see around corners." Electromagnetic waves tend to diffract at sharp angles, and when they encounter a curved surface, they can diffract around that surface as "creeping" waves (see Figure 8). As explained above, these effects are weak at higher frequencies (e.g., K and W bands), but they can be sufficient in the UHF and C (4-8 GHz) bands. The inventors had the foresight that this diffraction behavior, along with lower transmission loss in the low-frequency band, could be utilized to enable radar systems to detect objects around corners.

[0100] Radar signals traverse many paths, bouncing off many objects, and their paths to and from objects are tracked. In AD applications, radar returns (e.g., reflected signals 148) (referred to as multipath or multipath signals) resulting from these multiple paths may be affected by radar signals from other nearby automotive radar systems. Nevertheless, these multipath signals can provide information about a target 130 that is not in the line of sight of system 100 to the processors of system 100 (e.g., at least one processor 114 and / or at least one processor 128), for example, to reveal intersecting traffic that is hidden from direct detection. In other words, these multipath signals can be used for the detection of NLOS target 130.

[0101] A simulation of the tri-band version of System 100 can be used to demonstrate the ability to "see around corners." Consider an autonomous vehicle (AV) equipped with System 100, a tri-band radar (e.g., UHF, C, and K bands) with sparse large aperture transmit / receive arrays for each band, operating simultaneously on multiple bands. Assume the AV is approaching a city intersection surrounded by four tall concrete buildings, one at each corner of the intersection. For simplicity, assume the streets run north-south and east-west. Figures 19A and 19B show a scene including one person on the northeast side of the intersection and three vehicles approaching the intersection, with one vehicle 300A approaching from north to south, one vehicle 300B approaching from south to north, and one vehicle 300C approaching from west to east. The AV is moving from east to west and is not visible in Figure 19A, but its position is shown in Figure 19B. In the scenarios shown in Figures 19A and 19B, AV is 35 meters from the center of the intersection, and car 300C is approaching the intersection from the south. One car 300C is 40 meters from the center of the intersection and is not within AV's line of sight. Therefore, it cannot be detected by system 100 using the LOS technique.

[0102] In each of the three frequency bands, the radar system can estimate the distance and direction (angle of arrival) of target 130. For LOS target 130, the distance of target 130 (its distance from system 100) is equal to the speed of light multiplied by half the time it takes for the transmitted EM wave (e.g., transmitted waveform 140) to return to system 100. In other words, the distance corresponds to half the time of flight (ToF) of the transmitted EM wave returning to system 100. The direction of target 130 is determined from the angle of incidence of the wavefront received by system 100 (e.g., reflected signal 148). In other words, AoA is simply the angle of the waveform (e.g., reflected signal 148) received from a particular target 130. Furthermore, when target 130 is in LOS of system 100, this is simply the angle at which target 130 is positioned.

[0103] The situation is more complex when target 130 is not within the line of sight of system 100, as the signal (e.g., reflected signal 148) returns along multiple paths. When target 130 is NLOS, the distance cannot be interpreted as anything other than representing half of the ToF, as the EM wave (e.g., reflected signal 148) is received through multiple reflections or scattering, and therefore cannot be seen as the distance of target 130 from the radar. Furthermore, the AoA of NLOS target 130 is not necessarily related to the angle of target 130, but rather to the angle of the last return (through reflection or scattering) of the multipath to system 100. When interpreting information from each of the bands of system 100, the above considerations can be taken into account along with any conventional knowledge about the landscape (e.g., from cameras, LiDAR, maps, GPS, etc.) to infer, rather than directly measure, the location of target 130.

[0104] In some embodiments, returns (e.g., reflected signals 148) are classified as either line-of-sight (LOS), multipath, or through buildings (or NLOS) returns. For a given distance, multipath returns are typically weaker (due to multiple reflections) and have a different polarization than LOS returns. Returns through buildings are also weaker. In some embodiments, information about the environment (e.g., the location and materials of buildings and other stationary objects around the AD system) also contributes. A framework can be constructed to find possible locations of the true target. That framework can then be used to estimate the probability that the target is in each of several candidate locations.

[0105] As an example of how returns can be classified, a return may be classified as an LOS return if it is observed in multiple frequency bands (e.g., the majority of the frequency bands) with substantially similar flight times and angles of arrival, has a return intensity that exceeds a threshold, and / or has an angle and distance position that changes slowly over time (calculated assuming the target is LOS). A return may be classified as a multipath return if it is observed primarily or entirely in low frequency bands with substantially similar flight times and angles of arrival, has a return intensity that falls below a threshold, and / or has an angle and distance position that changes rapidly and irregularly over time (calculated assuming the target is LOS). Polarization information can also be considered to identify multipath returns. A return may be classified as a return through a building if it is observed primarily or entirely in low frequency bands with substantially similar flight times and angles of arrival, has a return intensity that falls below a threshold, and / or has an angle and distance position that changes slowly over time (calculated assuming the target is LOS). Information about the landscape (for example, the location of buildings from a map or LiDAR) can also be used to identify returns through buildings.

[0106] As the AV and target 130 move and system 100 collects additional data (for example, through reflected signal 148), the additional data can be used to update the probability that target 130 is in a candidate location. Each time system 100 updates the estimated probability, it narrows the area of ​​possible locations until the reliability of target 130's location is selected to be high enough and “ghost” target locations disappear. The performance of system 100 can be significantly enhanced by fusing information obtained from multiple bands.

[0107] The disclosed system 100 also offers advantages in congested EM environments where multiple radar systems of autonomous vehicles are operating simultaneously. The presence of multiple bandwidths adds an additional dimension (a concept called "orthogonalization") that further separates the transmitted waveforms (e.g., transmitted waveform 140) through frequency hopping or time sharing, for example, which allows for reduced interference from other radar systems and enables adjustment of system 100 for urban traffic environments and use in urban traffic environments.

[0108] Figure 19C shows ray tracing for the scenarios in Figures 19A and 19B at a point in time after the time shown in Figures 19A and 19B. The round-trip distances of some of the principal rays that originate from and return to System 100 are marked on each ray. For example, the round-trip distances to cars 300B ("V1") and 300A ("V2") are both within the LOS of System 100, and are 94.8m and 51.6m, respectively (as will be understood by those skilled in the art, the round-trip distance is a proxy for the round trip). The round-trip distances to pedestrians 302A ("P1") and 302B ("P2") are both within the LOS of System 100, and are 56.4m and 81.6m, respectively.

[0109] Figure 20A shows the magnitude of the channel's impulse response in dB for each of the three bandwidths (the signal is received at the receiving element when the impulse was transmitted at the transmitting element). Figure 20B is an enlarged version of the plot in Figure 20A, providing a clearer identification of the peaks in the impulse response. The plot is given as a function of round-trip distance, highlighting the distance the EM wave travels before returning to the receiver, as opposed to time. Round-trip distance can be easily converted to time by dividing them by the speed of light. The impulse response can be thought of as the output of an ideal correlated receiver applied to the received signal for the transmitted pulse of a radar system.

[0110] As can be seen from Figure 20A, the impulse responses at 5.8 GHz and 24 GHz are mostly silent after 160 m, whereas the impulse response at 900 MHz is not. The main reason is that 900 MHz allows for stronger reflections from walls and buildings. Therefore, the peaks observed after 160 m round trip correspond to such reflections. Since most of these reflections are absorbed and the signal is attenuated, they are not present at higher frequencies.

[0111] Referring to Figure 20B, for car 300B ("V1"), there is a peak at approximately 52m-53m in all three bands. For car 300A ("V2"), there are peaks at approximately 95m and 96m in all three bands. This indicates that both cars 300A and 300B are visible in all three bands. Pedestrians 302A ("P1") and 302B ("P2") are even more visible in the 900MHz signal, with distinct peaks at 56m and 82m, respectively. The corresponding peaks at 5.8GHz and 24GHz are less noticeable, presumably because the human RCS is much more robust to the orientation and gait of pedestrians at lower frequencies.

[0112] It should be noted that System 100 can also detect a vehicle 300C ("V3") located "around the corner" and blocked by a building. The impulse responses at 900 MHz and 5.8 GHz have a distinct peak at 66 MHz. This corresponds to an EM wave that penetrates the building, reflects off V3, and returns after a second penetration of the building. It should be noted that this peak is absent in the 24 GHz impulse response, as EM waves do not penetrate buildings at this high frequency.

[0113] Further examination of the ray tracing in Figure 19C reveals a second path through which vehicle 300C ("V3") can be observed. This corresponds to an EM wave that penetrates the building in the lower right corner, reflects to the left from vehicle 300C, and then reflects from the building in the lower left. Unlike the paths to and from vehicle 300C, which involve two building penetrations and one reflection, this path involves one building penetration and two reflections. The round-trip time for this path is 88.8m, as shown in Figure 19C. The impulse response at 900MHz has a distinct peak at 88m–89m. In the higher bandwidths of 5.8GHz and 24GHz, there is a slightly less distinct peak at approximately 87m–88m, which may also be attributable to this path.

[0114] There are several ways to perform sensor fusion (for example, to combine information obtained from different frequency bands). For example, for each frequency band, the probability that target 130 is at a candidate location (for example, as described above and further in the discussion of Figures 26A and 26B below) can be calculated for some or all of the return signals of that band (e.g., reflected signal 148, echo signal 142, etc.). This procedure yields a list of candidate locations for target 130, each associated with a probability. Then, for example, a "heatmap" can be created for each frequency band by placing a two-dimensional Gaussian distribution around each candidate location, where the mean of the Gaussian distribution is the candidate location of target 130, the covariance matrix is ​​determined by the accuracy of the estimated distance and angle of target 130, and the Gaussian intensity is determined by the probability that target 130 is located at the candidate location. Then, information from multiple bands can be fused, for example, by pointwise multiplication of the individual heatmaps from different bands to obtain a fused heatmap (assuming each frequency band provides an independent measurement of the landscape). As time passes and more measurements are collected, the fused heatmap evolves and becomes more accurate. As a result, ghost targets can be eliminated (for example, by eliminating candidate locations below a threshold probability), and the true location of target 130 can be determined.

[0115] Figures 21A–21I are exemplary heatmaps from the simulation, illustrating the advantages of fused information obtained using two radar bands (e.g., sensor fusion) and the ability of system 100 to view around corners. Vehicles equipped with exemplary system 100 are indicated by the right-hand circle in each figure. Figures 21A–21C show target 130 detected using the UHF band (915 GHz), and Figures 21D–21F show the results for the C band (5.8 GHz) (corresponding figures for the K band are not included, nor are figures for target 130 detected with a signal in the 24 GHz band, as the K band does not reveal objects around corners under the simulation conditions). The intensity of the detected target 130 is indicated by the scale on the right-hand side of each figure.

[0116] Figures 21A and 21D show target 130 detected by system 100 at time zero using signals transmitted simultaneously in the UHF and C bands (e.g., transmitted waveform 140 and reflected signal 148), respectively. As can be seen in Figures 21A and 21D, the four building corners are detected by system 100 in both the C and UHF bands. The buildings above and below AV have (and are visible) strong reflections in the UHF band, but not in the C band. In other words, faint returns are detected directly above and below AV, and these faint returns are visible only at UHF band frequencies, not C band frequencies. As shown in Figures 21A and 21D, car 300 approaching the intersection from the south (from AV's line of sight, car 300 approaches the intersection from the left) is not within system 100's LOS. Furthermore, at time zero, car 300 is not detected as NLOS target 130 by system 100. In other words, when the approaching vehicle, car 300, is 40m from the center of the intersection, it cannot be "seen" by system 100 at time zero.

[0117] The simulation shows that the exemplary system 100 can detect an approaching vehicle, starting at approximately 0.45 seconds, when the vehicle 300 is approximately 33 m from the center of the intersection. Figures 21B and 21E show the target 130 detected by system 100 at 0.6 seconds, and Figures 21C and 21F show the detected target 130 at 0.65 seconds. In both cases, the vehicle 300 is approximately 30 m from the center of the intersection. At both of these times, the vehicle 300 can be "seen" in both the UHF and C bands due to multiple reflections and EM waves penetrating buildings in the southeast quadrant of the intersection.

[0118] Note that at all the moments shown, the building southeast of the intersection (to the left of AV and system 100, and to the right of car 300) completely obstructs system 100 from directly seeing the approaching car 300, making it the NLOS target 130. The exemplary system 100 detects the approaching car 300 because system 100 is capable of detecting objects around the corner.

[0119] Finally, due to noise, clutter, etc., there are many “ghost targets” detected by system 100 in each band at each of the times shown in Figures 21A to 21F. The true target 130 can be distinguished from the ghost targets by fusing results from the UHF and C bands. Sensor fusing algorithms can be used to combine data from multiple bands (e.g., from a subset of bands or from all of the bands used by system 100). In some embodiments, system 100 (e.g., at least one processor 114 and / or at least one processor 128) fusing information from a diverse set of multimodal sources of information, including multiband radar (e.g., one or more of its bands), LiDAR, cameras, GPS coordinates, roadmaps, etc. For example, to determine the radar return distance and angle for each target 130, knowledge of the surrounding landscape (e.g., the location of plants, buildings, etc.) can be used to determine which signal returns (e.g., reflected signals 148) are LOS, which are NLOS, and which are the result of reflection, transmission, or both.

[0120] There are several approaches that can be used to perform sensor fusion. For example, a Bayesian network can be used to perform optimal detection and estimation based on prior knowledge and current measurements. As will be obvious to those skilled in the art, a Bayesian network (also known as a decision network) is a probabilistic graphical model that predicts the likelihood that any one of several possible known causes was the cause of an event by representing a set of variables and their conditional dependencies through a directed acyclic graph. Another option is to use a deep network (also known as a deep neural network), which is data-driven (e.g., uses very large datasets) and leverages past observations without the need to define features, models, or hypotheses presented for the problem. As will be obvious to those skilled in the art, a deep network can be thought of as a stacked neural network with many hidden layers (e.g., more hidden layers than one or two in a conventional neural network) and with weights, biases, nonlinear activations, and / or backpropagation. Furthermore, since the scene being reconstructed is dynamically changing, techniques such as particle filtering and scented Kalman filtering can be used. Figures 21G, 21H, and 21I show the results after system 100 has fused data from the UHF and C bands at time instances 0.00 second, 0.60 second, and 0.65 second (for example, Figure 21G follows the fusion of data corresponding to Figure 21A and data corresponding to Figure 21D; Figure 21H follows the fusion of data corresponding to Figure 21B and data corresponding to Figure 21E; and Figure 21I follows the fusion of data corresponding to Figure 21C and data corresponding to Figure 21F). As can be seen in Figures 21G, 21H, and 21I respectively, as a result of the fusion procedure, the only remaining significant targets 130 are the corners of buildings around the intersection (particularly those northwest and southwest of the intersection), as well as the cars 300 approaching the intersection from the south, located around the corners. Thus, Figures 21G-21I demonstrate that the sensor fusion process clears ghost targets from the scene.

[0121] Figures 21A-21I illustrate sensor fusion using two radar bands, but it should be understood that the disclosed technique can be used to fuse information from different sensor types. For example, the technique can be used to fuse information from radar subsystem 170 with information from LiDAR subsystem 320. Furthermore, information from more than two frequency bands can be fused. For example, system 100 may fuse information from more than two radar bands. As another example, system 100 may fuse information from two or more radar bands used by radar subsystem 170 with information from bands used by LiDAR subsystem 320.

[0122] The simulation results shown and described above demonstrate that the disclosed embodiment of system 100 can identify and locate targets 130, such as vehicles, that are obstructed by other objects, such as buildings. The simulation also demonstrates the value of system 100's ability to operate simultaneously across multiple radar bands. Low-frequency bands provide information unavailable in high-frequency bands because low-frequency EM waves penetrate buildings, reflect more effectively, and behave rather like light waves (and provide complementary information, unlike the information provided by LiDAR), generally have better propagation characteristics, and allow for longer-distance searches. Low-frequency bands are also more robust to weather conditions and the orientation of targets 130. High-frequency bands, on the other hand, provide even higher angular resolution and allow for the observation of finer details that may be missed at low frequencies. The use of multiple bands and the fusion of the information they provide allows system 100 to gain a precise understanding of the scene, enabling safe operation of autonomous vehicles.

[0123] System 100 may include various combinations of subsystems to provide the desired functionality. For example, for AD applications, System 100 may include, for example, (a) a multiband radar subsystem only, (b) a multiband radar subsystem and a LiDAR subsystem, or (c) a multiband radar subsystem, a LiDAR subsystem, and a camera subsystem. In any configuration, System 100 may also consider other information about stationary objects in the scene, such as buildings, fire hydrants, signs, and guardrails (e.g., their location, material properties, etc.). System 100 may also consider information about temporary hazards, such as construction areas, depressions, and scaffolding.

[0124] Figure 22 is a block diagram of an exemplary system 100 according to several embodiments. The exemplary system 100 comprises a sparse array 160 coupled to at least one processor 114. The sparse array 160 comprises a plurality of array elements 102. In Figure 22, the array elements 102 are labeled 1 to N, but it should be understood that there may be any number of at least two array elements 102. Each of the array elements 102 may be a transmit element, a receive element, or a combined transmit / receive element. For example, each of the array elements 102 may be a radar transmitter, a LiDAR transmitter, a radar receiver, a LiDAR receiver, a camera, etc. The sparse array 160 includes at least one transmit element and at least one receive element. The number of transmit elements may be the same as the number of receive elements, or there may be more or fewer transmit elements than receive elements.

[0125] The sparse array 160 may be distributed on the body of a vehicle, such as an automobile, as described above. The array elements 102 of system 100 (for example, transmit and receive antennas that transmit / receive at different frequencies, depending on the case) can be embedded in various locations on / within the body of the automobile (e.g., roof, hood, fenders, front, rear, sides, around the windshield, inside the windshield, etc.). The resulting sparse array 160 is likely to be a two-dimensional curve or a three-dimensional surface. As described above, the arrangement of the array elements 102 must be such that there are multiple distinct pairwise differences in the x, y, and z coordinates between the array elements 102. The greater the distinct pairwise differences, the better the performance of system 100 may be. Since the sparse array 160 is two-dimensional or three-dimensional, system 100 (for example, at least one processor 114) will be able to estimate the elevation and azimuth angles of the target 130. System 100 can jointly estimate the distance, azimuth, and elevation of target 130, for example, by using atomic norm minimization.

[0126] Figure 23 shows another exemplary system 100 according to several embodiments. The exemplary system 100 comprises a radar subsystem 170 and a LiDAR subsystem 320, both of which are coupled to at least one processor 114. The exemplary system 100 may include other components not shown in Figure 23. For example, the exemplary system 100 may include memory for storing data collected by the radar subsystem 170 and / or the LiDAR subsystem 320.

[0127] The radar subsystem 170 comprises a plurality of transmitting and receiving elements, including at least one radar transmitter 172 and at least one radar receiver 176, which may be arranged in a sparse array 160 (for example, on / inside the vehicle body). The number of radar transmitters 172 may be the same as the number of radar receivers 176, or there may be more or fewer radar transmitters 172 than radar receivers 176. Each radar transmitter 172 may comprise some or all of the components of transmitter 105, for example, shown and described in Figures 1-6. Each radar receiver 176 may comprise some or all of the components of receiver 120, for example, shown and described in Figures 1-6. The radar antenna 174 may be shared by the radar receivers 176, or the radar receivers 176 may have separate antennas (for example, as described in the discussion of Figure 9). The radar subsystem 170 may include components not shown in Figure 23, such as those shown in Figures 3 and 6 and described in their discussion. It may also include other components not described herein.

[0128] The LiDAR subsystem 320 comprises a plurality of transmitting and receiving elements, including at least one light source 322 and at least one photodetector 324, which may be arranged in an array (for example, on the vehicle body). At least one light source 322 may be, for example, a laser. Each of the at least one photodetector 324 may be, for example, a photon detector (for example, a photodiode such as an avalanche photodiode). The number of light sources 322 may be the same as the number of photodetectors 324, or there may be more or fewer light sources 322 than photodetectors 324. The LiDAR subsystem 320 may include components not shown in Figure 23, such as the components shown and described in U.S. Patent No. 11,047,982, which is incorporated by reference in its entirety for all purposes in this application. The LiDAR subsystem 320 may use a light source 322 and a photodetector 324 having a field of view that overlaps with a sparse pulse sequence with low cross-correlation, as described in detail in U.S. Patent No. 11,047,982, to enable the LiDAR subsystem 320 to detect a target 130 in the scene. Other components not described herein or in U.S. Patent No. 11,047,982 may also be included.

[0129] Therefore, in some embodiments, the system 100 for detecting a target in a scene comprises a radar subsystem 170, a LiDAR subsystem 320, and at least one processor 114 coupled to the radar subsystem 170 and the LiDAR subsystem 320. The LiDAR subsystem 320 comprises a light source configured to emit an optical signal and an optical detector configured to detect the reflection of the emitted optical signal. The radar subsystem 170 comprises a radar transmitter configured to transmit a radar signal and a radar receiver configured to detect the reflection of the transmitted radar signal. The at least one processor 114 is configured to execute at least one machine-executable instruction that, when executed, causes the at least one processor 114 to acquire a representation of the reflection of the optical signal emitted from the LiDAR subsystem 320, acquire a representation of the reflection of the radar signal transmitted from the radar subsystem 170 (e.g., a reflected signal 148), and determine the location of at least one target 130 in a scene, at least in part, based on the representation of the reflection of the emitted optical signal and the representation of the reflection of the transmitted radar signal.

[0130] The radar subsystem 170 may include at least one radio frequency signal generator and at least one antenna, as described above. In some embodiments, the radar subsystem 170 includes a sparse array 160 distributed on the body of the vehicle, the sparse array 160 comprising a plurality of radar transmitters 172 and a plurality of radar receivers 176. The sparse array 160 may be three-dimensional and / or non-uniformly distributed, as described herein.

[0131] In some embodiments, the radar subsystem 170 can operate simultaneously in at least two radar bands, such as two or more of the L, S, W, X, C, K, ka, Ku, or UHF bands. For example, the radar subsystem 170 may operate simultaneously in the W or X band and in the UHF band.

[0132] In some embodiments, the LiDAR subsystem 320 comprises an optical array having multiple optical components, the multiple optical components including multiple light sources 322 and multiple photodetectors 324. In some embodiments, at least two of the multiple optical components are non-collinear.

[0133] Figure 24 shows a portion of System 100, including an exemplary sparse array 160 according to several embodiments. The exemplary sparse array 160 comprises a plurality of radar transmitters 172 and a plurality of radar receivers 176. Figure 24 shows at least three radar transmitters 172 of the exemplary sparse array 160, namely radar transmitter 172A, radar transmitter 172B, and radar transmitter 172N. In some embodiments, the plurality of radar transmitters 172 of the sparse array 160 are capable of transmitting radar signals in multiple bands (e.g., at least two of L, S, X, C, K, Ka, Ku, W, UHF, etc.) as described above, in order to run a multiband radar system. The use of reference numbers 172A, 172B, and 172N for radar transmitters 172 in Figure 24 is for convenience and should be understood as not suggesting that the sparse array 160 necessarily includes exactly 14 radar transmitters 172. As described above, the sparse array 160 can have any number of radar transmitters 172. When the sparse array 160 is used in a system 100 that uses multiple radar bands simultaneously, the number of radar transmitters 172 is greater than one.

[0134] The exemplary sparse array 160 shown in Figure 24 also includes a plurality of radar receivers 176. Figure 24 shows at least three radar receivers 176 of the exemplary sparse array 160, namely radar receiver 176A, radar receiver 176B, and radar receiver 176M. In some embodiments, the plurality of radar receivers 176 of the sparse array 160 are capable of detecting radar signals in multiple bands (e.g., two or more of L, S, X, C, K, Ka, Ku, W, UHF, etc.) as described above, in order to run a multiband radar system. The use of reference numbers 176A, 176B, and 176M for radar receivers 176 in Figure 24 is for convenience and should be understood as not suggesting that the sparse array 160 does not necessarily include exactly 13 radar receivers 176. As described above, the sparse array 160 can have any number of radar receivers 176. When the sparse array 160 is used in a system 100 that uses multiple radar bands simultaneously, the number of radar receivers 176 is greater than 1.

[0135] Therefore, in some embodiments, the system 100 comprises at least one processor 114 and a sparse array 160 to emit a probing signal to detect reflected signals in the scene. The sparse array 160 may be distributed on the vehicle body. The sparse array 160 comprises a plurality of array elements, each capable of transmitting and / or receiving signals. Between the plurality of array elements are at least one element capable of transmitting signals (e.g., transmitter 105) and at least one element capable of receiving signals (e.g., receiver 120). In some embodiments, the first array element of the sparse array 160 is located at a first location having a first set of three-dimensional coordinates (X1, Y1, Z1), and the second array element of the sparse array 160 is located at a second location having a second set of three-dimensional coordinates (X2, Y2, Z2), where X1≠X2 and / or Y1≠Y2 and / or Z1≠Z2. In some embodiments, X1 ≠ X2, Y1 ≠ Y2, and Z1 ≠ Z2. In some embodiments, the first and second locations may be the roof, hood, front, bumper, fender, rear, trunk, left side, right side, windshield, or any other location on the vehicle.

[0136] In some embodiments, the sparse array 160 is three-dimensional. In some embodiments, the sparse array 160 is non-uniformly distributed, meaning that at least some of the pairwise distances between the nearest neighbor elements of the sparse array 160 are different from one another. In some embodiments, the pairwise distances between the nearest neighbor elements of the sparse array 160 are unique.

[0137] In some embodiments, the sparse array 160 is capable of simultaneously transmitting and receiving on multiple radar bands (e.g., two or more of L, S, W, X, C, K, ka, Ku, UHF, etc.). In some embodiments, the sparse array 160 comprises a first radar transmitter 172A and a first radar receiver 176A. In such embodiments, the sparse array 160 may similarly include a second radar transmitter 172B and a second radar receiver 176B, thereby configuring the radar transmitter 172A to transmit on a first radar band (e.g., K or C) and the radar receiver 176A to receive on a first radar band, and the radar transmitter 172B to transmit on a second radar band different from the first radar band (e.g., UHF) and the radar receiver 176B to receive on a second radar band.

[0138] At least one processor 114 is configured to execute at least one machine-executable instruction, when executed, to cause the at least one processor 114 to collect a plurality of reflected signals (e.g., reflected signal 148) detected by the sparse array 160, and to estimate the position (e.g., distance and / or angle (e.g., azimuth or elevation)) of at least one target 130 in the scene based at least partially on the plurality of reflected signals. In some embodiments, the at least one processor 114 jointly estimates the distance, azimuth, and elevation to estimate the position of at least one target 130. The joint estimation may include calculating or minimizing the atomic norm. In some embodiments, the at least one processor 114 partially estimates the position of at least one target 130 by denoising at least a portion of the plurality of reflected signals (e.g., by determining or minimizing the atomic norm), performing correlation of at least a portion of the denoised plurality of reflected signals, and identifying at least one peak in the correlation results. In some embodiments, at least one processor 114 partially estimates the location of at least one target in the scene by performing correlation and identifying at least one peak in the results of the correlation.

[0139] In some embodiments, system 100 may also include a LiDAR subsystem 320 having at least a first light source 322A and a first photodetector 324B. The first light source 322A may be configured to emit a first pulse sequence during a first time window. The first pulse sequence may be sparse. In some embodiments, the LiDAR subsystem 320 also includes a second light source 322B and a second first photodetector 324B. The second light source 322B may be configured to emit a second pulse sequence simultaneously with 322A emitting the first pulse sequence (for example, during a first time window). The second pulse sequence may be sparse. In some embodiments, the second pulse sequence is different from the first pulse sequence. The first and second pulse sequences may not be substantially correlated with each other (for example, they have low cross-correlation). The first and second pulse sequences may be substantially white.

[0140] Figure 25 shows a portion of System 100, including an exemplary optical array 164 of a LiDAR subsystem 320 according to several embodiments. The exemplary optical array 164 comprises a plurality of light sources 322 and a plurality of photodetectors 324. As described in U.S. Patent No. 11,047,982, it should be understood that the optical array 164 can provide high-precision target detection using a variety of numbers of light sources 322 and photodetectors 324 (for example, just one light source 322 or just one photodetector 324), as long as certain conditions regarding the total number of elements of the optical array 164 and their positions relative to each other are met. Figure 25 shows at least three light sources 322 of the optical array 164, namely light source 322A, light source 322B, and light source 322P. In some embodiments, multiple light sources 322 of the optical array 164 are capable of emitting optical signals having multiple wavelengths and / or using different pulse sequences, as described in U.S. Patent No. 11,047,982. In some embodiments, the optical array 164 assists in the operation of multiplexed-input (MIMO) LiDAR, as described in U.S. Patent No. 11,047,982. The use of reference numbers 322A, 322B, and 322P in Figure 25 is for convenience only and should be understood as not suggesting that the optical array 164 does not necessarily contain exactly 16 light sources 322. The optical array 164 may have any number of light sources 322 greater than 0, as described in U.S. Patent No. 11,047,982.

[0141] The optical array 164 shown in Figure 25 also includes a plurality of photodetectors 324. Figure 25 shows at least three photodetectors 324 of the optical array 164, namely photodetector 324A, photodetector 324B, and photodetector 324Q. In some embodiments, the plurality of photodetectors 324 of the optical array 164 can detect light at multiple wavelengths and according to heterogeneous pulse sequences (which may not be substantially correlated with each other, for example), as described in U.S. Patent No. 11,047,982. The use of reference numbers 324A, 324B, and 324Q in Figure 25 is for convenience only and should be understood as not suggesting that the optical array 164 necessarily includes exactly 17 photodetectors 324. The optical array 164 may have any number of photodetectors 324 greater than zero, as described in U.S. Patent No. 11,047,982.

[0142] Figure 26A is a flowchart of an exemplary method 200 for identifying the location of a target 130 in a given scene, according to several embodiments. The exemplary method 200 may be performed, for example, by one or more embodiments of the system 100 described herein. In block 202, a plurality of scanning signals (e.g., transmitted waveforms 140) are transmitted (e.g., by the radar subsystem 170). The scanning signals may be radar signals transmitted in separate bands of different frequencies (e.g., L, S, X, C, K, Ka, Ku, W, UHF, etc.), as described above, to enable the system 100 implementing the exemplary method 200 to look around a corner and provide high-precision detection of the target 130. For example, the first scanning signal may be transmitted in the C or K band, and the second scanning signal may be transmitted in the UHF band. The scanning signals may be transmitted multiple times or over multiple time intervals. Similarly, multiple reflected signals may be collected (e.g., received) multiple times or over multiple time intervals.

[0143] In block 204, multiple reflected signals (e.g., reflected signal 148) are collected (e.g., received by radar subsystem 170). In block 206, the classification of each of the smallest subsets of the multiple reflected signals is determined. The classification may result in each of the subsets of reflected signals being considered, for example, LOS returns, multipath returns, or returns through a building. Thus, in some embodiments, as a result of the classification, each reflected signal in at least one subset is classified as an LOS return, a multipath return, or a return through a building.

[0144] In block 208, the projection position of the target 130 in the scene (e.g., range / distance and angle (e.g., elevation and / or azimuth)) is determined based on at least one subset of multiple reflected signals and the classification of each signal in at least one subset of multiple reflected signals (e.g., LOS, multipath, through buildings). Optionally, information 210 about one or more stationary objects in the scene (e.g., the location, position, and / or orientation of buildings, utility poles, fire hydrants, and other fixed (permanent or temporary) obstacles, information about the materials used in the stationary objects, or characteristics of the stationary objects) may be considered in block 208 to determine the projection position of the target 130 in the scene. Method 200 can be used to detect multiple targets 130 in the scene.

[0145] Figure 26B is a flowchart of exemplary procedures that can be performed to make a decision in Figure 26A, 208, according to several embodiments. In block 212, a probability is determined for each of several candidate positions of target 130. Each probability corresponding to each candidate position is the likelihood that target 130 is in that candidate position. In other words, the result of block 212 is a set of candidate positions, each associated with a probability representing the likelihood that target 130 is in that candidate position. In block 214, the highest probability is identified. In block 216, the projected position of target 130 is selected as the candidate position corresponding to the highest probability identified in block 214. In other words, after determining the probability of target 130 in each candidate position, the candidate position with the highest probability is selected as the projected position of target 130. The probabilities may be updated based on reflected signals collected at different times or at different time intervals to describe the movement of the apparatus or system implementing Method 200, and / or changes in the position, location, or orientation of the target 130, other objects, and obstacles (for example, with respect to the apparatus or system implementing Method).

[0146] System 100 can implement exemplary method 200. System 100 may be integrated into a vehicle, for example, an automobile. In some embodiments, System 100 configured to perform exemplary method 200 comprises a radar subsystem 170 configured to transmit a plurality of scan signals and collect a plurality of reflected signals (for example, blocks 202 and 204 of exemplary method 200), and at least one processor 114 coupled to the radar subsystem 170 and configured to execute one or more machine-executable instructions that, when executed, cause at least one processor 114 to perform blocks 206 and 208 of exemplary method 200. The radar subsystem 170 may be capable of transmitting a first subset of the plurality of scan signals in a first radar band (e.g., a low-frequency band) during a first time period, and transmitting a second subset of the plurality of scan signals in a second (different) radar band (e.g., a high-frequency band) during a second time period. In some embodiments, the radar subsystem 170 comprises a plurality of transmitting elements (e.g., antennas, transmitters 105, etc.) and a plurality of receiving elements (e.g., antennas, receivers 120) located on and / or within the body of a vehicle (e.g., an automobile). In some embodiments, the radar subsystem 170 comprises a sparse array 160 comprising the plurality of transmitting elements and the plurality of receiving elements. In some embodiments, the sparse array 160 is three-dimensional. In some embodiments, the sparse array 160 is non-uniformly distributed, meaning that at least some of the pairwise distances between the nearest neighbor elements of the sparse array 160 are different from one another. In some embodiments, the pairwise distances between the nearest neighbor elements of the sparse array 160 are unique.

[0147] In some embodiments, a system 100 implementing exemplary method 200 also includes a LiDAR subsystem 320 configured to transmit multiple optical signals and collect a second set of multiple reflected signals, and at least one processor 114 determines the projection position of a target based on the second set of multiple reflected signals. In other words, in some embodiments, at least one processor 114 performs sensor fusion using both radar return signals and LiDAR return signals. The system 100 may be integrated into a vehicle, for example, an automobile.

[0148] In embodiments described herein, where the transmitter has multiple transmitter elements or the receiver has multiple receiver sensors, it should be understood that, in contrast to conventional antenna arrays, the spacing between adjacent elements or sensors (e.g., antenna elements) of the array (whether transmitting or receiving) does not have to be the same, but can be the same. In conventional systems, even slight changes in the distance between antenna elements result in undesirable sidelobes, which can cause many potential problems, such as false positives and vulnerability to the presence of strong interference signals (e.g., jammer signals). In contrast to conventional antenna arrays, where antenna elements are spaced half a wavelength apart from each other to minimize sidelobes (where the wavelength is the peak of the transmit / receive signal used to detect the target), in embodiments disclosed herein, the elements (transmitter elements or receiver sensors (e.g., antennas)) do not need to be at any specific distance from each other. Thus, the spacing may be regular (e.g., half a wavelength as in conventional systems), but instead, the sensors may be placed at any convenient interval. When transmitter elements (e.g., antennas) are not half a wavelength apart from each other, the systems and methods disclosed herein can utilize sidelobe energy as part of a broad transmit pulse in a wide spatial sense, in contrast to the narrow beam from the transmit beam shaper of a conventional radar system.

[0149] Similarly, in embodiments including more than one transmitter element or more than one receiver sensor, the transmitter elements and receiver sensors of the system disclosed herein do not need to be in the same location, but they may be. For example, some transmitter elements may be located in a first location, and other transmitter elements may be located in a second location. As another example, some receiver sensors may be located in a first location, and other receiver sensors may be located in a second location. For example, in embodiments where the receiver sensor array is mounted on a vehicle, some elements or sensors of the array may be mounted, for example, on the front bumper of the vehicle, others on, for example, the roof of the vehicle, and still others on, for example, the rear bumper of the vehicle. Receiver sensors may be located wherever is convenient. Similarly, transmitter elements may be located wherever is convenient.

[0150] Each of at least one processor 114 and at least one processor 128 may, for example, include a processing unit and a memory for storing program code executed by the processing unit to perform various methods and techniques of the above embodiments and to further constitute data or other information for performing various programmed or configuration settings according to the above embodiments. Note that the processing unit itself may be implemented as a general-purpose or dedicated processor (or a set of processing cores) and therefore may execute a sequence of programmed instructions to perform various operations of, for example, the radar subsystem 170 and the LiDAR subsystem 320 and other components of the system 100, which are associated with the control and / or synchronization of the operation of the components. Each of at least one processor 114 and at least one processor 128 may be implemented as a standalone processor (e.g., a digital signal processor (DSP)), a controller, a CPU, or custom-ordered hardware (e.g., an application-specific integrated circuit (ASIC)), or may be programmed on a programmable hardware device such as a field-programmable gate array, or any combination thereof inside or outside the system 100.

[0151] The techniques and / or user interfaces for configuring and managing them disclosed herein may be implemented by the machine execution of one or more sequence instructions (including relevant data necessary for the execution of appropriate instructions). Such instructions may be recorded on one or more non-temporary computer-readable media for later retrieval and execution within one or more processors of a special-purpose or general-purpose computer system or a home appliance or appliance, such as System 100 (for example, implemented in a vehicle such as an automobile). Computer-readable media on which such instructions and data may be embodied include, but are not limited to, non-volatile storage media such as fixed or removable magnetic, optical, or semiconductor-based recording media for storing executable code and relevant data, and volatile storage media such as static or dynamic RAM for storing more transient information and other variable data.

[0152] In the foregoing description and accompanying drawings, certain terms have been explained to provide a complete understanding of the embodiments disclosed. In some examples, terms or drawings may imply certain details that are not necessary to carry out the invention.

[0153] It should be understood that the use of Cartesian coordinate systems to illustrate various aspects of this disclosure is for convenience only and is not intended to limit the scope. Other coordinate systems may be used.

[0154] Although this disclosure is presented primarily in the context of autonomous driving, it should be understood that any system that uses reflected signals to identify a target, its distance from a sensor, its angle of arrival (abbreviated as AoA, and more commonly referred to in the art as direction of arrival (DOA)), its velocity (e.g., direction and / or velocity of movement from Doppler shift), and / or its composition of a material, can benefit from the disclosure herein, as will be obvious to those skilled in the art.

[0155] To avoid unnecessarily obscuring this disclosure, well-known components are shown in block diagrams and / or are not discussed in detail, or in some cases, not at all.

[0156] Unless otherwise defined herein, all terms are given the broadest possible interpretation, including the meaning implied by the specification and drawings, as well as the meaning understood by those skilled in the art and / or defined in dictionaries, papers, etc. As explicitly stated herein, some terms may not conform to their usual or customary meanings.

[0157] As used herein, the terms “target” and “objective” are interchangeable unless otherwise indicated by the context.

[0158] As used herein and in the appended claims, the singular forms “a,” “an,” and “the” do not exclude multiple referents unless otherwise specified. The word “or” should be interpreted as inclusive unless otherwise specified. Thus, the phrase “A or B” should be interpreted as meaning all of the following: “both A and B,” “A but not B,” and “B but not A.” The use of either “and / or” herein does not imply that the word “or” alone implies exclusivity.

[0159] As used herein and in the appended claims, the phrases “at least one of A, B, and C,” “at least one of A, B, or C,” “one or more of A, B, or C,” and “one or more of A, B, and C” are interchangeable and each encompasses all the meanings of “A only,” “B only,” “C only,” “A and B but not C,” “A and C but not B,” “B and C but not A,” and “all of A, B, and C.”

[0160] To the extent that the terms “including,” “having,” “possessing,” and their variations are used in the detailed description or claims, such terms are intended to be comprehensive in a manner similar to the term “equipped with,” i.e., “including, but not limited to, the following.”

[0161] The terms “example” and “embodiment” are used to describe examples, not as prerogatives or requirements.

[0162] The term “joined” is used herein to describe direct connection / attachment, as well as connection / attachment through one or more intervening elements or structures.

[0163] The terms “above,” “below,” “between,” and “above” are used herein to refer to the relative position of one feature to another. For example, one feature positioned “above” or “below” another feature may be in direct contact with the other feature or may have intervening material. Furthermore, one feature positioned “between” two features may be in direct contact with both features or may have one or more intervening features or materials. In contrast, a first feature “above” a second feature is in contact with the second feature.

[0164] The term "substantially" is used to describe a structure, configuration, dimensions, etc., that is roughly or nearly as described, but in practice, due to manufacturing tolerances, etc., the structure, configuration, dimensions, etc., may not always or necessarily be exactly as described. For example, describing two lengths as "substantially equal" means that the two lengths are the same for all practical purposes, but they may not be (and do not need to be) exactly equal on a sufficiently small scale (for example, if the unit of measurement is meters, two features having lengths of 1.000m and 1.001m are substantially equal in length). As another example, a structure that is "substantially vertical" is considered to be practically vertical even if it is not exactly 90 degrees to the horizontal.

[0165] The drawings are not necessarily to scale, and the dimensions, shapes, and sizes of features may differ substantially from how they are depicted in the drawings.

[0166] While specific embodiments have been disclosed, it will be apparent that various modifications and alterations may be made to them without departing from the broader spirit and scope of this disclosure. For example, any feature or aspect of an embodiment may be applied in combination with other embodiments, or in place of equivalent features or aspects, at least where feasible. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method (200) for identifying the location of a target in a scene, Transmitting multiple scanning signals (202), Collecting multiple reflected signals (204), Determining a classification for each signal of at least one subset of the plurality of reflected signals (206), wherein the classification includes returns through the building, (a) determining the projection position of the target in the scene based at least in part on the classification of each signal of the at least one subset of the plurality of reflected signals (208) including, method.

2. The method according to claim 1, wherein for each signal in the subset of the plurality of reflected signals, the classification is one of line-of-sight return, multipath return, or return through a building.

3. The method according to claim 1, wherein determining the classification of each signal in the at least one subset of the plurality of reflected signals includes determining, for a particular reflected signal in the at least one subset of the plurality of reflected signals: (a) the frequency of the band in which another reflected signal having substantially similar flight time and / or angle of arrival to the particular reflected signal is observed; (b) the number of bands in which the other reflected signal having substantially similar flight time and / or angle of arrival to the particular reflected signal is observed; (c) whether the intensity of the particular reflected signal exceeds a threshold; (d) the rate of change over time of the angle of arrival and / or flight time of the particular reflected signal; or (e) the polarization of the particular reflected signal.

4. The method according to claim 1, wherein determining the projection position of the target is further based on information (210) about at least one stationary object in the scene.

5. The method according to claim 4, wherein the information relating to the at least one stationary object in the scene includes at least one of the location of the at least one stationary object, the position of the at least one stationary object, the orientation of the at least one stationary object, or the material of the at least one stationary object.

6. The method according to claim 4, wherein the at least one stationary object includes a building.

7. The method according to claim 1, wherein the scanning signal is a radar signal.

8. The method according to claim 1, wherein collecting the plurality of reflected signals includes receiving the plurality of reflected signals.

9. The method according to claim 1, wherein the projection position represents distance and angle.

10. Determining the projection position of the target is Determining a plurality of probabilities for a plurality of candidate positions of the target (212), wherein each of the plurality of probabilities corresponds to one of the plurality of candidate positions, and each of the plurality of probabilities represents the possibility that the target is located at each of the candidate positions. Identifying the largest of the aforementioned multiple probabilities (214), The method according to claim 1, further comprising selecting one of the plurality of candidate positions corresponding to the largest of the plurality of probabilities as the projection position of the target (216).

11. The at least one subset of the plurality of reflected signals includes a first subset of reflected signals and a second subset of reflected signals, wherein each signal in the first subset of reflected signals corresponds to one or more scan signals transmitted in a first time or during a first time interval. The method according to claim 10, further comprising updating the plurality of probabilities based at least in part on the second subset of reflected signals, wherein each signal in the second subset of reflected signals corresponds to one or more scan signals transmitted in a second time or during a second time interval.

12. The method according to claim 1, wherein the first reflected signal of the plurality of reflected signals comprises a reflection of a first radar signal transmitted in a first frequency band, and the second reflected signal of the plurality of reflected signals comprises a reflection of a second radar signal transmitted in a second frequency band, and the first and second frequency bands are different.

13. The method according to claim 12, wherein the first frequency band is the L, S, W, X, C, K, ka, Ku, or UHF band.

14. The method according to claim 12, wherein the first frequency band is the C or K band, and the second frequency band is the UHF band.

15. A system (100) configured to perform the method described in any one of claims 1 to 14, A radar subsystem (170) configured to transmit the plurality of scanning signals and collect the plurality of reflected signals, When coupled to and executed in the radar subsystem, at least one processor (114) is configured The classification of each signal in at least one subset of the plurality of reflected signals is determined. (a) at least one processor (114) configured to execute at least one machine-executable instruction causing the projection position of the target to be determined based at least partially on the classification of each signal of the at least one subset of the plurality of reflected signals and (b) at least one machine-executable instruction to determine the projection position of the target. A system that includes these features.

16. The system according to claim 15, wherein the radar subsystem is capable of transmitting a first subset of the plurality of scan signals in a first radar band during a first time period, and transmitting a second subset of the plurality of scan signals in a second radar band during the first time period, and the first radar band is different from the second radar band.

17. The system according to claim 16, wherein the radar subsystem comprises a plurality of transmitters and a plurality of receivers located on and / or inside the vehicle body.

18. The system according to claim 15, wherein the radar subsystem comprises a sparse array (160), and the sparse array comprises a plurality of transmitting elements and a plurality of receiving elements.

19. The system according to claim 18, wherein the plurality of transmitting elements and the plurality of receiving elements are distributed on and / or inside the vehicle body.

20. The first array element of the sparse array is located at a first location having a first set of three-dimensional coordinates (X1, Y1, Z1), The second array element of the sparse array is located at a second location having a second set of three-dimensional coordinates (X2, Y2, Z2), The system according to claim 18, wherein X1 ≈ X2 and / or Y1 ≈ Y2 and / or Z1 ≈ Z2.

21. The system according to claim 18, wherein the sparse array is non-uniformly distributed.

22. The aforementioned plurality of reflected signals are the first plurality of reflected signals, The system further comprises a LiDAR subsystem (320) configured to transmit a plurality of optical signals and collect a second plurality of reflected signals, When executed, the at least one machine-executable instruction causes the at least one processor to determine the projection position of the target, based on the second plurality of reflected signals. The system according to claim 15.

23. A vehicle comprising the system described in claim 15.

Images