Fusion of depth imagers and radar systems in vehicles

Abstract

Systems and methods for performing sensor fusion using a depth imager and a radar system involve transmitting radio frequency (RF) energy from the radar system to an area and simultaneously transmitting light to the area using a light source, receiving reflected light at the depth imager aligned with the light source, and receiving RF reflections at the radar system. The reflected light is processed to obtain azimuth, elevation, range, range variance, and reflectivity for each pixel making up the area. The RF reflections are processed to provide azimuth, elevation, range, range variance, velocity, and velocity variance for a subset of pixels representing the area of interest. Performing sensor fusion includes using the azimuth, elevation, range variance, and reflectivity generated by the depth imager and the range, velocity, and velocity variance generated by the radar system.

Description

TECHNICAL FIELD

[0001] The subject disclosure relates to the fusion of a depth imager and a radar system in a vehicle.

[0002] Vehicles, such as cars, trucks, construction equipment, farm equipment, automated factory equipment, are increasingly using sensors to acquire information about the vehicle and its environment. Information from the sensors facilitates semi-autonomous operation (e.g., lane departure correction, automatic steering or braking) and autonomous operation of the vehicle. Exemplary sensors typically used to acquire information about the vehicle's environment include cameras, radio detection and ranging (radar) systems, and light detection and ranging (lidar) systems. In general, depth imagers are not considered suitable for automotive applications, but the sensors can be advantageous when their shortcomings are addressed. Thus, it is desirable to provide a fusion of a depth imager and a radar system in a vehicle. SUMMARY

[0003] In one exemplary embodiment, a method of performing sensor fusion using a depth imager and a radar system includes transmitting radio frequency (RF) energy from the radar system to an area and transmitting light using a light source to the area at the same time the RF energy is transmitted. The method also includes receiving, at a depth imager aligned with the light source, reflected light from the area resulting from the light transmitted by the light source and receiving, at the radar system, RF reflections resulting from reflections of the RF energy transmitted by the radar system by one or more objects in the area. The reflected light is processed to obtain an azimuth angle, an elevation angle, a distance, a distance variance, and a reflectivity for each of a plurality of pixels making up the area. The RF reflections are processed to obtain an azimuth angle, an elevation angle, a distance, a distance variance, a velocity, and a velocity variance for a subset of the plurality of pixels of the area corresponding to the one or more objects, the subset of the plurality of pixels of the area representing a region of interest. Sensor fusion is performed using the azimuth angle, the elevation angle, the distance variance, and the reflectivity produced by the depth imager and the distance, the velocity, and the velocity variance produced by the radar system within a region of interest determined by the radar system.

[0004] In addition to one or more of the features described herein, performing sensor fusion further includes obtaining a high resolution image in the region of interest.

[0005] In addition to one or more of the features described herein, obtaining a high resolution image in the region of interest further includes obtaining the azimuth angle, the elevation angle, and the reflectivity produced by the depth imager for the subset of the plurality of pixels corresponding to the region of interest determined by the radar system.

[0006] In addition to one or more of the features described herein, performing sensor fusion further includes determining a fusion-based distance for the one or more objects corresponding to the subset of the plurality of pixels.

[0007] In addition to one or more of the features described herein, determining the fusion-based distance includes using the reflected light-based distance, the RF reflection-based distance, and a maximum detectable distance of the depth imager.

[0008] In addition to one or more of the features described herein, the maximum detectable distance is based on a frequency of light emitted by the light source.

[0009] In another example embodiment, a method of performing sensor fusion in a vehicle having a depth imager and a radar system includes transmitting radio frequency (RF) energy from the radar system to an area and using a light source to emit light to the area while the RF energy is being transmitted. The method also includes receiving, at the depth imager aligned with the light source, reflected light from the area resulting from the light emitted by the light source and receiving, at the radar system, RF reflections resulting from reflections of the RF energy transmitted by the radar system by one or more objects in the area. The reflected light is processed to obtain an azimuth angle, an elevation angle, a distance, a distance variance, and a reflectivity for each of a plurality of pixels constituting the area. The RF reflections are processed to obtain an azimuth angle, an elevation angle, a distance, a distance variance, a velocity, and a velocity variance for a subset of the plurality of pixels of the area corresponding to the one or more objects, the subset of the plurality of pixels of the area representing a region of interest. The sensor fusion is performed using the azimuth angle, the elevation angle, the distance variance, and the reflectivity produced by the depth imager and the distance, the velocity, and the velocity variance produced by the radar system within the region of interest determined by the radar system. Information obtained based on the sensor fusion enables autonomous or semi-autonomous control of the vehicle.

[0010] In addition to one or more of the features described herein, performing the sensor fusion includes obtaining a high resolution image in the region of interest.

[0011] In addition to one or more of the features described herein, obtaining the high resolution image in the region of interest includes obtaining the azimuth angle, the elevation angle, and the reflectivity produced by the depth imager for the subset of the plurality of pixels corresponding to the region of interest determined by the radar system.

[0012] In addition to one or more of the features described herein, performing the sensor fusion includes determining a fusion-based distance for the one or more objects corresponding to the subset of the plurality of pixels.

[0013] In addition to one or more of the features described herein, determining the fusion-based distance includes using the reflected light-based distance, the RF reflection-based distance, and a maximum detectable distance of the depth imager.

[0014] In addition to one or more of the features described herein, the maximum detectable distance is based on a frequency of light emitted by the light source.

[0015] In another example embodiment, a system for performing sensor fusion in a vehicle includes a radar system of the vehicle to transmit radio frequency (RF) energy to an area and to receive RF reflections resulting from the radar system transmitted RF energy being reflected by one or more objects in the area, and an optical sensor including a light source to emit light toward the area at the same time the RF energy is emitted and a depth imager aligned with the light source to receive reflected light from the area resulting from the light source emitted light. A controller processes the reflected light to obtain an azimuth angle, an elevation angle, a distance, a distance variance, and a reflectivity for each of a plurality of pixels making up the area. The RF reflections are processed to obtain an azimuth angle, an elevation angle, a distance, a distance variance, a velocity, and a velocity variance for a subset of the plurality of pixels corresponding to the one or more objects, the subset of the plurality of pixels in the area representing a region of interest. Within the region of interest determined by the radar system, sensor fusion is performed using the azimuth angle, the elevation angle, the distance variance, and the reflectivity resulting from the depth imager and the distance, the velocity, and the velocity variance resulting from the radar system.

[0016] In addition to one or more of the features described herein, the controller obtains a high resolution image in the region of interest as part of the sensor fusion.

[0017] In addition to one or more of the features described herein, the controller obtaining the high resolution image in the region of interest includes obtaining the azimuth angle, the elevation angle, and the reflectivity resulting from the depth imager for the subset of the plurality of pixels corresponding to the region of interest determined by the radar system.

[0018] In addition to one or more of the features described herein, the controller determines a fusion based distance for the one or more objects corresponding to the subset of the plurality of pixels as part of the sensor fusion.

[0019] In addition to one or more of the features described herein, the controller determines the fusion based distance by using the distance based on the reflected light, the distance based on the RF reflections, and a maximum detectable distance of the depth imager.

[0020] In addition to one or more of the features described herein, the maximum detectable distance is based on a frequency of the light emitted by the light source.

[0021] In addition to one or more of the features described herein, the controller implements semi-autonomous control of the vehicle based on information obtained through the sensor fusion.

[0022] In addition to one or more of the features described herein, the controller implements autonomous control of the vehicle based on information obtained through the sensor fusion.

[0023] The above features and advantages and other features and advantages of the present application are readily apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, aspects, and details are described in only examples, and with reference to the drawings, in which:

[0025] Figure 1 is a block diagram of a vehicle using a depth imager in conjunction with a radar system according to one or more embodiments;

[0026] Figure 2 is a process flow of a method of performing fusion of a depth imager and a radar system in a vehicle according to one or more embodiments; and

[0027] Figure 3 shows exemplary emitted and reflected light used by an optical sensor according to one or more embodiments. DETAILED DESCRIPTION

[0028] The following description is merely exemplary in nature and is not intended to limit the present disclosure and its applications or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0029] As previously mentioned, depth imagers are not typically part of the sensors used in vehicles. Depth imagers are part of optical sensors, and more specifically, optical time-of-flight cameras, that work in synchronization with the light source of the optical sensor to capture the round trip time of light from the light source to one or more objects to the depth imager that captures any light reflected by the objects. The distance and reflectivity obtained by the depth imager provides an image of the scene within the field of view of the optical sensor. The maximum detectable distance of the depth imager is inversely related to the distance accuracy. That is, the distance accuracy increases as the frequency increases, but the maximum detectable distance decreases. This tradeoff between the maximum detectable distance and the distance accuracy in depth imagers is not suitable for obtaining information about the environment surrounding a vehicle.

[0030] Embodiments of the systems and methods detailed herein relate to the fusion of a depth imager and a radar system in a vehicle. Specifically, the depth imager is used in conjunction with the radar system to obtain the advantages of each sensor while mitigating the disadvantages of each sensor. The increase in distance accuracy when the depth imager has a low maximum distance is combined with the long maximum distance of the radar system. The reflectivity information from the depth imager helps to obtain a high density reflectivity image that is not possible using the radar system alone. Amplitude modulated continuous wave (AMCW) optical signals are used for the depth imager while frequency modulated continuous wave (FMCW) radio frequency (RF) signals are used for the radar system, as detailed.

[0031] According to exemplary embodiments,Figure 1 is a block diagram of a vehicle 100 in which a depth imager 110 is used with a radar system 130. Figure 1 The example vehicle 100 in is an automobile 101. The depth imager 110 is shown co-located with a light source 120. Together, the light source 120 and the depth imager 110 can be referred to as an optical sensor 105. The light source 120 can be a laser, a vertical cavity surface emitting laser (VCSEL) array, or a light emitting diode (LED) array, for example. According to an example embodiment, the light source 120 can emit AMCW light 125 such that the depth imager 110 is an indirect time-of-flight (i-ToF) depth imager based on AMCW. In Figure 1 In the scenario shown, the light 125 is reflected, for example, by an object 160. Figure 1 The example object 160 shown is a pedestrian 165. Instead, the object 160 can be another automobile, a building, a bush, or anything else in the environment of the vehicle 100.

[0032] The frequency f of the light 125 emitted by the light source 120 can be tens of megahertz (MHz). The maximum detectable distance Rmax of the depth imager 110 is given by:

[0033]

[0034] Thus, the maximum detectable distance Rmax of the depth imager 110 is about 15 meters (m). The distance accuracy is about 0.01 m. The spatial resolution can be 0.1 degree azimuth and 0.1 degree elevation. In comparison, the maximum detectable distance of the radar system 130 can be on the order of 200 m with a resolution on the order of 0.1 m. The depth imager 110 and the light source 120 need not be co-located as shown in the example case. However, the depth imager 110 must be aligned with the light source 120 such that the reflected light 115 is received (i.e., the depth imager 110 and the light source 120 must have a common field of view).

[0035] The radar system 130 is also shown co-located with the optical sensor 105. The radar system 130 emits RF energy 133 and receives RF reflections 135 based on reflections of some of the RF energy 133 by one or more objects 160 in its field of view. According to one or more embodiments, the optical sensor 105 and the radar system 130 are synchronized and have the same field of view, but need not be co-located as shown. As noted above, the radar system 130 can emit a frequency modulated continuous wave as the RF energy 133, while the light source 120 emits an amplitude modulated continuous wave as the light 125. Examples of the emitted AMCW light 125 and FMCW RF energy 133 are shown in Figure 1The values ​​are expressed in microseconds (μsec) over time T. The frequency f of the RF energy 133 is expressed in gigahertz (GHz), while the amplitude A of the light 125 is expressed in watts (W).

[0036] Vehicle 100 displays a controller 140 and one or more other sensors 150 (e.g., cameras, lidar systems, additional radar). The number and location of the sensors 150 are not limited. Figure 1 Limitations of the example shown. Controller 140 may acquire data from optical sensor 105 and radar system 130 to perform fusion according to one or more embodiments, as described in detail. Depth imager 110 and radar system 130 may perform processing of the received signals (115, 135) or may provide data to controller 140 for processing. Controller 140 may use the information generated by fusion, either alone or in addition to information from other sensors 150, to control aspects of autonomous or semi-autonomous operation of vehicle 100. Any processor within controller 140 and depth imager 110 and / or radar system 130 may include processing circuitry that may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped), and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components providing the described functionality.

[0037] Figure 2 This is a processing flow of a method 200 for fusing depth imager 110 and radar system 130 in vehicle 100 according to one or more embodiments. In block 210, light source 120 emits light 125, while radar system 130 emits RF energy 133. Light 125 can be AMCW, while RF energy 133 is FMCW, such as... Figure 1 As shown. In box 220, the depth imager 110 receives and processes the reflected light 115 to obtain the azimuth angle θop, elevation angle φop, distance Rop with uncertainty (i.e., variance) ±σop, and reflectivity (i.e., amplitude or intensity) for points (i.e., pixels) in the overlapping field of view of the light source 120 constituting the emitting light 125 and the depth imager 110 receiving the reflected light 115. That is, the depth imager 110, like any camera, obtains information about each pixel within the camera's field of view (i.e., region).

[0038] As previously mentioned, this processing can be performed by the processing circuitry within the depth imager 110 or by the controller 140, or a combination of both. The "op" designation distinguishes the optical sensor 105 from the radar system 130, with the values ​​for the radar system 130 identified by "r". Furthermore, higher-precision results (e.g., azimuth angles θop and elevation angles φop obtained via the optical sensor 105, rather than azimuth angles θr and elevation angles φr obtained via the radar system 130) are achieved in... Figure 2are indicated in bold. At this stage, no fusion according to one or more embodiments is involved, and only the processing of the reflected light 115 received at the depth imager 110 is known. Reference is made to Figure 3 Further discussion of distance and reflectivity determination.

[0039] At block 230, the radar system 130 receives the RF reflections 135 and processes the RF reflections 135 to obtain, for a set of points corresponding to reflectors on one or more objects 160 encountered by the transmitted RF energy 133, an azimuth angle θr, an elevation angle φr, a distance Rrwith an uncertainty σr, a velocity V with a variance ±σr. Unlike the depth imager 110 which obtains information for every pixel in the region constituting the field of view, the radar system 130 only obtains information for a subset of pixels in the field of view corresponding to the objects 160 that reflected the RF energy 113 as the RF reflections 135. This subset of pixels constitutes a region of interest within the entire region (i.e., the region of interest is the region occupied by the objects 160).

[0040] The processing of the RF reflections 135 can be performed by a processing circuit or controller 140 within the radar system 130 or a combination of both. As noted for the depth imager 110, at this stage there is no fusion according to one or more embodiments, and the processing of the RF reflections 135 is well known. Specifically, for a set of received RF reflections 135, a fast Fourier transform is performed along the range bins, followed by a second FFT on the first FFT result. This is followed by digital beamforming which provides a range-Doppler (i.e., relative velocity) map for each beam. In addition to the distance Rrwith an uncertainty σr, the velocity V with a variance ±σr, the azimuth angle θrand the elevation angle φrare obtained for each detected object 160, each detected object representing the region of interest, employing thresholding.

[0041] At block 240, obtaining a high resolution image representation in the region of interest is the first stage of fusion according to one or more embodiments. In particular, for pairs of azimuth angles θrand elevation angles φrobtained in the region of interest identified according to the radar system 130, the corresponding higher precision azimuth angle θopand elevation angle φopobtained by the depth imager 110 for each pixel within the region of interest and the associated reflectivity at the azimuth angle θopand the elevation angle φopare obtained as a high resolution image, as shown in Figure 2 .

[0042] At block 250, a higher precision distance R (i.e., a fusion-based distance) is computed for each pixel in the region of interest. The relatively higher precision distance Rr obtained by the radar system 130 and the relatively higher precision variance σop obtained by the depth imager 110 are used to obtain a distance R for each pixel i,j in the region of interest according to one or more embodiments:

[0043]

[0044] In Equation 2, Rmax is the maximum detectable distance using the depth imager 110 according to Equation 1.

[0045]

[0046] R opi，j = R i，j mod R max [Equation 4]

[0047] In Equation 4, each distance Rij mod (i.e., modulo) Rmax will give the remainder of dividing the distance Rij by Rmax. In Equations 2 and 3, Rr can be written as Rrij, but the pixel density of the distances determined via the radar system 130 is much lower than the pixel density obtained via the depth imager 110. Moreover, the radar system 130 does not have a fixed pixel grid like the depth imager 110. According to the processing at block 250, the higher precision distance R computed by fusion using Equation 2 and the corresponding higher precision distance variance σop obtained by the depth imager 110 result.

[0048] At block 260, the results from the first fusion step from blocks 240 and 250 are combined with the velocity V and corresponding variance ±σr from the radar system 130. Specifically, from block 240, the high resolution image obtained using the depth imager 110 in the region of interest identified by the radar system 130 (at block 240) is represented by reflectivity at the azimuth angle θop and elevation angle φop related to the region of interest. From block 250, the higher precision distance R computed by fusion using Equation 2 and the corresponding higher precision distance variance σop obtained by the depth imager 110 result. The controller 140 can control aspects of autonomous or semi-autonomous operation of the vehicle 100 based on the results. For example, the controller 140 can implement automatic braking, adaptive cruise control, or lane keeping in a semi-autonomous vehicle 100, or can implement trajectory planning in an autonomous vehicle 100 using the higher precision information resulting from fusion compared to either sensor (105, 130) alone.

[0049] Figure 3An exemplary emitted light 125 and reflected light 115 used by optical sensor 105 according to one or more embodiments is shown. The amplitude A of both signals is expressed in Watts, as a function of time T in seconds (sec). The emitted light 125 is an AMCW signal as shown. Four samples C1, C2, C3, C4 are shown over a period of reflected light 115. C1, C2, C3, and C4 are the integral values resulting from the integration of the respective samples of reflected signal 115. The C1, C2, C3, and C4 values are at the pixel level, such that four samples are obtained per pixel. Equations 5-7 are used to calculate reflectivity and distance for each pixel. The distance R op By way of introduction:

[0050]

[0051]

[0052] The amplitude A is given by:

[0053]

[0054] In Equation 5, R op is the distance, φ d is the modulation phase delay resulting from the round-trip time-of-flight to object 160 (i.e., a given pixel within the region of interest).

[0055] While the foregoing disclosure has been described in reference to exemplary embodiments, it will be understood by those skilled in the art that various changes can be made and equivalents can be substituted for elements thereof without departing from the scope of the present disclosure. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the present disclosure without departing from the central scope thereof. Therefore, the present disclosure is not limited to the particular embodiments disclosed, but includes all embodiments falling within the scope of the appended claims.

Claims

1. A method of performing sensor fusion in a vehicle having a depth imager and a radar system, the method comprising: transmitting radio frequency (RF) energy from the radar system to an area; emitting light toward the area using a light source while the RF energy is being transmitted; receiving, at a depth imager aligned with the light source, reflected light from the area resulting from the light emitted by the light source; receiving, at the radar system, RF reflections resulting from the RF energy transmitted by the radar system being reflected by one or more objects within the area; processing the reflected light to obtain an azimuth angle, an elevation angle, a range, a range variance, and a reflectivity for each of a plurality of pixels making up the area; processing the RF reflections to obtain an azimuth angle, an elevation angle, a range, a range variance, a velocity, and a velocity variance for a subset of the plurality of pixels corresponding to the one or more objects, the subset of the plurality of pixels representing a region of interest in the area; performing sensor fusion in the region of interest determined by the radar system based on the azimuth angle, the elevation angle, the range variance, and the reflectivity generated by the depth imager, and the range, the velocity, and the velocity variance generated by the radar system, to obtain a high resolution image of the region of interest, wherein performing sensor fusion includes assigning the azimuth angle and the elevation angle generated by the depth imager and the associated reflectivity at the azimuth angle and the elevation angle to each of the pixels in the region of interest; and implementing autonomous or semi-autonomous control of the vehicle based on information obtained from the sensor fusion.

2. The method of claim 1, wherein, the performing sensor fusion includes determining a fusion-based range for the one or more objects corresponding to the subset of the plurality of pixels.

3. The method of claim 2, wherein the determining a fusion-based distance comprises: the range based on the reflected light, the range based on the RF reflections, and a maximum detectable range of the depth imager are used, and the maximum detectable range is based on a frequency of the light emitted by the light source.

4. A system for performing sensor fusion in a vehicle, the system comprising: a radar system of the vehicle configured to transmit radio frequency (RF) energy to an area and to receive RF reflections resulting from the RF energy transmitted by the radar system being reflected by one or more objects within the area; an optical sensor including a light source configured to emit light toward the area while the RF energy is being transmitted, and a depth imager aligned with the light source and configured to receive reflected light from the area resulting from the light emitted by the light source; and a controller configured to process the reflected light to obtain an azimuth angle, an elevation angle, a distance, a distance variance, and a reflectivity for each of a plurality of pixels constituting the region, configured to process the RF reflections to obtain an azimuth angle, an elevation angle, a distance, a distance variance, a velocity, and a velocity variance for a subset of the plurality of pixels of the region corresponding to the one or more objects, the subset of the plurality of pixels of the region representing a region of interest, and configured to obtain a high resolution image of the region of interest based on performing sensor fusion in the region of interest determined by the radar system based on utilizing the azimuth angle, the elevation angle, the distance variance, and the reflectivity produced by the depth imager, and the distance, the velocity, and the velocity variance produced by the radar system, wherein performing the sensor fusion includes assigning the azimuth angle and the elevation angle produced by the depth imager and the associated reflectivity at the azimuth angle and the elevation angle to each pixel in the region of interest.

5. The system of claim 4, wherein, the controller is configured to determine a fused-based distance for the one or more objects corresponding to the subset of the plurality of pixels as part of the sensor fusion, and to determine the fused-based distance by using the distance based on the reflected light, the distance based on the RF reflections, and a maximum detectable distance of the depth imager, and the maximum detectable distance is based on a frequency of the light emitted by the light source.

6. The system of claim 4, wherein, the controller is configured to implement a semi-autonomous control or an autonomous control of the vehicle based on information obtained by the sensor fusion.

Images