Early fusion of camera and radar frames
By receiving and processing camera frames and radar frames in the vehicle, generating and connecting feature maps for object detection, the problem of indirect integration of early fusion level information is solved, efficient object detection is achieved, and the performance of the vehicle safety system is enhanced.
Patent Information
- Application Number
- CN201980077199.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-11-27
- Filing Date
- 2019-11-29
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-11-29
AI Technical Summary
The existing radar-vision fusion technology has insufficient information integration at the early fusion level, resulting in high computing costs and difficulty in achieving efficient object detection in vehicle safety applications.
By receiving camera frames and radar frames in an on-board computer, camera feature extraction and radar feature extraction are performed, feature maps are converted to the public space domain, and connected to generate connection feature maps, perform object detection, and adopt an encoder-decoder network Convert the camera feature map into the radar frame space domain to generate a combined feature map to achieve early fusion.
Improves the accuracy and efficiency of object detection, reduces calculation costs, and enhances the performance of vehicle safety systems, especially in applications such as adaptive cruise control, forward collision warning and lane departure warning.
Smart Images

Figure CN113167888B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This patent application claims the benefit of U.S. Provisional Application No. 62 / 774,020, filed on November 30, 2018, and entitled “EARLY FUSION OF CAMERA AND RADAR FRAMES,” and U.S. Non-Provisional Application No. 16 / 698,601, filed on November 27, 2019, and entitled “EARLY FUSION OF CAMERA AND RADAR FRAMES,” both of which are assigned to the assignee of this application, and the entire contents of which are expressly incorporated herein by reference. Technical Field
[0003] Aspects of the present disclosure generally relate to early fusion of camera frames and radar frames. Background Art
[0004] Radar sensors and camera sensors are commonly used in vehicles to enable systems to enhance vehicle safety, such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance through automatic braking, pre-collision functions (such as airbag activation or pre-activation), and lane departure warning (LDW). Systems using radar sensors and camera sensors can provide a high level of active safety capabilities and are increasingly used in production vehicles.
[0005] One or more camera sensors mounted on the vehicle capture images of the scene in front of the vehicle and may also capture images of the scene behind and to the sides of the vehicle. A digital signal processor (DSP) within the vehicle then attempts to identify objects within the captured images. Such objects can be other vehicles, pedestrians, road signs, objects within the path of travel, etc. Radar systems can also be used to detect objects on the path of travel. Radar systems use radio waves to determine the distance, height, direction, and / or speed of objects on the road. A transmitter sends out pulses of radio waves that reflect off objects in its path. The pulses reflected from the object return a small portion of the radio wave energy to a receiver that is usually located at the same location as the transmitter.
[0006] Radar systems provide fairly accurate measurements of object distance and velocity in all weather conditions. However, radar systems typically do not have sufficient resolution to identify features of detected objects. Camera sensors, on the other hand, typically have sufficient resolution to identify object features. Clues to object shape and appearance extracted from captured images can provide sufficient features for classification of different objects. Based on the complementary properties of the two sensors, data from both sensors can be combined in a single system (called "fusion") to improve performance.
[0007] Radar-vision fusion methods differ mainly in the level of fusion. Early fusion combines several raw data sources to produce an output that is expected to be more directly informative than any one input alone. In early fusion, various features, such as edges, corners, lines, texture parameters, etc., are combined into feature maps, which are then used in further processing stages. In advanced fusion, each input source produces a decision, which is then fused. Summary of the invention
[0008] A simplified overview of one or more aspects disclosed herein is presented below. The following overview should not be considered a broad overview of all expected aspects, nor should it be considered to identify key or important elements related to all expected aspects or to describe the scope related to any particular aspect. Therefore, the sole purpose of the following overview is to present certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form before the detailed embodiments given below.
[0009] In one aspect, a method for fusing camera frames and radar frames to perform object detection performed by an onboard computer of a host vehicle in one or more spatial domains includes: receiving multiple camera frames from a camera sensor of the host vehicle; receiving multiple radar frames from a radar sensor of the host vehicle; performing a camera feature extraction process on a first camera frame of the multiple camera frames to generate a first camera feature map; performing a radar feature extraction process on a first radar frame of the multiple radar frames to generate a first radar feature map; converting the first camera feature map and / or the first radar feature map to a common spatial domain; connecting the first radar feature map and the first camera feature map to generate a first connected feature map in the common spatial domain; and detecting one or more objects in the first connected feature map.
[0010] In one aspect, a method of fusing camera frames and radar frames to perform object detection performed by an onboard computer of a host vehicle in one or more spatial domains includes: receiving multiple camera frames from a camera sensor of the host vehicle; receiving multiple radar frames from a radar sensor of the host vehicle; applying an encoder-decoder network on a first camera frame to generate a first camera feature map in the spatial domain of the first radar frame; combining the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame; and detecting one or more objects in the first combined feature map.
[0011] In one aspect, an onboard computer of a host vehicle includes at least one processor configured to: receive a plurality of camera frames from a camera sensor of the host vehicle; receive a plurality of radar frames from a radar sensor of the host vehicle; perform a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map; perform a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map; convert the first camera feature map and / or the first radar feature map to a common spatial domain; connect the first radar feature map and the first camera feature map to generate a first connected feature map in the common spatial domain; and detect one or more objects in the first connected feature map.
[0012] In one aspect, an onboard computer of a host vehicle includes at least one processor configured to: receive a plurality of camera frames from a camera sensor of the host vehicle; receive a plurality of radar frames from a radar sensor of the host vehicle; apply an encoder-decoder network on a first camera frame to generate a first camera feature map in a spatial domain of the first radar frame; combine the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame; and detect one or more objects in the first combined feature map.
[0013] In one aspect, an onboard computer of a host vehicle includes a component for receiving a plurality of camera frames from a camera sensor of the host vehicle; a component for receiving a plurality of radar frames from a radar sensor of the host vehicle; a component for performing a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map; a component for performing a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map; a component for converting the first camera feature map and / or the first radar feature map to a common spatial domain; a component for connecting the first radar feature map and the first camera feature map to generate a first connected feature map in the common spatial domain; and a component for detecting one or more objects in the first connected feature map.
[0014] In one aspect, an onboard computer of a host vehicle includes components for receiving a plurality of camera frames from a camera sensor of the host vehicle; components for receiving a plurality of radar frames from a radar sensor of the host vehicle; components for applying an encoder-decoder network on a first camera frame to generate a first camera feature map in a spatial domain of the first radar frame; components for combining the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame; and components for detecting one or more objects in the first combined feature map.
[0015] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions includes computer-executable instructions, the instructions including at least one instruction instructing an onboard computer of a host vehicle to receive a plurality of camera frames from a camera sensor of the host vehicle; at least one instruction instructing the onboard computer to receive a plurality of radar frames from a radar sensor of the host vehicle; at least one instruction instructing the onboard computer to perform a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map; at least one instruction instructing the onboard computer to perform a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map; at least one instruction instructing the onboard computer to convert the first camera feature map and / or the first radar feature map to a common spatial domain; at least one instruction instructing the onboard computer to connect the first radar feature map and the first camera feature map to generate a first connected feature map in the common spatial domain; and at least one instruction instructing the onboard computer to detect one or more objects in the first connected feature map.
[0016] In one aspect, a non-transitory computer-readable medium storing computer-executable instructions includes computer-executable instructions including at least one instruction instructing an onboard computer of a host vehicle to receive a plurality of camera frames from a camera sensor of the host vehicle; at least one instruction instructing the onboard computer to receive a plurality of radar frames from a radar sensor of the host vehicle; at least one instruction instructing the onboard computer to apply an encoder-decoder network on a first camera frame to generate a first camera feature map in a spatial domain of the first radar frame; at least one instruction instructing the onboard computer to combine the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame; and at least one instruction instructing the onboard computer to detect one or more objects in the first combined feature map.
[0017] Other objects and advantages associated with the various aspects disclosed herein will be apparent to those skilled in the art based on the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings are presented to aid in describing various aspects of the disclosure and are intended solely for purposes of illustration of these aspects and not limitation thereof.
[0019] Figure 1 A top view of a vehicle with an integrated radar camera sensor behind the windshield according to various aspects.
[0020] Figure 2 An onboard computer architecture according to various aspects is shown.
[0021] Figure 3 It is a schematic diagram of the induction observation radar grid.
[0022] Figure 4 An exemplary difference between a camera image and a radar image of the same scene is shown.
[0023] Figure 5 is a block diagram illustrating how a radar-camera sensor module and / or one or more processors may process data from a camera sensor and a radar sensor in accordance with various aspects.
[0024] Figure 6 A road scene according to one aspect of the present disclosure is shown.
[0025] Figure 7 and Figure 8 Exemplary methods for fusing camera frames and radar frames to perform object detection in one or more spatial domains in accordance with aspects of the present disclosure are shown. DETAILED DESCRIPTION
[0026] Various aspects of the present disclosure are provided in the following description and related drawings, which are directed to various examples provided for illustrative purposes. Alternative aspects may be designed without departing from the scope of the present disclosure. In addition, well-known aspects of the present disclosure may not be described in detail or may be omitted to avoid obscuring more relevant details.
[0027] Those skilled in the art will appreciate that the information and signals described below may be represented using any of a variety of different techniques and technologies. For example, data, instructions, commands, information, signals, bits, symbols, and chips referenced in the following description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.
[0028] In addition, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be appreciated that the various actions described herein may be performed by specific circuits (e.g., application specific integrated circuits (ASICs)), by program instructions executed by one or more processors, or a combination of both. In addition, for each aspect described herein, the corresponding form of any such aspect may be implemented, for example, as logic that is "configured to" perform the described actions.
[0029] Reference now Figure 1, a vehicle 100 (referred to as the “ego vehicle” or “host vehicle”) is shown including a radar-camera sensor module 120 located in an interior cabin of the vehicle 100 behind a windshield 112. The radar-camera sensor module 120 includes a radar assembly configured to transmit radar signals through the windshield 112 in a horizontal coverage area 150 (shown by a dashed line) and receive reflected radar signals reflected by any object within the coverage area 150. The radar-camera sensor module 120 also includes a camera assembly for capturing images based on light waves seen and captured through the windshield 112 in a horizontal coverage area 160 (shown by a dashed line).
[0030] Although Figure 1 The radar assembly and the camera assembly are shown as examples of juxtaposed components in a common housing, but it is understood that they can be housed in different locations within the vehicle 100. For example, the camera can be as shown. Figure 1 As shown, the radar assembly may be located in the grille or front bumper of the vehicle 100. Figure 1 The radar-camera sensor module 120 is shown located behind the windshield 112, but it can also be located in the roof sensor array or elsewhere. Figure 1 Only a single radar-camera sensor module 120 is shown, but it is understood that the vehicle 100 may have multiple radar-camera sensor modules 120 pointing in different directions (side, front, rear, etc.). The various radar-camera sensor modules 120 may be under the "skin" of the vehicle (e.g., behind the windshield 112, door panels, bumpers, grilles, etc.) or in a rooftop sensor array.
[0031] The radar-camera sensor module 120 can detect one or more (or no) objects relative to the vehicle 100. Figure 1 In the example of FIG. 1 , there are two objects, namely vehicles 130 and 140, within horizontal coverage areas 150 and 160 that can be detected by the radar-camera sensor module 120. The radar-camera sensor module 120 can estimate parameters of the detected object(s), such as position, distance, direction, speed, size, category (e.g., vehicle, pedestrian, road sign, etc.), etc. The radar-camera sensor module 120 can be used on the vehicle 100 for automotive safety applications, such as adaptive cruise control (ACC), forward collision warning (FCW), collision mitigation or avoidance through automatic braking, lane departure warning (LDW), etc.
[0032] Collocating the camera frame and radar allows these components to share electronics and signal processing, particularly enabling early radar-camera data fusion. For example, the radar and camera can be integrated on a single board. Radar-camera joint alignment techniques can be used to align the radar and camera. However, collocation of the radar and camera is not required to practice the techniques described herein.
[0033] Figure 2 An on-board computer (OBC) 200 of a vehicle 100 is shown in accordance with various aspects of the present disclosure. In one aspect, the OBC 200 may be part of an ADAS or ADS. The OBC 200 includes a non-transitory computer-readable storage medium, namely a memory 204, and one or more processors 206 in communication with the memory 204 via a data bus 208. The memory 204 includes one or more storage modules that store computer-readable instructions executable by the processor(s) 206 to perform the functions of the OBC 200 described herein. For example, the processor(s) 206 together with the memory 204 may implement the various neural network architectures described herein.
[0034] One or more radar-camera sensor modules 120 are coupled to the OBC 200 (for simplicity, Figure 2 In some aspects, the radar-camera sensor module 120 includes at least one camera 212, at least one radar 214, and an optional light detection and ranging (LiDAR) sensor 216. The OBC 200 also includes one or more system interfaces 220 that connect the processor(s) 206 to the radar-camera sensor module 120 via a data bus 208, and optionally, to other vehicle subsystems (not shown).
[0035] At least in some cases, the OBC 200 also includes a wireless wide area network (WWAN) transceiver 230, which is configured to communicate via one or more wireless communication networks (not shown), such as NR networks, LTE networks, GSM networks, etc. The WWAN transceiver 230 can be connected to one or more antennas (not shown) for communicating with other network nodes (e.g., other vehicle UEs, pedestrian UEs, infrastructure access points, roadside units (RSUs), base stations (e.g., eNBs, gNBs), etc.) over a related wireless communication medium (e.g., a set of time / frequency resources in a specific spectrum) via at least one designated RAT (e.g., NR, LTE, GSM, etc.). The WWAN transceiver 230 can be configured differently for sending and encoding signals (e.g., messages, indications, information, etc.) according to the designated RAT, and conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, etc.).
[0036] At least in some cases, the OBC 200 also includes a wireless local area network (WLAN) transceiver 240. The WLAN transceiver 240 can be connected to one or more antennas (not shown) for communicating with other network nodes (e.g., other vehicle UEs, pedestrian UEs, infrastructure access points, RSUs, etc.) over an associated wireless communication medium via at least one designated RAT (e.g., cellular vehicle-to-everything (C-V2X), IEEE 802.11p (also known as Wireless Access for Vehicular Environments (WAVE)), dedicated short-range communications (DSRC), etc.). The WLAN transceiver 240 can be configured differently for sending and encoding signals (e.g., messages, indications, information, etc.) according to the designated RAT, and conversely, for receiving and decoding signals (e.g., messages, indications, information, pilots, etc.).
[0037] As used herein, a "transceiver" may include transmitter circuitry, receiver circuitry, or a combination thereof, but need not provide both transmit and receive functionality in all designs. For example, a low-function receiver circuit may be employed in some designs to reduce cost when full communications need not be provided (e.g., a receive chip or similar circuit that simply provides low-level sniffing).
[0038] At least in some cases, the OBC 200 also includes a global positioning system (GPS) receiver 250. The GPS receiver 250 can be connected to one or more antennas (not shown) for receiving satellite signals. The GPS receiver 250 can include any suitable hardware and / or software for receiving and processing GPS signals. The GPS receiver 250 requests appropriate information and operations from other systems and uses measurements obtained through any suitable GPS algorithm to perform calculations required to determine the location of the vehicle 100.
[0039] In one aspect, the OBC 200 can utilize the WWAN transceiver 230 and / or the WLAN transceiver 240 to download one or more maps 202, which can then be stored in the memory 204 and used for vehicle navigation. The map(s) 202 can be one or more high-definition (HD) maps that can provide accuracy within 7-10 centimeters absolute distance, highly detailed information of all fixed physical assets associated with the road (e.g., lanes, road edges, shoulders, dividers, traffic signals, signs, paint markings, poles), and other data useful for safe navigation of the vehicle 100 on the road and at intersections. The map(s) 202 can also provide electronic horizon prediction awareness, which enables the vehicle 100 to know what is ahead.
[0040] In one aspect, the camera 212 may capture images within the viewing area of the camera 212 at a periodic rate. Figure 1Similarly, radar 214 may capture images of a scene within the viewing area of radar 214 (e.g., horizontal coverage area 160). Figure 1 150). The periodicity rate at which camera 212 and radar 214 capture their respective frames may be the same or different. Each camera frame and radar frame may be time stamped. Thus, in the case of different periodicity rates, the time stamps may be used to select camera frames and radar frames captured simultaneously or nearly simultaneously for further processing (e.g., fusion).
[0041] Figure 3 The radar grid 300 is shown. The transmitter (eg, transmit antenna array) of the radar 214 transmits signals from the object(s) in the transmission path (eg, Figure 1 A portion of the electromagnetic RF waves reflected from the object(s) returns to a receiver (e.g., a receiving antenna array) of radar 214, which is typically located at the same location as the transmitter of radar 214.
[0042] In one aspect, radar 214 can be an imaging radar that uses beamforming to scan horizontally and vertically. Beamforming is a technique that aims the effective direction of a radar beam by varying the delays between different transmit antennas so that the signal constructively increases in a specified direction. Thus, radar 214 can scan an entire sensing area (e.g., horizontal coverage area 150) horizontally and vertically by using a transmitter that includes an electronically steered antenna array.
[0043] The return response measured by the radar 214 is characterized as an observation grid 340 having a plurality of observation cells 342. Each cell 342 represents a return response value measured at a specific distance (r) and angle / azimuth (θ). Each cell 342 is also alternatively referred to as a distance-angle bin. Features 344 are extracted from the cells 342 to determine whether the features 344 are objects (e.g., vehicles 130 / 140). Each feature 344 within each cell 342 can be identified as having up to four parameters: distance, Doppler, azimuth, and elevation. This is called a radar frame. As an example, the features 344 within the cell 342 can be a signal-to-noise ratio (SNR) calculated by a constant false alarm rate (CFAR) algorithm. However, it should be understood that other methods can be used to locate and identify the features 344 within the cell 342.
[0044] Compared to images (e.g., from camera 212), radar signals (e.g., from radar 214) have several unique characteristics. One is specular reflection, which manifests itself in that only certain surfaces on the target with favorable azimuths reflect the radar signal, which generally results in a small number of reflections. The second is non-uniformity, which manifests itself in that objects that appear far away in {range, azimuth} signal space are always smaller and have a different shape than objects that appear closer. This is not the case with camera images.
[0045] Figure 4 An exemplary difference between a camera image 410 and a radar image 420 of the same scene is shown. A radar image (e.g., radar image 420), which may also be referred to as a radar frame, may be a visual representation of a sensed occupancy radar grid (e.g., sensed occupancy radar grid 300). A radar image is a range-azimuth matrix / tensor (where the radar image is a two-dimensional radar image) or a range-azimuth-Doppler matrix / tensor (where the radar image is a three-dimensional radar image). The matrix / tensor represents the received power of the returned radar signal as values along these axes. For example, for a range-azimuth radar frame (e.g., radar image 420), each range-azimuth bit will have a power value (e.g., in dB). Typically, different power values are represented as different colors and / or shades, as shown in radar image 420.
[0046] refer to Figure 4 , camera image 410 may have been captured by camera 212, and radar image 420 may have been captured by radar 214. In camera image 410, three objects (herein, vehicles) have been detected, each surrounded by bounding boxes 412, 414, and 416. In radar image 420, the same three objects (vehicles) have been detected and are surrounded by bounding boxes 422, 424, and 426. Figure 4 As can be seen in FIG. 4 , objects / bounding boxes 422, 424, and 426 have different sizes and shapes in radar image 420 based on whether they are below (i.e., bounding box 422) or above (i.e., bounding boxes 424 and 426) centerline 428 of radar image 420. In addition, since the distance between radar 214 and the other objects is greater, fewer reflected radio waves are received at radar 214, so the bounding box associated with the closer object (i.e., bounding box 422) is larger than the bounding boxes associated with the farther object (i.e., bounding boxes 424 and 426).
[0047] Figure 5 5 is a block diagram 500 illustrating how radar-camera sensor module 120 and / or processor(s) 206 process data from camera 212 and radar sensor 214 in accordance with various aspects of the present disclosure. As noted above, radar sensor 214 may capture radar frames (e.g., Figure 1 Horizontal coverage area 150 is shown. Time domain (eg, I / Q) samples 502 are generated from the captured radar frames. Time domain samples 502 are raw data that are processed to generate tensors to be used during backend processing.
[0048] More specifically, the computational front end 504 performs tensor generation, windowing, subsampling, fast Fourier transform (FFT), motion compensation, etc. to generate a four-dimensional (4D) tensor 506 of features 344 within cells 342 of the occupancy grid 340 detected by the radar sensor 214. In one aspect, the radar sensor 214 is an imaging radar sensor that scans horizontally and vertically. Thus, the tensor 506 represents the distance (the distance from the radar 214 to the detected feature 344), the azimuth (the horizontal distance between the feature 344 and a reference RF ray emitted by the radar 214, such as the initial RF ray of the radar scan), the Doppler (indicating the velocity of the detected feature 344), and the elevation (the vertical direction from the radar 214 to the detected feature) of each detected feature 344. These tensors are then passed to the computational back end 508, which performs object detection, object classification, localization, and attribute estimation based on the tensors 506 and the undistorted camera frames 516 received from the computational front end 514 for the camera 212.
[0049] On the camera 212 side, the camera 212 and the radar 214 capturing radar frames of the same scene simultaneously (or nearly simultaneously) capture the image within the viewing area of the camera 212 (e.g., Figure 1 Shown is a raw camera frame 512 of a scene with horizontal footprint 160. A computational front end 514 for camera 212 corrects any distortion in the raw camera frame 512 to generate an undistorted camera frame 516. The undistorted camera frame 516 is passed to the computational back end 508 for processing with tensor 506.
[0050] The result 510 of the processing performed by the computational backend 508 is various attributes of the detected object(s), such as a bounding box in Cartesian xy coordinates that tightly surrounds the object(s), the type of the object(s) (e.g., car, truck, pedestrian, motorcycle, road sign, etc.), relative (e.g., for vehicle 100) speed, direction, softmax function scores (probability of a given output of a tensor in binary classification) for all attributes, predicted positions of the object(s) after a period of time (e.g., 50 milliseconds, 100 milliseconds, etc.), etc.
[0051] Referring to the computational front end 504 in more detail, the computational front end 504 receives as input I / Q samples 502 organized as radar frames, where each frame can be represented as a three-dimensional (3D) tensor representing virtual receive antenna indices (which indicate MIMO operation so as to virtually construct, for example, 31 of 2 transmit (Tx) antennas and 16 receive (Rx) antennas), chirp indices, and sample indices of the corresponding radar signal. The computational front end 504 can output a 4D complex-valued tensor representing range, azimuth, Doppler, and elevation. Optionally, the computational front end 504 can sub-select, group, compress the "argmax" (the point of the domain of a function where the function value is maximized) or "arbitrary function" of one or more of these dimensions. Alternatively, the computational front end 504 can output a 3D complex-valued tensor representing range, azimuth, and Doppler. As yet another alternative, computational front end 504 may output a set of two-dimensional (2D) complex-valued tensors representing one or more of range and azimuth, range and Doppler, Doppler and azimuth, range and elevation, Doppler and elevation, or azimuth and elevation. Significant information is sometimes contained in the frequency content of the radar signal, in which case further signal processing may be performed to extract that frequency information.
[0052] Referring to the computational backend 508 in more detail, the computational backend 508 may implement various "deep learning" techniques (machine learning methods based on learning data representations, as opposed to task-specific algorithms) to determine results 510. Deep neural networks (DNNs) (a type of deep learning that includes multiple layers of artificial neural networks (ANNs) between input and output layers) may be used to detect, classify, localize, and understand objects from radar signals and camera images.
[0053] Disclosed herein are techniques for early fusion of camera frames (e.g., undistorted camera frames 516) and radar frames to perform object detection in a bird's eye view (the view of radar 214). The "bird's eye view" captured by radar 214 is a top-down or aerial view of the environment in the direction in which radar 214 transmits radio wave pulses (e.g., Figure 1 ). In contrast, camera 212 captures an image of the same environment that is substantially orthogonal to radar 214's view of the environment. Because the two views are orthogonal to each other, the radar frame and camera frame of the same scene need to be brought into the same plane (i.e., fused) so that existing object detection techniques can be applied.
[0054] Fusion can be performed at different processing levels. Early fusion combines several raw data sources, such as data from the camera 212 (e.g., raw camera frame 512), radar 214 (e.g., I / Q samples 502), and / or LiDAR sensor 216, to produce a new output that is expected to provide more direct information than the individual inputs. In early fusion, various features, such as edges, corners, lines, texture parameters, etc. are combined into a feature map, which is then used in further processing stages. In contrast, in advanced fusion, each input source (e.g., camera 212, radar 214, and / or LiDAR sensor 216) produces a decision (e.g., object detection and / or classification), and these decisions are fused. The higher the level at which fusion is performed, the lower the subsequent computational cost (due to the compressed nature of the information). However, the accuracy may be lower. In contrast, the lower the level at which fusion is performed, the higher the subsequent computational cost. However, the accuracy can be much higher. Therefore, it would be beneficial to be able to fuse information from different sources at an early stage while reducing computational costs.
[0055] In one aspect, to perform early fusion, the selected camera frame and the corresponding radar frame are passed through respective feature extraction branches, and features from both branches are concatenated to allow for subsequent detection of objects in the combined frame. As noted above, due to the different periodicity rates at which the camera 212 and radar 214 capture their respective frames, the timestamps of the frames may not match. To match the camera frame to the radar frame, a camera frame (which may be captured at a slower periodicity rate) is selected, and the most recent timestamped radar frame (which may be captured at a faster periodicity rate) is selected as the corresponding frame. The camera frame is then passed through a feature extraction branch with multiple convolutional layers (representing the integral of the amount of overlap when one function moves over another; thus, it "mixes" one function with another) to generate a latent feature map representing the camera image. The latent feature map of the camera frame (i.e., the feature map after passing through multiple convolutional layers) is transformed to the same space as the radar frame (i.e., a bird's-eye view) by performing an explicit inverse perspective mapping transformation.
[0056] Alternatively, another way to create an abstract camera image representation in the coordinate space of the radar frame is to first transform the camera image to the same space as the radar image (i.e., a bird's eye view) and then run the result through multiple convolutional layers to generate a latent feature map for the camera frame. Regardless of how the camera frame is transformed to radar space, the corresponding radar frame is run through its own feature extraction branch (with multiple convolutional layers) to generate a latent feature map representing the radar frame. The features from both branches are concatenated, and the new tensor (representing the concatenated frame) is run through a new set of convolutional layers before running the resulting representation through an object detection algorithm module (e.g., single-shot detection (SSD)) to achieve the final object detection. This process is then repeated for the next pair of camera and radar frames.
[0057] In alternative aspects, the radar frame can be brought into the camera frame, or the two frames can converge somewhere in the middle, rather than matching the camera frame to the radar frame. For example, the two frames can be tilted towards each other first, and then the result can be tilted back to a bird's eye view.
[0058] SSD is a method for detecting objects in images using a single deep neural network. SSD discretizes the output space of bounding boxes into a set of default boxes (also called "priority boxes" or "anchor boxes") at different aspect ratios and scales at each feature map location. At prediction time, the network generates scores for each object category present in each default box and adjusts the box to better match the object shape. In addition, the network combines predictions from multiple feature maps of different resolutions to naturally handle objects of various sizes. The SSD header indicates the detection header and regression header of the SSD network. The SSD model eliminates the proposal generation and subsequent pixel or feature resampling stages and encapsulates all computations in a single network. This makes SSD easy to train and directly integrated into systems that require a detection component.
[0059] As noted above, the above fusion techniques are performed for each frame (at least for frames captured at a slower periodic rate). In this way, it would be beneficial to reduce the per-frame computations of radar-based or vision-based object detection networks. This can be achieved by using block hashing. For example, in autonomous driving applications, a large portion of the frames (camera or radar) do not change (absolutely or relatively) from one frame to another. For example, a portion of each frame may capture the dashboard, sky, road, etc., and these areas may not change significantly from one frame to another, or may not change at all. Therefore, processing every complete frame in a series of frames may be unnecessary redundant.
[0060] Based on this observation, the radar-camera sensor module 120 can perform exact or perceptual hashing on the blocks of each input frame to identify which blocks have not changed from the previous frame to the current frame. For example, each frame can be divided into an 8×8 grid with 64 blocks. Next, the radar-camera sensor module 120 can use this information to simply copy the calculated feature map values from the previous frame to the current frame, rather than recalculating the same (or nearly the same) values. This information can be further propagated through the neural network to prevent the network from performing redundant calculations in later layers of the network (i.e., through the cone propagation of the feature map).
[0061] In one aspect, a pre-trained camera-based object detection network can be used to achieve bird's-eye view object detection and localization in radar frames. For example, a pre-trained object detection network for camera images, such as ResNeXt (a modular network architecture for image classification) or Inception-Resnet-V2, can be used to perform object detection in camera frames. The same network can be augmented to estimate the length of objects captured in camera frames (by augmenting the SSD header). For example, in the case where the object is a vehicle, the make and model of the vehicle, as well as the depth perceived in the camera frame, can be used to estimate the length / depth of the vehicle.
[0062] For illustrative purposes, Figure 6 An exemplary scenario is shown in which there are three objects in front of a host vehicle (e.g., vehicle 100) on a highway. The objects shown are a tractor trailer, a sport utility vehicle, and a motorcycle. Figure 6 As shown, each object is outlined by a bounding box 602, 604, and 606, respectively.
[0063] In one aspect, after inverse perspective mapping, the 2D bounding boxes (e.g., bounding boxes 602, 604, and 606) overlaid on the camera frame can be used to estimate the width of the object in the bird's eye view. This can be done using deep learning (via learned features) by the camera branch. The estimated length and width of the object can be used to form a complete bounding box in the bird's eye view.
[0064] In one aspect, instead of performing an explicit inverse perspective mapping that is very sensitive to the calibration parameters of the radar 214 and camera 212, newer techniques such as bilateral convolutional networks, deformable convolutional networks (convolutional neural networks (CNNs) in which a 2D offset is added to achieve a deformed sampling grid), or spatial transformer networks (CNNs that use spatial transformers to learn invariance to translation, scaling, rotation, and more general distortions) can be used at a later stage in the base network.
[0065] Given the difficulty of training the fusion network (i.e., the neural network that fuses the camera frames and the radar frames), the camera branch (i.e., the neural network used to detect objects in the camera frames) is unlikely to be weakened. This is mainly due to the fact that there is no simple (e.g., affine) mapping / transformation from the camera frame to the bird's-eye view. This means that the radar branch (i.e., the neural network used to detect objects in the radar frames) will play a major role in the processing because it can be trained relatively easily. One way to solve this problem is to use an encoder-decoder network to transform the camera frames into frames that are more like radar frames. In this way, the feature maps of the camera frames will be similar to the corresponding radar frames. This allows the camera network to determine an output that is at least similar to the bird's-eye view of the radar network, where the final result of the fusion is expected. Alternatively, the camera branch can be pre-trained. In early camera-radar fusion training, the weights of this pre-trained network can be loaded (while being frozen) into the camera branch.
[0066] Note that the encoder side of an encoder-decoder network is the network that takes an input (e.g., a camera frame) and outputs a feature map / vector / tensor. The decoder side of an encoder-decoder network is the network (usually the same network structure as the encoder, but in the opposite direction) that takes a feature vector from the encoder and gives the closest match to the actual input or expected output (e.g., a camera frame like a radar frame).
[0067] One advantage of using an encoder-decoder network to take camera frames and make them similar to radar frames is that no annotations are required to train this new camera branch. The final result of this network (with pre-trained weights) can then be concatenated with the radar input and / or features and run through the final layers of the network.
[0068] Figure 7 An exemplary method 700 of fusing camera frames and radar frames to perform object detection in one or more spatial domains according to aspects of the present disclosure is shown. In one aspect, the method 700 may be performed by an onboard computer (eg, OBC 200) of a host vehicle (vehicle 100).
[0069] At 710, the OBC receives a plurality of camera frames from a camera sensor of the host vehicle (e.g., camera 212). In one aspect, operation 710 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206, any or all of which may be considered means for performing the operation.
[0070] At 720, the OBC receives a plurality of radar frames from a radar sensor of the host vehicle (e.g., radar 214). In one aspect, operation 720 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206, any or all of which may be considered means for performing the operation.
[0071] At 730, the OBC performs a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map. In one aspect, operation 730 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0072] At 740, the OBC performs a radar signature extraction process on a first radar frame of the plurality of radar frames to generate a first radar signature map. In one aspect, operation 740 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0073] At 750, the OBC converts the first camera signature map and / or the first radar signature map to the common spatial domain. In one aspect, operation 750 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0074] At 760, the OBC connects the first radar signature map and the first camera signature map to generate a first connected signature map in the common spatial domain. In one aspect, operation 760 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0075] At 770, the OBC detects one or more objects (eg, vehicles) in the first connectivity feature map. In one aspect, operation 770 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0076] Figure 8 An exemplary method 800 of fusing camera frames and radar frames to perform object detection in one or more spatial domains according to aspects of the present disclosure is shown. In one aspect, the method 800 may be performed by an onboard computer (eg, OBC 200) of a host vehicle (vehicle 100).
[0077] At 810, the OBC receives a plurality of camera frames from a camera sensor of the host vehicle (e.g., camera 212). In one aspect, operation 810 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206, any or all of which may be considered means for performing the operation.
[0078] At 820, the OBC receives a plurality of radar frames from a radar sensor of the host vehicle (e.g., radar 214). In one aspect, operation 820 may be performed by system interface(s) 210, data bus 208, memory 204, and / or processor(s) 206, any or all of which may be considered means for performing the operation.
[0079] At 830, the OBC applies an encoder-decoder network on the first camera frame to generate a first camera feature map in the spatial domain of the first radar frame. In one aspect, operation 830 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0080] At 840, the OBC combines the first radar frame and the first camera feature map to generate a first combined feature map in the spatial domain of the first radar frame. In one aspect, operation 840 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0081] At 850, the OBC detects one or more objects (eg, vehicles) in the first combined feature map. In one aspect, operation 850 may be performed by processor(s) 206, which may be considered a means for performing the operation.
[0082] It should be noted that although specific frequencies, integrated circuits (ICs), hardware, and other features are described in various aspects herein, alternative aspects may vary. That is, alternative aspects may utilize additional or alternative frequencies (e.g., frequency bands other than the 60 GHz and / or 28 GHz bands), antenna elements (e.g., arrays of antenna elements of different sizes / shapes), scanning cycles (including static and dynamic scanning cycles), electronic devices (e.g., mobile phones, tablet computers, personal computers (PCs), etc.), and / or other features. Those of ordinary skill in the art will understand these variations.
[0083] It should be understood that any reference to an element using indications such as "first", "second", etc. herein generally does not limit the number or order of these elements. On the contrary, these indications can be used as a convenient method to distinguish two or more elements or element instances in this article. Therefore, the reference to the first and second elements does not mean that only two elements can be used, or that the first element must be in some way before the second element. In addition, unless otherwise specified, a group of elements may include one or more elements. In addition, the term form "at least one of A, B or C" or "one or more of A, B or C" or "at least one of the group consisting of A, B and C" used in the specification or claims refers to "A or B or C or any combination of these elements". For example, this term may include A, or B, or C, or A and B, or A and C, or A and B and C, or 2A, or 2B, or 2C, etc.
[0084] In view of the above description and explanation, it will be appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the various aspects disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above according to their functions. Whether this function is implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the present disclosure.
[0085] Thus, it will be understood that, for example, a device or any component of a device may be configured (or made operable or adapted) to provide functionality as taught herein. This may be accomplished, for example, by: producing (e.g., manufacturing) the device or component so that it provides the functionality; by programming the device or component so that it provides the functionality; or by using some other suitable implementation technique. As an example, an integrated circuit may be manufactured to provide the necessary functionality. As another example, an integrated circuit may be manufactured to support the necessary functionality and then configured (e.g., by programming) to provide the necessary functionality. As yet another example, a processor circuit may execute code to provide the necessary functionality.
[0086] In addition, the methods, sequences and / or algorithms described in conjunction with the various aspects disclosed herein may be directly embodied in hardware, software modules executed by a processor, or a combination of the two. The software module may reside in a random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated into the processor (e.g., a cache memory).
[0087] It will also be appreciated, therefore, that, for example, certain aspects of the present disclosure may include a computer-readable medium embodying a method for personalized facial verification utilizing radar images.
[0088] Although the foregoing disclosure shows various illustrative aspects, it should be noted that various changes and modifications may be made to the examples shown without departing from the scope defined by the appended claims. The present disclosure is not intended to be limited to the specifically shown examples. For example, unless otherwise stated, the functions, steps and / or actions of the method claims according to the various aspects of the disclosure herein do not need to be performed in any particular order. In addition, although some aspects may be described or claimed in the singular, the plural is also contemplated unless a limitation to the singular is explicitly stated.
Claims
1. A method of performing early fusion of camera frames and radar frames to perform object detection in one or more spatial domains performed by an onboard computer of a host vehicle, comprising: receiving a plurality of camera frames from a camera sensor of the host vehicle; receiving a plurality of radar frames from a radar sensor of the host vehicle; performing a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map; performing a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map, wherein the first radar frame corresponds in time to the first camera frame; converting the first camera signature map, the first radar signature map, or both to a common spatial domain; Connecting the first radar feature map and the first camera feature map to generate a first connected feature map in a common spatial domain; performing object detection on the first connected feature map to detect one or more objects in the first connected feature map without performing object detection on the first camera feature map or the first radar feature map; as well as After the inverse perspective mapping, a width, a length, or both of the one or more objects in the bird's eye view are estimated based on a bounding box in the first camera frame encapsulating each of the one or more objects. The method of claim 1 , wherein the common spatial domain is a spatial domain of the radar sensor.
3. The method according to claim 1, wherein: Converting the first camera signature map, the first radar signature map, or both to the common spatial domain comprises converting the first camera signature map to the common spatial domain, and Converting the first camera feature map to the common spatial domain includes performing an explicit inverse perspective mapping transformation on the first camera feature map.
4. The method according to claim 1, wherein: Converting the first camera signature map, the first radar signature map, or both to the common spatial domain comprises converting the first camera signature map to the common spatial domain, and Converting the first camera feature map to the common spatial domain occurs during execution of the camera feature extraction process.
5. The method according to claim 1, further comprising: hashing a plurality of blocks of the first camera frame to identify one or more blocks that have not changed between a previous camera frame of the plurality of camera frames and the first camera frame; as well as The feature map values of the second camera feature map of the previous camera frame are copied to the corresponding feature map values of the first camera feature map. 6 . The method of claim 1 , wherein a width, a length, or both of the one or more objects is estimated based at least in part on a make, a model, or both of the one or more objects.
7. The method according to claim 1, further comprising: performing the camera feature extraction process on a second camera frame among the plurality of camera frames to generate a second camera feature map; performing the radar feature extraction process on a second radar frame among the plurality of radar frames to generate a second radar feature map; converting the second camera signature map to the common spatial domain to generate a converted camera signature map, converting the second radar signature map to the common spatial domain to generate a converted radar signature map, or both; as well as connecting the converted second radar feature map, the converted second camera feature map, or both to generate a second connected feature map, Wherein detecting the one or more objects is further based on the second connection feature map. 8 . The method of claim 1 , wherein the radar sensor and the camera sensor are collocated in a common housing in the host vehicle.
9. The method according to claim 1, further comprising: An autonomous driving operation is performed based on detecting the one or more objects.
10. The method of claim 9, wherein the autonomous driving operation is one or more of braking, accelerating, steering, adjusting cruise control settings, or signaling.
11. A method of performing early fusion of camera frames and radar frames to perform object detection in one or more spatial domains performed by an onboard computer of a host vehicle, comprising: receiving a plurality of camera frames from a camera sensor of the host vehicle; receiving a plurality of radar frames from a radar sensor of the host vehicle; applying an encoder-decoder network on a first camera frame to generate a first camera feature map in a spatial domain of a first radar frame, wherein the first radar frame corresponds in time to the first camera frame; combining the first radar frame and the first camera feature map to generate a first combined feature map in a spatial domain of the first radar frame; performing object detection on the first combined feature map to detect one or more objects in the first combined feature map without performing object detection on the first camera feature map; as well as After the inverse perspective mapping, a width, a length, or both of the one or more objects in the bird's eye view are estimated based on a bounding box in the first camera frame encapsulating each of the one or more objects.
12. The method according to claim 11, further comprising: The first combined feature map is provided to a neural network.
13. The method according to claim 11, further comprising: An autonomous driving operation is performed based on detecting the one or more objects.
14. The method of claim 13, wherein the autonomous driving operation is one or more of braking, accelerating, steering, adjusting cruise control settings, or signaling.
15. A vehicle-mounted computer of a host vehicle, comprising: At least one processor configured to: receiving a plurality of camera frames from a camera sensor of the host vehicle; receiving a plurality of radar frames from a radar sensor of the host vehicle; performing a camera feature extraction process on a first camera frame of the plurality of camera frames to generate a first camera feature map; performing a radar feature extraction process on a first radar frame of the plurality of radar frames to generate a first radar feature map, wherein the first radar frame corresponds in time to the first camera frame; converting the first camera signature map, the first radar signature map, or both to a common spatial domain; connecting the first radar feature map and the first camera feature map to generate a first connected feature map in the common spatial domain; performing object detection on the first connected feature map to detect one or more objects in the first connected feature map without performing object detection on the first camera feature map or the first radar feature map; as well as After the inverse perspective mapping, a width, a length, or both of the one or more objects in the bird's eye view are estimated based on a bounding box in the first camera frame encapsulating each of the one or more objects.
16. The onboard computer of claim 15, wherein the common spatial domain is a spatial domain of the radar sensor.
17. The vehicle-mounted computer according to claim 15, wherein: The at least one processor is configured to convert the first camera signature map, the first radar signature map, or both to the common spatial domain comprises converting the first camera signature map to the common spatial domain, and The at least one processor being configured to convert the first camera characteristic map to the common spatial domain includes the at least one processor being configured to perform an explicit inverse perspective mapping transformation on the first camera characteristic map.
18. The vehicle-mounted computer according to claim 15, wherein: The at least one processor is configured to convert the first camera signature map, the first radar signature map, or both to the common spatial domain comprises converting the first camera signature map to the common spatial domain, and The at least one processor is configured to convert the first camera feature map to the common spatial domain during execution of the camera feature extraction process.
19. The vehicle-mounted computer of claim 15, wherein the at least one processor is further configured to: hashing a plurality of blocks of the first camera frame to identify one or more blocks that have not changed between a previous camera frame of the plurality of camera frames and the first camera frame; and The feature map values of the second camera feature map of the previous camera frame are copied to the corresponding feature map values of the first camera feature map.
20. The onboard computer of claim 15, wherein a width, a length, or both of the one or more objects is estimated based at least in part on a make, a model, or both of the one or more objects.
21. The vehicle-mounted computer of claim 15, wherein the at least one processor is further configured to: performing the camera feature extraction process on a second camera frame among the plurality of camera frames to generate a second camera feature map; performing the radar feature extraction process on a second radar frame among the plurality of radar frames to generate a second radar feature map; converting the second camera signature map to the common spatial domain to generate a converted camera signature map, converting the second radar signature map to the common spatial domain to generate a converted radar signature map, or both; as well as connecting the converted second radar feature map, the converted second camera feature map, or both to generate a second connected feature map, Wherein detecting the one or more objects is further based on the second connection feature map.
22. The onboard computer of claim 15, wherein the radar sensor and the camera sensor are collocated in the host vehicle.
23. The vehicle-mounted computer of claim 15, wherein the at least one processor is further configured to: An autonomous driving operation is triggered based on detecting the one or more objects.
24. The onboard computer of claim 23, wherein the autonomous driving operation is one or more of braking, accelerating, steering, adjusting cruise control settings, or signaling.
25. A vehicle-mounted computer of a host vehicle, comprising: At least one processor configured to: receiving a plurality of camera frames from a camera sensor of the host vehicle; receiving a plurality of radar frames from a radar sensor of the host vehicle; applying an encoder-decoder network on a first camera frame to generate a first camera feature map in a spatial domain of a first radar frame, wherein the first radar frame corresponds in time to the first camera frame; combining the first radar frame and the first camera feature map to generate a first combined feature map in a spatial domain of the first radar frame; as well as performing object detection on the first combined feature map to detect one or more objects in the first combined feature map without performing object detection on the first camera feature map; as well as After the inverse perspective mapping, a width, a length, or both of the one or more objects in the bird's eye view are estimated based on a bounding box in the first camera frame encapsulating each of the one or more objects.
26. The vehicle-mounted computer of claim 25, wherein the at least one processor is further configured to: The first combined feature map is provided to a neural network.
27. The vehicle-mounted computer of claim 25, wherein the at least one processor is further configured to: An autonomous driving operation is triggered based on detecting the one or more objects.
28. The onboard computer of claim 27, wherein the autonomous driving operation is one or more of braking, accelerating, steering, adjusting cruise control settings, or signaling.
