Extracting features from sensor data

Abstract

In a first aspect, a computer-implemented method of fusing visual features obtained with multiple sensor modalities comprises extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non-camera sensor modality; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non- parallel to the camera image plane in 3D space; and computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, without applying monocular depth detection to the camera image. In a second aspect, the fused feature vectors are computed using a transformer neural network cross-attention between the camera feature vectors and the second feature vectors.

Description

EXTRACTING FEATURES FROM SENSOR DATA TECHNICAL FIELD

[0001] The present disclosure pertains generally to computer-implemented perception, and more particularly to methods and systems for extracting features from spatial sensor data (e.g., image data, lidar data, radar data etc.). Certain embodiments herein extend to extracting and fusing visual features from multi-sensor data (sensor data captured using multiple sensors) such as multi-modal sensor data (e.g., two or more of image data, lidar data, radar data etc.). BACKGROUND

[0002] The term herein “perception” refers to techniques for perceiving structure in spatial sensor data (e.g., camera image data, lidar data, radar data etc.). Such sensor data may be represented in a variety of formats, such as image formats (2D pixel arrays), voxel formats (3D voxel arrays), 2D or 3D point clouds, surface meshes etc. Sensor data may be converted between different formats (e.g. a lidar or radar point cloud of unstructured points may be ‘discretised’ to form a 2D image or 3D voxel representation). Examples of perception tasks include 2D or 3D bounding box detection, location detection, pose detection, motion detection etc. In the context of image processing, such techniques are often classed as “computer vision”, but the term perception encompasses a broader range of sensor modalities. Increasingly, machine learning (ML) is used to implement perception tasks to high accuracy and precision. Perception may be used in diverse fields, such as robotics, medical imaging etc. An increasingly relevant application is that of autonomous driving or mobile robotics more generally.

[0003] When performing perception tasks (such as object detection etc.) using different sensors, it may be desirable to bring together different features from the different sensors. However, with multiple sensors, different features typically exist in different spaces. For instance, with a sensor-equipped mobile robots, camera features may be obtained in perspective view (e.g. in a vertical camera image plane lying approximately vertically) while LiDAR / radar features are typically in a horizontal ‘bird’s-eye view’ (BEV) plane (that is, 'top down' space). The resulting view discrepancy poses a challenge for multi-modal perception since the same element in different feature tensors might correspond to different spatial locations within their respective spaces.

[0004] The term ‘fusion’ generally refers to a process or combining features from different sensors in a common space. The goal of multi-sensor fusion is to learn a unified representation of a scene derived from multiple views by different sensors. For example, given a camera image and a time-associated lidar point cloud captured with the above setup, a goal might be to determine a set of fused features in the BEV plane.

[0005] Recent state-of-the-art methods learn a unified representation in BEV space in the form of a BEV feature grid which is further processed by a ‘vanilla’ detection head to produce 3D detections. While the lidar representation can be naturally obtained in BEV, the multi-view cameras features are defined in their respective image planes which are scattered around the scene. In this context, a mechanism must then be introduced in order to embed these camera features onto a BEV grid, which can finally be fused together with the lidar features.

[0006] The transformation to BEV keeps both geometric structure (from LiDAR features) and semantic density (from camera features). On the one hand, the LiDAR-to-BEV projection flattens the sparse LiDAR features along the height dimension, thus does not create geometric distortion. On the other hand, camera-to-BEV projection casts each camera feature pixel back into a ray in the 3D space, which can result in a dense BEV feature map that retains full semantic information from the cameras.

[0007] A particular challenge with ‘mono’ camera data is the lack of depth data that would enable a straightforward projection of camera features into the BEV plane. One approach is to use ‘stereo’ imaging techniques to explicitly estimate pixel depths. However, this requires at least two cameras in a stereo imaging arrangement, with additional processing required to estimate the depth of each pixel in one image based on disparity between that pixel and a corresponding pixel in the other image.

[0008] An algorithm referred to as “LiftSplatShoot” (LSS) is the currently dominant method for transforming 2D camera features (without explicit depth information) into a different space, such as a BEV plane. In a “Lift” step, the 2D camera features are assigned a depth distribution over a fixed set of depth bins using a pixel depth estimator (referred to as ‘lifting’ 2D camera features into 3D space). The outer product of the depth distribution and the camera feature map results in a 3D pseudo-point cloud of camera features in 3D space. This point cloud is then projected onto the 2D BEV plane and all 3D camera features that projectinto a given cell in a BEV grid are summed in a “Splat” step to obtain the BEV features. Together, the ‘lift’ and ‘splat’ steps transform 2D camera features to 2D BEV space.

[0009] The LSS method is described in Philion et al., “Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D” which is incorporated herein by reference in its entirety (‘Philion’ hereinafter).

[0010] The principle of LSS is that, although the 2D camera features have unknown depths, a probability distribution over multiple possible depths (corresponding to multiple locations in 3D space) can be estimated for each 2D camera feature. Each 2D camera feature can then ‘lifted’ from 2D camera space into those multiple locations in 3D space, weighted by depth probability. The estimation of the depth distribution amounts to probabilistic mono depth estimation (mono depth estimation referring to the estimation of 2D camera feature in the absence of explicit depth data). SUMMARY

[0011] Combining complementary sensor modalities is important to providing robust perception for safety-critical robotics applications such as autonomous driving (AD). For example, recent state-of-the-art sensor modality fusion methods for AD rely on monocular depth estimation which is a notoriously difficult task compared to using depth information from the sensor directly. Similar challenges arise when fusing camera with other non-camera modalities, such as radar.

[0012] The inventors have found that, surprisingly, the accuracy of the mono depth detection has limited impact on overall performance in such contexts. This conclusion is supported by experimental results detailed herein below. The inventors’ experiments show that state-of- the-art fusion techniques based on monocular depth detection do not leverage depth as expected and that naively improving depth estimation does not, in fact, lead to improvements in downstream performance. Surprisingly, the experiments demonstrate that removing depth estimation altogether does not materially degrade performance of tasks such as object detection applied to fused sensor data.

[0013] Informed by the results of the experiments, various aspects herein leverage the insight that monocular depth is an unnecessary architectural bottleneck during camera and non- camera fusion.

[0014] A first aspect herein provides a computer-implemented method of fusing visual features obtained with multiple sensor modalities, the method comprising: extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non- camera sensor modality, the sensor data being time-associated with the camera image; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non-parallel to the camera image plane in 3D space; and based on the camera feature vectors and the second feature vectors, computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, without applying monocular depth detection to the camera image.

[0015] In embodiments, computing the plurality of fused feature vectors may comprise performing, using a transformer neural network, cross-attention between the plurality of camera feature vectors and the plurality of second feature vectors.

[0016] A second aspect herein provides a computer-implemented method of fusing visual features obtained with multiple sensor modalities, the method comprising: extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non- camera sensor modality, the sensor data being time-associated with the camera image; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non-parallel to the camera image plane in 3D space; and based on the camera feature vectors and the second feature vectors, computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, wherein computing the plurality of fused feature vectors comprises performing, using a transformer neural network cross-attention between the plurality of camera feature vectors and the plurality of second feature vectors.

[0017] In embodiments, a plurality of intermediary fused feature vectors in an intermediary plane may be determined based on a projection of the second feature vectors from the target plane into the intermediary plane. The plurality of fused feature vectors may be computed by projecting the plurality of intermediary fused feature vectors into the target plane.

[0018] The intermediary plane may be a projected camera horizon plane defined by one or more predetermined camera properties associated with the camera image.

[0019] The cross-attention may be performed by: projecting the second feature vectors into a plurality of frustum portions in the intermediary plane, each frustum portion corresponding to a camera feature column lying perpendicular to the intermediary plane and containing a subset of the camera feature vectors in the camera image plane; and inputting to the transformer neural network each camera feature column in association with the corresponding frustum portion.

[0020] The second feature vectors may be projected onto the intermediary plane based on bilinearly sampling a grid of the target plane at the locations of the projected centres of cells of a grid of the intermediary plane; and the fused feature vectors may be projected onto the target plane based on bilinearly sampling the grid of the target plane at the locations of the projected centres of cells of the grid of the intermediary plane.

[0021] An encoder-decoder may be used to compute the fused feature vector at each intermediary plane location based on the camera feature vector at the intermediary plane location and the second feature vector at the intermediary plane location.

[0022] The transformer neural network may comprise the encoder-decoder wherein each camera feature column may be inputted to an encoder of the transformer neural network, and passed to a decoder of the transformer neural network, wherein the decoder may use associated frustum portion to attend to the camera features.

[0023] A third aspect herein provides a computer-implemented method of extracting, in a horizon plane, camera features from a camera image, the method comprising: extracting from the camera image using a visual feature extractor a plurality of camera feature vectors in a camera image plane, the camera image plane lying non-parallel to the horizon plane in 3D space; aggregating the plurality of camera feature vectors along a height dimension in the camera image plane, the height dimension extending perpendicular to the horizon plane in 3D space, resulting in a plurality of aggregate feature vectors at respective camera image plane locations in the camera image plane; and populating a plurality of horizon plane locations in the horizon plane with respective camera features based on the plurality of aggregate feature vectors and a predetermined geometric relationship between each horizon plane location and the respective camera image plane locations.

[0024] In embodiments, aggregating the plurality of camera feature vectors may comprise determining a plurality of importance-weighted camera feature vectors, by weighting theplurality of camera feature vectors by respective importance weights, and applying an aggregation function to the importance-weighted camera feature vectors.

[0025] Each horizon plane location may be pre-associated with a first of the camera image plane locations and a second of the camera image plane locations, and may be populated with the visual features in the form of an interpolated camera feature vector based on: a first of the aggregate feature vectors at the first camera image plane location, a first predetermined distance between the first camera image plane location and the horizon plane location projected into the camera image plane, a second of the aggregate feature vectors at the second of camera image plane location, and a second predetermined distance between the second camera image plane location and the horizon plane location projected into the camera image plane.

[0026] The respective camera image plane locations may be arranged in a first uniform grid within the camera image plane.

[0027] The camera image plane may lie perpendicular to the horizon plane in 3D space such that the height dimension is aligned with a horizontal or vertical dimension of the first uniform grid.

[0028] The plurality of horizon plane locations may be arranged in a second uniform grid within the horizon plane.

[0029] A perception component may be applied to the fused visual features to compute a perception output.

[0030] The plurality of fused feature vectors may be computed based on an assumption that each camera plane location corresponds to multiple depth locations in 3D space with equal weight.

[0031] A fourth aspect herein provides a computer system comprising one or more computers configured to implement the method of any above aspect or embodiment.

[0032] A fifth aspect herein provides transitory or non-transitory media embodying computer-readable instructions configured when executed by one or more processors to cause the one or more processors to implement the method of any above aspect or embodiment.

[0033] Note, ‘height’ in this context is an arbitrary label that is simply used to a dimension perpendicular to the horizon plane. BRIEF DESCRIPTION OF FIGURES

[0034] Particular embodiments will now be described, by way of example only, with reference to the following schematic figures, in which:

[0035] Figure 1 shows a schematic block diagram of a multi-sensor fusion system.

[0036] Figure 2 shows the LiftSplat projection, taken from LiftSplatShoot (LSS), with the “shoot” omitted.

[0037] Figure 3 shows results demonstrating the impact of the quality of the monocular depth prediction on the object detection performance of BEVFusion on the nuScenes val set.

[0038] Figure 4 shows depth maps obtained with various depth prediction models.

[0039] Figure 5 shows the horizontal line passing through the centre of the image and the plane corresponding to its projection in 3D, the projected horizon of the camera.

[0040] Figure 6 shows certain geometric principles underpinning a ‘LiftAttendSplat’ approach to camera-lidar fusion in a first embodiment.

[0041] Figure 7 shows an example transformer encoder-decoder.

[0042] Figure 8 illustrates certain principles behind a ‘SwiftBlat’ method in a second embodiment.

[0043] Figure 9 shows how the column dimension of a camera feature map may be reduced to a single row following feature modulation in the SwiftBlat method.

[0044] Figure 10 shows an additional step in the SwiftBlat method for a camera which is not axis aligned.

[0045] Figure 11 shows an example camera-to-BEV projection using the SwiftBlat method for two example points on a camera image.

[0046] Figure 12 illustrates the processing of the camera image in the SwiftBlat method.

[0047] Figure 13 shows the aggregation of camera features in the height dimension in the SwiftBlat method.

[0048] Figure 14 shows the method of interpolating grid cell values during projection in the SwiftBlat method.

[0049] Figure 15 shows a flow chart outlining the steps in the SwiftBlat method for camera- to-BEV projection.

[0050] Figure 16 shows object detection performance measured using mAP for objects at different distances from the ego and of different sizes.

[0051] Figure 17 shows a visualisation of where camera features of ground-truth objects are projected onto the BEV grid for the present method compared to BEVFusion.

[0052] Figure 18 shows activations in BEV space derived by summing up feature maps along the channel dimension.

[0053] Figure 19 shows a comparison of saliency maps for the method of the present invention compared to BEVFusion.

[0054] Figure 20 shows a comparison of saliency maps for the method of the present invention compared to BEVFusion.

[0055] Figure 21 shows examples of the weight of projected camera features onto the BEV space.

[0056] Figure 22 shows a camera image with annotations compared to the predictions of the model of the present invention.

[0057] Figure 23 shows the distribution of the L2 velocity errors for each class in the nuScenes validation set. DETAILED DESCRIPTION

[0058] Integrating information from different modalities efficiently and effectively is especially important in safety-critical applications such as autonomous driving, where different sensor modalities are complementary and combining them adequately is crucial toguarantee safety. For example, cameras capture rich semantic information of objects up to far away distances, while lidars provide extremely accurate depth information but are sparse at large distances. For this reason, many modern self-driving platforms have a large number of different sensors which must be combined in order to provide accurate and reliable perception of the surrounding scene and allow safe deployment of these vehicles in the real world.

[0059] Multimodal sensor fusion — learning a unified representation of a scene derived from multiple sensors — offers a plausible solution to this problem. However, training such multimodal models can be challenging, especially when modalities are as different as cameras (RGB images) and lidars (3D point clouds). For instance, it is known that different modalities overfit and generalise at different rates and that training all modalities jointly can lead to underutilisation of the weaker modalities and [Ref.57] even to inferior results compared to unimodal models in some situations [Ref.40].

[0060] In the context of autonomous driving, many of the recent state-of-the-art methods for camera-lidar fusion [Ref.16, 31, 36] are based on the Lift-Splat (LS) paradigm [Ref.41]. The “shoot” component of “Lift, Splat, Shoot” [Ref.41] relates to trajectory prediction and is not considered here. In this approach, the camera features are projected in bird’s-eye-view (BEV) — or top-down space — using monocular depth before being fused with the lidar features. As a result, the location of the camera features in BEV is highly dependent on the quality of the monocular depth prediction and it has been argued that its accuracy is critical [Ref.16, 31]. This prevailing view is challenged and it is shown that the monocular depth prediction inside these models is of poor quality and cannot account for their success. In particular, results showing that methods based on Lift-Splat perform equally well when the monocular depth prediction is replaced by direct depth estimation from the lidar point cloud or removed completely, are presented. This leads to the suggestion that relying on monocular depth when fusing camera and lidar features is an unnecessary architectural bottleneck and that Lift-Splat could be replaced by a more effective projection.

[0061] Most 3D object detection benchmarks are dominated by methods using lidar point clouds due to their highly accurate range measurement allowing for better placement of objects in 3D compared to methods using cameras or radars only. Deep learning methods for classification on point clouds were pioneered in the seminal works of [Ref.43, 44] and early works have been applying similar ideas to 3D object detection [Ref 45, 48]. A more recentfamily of methods is based on direct voxelisation of the 3D space [Ref.64, 70] or compression of the lidar representation along the z-direction into “pillars” [Ref.25, 65]. These approaches have been very successful and are the basis of many follow-up works [Ref. 17, 22, 67].

[0062] The task of 3D object detection has also been tackled from multiple cameras alone. Early works have mostly been based on various two-stage approaches [Ref.4, 23, 45, 58], while recent methods have been leveraging monocular depth estimation directly [Ref.3, 24, 46]. This task is difficult when lidar is absent because 3D information must be estimated using images only, which is a challenging problem. However, recent works have shown impressive performance by borrowing ideas from lidar detection pipelines [Ref.8, 12, 18], by improving position embeddings [Ref.34] and 3D queries [Ref.20], as well as by leveraging temporal aggregation [Ref.13, 28, 33, 35, 56, 72] or 2D semantic segmentation [Ref.69].

[0063] Perception quality can be improved by jointly leveraging cameras and lidars when available. Recent fusion methods can be broadly classified into three categories: point decoration methods, methods that leverage task-specific object queries and architectures, and projection-based methods. Point decoration methods augment the lidar point cloud using semantic segmentation data, camera features [Ref.54], or even create new 3D points using object detections in the image plane [Ref.68]. Such methods are relatively easy to implement but suffer from the fact that they require lidar points to fuse camera features. TransFusion [Ref.1] is a recent example of a method that leverages task-specific object queries generated using the lidar point cloud. Final detections are made directly without explicit projection of camera features into BEV space. Similar methods utilising two-way modality interactions perform even better [ref.62, 66]. Fusion can also be performed earlier in the model, for example at the level of the 3D voxels [Ref.6, 7], or by sharing information between the camera and lidar backbones [Ref.19, 29, 42] MSMDFusion [Ref.21] fuses camera and lidar features at multiple scales using lidar points to estimate the 3D position of camera features, while UniTR [Ref.55] pre-assigns depth to each camera feature. FUTR3D [Ref.5] fuses features by selecting modality-agnostic 3D reference points. Finally, projection-based methods project camera features into 3D before fusing them with the lidar (see below).

[0064] Of most interest are camera-lidar fusion methods based on projecting camera features into 3D. Many recent state-of-the-art camera-lidar fusion methods [Ref.16, 31, 36] project camera features in 3D using monocular depth estimation even though depth information isavailable from the lidar. It has also been shown that the projection method is less important than other aspects of training in the camera-only setting [Ref.14]. It was shown in [Ref.16] that the performance of [Ref.30, 36] can be boosted significantly by including well-crafted features derived from the ground-truth lidar depth map into the cam era stream prior to monocular depth estimation. These approaches are relatively simple yet powerful and rank among the top entries in the nuScenes leaderboard [2]. Monocular 146 depth estimation, however, is a difficult task and it is shown that predicting depth from cameras alone introduces an architectural bottleneck when lidar is available. An alternative approach is to project camera features directly into BEV space using the known correspondence between lidar points and camera features [Ref.9, 26, 59]. However, the sparsity of the lidar point cloud can limit which camera features are projected, as described in [Ref.36]. Finally, learning to project camera features in BEV without explicit depth can be achieved when lidar is absent using a transformer, as shown in [Ref 28, 47]. In the present invention, all camera features are considered for projection, alleviating the lidar sparsity issue. Furthermore, the network is not constrained to predict monocular depth, allowing it more flexibility in choosing where best to project camera features in BEV.

[0065] An algorithm referred to as “LiftSplatShoot” (LSS) is the currently dominant method for transforming 2D camera features (without explicit depth information) into a different space, such as a BEV plane. In a “Lift” step, the 2D camera features are assigned a depth distribution over a fixed set of depth bins using a pixel depth estimator (referred to as ‘lifting’ 2D camera features into 3D space). The outer product of the depth distribution and the camera feature map results in a 3D pseudo-point cloud of camera features in 3D space. This point cloud is then projected onto the 2D BEV plane and all 3D camera features that project into a given cell in a BEV grid are summed in a “Splat” step to obtain the BEV features. Togher, the ‘lift’ and ‘splat’ steps transform 2D camera features to 2D BEV space.

[0066] The LSS method is described in Philion et al., “Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D” which is incorporated herein by reference in its entirety (‘Philion’ hereinafter).

[0067] The principle of LSS is that, although the 2D camera features have unknown depths, a probability distribution over multiple possible depths (corresponding to multiple locations in 3D space) can be estimated for each 2D camera feature. Each 2D camera feature can then ‘lifted’ from 2D camera space into those multiple locations in 3D space, weighted by depthprobability. The estimation of the depth distribution amounts to probabilistic mono depth estimation (mono depth estimation referring to the estimation of 2D camera feature in the absence of explicit depth data)

[0068] One application of the LSS camera-to-BEV projection steps is to fuse camera features in 2D BEV space with LiDAR features in 2D BEV space. 2D BEV lidar features may be obtained in a LiDAR-to-BEV step within a camera-LiDAR fusion framework. In this context, the LSS method produces a useful set of features ^′^^^^^^∊× ^ × ^in an ^ × ^ birds eye view (BEV) grid from a set of camera features∊ ℝ^ × ^ × ^, where ^, ^ are the height and width of the camera image feature map respectively.

[0069] As discussed, the conventional LSS method leverages probabilistic mono depth estimation as part of the 2D camera feature transformation. However, experimental results detailed herein demonstrate that camera-lidar fusion methods based on the “LiftSplat” paradigm are not leveraging depth as expected. In particular, it is shown that such methods perform equivalently or better if monocular depth prediction is removed completely.

[0070] Expanding on the above, whilst the LSS method takes depth of camera features as input, it is found that the role of LSS in camera-LiDAR fusion is misunderstood. It does seem to predict depth but not very accurately. Experiments conducted with pretrained depth models show no positive correlation between depth prediction accuracy and final 3D detection accuracy. Additionally, when replacing predicted depth distributions with a uniform distribution of 1s in LSS training, experiments show no drop in the final precision. Motivated by this observation, a new methodology is disclosed herein that does not require any explicit depth prediction on camera features to be performed and, therefore, offers a much improved trade-off between efficiency and inference accuracy.

[0071] A novel fusion method is described. The method that bypasses monocular depth estimation altogether and instead selects and fuses camera and lidar features in a bird’s-eye- view grid using a simple attention mechanism. It is shown that the model can modulate its use of camera features based on the availability of lidar features and that it yields better 3D object detection on the nuScenes dataset than baselines relying on monocular depth estimation.

[0072] There is described methods and systems for fusing visual features obtained with multiple sensor modalities using cross-attention in a transformer neural network between camera features and features of another sensor modality (such as lidar or radar).

[0073] A novel approach is described for camera-lidar fusion called “Lift-Attend-Splat” that bypasses monocular depth estimation altogether and instead selects and fuses camera and lidar features in BEV using a simple transformer. Evidence is presented showing that the present method produces better camera utilisation compared to the methods based on monocular depth estimation and that it improves object detection performance.

[0074] It is shown that camera-lidar fusion methods based on the Lift-Splat paradigm are not leveraging depth as expected. In particular, it is shown that they perform equivalently or better if monocular depth prediction is removed completely.

[0075] A novel camera-lidar fusion method that fuses camera and lidar features in BEV is described using a simple attention mechanism. It is shown that it leads to better camera utilisation and improves 3D object detection compared to models based on the Lift-Splat paradigm.

[0076] By way of example, two embodiments of the new methodology are described, referred to respectively as “SwiftBlat” (SB) and “LiftAttendSplat” (LAS). Each method provides a concrete improvement over the “lift-splat” steps of the conventional LSS method.

[0077] Figure 1 shows a highly schematic camera-LiDAR fusion framework where two separate streams are used to separately extract features and transform them into the same BEV space: i) camera view features 101 are encoded to generate camera BEV features 102; LiDAR BEV features 104 are extracted from LiDAR point clouds 103. Then a fusion module 105 integrates the BEV features from the two modalities. Downstream tasks may be performed based on the fused BEV features 106.

[0078] As noted, in the conventional LSS method, the transformation from the camera images 101 to the camera BEV features prior to fusion involves an expensive lifting of 2D camera features into 3D space based on probabilistic mono depth estimation.

[0079] The SwiftBlat method falls within the fusion framework of Figure 1, but avoids the need for mono depth estimation. In a preferred implementation of SwiftBlat, a 2D array ofcamera features is ‘collapsed’ into a 1D array of camera features prior to lifting, by aggregating 2D camera features along a height dimension perpendicular (or approximately perpendicular) to the BEV plane. The 1D array of camera features can, in turn, be directly mapped to a 2D grid of BEV location (essentially combining the lift and splat steps in a single transformation). This aggregation prior to lifting is only possible because of the assumption of a uniform depth distribution for each 2D camera feature. A specific benefit of the SB method is improved computational efficiency compared with the lift-splat steps of LSS. In practical fusion settings, the LSS transformation from 2D camera feature space to another space (such as BEV space) in LSS can become a computational bottleneck.

[0080] The SwiftBlat method is motivated by the findings concerning mono depth detection in LSS. In the SwiftBlat method, a simplified 2D camera feature transformation (from camera space to some other target space) is provided, based on an assumption that each 2D camera feature corresponding to multiple depth locations in 3D space with equal weight (that is, with uniform depth weightings). This transformation can, in turn, be implemented more efficiently than the equivalent ‘lift-splat’ steps of LSS or similar methodologies.

[0081] The LAS method also broadly falls within the framework of Figure 1, but with an additional dependency arising between the camera BEV features 102 and the lidar BEV features 104. As described in more detail below, a transformer neural network architecture is used to derive the BEV camera features 102, which involves attending over 2D camera features based on the BEV lidar features 104 in an intermediary projected horizon plane. Hence the camera BEV features 102 computed prior to fusion are dependent on the lidar BEV features 104. The LAS method is comparable in cost to the LSS method but achieves better performance.

[0082] The SB and LAS methods are described in detail below. First, to provide relevant context, further details of the lift-splat steps of the conventional LSS method are described.

[0083] Whilst the following examples consider fusion of 2D camera features with lidar features in a BEV plane, the techniques can be applied with other sensor modalities, such as radar (e.g., with a radar point cloud substituted for the lidar point cloud). Moreover, the techniques can be used to transform 2D camera features from a 2D camera plane into any other target plane lying non-parallel to the camera plane in 3D space, and the BEV plane is merely one example of such a target plane. The technique can be used to fuse 2D camera features with features derived from one or more non-camera modalities (such as lidar, radaretc.) from camera and non-camera sensor data captured at the same or similar times Alternatively or additionally, the techniques may be used to fuse 2D camera features across different cameras, by transforming 2D camera features from one or more of those cameras into a common target plane (e.g. the camera plane of another camera).

[0084] Recent camera-lidar fusion methods based on the Lift-Splat paradigm [Ref.31, 36] learn a unified representation in the form of a BEV grid by projecting camera features in BEV space. The resulting feature map is then merged with the lidar features using concatenation [Ref.36]. In this paradigm, the monocular depth distribution prediction is learned indirectly from the downstream task without explicit depth supervision.

[0085] As mentioned above, the LiftSplat projection is the dominant method for lifting 2D camera features into 3D space. Figure 2 shows the LiftSplat projection, taken from LiftSplatShoot (LSS), with the “shoot” omitted (the ‘shoot’ step of LSS related to a particular end-to-end motion planning application, which is not directly relevant to the novel methodology described herein). The LSS method uses monocular depth estimation and takes as input a number of images 201-206 and their corresponding extrinsic and intrinsic parameters.

[0086] As described in further detail in Philion, a pinhole camera model is assumed. Each ray projecting outside of the camera feature map at the location of its individual pixels is divided into a predetermined set of bins. State-of-the-art methods [Ref.36, 31, 16] learn a unified representation in the form of a BEV grid by projecting camera features, where ^ indexes the cameras, along their corresponding ray into a predetermined set of depth bins.

[0087] In the LiftSplat projection in Figure 2 the camera features 210,are passed through some convolutional layers 220 to generate the image context vectors 230 , and the depth classification weightswith ^^being the number of predetermined depth bins. Depth ^^is normalised in the ^^dimension using the softmax function and can be interpreted as the probability of a given camera feature 210 existing at a particular depth ^^for ^ ∈ [0, ^^) along its ray. The ^^are hyperparametrs of the model.

[0088] Camera features are then “lifted” onto a 3D point cloud along the corresponding ray using these bins.

[0089] The outer productis a pointcloud 240 of weighted camera features 230 in 3D space, with (^, ^, ^) positions given by the camera intrinsics, depth bin ^^and (^, ^) index in the feature map. Thus, in the “Lift” step, a frustum-shaped pointcloud 240 is generated for each individual image. The extrinsics and intrinsics are then used to “Splat” each frustum onto the BEV ^ = 0 plane.

[0090] Camera features are then “splatted” onto the BEV grid by projecting each point downwards onto the road plane.

[0091] The pointcloud is projected downwards onto the ^ = 0 BEV plane and points in the same BEV grid cell reduced using ∑ or max to yield the BEV projected image features 250, ^′^^^^^^.

[0092] The estimated depth distribution can be written as Proj^^^^^^^^^^= Splat(^′^^^⊗ ^), (1) where ^′^^^∈ ℝ^^^×^×^a context vector obtained from the camera features ^^^^∈ ℝ^^×^×^, ^ ∈ ℝ^^×^×^is a normalised distribution over predetermined depth bins and Splat denotes the operation of projecting each point downwards into the ^ = 0 plane, see for details.

[0093] An example overall projection can be written aswhere Binz=0 denotes the operation of binning the lifted point clouds onto the BEV grid. Points in the same BEV grid cell are summed over. This grid is then merged with the lidar featuresgivewhere Proj is the above projection and Merge denotes the mechanism by which the lidar and camera BEV grids are merged together. This module varies slightly from method to method,e.g. simple concatenation in [Ref.36] and gated attention in [Ref.31]. Note that the projection depends on the lidar features but also implicitly on the camera location and properties.

[0094] As a result, the location of the camera features in BEV is highly dependent on the quality of the monocular depth prediction and the accuracy of the monocular depth is thought to be critical to the success of this approach.

[0095] Many of the recent state-of-the-art methods for camera-lidar fusion [Ref.36, 31.16] are based on this “LiftSplat” (LS) paradigm and many have argued that the quality of the monocular depth prediction is crucial to the success of these approaches. This prevailing view is challenged and it is shown that the monocular depth prediction inside these models is often poor and cannot account for their success. In particular, results are presented indicating that these methods perform similarly or better when the “lift” step uses the lidar point cloud directly instead of the monocular depth prediction, or when the monocular depth prediction is removed altogether.

[0096] Experiments show that replacing predicted depth distributions with a uniform distribution of 1s in LSS training (LSS(1)) gives no drop in the final precision.

[0097] In order to understand how the accuracy of the monocular depth prediction impacts the detection performance of these methods, the depth predicted by BEV Fusion [Ref 36] is analysed.

[0098] The impact of the quality of the monocular depth prediction on the object detection performance of BEVFusion is evaluated on the nuScenes val set. Both are compared qualitatively and quantitatively, and the quality of the monocular depth predicted by BEV Fusion is compared to both the lidar depth maps and to models fine-tuned with an additional depth supervision loss at various values of the weight λ (supervised). Models with a pretrained depth classifier (pretrained) and models using the lidar ground truth directly (lidar) are also considered. Finally, results that use the simplified camera BEV projection that projects camera features to all depths uniformly (no depth) are also shown. In experiments, more accurate depth does not translate to better detection performance and the original model is on-par with using the lidar points directly as a source of depth. The model with the best detection performance overall is the no depth model, clearly indicating that accurate monocular depth is not necessary for BEVFusion [6] to achieve its performance.

[0099] All experiments are performed using the method presented in [Ref.36] and use the original repository. The depth classifier is supervised by introducing the following loss alongside the original detection losses,which is a cross-entropy loss between the LiDAR depth distribution and predicted depth distribution, taken over all cells for which the LiDAR depth is available.is the normalised predicted depth distribution from the LiftSplat model for the nth cell, 1n is the one-hot encoded LiDAR depth distribution for the nth cell and λ is a weight controlling the strength of the depth supervision loss with respect to the detection losses.

[0100] Experiments with pretraining the depth estimation module within LiftSplat are also performed. First, the camera stream in [Ref.36]is trained supervising only the depth distribution using the LiDAR depth map. During this pretraining, the entire camera pipeline is unfrozen. After the depth predictor has been pretrained, the lidar stream is added and the model is trained end-to-end as in [Ref.36], with no depth supervision (λ = 0) and all modules unfrozen.

[0101] The quality of the depth distribution predicted by BEVFusion is analysed by comparing its mean value to lidar depth maps, both qualitatively and quantitatively using the absolute relative (Abs. Rel.) and root mean squared errors (RMSE).

[0102] Figure 3 shows the impact of the quality of the monocular depth prediction on the object detection performance of BEVFusion [Ref.36] on the nuScenes val set.

[0103] The monocular depth maps shown on Figure 3 are generated by calculating the weighted average of the depth bins by the depth distribution as . This depth map is constrained by construction torange is mapped onto the Turbo colour map for visualisation. The LiDAR depth map is similarly colourised, but with values calculated by projecting the 3D LiDAR point cloud into the image plane of every camera and min-pooling the depths of the LiDAR points within each camera feature cell. In general the LiDAR yields sparse depth maps and cells for which there is no depth information from the LiDAR are visualised in grey. Furthermore, this process will yieldcertain cells in the LiDAR depth maps to have values outside the range covered by the depth bins. These cells are considered to be invalid and coloured grey and discarded. For each camera image i ground depth distributionsare computed by projecting the 3D lidar point cloud onto the image plane and binning each point onto the pixels of the feature map. For non-empty cells, the depth is chosen to be the minimum distance (from the camera plane) of all the points in the cell, while the depth is set as non-specified for empty cells and those for which the minimum yields a depth value which is outside the range of the model’s depth bins. Finally, the depth distribution is calculated as the one-hot encoding of this minimum distance according to buckets defined by the model’s depth bins, a hyperparameter.

[0104] In Figure 3, it is seen that the BEV fusion depth map 301 is of poor quality compared to the LiDAR depth maps 302 and 303, despite giving state of the art fusion performance. This is highly surprising because the model has been structured to benefit from accurate monocular depth and one would therefore expect it to learn fairly robust depth, even without direct supervision. The discrepancy of the monocular depth compared to the LiDAR depth is characterised more quantitively using the relative error (Abs. Rel.) and the root mean squared error metrics, which are popular monocular depth estimation metrics and show the present results in the table below.

[0105] As shown in Figure 3, the mean depth prediction does not accurately reflect the structure of the scene and is markedly different from the lidar depth map which suggests that monocular depth is not leveraged as expected in [Ref 36].

[0106] The results reveal that the difference between the learnt depth and the LiDAR depth is almost three times as large as the LiDAR depth itself, and further that the root-mean-squared difference in depth is on average over 18m. It is to be noted that using LiDAR point clouds as a source of depth can be problematic in the context of sensor fusion if sensors have slightly different viewpoints, but it is not expected that these effects will be prominent here.

[0107] It is investigated whether improving the depth prediction quality can boost object detection performance. To do so, a supplementary depth supervision objective is introduced with the LiDAR depth map as a target using a simple cross-entropy loss with weight λ and optimise this loss jointly with the detection loss. To do so, the model from [Ref.36] is retrained with the following loss:where ^^^^is the original 3D object detection loss and ^^^^^^is a simple cross-entropy loss for the depth estimation that uses one-hot encoded lidar depth as a target. By changing the hyper-parameter ^, the quality of the depth prediction can be controlled and its impact detection performance can be explored.

[0108] In Table 1, it can be seen that while depth supervision indeed leads to much more accurate depth maps both visually and quantitatively, detection performance — measured using mean average precision (mAP) — degrades from the baseline as the weight of the depth supervision is increased. This suggests that the method is unable to take advantage of more accurate depth prediction. Experiment where the depth module is pretrained are also conducted. Since training on the multi-task loss Eq. (7) is likely to degrade object detection performance at high values of ^, experiments are also conducted with two more variants: (i) pretraining the depth supervision module separately and (ii) using the lidar point cloud directly to bypass the depth supervision module altogether.

[0109] The result in Table 1 shows that pretraining leads to more accurate depth prediction but degrades detection performance relative to the baseline, while using the lidar directly does not change the detection performance compared to the baseline, even though all depth metrics are close to zero. They are not exactly zero because of the depth quantisation introduced by the one-hot encoding of the lidar depth.

[0110] Finally, the LiDAR point cloud is used directly in place of the monocular depth in order to compute the depth classification weights D. This is similar to earlier works. As can be seen from the results table above, detection performance remains unchanged compared to the baseline even though all depth metrics are close to zero. (They are not exactly zero because of the depth quantisation introduced by the one-hot encoding).

[0111] This leads to the hypothesis that, contrary to common beliefs, accurate monocular depth is not leveraged in fusion paradigms based on the “LiftSplat” projection method.

[0112] To test this, the monocular depth prediction is removed completely and the projection (1) is replaced by Proj^^^^^^^^= Splat(^′^^^⊗ 1), (3)orwhere the tensor of the same shape as Diwith all entries equal to 1 is denoted by 1. This projection projects the camera features to all depths uniformly. Strikingly, it can be seen in Table 1 that this projection leads to a slight improvement in detection performance compared to the original LS projection, showing that accurate depth estimation is not necessary in these methods in order to achieve state of the art fusion performance.

[0113] This finding appears contrary to previous work within the camera-only paradigm which show that accurate depth projection is crucial for BEV projection. However, the present experiments do not contradict this prevailing view, and instead suggest that they do not extend to the camera-LiDAR fusion paradigm as might have otherwise been assumed.

[0114] A possible explanation is that the importance of monocular depth is greatly diminished when lidar features are available since lidar is a much more precise source of depth information and the model is able to easily suppress camera features projected at the wrong location. This suggests that relying on monocular depth estimation could be an unnecessary architectural bottleneck and lead to underutilisation of the camera. The same rationale applies to other modalities with depth information, such as radar (e.g. where depth information is captured in radar return times).

[0115] Leveraging the insight that non-camera features with better depth information (e.g. LiDAR, radar etc.) remove the need for accurate depth, the projection is modified to inject non-camera information (e.g., LiDAR, radar etc.) earlier in the pipeline, yielding better results. As accurate depth estimation is not needed, deployments of these detection models can be optimised for better runtime performance and speed by switching the expensive psuedo-pointcloud generation for the Projno-depth described above. Table 1: Impact of the quality of the monocular depth prediction on the object detection performance of BEVFusion [Ref.36] on the nuScenes validation set. BEVFusion and four different variants are compared: adding depth supervision using Eq. (2) with various weights λ, using lidar depth maps instead of monocular depth estimation (lidar), using a pretrained and frozen depth classifier (pretrained), and finally removing depth estimation altogether byprojecting camera features at all depths uniformly using Eq. (3) (uniform depth). In the present experiments, more accurate depth does not translate to better detection performance and the original model is on-par with using the lidar points directly as a source of depth. Equivalent detection performance was achieved using the uniform depth model, clearly indicating that accurate monocular depth is not necessary for BEVFusion [Ref.36] to achieve its performance. Abs. Rel. ↓ RMSE ↓ mAP ↑ BEVFusion 2.75 17.40 68.5 [Ref.30] BEVFusion [Ref.36] w / Eq. (2): ^ = 0 2.83 18.54 68.4 ^= 0.010.76 8.09 68.0 ^= 10.22 4.77 68.1 ^= 1000.16 4.55 64.6 Lidar 0.04 0.29 68.4 Pretrained 0.64 7.87 67.4 Uniform – – 68.5 depth

[0116] The performance of the depth classifier is evaluated using five of the metrics proposed in [Ref.10]: root mean squared error (RMSE), root mean squared logarithmic error (RMSLE), mean absolute relative error (Abs. Rel.), mean squared relative error (Sq. Rel.) and fraction outside 125% (Frac.125). All the metrics for 2 different methods of translating the classification output into a depth map are shown: “mode” — in which the bin with maximum probability is used, and “mean” — where the predicted depth is the weighted average of all the bins. These quantities are averaged over all predictions made by the depth classifier for which ground truth is available to compute the metrics. Camera feature cells for which lidar depth is unspecified are ignored. BEVFusion is compared to four different variants: adding depth supervision using Eq. (2) with various weights ^, using lidar depth maps instead of monocular depth estimation (lidar), using a pretrained and frozen depth classifier (pretrained), and finally removing depth estimation altogether by projecting camera features at all depths uniformly using Eq. (3) (uniform depth).

[0117] Figure 4 shows the impact of the quality of the monocular depth prediction on the object detection performance of BEVFusion [Ref.36] on the nuScenes val set. In Figure 4, theBEV fusion depth map 301 and the LiDAR depth maps 302 and 303 are the same ones in Figure 3 discussed above. The depth maps 401-406 are BEV fusion depth maps with depth supervision using Eq. (2) with weights 0, 0.001, 0.01, 0.1, 5, and 100 respectively. Quantitative results can be seen in Table 2 corresponding to the qualitative comparisons in Figure 4.

[0118] In the present experiments, more accurate depth does not translate to better detection performance and the original model is on-par with using the lidar points directly as a source of depth. The model with the best detection performance overall is the uniform model, clearly indicating that accurate depth is not necessary in these models.

[0119] Table 2 shows an extended analysis of the monocular depth quality provided by different variations of the “LiftSplat” camera feature projection. All experiments are performed using the original BEVFusion code

[0033] , with the code path that provides the monocular depth prediction module with ground-truth lidar depth removed. The performance of the depth classifier is evaluated using five metrics common in the literature: root mean squared error (RMSE), root mean squared logarithmic error (RMSLE), mean absolute relative error (Abs. Rel.), mean squared relative error (Sq. Rel.) and fraction outside 125% (Frac.125). The 3D detection mAP and depth quality of the following models are compared: the original model (no supervision), models fine-tuned with the additional depth supervision loss Eq. (4) at various values of the weight λ, a model with a pretrained depth classifier (pretrained), a model that uses the lidar ground truth directly (lidar), and the simplified projection Eq. (3) (no depth). In the present experiments, more accurate depth does not translate to better detection performance and the original model is on-par with using the lidar points directly as a source of depth. The model with the best detection performance overall is the no depth model, clearly indicating that accurate depth is not necessary in these models Table 2: Extended analysis of the monocular depth quality provided by different variations of the “LiftSplat” camera feature projection. 3*Loss mode mean Weight 3*mAP Relative RMSE 2*Frac. Relative RMSE 2*Frac. 125 125Abs. Sq. Linear Log Abs. Sq. Linear Log BEVFusion 68.5 2.95 133.76 25.95 1.87 0.97 2.75 61.31 17.40 1.30 0.88 [Ref.36] ^= 068.4 3.69 176.09 30.22 1.90 0.96 2.83 68.73 18.54 1.34 0.87 0.001 68.1 1.79 65.63 20.16 1.77 0.94 3.14 79.87 19.91 1.39 0.88 0.01 68.0 0.61 11.78 11.54 1.03 0.63 0.76 10.30 8.09 0.68 0.61 0.1 68.1 0.38 5.53 9.28 0.77 0.41 0.43 4.97 6.47 0.46 0.37 1 68.1 0.21 2.48 5.78 0.37 0.20 0.22 2.23 4.77 0.33 0.19 5 66.6 0.19 2.01 4.77 0.33 0.17 0.19 1.95 4.53 0.32 0.17 100 64.6 0.16 1.15 4.64 0.33 0.17 0.16 1.12 4.55 0.32 0.17 Pretrained 67.4 0.54 8.10 9.95 0.86 0.61 0.64 7.91 7.87 0.66 0.57 Lidar 68.4 0.04 0.01 0.29 0.05 0.00 0.04 0.01 0.29 0.05 0.00 Uniform 68.5 – – – – – – – – – – depth

[0120] For qualitative evaluation visualisations of the depth maps in Figure 4 are included. Camera feature cells for which LiDAR depth values unavailable are ignored.

[0121] A camera-lidar fusion method is presented that bypasses monocular depth estimation altogether and instead fuses camera and lidar features in bird’s-eye-view using a simple transformer [Ref.51]. A naive application of the transformer architecture to the problem of camera-lidar fusion is difficult, however, due to the large number of camera and lidar features and the quadratic nature of attention. As shown in [Ref.47], it is possible to use the geometry of the problem to drastically restrict the scope of the attention when projecting camera features in BEV, since camera features should only contribute to locations along their corresponding rays. This idea is adapted to the case of camera-lidar fusion and a simple method that uses cross-attention between columns in the camera plane and polar rays in the lidar BEV grid are used. Instead of predicting monocular depth, cross-attention learns which camera features are the most salient given context provided by the lidar features along its ray.

[0122] Further details of the LAS embodiment will now be described. The fact that accurate monocular depth is not crucial to the success of LSS suggests that lidar features provide amuch better indication of object depth than can be obtained from the camera alone. This was shown by using the lidar depth directly to project the camera features in 3D space, leading to results similar or better compared to ad-hoc monocular depth prediction of the LS projection.

[0123] The LAS embodiment is, therefore, constructed to allow lidar features to guide the fusion in selecting appropriate camera features for the object detection task. Architectures based on Transformer neural network architectures [9] have shown great success in multimodal fusion, language, etc.

[0124] A naive application of the Transformer architecture to the problem of camera-lidar fusion is difficult, however, due to the large number of camera and lidar features and the quadratic nature of attention, especially for resolutions required for state-of-the-art results on AD datasets. However, it is possible to use the geometry of the problem to drastically restrict the scope of the attention between cameras and lidar, since not all possible pairs of camera and lidar features need to attend to each other. A novel architecture is described for camera- lidar fusion that uses cross-attention between columns in the camera plane and polar rays in the lidar BEV. In the absence of monocular depth, this correspondence between columns and rays is the most general and depth is learned implicitly in the present model. Underpinning the described implementation is an assumption that attention is necessary only to resolve the inherent depth ambiguity of camera images, and that the remaining correspondence between the image plane and BEV space is determined by known camera intrinsics and extrinsics.

[0125] Except for the projection of the camera features in BEV, the present model shares a similar overall architecture to methods based on the Lift-Splat paradigm [Ref.16, 31, 36. It consists of the following modules: the camera and lidar backbones which produce features for each modality independently, a projection and fusion module that embeds the camera features into BEV and fuses them with the lidar, and finally a detection head. When considering object detection, the final output of the model is the property of objects in the scene represented as 3D bounding boxes with position, dimension, orientation, velocity and classification information. In what follows the architecture of the present projection and fusion modules is described in detail.

[0126] Figure 5 shows a camera image feature map 501 (a feature map extracted from a camera image) associated with a particular camera. A horizon plane 502 is defined by a horizontal line passing through the centre of the image and the plane corresponding to its projection in 3D. With multiple cameras, a horizon plane is determined for each camera inthis manner. This plane 502 is referred to herein as the projected horizon of the camera. Note that this plane is not in general parallel to the BEV grid 503, its exact orientation being defined by the camera’s extrinsic parameters. The location of any point on the projected horizon can be described by a position on the corresponding horizontal line in the image plane and a depth. Using these coordinates, a regular grid is defined on the projected horizon that is aligned with the 2D grid of features in the image plane by tracing out rays from the intersection of the horizontal line with the edges of the feature columns in the image plane, and then separating these rays into a set of predetermined depth bins, similarly to LS based methods [Ref.41, 36, 31, 16]. Reference number 504 denotes a feature column (column of camera image features) within the camera image feature map 501.

[0127] The feature column 502 is associated with a ‘ray’ of lidar features 505, which have been lifted out of the square BEV 503 grid. The lidar feature ray 505 is an array of lifted lidar features (liar feature vectors) lying within a portion the frustrum defined by tracing rays though the vertical edges of the feature column 502. Whereas each original lidar feature is associated with a square grid cell within the BEV plane 503, the lidar ray is an array of numerical lidar features (feature vectors), each of which is associated with a non-square sub- region within the frustum.

[0128] In the following examples, the camera image feature column 501 is fed into the encoder side of a transformer, and the lidar feature array is fed into the decoder side of that transformer. Processed together, the result is a corresponding "ray" of projected camera features in the projected horizon 502, where the lidar ray 505 has guided the allocation of camera features to depth bins.

[0129] For each camera, the horizontal line passing through the centre of the image and the plane corresponding to its projection in 3D is considered. This plane is called the projected horizon of the camera. It can easily be described using homogeneous coordinates as the set of points ^ ∈ ℝ^for which there exists a ^ ∈ ℝ such that ^^ ∼ (^, ℎ / 2,1), (4) where ^ is the 3 × 4 camera projection matrix (intrinsic and extrinsic), and ℎ is the height of the image. Note that this plane is not in general parallel to the BEV grid, its relative orientation being defined by the camera’s extrinsic parameters. A regular grid on each camera’s projected horizon is defined that is aligned with the 2D grid of features in its imageplane by tracing out rays from the intersection of the horizontal line with the edges of the feature columns in the image plane, and then separating these rays into a set of predetermined depth bins (similarly to [Ref.31]). Final projected horizon is a set of 289 points in 3d space. Features on this grid can be represented by a matrix ^ ∈where each row corresponds to a specific column in the camera feature map ^^^^geometry of a projected horizon can be seen in Figure 5. The projected horizon allows for a consistent definition of depth between differently pitched cameras. Features from the camera and lidar backbones are fused together and merged before being passed to a detection head. In the geometry of the 3D projection, the “Lift” step embeds the lidar BEV features into the projected horizon by lifting the lidar features along the z-direction using bilinear sampling. The “Splat” step corresponds to the inverse transformation in that it projects features from the projected horizon back onto the BEV grid using bilinear sampling, again along the z-direction. (right) Details of the projection module: the “Attend” step in the present method lets the lifted lidar features ^^^^^ ^^^ ^ attend to the camera features ^^in the corresponding column using a simple encoder-decoder transformer architecture to produce fused features ^(^^^^^ ^^^ ^ , ^(^^)) in frustum space.

[0130] A correspondence between points on a projected horizon and points on the BEV plane can easily be defined by projecting them along a direction perpendicular to the BEV plane, the z-direction, in 3D space. As cameras are in general tilted with respect to the ground, this correspondence depends on each camera’s extrinsic parameters. Tilting of the camera with respect to the ground may be accounted for using knowledge of each camera’s extrinsic parameters. Lidar features are transferred from the BEV grid to a camera’s projected horizon grid (and vice versa) through bi-linear sampling of the BEV grid at the locations of the down-projected cell-centers of the projected horizon on the target grid. This process is called “lifting” and is denoted as Lift^for the projected horizon of camera ^, in the context of the LAS method, and the inverse is referred to as “splatting”. Similarly, camera features can be transferred in the opposite direction, from a projected horizon to the BEV grid, by bi-linearly sampling the projected horizon at the locations of the projected cell-centers of the BEV grid. This operation is denoted as Splat^, similarly to [Ref.31, 36, 41]. Fusion of lidar features with splatted camera features takes place in BEV space, as is common [Ref.31, 36]. The latter is similar to the “splat” step in LSS [Ref.41], with the difference that LSS’s splat doesn’t make use of interpolation. Note that the “lift“ step here refers to the resampling ofBEV features in the projected horizon grid and not to the creation of a pseudo-point cloud of camera features via depth estimation, in contrast to the more conventional LSS terminology.

[0131] The present projection module can be broken down into three simple steps: (i) first lift the BEV lidar features ^^^^onto the projected horizon of camera ^, producing “lifted” lidar features ^^^^^ ^ , (ii) let the “lifted” lidar features attend to the camera features in the corresponding column using a simple transformer encoder-decoder, producing fused features^^^^^ ^ on the ^^^projected horizon, and finally (iii) splat these features back onto the BEV grid to produce ^^^^ ^ . During the attend step, the camera features in each column are encoded by a transformer encoder ^ and passed as keys and values to a transformer decoder ^ which uses the frustum lidar features as queries.

[0132] Figure 6 shows the LiftAttendSplat approach to camera-lidar fusion and can be broadly characterized in three steps.

[0133] In Step 603, a Lidar encoder 602 lifts the BEV lidar features ^^^^601 onto the projected horizon of camera ^, producing “lifted” lidar features ^^ ^^^. During the “Lift” step 603, the lidar BEV features ^^^^`are embedded into the projected horizon of a camera by projecting the lidar features along the z-direction using bilinear sampling.

[0134] In Step 604, the “lifted” lidar features attend to the camera features 600 in the corresponding column using a transformer encoder-decoder (602, 605), producing fused features on the projected horizon.

[0135] In Step 606, these features are splatted back onto the BEV grid to produceStep 606 can be expressed mathematically aswhere Lift^and Splat^project the BEV features onto the projected horizon of camera ^ (and vice versa) as described above. E and D are the transformer encoder and decoder detailed below

[0136] Finally, a simple fusion module is applied where the projected features are summed from different cameras together, concatenate them with the lidar features and apply a convolutional block to obtain the final features in BEV.

[0137] Finally, in step 607, the fused features from different cameras are summed together and concatenated with the lidar and a linear layer 608 is applied to obtain the final fused features in BEV aswhere. These fusion features 609 can then be used for any object detector in BEV space. All weights are shared across columns of the camera feature space as well as across different cameras.

[0138] Note that, in the case the camera is sufficiently aligned with the BEV plane, the BEV may be parallel to the projected horizon. In this case, the lidar features are still lifted from the square grid of the BEV plane to the non-square projected horizon, and the fused features splatted back to the square BEV grid, as per Figure 5.

[0139] This simple architecture allows the camera features to be projected from the image plane onto the BEV grid without requiring monocular depth estimation. A single set of transformer weights are shared across all column-frustum pairs and cameras. For simplicity, a single transformer encoder and decoder is used but it is shown later that adding more can be beneficial. All camera features participate in the attention, not just a small number of reference points as is the case in [Ref.28]. This learnt set of salient features initialises the object detection queries, rather than the fixed maxpool of [Ref.1].

[0140] A first example implementation of the “Attend” step will now be described. Note this is merely one example, chosen largely for simplicity. The present LiftAttendSplat approach is a general framework for camera lidar fusion that can accommodate more complex architectures. The present implementation uses the Transformer architecture of [Ref.51].

[0141] Figure 7 shows how the projected horizon lidar features are used attend to the column camera features using an encoder-decoder architecture. The camera features 704 are first encoded using a transformer encoder 706 and then passed to a Transformer decoder (D) 707 which takes the frustum lidar features as inputs and attends to the encoded column camera features using cross-attention. This allows the lidar to select the relevant camera features dynamically. All the weights of the column-frustum transformer are shared among columns and cameras.

[0142] A neural attention function is applied to a query vector q and a set of key-value pairs. Each key-value pair is formed of a key vector ^^and a value vector ^^, and the set of key- value pairs is denoted {^^, ^^}. An attention score for the ^th key-value pair with respect to the query vector q is computed as a softmax of a dot product of the query vector with the ^th key value, ^ ⋅ ^^. An output is computed as a weighted sum of the value vectors, {^^}, weighted by the attention scores.

[0143] In a ‘self-attention’ attention layer of a transformer, query, key and value vectors are all derived from an input sequence (inputted to a self-attention layer) through matrix multiplication. The input sequence comprises multiple input vectors at respective sequence positions, and may be an input to the transformer (e.g., tokenized and embedded text, image, audio etc.) or a ‘hidden’ input from another layer in the transformer. For each input vector ^^in the input sequence, a query vector ^^, a key vector ^^and a value vector ^^are computed through matrix multiplication of the input vector ^^with learnable matrices ^^, ^^, ^^. An attention score ^^,^for every input vector ^^with respect to position ^ (including ^ = ^) is given by the softmax of ^^⋅ ^^. An output vector ^^for token ^ is computed as a weighted sum of the values ^^, ^^, …, weighted by their attention scores: ^^= ∑^^^,^^^. The attention score ^^,^captures the relevance (or relative importance) of input vector ^^to input vector ^^.

[0144] However, in the present context, ‘cross-attention’ is used. In one form of cross- attention, self-attention is performed at encoding with cross-attention at decoding. Thus, in the encoder step of a transformer encoder-decoder, the output vector ^^=^^,^^^describes the relative importance of input vector ^^to input vector ^^(self-attention) in the same way as described above. The relative importance (cross-attention) of input vector ^^to a second input vector ^^from a different set of input vectors may be found using a decoder step. The output attention matrix ^^=^^,^^^from the encoder step is multiplied with learnable matrix ^ ^ of the decoder step to obtain the query vec ^ ^ tor ^^. Input vector ^^is multiplied with learnable matrices ^^ and ^^ of the de ^ ^^coder step to obtain the key vector ^^and the value vector ^^ ^ . Cross-attention scores ^^,^are similarly computed by the softmax of the dot product of the query vector ^^ ^ ^ and the transpose of the key vector ^^. An output vector ^^for token ^ is computed as a weighted sum of the values ^^ ^ ^ , ^^, …, weighted by their attentionscores: ^^= ∑^^^,^^^ ^ . The attention score ^^,^captures the relevance (or relative importance) of input vector ^^to input vector ^^.

[0145] In the present context, the 2D camera features are attended to in their original space (the camera image plane). It is the process of attention that "pulls" them onto the projected horizon.. The ‘query-key-value’ terminology reflects parallels with a data retrieval mechanism, in which a query is matched with a key to return a corresponding value. As noted above, in traditional neural attention, the query is represented by a single embedding vector ^. In this context, an attention layer is, in effect, querying knowledge that is captured implicitly (in a non-interpretable, non-verifiable and non-correctable manner) in the weights of the neural network itself.

[0146] A camera-to-BEV attention map Attnmay computed by specifying the projection mechanism. For the present method, the attention map of the last transformer decoder block is extracted by averaging over all heads Attn(^^^^→ ^^^^^^^^) ∈ ℝ^×^×^×^^, where ^ × ^′ corresponds to the frustrum dimension. Finally Attn (^^^^→ ^^^) is constructed by mean scattering the frustrum attention values onto the BEV grid. The LiftSplat projection lifts the camera features onto a 3D point cloud using the normalized depth classification weight ^^onto the BEV grid, resulting in the camera-to-BEV attention map Attn (^^^^→ ^^^). For both projections, given the camera image ^^, ^^^^(^)∈ {0,1}^×^is created by in-paint drawing annotations. Finally the BEV attention ^^^^(^^^)∈ ℝ^×^illustrates where visual camera features of ground truth objects get projected onto the BEV grid:

[0147] The differences between the present approach and the approach of predicting monocular depth directly are discussed. When using monocular depth, each feature in the camera feature map is projected into BEV at multiple locations weighted by a normalised depth distribution. This normalisation limits each feature to be projected either into a single location or smeared with lower intensity across multiple depths. In “LiftSplat” this means that the depth bins D are normalised along the corresponding ray for each camera feature. However, in the present approach, the attention between camera and lidar is such that the same camera feature can contribute fully to multiple locations in the BEV grid. This is possible because attention is normalised over keys, which correspond to different heights inthe camera feature map, rather than queries, which correspond to different distances along the ray. Furthermore, the present model has access to lidar features in BEV when choosing where to project camera features, which gives it greater flexibility when selecting camera features to be fused to the lidar. Finally, the projection requires fewer parameters than competing methods: 0.9M for the attention-based module compared to 1.6M in the equivalent component of [Ref.36].

[0148] The effectiveness of the present approach against recent camera-lidar fusion methods that use the Lift-Splat paradigm [Ref.31, 36] is measured. In all of the present experiments, 3D object detection using the nuScenes dataset [Ref.2] is used, which is a large-scale dataset for autonomous driving which contains 40,157 annotated sample organised in 1000 runs, each sample consisting of six monocular camera images with 360- degree FoV and a 32- beam LiDAR point cloud. It contains up to 1.4 million annotated 3D bounding boxes for 10 classes. The nuScenes detection score (NDS) and mean average precision (mAP) as evaluation metrics for object detection are used. For a fair comparison, the extension of [Ref. 31, 36] presented in [Ref.16] is not considered as it introduces two supplementary dense depth supervision losses on the camera path to significantly boost the performance of the underlying methods. Solely the 3D object detection losses present in [Ref.1, 31, 36] are used and the application of the framework of [Ref.16] to the present method is not performed .

[0149] Dual-Swin-Tiny [Ref.30] is used with a feature pyramid network [Ref.32] and VoxelNet [Ref.70] as the present camera and lidar encoders respectively. The object detection head is the transformer-decoder-based module from TransFusion-L [Ref.1]. The Lift-Attend-Splat method is used to project camera features into BEV space. Camera and lidar features are then fused using simple concatenation and convolution. Following [Ref.36] the RPN part of VoxelNet is applied to the merged feature. Alternative choices for the fusion architecture are also ablated.

[0150] Inputs to the camera encoder have resolution 800x448, which it downsamples by 8x into per-camera feature maps of shape 100x56. For VoxelNet, the settings of [Ref.31] are followed. A maximum of 90k non-empty voxels during training is used, increased to 180k for inference. An ego-centric BEV grid with dimensions 108m × 108m and 0.075m cell size is used. This is downsampled 8x by the lidar encoder to the 180 × 180 grid into which the camera features are projected. The intermediate projected horizon is constructed with 143 uniformly spaced depth bins ranging from 1m to 72m. For the projection of camera featuresinto BEV, original transformer [Ref.51] is used as encoder-decoder architecture, with one encoder layer, one decoder layer, ^^^^^^= 256 and ^^^= 512. The ReLU [Ref.39] activation function is replaced with GeLU [Ref.15], using learnable position embeddings [Ref.11] in place of sinusoidal encodings and normalise features before each sublayer [Ref. 60]. The parameters are tied in each of the 8 heads of the attention blocks. For the object detection head, 200 and 300 queries are used during training and inference respectively.

[0151] The lidar backbone is pretrained on 8 GPUs with batch-size of 1 / GPU following the schedule presented in [Ref.1], with CBGS [Ref.71]. The camera backbone is initialised with weights pretrained on nuImages [Ref.2] by [Ref.31]. The lidar backbone is frozen and the camera backbone trained, with projection, fusion and detection head for 20 epochs using 8 GPUs with a batch size of 4 / GPU. The method uses the AdamW optimiser [Ref.37] with a maximum learning rate of 5 × 10^^for the camera backbone and 1 × 10^^for all other components. The method applies the following augmentations: mirroring in the Y dimension, global rotation and scale, and camera-lidar copy-paste [Ref.53]. The LiDAR stream is first pre-trained separately for 20 epochs, following [Ref.1, 71, 68]. The LiDAR backbone is then frozen and the camera backbone is trained with fusion and detection heads for another 10 epochs.

[0152] Results for the task of 3D object detection are presented in Table 3. Table 3: Object detection performance on the validation and test splits of the nuScenes dataset. TFA: Temporal Feature Aggregation. denotes test-time augmentations and model ensembling. val. test mAP NDS mAP NDS BEVFusion [Ref.36] 68.5 71.4 70.2 72.9 BEVFusion [Ref.31] 69.6 72.1 71.3 73.3 FUTR3D 64.2 68.0 69.4 72.1 [Ref.5] TransFusion [Ref.1] 67.5 71.3 68.9 71.6 DeepInteraction [Ref.66] 69.9 72.6 70.8 73.4MSMDFusion [Ref.21] - - 71.5 74.0 CMT [Ref.62] 70.3 72.9 72.0 74.1 UniTR [Ref.55] 70.5 73.3 70.9 74.5 Ours 71.2 72.7 71.5 73.6 Ours w / TFA 72.1 73.8 - - BEVFusion [Ref.36] 73.7 74.9 75.0 76.1Ours74.6 75.1 - -Ours w / TFA75.7 76.0 75.5 74.9

[0153] Compared to baselines based on the Lift-Splat projection [Ref.31, 36], the present method, labelled “Ours”shows improvements on the validation and test splits of the nuScenes dataset. In particular, substantial improvements in both mAP (+1.1) and NDS (+0.4) on the test split are shown. The lidar backbone is frozen, showing that the present model is better able to leverage camera features. The present method outperforms the more recent fusion algorithms TransFusion, DeepInteraction, and FUTR3D by a substantial margin, performs similarly to UniTR and MSMDFusion, and slightly underperforms CMT. These results show that the present simple modification of BEVFusion is successful in raising its performance on par with recent SOTA methods. The present method achieves even better results with the addition of test-time-augmentations and via ensembling multiple models of varying grid size The step of extending the present method with multi-scale feature fusion (used by MSMDFusion) and query denoising (used by CMT) is not performed here.

[0154] Further details of the SwiftBlat (SB) embodiments will now be described.

[0155] The SB method attempts to find, for a given BEV grid cell, where the centre of that BEV grid cell would project into in the camera image. Figure 8 shows a BEV grid cell 801 in a BEV grid 800. Conceptually, this BEV grid cell 801 is projected in the vertical direction into a column 802 of infinite height. A camera 803, with its axes perpendicular to the BEV grid 800 will capture a column 804 in the camera image 805. The camera features in that column 804 may be summed to obtain the features in the BEV grid cell 801. In practice, there are two additional steps before the BEV features are computed. First, feature modulation isperformed on the camera image and, second, the camera image is reduced to a single row (of super pixels).

[0156] The LiftSplat image context vectors ^′ ^^^ ^^^are first modulated using learnt weights ^ ∈ generated from ^ ^^^ ^^^. These weights are constrained to be in [0, 1] and allow the network to discard camera features that are not useful for downstream tasks before reduction. Figure 9 shows that there may be a lot of sky 901 or ground 902 in the camera image 900 which do not necessarily need to be projected onto the BEV grid. This helps to simplify the projection task.

[0157] The modulated context vectors are then reduced to a single row 905 of superpixels. This is achieved by noting that for an axis-aligned camera (camera axes are perpendicular to BEV grid) and ^ a tensor of ones, every cell in a column of camera features is projected to the same BEV location. Therefore, it is possible to reduce the column (^) dimension from the camera feature map before projection and achieve an equivalent result. This also assumes that the speed of the vehicle is slow compared to the speed at which the camera images are being taken. Figure 10 shows how the modulated context vectors are reduced to a single row for a camera which is not axis-aligned i.e., the image plane and the projection plane are not perpendicular. In this case, reference lines are drawn perpendicular to the projection plane and the cells in the camera image are aggregated in step 1001 along a reference line 1002 to obtain a corresponding superpixel 1003 in the projection plane.

[0158] Mathematically, the modulated context vectors are reduced to a single row viayielding ^′^^^^^^ ∈.

[0159] A BEV grid cell centre may not project exactly into the centre of a camera pixel. Thus, the BEV grid feature map ^′^^^^^^is constructed from a linear interpolation of the two closest ^′^^^^^^elements of the projected center (^, ^, 0) for each BEV cell visible in the camera.

[0160] Note that camera images may overlap in the BEV grid, i.e the frustums in Figure 2 are not necessarily disjoint if multiple cameras have a field of view that overlaps. In this case, it is possible to sum or take the maximum of the camera features that project into a given BEV grid cell.

[0161] Figure 11 exemplifies the SwiftBlat method for two example BEV grid cell centres 1101 and 1104. It is attempted to find where the centres, 1101 and 1104, of two example BEV grid cells would project into in the camera image. Conceptually, the centres 1101 and 1104 are each projected in the vertical direction into a column of infinite height. The column projected from the centre 1101 contains points, such as the points 1102 and 1103, at different heights ℎ^and ℎ^in a point cloud. The column projected from the centre 1104 contains the points, such as 1105 and 1106, at different heights ℎ^and ℎ^in the point cloud. A camera, with its axes perpendicular to the BEV grid will capture the projected columns as a column of the camera image. The BEV grid cell centres 1101 and 1104 thus project into the pixel centres 1107 and 1108, both in one column of the camera image.

[0162] The weighted camera features in the two pixels with centres 1107 and 1108 are denoted and ^′^,^^respectively. The point cloud is created assigning a depth, ^, to the camera features. Thus the camera features ^′^,^^are projected into points, such as 1103 and 1106, at depths ^ and ^ respectively. These points, obtained by projectingare denoted by ^^,^,^^^′^,^^and ^^,^,^^^′^,^^for the depths ^ and ^ respectively. Similarly, the camera features ^′^,^^are projected into points, such as 1103 and 1106, at depths ^ and ^ respectively. These points, obtained by projecting ^′^,^^, are denoted by ^^,^,^^^′^,^^and ^^,^,^^^′^,^^for the depths ^ and ^ respectively. The features falling in the BEV grid cell with centre 1101 at depth ^ are obtained by summing the points in the point cloud, such as ^^,^,^^^′^,^^and ^^,^,^^^′^,^^, along the corresponding column of infinite height. Also noting that features are weighted uniformly along the height dimension, ^^,^,^= ^^,^. Thus, the features falling in the BEV grid cell with centre 1101 at depth ^ are obtained by the expression^^,^,^′^,^, which is equivalent to ^^,^,∑^∊^^′^,^, Similarly, the features falling in the BEV grid cell with centre 1104 at depth j are obtained by the expression^^,^,^′^,^which is equivalent to ^^,^,∑^∊^^′^,^. It is noted that ^′^,^is in fact the sum of the features in the column of the camera image containing the pixels with centres 1107 and 1108. Thus, the features in that column in the camera image may be summed to obtain the features in the BEV grid cells.

[0163] Figure 12 summarises the steps in processing the camera image. The camera features are passed through convolutional layers to generate image context vectors ^^^^^ ^^^and the depth classification ^, denoting the probability of a given camera feature existing at a particular depth. The outer product of ^^^^^ ^^^cam and ^ is a point cloud of weighted camera features in 3D space. The point cloud is then aggregated in the height dimension to give ^^^^^ ^^^. The reference numeral 1202 denotes the aggregate of the feature vector for a superpixel in a row of the camera image.

[0164] Figure 13 shows the steps in processing the camera image using uniform depth weighting. The camera features in the camera feature tensor of dimensions HxW are assigned a uniform depth weighting. The camera features are aggregated over the height dimension (perpendicular to the BEV plane). Each aggregated camera feature is projected to multiple locations in BEV plane.

[0165] Figure 14 summarises the projection method used in the SwiftBlat method. A line, such as lines 1402 and 1404, is extended in 3D space through the center of each BEV grid cell, in a direction perpendicular to the BEV plane 1400. Each of the lines 1402 and 1404 are projected into the image plane to obtain lines 1402’ and 1404’ respectively. For each line projection, two closes pixel columns of the camera feature map is identified, and the distance to the center of each closest pixel column is computed. For example, projected line 1402’ is at a distance ^ from centre ^^of pixel column ^ and at a distance ^ from centre ^^^^of pixel column ^ + 1, where ^ < ^. Projected line 1404’ is at a distance ^′ from centre ^^of pixel column ^ and at a distance ^’ from centre ^^^^of pixel column ^ + 1, where ^ < ^ Thus, projected line 1402’ is closest to the centre ^^of pixel column ^, and projected line 1404’ is closest to the centre ^^^^of pixel column ^ + 1. Interpolation weights ^^,^and ^^^,^^are computed using the camera features ^^and ^^^^in the array of aggregate camera feature vectors, corresponding to pixel columns ^ and ^ + 1. The interpolation weights ^^,^and ^^^,^^are computed based on the predetermined geometric relationship between each BEV grid cell and the aggregate camera feature array asThe BEV grid cells are populated with the camera features based on the computed interpolation weights.

[0166] Figure 15 summarises the steps in the SwiftBlat method. The LiftSplat convolutional neural network (CNN) is applied to the camera features 1501 to give image context vectors 1502. These image context vectors are modulated to give modulated image context vectors 1503 in the camera image. The columns of pixels in the camera image are then reduced to a single row of superpixels 1504. For a given BEV grid cell 1505 in a BEV grid, it is found where in the row of superpixels 1504 the BEV grid cell centre 1506 is projected. The features in the two closest superpixels 1508 to the projected centre 1506 of the BEV grid cell 1505 are linearly interpolated to obtain the BEV grid cell features 1509.

[0167] The performance of the present model is analysed further by clustering objects (of the car class) together depending on their distances from the ego and on their sizes, and on the number of lidar points falling into their ground-truth boxes

[0168] Figure 16 shows object detection performance measured using mAP for objects at different distances from the ego and of different sizes. In bar chart 1600, it can be seen that the mAP for the lidar-only model at object distances ≤ 20^, 20 − 30^, > 30^ is approximately 78, 60, and 42 as seen from bars 1601, 1602 and 1603 respectively. The mAP for the BEVFusion model from [Ref.31] at object distances ≤ 20^, 20 − 30^, > 30^ is approximately 80, 66, and 52 as seen from bars 1604, 1605 and 1606 respectively. The mAP for the BEVFusion model from [Ref.36] at object distances ≤ 20^, 20 − 30^, > 30^ is approximately 80, 67, and 50 as seen from bars 1607, 1608 and 1609 respectively. The mAP for the model of the present invention, termed “Ours” at object distances ≤ 20^, 20 − 30^, > 30^ is approximately 82, 70, and 55 as seen from bars 1610, 1611 and 1612 respectively. The mAP for objects of size ≤ 4^ is approximately 60, 67, 67, 70 for the lidar- only model, the BEVFusion model from [Ref.31], the BEVFusion model from [Ref.36], and the present model (“Ours”) respectively as seen from the bars 1613, 1614, 1615 and 1616 respectively on chart 1621. The mAP for objects of size > 4^ is approximately 55, 60, 60, 63 for the lidar-only model, the BEVFusion model from [Ref.31], the BEVFusion model from [Ref.36], and the present model (“Ours”) respectively as seen from the bars 1616, 1617, 1618 and 1619 respectively on chart 1621.

[0169] Therefore, the present model consistently outperforms baselines based on Lift-Splat, especially at large distances and for small objects. It can be seen that the bulk of the improvements comes from objects located at large distances and of small sizes. These are situations for which monocular depth estimation is particularly difficult which could explain why the present model fares better in these cases. It is hypothesised that this is due to the better complementarity between camera and lidar illustrated in the previous section.

[0170] Note that even though far-away and small objects contain fewer lidar points, the present model is still able to leverage camera features effectively even though the context given by the lidar is weaker.

[0171] Results that use test-time-augmentations (TTA) and model ensembling are shown at the bottom of Table 3. TTA is performed over a combination of mirror and rotation augmentations and ensemble models with cell resolutions of 0.05m, 0.075m and 0.10m. TTA is first applied at each cell resolution and then merge the resulting boxes using Weighted Boxes Fusion (WBF) [Ref.50] on the outputs from TTA of each model to get the final detections which are used for evaluations. Unsurprisingly, the present method shows excellent scaling for these techniques and outperforms BEVFusion [Ref.36] on the nuScenes validation set.

[0172] The locations where camera features are projected onto the BEV grid are visualised, and the present method is compared to BEVFusion [Ref.36].

[0173] For the present method, the attention map of the final cross-attention block in the transformer is examined, averaged over all attention heads. For BEVFusion, the monocular depth estimate is used to establish the strength of correspondence between positions in camera and BEV space. Only the pixels corresponding to ground-truth objects are considered when calculating the total weight of projected camera features in BEV.

[0174] Figure 17 shows a visualisation of where camera features of ground-truth objects are projected onto the BEV grid for the present method compared to BEVFusion [Ref.36]. It is observed that the present method is able to place camera features around objects more narrowly than BEVFusion, which is based on monocular depth estimation.

[0175] The group of images 1700 represent the visualisation with the method of the present application. The present method places camera features predominantly in regions where ground-truth bounding boxes (1701-1708) are present. This indicates that it can effectivelyleverage the lidar point cloud as a context to project camera features at the relevant location in BEV. The group of images 1709 represent the visualisation with the BEVFusion method with ground truth bounding boxes 170-1717. Compared to BEVFusion, the distribution of features in images in group 1700 appears more narrowly localised and stronger around objects. This could be because the present projection mechanism does not require the weights of the camera features to be normalised along their ray, giving the present model more flexibility to place features at the desired location. Note that, even though the present method also projects camera features outside of ground-truth boxes in BEV, the strength of the activation in these regions is suppressed by the fusion module.

[0176] Figure 18 shows activations in BEV space derived by summing up feature maps along the channel dimension. The image 1801 uses the channel-wise concatenation of lidar and projected camera features. The image 1802 uses the output of the fusion module, demonstrating its efficacy in suppressing background activations.

[0177] This is consistent with previous findings described above, where it was shown that the latter part of the model can effectively suppress camera features at the wrong location.

[0178] Which pixels in the camera images are most attended to using saliency maps [Ref.49] are further explored. These are derived by computing the gradient of the maximum class logit with respect to a camera image ^^, given object query index ^ and probabilities ^, as ^^^,^̂ / ^^|^^where ^̂ = argmax^^^,^. They allow the visualisation of the contribution of individual pixels to the final detection for a selected object.

[0179] Figure 19 shows a comparison of saliency maps, cropped to aid visualisation, given the camera image 1900 for models trained with camera-lidar (maps 1901, 1902) and camera only (maps1903, 1904). The region for selecting camera features is labelled 1905 - 1908 in saliency maps 1901 – 1904 respectively. When trained with both camera and lidar, the present model, “Ours” (maps 1901 and 1903) selects camera features in an area that is different than when trained with camera only, while [Ref.36] behaves similarly in both settings.

[0180] Interestingly, it is observed that when trained with both camera and lidar, the present model tends to select camera features at different locations than when trained with cameras only. In the absence of lidar, the present method selects camera features across the entirety of the object, while in the presence of both lidar and cameras, the model selects camera featuresmainly from the upper part of the object. It is observed that this pattern is mostly prevalent for objects close to the ego which are well-represented by lidar point clouds but fades away for far-away objects or objects with few lidar points such as pedestrians.

[0181] Figure 20 shows saliency maps for images 2001-2004. Image 2001 shows a close object (a car) which is well-represented by the lidar point cloud. Image 2002 shows the same object as in image 200, but at a later frame when the car moved further away from the ego, the present model attends to the same area when trained with camera and lidar (maps 2005 and 2007) as when trained with camera only (maps 2006 and 2008). Image 2003 shows an occluded object, whose representation in the lidar point cloud is weaker, the present model attends to the entire unoccluded area in both settings, i.e. camera and lidar (map 2009), and camera only (map 2010).. BEVFusion [Ref.36] (in maps 2011-2016) appears to consistently attend to a larger neighbourhood of pixels. Image 2004 shows a pedestrian, who is not well- represented by the lidar point cloud, and is fully attended by the present model (maps 2017, 2018) in the presence of lidar and camera (map 2018). The present method selects features from regions where the object point cloud is assumed to be sparser. The occluded object is fully attended by camera in any setting due to the sparsity of the point cloud. The pedestrian, underrepresented by the lidar point cloud, is precisely attended by the present model trained on both camera and lidar which behaves similarly to the camera-only setting.

[0182] Figure 21 shows additional examples showcasing the weight of projected camera features onto the BEV space. All examples presented in the analysis are from the validation set. The images 2101-2104 are similar to the visualisations shown for Figure 17. Figure 21 shows two comparisons, namely for map 2021 (obtained with “Ours” model) with map 2103 (obtained with the BEVFusion model) for a given BEV grid, and for map 2002 (obtained with “Ours” model) with map 2104 (obtained with the BEVFusion model) for a different given BEV grid. Similar conclusions as with Figure 17 can be drawn.

[0183] It is hypothesised that the present projection architecture enables the model to select camera features that best complement the information encoded in lidar, resulting in differing attention patterns between camera-only and fusion settings. This pattern is less present in BEVFusion [Ref.36], which attends to the broader neighbourhood of pixels surrounding the selected object in both cases.

[0184] Because the present method fuses camera and lidar features onto a BEV grid, past information can easily be leveraged using temporal feature aggregation (TFA). To achievethis, the simple autoregressive procedure of VideoBEV [Ref.13] is implemented the fused BEV features ^^^^.Are aggregated instead of the camera features. The present method is as follows: (i) save the fused BEV features from the previous timestep, (ii) apply ego motion compensation to align these features with the current timestep, using bilinear sampling to construct the new feature grid, (iii) concatenate these features with the fused BEV features of the current timestep and merge them using a simple 3 × 3 convolutional block.

[0185] TFA models are trained on sequences of 3 frames for 10 epochs starting from the present single frame model’s object detection head, lidar and camera backbones. During training, the lidar and camera backbones are frozen. For inference, BEV features are accumulated for the entire length of a run, yielding detections at each time step. Table 2 shows that temporal feature aggregation boosts object detection performance significantly in all configurations.

[0186] Some design choices for the present method are ablated their impact on object detection performance is shown in Table 4.

[0187] All ablation experiments use a simpler training schedule with 10 epochs, batch accumulation instead of full batch training and no camera augmentations. The effect of different fusion modules is first analysed: a simple skip connection (add), a small concatenation and convolution layer (Cat+Conv as in [Ref.36]) and a gated sigmoid block [Ref.31] are compared. All are found to perform similarly, with Cat+Conv achieving the best mAP, contrary to findings of [Ref.31]. The number of transformer decoder blocks are also ablated in the “Attend” stage of the projection and show that increasing their number leads to a small improvement in mAP. This suggests that the present method’s performance scales with compute. A single decoder block is used in the main experiments to balance quality and performance. Finally, good improvement is seen in NDS with an increasing number of frames in TFA during training

[0188] Table 4 below shows the impact of different ways of fusing camera and Li-DAR modalities. Contradictory to [Ref.31] it is found that the simplest catenation and projection of both features perform the best. When no additional fusion module was used at all, given the present projection mechanism is already attending to the LiDAR features, it was found that the performance is worse. Table 4 below shows that it’s beneficial to put a LiDAR backbone neck after the fusion module (following [Ref.36]). It is hypothesised that in this configuration it is placed in the same place as in LiDAR-only pretraining and can take fulladvantage of the pretrained weights. Additionally, it provides more capacity for the model to correct any misalignments of camera-lidar feature during the projection, before the final detection head. Adding layers to the projection transformer does not improve model performance. Table 4: Impact of model modifications on 3D object detection performance: (i) feature fusion module, (ii) number of transformer decoder blocks in the “Attend” stage, (iii) number of frames in Temporal Feature Aggregation (TFA).∗frozen camera backbone. mAP NDS Fusion module Cat+Conv 70.43 71.9 Gated sigmoid [Ref.31] 70.12 71.9 Add 70.32 72.1 # decoder 1 block 70.29 71.9 blocks∗2 blocks 70.40 72.0 4 blocks 70.49 71.9 # TFA frames 1 frame (no TFA) 71.2 72.8 2 frames 72.1 73.3 3 frames 72.1 73.8

[0189] For each camera image the ground depth map ^^^is computed by projecting the 3D lidar point cloud onto the image plane and binning each point within the pixels of the camera feature map. For non-empty cells, [Ref.27] is followed and the depth is chosen to be the minimum distance (from the camera plane) of all the points in the cell, leaving the depth unspecified for empty cells and those for which the minimum yields a depth value which is outside the range of the model’s depth bins. This depth map is suitable for visualisation and depth metric evaluation, but for depth supervision it is necessary to calculate the one-hot encoding of ^^^according to buckets defined by the model’s depth bins ^.

[0190] The monocular depth maps shown in Figure 3 are generated by calculating the weighted average of the model’s depth bins ^ with the predicted depth distribution ^^^^^∈ ℝ^^×^×^^^^^^ ^,^= ∑^^^^^^ ^ ^^^^,^,^.

[0191] This depth map is constrained by construction to [min(^), max(^)] and this range onto is mapped into the Turbo colour map [Ref.38] for visualisation.

[0192] The lidar depth map ^^^is similarly colourised, except for cells where the depth is unspecified as described above (see Figure 4, top-right) which are coloured grey.

[0193] All of the present experiments are performed using the method presented in [Ref.36] and use the original repository (https: / / github.com / mit-han-lab / bevfusion). The vanilla Lift- Splat transform is used, implemented in the class LSSTransform with default parameters provided in the original work. The depth classifier is supervised by introducing the following loss alongside the original detection losses, ^ ^^^^^^= − ^ ∑^^log(^^⋅ 1^), (7) which is a cross-entropy loss between the lidar depth distribution and predicted depth distribution, taken over all cells for which the lidar depth is available. ^^is the normalised predicted depth distribution from the LiftSplat model for the ^th cell, 1^is the one-hot encoded lidar depth distribution for the ^th cell. The model is trained end-to-end with all components unfrozen as in [Ref.36] and hyper-parameter ^ controlling the strength of the depth supervision loss with respect to the detection losses.

[0194] The depth estimation module is also pretrained within LiftSplat. First, the camera stream in [Ref.36] is trained by supervising only the depth distribution with the whole camera pipeline unfrozen. Following this pretraining, the lidar components are added and the full model is trained end-to-end as in [Ref.36], with no depth supervision (^ = 0) and all modules unfrozen.

[0195] In Table 5 per-class detection scores are presented and the present model is compared to other state-of-the-art models on the validation and test splits of the nuScenes dataset. The present method outperforms baselines based on the LiftSplat paradigm. Additionally, improvements from test-time-augmentations and temporal feature aggregation are shown.

[0196] The performance of the present method is analysed compared to BEVFusion [Ref.30] and the lidar-only baseline on the nuScenes validation split for the car class. For each method, the ground truth and detections are divided into groups according to the number of lidar boxes inside each object, and the mAP is calculated separately for each group. This analysis shows that the present method performs best across all lidar points buckets, but of most interest is the boost in performance it achieves over the baselines in buckets which represents the objects most well represented by lidar points Table 5: Per-class object detection scores on the nuScenes validation set (top) and test set (bottom). TFA: Temporal Feature Aggregation. indicates ensembling + TTA. Model barrier trailer mAP bicycle bus car CV MC ped TC truck NDS Ours 74.1 70.0 81.3 90.3 33.8 80.8 89.3 79.7 44.0 68.2 71.2 72.7 BEVFusion 73.5 67.5 77.7 89.1 30.9 79.0 89.4 79.3 42.6 66.7 69.6 72.1 [Ref.31] DeepInteraction 78.1 52.9 68.3 87.1 33.1 73.6 88.4 86.7 60.8 60.0 69.9 72.6 [Ref.66] Ours77.5 75.2 82.3 91.2 40.0 85.6 90.6 80.2 50.1 72.2 74.6 75.1 Ours w / TFA 74.4 72.4 81.6 90.8 33.7 82.5 89.8 79.6 45.8 70.1 72.1 73.8 Ours w / TFA78.6 78.2 84.3 91.6 39.9 87.5 91.4 80.7 51.2 73.3 75.7 76.0 Ours 78.0 54.9 72.1 89.0 38.9 75.3 90.3 87.0 65.3 64.2 71.5 73.6 Ours w / TFA79.7 65.2 75.2 90.3 43.5 82.8 92.0 87.1 70.1 68.9 75.5 74.9 BEVFusion 78.3 56.5 72.0 88.5 38.1 75.2 90.0 86.5 64.7 63.1 71.3 73.3 [Ref.31] DeepInteraction 80.4 54.5 70.8 87.9 37.5 75.4 91.7 87.2 63.8 60.2 70.8 73.4 [Ref.66]

[0197] To obtain Figure 17 for the present method, the full camera-to-BEV attention map Attn(^^^^→^^^)∈ ℝ^×^×^×^is first computed. To do so, the attention map of the last transformer decoder block is extracted by averaging over all heads, Attn(^^^^→^^^^^^^)∈where (^ × ^′) corresponds to the frustum dimension. Attn(^^^^→^^^)isconstructed by scattering the frustum attention values onto the BEV grid. Given camera image ^^, Mask(^)∈is created by in-paint drawing the annotations, Figure 21, and obtain the BEV attention Attn(^^^)∈ ℝ^×^shown on Figure 17a by projecting these camera features onto the BEV grid: Attn

[0198] Figure 22 shows Camera image ^^(2201) with annotations (2202) of the present model’s predictions (2203). Figure 22 also shows a binary image Mask(^)(2204) created by in-painting the annotations.

[0199] To obtain a similar visualisation for the “LiftSplat” projection, see Figure 17, the implementation of [Ref 36] is adjusted but the same model weights are used. Firstly, the feature map of image ^^is replaced with Mask(^)and that is used as input to the projection. This binary mask is “lifted” onto a 3D point cloud using the normalised depth classification weights ^^for which the first 5 and last 5 depth bins are clipped. Mask(^)thus acts as an indicator function and ^^specifies the strength of correspondence between pixels and the 3D point cloud ^ ∈ ℝ^×^×^^×^. Secondly, during “splatting”, points are projected onto the z = 0 plane and pooled using max. This operation ensures that the weight of attention for large objects in the final visualisation does not overpower that of smaller objects.

[0200] For test-time-augmentation (TTA) and model ensembling, WBF [Ref.50] is usedbased on L2 distance metric per object category to decide which of the boxes to fuse. TTA (using mirror and rotation augmentations) with WBF is first performed for each cell resolution, and then another WBF is applied on the outputs from TTA of each model to get the final detections which are used for evaluations. For rotation augmentation, (-12.5, -6.25, 0, 6.25, 12.5) degrees are used. Tracking

[0201] The tracking performance of the present model is evaluated by implementing the method of CenterPoint

[0067] on top of the present model’s object detection architecture. Detections are assigned to tracks using distance based association between predictions at the current timestep and the projection of tracks from the previous timestep using their predicted velocity. Class-specific distance thresholds are used for the association algorithm with values obtained by maximising the tracking performance on the validation set of the nuScenesdataset. These thresholds are optimised as follows. Figure 23 shows the distribution of the L2 velocity errors for each class in the nuScenes validation set and the distance threshold corresponding to the velocity error at each of the 80th, 90th, 95th, 99th and 99.9th percentiles in these distributions. Finally, the tracking performance is evaluated for each class at every threshold and the per-class threshold is chosen to be that which maximises AMOTA. The distance thresholds corresponding to the velocity error at each of the 80th, 90th, 95th, 99th and 99.9th percentiles is shown by lines 2301-2305 (for class bicycle), 2306-2310 (for class bus), 2310-2315 (for class car), 2316-2320 (for class motorcycle), 2321-2325 (for class pedestrian), 2325-2330 (for class trailer) and 2330-2335 (for class truck) respectively.

[0202] References herein to components, functions, modules and the like, denote functional components of a computer system which may be implemented at the hardware level in various ways. A computer system comprises execution hardware which may be configured to execute the method / algorithmic steps disclosed herein and / or to implement a model trained using the present techniques. The term execution hardware encompasses any form / combination of hardware configured to execute the relevant method / algorithmic steps. The execution hardware may take the form of one or more processors, which may be programmable or non-programmable, or a combination of programmable and non- programmable hardware may be used. Examples of suitable programmable processors include general purpose processors based on an instruction set architecture, such as CPUs, GPUs / accelerator processors etc. Such general-purpose processors typically execute computer readable instructions held in memory coupled to or internal to the processor and carry out the relevant steps in accordance with those instructions. Other forms of programmable processors include field programmable gate arrays (FPGAs) having a circuit configuration programmable through circuit description code. Examples of non- programmable processors include application specific integrated circuits (ASICs). Code, instructions etc. may be stored as appropriate on transitory or non-transitory media (examples of the latter including solid state, magnetic and optical storage device(s) and the like). References Each of the following is incorporated herein by reference in its entirety.[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew- Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 1090–1099, 2022. [2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. [3] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2147–2156, 2016. [4] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017. [5] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 172–181, 2023. [6] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 5428–5437, 2022. [7] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao. Autoalign: pixel-instance feature aggregation for multi-modal 3d object detection. arXiv preprint arXiv:2201.06493, 2022. [8] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. Monodistill: Learning spatial features for monocular 3d object detection. arXiv preprint arXiv:2201.10830, 2022. [9] Florian Drews, Di Feng, Florian Faion, Lars Rosenbaum, Michael Ulrich, and Claudius Gläser. Deepfusion: A robust and modular 3d object detector for lidars, cameras and radars. In 2022 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pages 560–567. IEEE, 2022.

[0010] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2014.

[0011] Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International conference on machine learning, pages 1243–1252. PMLR, 2017.

[0012] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 3153–3163, 2021.

[0013] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303.05970, 2023.

[0014] Adam W Harley, Zhaoyuan Fang, Jie Li, Rares Ambrus, and Katerina Fragkiadaki. Simple-bev: What really matters for multi-sensor bev perception? In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2759–2765. IEEE, 2023.

[0015] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.

[0016] Haotian Hu, Fanyi Wang, Jingwen Su, Yaonong Wang, Laifeng Hu, Weiye Fang, Jingwei Xu, and Zhiwang Zhang. Ea-lss: Edge-aware lift-splat-shot framework for 3d bev object detection, 2023.

[0017] Dihe Huang, Ying Chen, Yikang Ding, Jinli Liao, Jianlin Liu, Kai Wu, Qiang Nie, Yong Liu, Chengjie Wang, and Zhiheng Li. Rethinking dimensionality reduction in grid-based 3d object detection. arXiv preprint arXiv:2209.09464, 2022.

[0018] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High- performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.

[0019] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In Computer Vision–ECCV 2020: 16thEuropean Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16, pages 35–52. Springer, 2020.

[0020] Xiaohui Jiang, Shuailin Li, Yingfei Liu, Shihao Wang, Fan Jia, Tiancai Wang, Lijin Han, and Xiangyu Zhang. Far3d: Expanding the horizon for surround-view 3d object detection. arXiv preprint arXiv:2308.09616, 2023.

[0021] Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Msmdfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 21643–21652, 2023.

[0022] Junho Koh, Junhyung Lee, Youngwoo Lee, Jaekyum Kim, and Jun Won Choi. Mgtanet: Encoding sequential lidar points using long short-term motion-guided temporal attention for 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1179–1187, 2023.

[0023] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In 2018 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1–8. IEEE, 2018.

[0024] Jason Ku, Alex D Pon, and Steven L Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11867–11876, 2019.

[0025] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 12697– 12705, 2019.

[0026] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 17182–17191, 2022.

[0027] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d objectdetection. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1477– 1485, 2023.

[0028] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1–18. Springer, 2022

[0029] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European conference on computer vision (ECCV), pages 641–656, 2018.

[0030] Tingting Liang, Xiaojie Chu, Yudong Liu, Yongtao Wang, Zhi Tang, Wei Chu, Jingdong Chen, and Haibin Ling. CBNet: A composite backbone network architecture for object detection. IEEE Transactions on Image Processing, 31:06893–6906, 2022.

[0031] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35:010421–10434, 2022.

[0032] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.

[0033] Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, and Zhizhong Su. Sparse4d: Multi-view 3d object detection with sparse spatial-temporal fusion. arXiv preprint arXiv:2211.10581, 2022.

[0034] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision, pages 531–548. Springer, 2022.

[0035] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 3262– 3272, 2023.

[0036] Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2774– 2781. IEEE, 2023.

[0037] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.

[0038] Anton Mikhailov. Turbo, an improved rainbow colormap for visualization. https: / / blog.research.google / 2019 / 08 / turbo-improved-rainbow-colormap-for.html , 2019. Accessed: 20th October 2023.

[0039] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML- 10), pages 807–814, 2010.

[0040] Kranti Parida, Neeraj Matiyali, Tanaya Guha, and Gaurav Sharma. Coordinated joint multimodal embeddings for generalized audio-visual zero-shot classification and retrieval of videos. In Proceedings of the IEEE / CVF winter conference on applications of computer vision, pages 3251–3260, 2020.

[0041] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16, pages 194–210. Springer, 2020.

[0042] AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova. 4d-net for learned multi-modal alignment. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 15435–15445, 2021.

[0043] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.

[0044] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017 b .

[0045] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 918–927, 2018.

[0046] Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lidar for image-based 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 5881–5890, 2020.

[0047] Avishkar Saha, Oscar Mendez, Chris Russell, and Richard Bowden. Translating images into maps. In 2022 International conference on robotics and automation (ICRA), pages 9200–9206. IEEE, 2022.

[0048] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 770–779, 2019.

[0049] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.

[0050] Roman A. Solovyev and Weimin Wang. Weighted boxes fusion: ensembling boxes for object detection models. CoRR, abs / 1910.13302, 2019.

[0051] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[0052] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 4604–4612, 2020.

[0053] Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11794–11803, 2021 a .

[0054] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross- modal augmentation for 3d object detection. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 11794–11803, 2021 b .

[0055] Haiyang Wang, Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, and Liwei Wang. Unitr: A unified and efficient multi-modal transformer for bird’s-eye-view representation. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 6792–6802, 2023.

[0056] Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object- centric temporal modeling for efficient multi-view 3d object detection. arXiv preprint arXiv:2303.11926, 2023.

[0057] Weiyao Wang, Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 12695–12705, 2020.

[0058] Zhixin Wang and Kui Jia. Frustum convnet: Sliding frustums to aggregate local point- wise features for amodal 3d object detection. In 2019 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1742–1749. IEEE, 2019.

[0059] Zining Wang, Wei Zhan, and Masayoshi Tomizuka. Fusing bird view lidar point cloud and front view camera image for deep object detection, 2018.

[0060] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524–10533. PMLR, 2020.

[0061] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 3047–3054. IEEE, 2021.

[0062] Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal transformer: Towards fast and robust 3d object detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 18268– 18278, 2023.

[0063] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors (Basel, Switzerland), 18, 2018.

[0064] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 180 (10):03337, 2018.

[0065] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652–7660, 2018.

[0066] Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. Advances in Neural Information Processing Systems, 35:01992–2005, 2022.

[0067] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021.

[0068] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. Advances in Neural Information Processing Systems, 34:016494–16507, 2021.

[0069] Jinqing Zhang, Yanan Zhang, Qingjie Liu, and Yunhong Wang. Sa-bev: Generating semantic-aware bird’s-eye-view feature for multi-view 3d object detection. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 3348–3357, 2023.

[0070] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018.

[0071] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection, 2019.

[0072] Zhuofan Zong, Dongzhi Jiang, Guanglu Song, Zeyue Xue, Jingyong Su, Hongsheng Li, and Yu Liu. Temporal enhanced training of multi-view 3d object detector via historical object prediction. arXiv preprint arXiv:2304.00967, 2023.

[0073] Xie, E., Yu, Z., Zhou, D., Philion, J., Anandkumar, A., Fidler, S., Luo, P. and Alvarez, J.M., 2022. Multi-camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204.05088.

Claims

CLAIMS 1. A computer-implemented method of fusing visual features obtained with multiple sensor modalities, the method comprising: extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non-camera sensor modality, the sensor data being time- associated with the camera image; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non-parallel to the camera image plane in 3D space; and based on the camera feature vectors and the second feature vectors, computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, without applying monocular depth detection to the camera image.

2. The method of claim 1, wherein computing the plurality of fused feature vectors comprises performing, using a transformer neural network cross-attention between the plurality of camera feature vectors and the plurality of second feature vectors.

3. A computer-implemented method of fusing visual features obtained with multiple sensor modalities, the method comprising: extracting from a camera image using a visual feature extractor a plurality of camera feature vectors at respective camera plane locations in a camera image plane; receiving sensor data of a non-camera sensor modality, the sensor data being time- associated with the camera image; processing the sensor data to compute a plurality of second feature vectors in a target plane lying non-parallel to the camera image plane in 3D space; and based on the camera feature vectors and the second feature vectors, computing a plurality of fused feature vectors at a plurality of target plane locations in the target plane, wherein computing the plurality of fused feature vectors comprises performing, using a transformer neural network cross-attention between the plurality of camera feature vectors and the plurality of second feature vectors.

4. The method of claim 1, 2 or 3, comprising determining a plurality of intermediary fused feature vectors in an intermediary plane based on a projection of the second featurevectors from the target plane into the intermediary plane, the plurality of fused feature vectors computed by projecting the plurality of intermediary fused feature vectors into the target plane.

5. The method of claim 4, wherein the intermediary plane is a projected camera horizon plane defined by one or more predetermined camera properties associated with the camera image.

6. The method of claim 5, wherein the cross-attention is performed by: projecting the second feature vectors into a plurality of frustum portions in the intermediary plane, each frustum portion corresponding to a camera feature column lying perpendicular to the intermediary plane and containing a subset of the camera feature vectors in the camera image plane; and inputting to the transformer neural network each camera feature column in association with the corresponding frustum portion 7. The method of claim 6, wherein the second feature vectors are projected onto the intermediary plane based on bilinearly sampling a grid of the target plane at the locations of the projected centers of cells of a grid of the intermediary plane; and wherein the fused feature vectors are projected onto the target plane based on bilinearly sampling the grid of the target plane at the locations of the projected centers of cells of the grid of the intermediary plane.

8. The method of claim 7, wherein an encoder-decoder is used to compute the fused feature vector at each intermediary plane location based on the camera feature vector at the intermediary plane location and the second feature vector at the intermediary plane location.

9. The method of claim 8 when dependent on claim 2, wherein the transformer neural network comprises the encoder-decoder wherein each camera feature column is inputted to an encoder of the transformer neural network, and passed to a decoder of the transformer neural network, wherein the decoder uses associated frustum portion to attend to the camera features.

10. A computer-implemented method of extracting, in a horizon plane, camera features from a camera image, the method comprising: extracting from the camera image using a visual feature extractor a plurality of camera feature vectors in a camera image plane, the camera image plane lying non-parallel to the horizon plane in 3D space; aggregating the plurality of camera feature vectors along a height dimension in the camera image plane, the height dimension extending perpendicular to the horizon plane in 3D space, resulting in a plurality of aggregate feature vectors at respective camera image plane locations in the camera image plane; and populating a plurality of horizon plane locations in the horizon plane with respective camera features based on the plurality of aggregate feature vectors and a predetermined geometric relationship between each horizon plane location and the respective camera image plane locations.

11. The method of claim 10, wherein aggregating the plurality of camera feature vectors comprises determining a plurality of importance-weighted camera feature vectors, by weighting the plurality of camera feature vectors by respective importance weights, and applying an aggregation function to the importance-weighted camera feature vectors.

12. The method of claim 10 or 11, wherein each horizon plane location is pre-associated with a first of the camera image plane locations and a second of the camera image plane locations, and is populated with the visual features in the form of an interpolated camera feature vector based on: a first of the aggregate feature vectors at the first camera image plane location, a first predetermined distance between the first camera image plane location and the horizon plane location projected into the camera image plane, a second of the aggregate feature vectors at the second of camera image plane location, and a second predetermined distance between the second camera image plane location and the horizon plane location projected into the camera image plane.

13. The method of any of claims 10 to 12, wherein the respective camera image plane locations are arranged in a first uniform grid within the camera image plane.

14. The method of claim 13, wherein the camera image plane lies perpendicular to the horizon plane in 3D space such that the height dimension is aligned with a horizontal or vertical dimension of the first uniform grid.

15. The method of any of claims 10 to 14, wherein the plurality of horizon plane locations are arranged in a second uniform grid within the horizon plane.

16. The method of any preceding claim, comprising applying a perception component to the fused visual features to compute a perception output.

17. The method of claim 1, wherein the plurality of fused feature vectors are computed based on an assumption that each camera plane location corresponds to multiple depth locations in 3D space with equal weight.

18. A computer system comprising one or more computers configured to implement the method of any preceding claim.

19. Transitory or non-transitory media embodying computer-readable instructions configured when executed by one or more processors to cause the one or more processors to implement the method of any of claims 1 to 17.

Images