Cross-domain spatial matching for monocular 3D object detection and / or low-level sensor fusion

Abstract

There is described a method for performing cross-domain spatial matching (CDSM) for monocular object detection in a 3D free space around a vehicle, the method comprising: receiving at least one input image from a camera of the vehicle into an image processing network; determining a set of 2D image features of potential targets around the vehicle from the input image; transforming the set of 2D image features into a set of 3D image features of the potential target by aligning horizontal and longitudinal axes of the 2D image features with a tensor height axis and a tensor width axis of the 3D aerial view mesh, respectively, and extrapolating depth information by applying CDSM aggregation to the 3D image features; generating a set of aggregated 3D features of the potential target; and detecting one or more objects associated with the potential target based on the aggregated 3D features.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 512,045, filed on July 5, 2023. The entire disclosure of the above application is incorporated herein by reference. Technical Field

[0002] The present disclosure relates to techniques for performing Cross-Domain Spatial Matching (CDSM) for monocular three-dimensional (3D) object detection and / or low-level sensor fusion (e.g., camera and radar fusion). Background Art

[0003] The challenge encountered when detecting monocular objects in 3D space is the lack of third dimension or depth information within a single image, which is represented by pixels in two-dimensional (2D) space. Without information about the third dimension, it is difficult to accurately locate and understand objects in 3D space from processing only 2D images. There are many methods to address this problem. Some methods aim to enhance the performance and reliability of monocular object detection.

[0004] One way to address the lack of depth information in monocular object detection is to decouple the 2D and 3D parameters and predict them separately. For example, instead of using bounding boxes, CenterNet predicts only the center of the object while regressing other attributes such as depth, 3D size, and rotation angle relative to this key center point. Another approach is called FCOS3D. In this example, a set of predefined 3D locations in the image are maintained to perform 2D center prediction. In this approach, the image is projected back into a two-and-a-half-dimensional (2.5D) space with predicted depth based on the 2D locations and their corresponding ones. The remaining parameters are also regressed in 3D space to generate the final object prediction.

[0005] Another previous approach treats the problem as a depth estimation problem with a solution derived from a single image. Another approach, known as PGD, performs depth estimation of 2D image pixels as a distribution and incorporates a probabilistic representation to capture uncertainty. Furthermore, a geometric relationship graph is exploited to improve the estimation by considering contextual connections between 3D objects. EPro-PnP is another approach that adopts a perspective-n-points (PnP) approach. This solves the pose from a set of 3D points in object space and their corresponding 2D projection in image space. EPro-PnP proposes a differentiable layer using the PnP approach that trains the network in an end-to-end manner to learn weighted 2D-3D point correspondences. The output is interpreted as a probability distribution suitable for 3D object detection.

[0006] Other examples include Transformer-based vision networks for predicting 3D objects from 2D images, such as BEVFormer. BEVFormer exploits spatial and temporal information to learn a unified bird's-eye-view (BEV) representation by performing predefined grid-shaped birds-eye-view (BEV) queries on image features. BEVFormer predictions are currently being used by the NuScenes3D object detection benchmark. However, current implementations of these and similar Transformers are computationally intensive and are not yet suitable for embedded deployment (e.g., on an embedded computer or controller in a vehicle).

[0007] Fusion algorithms are used to enhance the performance of advanced vehicle perception systems. Fusion algorithms combine data from individual sensors and generate a final perception result. The fusion process leverages the strengths of each sensor and aims to achieve a more robust result. In addition, fusion algorithms provide an additional layer of safety in situations involving partial sensor blockage or other failure modes. By relying on more reliable sensor readings, it can help mitigate potential hazards.

[0008] Fusion algorithms can be classified as high-level or low-level. In high-level fusion, information about detected objects from different sensors is fused at the object level. Low-level fusion operates closer to the raw input data, directly utilizing information from each sensor's data stream. The key difference between the two lies in the stage at which they operate. High-level fusion applies to processed sensor detections, while low-level fusion applies to the raw or minimally processed data stream itself. Therefore, low-level fusion neural networks can potentially identify patterns in cross-sensor data that may not be accessible at a higher level (e.g., object level). However, as the sensor readout domains differ significantly (e.g., 2D view from an image camera vs. 3D surrounding view from a point cloud), correlating or combining these two sources of information presents many challenges.

[0009] Existing methods such as AVOD, MV3D, and PointFusion attempt to implement different multi-view approaches using complex domain mapping techniques. Newer methods such as CenterFusion and FUTR3D include different architectures where the fusion is performed on camera images processed similar to the CenterNet pure vision model approach, but also includes additional information from radar detections. First, 2D center points and object features are predicted in the image, which are then associated with the extracted radar features via the Frustum association mechanism. The fusion of the two sensor feature maps leads to the final 3D prediction.

[0010] Existing sensor fusion methods are computationally expensive to implement in certain systems. Existing methods of deriving 3D data from 2D images may be overly complex and unsuitable for embedded implementations where computing resources (e.g., processor bandwidth, memory) may be limited (such as in a vehicle). Summary of the invention

[0011] This section provides a general summary of the disclosure, but is not a comprehensive disclosure of its full scope or all of its features.

[0012] A computer-implemented method for a vehicle system for performing cross-domain spatial matching CDSM for monocular object detection in a 3D free space of a physical environment surrounding the vehicle, the method comprising: receiving at least one input image from a camera of the vehicle into an image processing network, the at least one input image comprising a 2D array of pixel information of the physical environment; determining a set of 2D image features of potential targets in the physical environment from the 2D array of pixel information of the input image; transforming the set of 2D image features into a set of 3D image features of the potential targets by applying CDSM rotations to the 2D image features, the CDSM rotations aligning the horizontal and vertical axes of the 2D image features with a tensor height axis and a tensor width axis of a 3D bird's eye view (BEV) grid, respectively, and extrapolating depth information of tensor channel axes perpendicular to the tensor width axis and the tensor height axis by applying CDSM aggregation to the 3D image features; generating a set of aggregated 3D features of the potential targets, the set of aggregated 3D features comprising depth information extrapolated for the tensor channel axes; and detecting one or more objects associated with the potential targets based on the aggregated 3D features.

[0013] In other features, at least one of the one or more objects is determined to be located in a path of travel of a vehicle.

[0014] In other features, the vehicle is controlled to avoid collision with at least one of the one or more objects.

[0015] In other features, receiving at least one input image includes centering a Cartesian global camera coordinate system GCCS at a center of a camera sensor field of view of the camera.

[0016] In other features, the GCCS includes: an X-axis defined along a path of travel of the vehicle; a Y-axis orthogonal to the X-axis, the Y-axis defining a width of the vehicle; and a Z-axis defining a tensor channel axis.

[0017] Among other features, the tensor height axis corresponds to the X-axis of the GCCS, and the tensor width axis corresponds to the Y-axis of the GCCS.

[0018] In other features, applying the CDSM rotation to the 2D image feature includes aligning the 2D image feature with the GCCS by rotating the 2D image feature about a Z axis by a first distance and rotating the GCCS about a Y axis by a second distance.

[0019] Among other features, a 2D convolution layer is applied to the BEV grid to generate refined 2D image features; and the refined 2D image features are passed to a 3D prediction head to process the refined 2D image features into 3D image features.

[0020] A computer system for performing cross-domain spatial matching CDSM for monocular object detection in a 3D free space of a physical environment around a vehicle, the computer system comprising: an image processing network, the image processing network being used to receive at least one input image from a camera of the vehicle into a network backbone, the at least one input image comprising a 2D array of pixel information of the physical environment; a bidirectional feature pyramid network, the bidirectional feature pyramid network being used to determine a set of 2D image features of potential targets in the physical environment from the 2D array of pixel information of the input image; and a CDSM system, the CDSM system being used to transform the set of 2D image features into a set of 3D image features of the potential targets by applying CDSM rotations to the 2D image features, the CDSM rotations aligning the horizontal and vertical axes of the 2D image features with the tensor height axis and the tensor width axis of a 3D bird's eye view (BEV) grid, respectively, and extrapolating depth information of tensor channel axes perpendicular to the tensor width axis and the tensor height axis by applying CDSM aggregation to the 3D image features.

[0021] Among other features, the CDSM system: generates a set of aggregated 3D features of a potential target, the set of aggregated 3D features including depth information extrapolated with respect to tensor channel axes; and detects one or more objects associated with the potential target based on the aggregated 3D features.

[0022] In other features, a point cloud processing network is connected to at least one of the lidar system and the radar system, and the point cloud processing system receives 3D point cloud data representing the physical environment surrounding the vehicle from at least one of the lidar system and the radar system.

[0023] Among other features, a voxel feature extractor (VFE), the VFE is coupled to a point cloud processing network, and the point cloud processing network and the VFE generate multiple 3D feature maps of the physical environment around the vehicle.

[0024] Among other features, the CDSM system fuses 2D image features with 3D feature maps in a CDSM fusion block to generate a set of aggregated 3D features of potential objects.

[0025] Among other features, the CDSM system generates unified output predictions across a grid of BEVs of potential targets.

[0026] In other features, the CDSM system aligns spatial information in a first domain representing 2D image features with spatial information in a second domain different from the first domain representing a 3D feature map in a CDSM alignment block to fuse the 2D image features with the 3D feature map.

[0027] In other features, the CDSM system determines that at least one of the one or more objects associated with the potential target is located in a path of travel of the vehicle.

[0028] In other features, the CDSM system controls the vehicle to avoid a collision with at least one of the one or more objects.

[0029] In other features, when at least one input image is received into the network backbone, the CDSM system centers a Cartesian global camera coordinate system GCCS at a center of a camera sensor field of view of the camera.

[0030] Among other features, the CDSM system defines the GCCS as: an X-axis extending along a vehicle travel path; a Y-axis orthogonal to the X-axis defining a width of the vehicle; and a Z-axis defining a tensor channel axis.

[0031] Among other features, the tensor height axis in the CDSM corresponds to the X-axis of the GCCS, and the tensor width axis in the CDSM corresponds to the Y-axis of the GCCS.

[0032] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0034] Figure 1 Depicted is a computing system implementing cross-domain spatial matching (CDSM) for monocular 3D object detection according to a non-limiting example;

[0035] Figure 2 is a block diagram depicting a system for implementing a CDSM for monocular object detection for vehicles according to a non-limiting example;

[0036] Figure 3 Depicted according to a non-limiting example Figure 1an input image to a computing system of FIG. 1 including bounding boxes drawn around four potential targets near a vehicle and a bird's eye view (BEV) grid showing potential target detection based on a Karlsruhe Institute of Technology and Toyota Technological Institutes of Chicago (KITTI) dataset;

[0037] Figure 4 Depicted according to a non-limiting example Figure 1 An input image to a computing system of FIG. 1 including bounding boxes drawn around four potential targets near a vehicle and a bird's eye view (BEV) grid showing potential target detection based on a NuScenes dataset;

[0038] Figure 5 Depicted is a computing system implementing a cross-domain spatial matching (CDSM) system for low-level sensor fusion according to a non-limiting example;

[0039] Figure 6 depicts an input model including an image portion and a point cloud portion of a CDSM system according to a non-limiting example;

[0040] Figure 7 Depicted are additional fusion techniques for a CDSM system according to non-limiting examples;

[0041] Figure 8 depicts a network for fusing aligned CDSM features from different sensors into a fused output according to a non-limiting example;

[0042] Fig. 9 Figure illustrates CDSM fusion output results from a CDSM system according to a non-limiting example;

[0043] Fig.10 FIG. 1 illustrates another CDSM fusion output result from a CDSM system according to a non-limiting example;

[0044] Fig.11 is a block diagram depicting a vehicle including a CDSM system according to a non-limiting example; and

[0045] Fig.12 A flow chart illustrating a method for performing monocular 3D object detection and / or low-level CDSM sensor fusion according to non-limiting examples is depicted.

[0046] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0047] Figure 1 A computing system 100 is illustrated that implements the CDSM layer approach for vision sensor data interpretation. Figure 1 The 3D global camera coordinate system (GCCS) and tensor dimension indicators are shown to aid understanding. Computing system 100 represents a layer or portion of a computer model or network that executes on a processor (e.g., an embedded vehicle system) to perform object detection functions for a vehicle or other embedded system application. For example, a processor, controller, or computing system on a vehicle may execute a program that includes Figure 1 A computer model or artificial neural network of the only CDSM transformation layer shown in FIG.

[0048] As described above, CDSM technology can be used to overcome the coordinate system differences between the 2D model architecture and the desired 3D domain. As shown, the computing system 100 receives image data from a camera of a vehicle. For example, at least one input image 102 is received from a camera of a vehicle, and the at least one input image 102 includes a 2D array of pixel information of the physical environment around the vehicle. In some examples, the Cartesian 3D GCCS on the right is centered on the camera sensor field of view. The X-axis can point forward (e.g., in the direction of the vehicle's travel), the Y-axis points to the left side of the camera, and the Z-axis is orthogonal to the XY plane (e.g., above and below the camera's center line of sight). A tensor of a 3D prediction grid, such as a bird's eye view (BEV) grid 116, is aligned with the GCCS, wherein the tensor height dimension corresponds to the GCCS X-axis, the tensor width corresponds to the GCCS Y-axis, and the tensor channel corresponds to the GCCS Z-axis.

[0049] Based on the 2D array of pixel information of at least one input image 102, a set of 2D image features 106 of potential targets in the physical environment can be determined. The 2D image features 106 are extracted from the input image 102 by the backbone and BiFPN (bidirectional feature pyramid network) block 104. Block 104 is configured to process at least one input image using a backbone and multiple feature pyramid network (FPN) subnetworks or subnetworks. For example, at least one input image is fed to the FPN subnetwork of block 104, and the FPN subnetwork extracts a 2D feature map based on the at least one input image. The feature map based on the 2D image is output as a set of 2D image features 106 identified from the at least one input image.

[0050] The two dimensions of the tensor of 2D image features 106 are aligned with the GCCS ZY plane, while the tensor channels are aligned with the X axis defined perpendicular to the XY plane. To reconcile these differences, a spatial transformation called CDSM is performed. The CDSM block consists of two steps, as shown in blocks 108 and 112. Rather than relying on display projections or depth estimates of at least one input image as done in these other solutions, the 2D image features 106 are identified from at least one input image. Maintaining the 2D image features 106 requires less memory than the at least one input image itself.

[0051] The CDSM rotation block 108 transforms the 2D feature map into a 3D feature map compatible with the BEV grid format. Block 108 is implemented to align the 2D feature tensor with the spatial orientation of the GCCS. This may involve performing a quaternion rotation to calculate a new index and distributing the input tensor values ​​to the rotated output tensor accordingly. The parameters and rotation order of the 2D image features 106 are 180 degrees around the first dimension (GCCS Z axis) and then 90 degrees around the second dimension (GCCS Y axis). Such a combined rotation ensures the correct orientation and alignment of the GCCS center with the 3D prediction grid tensor. For example, the 2D image features 106 are passed to the CDSM rotation block 108 to match two of the three 3D spatial representations of potential targets at different positions within the BEV grid 116. The rotation tensors from different scale levels of the FPN are aggregated on a single BEV grid 116 according to their corresponding features at that level. For example, the block 108 adapts a set of 2D image features 106 into a set of 3D image features 110 of potential targets by applying a CDSM rotation block 108 to the 2D image features 106, which aligns the horizontal (Y) axis and the vertical (Z) axis of the 2D image features 106 with the tensor height axis and the tensor width axis, respectively, of the 3D bird's-eye view as the BEV grid 116. The CDSM aggregation 112 is applied to the 3D image features 110 to extrapolate the depth information of the tensor channel axis perpendicular to the tensor width axis and the tensor height axis (e.g., along the stacked depth (X) axis of the 2D image features).

[0052] The set of 3D image features 110 is sent to a CDSM aggregation block 112 to generate a set of aggregated 3D features 114 of potential targets, which includes depth information extrapolated for the tensor channel axes. This second step involves aggregating the rotated 2D feature maps onto a BEV grid 116. By applying this technique, the correlation between the distance of an object from the sensor and the scale level at which it is detected is apparent. Therefore, close objects appear larger in terms of pixels and are detected on a smaller grid, while distant objects appear smaller in terms of pixels and are detected at a finer scale. To capture this correlation, various scale feature tensors covering the field of view of the camera sensor are aggregated and applied to the 2D image features 106 based on the distance correlation of the object at each level. For example, the 3D image features 110 are aggregated and grouped into a set of aggregated 3D features 114, which are arranged in a BEV grid 116 or other suitable format for implementing functions related to object detection based on perception sensors.

[0053] Finally, a set of 2D convolutional layers are applied to the aggregated BEV feature maps to refine the 2D image features 106 and obtain additional grid scales in the domain. These refined 2D features 106 are then passed to the 3D prediction head (e.g., Figure 2 The prediction head 202 shown in FIG. 1 is used for further processing into 3D image features 110 .

[0054] The previously described solution for 3D object detection is different from the approach taken by the computing system 100. Instead of relying on explicit projections or depth estimation, 2D image features 106 are utilized, and a CDSM rotation block 108 is used to adapt them to the 3D domain, and a CDSM aggregation block 112 further adapts them to a specific BEV grid 116. In this way, the computing system 100 enables direct prediction of objects in a tensor 3D space based on aggregated 3D features 114 derived from at least one input image, rather than consuming additional resources to process the image 102 at a single or grouped pixel level (rather than an abstract feature level as described herein).

[0055] As described below, object detection may be performed after further aggregation with other types of sensor data mapped to the same BEV grid 116. For example, in addition to monocular object detection with a camera, the CDSM layer provided by the computing system 100 may be adapted to implement low-level sensor fusion of different types of sensor data (e.g., camera and radar). Various aspects of the CDSM layer (e.g., blocks 108 and 112) may be used in these and other implementations to reduce the model and / or sensor data set to a more manageable / more suitable size for use in embedded applications (e.g., on a vehicle). That is, because no explicit projection or depth estimation is used within the computing system 100 and its entire computer model, the computing system 100 may require less memory and less complex processing resources than existing solutions. For example, one or more objects associated with a potential target may be detected based on the aggregated 3D features 114. In response to determining that at least one of the objects is near the path of travel of the vehicle, the vehicle may be controlled based on the information in the BEV grid 116 to avoid collision with the object.

[0056] The CDSM layer of the computing system 100 allows the network to learn a spatial mapping of the 2D image features 106 to apply them directly to the BEV mesh 116 as aggregated 3D features 114. This mapping and aggregation process is guided by the specific way in which 2D feature tensors from different FPN levels are aggregated onto the BEV mesh 116. The main conditioning of the computing system 100 can be performed by using trainable convolutional weights; the weights are adjusted by back-propagation through the network to match the provided 3D target.

[0057] Figure 2 An example computer model or network implemented on a vehicle system for performing a CDSM for monocular object detection is shown. Figure 2 The entire model architecture 200 is presented in.

[0058] Block 104-1 is illustrated as an example of block 104, which includes an EfficientNetV2 model used as the backbone of the architecture 200. Block 104-1 can utilize a model selected for compatibility with existing 2D image processing architectures, and its optimized depth, layer width, and resolution representation at different levels, resulting in fast inference times and satisfactory performance (e.g., for embedded vehicle applications). The official (e.g., predefined) model weights are pre-trained on ImageNet as a starting point for training. The backbone generates three feature maps at levels P3, P4, and P5, corresponding to one-eighth (1 / 8), one-sixteenth (1 / 16), and one-thirty-second (1 / 32) of the size of the input image 102-1, respectively.

[0059] The second stage of block 104-1 includes BiFPN. BiFPN is described with reference to the EfficientNetV2 model used as FPN. Figure 2 As shown, the BiFPN of block 104-1 consists of multiple iterations of complex connections between input feature maps, which aggregate and refine the 2D image features 106-1 in a more complex way than the standard U-shaped FPN. The backbone outputs P3-P5 and artificially generated P6 and P7, which use only one-sixty-fourth (1 / 64) or one-hundred and twenty-eighth (1 / 128) of the processed image 102-1, respectively. The 2D image features 106-1 can be output as five associated new feature maps of five different scales.

[0060] The single CDSM block includes a CDSM rotation block 108-1 with a CDSM aggregation block 112-1, which are examples of blocks 108 and 102, respectively. This combination enables the CDSM layer of the architecture 200 to be applied to process image features rather than image pixels, and transform potential targets into a BEV grid 116-1 for downstream processing to detect objects associated with the potential targets. For example, the BiFPN feature map 106-1 is an example of a 2D image feature 106, which is in a 2D image domain and is mapped to a 3D BEV grid space defined by the BEV grid 116-1. This transformation enables the 2D image feature 106-1 to be aligned with the desired 3D domain defined by the tensor dimensions (e.g., tensor height, width, channels).

[0061] Finally, there are two separate heads at the output of the aggregation block 112-1, called the classification head and the regression head prediction 202. The transformed 3D feature map 114-1 is processed by these heads to perform the final refinement and generate object detection. For example, the 3D prediction 202 is output from the classification head, which predicts the objectness and category score of each grid cell anchor, while the regression head predicts the 3D bounding box increment (center, size and rotation) relative to the anchor. In some cases, anchors are dynamically generated for each cell of the BEV grid 116-1, thereby generating nine anchors per cell using three size scale parameters and three ratio parameters. The anchor center can also be adjusted by the grid cell position offset. This method produces different sets of anchors distributed on five parts of a single BEV grid 116-1 or distributed on five different BEV grids 116-1, each with a different size. In fact, the architecture 200 captures a wide range of possible object geometries in the 3D prediction 202. Although object prediction can occur separately on each BEV grid 116-1, the weights of the two heads are shared across scales.

[0062] Object predictions from all levels P3 to P7 may be merged into a single list using the output of the 3D prediction 202. Downstream tracking functions, such as a Non-Max Suppression (NMS) algorithm, may be applied to the 3D prediction 202 to filter out overlapping detections. Figure 3 and Figure 4 The figure shows the results of performing the described CDSM technique using different example datasets. The model implemented by the computing system 100-1 can be trained on the KITTI and NuScenes datasets respectively. For example, the detection results on the KITTI dataset are as follows: Figure 3 As shown; the detection results on the NuScenes dataset are shown Figure 4 shown.

[0063] exist Figure 3 In FIG. 1 , computing system 100 and / or architecture 200 is implemented as computing system 100 - 1 on a vehicle. Figure 3 In FIG. 3 , an input image 302 is shown with bounding boxes 304 - 1 to 304 - 4 drawn around four different potential targets. A BEV grid 306 is shown with 3D features of potential targets 308 - 1 to 308 - 4 corresponding to the objects of the bounding boxes 304 - 1 to 304 - 4 that overlap relative to the vehicle position. Figure 4 , an input image 402 is shown with bounding boxes 404-1 to 404-5 drawn around four different potential targets. A BEV grid 406 shows 3D features of potential targets 408-1 to 408-5 that correspond to objects of bounding boxes 404-1 to 404-4 that overlap relative to the vehicle's position.

[0064] The relevant key performance indicators (KPIs) can be calculated on different test samples, e.g., samples different from those used during training. For example, Table I provides widely used metrics for measuring the performance of object detection. Table I For the KITTI dataset, the IOU 70 method recommended in the official ranking is applied, where a label prediction pair is considered a true positive if the intersection over union (IoU) of the associated 3D cuboids exceeds seventy percent (70%). This criterion is a known challenge for feature matching, especially for monocular camera-only methods, as reflected by the relatively low scores of FCOS3D and the example CDSMNet method. However, the results between the two methods are closely consistent. Additional KPI calculations can be performed using slightly less stringent association criteria (i.e., IOU50 and DIST2). Figure 3 and Figure 4 The results in show that the CDSM-based model described in this paper accurately predicts most of the objects, while the problem of low mAP scores can be attributed to the required precision of the demanding association method.

[0065] The NuScenes dataset, which is considered more challenging overall, adopts an official association method based on the distance (in meters) between the label and the predicted center, resulting in relatively better mAP scores. To adhere to the official method, mAP is calculated for four different DIST association thresholds: 0.5m, 1m, 2m, and 4m, and the final metric is the average of these values. There is a clear gap between the state-of-the-art solutions and our model; however, achieving an mAP of around 0.5 is a positive result considering the simplicity and minimalist design of our approach to such a complex task. Moreover, similar to the KITTI results, the model achieves comparable performance at lower association criteria, providing evidence that the CDMS transform solution is operating correctly.

[0066] exist Figure 3 and Figure 4 In both the image shown and the corresponding 3D view, the predicted bounding boxes are highlighted (e.g., using different colors) based on their classification as true positives, false detections, and missed objects. The association can be performed using the DIST2 metric. Additionally, a lidar point cloud dataset is included in the BEV format for reference.

[0067] The NuScenes dataset is considered more challenging than the KITTI dataset, particularly due to the presence of labeled distant objects. Despite this difficulty, the CDSM-based model described herein is still able to correctly predict most of these objects, with only some or limited inaccuracies in depth estimates. While most detections may be correct, there may be some slight inaccuracies in the regressed bounding box coordinates and sizes. For example, a car on the left is detected that is too far from the label, resulting in a false detection. Although the predictions in input image 302 and input image 402 appear to match the objects in the BEV meshes 306 and 406, the estimated depth in some parts of the 3D space may be inaccurate.

[0068] Table II presents some of the more important KPIs, including multiple parameters, floating-point operations, and GPU memory required for inference. Table II According to Table II, even considering the theoretical performance of multi-view methods from the perspective of a single camera view, the CDSM model has a great advantage in terms of resource usage. This can be attributed to the small input image size (e.g., using 2D image features instead of the entirety of image pixel information) and the simplicity of CDSM blocks 108 and 112. Nevertheless, the CDSM model achieves competitive results, enabling its adoption in a wider range of vehicles and embedded systems with lower cost and less powerful onboard computer systems.

[0069] Figure 5 An example of a CDSM computer model 500 for a low-level sensor fusion application is shown. For example, vehicles typically use multiple sensor types (e.g., radar, lidar, camera, ultrasonic) to sense objects that threaten collision while driving. The above CDSM techniques can be adapted to enable 3D image features to be presented along with other types of 3D sensor features that are aggregated on a shared BEV grid relative to the vehicle. Figure 5 The CDSM computer model 500 illustrated in FIG. 5 includes two parts that feed the CDSM fusion output. The CDSM computer model 500 represents the concept of a multi-part setup in the fusion method. A separate network architecture is used to initially process camera images and point cloud data (e.g., radar, lidar) separately. The CDSM computer model 500 includes a CDSM layer 100-2, which is an example of a CDSM layer from a computing system 100. The CDSM layer 100-2 receives an image 102-2 from a camera (e.g., a 2D camera). In addition, the model includes a point cloud or non-image-based feature extraction layer 504. The point cloud layer 504 can receive point cloud data as input from a radar or lidar sensor on a vehicle.

[0070] Layer 100-2 and layer or network 502 can be separate neural networks or separate subnetworks of a single network in the CDSM computer model 500. In some examples, these layers 100-2 and 502 can be different neural network layers that feed 3D features extracted from multiple sensors to a common sensor fusion processing pipeline that uses a common BEV grid as a reference. The image 102-2 (e.g., camera input) is processed in the 2D image domain, while the point cloud data is processed in an enhanced 3D space defined by the BEV grid. For example, layer 502 includes a voxel feature extractor (VFE) 506 and a point cloud processing network 508 for outputting multiple point clouds, a 3D feature map 510.

[0071] Both neural networks 100-2 and 502 may generate outputs and predictions in their respective domains. However, to facilitate low-level sensor fusion, the CDSM alignment block (e.g., including block 108-2 and block 112-2) feeds a CDSM fusion block 514 that fuses the 3D feature maps from the intermediate layers of networks 100-2 and 502 into a unified output of a fused prediction 516 in the 3D space of the BEV mesh.

[0072] The main challenge of fusing feature maps originating from completely different domains (e.g., 2D camera and 3D BEV) can be overcome by utilizing identifiable features after spatially aligning their feature maps before fusion using a single CDSM fusion block 514, enabling more efficient matching of features from different sensor data sets. Therefore, the CDSM fusion block 514 is used to align spatial information present in different domains of sensor data (e.g., obtained from camera images and radar point clouds). Image prediction and / or point cloud prediction are optionally provided depending on the application. For example, some functions can directly perform processing on the input image 102-2 and / or point cloud data 504 without waiting for the fused prediction 516.

[0073] Figure 6 The diagram shows input 600 to CDSM computer model 500. Input 600 includes image 602, which is an example of image 102, and radar point cloud 604, which is an example of point cloud data. Figure 6 The figure shows how the feature maps initially derived from the input 600 from the intermediate layers of each network 100-2 and 502 are misaligned. The CDSM can be used to align and fuse these sets of 3D features into a common BEV grid associated with a fused prediction 516. After each sensor domain is aligned by the CDSM or has been aligned to a common BEV grid, the CDSM fusion block 514 can apply further CDSM techniques to aggregate and output a fused prediction 516 derived from their combined sensor outputs.

[0074] Figure 7 The diagram illustrates additional concepts related to the CDSM fusion technique described herein for low-level sensor fusion of sensor data from different domains. An example of a CDSM computer model 500 is provided by a neural network 700. The neural network 700 includes a CDSM layer and a radar network point cloud layer. The CDSM layer outputs a feature map of a 2D image that is aligned with a BEV mesh associated with a vehicle and aligned with a point cloud feature map output from the radar layer.

[0075] Before fusing information from both radar and camera views, their tensor dimensions are aligned in neural network 700 to match their spatial orientation in the vehicle coordinate system (e.g., BEV grid).

[0076] To achieve this, a custom CDSM rotation layer is implemented to perform this operation. The final rotation matrix is ​​calculated using a quaternion rotation chain and applied to the tensor index via matrix multiplication. Additionally, since some rotations may produce negative index values, the new index is offset by the calculated offset to align the tensor index (0, 0, 0). Subsequently, the values ​​from the old index are collected and scattered on the rotated output tensor based on the new index. The CDSM rotation layer can be used to align the camera and radar feature map tensors with respect to the vehicle coordinate system (VCS). For example, the rotation can be applied to the camera feature tensor as follows: first, rotate 180 degrees around the first dimension (VCS Z axis) and then rotate 90 degrees around the second dimension (VCS Y axis). The combination and order of properly selected rotation operations can not only ensure that the two tensors have the same orientation relative to the VCS, but also align the centers of their VCS at the same center position. This alignment may not be achieved by any combination of input tensor dimension permutations and / or transpositions.

[0077] Figure 8 The diagram shows a network 800 that implements the CDSM fusion block and fuses aligned CDSM features from different sensors into a fused output. The CDSM fusion block can be divided into three stages: aggregation, BiFPN processing, and prediction head evaluation. The network 800 enables low-level merging of information from both camera and radar views.

[0078] First, camera feature maps at different scale levels are acquired and aggregated onto a single BEV map. This is done to detect objects of various sizes in the camera plane. Due to perspective mapping, these feature maps correspond to specific regions in the BEV domain at a certain distance from the camera sensor. The distance range of the features to each scale level is determined using the GradCAM visualization technique described above. Additionally, the relationship between the field of view of the camera sensor and the output BEV grid is taken into account during the aggregation of the feature maps. This ensures that features are not placed in 3D space that is not visible in the image.

[0079] After aggregation, a feature refinement step is performed. This step involves several 2D convolutional layers in the BEV domain. Similar to the concept of the backbone, the features are processed from detailed to more general, creating smaller grid representations of the same BEV area. This enables the extraction of relationships between different feature maps in an end-to-end manner throughout the training process, rather than manually calling them onto the model. It also generates higher-level features that capture larger areas in the BEV.

[0080] Ultimately, the result of the refinement step is a set of three different BEV grid feature maps from the camera sensor in the 3D domain, which can be directly fused with the point cloud feature map. The fusion of camera and point cloud features now becomes a relatively simple task as they have been transformed into the same coordinate system. The aggregated and refined camera feature maps in BEV correspond spatially to the feature maps obtained during the radar data processing. During the architecture design, compatible grid sizes were ensured so that the two grid tensors can be cascaded together, stacking the camera and radar features for each grid cell along the channel dimension. Another BiFPN block is applied to the cascaded feature maps at different levels to further combine the information from the two sensors into a single 3D internal representation. This representation is used in the prediction head to generate the final 3D object prediction.

[0081] Fig. 9 and Fig.10 The figure shows an example result of performing CDSM fusion according to the described techniques. Fig. 9 As shown, and the result 1000 is as Fig.10 ; Results 900 and 1000 each describe, from top to bottom, a camera-only test, a radar-only test, and a CDSM-fusion test, respectively. In obtaining results 900 and 1000, the vehicle system or simulation executed three different computer models that were trained and evaluated for performance using the NuScenes dataset. Each shows, from top to bottom, a 2D image side-by-side with a corresponding 3D BEV populated with object predictions from one of the three different models: a camera-only model, a radar-only model, and a CDSM-fusion model, as described above (e.g., in Figure 5 , Figure 6 , Figure 7 and Figure 8 Each model is trained on the NuScenes dataset and then individually evaluated based on one or more available KPIs, including official rankings (e.g., NuScenes Rank), to compare against other state-of-the-art systems used in 3D object detection.

[0082] Results 900 include images 902 and corresponding 3D BEV 904 representations generated by each individual model tested. Lidar point cloud data is shown added to the 3D BEV 904 for further reference in the BEV view. Table III includes a comparison between camera only, radar only, and different fusion models (including an example CDSM fusion model) when processing images 902 and / or 3D radar data to derive a 3D BEV 904. Table III The two different modalities are marked as C (for camera), R (for radar), or C+R (for camera and radar together) in Table III.

[0083] Results 1000 include images 1002 and corresponding 3D BEV 1004 representations generated by each individual model tested, for Fig. 9 The lidar point cloud data is shown as being added to the 3D BEV 1004 for further reference in the BEV view. Table IV includes a comparison between lidar-only, camera-only, and various fusion models, including an example CDSM fusion model that combines processing of camera and radar. Table IV When processing images 1002 and / or 3D radar data, these three different modalities are labeled C (for camera), R (for radar), L (for lidar), or C+R (for camera and radar together). Even when a single sensor model cannot predict all objects in a scene, different fusion models utilize data from both sensors to improve each of them. The CDSM fusion model described herein achieves state-of-the-art performance comparable to more complex systems that are too computationally resource-intensive for embedded (e.g., vehicle) applications.

[0084] Fig.11 An example configuration 1100 of a vehicle 1102 in which the techniques described herein may be implemented to perform tasks for monocular 3D object detection and / or low-level sensor fusion is illustrated. The vehicle 1102 includes one or more sensors 1104, one or more communication devices 1106, one or more processors 1108, and a computer-readable storage medium (CRM) 1110. The vehicle 1102 is further equipped with an autonomous driving system 1112 that relies on object information from a perception system 1114. The perception system 1114 includes a neural network 1116 trained to perform the techniques described in relation to CDSM alignment and CDSM fusion. The neural network 1116 may be divided into several layers, wherein each upper layer uses information provided by a lower layer. At least one of the layers of the neural network 1116 includes the computing system 100 and / or the architecture 200. The CRM 1110 stores instructions associated with the autonomous driving system 1112, the perception system 1114, and the neural network 1116.

[0085] Sensor 1104 provides sensor data to neural network 1106. Sensor 1104 can be mounted to any part of vehicle 1102 or integrated into any part of vehicle 1102 to provide instrument field of view. Sensor 1104 generates raw information about vehicle 1102 and surrounding physical environment (e.g., low-level information, not higher object level information). As used herein, sensor 1104 may include a camera system, a radar system, a lidar system, or an ultrasonic system to detect and track objects in the environment surrounding vehicle 1102, and detect objects in the drivable area around vehicle 1102. Radar systems can provide information about the distance, radial velocity, and azimuth of nearby objects. Lidar systems and ultrasonic systems can provide maps of reflective objects and indicate distance information associated with these objects. Camera systems can be used to provide distance information or determine object categories associated with nearby objects.

[0086] Sensors 1104 may provide sensor data about objects, including 2D camera images, radar and / or lidar point cloud data, or other forms of low-level or raw sensor data. The sensor data may, for example, indicate the relative position (e.g., distance and lateral positioning), heading, or speed of an object relative to the vehicle 1102, including stationary objects that constitute the boundaries of free space. In some cases, sensors 1104 include a combination of different types of sensors, such as a radar system and a camera, which provides a larger instrument field of view or improved detection of an object compared to using only one type of sensor.

[0087] The perception system 1114 may include a vehicle state estimator and an environmental perception system. The vehicle state estimation layer uses raw data from the sensor 1104 to obtain accurate information about the vehicle 1102, including its position, speed, orientation (e.g., yaw), or yaw rate. The environmental perception layer also uses data from the sensor 1104 to provide information about the surrounding environment, including the dynamic world (e.g., moving cars and pedestrians) and the static world (e.g., guardrails). The top layer includes: an autonomous driving system 1112, which provides features and functions that support daily driving (e.g., lane keeping assist); and a neural network 1116, which is responsible for processing sensor data from the sensor 1104 to detect objects and output a bounding box relative to the 3D space (e.g., GCCS) defined by the autonomous driving system 1112; a safe driving path from point A to point B can be inferred from the information output by the neural network 1116 to avoid collisions with any detected objects.

[0088] The communication device 1106 includes any radio or transceiver configured to send or receive vehicle signals. The communication device 1106 may include a telecommunications radio, a satellite transceiver, or the like.

[0089] The processor 1108 (e.g., an energy processing unit or an electronic control unit) may be a microprocessor or a system on a chip. The processor 1108 executes instructions stored in the CRM 1110. For example, the processor 1108 may process sensor data from the sensor 1104 by inputting an image from a 2D camera and converting the input into a set of 3D features aggregated on the BEV grid. These 3D features may be combined using CDSM fusion to determine a combined feature identified by the sensor 1104, each combined feature relative to the same reference frame of the vehicle 1102 traveling in free space. Based on the 3D features, the neural network 1116 of the perception system 1114 or other parts may be used to identify the characteristics (e.g., relative positioning, distance, and predicted trajectory) and classification of observable objects in the surrounding environment. The processor 1108 may then control the operation of the autonomous driving system 1112, the perception system 1114, and the neural network 1116 to enable the vehicle 1102 to navigate in the environment and avoid potential collisions or unsafe driving. For example, processor 1108 may control perception system 1114 to request information about an object and its location and size based on data from sensor 1104 to determine a navigation path for vehicle 1102 to avoid the object.

[0090] Fig.12 The diagram illustrates an overview of techniques associated with performing CDSM for monocular 3D object detection and / or low-level CDSM sensor fusion. Fig.12 The diagram illustrates process 1200 described in the context of execution on a vehicle 1102 using computing system 100, architecture 200, CDSM computer model 500, neural network 700, and network 800. For example, processor 1108 can be configured to execute a CDSM 2D image feature extractor and aggregator that outputs 3D features for use in BEV processing functions (e.g., assisted driving). For ease of description, process 1200 is shown as optionally used as an entry to a CDSM fusion algorithm. Process 1200 can be performed without executing the steps of the CDSM fusion algorithm (e.g., as an example implementation of monocular 3D object detection). When steps 1214 and 1216 are performed as shown to provide input to step 1210, process 1200 is an example of low-level CDSM sensor fusion that builds on the CDSM transform of the input image to generate a more comprehensive view of all 3D features, which can then be used to classify and / or detect objects.

[0091] At step 1202, at least one input image is received from a camera of a vehicle, the input image comprising a 2D array of pixel information of a physical environment. For example, input image 102 is provided to computing system 100 (eg, from a camera of sensor 1104).

[0092] At step 1204, a set of 2D image features of potential targets in the physical environment are determined from the 2D array of pixel information of the input image. For example, block 104 generates 2D image features 106, which include less information than the 2D array of pixel information. 2D image features 106 may characterize portions of the pixel array, but do not include red, green, and blue values; instead, they provide a 2D image feature map that indicates potential targets and their relative size and location relative to image 102.

[0093] At step 1206, the set of 2D image features is adapted to a set of 3D image features of the potential target. For example, block 108 applies CDSM rotation block 108 to 2D image features 106, which aligns the horizontal and vertical axes of 2D image features 106 with the tensor height axis and tensor width axis, respectively, of 3D BEV grid 116. Block 112 then applies CDSM aggregation 112 to extrapolate the depth information of the tensor channel axis perpendicular to the tensor width axis and the tensor height axis, which produces 3D image features 110.

[0094] At step 1208, a set of aggregated 3D features of the potential target is generated, the set of aggregated 3D features including the extrapolated depth information for the tensor channel axis. For example, the aggregated 3D features 114 are generated by aggregating the 3D image features 110 to fit the BEV grid 116 based on the extrapolated depth information of the 3D image features 110.

[0095] At step 1210 , one or more objects associated with a potential target are detected and / or classified based on the aggregated 3D features. For example, another vehicle may be detected from the BEV grid 116 .

[0096] At step 1212, the vehicle is controlled to avoid collision with the object. For example, in response to determining that another vehicle is included in at least one of the objects and is near the vehicle's path of travel, the processor 1108 may control the vehicle 1102 to avoid collision with the object.

[0097] At step 1214, 3D point cloud data is received from a radar or lidar of the vehicle. For example, the sensor 1104 includes a radar configured to output radar data processed by the perception system 1114. Features are extracted from the 3D point cloud data for aggregation onto the BEV grid 116.

[0098] At step 1216, the 3D image features are aggregated with the 3D point cloud features to generate a set of fused aggregated 3D features of the potential target. For example, the aggregated 3D features 114 are produced by aggregating the 3D image features 110 based on the extrapolated depth information of the 3D image features 110 to fit the BEV grid 116 in conjunction with the 3D feature map 510 output from the CDSM computer model 500. In this way, the two sensor modalities can be used in combination for object classification and / or object detection using the BEV grid 116.

[0099] Steps 1210 and 1212 may be performed in response to execution of step 1216. For example, by utilizing information derived from 3D features found in the input image 102 and point cloud data, another vehicle may be detected with greater accuracy from the BEV grid 116. The vehicle 1102 is controlled to avoid collision with objects near the path of travel.

[0100] The technology according to the present invention can be implemented according to a computer program, which can be executed on any suitable data processing device, which includes a device configured accordingly (e.g., a memory and one or more processors operably coupled to the memory). The computer program can be stored as computer executable instructions on a non-transitory computer readable medium.

[0101] The embodiments of the present disclosure may be implemented in any of a variety of forms. For example, in some embodiments, the present invention may be implemented as a computer-implemented method, a non-transitory computer-readable storage medium, or a computer system.

[0102] In some embodiments, a non-transitory computer-readable storage medium may be configured to store program instructions and / or data, where the program instructions, if executed by a computer system, cause the computer system to perform a method, such as any of the techniques described herein, or any combination of the techniques described herein, or any subset of any of the techniques described herein, or any combination of these subsets.

[0103] In some embodiments, a computing device may be configured to include a processor (or a set of processors) and a memory medium, wherein the memory medium stores program instructions, and the processor is configured to read and execute the program instructions from the memory medium, wherein the program instructions may be executed to implement any of the various techniques described herein (or any combination of the techniques described herein, or any subset of any of the techniques described herein, or any combination of these subsets). The device may be implemented in various forms.

[0104] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the present disclosure, even with respect to a single embodiment described with respect to a particular feature. Unless otherwise stated, the examples of features provided in the present disclosure are intended to be illustrative rather than restrictive. The above description is intended to encompass alternatives, modifications, and equivalents that are obvious to those skilled in the art having the benefit of the present disclosure.

[0105] The scope of the present disclosure includes any feature or combination of features disclosed herein (explicitly or implicitly), or any generalization thereof, whether or not it alleviates any or all of the problems mentioned herein.

[0106] Although various embodiments of the present disclosure are described in the foregoing description and shown in the accompanying drawings, it should be understood that the present disclosure is not limited thereto, but can be implemented in various ways within the scope of the following claims. From the foregoing description, it will be apparent that various changes can be made without departing from the scope of the present disclosure defined by the following claims.

Claims

1. A computer-implemented method for a vehicle system for performing cross-domain spatial matching (CDSM) for monocular object detection in a 3D free space of a physical environment surrounding the vehicle, the method comprising: receiving at least one input image from a camera of the vehicle into an image processing network, the at least one input image comprising a 2D array of pixel information of the physical environment; determining a set of 2D image features of potential objects in the physical environment from the 2D array of pixel information of the input image; transforming the set of 2D image features into a set of 3D image features of the potential target by applying a CDSM rotation to the 2D image features, the CDSM rotation aligning the horizontal axis and the vertical axis of the 2D image features with the tensor height axis and the tensor width axis of the 3D bird's eye view (BEV) grid, respectively, and extrapolating depth information of the tensor channel axis perpendicular to the tensor width axis and the tensor height axis by applying a CDSM aggregation to the 3D image features; generating a set of aggregated 3D features of the potential object, the set of aggregated 3D features comprising the depth information extrapolated with respect to the tensor channel axes; and One or more objects associated with the potential target are detected based on the aggregated 3D features.

2. The computer-implemented method of claim 1 , further comprising: It is determined that at least one of the one or more objects is located in a path of travel of the vehicle.

3. The computer-implemented method of claim 2, further comprising: The vehicle is controlled to avoid collision with the at least one of the one or more objects.

4. The computer-implemented method of claim 1, wherein: Receiving the at least one input image comprises placing a Cartesian global camera coordinate system GCCS at the center of a camera sensor field of view of the camera.

5. The computer-implemented method of claim 4, wherein: The GCCS includes an X-axis defined along a vehicle travel path, a Y-axis orthogonal to the X-axis, the Y-axis defining a width of the vehicle, and a Z-axis defining an axis of the tensor channel.

6. The computer-implemented method of claim 5, wherein: The tensor height axis corresponds to the X-axis of the GCCS, and the tensor width axis corresponds to the Y-axis of the GCCS.

7. The computer-implemented method of claim 5, wherein: Applying the CDSM rotation to the 2D image feature includes aligning the 2D image feature with the GCCS by rotating the 2D image feature about the Z axis by a first distance and rotating the GCCS about the Y axis by a second distance.

8. The computer-implemented method of claim 7, further comprising: Applying a 2D convolutional layer to the BEV mesh to generate refined 2D image features; as well as The refined 2D image features are passed to a 3D prediction head to process the refined 2D image features into 3D image features.

9. A computer system for performing cross-domain spatial matching (CDSM) for monocular object detection in a 3D free space of a physical environment around a vehicle, the computer system comprising: an image processing network for receiving at least one input image from a camera of the vehicle into a network backbone, the at least one input image comprising a 2D array of pixel information of the physical environment; a bidirectional feature pyramid network for determining a set of 2D image features of potential targets in the physical environment from a 2D array of pixel information of the input image; and A CDSM system is provided for transforming the set of 2D image features into a set of 3D image features of the potential target by applying CDSM rotation to the 2D image features, wherein the CDSM rotation aligns the horizontal axis and the vertical axis of the 2D image features with the tensor height axis and the tensor width axis of the 3D bird's eye view (BEV) grid, respectively, and extrapolating the depth information of the tensor channel axis perpendicular to the tensor width axis and the tensor height axis by applying CDSM aggregation to the 3D image features.

10. The computer system according to claim 9, characterized in that: The CDSM system: generates a set of aggregated 3D features of the potential target, the set of aggregated 3D features including the depth information extrapolated for the tensor channel axis; and detects one or more objects associated with the potential target based on the aggregated 3D features.

11. The computer system of claim 10, further comprising: A point cloud processing network is connected to at least one of the lidar system and the radar system, and the point cloud processing system receives 3D point cloud data representing the physical environment surrounding the vehicle from at least one of the lidar system and the radar system.

12. The computer system of claim 11, further comprising: A voxel feature extractor (VFE) is coupled to the point cloud processing network, and the point cloud processing network and the VFE generate a plurality of 3D feature maps of the physical environment around the vehicle.

13. The computer system according to claim 12, characterized in that: The CDSM system fuses the 2D image features with the 3D feature map in a CDSM fusion block to generate the set of aggregated 3D features of the potential object.

14. The computer system according to claim 13, characterized in that: The CDSM system generates a unified output prediction among the BEV grid of the potential targets.

15. The computer system according to claim 13, characterized in that: The CDSM system aligns spatial information in a first domain representing the 2D image features with spatial information in a second domain different from the first domain representing the 3D feature map in a CDSM alignment block to fuse the 2D image features with the 3D feature map.

16. The computer system according to claim 10, characterized in that: The CDSM system determines that at least one of the one or more objects associated with the potential target is located in a path of travel of a vehicle.

17. The computer system according to claim 16, characterized in that: The CDSM system controls the vehicle to avoid collision with the at least one of the one or more objects.

18. The computer system according to claim 17, characterized in that: When the at least one input image is received into the network backbone, the CDSM system centers a Cartesian global camera coordinate system GCCS at a center of a camera sensor field of view of the camera.

19. The computer system according to claim 18, characterized in that: The CDSM system defines the GCCS as: an X-axis extending along a vehicle travel path; a Y-axis orthogonal to the X-axis defining a width of the vehicle; and a Z-axis defining the tensor channel axis.

20. The computer system according to claim 19, characterized in that The tensor height axis in the CDSM corresponds to the X-axis of the GCCS, and the tensor width axis in the CDSM corresponds to the Y-axis of the GCCS.

Images