Efficient cloud-based dynamic multi-vehicle bird's eye view (BEV) feature fusion for extended robust co-awareness

Abstract

An example system includes one or more memories to store meshless vehicle data and one or more processors configured to determine one or more first features from first vehicle data of the vehicle data, the first vehicle data being from a first vehicle of a plurality of vehicles. The one or more processors are configured to determine one or more second features from second vehicle data of the vehicle data, the second vehicle data being from a second vehicle of the plurality of vehicles. The one or more processors are configured to fuse the one or more first features and the one or more second features to generate fused features. The one or more processors are configured to generate a BEV representation based on the fused features.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 608,250, filed March 18, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to systems, including systems for collaborative sensing of multiple devices, such as multiple vehicles. Background Technology

[0003] Automated vehicles are vehicles configured to sense their surroundings (such as the presence and location of other objects) and operate without human control or with limited human control. Automated vehicles may include light detection and ranging (LiDAR) systems, camera systems, and / or other sensor systems for sensing data indicating the presence and location of other objects in the driver's environment. In some examples, such automated vehicles may be referred to as self-driving vehicles. Vehicles with advanced driver assistance systems (ADAS) are vehicles that include systems that assist a driver in operating the vehicle (such as parking or driving it). For safety and / or other reasons, automated vehicles may or may not share sensors or other data with other automated vehicles. Summary of the Invention

[0004] This disclosure relates in general to techniques and apparatus for processing data from multiple sensor systems of multiple devices (e.g., vehicles, robots, virtual reality (VR) devices, etc.) to compress and fuse extended and robust collaborative perception of spatial features in an environment for the multiple devices. While the techniques disclosed herein are primarily discussed in relation to vehicles, it should be understood that these techniques are adaptable for use with other devices, such as robots, VR devices, or other devices that may desire collaborative perception of space.

[0005] Single-agent (e.g., single-vehicle) multi-sensor BEV frameworks have some limitations that can impact performance in certain scenarios. For example, due to the number and location of cameras in the device's camera system, a single-agent BEV framework may have limited resolution and coverage. This limited resolution can lead to less accurate object detection, missed objects, and incomplete or inaccurate segmentation compared to multi-agent (e.g., multi-vehicle) frameworks. Single-agent LiDAR and camera-based BEV frameworks can also have limited accuracy and scalability in adverse weather conditions. Therefore, leveraging collaborative BEV perception using multi-sensor and multi-vehicle systems is desirable.

[0006] However, cooperative (e.g., collaborative) BEV perception has its own challenges. Cooperative perception of the environment surrounding autonomous vehicles (AVs) using vehicle-to-vehicle (V2V) and vehicle-to-roadside unit (RSU) (V2X) interactions is being developed. Top-down “bird’s-eye view” (BEV) sensor data can be used, but existing technologies do not handle adaptive fusion of multimodal (e.g., camera-LIDAR) sensor combinations. Additionally, the variable resolution of the BEV mesh across different sensor systems and different vehicles is a complicating factor.

[0007] The technology disclosed herein offers numerous advantages over existing technologies. For example, the technology disclosed herein can provide support for heterogeneous transportation. The technology described herein allows different types of transportation to participate in collaborative perception tasks regardless of their perception capabilities. Each transportation can contribute a shared map (e.g., a neighborhood map) using its own sensor data, and the system, residing on one or more servers in a cloud computing environment, can perform inferences to obtain shared perception of the environment.

[0008] The techniques disclosed herein can provide communication efficiency. In some examples, the system can send compact BEV features instead of raw sensor data to the vehicle. Therefore, the techniques disclosed herein can reduce or minimize communication overhead between the vehicle and the system. Additionally, using soft BEV features (described in further detail later herein) can further reduce the amount of data transmitted because the probability or likelihood of the object's presence or object attributes is more compact than that of hard features and / or raw BEV feature data.

[0009] The techniques disclosed herein can provide robustness and accuracy on BEV maps. By fusing sensor data from multiple vehicles and performing inference on one or more servers in a cloud computing environment, the techniques disclosed herein can improve the robustness and accuracy of shared maps. Vehicles with incomplete or noisy sensor data can benefit from information provided to the system by other vehicles, thereby obtaining a more comprehensive and accurate representation of the environment, thus increasing navigation safety for all vehicles utilizing shared maps.

[0010] The techniques disclosed herein provide privacy and security. Using feature normalization and compression, along with communication encryption, can improve the privacy and security of data transmitted between vehicles and the cloud. Privacy and security are concerns given the sensitive nature of sensor data and the potential for malicious attacks.

[0011] Overall, the technology disclosed herein provides a scalable, efficient, and secure approach to collaborative perception in multi-vehicle environments.

[0012] In one example, a system includes: one or more memories for storing vehicle data from each of a plurality of vehicles, the vehicle data being meshless; and one or more processors communicating with the one or more memories, the one or more processors being configured to: determine one or more first features from first vehicle data in the vehicle data, the first vehicle data being from a first vehicle of the plurality of vehicles; determine one or more second features from second vehicle data in the vehicle data, the second vehicle data being from a second vehicle of the plurality of vehicles; fuse the one or more first features and the one or more second features to generate fused features; and generate a BEV representation based on the fused features.

[0013] In another example, one approach includes: obtaining vehicle data from each of a plurality of vehicles, the vehicle data being gridless; determining one or more first features from first vehicle data, the first vehicle data being from a first vehicle of the plurality of vehicles; determining one or more second features from second vehicle data, the second vehicle data being from a second vehicle of the plurality of vehicles; fusing the one or more first features and the one or more second features to generate a fused feature; and generating a BEV representation based on the fused feature.

[0014] In another example, a computer-readable medium stores instructions that, when applied by processing circuitry, cause the processing circuitry to obtain vehicle data from each of a plurality of vehicles, the vehicle data being meshless; determine one or more first features from first vehicle data, the first vehicle data originating from a first vehicle among the plurality of vehicles; determine one or more second features from second vehicle data, the second vehicle data originating from a second vehicle among the plurality of vehicles; fuse the one or more first features and the one or more second features to generate a fused feature; and generate a BEV representation based on the fused feature.

[0015] In another example, a system includes: components for obtaining vehicle data from each of a plurality of vehicles, the vehicle data being meshless; components for determining one or more first features from first vehicle data in the vehicle data, the first vehicle data being from a first vehicle of the plurality of vehicles; components for determining one or more second features from second vehicle data in the vehicle data, the second vehicle data being from a second vehicle of the plurality of vehicles; components for fusing one or more first features and one or more second features to generate a fused feature; and components for generating a BEV representation based on the fused feature.

[0016] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the description, drawings, and claims. Attached Figure Description

[0017] Figure 1 This is a block diagram illustrating an example processing system according to one or more techniques of this disclosure.

[0018] Figure 2 This is a conceptual diagram illustrating multiple vehicles, each with its own corresponding BEV grid.

[0019] Figure 3 This is a block diagram of an example single-agent BEV converged architecture.

[0020] Figure 4 This is a block diagram of an example collaborative multi-vehicle late-stage BEV fusion architecture.

[0021] Figure 5 This is a block diagram of an example collaborative multi-vehicle dynamic BEV mid-term feature fusion architecture based on one or more aspects of this disclosure.

[0022] Figure 6 This is a conceptual diagram illustrating sensor data captured by two example vehicles according to one or more aspects of this disclosure.

[0023] Figure 7 This is a block diagram illustrating an example architecture of a collaborative multi-vehicle dynamic BEV mid-term feature fusion system according to one or more aspects of this disclosure.

[0024] Figure 8 This is a conceptual diagram illustrating a multimodal dynamic mask / frustum generator according to one or more aspects of this disclosure.

[0025] Figure 9 This is a block diagram illustrating a second encoder-decoder architecture for processing image data and position data in a continuous space to generate an output, according to one or more techniques of this disclosure.

[0026] Figure 10 This is a block diagram illustrating an example vehicle radial neighborhood embedding system according to one or more aspects of this disclosure.

[0027] Figure 11 This is a block diagram illustrating an example dynamic multimodal mesh fusion system according to one or more aspects of this disclosure.

[0028] Figure 12 This is a block diagram illustrating an example of trajectory-sensitive subsampling according to one or more aspects of this disclosure.

[0029] Figure 13 This is a flowchart illustrating an example communication between a C2C server and a vehicle according to one or more aspects of this disclosure.

[0030] Figure 14 This is a flowchart illustrating an example of a multi-vehicle (BEV) feature fusion technique according to one or more aspects of this disclosure. Detailed Implementation

[0031] Multi-sensor systems, such as cameras and light detection and ranging (LiDAR) systems, can be used in a wide variety of robotics, transportation, and virtual reality (VR) applications. One such transportation application is Advanced Driver Assistance Systems (ADAS). ADAS is a system that utilizes multi-sensor systems, such as cameras and LiDAR sensor systems, to improve driving safety, comfort, and overall vehicle performance. Such systems combine the strengths of both sensors to provide a more comprehensive view of the vehicle's surroundings, enabling ADAS to better assist drivers in various driving scenarios.

[0032] In some examples, camera-based systems are responsible for capturing and processing high-resolution images in real time. The output images from such camera-based systems can be used in applications such as depth estimation, object detection, and / or pose detection, including object detection and recognition, such as other vehicles, pedestrians, traffic signs, and lane markings. Cameras may be particularly adept at capturing color and texture information, which is useful for accurate object recognition and classification.

[0033] LiDAR sensors emit laser pulses to measure the distance, shape, and relative speed of objects around a vehicle. LiDAR sensors provide three-dimensional (3D) data, enabling ADAS to create detailed maps of the surrounding environment. LiDAR can be particularly effective in low-light or adverse weather conditions where camera performance may be hampered. In some examples, the output of a LiDAR sensor can be used as partial ground truth data to determine neural network-based depth information on the corresponding camera image.

[0034] By fusing data collected from both cameras and LiDAR sensors, ADAS, or another system, can provide enhanced environmental perception and improved decision-making capabilities. This enables a variety of driver assistance features, such as adaptive cruise control, lane keeping assist, pedestrian detection, automatic emergency braking, and parking assist. Combined systems can also contribute to the development of semi-autonomous and fully autonomous driving technologies, resulting in a safer and more efficient driving experience.

[0035] However, single-agent (e.g., single-vehicle) multi-sensor BEV frameworks have some limitations that can affect performance in certain scenarios. For example, a single-agent BEV framework may have limited resolution due to the number and location of cameras in the vehicle's camera system. This limited resolution may result in lower accuracy for object detection and segmentation compared to multi-agent (e.g., multi-vehicle) frameworks. A single-agent (or limited-number-of-cameras) BEV framework may also have limited coverage. For example, a single or limited number of cameras on a single vehicle may not be able to capture the entire field of view required for accurate BEV segmentation. This can lead to missed objects or incomplete segmentation. Single-agent BEV frameworks may also have limited accuracy in complex scenes. For example, such frameworks may struggle to accurately segment objects in complex scenes that include occlusion and / or crowded environments. Single-agent LiDAR and camera-based BEV frameworks may have limited accuracy in adverse weather conditions. For example, single-agent LiDAR and camera-based BEV frameworks can be affected by adverse weather conditions such as rain, snow, or fog, which can reduce accuracy and make object detection more difficult. Single-agent BEV frameworks may also have limited scalability. For example, a single-agent BEV framework may have lower scalability than a multi-agent framework because adding additional sensors may not necessarily improve the overall performance of the framework. Single-agent multi-sensor BEVs have limitations including limited resolution, coverage, accuracy, and scalability. Therefore, leveraging collaborative BEV perception using multiple sensors and multi-vehicle systems is desirable.

[0036] Collaborative perception of the surrounding environment of autonomous vehicles (AVs) using vehicle-to-vehicle (V2V) and vehicle-to-roadside unit (RSU) (V2X) interactions is under development. Top-down bird's-eye view (BEV) sensor data can be used, but existing technologies do not handle adaptive fusion of multimodal (e.g., camera-LIDAR) sensor combinations. Additionally, the variable resolution of the BEV mesh is a complicating factor.

[0037] Generally, collaborative sensing can be categorized into three distinct levels: early collaboration, mid-stage collaboration, and late collaboration. In early collaboration, different vehicles can share raw, collected data. In mid-stage collaboration, different vehicles can process the data to extract features and share the extracted features. In late collaboration, different vehicles can process the data and features to generate outputs (e.g., BEV grids) and share the outputs. Mid-stage collaboration may include data whose geometric features are relatively easy to compress and maintain. BEV features typically include common representations across different vehicles.

[0038] Now we will discuss cooperative adaptive feature fusion. Mid-level feature fusion is an important issue when considering dynamically interacting connected automated vehicles (CAVs). Qiao, D. and Zulkernine, F.'s paper "Adaptive Feature Fusion for Cooperative Perception using LiDAR Point Clouds" (available at arxiv.org / pdf / 2208.00116.pdf) (hereinafter referred to as "Qiao et al.") published at Queen's University, Canada, describes different adaptive late-level fusion techniques while maintaining a maximum static number of interacting CAVs to avoid having to process a dynamic number of CAVs. However, the authors do not mention adaptive fusion in the case of multimodal camera-LiDAR, where there are various sensor combinations across vehicles (camera-LiDAR, camera-radar, camera-camera, LiDAR-LiDAR, LiDAR-radar, etc.) and do not mention variable resolution of the BEV mesh that can indicate or be based on the sensor resolution of different sensors.

[0039] However, enhancing perception tasks such as 3D detection or segmentation by dynamically fusing BEV features across interactive vehicles with multimodal sensor sets (e.g., cameras / LiDAR / radar, possibly equipped with different sets of sensors) presents challenges. Therefore, addressing several of these challenges is desirable when implementing multi-vehicle BEV systems.

[0040] Challenges in implementing a multi-vehicle BEV system may include performing relatively early feature fusion, handling variations in the self-attitude orientation and position of multiple vehicles, overcoming differences in BEV mesh resolution between vehicles (e.g., due to sensor ensemble characteristics, LiDAR range, camera and LiDAR resolution), handling dynamic objects between sensor inputs from different vehicles, and handling distortion aspects and thus variations in BEV features caused by different positions of objects in the field of view and / or in the cameras in a multi-camera setup across different vehicles.

[0041] According to the technology disclosed herein, a multi-vehicle BEV system may include relative early dynamic BEV feature fusion, which integrates BEV features obtained from adjacent vehicles. The system can establish a unified representation of the BEV features that incorporates information from multiple CAV BEV grids and is usable by multiple CAVs. The system can handle changes in attitude within the vehicle (e.g., rotation and translation). For example, given a neighborhood within a CAV, the CAV may have different relative changes in rotation and translation. Such a neighborhood may include a geographic area around one or more CAVSs or RSUs in a CAVS, including, for example, the current location of each cooperating CAV in a cooperating CAV. To successfully fuse features, the system can align the grid across a special Euclidean group 3 (SE3) transformation group, even if all CAVs have the same BEV grid size and resolution.

[0042] This system can fuse BEV features from grids with different resolutions. For example, each vehicle within the group may have a different BEV grid resolution, which refers to the size and granularity of the grid used to represent the area around the vehicle. These variations can be factored in to ensure consistent and accurate fusion of BEV features.

[0043] This system can handle dynamic objects between vehicle perception inputs. For example, when a vehicle moves across an area, the scene may contain dynamic objects such as pedestrians or other vehicles that may not be part of the group. These dynamic objects can introduce complexity when fusing BEV features because their position and appearance can change between perception inputs from two or more vehicles.

[0044] This system can handle variations in distortion and camera settings. For example, in a multi-camera setup, the distortion and camera parameters may differ between vehicle pairs. Objects located at different positions within the field of view (FOV) can cause different distortions (e.g., positive and / or negative scaling based on CAV relative motion), thus affecting the BEV features acquired from the cameras. Accurate fusion of features between vehicles may depend on handling these variations.

[0045] This disclosure describes real-time cooperative perception using a BEV fusion model of an autonomous driving system. The techniques disclosed herein allow different vehicles to have different sensor combinations (e.g., multiple cameras only, multiple cameras and LiDAR, LiDAR only, cameras and radar, etc.), and allow BEV meshes of multiple vehicles to intersect. This disclosure as a whole relates to techniques and apparatus for generating BEV features based on multiple vehicles. It should be noted that the techniques disclosed herein can be implemented in one or more servers, one or more RSUs, and / or one or more vehicles.

[0046] Figure 1 This is a block diagram illustrating an example processing system 100 according to one or more technologies of this disclosure. Processing system 100 can be used in vehicles, such as autonomous vehicles or driver-assisted vehicles (e.g., vehicles with ADAS or "self-driving vehicles"). In such examples, processing system 100 may represent ADAS. In other examples, processing system 100 can be used in robotics applications, VR applications, or other kinds of applications that may include multiple sensor systems, such as cameras and LiDAR systems. The technologies of this disclosure are not limited to vehicle applications. The technologies of this disclosure can be used by any system application that processes data from multiple devices or systems.

[0047] The processing system 100 may include a LiDAR system 102, a camera 104, a controller 106, one or more sensors 108, an input / output device 120, a wireless connectivity component 130, and a memory 160. The LiDAR system 102 may include one or more light emitters (e.g., lasers) and one or more light sensors. In some cases, the LiDAR system 102 may be deployed in or around a vehicle. For example, the LiDAR system 102 may be mounted on the roof of a vehicle, in the bumper of a vehicle, and / or in other locations on the vehicle. The LiDAR system 102 may be configured to emit light pulses and sense light pulses reflected from objects in the environment. The LiDAR system 102 is not limited to deployment in or around a vehicle. The LiDAR system 102 may be deployed in or around another object.

[0048] In some examples, one or more light emitters of the LiDAR system 102 may emit such pulses in a 360-degree field around the vehicle to detect objects within the 360-degree field by detecting the reflected pulses using one or more light sensors. For example, the LiDAR system 102 may detect objects in front of, behind, or beside the LiDAR system 102. Although described herein including the LiDAR system 102, it should be understood that another distance or depth sensing system may be used instead of the LiDAR system 102. The output of the LiDAR system 102 is referred to as a point cloud or point cloud frame.

[0049] A point cloud frame output by LiDAR system 102 is a collection of 3D data points representing the surface of an object in the environment. The LiDAR processing circuitry of LiDAR system 102 can generate one or more point cloud frames based on one or more light signals emitted by one or more light emitters of LiDAR system 102 and one or more reflected light signals sensed by one or more light sensors of LiDAR system 102. These points are generated by measuring the time it takes for a laser pulse to travel from the light emitter to the object and back to the photodetector. Each point in the cloud has at least three attributes: x, y, and z coordinates, which represent its position in a Cartesian coordinate system. Some LiDAR systems also provide additional information for each point, such as intensity, color, and classification.

[0050] Intensity (also known as reflectivity) is a measure of the strength of the returned laser pulse signal at each point. The value of the intensity attribute depends on various factors, such as the reflectivity of the object's surface, the distance from the sensor, and the angle of incidence. Intensity values ​​can be used for several purposes, including distinguishing different materials and enhancing visualization: intensity values ​​can be used to generate grayscale images of point clouds, thus helping to highlight structure and features in the data.

[0051] Color information in point clouds is typically obtained from other sources, such as a digital camera mounted on the same platform as the LiDAR sensor, and then combined with the LiDAR data. In some examples, the camera used to capture the color information of the point cloud data may be separate from camera 104. Color attributes include color values ​​(e.g., red, green, and blue (RGB)) for each point. Color values ​​can be used to improve visualization and help enhance classification (e.g., color information can help classify objects and features in a scene, such as vegetation, buildings, and roads).

[0052] Classification is the process of assigning each point in a point cloud to a particular category or class based on its characteristics or its relationship to other points. Classification attributes can be integer values ​​representing the category of each point, such as ground, vegetation, buildings, water, etc. Various algorithms can be used to perform classification, typically relying on machine learning techniques or rule-based methods.

[0053] Camera 104 may include any type of camera configured to capture video or image data of the environment surrounding processing system 100 (e.g., around a vehicle). In some examples, processing system 100 may include multiple cameras 104. For example, camera 104 may include forward-facing cameras (e.g., front bumper camera, windshield camera, and / or dashcam), rear-facing cameras (e.g., reversing camera), and side-facing cameras (e.g., cameras mounted in side mirrors). Camera 104 may include color cameras or grayscale cameras. In some examples, camera 104 may be a camera system including more than one camera sensor. Sensor 108 may include radar sensors, position sensors, sonar sensors, infrared cameras, and / or time-of-flight (ToF) cameras.

[0054] In some examples, LiDAR system 102 may be configured to collect 3D point cloud frames 166. In some examples, camera 104 may be configured to collect 2D camera images 168. The importance of data input modalities (such as 3D point cloud frames 166 and 2D camera images 168) to indicate one or more characteristics of objects in a 3D environment can vary. For example, when color and texture are important characteristics of a first object and when color and texture are not important characteristics of a second object, 2D camera images 168 may be more important for identifying characteristics of the first object than 3D point cloud frames 166 used to identify characteristics of the second object. When generating BEV features corresponding to 3D point cloud frames 166 and / or generating BEV features corresponding to 2D camera images 168, it may be beneficial to consider the importance of 3D point cloud frames 166 and 2D camera images 168 to indicate characteristics of the 3D environment.

[0055] The wireless connectivity component 130 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. The wireless connectivity component 130 is further connected to one or more antennas 135. The processing system 100 may communicate with the processing systems of external processing systems and / or other devices (e.g., other vehicles) via the wireless connectivity component 130.

[0056] The processing system 100 may also include one or more input / output devices 120, such as a screen, touch surface (including a touch display), physical buttons, speakers, microphones, etc. Input / output devices 120 (e.g., they may include I / O controllers) manage input and output signals for the processing system 100. In some cases, input / output devices 120 may represent physical connections or ports to external peripheral devices. In some cases, input / output devices 120 may utilize an operating system. In other cases, input / output devices 120 may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, input / output devices 120 may be implemented as part of a processor (e.g., the processor of processing circuitry 110). In some cases, a user may interact with the device via input / output devices 120 or via hardware components controlled by input / output devices 120.

[0057] Controller 106 may be an automated or driver assistance controller (e.g., ADAS) configured to control the operation of processing system 100 (e.g., including the operation of a vehicle). For example, controller 106 may control the vehicle to accelerate, brake, and / or navigate through its surroundings. Controller 106 may include one or more processors, such as processing circuitry 110. Controller 106 is not limited to controlling vehicles. Controller 106 may additionally or alternatively control any kind of controllable device, such as robotic components. Processing circuitry 110 may include one or more central processing units (CPUs), such as single-core or multi-core CPUs, graphics processing units (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), neural processing units (NPUs), multimedia processing units, etc. Instructions applied by processing circuitry 110 may be loaded, for example, from memory 160 and may cause processing circuitry 110 to perform operations attributed to processors in this disclosure.

[0058] NPUs are typically configured to implement dedicated circuitry for the control and arithmetic logic used to execute machine learning algorithms, such as those for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RF), kernel methods, etc. NPUs are sometimes alternatively referred to as neural signal processors (NSPs), tensor processing units (TPUs), neural network processors (NNPs), intelligent processing units (IPUs), or vision processing units (VPUs).

[0059] The processing circuitry 110 may also include one or more sensor processing units associated with the LiDAR system 102, camera 104, and / or sensor 108. For example, the processing circuitry 110 may include one or more image signal processors associated with camera 104 and / or sensor 108, and / or a navigation processor associated with sensor 108, which may include satellite-based positioning system components (e.g., Global Positioning System (GPS) or Global Navigation Satellite System (GLONASS)) and inertial positioning system components. Sensor 108 may include a direct depth sensing sensor capable of determining the depth or distance to objects in the environment surrounding the processing system 100 (e.g., around a vehicle).

[0060] The processing system 100 also includes a memory 160, which represents one or more static and / or dynamic memories, such as dynamic random access memory, flash-based static memory, etc. In this example, the memory 160 includes computer-executable components that can be used by one or more of the aforementioned components of the processing system 100.

[0061] Examples of memory 160 include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disc ROM (CD-ROM), and / or another type of hard disk. Examples of memory 160 include solid-state memory and / or hard disk drive. In some examples, memory 160 is used to store computer-readable, computer-executable software including instructions that, when applied, cause one or more processors to perform the various functions described herein. In some cases, among others, memory 160 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates on the memory cells of memory 160. For example, the memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within memory 160 store information in the form of logical states.

[0062] The processing system 100 may be configured to perform techniques for extracting features from image data and location data, processing those features, fusing those features, or any combination thereof. In some examples, the processing system 100 may perform one part of such techniques, while an external processing system 180 may perform another part of such techniques.

[0063] Processing circuitry 110 may include a BEV unit 140. The BEV unit 140 may be implemented using software, firmware, and / or any combination of hardware described herein. As will be described in more detail below, the BEV unit 140 may be configured to receive a plurality of 2D camera images 168 captured by camera 104; and to receive a plurality of 3D point cloud frames 166 captured by LiDAR system 102. The BEV unit 140 may be configured to receive the 2D camera images 168 directly from camera 104 and LiDAR system 102, or from memory 160, respectively. In some examples, the plurality of 3D point cloud frames 166 may be referred to herein as “position data.” In some examples, the plurality of 2D camera images 168 may be referred to herein as “image data.”

[0064] In the case of a single agent system, BEV unit 140 can fuse features corresponding to multiple 3D point cloud frames 166 and features corresponding to multiple 2D camera images 168 to combine image data corresponding to one or more objects in 3D space and location data corresponding to one or more objects. For example, each of the multiple 2D camera images 168 may include a 2D pixel array that includes image data corresponding to one or more objects. Each of the multiple 3D point cloud frames 166 may include a 3D multidimensional point array corresponding to one or more objects. Since one or more objects are located in the same 3D space where processing system 100 is located, it may be beneficial to fuse features in the image data present in the 2D camera images 168 that indicate information corresponding to the identity of one or more objects; and features in the location data present in the 3D point cloud frames 166 that indicate the location of one or more objects in 3D space. This is because image data may include at least some information not included in location data, and location data may include at least some information not included in image data.

[0065] Compared to analyzing the features of image data and the features of location data separately, fusing the features of image data and the features of location data can provide a more comprehensive view of the 3D environment corresponding to the processing system 100. For example, multiple 3D point cloud frames 166 can indicate an object in front of the processing system 100, and the BEV unit 140 may be able to process multiple 3D point cloud frames 166 to determine that the object is a traffic light. This is because multiple 3D point cloud frames 166 can indicate that the object comprises three circular components oriented vertically and / or horizontally relative to the surface of the road intersection, and that the multiple 3D point cloud frames 166 can indicate that the size of the object is within the size range typically occupied by a traffic light. However, multiple 3D point cloud frames 166 may not include information indicating which of the three lights of the traffic light is on and which of the three lights is off. For example, a 2D camera image 168 may include image data indicating that the green light of the traffic light is on. This means that fusing the features of image data and the features of location data can be beneficial, allowing the BEV unit 140 to analyze the image data and location data to determine the characteristics of one or more objects within the 3D environment.

[0066] Fusing BEV features from image data and BEV features from location data can involve associating BEV features from image data with BEV features from location data corresponding to the image data features. For example, processing system 100 can fuse BEV features from image data indicating the color and identity of a traffic light with BEV features from location data indicating the location of the traffic light. This means that the fused set of BEV features can include information important to the generated output from both the image data and location data corresponding to the traffic light. Some systems can fuse BEV features from image data and BEV features from location data by generating a “mesh” of BEV features from image data and BEV features from location data and fusing these meshes. The BEV feature mesh can correspond to 2D BEVs in a 3D environment. Each “cell” of the BEV feature mesh can include features corresponding to a portion of the 3D environment that corresponds to that cell. This allows the system to fuse BEV features from image data with BEV features from location data corresponding to the same portion of the 3D environment. In some examples, the mesh itself (e.g., rather than the BEV features that may be within the mesh) can be referred to as mesh information.

[0067] BEV mesh resolution can be chosen in a manner similar to selecting the stereo pixel mesh size in a point cloud DNN architecture to reduce memory consumption. In some examples, a fixed mesh size can be 128 cells by 128 cells, where each cell in the mesh has a resolution of 80 centimeters (cm). Low-resolution BEV meshes can lead to the extraction of coarse features with poor resolution. This can result in poor image detection. Features extracted from BEV meshes may be sensitive to mesh resolution. Poor resolution can lead to poor detection of pedestrians, traffic signs, and other thin, flat objects, while lanes / road segments with larger surface areas in BEVs can be detected better. BEV fusion models can use upsampling layers to handle low-resolution meshes.

[0068] According to the technology disclosed herein, a cooperative multi-agent (e.g., multi-vehicle) system can provide improved resolution to vehicles with lower-resolution BEV systems, as features from higher-resolution systems from other vehicles can be fused into a shared neighborhood map that retains higher resolution. The shared neighborhood map may include a geographic area surrounding one or more CAVSs or RSUs in the CAVS, including, for example, a representation of the current location of each cooperative CAV in the cooperative CAVs.

[0069] External processing system 180 may represent one or more servers and / or roadside units in a cloud computing environment. External processing system 180 may obtain (e.g., receive) data from processing system 100 and similar processing systems and process the received data, for example, to fuse BEV characteristic data from multiple vehicles.

[0070] It should also be noted that in some examples, processing system 100 may obtain (e.g., receive) data from another processing system 100 and similar processing systems and process the received data, for example, to fuse BEV characteristic data from multiple vehicles, including, for example, the vehicle in which processing system 100 is located. In such cases, BEV unit 140 may also include a multi-vehicle BEV unit, such as the multi-vehicle BEV unit 194 discussed below.

[0071] External processing system 180 may include processing circuitry 190, which may be any type of processor described above with respect to processing circuitry 110. Processing circuitry 190 may include a multi-vehicle BEV unit 194 configured to fuse BEV features from multiple vehicles for cooperative perception. Processing circuitry 190 may obtain data from controller 106 or from memory 160. External processing system 180 may also include memory 198 configured to store data obtained from processing system 100 and other similar processing systems. Memory 198 may also be configured to store training data (similar to training data 170) and model outputs (similar to model output 172) for encoders, decoders, or other models that are part of multi-vehicle BEV unit 194. Memory 198 may include any type of memory described above with respect to memory 160.

[0072] Wireless connectivity component 182 facilitates communication between external processing system 180 and processing system 100. Wireless connectivity component 182 may include sub-components for, for example, third-generation (3G) connectivity, fourth-generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth-generation (5G) connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards.

[0073] The Multi-Vehicle Vehicle (BEV) unit 194 can generate a fused BEV representation based on information received from processing system 100 and other similar systems. Processing circuitry 190 can transmit or transfer the fused BEV representation to processing system 100 via wireless connectivity component 182. Processing circuitry 110 can apply control unit 142 to control a device (e.g., a vehicle, robotic arm, or other device) corresponding to processing system 100 based on the fused BEV representation from external processing system 180. In some examples, the fused BEV representation can be further processed by BEV unit 140. Control unit 142 can control the device based on information included in the fused BEV representation related to one or more objects within a 3D space including processing system 100. For example, the fused BEV representation and / or the output of BEV unit 140 may include the identifiers of one or more objects, the positions of one or more objects relative to processing system 100, the motion characteristics of one or more objects (e.g., velocity, acceleration), or any combination thereof. Based on this information, control unit 142 can control the device corresponding to processing system 100. The fused BEV representation can be stored in memory 160 as model output 172.

[0074] In some examples, processing circuitry 110 may be configured to train one or more encoders, decoders, or any combination thereof applied by BEV unit 140 using training data 170. For example, training data 170 may include one or more training point cloud frames and / or one or more camera images. Training data 170 may additionally or alternatively include features known to accurately represent one or more point cloud frames and / or features known to accurately represent one or more camera images. This allows processing circuitry 110 to train one or more encoders to generate features accurately representing point cloud frames and to train one or more encoders to generate features accurately representing camera images. Processing circuitry 110 may also use training data 170 to train one or more decoders. In some examples, training data 170 may be stored separately from processing system 100. In some examples, processing circuitry other than processing circuitry 110 and / or processing circuitry 190 and separate from processing system 100 may use training data 170 to train one or more encoders, decoders, or any combination thereof applied by BEV unit 140.

[0075] In some examples, processing circuitry 190 may be configured to train one or more encoders, decoders, or any combination thereof applied by the multi-vehicle BEV unit 194 using training data (such as training data 170). For example, training data 170 may include one or more training point cloud frames and / or one or more camera images. Training data 170 may additionally or alternatively include features known to accurately represent one or more point cloud frames and / or features known to accurately represent one or more camera images. This allows processing circuitry 190 to train one or more encoders to generate features accurately representing point cloud frames and to train one or more encoders to generate features accurately representing camera images. Processing circuitry 190 may also use training data 170 to train one or more decoders. In some examples, training data 170 may be stored in memory 198.

[0076] Figure 2 This is a conceptual diagram illustrating multiple vehicles, each with its own corresponding BEV grid. Figure 2In the example, vehicles 212, 214, 216, and 218 are shown traveling on street 220. Vehicle 212 has an associated BEV grid 202 that can be generated by one or more processors (e.g., processing circuitry 110 and / or processing circuitry 190) based on sensor data of vehicle 212. For example, vehicle 212 may include a LiDAR system 102, camera 104, and / or sensor 108 that can capture information (e.g., image data, point cloud data, etc.) that can be fused to establish BEV grid 202. Similarly, vehicle 214 has an associated BEV grid 204 that can be generated by one or more processors based on sensor data of vehicle 214. Vehicle 216 has an associated BEV grid 206 that can be generated by one or more processors based on sensor data of vehicle 216. Vehicle 218 has an associated BEV grid 208 that can be generated by one or more processors based on sensor data of vehicle 218. BEV grids 202, 204, 206, and 208 may differ in terms of grid cell size, number of grid cells, and / or overall size, for example, because Figure 2 Differences in resolution and / or settings of the sensor systems of the vehicles. In the example where each of vehicles 212, 214, 216, and 218 is a single agent, one or more processors that generate each of BEV meshes 202, 204, 206, and 208 are typically included in one or more processors on the respective associated vehicle.

[0077] Vehicle 212 is also shown, having a travel vector 222 indicating speed and direction of travel. Vehicle 214 has a travel vector 224, vehicle 216 has a travel vector 226, and vehicle 218 has a travel vector 228. As time progresses, vehicles 212, 214, 216, and 218 move relative to each other according to their respective travel vectors. Additionally, when vehicles are traveling on street 220, travel vectors 222, 224, 226, and / or 228 may change over time based on the activity of the steering, braking, and / or acceleration systems of vehicles 212, 214, 216, and / or 218. Since each of the vehicles 212, 214, 216 and 218 moves in various directions as indicated by their associated travel vectors, the attitude and position of each vehicle relative to the other vehicles change over time, thus complicating the effective sharing of sensed information among the vehicles 212, 214, 216 and / or 218.

[0078] As can be seen, BEV grids 202 and 204 overlap. Additionally, BEV grids 208 and 204 overlap. This overlap of BEV grids can be utilized to provide additional accuracy for object detection by leveraging the multi-agent (multi-vehicle) techniques of this disclosure when compared to the individual BEV grids for each vehicle. For example, BEV grid 204 has a lower resolution than BEV grids 202 and 208, making it less likely that objects will be placed accurately within BEV grid 204 as they would within BEV grids 202 and 208.

[0079] Additionally, by using the multi-agent technology of this disclosure, information from the sensor systems of each vehicle within a specific area can be shared. For example, even if BEV grid 208 does not overlap with BEV grid 202, when vehicle 218 is traveling on street 220, vehicle 218 may encounter objects sensed by vehicle 212 (e.g., objects within BEV grid 202). Even if BEV grid 206 does not overlap with... Figure 2 Any other BEV mesh overlaps in the vehicle and travel vector 226 indicates that vehicle 216 is leaving other vehicles. Since such objects may themselves be moving and may be moving toward vehicles 212, 214 and / or 218, information about objects sensed by vehicle 216 may be meaningful to vehicles 212, 214 and / or 218.

[0080] Figure 3 This is a block diagram of an example single-agent BEV fusion architecture. For example, vehicle 212 may include architecture 300 that can be used to generate BEV mesh 202. Vehicle 212 may draw from, for example, LiDAR system 102 ( Figure 1 Point cloud data 302 is obtained. Encoder 306 encodes the point cloud data 302 to extract 3D sparse features 310. Vehicle 212 can flatten the projection via flattening projection unit 314, for example, flattening from 3D to 2D. Vehicle 212 can obtain LiDAR BEV features 318 from it.

[0081] Vehicle 212 can also be equipped with, for example, camera 104 ( Figure 1 Image data 304 is obtained. Encoder 308 can encode image data 304 to obtain perspective features 312. Vehicle 212 can use PV to BEV projection unit 316 to perform perspective to BEV projection transformation to obtain camera BEV features 320.

[0082] The feature fusion unit 319 can fuse LiDAR BEV features 318 and camera BEV features 320. The output of the feature fusion unit 319 can be decoded by the decoder 322 to generate a 3D bounding box 326, and then decoded by the decoder 324 and further decoded by the segmentation fusion decoder 328.

[0083] Encoders 306 and 308 can be configured to extract information from input data and process the extracted information to generate output. Typically, an encoder is configured to receive data as input and extract one or more features from the input data. Features are the output of the encoder. Features may include one or more vectors of numerical data that can be processed by a machine learning model. These vectors of numerical data can represent the input data in a way that provides information about the characteristics of the input data. In other words, the encoder is configured to process the input data to identify the characteristics of the input data.

[0084] Encoders 306 and 308 may represent encoders of neural networks, such as convolutional neural networks (CNNs), another class of artificial neural networks (ANNs), or another type of model comprising one or more layers and / or nodes configured to extract information from input data and process the extracted information to generate an output. Examples of layers include an input layer, an output layer, and hidden layers between the input and output layers. For example, a CNN may include one or more convolutional layers comprising convolutional filters. Each convolutional filter may perform one or more mathematical operations on the input data to detect one or more features, such as edges, shapes, textures, or objects. A CNN may additionally or alternatively include: activation functions that identify complex relationships between elements of point cloud data 302 and / or image data 304; and pooling layers that recognize patterns regardless of their location within the point cloud data 302 and / or image data 304.

[0085] Architecture 300 may include one or more encoders (e.g., encoder 306, encoder 308, flattening projection unit 314, PV-to-BEV projection unit 316) and one or more decoders (e.g., decoder 322, decoder 324, segmentation-fusion decoder 328). Architecture 300 may be configured to process image data 304 and location data (e.g., point cloud data 302). The encoder-decoder architecture for image feature extraction can be used in computer vision tasks such as image annotation, image-to-image translation, and image generation. The encoder-decoder architecture transforms input data into a compact and meaningful representation, called a feature vector, which captures significant visual information from the input data. The encoder extracts features from the input data, while the decoder reconstructs the input data from the learned features.

[0086] In some cases, encoders (e.g., encoder 306, encoder 308, flattening projection unit 314, PV-to-BEV projection unit 316) are constructed using CNN layers to analyze the input data in a hierarchical manner. CNN layers can apply filters to capture local patterns and progressively combine them to form higher-level features. Each convolutional layer extracts increasingly complex visual representations from the input data. These representations can be compressed and downsampled through operations such as pooling or strided convolutions, thereby reducing spatial dimensionality while preserving desired information. The final output of the encoder can represent a flattened feature vector encoding the high-level visual features of the input data.

[0087] Decoders (e.g., decoder 322, decoder 324, segmentation-fusion decoder 328) can be constructed using transposed convolutional layers or fully connected layers and can reconstruct input data from learned feature representations. The decoder takes a feature vector obtained from the encoder as input and processes it to generate an output similar to the input data. The decoder can upsample and expand the feature vector, gradually restoring the spatial dimensions lost during encoding. The decoder can apply transformations (such as transposed convolution or deconvolution) to reconstruct the input data. Decoder layers progressively refine the output, incorporating details and structure until a visually plausible image is generated.

[0088] During training, a loss function is used to train the encoder-decoder architecture for feature extraction. This loss function measures the difference between the reconstructed image and the ground truth image. This loss guides the learning process, encouraging the encoder to capture meaningful features and the decoder to produce accurate reconstructions. The training process may involve minimizing the difference between the generated image and the ground truth image, typically using backpropagation and gradient descent techniques. The encoder and decoder of architecture 300 can be used by... Figure 1 The training data 170 stored in memory 160 is used for training. Alternatively or additionally, the encoder and decoder of architecture 300 can be trained using training data stored separately from memory 160.

[0089] As illustrated, the encoder of architecture 300 extracts high-level features from the input data, and the decoder of architecture 300 reconstructs the input data from the learned features. This architecture allows input data to be transformed into a compact and meaningful representation. The encoder-decoder architecture enables the model to learn and utilize important visual and positional features, thereby facilitating tasks such as image generation, annotation, and transformation.

[0090] Encoder 306 can encode point cloud data 302 to extract 3D sparse features 310. Flattening projection unit 314 can represent an encoder configured to perform one or more tasks involving one or more spatial transformations, such as flattening the projection of 3D sparse features 310. For example, flattening projection unit 314 can include a Spatial Transformer Network (STN) that includes localization and sampling generation modules for spatial transformation of the input data. The STN (such as flattening projection unit 314) can perform geometric transformations on the 3D sparse features 310 to improve the alignment of the 3D sparse features 310 and / or adjust the 3D sparse features 310 for a specific task. Flattening projection unit 314 can output LiDAR BEV features 318.

[0091] Encoder 308 can encode image data 304 to extract perspective features 312. PV-to-BEV projection unit 316 can represent an encoder configured to perform one or more tasks involving one or more spatial transformations, such as projecting perspective features into BEV format, and can output camera BEV features 320. For example, PV-to-BEV projection unit 316 can include an STN that includes a localization and sampling generation module for spatial transformation of the input data. STNs (such as PV-to-BEV projection unit 316) can perform geometric transformations on perspective features 312 to improve the alignment of perspective features 312 and / or adjust the perspective features for a specific task.

[0092] The feature fusion unit 319 can be configured to fuse LiDAR BEV features 318 and camera BEV features 320 to generate a fused 3D feature set. In some examples, the feature fusion unit 319 can use a cascading operation to fuse LiDAR BEV features 318 and camera BEV features 320 to generate a 3D feature set, such that the fused 3D feature set includes useful information present in each of the fused LiDAR BEV features 318 and camera BEV features 320.

[0093] Figure 4 This is a block diagram of an example collaborative multi-vehicle late-stage BEV convergence architecture. The collaborative BEV baseline is now described. In this example, three vehicles participate in the collaborative BEV process.

[0094] Each vehicle can determine the attitude of its corresponding vehicle camera and LiDAR sensor. For example, a first vehicle can determine the attitude 402A of its associated vehicle camera-LiDAR, a second vehicle can determine the attitude 402B of its associated vehicle camera-LiDAR, and a third vehicle can determine the attitude 402C of its associated vehicle camera-LiDAR. In other words, each vehicle can determine the attitude of its own camera and / or LiDAR system.

[0095] Each of the three vehicles can perform BEV fusion of its own camera and LiDAR (or other sensor) data (e.g., BEV Fusion 404A, BEV Fusion 404B, and BEV Fusion 404C). Each of the three vehicles can generate its own BEV mesh (e.g., BEV Mesh 406A, BEV Mesh 406B, and BEV Mesh 406C). For example, each of the three vehicles can include data from... Figure 3 The architecture is similar to that of other BEVs, and they generate their own BEV meshes based on their own sensor data (e.g., point cloud data and image data).

[0096] BEV meshes 406A to 406C can be obtained through a cloud-based computing system (e.g., one or more servers) capable of performing cloud mesh fusion 408 to fuse BEV meshes 406A to 406C. In this example, a maximum number of vehicles (e.g., three) can be part of a group whose BEV meshes are fused. Each vehicle can then obtain the fused BEV mesh from the cloud mesh fusion 408. Since any of the three vehicles may have moved and the pose of the camera and / or LiDAR system may have changed, this process can then be repeated. For example, a first vehicle can determine vehicle camera-LiDAR pose 410A, a second vehicle can determine vehicle camera-LiDAR pose 410B, and a third vehicle can determine vehicle camera-LiDAR pose 410C.

[0097] This baseline includes late-stage fusion of the BEV mesh for each vehicle and adaptive feature fusion of LiDAR as described by Qiao et al. Such a baseline is relatively simple to implement and can include BEV detection and segmentation directly into a global coordinate system that can be shared with other vehicles and / or participants in the scene. Data sharing is relatively compact, with only the output of the BEV fusion model shared.

[0098] However, objects in the FOV of cameras from two different vehicles may have different characteristics. For example, a single object can appear very different across different viewpoints. The scale of an object can vary depending on the camera viewpoint of the different vehicles. In this baseline system, there is no way to correlate these viewpoints and fuse features to obtain a more complete 3D representation of the object. BEV features with different resolutions cannot be combined and reused between vehicles. For example, a high-resolution BEV mesh vehicle will replicate a low-resolution mesh, which may be undesirable. No information about mesh uncertainty is provided to perform fusion. When reusing BEV meshes from other vehicles, the self-contained vehicle may combine information between BEV meshes and therefore may lose information from the higher-resolution mesh.

[0099] Figure 5 This is a block diagram of an example collaborative multi-vehicle dynamic BEV mid-term feature fusion architecture based on one or more aspects of this disclosure. Figure 5 In the example, each of a dynamically numbered number of vehicles (1 to N) can determine the orientation of the vehicle camera and LiDAR sensor. For example, the first vehicle (which may be a reference vehicle) can determine the vehicle camera-LiDAR and position (e.g., via sensor 108). Figure 1 The GPS sensor of the second vehicle (502A) determines the attitude of its camera-LiDAR and position (502B), and so on. The Nth vehicle (502N) determines the attitude of its camera-LiDAR and position. Each vehicle can determine the attitude of its own camera system, LiDAR system, and / or position.

[0100] Each vehicle in the vehicle can use its own corresponding encoder (502A to 502N) to encode its own corresponding vehicle camera-LiDAR and position and attitude.

[0101] Encoded vehicle camera-LiDAR and position / pose data for each vehicle in the vehicle system can be obtained via a cloud-based computing system (e.g., one or more servers) that performs cooperative dynamic neighborhood feature fusion 506 to fuse the encoded vehicle camera-LiDAR and position / pose data. Each vehicle in the vehicle system can then obtain a corresponding fused BEV mesh 508A to 508N from the cooperative dynamic neighborhood feature fusion 506. These fused BEV meshes 508A to 508N may include [data missing - likely related to a specific feature]. Figure 4Examples of higher-resolution data, where the resolution of the fused BEV mesh can be the lowest resolution of the overlapping BEV meshes in the overlapping region of BEV meshes 506A to 506C. Since any of the N vehicles may have moved, and the pose of the camera, LiDAR, and / or positioning system may have changed, and vehicles may have left the group, or additional vehicles may have been added to the group, the process can then be repeated.

[0102] Figure 5 Examples and Figure 4 The examples differ. For instance, sensor data for each vehicle can be included in the BEV grid generated for any given vehicle, without requiring a given vehicle to generate its own BEV grid based solely on its own sensor data before generating a BEV grid based on sensor data from multiple vehicles. In other words, each vehicle does not need to generate its own BEV grid based solely on its own sensor data. Additionally, Figure 5 Examples could include the number of vehicles changing over time and unlike Figure 4 In some examples, the model remains static (e.g., with 3 vehicles). Therefore, the amount of sensor data used to generate the BEV mesh can be dynamic and increases as the scene becomes more complex with the addition of other vehicles. While Cooperative Dynamic Neighborhood Feature Fusion 506 is described as cloud-based, in some examples, it may be based on a reference vehicle and / or another vehicle.

[0103] Figure 6 This is a conceptual diagram illustrating sensor data captured by two example vehicles according to one or more aspects of this disclosure. Vehicles 600 and 610 are depicted. Vehicle 600 may include various sensors, such as multiple fisheye cameras and a front-facing camera. For example, four fisheye cameras have fields of view (FOV) 604A to 604B and capture image data of FOVs 604A to 604B. The front-facing camera may have an FOV 602 and capture image data of FOV 602.

[0104] Vehicle 610 may have a LiDAR system, a front-facing camera, and a rear-facing camera. The LiDAR system may have a 360-degree field of view (FOV) 612 surrounding vehicle 610 and may capture point cloud data of FOV 612. The front-facing camera may have an FOV 614 and may capture image data of FOV 614. The rear-facing camera may have an FOV 616 and may capture image data of FOV 616. As can be seen, the FOVs of the sensors of vehicle 600 overlap with each other, as do the FOVs of the sensors of vehicle 610. Additionally, some FOVs of the sensors of vehicle 600 overlap with the FOVs of the sensors of vehicle 610.

[0105] Object 620 represents a dynamic (e.g., moving) object that can be sensed by vehicle 600 in FOV 602 or by vehicle 610 in FOV 616. Relative to vehicle 600, object 620 may be in front of vehicle 600 at a certain distance and angle from vehicle 600 in its direction of travel. Relative to vehicle 610, object 620 may be behind vehicle 610 at different distances and angles from vehicle 610 in its direction of travel. Because object 620 is a dynamic object, and each of vehicles 600 and 610 may also be moving, the position, distance, and angle of object 620 relative to vehicle 600 and vehicle 610 can change drastically over time. The system of this disclosure can generate a mask / view frustum 630 based on the overlapping region. The mask / view frustum 630 can be used as part of the fused BEV features, as described later herein.

[0106] Figure 7 This is a block diagram illustrating an example architecture of a collaborative multi-vehicle dynamic BEV mid-term feature fusion system according to one or more aspects of this disclosure. The techniques of this disclosure include dynamic BEV mid-term feature fusion to address relative to... Figure 4 The discussion addresses the issues of late-stage feature fusion baselines described above. System 700 can be implemented in one or more servers (e.g., in a cloud computing environment), one or more PSUs, and / or one or more vehicles.

[0107] System 700 can obtain vehicle attitude information from multiple vehicles. For example, system 700 can obtain vehicle attitudes 702A to 702N (collectively referred to as "CAV vehicle attitudes 702"). Encoders 706A to 706N can each encode information (such as sensor system data, attitude, and / or position data) from the corresponding CAV. In some examples, encoders 706A to 706N can represent Figure 5The BEV encoders 504A to 504N are used, in which case encoders 706A to 706N may be located in the corresponding CAV. Encoders 706A to 706N and / or other encoders (such as flattening projection unit 314 and / or PV to BEV projection unit 316) may be part of feature compression unit 732 capable of generating compressed BEV features 718A to 718N. Multimodal dynamic mask / frust generator 730 may generate masks or frustums based on CAV vehicle attitude 702. (See reference...) Figure 8 The multimodal dynamic mask / frust generator 730 is described in more detail.

[0108] Feature compression unit 732 can be used to address the problems of high communication overhead and potential latency; one possible solution is to use a compressed version of the original BEV features. Many methods exist for compressing neural network feature vectors, and the optimal method depends on the specific use case and constraints. Potential ways to compress BEV features include quantization, pruning, hashing, transform decoding, etc. In some examples, the feature compression unit is the multi-vehicle BEV unit 194 (…). Figure 1 The feature compression unit is part of the BEV unit 140 (in some examples). Figure 1 ( ) part.

[0109] Quantization involves reducing the number of bits used to represent each feature in a vector. For example, a system can represent each feature using an 8-bit integer instead of a 32-bit floating-point number. This reduces the storage required for each feature and can speed up computation by reducing memory access time. However, this can come at the cost of reduced accuracy.

[0110] Pruning involves removing some less important features from a vector. Pruning can be achieved by setting small weights to zero or by grouping similar features together using a clustering algorithm. Pruning reduces the storage required for feature vectors and can speed up computation by reducing the amount of computation required. However, this can come at the cost of reduced accuracy.

[0111] Hashing involves using a hash function to map each feature in a vector to a fixed-length code. This reduces the storage required for the feature vector and speeds up computation by reducing memory access time. However, this can come at the cost of data collisions where different features map to the same code, which can reduce accuracy.

[0112] Transform decoding involves transforming the eigenvector using linear or nonlinear transforms, such as the Discrete Cosine Transform (DCT) or Wavelet Transform. The transformed coefficients can then be quantized and encoded using entropy decoding techniques, such as Huffman decoding. This reduces the storage required for the eigenvector and provides good compression performance. However, this technique can be computationally expensive and may require more storage than the other techniques described.

[0113] In some examples, the feature compression unit 732 may apply one or more of these techniques (including any combination of these techniques, such as two or more techniques) to generate compressed BEV features 718A to 718N. By utilizing the feature compression unit 732, the system 700 can achieve a desired balance between compression performance and accuracy level.

[0114] The trajectory-sensitive BEV feature subsampler 734 can utilize one or more masks or view frustums (e.g., Figure 6 The compressed BEV features 718A to 718N are subsampled using a mask / view frustum 630. For example, subsampling may be desirable due to bandwidth limitations in system 700. A trajectory-sensitive BEV feature subsampler 734 can subsample the compressed BEV features 718A to 718N based on the corresponding trajectories of vehicles CAV1 to CAVN. The operation of the trajectory-sensitive BEV feature subsampler is relative to... Figure 12 Let's discuss this in more detail.

[0115] The output of the trajectory-sensitive BEV feature subsampler 734 can be obtained through the vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736. In some examples, the vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736 can align the meshes associated with some or all of the CAVs in the CAV, for example, across the special Euclidean group 3 (SE3) transform group, even if all CAVs have the same BEV mesh size and resolution.

[0116] The output of the vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736 can be provided back to CAV. A static 3D on-device fusion decoder 722A (which may be, for example, a static 3D on-device camera / LiDAR fusion decoder of CAV1) can decode the output of the vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736. Similarly, a static 3D on-device fusion decoder 722N (which may be, for example, a static 3D on-device camera / LiDAR fusion decoder of CAVN) can decode the output of the vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736. The static 3D on-device fusion decoders 722A to 722N can thus generate a unified BEV mesh 738.

[0117] Figure 8This is a conceptual diagram illustrating a multimodal dynamic mask / frust generator according to one or more aspects of this disclosure. For example, a multimodal dynamic mask / frust generator 830 (which may be...) Figure 7 An example of a multimodal dynamic mask / view frustum generator 730 can be configured to generate dynamic masks and / or view frustums based on overlapping regions represented in the CAV1 vehicle posture 802A and the CAV2 vehicle posture 802B. For example, in Figure 6 In the example, two vehicles depict overlapping sensor FOVs. For example, a multimodal dynamic mask / frust generator 830 can generate a dynamic mask / frust camera-camera 842 that can represent the overlapping areas of the cameras' FOVs, such as... Figure 6 The overlapping areas of FOVs 602 and 616. The multimodal dynamic mask / cone generator 830 can generate a dynamic mask / cone camera-LiDAR 844, which can represent the overlapping areas of the FOVs of the camera and LiDAR system, such as the overlapping areas of the FOVs of 604A and FOV 612. Other examples of dynamic masks / cones that can be generated by the multimodal dynamic mask / cone generator 830 include LiDAR-LiDAR 846 (which can represent the overlapping FOV areas of different LiDAR systems), LiDAR-Radar 848 (which can represent the overlapping FOV areas of LiDAR and radar systems), Radar-Radar 850 (which can represent the overlapping FOV areas of different radar systems), Camera-LiDAR-Radar (CLR)-CLR 852 (which can represent the overlapping FOV areas of different camera, LiDAR, and radar systems), etc. It should be understood that the multimodal dynamic mask / view cone generator 830 can generate dynamic masks / view cones of overlapping FOVs for any combination of different sensor systems, including overlapping FOVs of sensor systems of more than two sensor systems and / or more than two vehicles.

[0118] Such dynamic masks / view frustums can be transmitted to downstream dynamic BEV fusion units (not shown). Figure 8In this context, the downstream dynamic BEV fusion unit uses any of a variety of techniques (e.g., channel, maximum, or self-attention-based techniques) to construct these features. Dynamic masks / frustes can be used to dynamically inform the downstream dynamic BEV fusion unit, based on the relative position of the CAV, which combination of sensors has overlapping FOVs. Since FOV overlap changes over time, the number of feature inputs can be dynamic. The downstream dynamic BEV fusion unit can fuse features based on dynamic masks / frustes. For example, if the FOVs of two cameras overlap, the downstream dynamic BEV fusion unit can use better BEV features to address scale issues, and if the FOVs of two LiDAR systems overlap, the downstream dynamic BEV fusion unit can aggregate BEV features using a better BEV grid resolution, and so on.

[0119] These dynamic masks / view cones can also provide camera-to-camera masks (e.g., dynamic mask / view cone camera-to-camera 842), in which distortion in the FOV of a camera can differ across different vehicles / cameras for a common object. This allows distances and angular distances from the principal point (e.g., direction of travel, camera focus, etc.) to be used as inputs to the feature fusion process for better feature fusion across cameras.

[0120] Figure 9 This is a block diagram illustrating a second encoder-decoder architecture 900 for processing image data and position data in a continuous space to generate an output, according to one or more techniques of this disclosure. In some examples, the second encoder-decoder architecture 900 may be... Figure 1 A part of BEV unit 140 and / or multi-vehicle BEV unit 194. Figure 9 Examples include a 2D camera image 902, a first feature extractor 904, a perspective feature 906, a 3D point cloud frame 922, a second feature extractor 924, a 3D feature 926, and a flattening unit 928. Figure 9 Examples also include ray to BEV unit 930, camera radial feature embedding 932, and point cloud radial feature embedding 934.

[0121] 2D camera image 902 can be Figure 1Examples of 2D camera images 168. In some examples, 2D camera image 902 may represent a set of camera images from 2D camera image 168, and 2D camera image 168 may include one or more camera images not present in 2D camera image 902. In some examples, 2D camera image 902 may be received from multiple cameras at different locations and / or different fields of view, and these fields of view may overlap. In some examples, a second encoder-decoder architecture 900 processes 2D camera image 902 in real-time or near real-time, such that when camera 104 captures 2D camera image 902, the second encoder-decoder architecture 900 processes the captured camera image. In some examples, 2D camera image 902 may represent one or more perspective views of one or more objects within the 3D space in which processing system 100 is located. That is, one or more perspective views may represent views seen from the perspective of processing system 100.

[0122] In some examples, the second encoder-decoder architecture 900 can be configured to convert a set of 2D camera images 902 into a first 3D representation of the 3D environment corresponding to the 2D camera images 902 and the 3D point cloud frame 922. The 3D point cloud frame 922 can represent a second 3D representation of the 3D environment. In this way, the second encoder-decoder architecture 900 enables both the 2D camera images 902 and the 3D point cloud frame 922 to become 3D representations of the 3D environment. This allows the second encoder-decoder architecture 900 to process features extracted from the 2D camera images 902 and the 3D point cloud frame 922 in a continuous space without using discrete processing of feature-dependent 2D meshes. For example, the second encoder-decoder architecture 900 can generate a set of BEV features. Although this set of BEV features may include a 2D representation of the 3D environment, this set of BEV features may not depend on a 2D mesh of BEV feature cells that place features corresponding to many different objects in the same cell.

[0123] The first feature extractor 904 may represent an encoder of a neural network or another type of model configured to extract information from input data and process the extracted information to generate an output. In some examples, the first feature extractor 904 represents a CNN, another type of ANN, or another type of model including one or more layers and / or nodes. Each of the one or more layers may include one or more nodes. Examples of layers include an input layer, an output layer, and hidden layers between the input and output layers. For example, a CNN may include one or more convolutional layers, which include convolutional filters. Each convolutional filter may perform one or more mathematical operations on the input data to detect one or more features, such as edges, shapes, textures, or objects. The CNN may additionally or alternatively include activation functions that identify complex relationships between elements of an image and pooling layers that identify patterns at any location within an image frame.

[0124] The first feature extractor 904 can extract perspective features 906 from the 2D camera image 902. Perspective features 906 can provide information from the viewpoint of the camera 104 that captured the 2D camera image 902, corresponding to one or more objects depicted in the 2D camera image 902. For example, perspective features 906 can include vanishing points and vanishing lines, indicating points where parallel lines converge or disappear, the direction of dominant lines, the structure or orientation of an object, or any combination thereof. Perspective features 906 can include color information. Additionally or alternatively, perspective features 906 can include keypoints matched on a group of two or more camera images of the 2D camera image 902. Keypoints can allow the second encoder-decoder architecture 900 to determine one or more characteristics of the motion and pose of an object. Perspective features 906 can include any image feature or a combination thereof that indicates characteristics of the 2D camera image 902.

[0125] In some examples, perspective feature 906 can represent 2D features. That is, perspective feature 906 can indicate the characteristics of one or more objects within a 3D environment, corresponding to a 2D camera image 902 and a 3D point cloud frame 922, which corresponds to the position on the 2D camera image as viewed from the perspective of camera 104. One technique for converting perspective feature 906 into BEV features is to project perspective feature 906 onto a 2D mesh of BEV cells. This may involve discrete processing of placing features corresponding to many different objects into the same cell, which can lead to blurring and poor resolution. A second encoder-decoder architecture 900 can convert perspective feature 906 into a 3D representation of the 3D environment, such that the encoder-decoder architecture 900 is configured to generate BEV features using sequential processing rather than discrete processing.

[0126] 3D point cloud frame 922 can be Figure 1 Examples of 3D point cloud frames 166. In some examples, 3D point cloud frame 922 may represent a set of 3D point cloud frames from 3D point cloud frame 166, and 3D point cloud frame 166 may include one or more 3D point cloud frames not present in 3D point cloud frame 922. In some examples, the second encoder-decoder architecture 900 processes 3D point cloud frame 922 in real-time or near real-time, such that when LiDAR system 102 generates 3D point cloud frame 922, the second encoder-decoder architecture 900 processes the captured 3D point cloud frame. In some examples, 3D point cloud frame 922 may represent a set of point coordinates (e.g., x, y, z coordinates in Cartesian space) within the 3D space in which LiDAR system 102 resides. Since LiDAR system 102 is configured to emit light signals and receive light signals reflected from the surfaces of one or more objects, the set of point coordinates may indicate the shape and position of the surfaces of one or more objects in 3D space.

[0127] The second feature extractor 924 may represent an encoder of a neural network or another type of model configured to extract information from input data and process the extracted information to generate output. The second feature extractor 924 may be similar to the first feature extractor 904, since both the first and second feature extractors 904 are configured to process input data to generate output features. However, in some examples, the first feature extractor 904 is configured to process 2D input data and the second feature extractor 924 is configured to process 3D input data. In some examples, the processing system 100 is configured to train the first feature extractor 904 using a training dataset 170 that includes one or more training camera images, and the processing system 100 is configured to train the second feature extractor 924 using a training dataset 170 that includes one or more point cloud frames. In other words, the processing system 100 can train a first feature extractor 904 to identify one or more patterns in a camera image corresponding to certain camera image perspective features, and the processing system 100 can train a second feature extractor 924 to identify one or more patterns in a point cloud frame corresponding to certain 3D sparse features. In some examples, processing circuitry separate from the processing system 100 is configured to train one or more elements of a second encoder-decoder architecture 900 using training data stored separately from the processing system 100.

[0128] The second feature extractor 924 can generate a set of 3D features 926 based on a 3D point cloud frame 922. The 3D features 926 can provide information corresponding to one or more objects indicated by the 3D point cloud frame 922 in a 3D space, which includes a LiDAR system 102 that captures the 3D point cloud frame 922. The 3D features 926 can include keypoints within the 3D point cloud frame 922 that indicate unique characteristics of one or more objects. For example, keypoints can include corners, straight edges, curved edges, and vertices of curved edges. The second encoder-decoder architecture 900 can identify one or more objects based on the keypoints. The 3D features 926 may additionally or alternatively include descriptors that allow the second feature extractor 924 to perform group comparisons and track keypoints across two or more point cloud frames of the 3D point cloud frame 922. Other types of 3D features 926 include stereo pixels and superpixels.

[0129] Flattening unit 928 can transform 3D feature 926 into 2D feature. Since 3D point cloud frame 922 represents a multidimensional array of Cartesian coordinates, flattening unit 928 can transform 3D feature 926 into 2D feature by compressing one dimension of the x, y, z Cartesian space into a flat plane without compressing the other two dimensions. That is, a point in a column of points parallel to one dimension of the x, y, z Cartesian space can be compressed into a single point in the 2D space formed by the two uncompressed dimensions. On the other hand, perspective feature 906 extracted from 2D camera image 902 may not include Cartesian coordinates. This means that transforming perspective feature 906 into a 3D representation of the 3D environment can be beneficial, allowing both features extracted from image data and features extracted from location data to be processed in a continuous space without the need for discrete processing using a 2D mesh dependent on BEV cells.

[0130] The ray-to-BEV unit 930 may receive perspective features 906 captured from a 2D camera image 902. The ray-to-BEV unit 930 may receive output from a flattening unit 928 representing flattened 3D features 926 extracted from a 3D point cloud frame 922. The ray-to-BEV unit 930 may receive both the 2D camera image 902 and the 3D point cloud frame 922. Based on any or a combination of the perspective features 906, the output from the flattening unit 928, the 2D camera image 902, and the 3D point cloud frame 922, the ray-to-BEV unit 930 may generate a camera radial feature embedding 932 and a point cloud radial feature embedding 934.

[0131] In some examples, the ray-to-BEV unit 930 can generate a camera radial feature embedding 932 by converting perspective features 906 and a 2D camera image 902 into a 3D representation of the 3D environment corresponding to the 2D camera image 902 and a 3D point cloud frame 922. For example, the ray-to-BEV unit 930 can establish rays traversing 3D space based on each camera image pixel of the 2D camera image 902. The ray-to-BEV unit 930 can identify one or more points in 3D space for each ray corresponding to each camera image pixel of the 2D camera image 902. The ray-to-BEV unit 930 can establish a depth distribution for one or more points corresponding to each ray of the 2D camera image 902. The ray-to-BEV unit 930 can generate a 3D feature volume based on the 3D space and the depth distribution corresponding to each ray of the 2D camera image 902. The 3D feature volume can represent a 3D representation of the 3D environment corresponding to the 2D camera image 902 and a 3D point cloud frame 922. In some examples, the ray to BEV unit 930 can generate a point cloud radial feature embedding 934 based on a 3D point cloud frame 922 and 3D features 926 extracted from the 3D point cloud frame 922.

[0132] The ray-to-BEV unit 930 can be configured to generate a camera radial feature embedding 932 and a point cloud radial feature embedding 934. The ray-to-BEV unit 930 includes a first 3D space 952, a first reference point 954, a first spatial neighborhood 956, a second 3D space 962, a second reference point 964, and a second spatial neighborhood 966.

[0133] The ray-to-BEV unit 930 can construct a kernelized continuous representation of the feature map in BEV space using a splatter point cloud representation derived from the camera pixel-ray background representation in Local Self-Similarity (LSS). To generate accurate depth-to-center feature embeddings, the ray-to-BEV unit 930 can use a 3D reference point from the LiDAR point cloud and examine the spherical neighborhood around that reference point to learn a multivariate Gaussian mixture with mean / covariance estimates. The perspective feature regions corresponding to the 3D reference point and their corresponding rays can be used to evaluate the variational autoencoder (VAE) parameters and / or GMM parameters.

[0134] The ray-to-BEV unit 930 can generate a first 3D space 952 based on the 2D camera image 902. In some examples, the ray-to-BEV unit 930 can generate the first 3D space 952 by generating rays corresponding to each pixel of the 2D camera image 902. The rays corresponding to each pixel of the 2D camera image 902 may include a set of points. In some examples, the ray-to-BEV unit 930 can generate a camera radial feature embedding 932. By generating the first 3D space 952 based on the 2D camera image 902, the ray-to-BEV unit 930 can convert the 2D camera image 902 into a 3D representation of the 3D environment corresponding to the 2D camera image 902 and the 3D point cloud frame 922. Converting the 2D camera image 902 into a 3D representation of the 3D environment allows the ray-to-BEV unit 930 to generate the camera radial feature embedding 932 in a fixed mesh manner independent of BEV features.

[0135] In some examples, the ray-to-BEV unit 930 can generate a camera radial feature embedding 932 by filling a first 3D space with perspective features of perspective feature 906. For example, the ray-to-BEV unit 930 can select a first reference point 954 based on a 3D point cloud frame 922 and fill a first spatial neighborhood 956 with perspective features of perspective feature 906 corresponding to the first reference point 954. In some examples, the first reference point 954 can be a reference point corresponding to a set of reference points in the first 3D space 952. The ray-to-BEV unit 930 can fill the spatial neighborhood corresponding to each reference point with perspective features located in the spatial neighborhood of the reference point.

[0136] In some examples, the second 3D space 962 corresponds to the 3D space of the 3D point cloud frame 922. Since the 3D point cloud frame 922 includes points with 3D coordinates, it may already represent a 3D space without needing to be converted to 3D. The ray-to-BEV unit 930 can utilize the 3D features of the 3D feature 926 to fill the second 3D space 962. For example, the ray-to-BEV unit 930 can select a second reference point 964 based on the 3D point cloud frame 922 and fill the second spatial neighborhood 966 using the 3D features of the 3D feature 926 corresponding to the second spatial neighborhood 966. In some examples, the second reference point 964 represents a reference point in a set of reference points corresponding to the second 3D space 962. The ray-to-BEV unit 930 can utilize the corresponding 3D features of the 3D feature 926 to fill the spatial neighborhood corresponding to each reference point.

[0137] Ray-to-BEV unit 930 can use radial neighborhood feature selection to generate camera radial feature embedding 932 and point cloud radial feature embedding 934. Camera BEV features can be “splashed” onto the point cloud, using neighbors centered on the LiDAR point cloud for transformation. BEV features can be embedded continuously using VAE GMM. Ray-to-BEV unit 930 can convert point cloud representations from camera perspective features into BEV features. More information regarding radial feature embedding can be found in U.S. Patent Application No. 18 / 466,460, filed September 13, 2023, the entire contents of which are incorporated herein by reference. It should be noted that the obtained BEV radial feature embedding is not limited to point cloud radial feature embedding or camera radial feature embedding, but may also include radial feature embeddings from other sensor systems, such as radar.

[0138] Figure 10 This is a block diagram illustrating an example vehicle radial neighborhood embedding system according to one or more aspects of this disclosure. System 1000 may determine or otherwise obtain BEV radial feature embeddings, such as point cloud radial feature embeddings 934 and camera radial feature embeddings 932 for multiple vehicles. System 1000 may be implemented in one or more servers (e.g., in a cloud computing environment), one or more PSUs, and / or one or more vehicles.

[0139] For example, System 1000 can obtain CAV1 BEV radial feature embedding 1050A and CAV2 BEV radial feature embedding 1050B. System 1000 can generate a meshless kernel 1052A based on CAV1 BEV radial feature embedding 1050A, and a meshless kernel 1052B based on CAV2 BEV radial feature embedding 1050B. For example, System 1000 can be configured to generate meshless BEV feature kernels 1052A to 1052B in a continuous space without relying on a 2D mesh of BEV features. Meshless kernels 1052A to 1052B can be variational autoencoder-Gaussian mixture model (VAE-GMM) kernels, and each kernel can be a per-vehicle BEV feature. The VAE-GMM kernel can include VAE components and GMM components. The VAE component can map input data to a probability distribution in the latent space and include data samples from points in the latent space. The GMM component can represent a mixture of multiple Gaussian distributions, which can be used for clustering, density estimation, and generative modeling. In some examples, the meshless kernels 1052A to 1052B may not include mesh information associated with CAV1 and CAV2. For example, the CAV1 BEV radial feature embedding 1050A and / or the CAV2 BEV radial feature embedding 1050B may not include mesh information. Where the CAV1 BEV radial feature embedding 1050A and / or the CAV2 BEV radial feature embedding 1050B may include mesh information, the system 1000 may remove such mesh information when generating the meshless kernels 1052A to 1052B.

[0140] Any mask / frust input from the meshless kernels 1052A to 1052B and the multimodal dynamic mask / frust generator 1030 (which may be an example of the multimodal dynamic mask / frust generator 830) can be fed into the multi-vehicle dynamic radial fusion unit 1054. The multi-vehicle dynamic radial fusion unit 1054 may include a dynamic BEV fusion unit or module that performs intermediate feature fusion to robustly extend perception not only within the intersection of the sensor's FOV but also across vehicles. The multi-vehicle dynamic radial fusion unit 1054 can also operate to fuse BEV features even if different vehicles and / or sensor systems do not output BEV meshes with the same reference frame, orientation, and / or resolution. The multi-vehicle dynamic radial fusion unit 1054 can be implemented in a cloud computing environment, leveraging existing BEV encoders on each vehicle. The multi-vehicle dynamic radial fusion unit 1054 may include a neural network (such as a deep neural network) trained for fusion across sparse dynamic vehicle interactions.

[0141] The multi-vehicle dynamic radial fusion unit 1054 can fuse meshless kernels 1052A and 1052B to generate a fused meshless kernel 1062. Since neither meshless kernels 1052A nor meshless kernels 1052B depend on a fixed mesh of BEV features, the multi-vehicle dynamic radial fusion unit 1054 can fuse meshless kernels 1052A and 1052B without relying on discrete processing.

[0142] System 1000 can use GMM to make the meshless core 1052A and meshless core 1052B with the center from (e.g., reference vehicle). Figure 1 The 3D point cloud frame 166 is adapted to the continuous 2D spatial coordinates of the reference points. Meshless kernels 1052A and 1052B can each represent continuous GMM features from multiple sensor systems.

[0143] The multi-vehicle dynamic radial fusion unit 1054 can combine meshless kernels 1052A and 1052B to generate a meshless kernel 1062 by combining GMM estimates via resampling to estimate a unified meshless representation. Reference ground truth discretization can be avoided by using a kernelized continuous segmentation representation centered around a reference point in the 3D point cloud frame 166 of the reference vehicle. BEV features, reference ground truth, and BEV output prediction can thus be represented as a 2D BEV hybrid model that avoids discretization artifacts.

[0144] A multi-vehicle dynamic radial fusion unit 1054 (which may be cloud-based) processes input to generate a meshless kernel 1062. The meshless kernel 1062 may be a VAE-GMM kernel, and each kernel may be a per-vehicle BEV feature. A BEV feature mesh scheduling discretizer 1064 discretizes the meshless kernel 1062 for each vehicle and transmits the discretized kernel to each of vehicles 1066A through 1066N. For example, the discretized kernel may be represented as a CAV neighborhood dynamic graph 1070. The BEV feature mesh scheduling discretizer 1064 allows updating the BEV mesh of the reference vehicle using information from neighboring meshes at the correct resolution in the reference vehicle.

[0145] System 1000 can use nucleated BEV fusion technology to perform dynamic (e.g., variable in size due to the dynamic number of CAVs around the reference CAV) feature fusion across vehicles based on query radius. This fusion can be performed collaboratively across all interacting CAVs, such that typically all CAVs benefit from the resulting improved perception and fused features. Radial embedding also allows fusion when CAVs differ in rotation, translation, and / or BEV mesh resolution. The final fused result can be dynamically stored and maintained in a cloud computing environment, such as a 5G cloud-based feature fusion network that can be implemented independently in the cloud. The final nucleated BEV features can be re-discretized and transmitted to the target vehicle at the correct or preferred BEV mesh resolution in the reference (e.g., rotation / translation) of the target vehicle. For example, the preferred BEV mesh resolution of the target vehicle can be based on the range and / or resolution of the vehicle's sensors (e.g., the range and / or resolution of such sensors (LiDAR, cameras, radar)) and the incoming cloud-based BEV mesh. For example, the highest resolution can be based on a combination of both, rather than one alone. In some examples, the preferred BEV mesh resolution may not necessarily be the highest possible resolution. For instance, a vehicle leaving a CAV neighborhood (e.g., in the CAV neighborhood dynamic graph 1070) may be less likely to collide with other vehicles in the group and may not require the same high resolution as vehicles still in or entering the CAV group. Therefore, the BEV feature mesh scheduling discretizer 1064 can adaptively send BEV features at different resolutions to reduce the bandwidth used for sending BEV features.

[0146] Figure 11 This is a block diagram illustrating an example dynamic multimodal mesh fusion system according to one or more aspects of this disclosure. Figure 10 In the example, system 1100 may determine or otherwise obtain a meshless kernel, such as CAV1 meshless kernel 1152A (which may be an example of CAV1 meshless kernel 1052A) and CAV2 meshless kernel 1152B (which may be an example of CAV2 meshless kernel 1052B). A multimodal dynamic mask / frust generator 1130 (which may be an example of multimodal dynamic mask / frust generator 730) can generate masks / frustes as discussed herein. A trajectory-sensitive BEV feature subsampler 1134 (which may be an example of trajectory-sensitive BEV feature subsampler 734) can use as relative to... Figure 7 The mask / frustum pair discussed above is used to subsample the compressed BEV features. Figure 11 In the example, the multi-vehicle dynamic radial fusion unit 1154 (which can be) Figure 10An example of a multi-vehicle dynamic radial fusion unit 1054 may include a dynamic fusion layer and cross-vehicle cross-camera dynamic object fusion with dynamic CAV intra-trajectory distance.

[0147] The dynamic fusion layer of the multi-vehicle dynamic radial fusion unit 1154 can take a variable number of BEV mesh features as input and adaptively fuse such BEV mesh features. This fusion layer uses dynamic masks generated by the multimodal dynamic mask / cone generator 1130, which display camera-camera, camera-LiDAR, camera-LiDAR-radar, LiDAR-radar, LiDAR-LiDAR, radar-radar, etc. interactions, and accordingly uses uncertainties based on sensor / sensor set to fuse BEV mesh features. For example, each sensor / sensor set may have a corresponding uncertainty that may differ from another sensor / sensor set. For example, camera-camera may have one uncertainty, while camera-LiDAR may have another uncertainty that differs from camera-camera uncertainty. For example, sensor resolution may affect BEV mesh resolution and downstream fusion, and therefore each sensor / sensor set may have its own corresponding uncertainty.

[0148] The dynamic object fusion across vehicles and cameras with dynamic intra-CAV trajectory distances can be performed as follows. Object 1120 (which can be...) Figure 6 (Example 620 in the text) can become visible across different scales / distances and positions in the camera viewpoints of two vehicles. If the CAVs move toward each other, the relative changes in scale of object 1120 will remain roughly similar. However, if the CAVs move in a manner that increases the distance between them, the object scales change in a relatively negatively correlated way. Therefore, cross-vehicle cross-camera dynamic object fusion with dynamic intra-CAV trajectory distances can use a reference to combine features with the relative trajectories between adjacent CAV vehicles. This provides better multi-scale features for the dynamic fusion layer to extract dynamic objects such as object 1120.

[0149] Figure 12 This is a block diagram illustrating an example of trajectory-sensitive subsampling according to one or more aspects of this disclosure. System 1200 can determine or otherwise acquire (e.g., a dynamic graph of the CAV neighborhood 1270) various vehicle attitudes and future path trajectory information of a dynamic number of vehicles. System 1200 can also acquire or determine meshless kernels 1252A to 1252N, which may be Figure 10 Meshless core 1052A or 1052B or Figure 11 Examples of meshless cores 1152A or 1152B.

[0150] A trajectory-sensitive BEV feature subsampler 1234 (which may be an example of a trajectory-sensitive BEV feature subsampler 734) can subsample the vehicle attitude and future path trajectory to select a subset of meshless kernels 1252A to 1252N that produce selected BEV features 1272 for fusion. For example, the selected BEV features 1272 for fusion may include meshless kernels 1252A and 1252B.

[0151] For example, vehicle interactions where future trajectories within a given time interval do not interact for more than K seconds may not be included in the selected BEV features 1272 for fusion and may therefore be discarded from or otherwise excluded from the neighborhood graph that can be transmitted to vehicles 1266A through 1266N (e.g., CAV neighborhood dynamic graph 1270 with future path trajectories). The future time interval (K seconds) may be programmable or dynamic, for example, based on the speed and / or direction of one or more vehicles in the neighborhood. For example, as shown in the CAV neighborhood dynamic graph 1270 with future path trajectories, there may be an impending interaction between vehicle 1266A and vehicle 1266B, but no impending interaction between vehicle 1266A or vehicle 1266B and vehicle 1266N. Therefore, system 1200 may discard features from vehicle 1266N and not include such features in the selected BEV features 1272 for fusion.

[0152] As discussed herein, the dynamic feature techniques of this disclosure may include the use of a multimodal dynamic mask / cone generator (e.g., multimodal dynamic mask / cone generator 730). Vehicle interactions are dynamic and can be confined to a neighborhood of a fixed radius selected by a cloud-based fusion module (e.g., external processing system 180). The multimodal dynamic mask / cone generator can acquire and evaluate BEV features from cameras, LiDAR, radar, and their intersections.

[0153] The dynamic feature techniques disclosed herein may include adaptive fusion using vehicle attitude and mesh resolution using kernelized BEV. For example, kernelized BEV representation can be used to fuse meshes with different translations, rotations, and / or orientations. Each vehicle may have a different BEV mesh resolution, which refers to the size and granularity of the mesh used to represent that region. These variations can be accounted for to ensure consistent and accurate fusion of BEV features. Kernelizing the BEV output provides a location-sensitive embedding method that is relatively easy to decode in the global BEV mesh.

[0154] The dynamic feature fusion technique disclosed herein may include dynamic feature fusion with variable FOV cross regions. Dynamic feature fusion can be implemented using a dynamic neural network architecture implemented in the cloud. This layer can perform robust fusion using uncertainties from a single sensor and any combination of sensors. Because the neural network is dynamic at inference time, the resulting inference may be more accurate in dynamic scenarios, such as those in autonomous driving scenarios. In some examples, the dynamic neural network can learn over time as new vehicles with different sensor system settings connect to the cloud. Dynamic fusion is performed on meshless radial embeddings, which helps to combine mesh features of different resolutions without loss of resolution in the process.

[0155] The dynamic feature fusion technique disclosed herein may include a BEV feature grid scheduling discretizer (e.g., Figure 7 Vehicle radial dynamic neighborhood radial fusion feature fusion and BEV mesh scheduler hard / soft parameter sharing unit 736 or Figure 10 The BEV feature grid scheduling discretizer 1064 can dynamically maintain the final fused result in the cloud, such as in a 5G cloud-based feature fusion network. For example, an external processing system 180 can store the final fused result in memory 198. Figure 1 The BEV feature grid scheduling discretizer can re-discretize features and transfer the re-discretized features to the target vehicle in its own reference (rotation / translation) at the correct BEV grid resolution.

[0156] The dynamic feature fusion technique disclosed herein may include a bandwidth-limited trajectory-sensitive subsampler (e.g., a trajectory-sensitive BEV feature subsampler 1234). The trajectory-sensitive subsampler may discard features of vehicles or vehicles whose future trajectories within a given time interval do not interact with other vehicles for more than K seconds from the neighborhood graph, and the associated features may not be transmitted from the cloud to the vehicle.

[0157] The techniques disclosed herein may also include feature compression. Each vehicle can use images obtained from one or more vehicle cameras in its vehicle cameras to obtain a common BEV feature vector of a pre-configured size. Regardless of the number of cameras / LiDAR / radars possessed by each vehicle (e.g., which may differ for each vehicle), the BEV features transmitted by the vehicle can have the same size. This will enable the cloud to perform global BEV inference across heterogeneous vehicles. In one case, the BEV feature vector size is specified by camera-to-camera (C2C) during initial connection setup.

[0158] For example, a vehicle obtains BEV features corresponding to images and point clouds from each of its cameras and LiDAR system. N The first vehicle using a single camera employs a first multilayer perceptron (MLP) feature normalization network, while possessing... M The second vehicle with a camera uses a second MLP feature normalization network to obtain BEV feature vectors with the same pre-configured BEV feature vector size. The output of the MLP feature normalization represents the original BEV features transmitted. R To minimize communication overhead, C2C provides the model to the MLP compression encoder. In cases where the vehicle uses a compression encoder, C2C can provide an index of the encoder used to obtain the compressed features (details below). Relative to Figure 13 Describes the signaling between C2C and vehicles.

[0159] Figure 13 This is a flowchart illustrating example communication between a C2C server and a vehicle. The following signaling details provide information about the communication between C2C server 1380 and vehicle 1382, particularly regarding BEV feature vectors. These details are divided into two parts: information transmitted from the C2C server to the vehicle; and information transmitted from the vehicle to the C2C server. Although referred to as a server, it should be understood that C2C server 1380 may include one or more servers.

[0160] For example, C2C server 1380 may transmit BEV feature configuration information 1384, which may include any or any combination of the information described below, to vehicle 1382. BEV feature configuration information 1384 may include BEV feature dimensions. ×ℎ× w The BEV feature configuration information 1384 specifies the size of the BEV feature vector that the vehicle 1382 should send to the C2C server 1380. The BEV feature configuration information 1384 may include a model index, such as a multilayer perceptron (MLP) model index, or the model itself for 2D-to-BEV transformation. This information specifies the MLP model or MLP model index that the vehicle 1382 should use to transform its 2D camera images into BEV feature vectors. The BEV feature configuration information 1384 may include an MLP model or model index for BEV feature compression. This information specifies the MLP model or MLP model index that the vehicle 1382 should use to compress the BEV feature vectors before sending them to the C2C server 1380. In some examples, the MLP model may be a model that the vehicle uses as a BEV encoder (e.g., Figure 5 (Model of BEV encoder 604).

[0161] Vehicle 1382 may send feature transmission 1386. Feature transmission 1386 may include any or a combination of the following information. Feature transmission 1386 may include information having dimensions ×ℎ× w BEV Features: This is the raw BEV feature vector that vehicle 1386 has obtained from its camera images. Feature transmission 1386 may include bits or flags indicating whether the BEV features are compressed. For example, a flag may indicate whether the BEV feature vector has been compressed. If the BEV feature vector is compressed, vehicle 1382 may also transmit an index of the MLP compression encoder used to compress the BEV feature vector. Feature transmission 1386 may include an index indicating the MLP index used for feature compression. For example, if the BEV feature vector is compressed, this may be an index of the MLP compression encoder used to compress the BEV feature vector.

[0162] Sending the raw BEV characteristics from vehicle 1382 to C2C server 1380 as part of a dedicated message or an existing message is a simple and straightforward method. However, this can result in higher communication overhead and potential latency in message delivery. Furthermore, without additional processing, the raw BEV characteristics may not be readily usable by other vehicles or the cloud.

[0163] In some examples, the vehicle 1382 may perform feature compression as described above with respect to feature compression unit 732, either as an alternative to or as a supplement to feature compression unit 732.

[0164] In some examples, vehicle 1382 may send soft BEV features or hard BEV features instead of the original BEV features. Soft BEV features are the probability or likelihood of an object's presence or an object's attribute, while hard BEV features are a binary or categorical representation of an object's presence or an object's attribute. By sending soft or hard BEV features instead of the original BEV features, the amount of data that needs to be sent can be reduced, resulting in lower communication overhead. Furthermore, soft or hard BEV features may be easier to process and integrate into existing sensor fusion algorithms. Finally, to further reduce communication overhead, only the BEV features of a specified object or region of interest (ROI) can be sent. This allows the cloud or receiving vehicle to enhance alignment accuracy, resulting in better BEV accuracy. Additionally, the cloud may request BEV features of the ROI or object from the first vehicle based on BEV inferences obtained from a second or third vehicle, which further reduces communication overhead.

[0165] The concept of hard BEV features is to indicate the presence of objects in a local BEV mesh as perceived by a vehicle. When sending hard BEV features, the information sent may include the center coordinates and dimensions of each object. By sending only this information, communication overhead can be drastically minimized. For example, if two objects exist in a local BEV mesh as perceived by a vehicle, the vehicle can report the hard BEV features of each object as follows: Object 1: Center coordinates ( x _1, y _1), and size _1 × w _1 Object 2: Center coordinates ( x _2, y _2), and size _2 × w _2.

[0166] In addition to the details of signaling notification, C2C 1380 (e.g., as part of BEV feature configuration information 1384) can provide vehicle 1382 with the BEV grid size on which hard features will be reported. This ensures that hard features are consistent across all vehicles and can be processed uniformly. It is worth noting that in some cases, the object center and size can be normalized to the BEV grid size for signaling notification. This means that the center and size can be scaled to fit within the BEV grid size provided by C2C 1380.

[0167] Soft BEV features are probabilistic measures of the likelihood of an object's presence or object attributes within a local BEV grid perceived by a vehicle. These features can be represented as probability distributions over object categories and attributes and can be obtained using various machine learning models, such as object detectors, trackers, and / or classifiers. For example, vehicle 1382 can use an object detection model to detect objects within its local field of view and generate soft BEV features for each detected object. These features can represent the probability or likelihood that the detected object belongs to a specific category (e.g., car, pedestrian, bicycle) and has certain attributes (e.g., size, orientation, speed).

[0168] Soft BEV features can also be used to estimate uncertainties associated with object detection and to refine object detection outputs from local BEV meshes of vehicles. For example, a C2C server 1380 can fuse soft BEV features from multiple vehicles to generate a more accurate and reliable global BEV map. Overall, because soft BEV features capture the uncertainties and variability associated with object detection, tracking, and / or classification, they can provide a richer representation of the local BEV mesh than hard BEV features. This enables more robust and accurate perception of the environment and enhances the performance of downstream tasks such as motion planning and control.

[0169] Figure 14 This is a flowchart illustrating an example of a multi-vehicle (BEV) feature fusion technique according to one or more aspects of this disclosure. Figure 14 Multi-vehicle BEV feature fusion technology can be implemented in one or more servers, one or more RSUs, and / or one or more vehicles. External processing system 180 or processing system 100 can acquire vehicle data. For example, external processing system 180 or processing system 100 can receive attitude, trajectory, position, BEV features, meshless kernel, and / or other information from each of the multiple vehicles. In some examples, the received vehicle data may include meshless data. In some examples, the received vehicle data may not include meshless data.

[0170] External processing system 180 or processing system 100 can determine one or more first features from first vehicle data in vehicle data, the first vehicle data being derived from a plurality of vehicles (1402). For example, external processing system 180 or processing system 100 can determine features in the first vehicle data based on a meshless kernel 1052A. External processing system 180 or processing system 100 can determine one or more second features from second vehicle data in vehicle data, the second vehicle data being derived from a plurality of vehicles (1404). For example, external processing system 180 or processing system 100 can determine features in the second vehicle data based on a meshless kernel 1052B.

[0171] External processing system 180 or processing system 100 may fuse one or more first features and one or more second features to generate a fused feature (1406). For example, external processing system 180 or processing system 100 may fuse meshless core 1052A and meshless core 1052B to generate meshless core 1062.

[0172] External processing system 180 or processing system 100 may generate a BEV representation (1408) based on the fused features. For example, external processing system 180 or processing system 100 may generate a CAV neighborhood dynamic graph 1070 that may be a BEV representation.

[0173] In some examples, external processing system 180 or processing system 100 may transmit a BEV representation to at least one of a plurality of vehicles. For example, external processing system 180 or processing system 100 may transmit a CAV neighborhood dynamic graph 1070 to vehicle 1066A, vehicle 1066B, and / or vehicle 1066N.

[0174] In some examples, at least one of the processors configured to generate BEV representations is located in a first vehicle or a second vehicle. For example, at least one processor of processing system 100 (located in a first vehicle or a second vehicle) can generate BEV representations located in the first vehicle or the second vehicle.

[0175] In some examples, at least one of the one or more processors configured to generate a BEV representation is located outside the first and second vehicles. For example, at least one processor of the external processing system 180 may generate a BEV representation.

[0176] In some examples, the external processing system 180 or the processing system 100 may perform at least one of the following: receiving a first instruction for one or more first features; or receiving a second instruction for one or more second features. In some examples, the processing system 100 may perform at least one of the following: transmitting a first instruction for one or more first features; or receiving a second instruction for one or more second features.

[0177] In some examples, the first vehicle data includes a first meshless kernel, and the second vehicle data includes a second meshless kernel. In some examples, as part of fusing one or more first features and one or more second features, the external processing system 180 may fuse the first and second meshless kernels. In some examples, the first and second meshless kernels include variational autoencoder-Gaussian mixture model (VAE-GMM) kernels.

[0178] In some examples, the vehicle data includes at least one of vehicle attitude, vehicle position, or vehicle trajectory. In some examples, external processing system 180 or processing system 100 may determine not to fuse a third or more features from the third vehicle data. For example, external processing system 180 or processing system 100 may determine not to fuse features of the meshless kernel 1252N. In some examples, the third vehicle data is from a third vehicle (e.g., vehicle 1266N) among multiple vehicles, as part of generating a BEV representation based on at least one of vehicle attitude, vehicle position, or vehicle trajectory. In some examples, as part of determining not to fuse the third or more features, external processing system 180 or processing system 100 may determine the third or more features based on determining that the third vehicle will be outside a neighborhood for less than or equal to a predetermined threshold time amount, the neighborhood including a geographic area that includes multiple vehicles. For example, external processing system 180 or processing system 100 may determine that vehicle 1266N will be outside the neighborhood for less than K seconds.

[0179] In some examples, the external processing system 180 or the processing system 100 may generate a mask based on the overlapping field of view of at least one sensor system of the first vehicle and at least one sensor system of the second vehicle, and apply the mask to a plurality of first features as part of determining one or more first features and apply the mask to a plurality of second features as part of determining one or more second features.

[0180] In some examples, the BEV representation is a unified BEV representation for multiple vehicles. For instance, the unified BEV representation may be based on information from multiple CAVs (e.g., multi-model BEV information) and may be used by one or more of the multiple CAVs, for example, for navigation. In some examples, first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle, and second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein the first sensor data and the second sensor data have different resolutions.

[0181] In some examples, the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle, and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein at least one sensor system in the first plurality of sensor systems has a different type (e.g., camera, LiDAR, radar, etc.) than each sensor system in the second plurality of sensor systems. For example, the first vehicle may include a multimodal sensor set different from the multimodal sensor set of the second vehicle. For example, the multimodal sensor set of the first vehicle may include a radar system, while the multimodal sensor set of the second vehicle may not.

[0182] In some examples, as part of acquiring vehicle data, the external processing system 180 may acquire vehicle data generated over time. In such examples, the vehicle data includes information indicating at least one of a change in the attitude of a first vehicle or a change in the attitude of a second vehicle. The change in the attitude of the first vehicle or the second vehicle may include a change in at least one of rotation or translation.

[0183] In some examples, a first feature in one or more first features corresponds to a second feature in one or more second features. In some examples, the first feature is represented with a different degree of distortion than the second feature.

[0184] In some examples, the first characteristic of one or more first characteristics represents a dynamic object. In some examples, at least a portion of the external processing system 180 resides in a cloud computing environment.

[0185] In some examples, external processing system 180 (or processing system 100 of the third vehicle) may send BEV feature configuration information to the first and second vehicles for configuring BEV processing of the first and second vehicles. For example, BEV feature configuration information 1384 includes at least one of the following: BEV feature vector dimensions; a model index of a model that transforms a 2D camera image into a BEV feature vector; or the model itself. In some examples, external processing system 180 or processing system 100 may receive a BEV feature vector from the first vehicle according to the BEV feature vector dimensions. The BEV feature vector may include an original BEV feature vector or a compressed BEV feature vector. In some examples, the compressed BEV feature vector is compressed using at least one of quantization, pruning, hashing, or transformation. In some examples, the compressed BEV feature vector includes soft BEV features or hard BEV features, wherein soft BEV features include the probability or likelihood of an object's presence or object attributes, and wherein hard BEV features include a binary or categorical representation of an object's presence or object attributes.

[0186] In some examples, before fusing one or more first features and one or more second features, external processing system 180 or processing system 100 may use at least one of quantization, pruning, hashing or transformation to compress one or more first features and one or more second features.

[0187] In some examples, external processing system 180 or processing system 100 includes a mid-term collaborative system. For example, external processing system 180 (or processing system 100 of a third vehicle) may receive instructions for one or more first features from a first vehicle and instructions for one or more second features from a second vehicle. In some examples, as part of generating a BEV representation, external processing system 180 or processing system 100 may discretize at least one vehicle without a mesh kernel. In some examples, external processing system 180 or processing system 100 aligns the meshes associated with the first and second vehicles before generating the BEV representation or as part of generating the BEV representation.

[0188] Additional aspects of this disclosure are detailed in the following numbered clauses.

[0189] Clause 1. A system for processing data from a plurality of vehicles, the system comprising: one or more memories for storing vehicle data from each of the plurality of vehicles, the vehicle data being meshless; and one or more processors communicating with the one or more memories, the one or more processors being configured to: determine one or more first features from first vehicle data in the vehicle data, the first vehicle data being from a first vehicle in the plurality of vehicles; determine one or more second features from second vehicle data in the vehicle data, the second vehicle data being from a second vehicle in the plurality of vehicles; fuse the one or more first features and the one or more second features to generate a fused feature; generate a bird's-eye view (BEV) representation based on the fused feature; and transmit the BEV representation to at least one of the plurality of vehicles.

[0190] Clause 2. The system according to Clause 1, wherein the one or more processors are further configured to: transmit the BEV representation to at least one of the plurality of vehicles.

[0191] Clause 3. The system according to Clause 1 or Clause 2, wherein at least one of the one or more processors configured to generate the BEV representation is located in the first vehicle or the second vehicle.

[0192] Clause 4. The system according to Clause 1 or Clause 2, wherein at least one of the one or more processors configured to generate the BEV representation is located outside the first vehicle and the second vehicle.

[0193] Clause 5. The system according to any one of Clauses 1 to 4, wherein the one or more processors are further configured to perform at least one of: receiving a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

[0194] Clause 6. The system according to any one of Clauses 1 to 5, wherein the one or more processors are further configured to perform at least one of: transmitting a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

[0195] Clause 7. The system according to any one of Clauses 1 to 7, wherein the first gridless vehicle data includes a first gridless core, wherein the second gridless vehicle data includes a second gridless core, and wherein, as part of fusing the one or more first features and the one or more second features, the one or more processors are configured to: fuse the first gridless core and the second gridless core.

[0196] Clause 8. The system according to Clause 7, wherein the first meshless kernel and the second meshless kernel comprise variational autoencoder-Gaussian mixture model (VAE-GMM) kernels.

[0197] Clause 9. The system according to any one of Clauses 1 to 8, wherein the vehicle data includes at least one of vehicle attitude, vehicle position, or vehicle trajectory, and wherein the one or more processors are further configured to: determine not to fuse a third or more features from third vehicle data, said third vehicle data being from a third vehicle among the plurality of vehicles, as part of generating the BEV representation based on said at least one of vehicle attitude, vehicle position, or vehicle trajectory.

[0198] Clause 10. The system according to Clause 9, wherein, as part of determining that the third or more features are not integrated, the one or more processors are configured to: determine the third or more features based on determining that the third vehicle will be outside a neighborhood for a time period less than or equal to a predetermined threshold, the neighborhood comprising a geographic area containing the plurality of vehicles.

[0199] Clause 11. The system according to any one of Clauses 1 to 10, wherein the one or more processors are further configured to: generate a mask based on the overlapping field of view of at least one sensor system of the first vehicle and at least one sensor system of the second vehicle; and apply the mask to a plurality of first features as part of determining the one or more first features, and apply the mask to a plurality of second features as part of determining the one or more second features.

[0200] Clause 12. The system pursuant to any one of Clauses 1 to 11, wherein the BEV representation is a uniform BEV representation of the plurality of vehicles.

[0201] Clause 13. The system according to any one of Clauses 1 to 12, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein the first sensor data and the second sensor data have different resolutions.

[0202] Clause 14. The system according to any one of Clauses 1 to 13, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein at least one sensor system in the first plurality of sensor systems has a different type from each sensor system in the second plurality of sensor systems.

[0203] Clause 15. The system according to any one of Clauses 1 to 14, wherein, as part of obtaining vehicle data, the one or more processors are configured to: obtain vehicle data generated over time, wherein the vehicle data includes information indicating at least one of a change in the attitude of the first vehicle or a change in the attitude of the second vehicle, and wherein the change in the attitude of the first vehicle or the change in the attitude of the second vehicle includes a change in at least one of rotation or translation.

[0204] Clause 16. The system according to any one of Clauses 1 to 15, wherein the first feature of the one or more first features corresponds to the second feature of the one or more second features, and wherein the first feature is represented with a different degree of distortion than the second feature.

[0205] Clause 17. The system according to any one of Clauses 1 to 16, wherein the first feature of the one or more first features represents a dynamic object.

[0206] Clause 18. A system pursuant to any one of Clauses 1 to 17, wherein at least a portion of said system resides in a cloud computing environment.

[0207] Clause 19. The system according to any one of Clauses 1 to 18, wherein the one or more processors are further configured to: send BEV feature configuration information to the first vehicle and the second vehicle for configuring BEV processing of the first vehicle and the second vehicle, wherein the BEV feature configuration information includes at least one of: BEV feature vector size; a model index of a model that transforms a two-dimensional camera image into a BEV feature vector; or the model.

[0208] Clause 20. The system according to Clause 19, wherein the one or more processors are further configured to: receive from the first vehicle a BEV feature vector according to the BEV feature vector size, wherein the BEV feature vector includes the original BEV feature vector or a compressed BEV feature vector.

[0209] Clause 21. The system according to Clause 20, wherein the compressed BEV feature vector is compressed using at least one of quantization, pruning, hashing, or transformation.

[0210] Clause 22. The system according to Clause 20 or Clause 21, wherein the compressed BEV feature vector comprises soft BEV features or hard BEV features, wherein the soft BEV features comprise the probability or likelihood of the existence of an object or an object attribute, and wherein the hard BEV features comprise a binary or categorical representation of the existence of an object or an object attribute.

[0211] Clause 23. The system according to any one of Clauses 1 to 22, wherein the one or more processors are further configured to: compress the one or more first features and the one or more second features using at least one of quantization, pruning, hashing, or transformation before fusing the one or more first features and the one or more second features.

[0212] Clause 24. The system according to any one of Clauses 1 to 23, wherein the system includes a mid-term collaboration system, wherein the one or more processors are further configured to: receive instructions for the one or more first features from the first vehicle; and receive instructions for the one or more second features from the second vehicle.

[0213] Clause 25. The system according to any one of Clauses 1 to 24, wherein, as part of generating the BEV representation, the one or more processors are configured to: discretize the meshless kernel of at least one of the plurality of vehicles.

[0214] Clause 26. The system according to any one of Clauses 1 to 25, wherein the one or more processors are further configured to: align the grid associated with the first vehicle and the second vehicle before generating the BEV representation or as part of generating the BEV representation.

[0215] Clause 27. A method for processing data from a plurality of vehicles, the method comprising: obtaining vehicle data from each of the plurality of vehicles, the vehicle data being gridless; determining one or more first features from first vehicle data in the vehicle data, the first vehicle data being from a first vehicle in the plurality of vehicles; determining one or more second features from second vehicle data in the vehicle data, the second vehicle data being from a second vehicle in the plurality of vehicles; fusing the one or more first features and the one or more second features to generate a fused feature; and generating a bird's-eye view (BEV) representation based on the fused feature.

[0216] Clause 28. The method according to Clause 27, the method further comprising: transferring the BEV representation to at least one of the plurality of vehicles.

[0217] Clause 29. The method described in accordance with Clause 27 or Clause 28, wherein the generation of the BEV means that it occurs in the first vehicle or the second vehicle.

[0218] Clause 30. The method described in accordance with Clause 27 or Clause 28, wherein the generation of the BEV means that it occurs outside the first vehicle and the second vehicle.

[0219] Clause 31. The method according to any one of Clauses 27 to 30, the method further comprising at least one of: receiving a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

[0220] Clause 32. The method according to any one of Clauses 27 to 31, the method further comprising at least one of: transmitting a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

[0221] Clause 33. The method according to any one of Clauses 27 to 32, wherein the vehicle data includes a first gridless kernel, wherein the second vehicle data includes a second gridless kernel, and wherein fusing the one or more first features and the one or more second features includes fusing the first gridless kernel and the second gridless kernel.

[0222] Clause 34. The method according to Clause 33, wherein the first meshless kernel and the second meshless kernel comprise a variational autoencoder-Gaussian mixture model (VAE-GMM) kernel.

[0223] Clause 35. The method according to any one of Clauses 27 to 34, wherein the vehicle data includes at least one of vehicle attitude, vehicle position, or vehicle trajectory, the method further comprising: determining a third or more features not fused from third vehicle data, said third vehicle data being from a third vehicle among the plurality of vehicles, as part of generating the BEV representation based on said at least one of vehicle attitude, vehicle position, or vehicle trajectory.

[0224] Clause 36. The method according to Clause 35, wherein determining that the third or more features are not integrated comprises: determining that the third vehicle will be outside the neighborhood for a time period less than or less than or equal to a predetermined threshold, the neighborhood comprising a geographic area containing the plurality of vehicles.

[0225] Clause 37. The method according to any one of Clauses 27 to 36, the method further comprising: generating a mask based on the overlapping fields of view of at least one sensor system of the first vehicle and at least one sensor system of the second vehicle; and applying the mask to a plurality of first features as a portion of determining the one or more first features, and applying the mask to a plurality of second features as a portion of determining the one or more second features.

[0226] Clause 38. The method according to any one of Clauses 27 to 37, wherein the BEV representation is a uniform BEV representation of the plurality of vehicles.

[0227] Clause 39. The method according to any one of Clauses 27 to 38, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein the first sensor data and the second sensor data have different resolutions.

[0228] Clause 40. The method according to any one of Clauses 27 to 39, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein at least one sensor system in the first plurality of sensor systems has a different type from each sensor system in the second plurality of sensor systems.

[0229] Clause 41. The method according to any one of Clauses 27 to 40, wherein obtaining vehicle data comprises: obtaining vehicle data generated over time, wherein the vehicle data includes information indicating at least one of a change in the attitude of the first vehicle or a change in the attitude of the second vehicle, and wherein the change in the attitude of the first vehicle or the change in the attitude of the second vehicle includes a change in at least one of rotation or translation.

[0230] Clause 42. The method according to any one of Clauses 27 to 41, wherein the first feature of the one or more first features corresponds to the second feature of the one or more second features, and wherein the first feature is represented with a different degree of distortion than the second feature.

[0231] Clause 43. The method according to any one of Clauses 27 to 42, wherein the first feature of the one or more first features represents a dynamic object.

[0232] Clause 44. The method according to any one of Clauses 27 to 43, wherein at least a portion of the method is implemented in a cloud computing environment.

[0233] Clause 45. The method according to any one of Clauses 27 to 44, the method further comprising: sending BEV feature configuration information to the first vehicle and the second vehicle for configuring BEV processing of the first vehicle and the second vehicle, wherein the BEV feature configuration information includes at least one of: BEV feature vector size; a model index of a model that transforms a two-dimensional camera image into a BEV feature vector; or the model.

[0234] Clause 46. The method according to Clause 45, the method further comprising: receiving a BEV feature vector from the first vehicle, wherein the BEV feature vector includes an original BEV feature vector or a compressed BEV feature vector according to the size of the BEV feature vector.

[0235] Clause 47. The method according to Clause 46, wherein the compressed BEV feature vector is compressed using at least one of quantization, pruning, hashing, or transformation.

[0236] Clause 48. The method according to Clause 46 or Clause 47, wherein the compressed BEV feature vector comprises soft BEV features or hard BEV features, wherein the soft BEV features comprise the probability or likelihood of the existence of an object or an object attribute, and wherein the hard BEV features comprise a binary or categorical representation of the existence of an object or an object attribute.

[0237] Clause 49. The method according to any one of Clauses 27 to 48, the method further comprising: compressing the one or more first features and the one or more second features using at least one of quantization, pruning, hashing, or transformation before fusing the one or more first features and the one or more second features.

[0238] Clause 50. The method according to any one of Clauses 27 to 49, wherein the method includes a mid-term collaboration method, and wherein obtaining the vehicle data includes: receiving an indication from the first vehicle for the one or more first features; and receiving an indication from the second vehicle for the one or more second features.

[0239] Clause 51. The method according to any one of Clauses 27 to 50, wherein generating the BEV representation comprises: discretizing at least one of the plurality of vehicles without a mesh kernel.

[0240] Clause 52. The method according to any one of Clauses 27 to 51, the method further comprising: aligning the grid associated with the first vehicle and the second vehicle before generating the BEV representation or as part of generating the BEV representation.

[0241] Clause 53. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: obtain vehicle data from each of a plurality of vehicles, the vehicle data being gridless; determine one or more first features from first vehicle data in the vehicle data, the first vehicle data originating from a first vehicle in the plurality of vehicles; determine one or more second features from second vehicle data in the vehicle data, the second vehicle data originating from a second vehicle in the plurality of vehicles; fuse the one or more first features and the one or more second features to generate a fused feature; and generate a bird's-eye view (BEV) representation based on the fused feature.

[0242] Clause 54. A system for processing data from a plurality of vehicles, the system comprising: components for obtaining vehicle data from each of the plurality of vehicles, the vehicle data being gridless; components for determining one or more first features from first vehicle data in the vehicle data, the first vehicle data being from a first vehicle in the plurality of vehicles; components for determining one or more second features from second vehicle data in the vehicle data, the second vehicle data being from a second vehicle in the plurality of vehicles; components for fusing the one or more first features and the one or more second features to generate a fused feature; and components for generating a bird's-eye view (BEV) representation based on the fused feature.

[0243] It should be recognized that, based on the examples, certain actions or events of any technique described herein may be performed in a different sequence, and may be added, combined, or omitted entirely (e.g., not all actions or events described are necessary for implementing the technique). Furthermore, in some examples, actions or events may be performed concurrently (e.g., through multithreading, interrupt handling, or multiple processors) rather than sequentially.

[0244] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted via a computer-readable medium and applied by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium (which corresponds to a tangible medium such as a data storage medium) or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products may include computer-readable media.

[0245] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies (such as infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies (such as infrared, radio, and microwave) are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs utilize lasers to optically reproduce data. The combinations described above should also be included within the scope of computer-readable media.

[0246] Instructions can be applied by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms "processor" and "processing circuit" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0247] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but implementation by different hardware units is not necessarily required. Rather, as described above, various units can be combined in a codec hardware unit, or various units can be provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0248] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A system for processing data from multiple vehicles, the system comprising: One or more memories are provided for storing vehicle data from each of the plurality of vehicles, the vehicle data being gridless; and One or more processors, which communicate with one or more memories, are configured to: One or more first features are determined from the first vehicle data in the vehicle data, wherein the first vehicle data comes from the first vehicle among the plurality of vehicles; One or more second features are determined from the second vehicle data in the vehicle data, wherein the second vehicle data is derived from a second vehicle among the plurality of vehicles; The one or more first features and the one or more second features are fused to generate a fused feature; as well as A bird's-eye view (BEV) representation is generated based on the fused features.

2. The system of claim 1, wherein the one or more processors are further configured to: transmit the BEV representation to at least one of the plurality of vehicles.

3. The system of claim 1, wherein at least one of the one or more processors configured to generate the BEV representation is located in the first vehicle or the second vehicle.

4. The system of claim 1, wherein at least one of the one or more processors configured to generate the BEV representation is located outside the first vehicle and the second vehicle.

5. The system of claim 1, wherein the one or more processors are further configured to perform at least one of the following: receiving a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

6. The system of claim 1, wherein the one or more processors are further configured to perform at least one of the following: transmitting a first instruction for the one or more first features; or receiving a second instruction for the one or more second features.

7. The system of claim 1, wherein the first vehicle data includes a first meshless core, wherein the second vehicle data includes a second meshless core, and wherein, as part of fusing the one or more first features and the one or more second features, the one or more processors are configured to fuse the first meshless core and the second meshless core.

8. The system of claim 7, wherein the first meshless kernel and the second meshless kernel comprise variational autoencoder-Gaussian mixture model (VAE-GMM) kernels.

9. The system of claim 1, wherein the vehicle data includes at least one of vehicle attitude, vehicle position, or vehicle trajectory, and wherein the one or more processors are further configured to: determine not to fuse a third or more features from third vehicle data, said third vehicle data being from a third vehicle among the plurality of vehicles, as part of generating the BEV representation based on said at least one of vehicle attitude, vehicle position, or vehicle trajectory.

10. The system of claim 9, wherein, as part of determining that the third or more features are not integrated, the one or more processors are configured to: determine the third or more features based on determining that the third vehicle will be outside a neighborhood for a time period less than or equal to a predetermined threshold, the neighborhood comprising a geographical region containing the plurality of vehicles.

11. The system of claim 1, wherein the one or more processors are further configured to: A mask is generated based on the overlapping fields of view of at least one sensor system of the first vehicle and at least one sensor system of the second vehicle; and The mask is applied to a plurality of first features as part of determining the one or more first features, and the mask is applied to a plurality of second features as part of determining the one or more second features.

12. The system of claim 1, wherein the BEV representation is a unified BEV representation of the plurality of vehicles.

13. The system of claim 1, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein the first sensor data and the second sensor data have different resolutions.

14. The system of claim 1, wherein the first vehicle data is based on first sensor data from a first plurality of sensor systems of the first vehicle and the second vehicle data is based on second sensor data from a second plurality of sensor systems of the second vehicle, wherein at least one sensor system in the first plurality of sensor systems has a different type from each sensor system in the second plurality of sensor systems.

15. The system of claim 1, wherein the one or more processors are configured to: obtain vehicle data generated over time, wherein the vehicle data includes information indicating at least one of a change in the attitude of the first vehicle or a change in the attitude of the second vehicle, and wherein the change in the attitude of the first vehicle or the change in the attitude of the second vehicle includes a change in at least one of rotation or translation.

16. The system of claim 1, wherein the first feature of the one or more first features corresponds to the second feature of the one or more second features, and wherein the first feature is represented with a different degree of distortion than the second feature.

17. The system of claim 1, wherein the first feature of the one or more first features represents a dynamic object.

18. The system of claim 1, wherein at least a portion of the system resides in a cloud computing environment.

19. The system of claim 1, wherein the one or more processors are further configured to: send BEV feature configuration information to the first vehicle and the second vehicle for configuring BEV processing of the first vehicle and the second vehicle, wherein the BEV feature configuration information includes at least one of the following: BEV feature vector size; a model index of a model that transforms a two-dimensional camera image into a BEV feature vector; or the model.

20. The system of claim 19, wherein the one or more processors are further configured to: receive from the first vehicle a BEV feature vector according to the BEV feature vector size, wherein the BEV feature vector includes an original BEV feature vector or a compressed BEV feature vector.

21. The system of claim 20, wherein the compressed BEV feature vector is compressed using at least one of quantization, pruning, hashing, or transformation.

22. The system of claim 20, wherein the compressed BEV feature vector comprises soft BEV features or hard BEV features, wherein the soft BEV features comprise the probability or likelihood of object presence or object attributes, and wherein the hard BEV features comprise a binary or categorical representation of object presence or object attributes.

23. The system of claim 1, wherein the one or more processors are further configured to: compress the one or more first features and the one or more second features using at least one of quantization, pruning, hashing, or transformation before fusing the one or more first features and the one or more second features.

24. The system of claim 1, wherein the system includes a mid-term cooperative system, and wherein the one or more processors are further configured to: Receive instructions for one or more of the first features from the first vehicle; and Receive instructions for one or more of the second features from the second vehicle.

25. The system of claim 1, wherein, as part of generating the BEV representation, the one or more processors are configured to: discretize the meshless kernel of at least one of the plurality of vehicles.

26. The system of claim 1, wherein the one or more processors are further configured to: align the grid associated with the first vehicle and the second vehicle before generating the BEV representation or as part of generating the BEV representation.

27. A method for processing data from multiple modes of transportation, the method comprising: Obtain vehicle data from each of the plurality of vehicles, wherein the vehicle data is gridless; One or more first features are determined from the first vehicle data in the vehicle data, wherein the first vehicle data comes from the first vehicle among the plurality of vehicles; One or more second features are determined from the second vehicle data in the vehicle data, wherein the second vehicle data is derived from a second vehicle among the plurality of vehicles; The one or more first features and the one or more second features are fused to generate a fused feature; A bird's-eye view (BEV) representation is generated based on the fused features; as well as The BEV represents the vehicle being transported to at least one of the plurality of vehicles.

28. The method of claim 27, wherein the first vehicle data includes a first gridless kernel, the second vehicle data includes a second gridless kernel, and wherein fusing the one or more first features and the one or more second features includes: The first meshless kernel and the second meshless kernel are fused together.

29. The method of claim 28, wherein the first meshless kernel and the second meshless kernel comprise a variational autoencoder-Gaussian mixture model (VAE-GMM) kernel.

30. A system for processing data from multiple vehicles, the system comprising: Components for obtaining vehicle data from each of the plurality of vehicles, the vehicle data being gridless; Components for determining one or more first features from first vehicle data in the vehicle data, wherein the first vehicle data is derived from a first vehicle among the plurality of vehicles; Components for determining one or more second features from second vehicle data in the vehicle data, wherein the second vehicle data is derived from a second vehicle among the plurality of vehicles; Components for fusing the one or more first features and the one or more second features to generate a fused feature; Components used to generate BEV representations based on the fused features; and A component for transferring the BEV representation to at least one of the plurality of vehicles.

Images