A method and system for dynamic scene reconstruction using online mapping pre-training and sensor generalization

By classifying sensor data into static and dynamic objects and generating four-dimensional representations, and combining Gaussian stitching and stream matching techniques, the mapping delay and adaptability issues of autonomous driving systems under different sensor configurations are solved, achieving efficient environmental representation and real-time updates.

CN123071608APending Publication Date: 2026-11-10ROBERT BOSCH GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610622983.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2025-05-08
Filing Date
2026-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing autonomous driving systems struggle to achieve seamless integration and high-fidelity environmental representation when faced with different sensor configurations, resulting in latency and insufficient adaptability in environmental mapping, especially in dynamic scenarios where generalization and robustness are inadequate.

Method used

By receiving previously generated sensor datasets, classifying them into static and dynamic object data, performing volumetric scene reconstruction, generating a four-dimensional representation, and converting it into a synthetic dataset aligned with new sensor specifications, rasterization and smoothing are performed using Gaussian stitching and diffusion/flow matching techniques to ensure accurate data synthesis.

Benefits of technology

It enables efficient online mapping and pre-training across different sensor configurations, enhancing the adaptability and robustness of autonomous driving systems, ensuring the accuracy and consistency of environmental representation, and supporting real-time environmental updates and safe navigation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071608A_ABST
    Figure CN123071608A_ABST
Patent Text Reader

Abstract

A method and system for dynamic scene reconstruction using online mapping pre-training and sensor generalization. The method and system for synthesizing sensor data for online mapping pre-training and sensor generalization address the challenge of adapting to different sensor configurations. Previously generated sensor datasets are classified into datasets of static and dynamic object data, and volumetric scene reconstruction is performed to produce a four-dimensional scene representation. New sensor datasets are synthesized by aligning the derived scene representation with new sensor specifications, thereby facilitating improved generalization and enhanced adaptability in autonomous driving systems.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure generally relates to autonomous driving technology. In specific embodiments, this disclosure relates to methods and systems for dynamic scene reconstruction and sensor data synthesis using online mapping pre-training and sensor generalization. Background Technology

[0002] In the fields of autonomous driving and advanced mapping systems, vehicles increasingly rely on sophisticated sensor arrays to generate accurate, real-time depictions of their surroundings. Modern mapping methods play a crucial role in interpreting dynamic environments that encompass both static structures and transient elements such as moving vehicles and pedestrians. However, the variability in sensor placement, calibration, and configuration across different platforms often results in heterogeneous data representations. These inconsistencies challenge the reliability of environment modeling and hinder the seamless integration of sensor data into coherent and actionable maps.

[0003] Existing methods that utilize datasets generated under fixed sensor conditions often require significant manual intervention or rigid update protocols when adapting to new sensor configurations. This reliance can introduce latency and limit adaptability under rapidly changing conditions where the dynamic nature of the environment needs to be mapped quickly and accurately. Furthermore, conventional methods may struggle to capture the full complexity of dynamic scenes, sometimes resulting in models that are inadequate in terms of generalization and robustness. Coordinating sensor data across various configurations while ensuring high-fidelity environmental representations under disparate operating conditions is both necessary and challenging. Summary of the Invention

[0004] In one embodiment, a method is provided for synthesizing sensor data for online mapping pre-training and sensor generalization. The method includes: receiving a previously generated sensor dataset associated with a scene, the dataset having been captured by a first plurality of sensors having associated sensor specifications; classifying the sensor data into static object data corresponding to static objects and dynamic object data corresponding to dynamic objects; performing volumetric scene reconstruction on the static and dynamic object data to generate a four-dimensional representation of the scene; generating a new sensor dataset for the scene, including sensor data captured by a second plurality of sensors having sensor specifications different from those of the first plurality of sensors; and synthesizing the new sensor dataset by aligning the four-dimensional representation to the new sensor specifications to produce a synthesized dataset, wherein in some embodiments, the synthesis further includes operations such as rasterization of the four-dimensional representation to a two-dimensional format, smoothing and interpolating the diffusion of rasterized data, and optionally stream matching operations to align the streams of data points; some embodiments incorporate the generation of a four-dimensional simulation of the scene and utilize previously generated sensor datasets, including spatiotemporal data captured by cameras, LiDAR (Light Detection and Ranging) or radar sensors.

[0005] In one embodiment, a system is provided for synthesizing sensor data for online mapping pre-training and sensor generalization. The system includes: a storage device configured to store previously generated sensor datasets associated with a scene; a second plurality of sensors located around a vehicle; and a processor coupled to the storage device and the sensors; the processor is programmed to receive the previously generated sensor datasets; process the sensor data to classify the data into static object data and dynamic object data; perform volumetric scene reconstruction on the classified data to generate a four-dimensional representation of the scene; generate a new sensor dataset associated with the scene, wherein the new dataset is captured by the second plurality of sensors having sensor specifications different from those of the first plurality of sensors; and synthesize the new sensor dataset by aligning the four-dimensional representation to the new sensor specifications to produce a synthesized dataset. In various embodiments, the synthesis further includes: rasterization of the four-dimensional representation into a two-dimensional format; smoothing and interpolating the diffusion of the rasterized data; stream matching operations to align the streams of data points; generation of a four-dimensional simulation of the scene; updating an HD map; and, in some cases, the operation of the system utilizes previously generated sensor datasets, which include spatiotemporal data captured by camera, LiDAR, or radar sensors.

[0006] In one embodiment, a non-transitory computer-readable medium is provided, the medium including instructions that, when executed by a processor, cause the processor to perform the following operations: receiving a previously generated sensor dataset associated with a scene, the dataset having been captured by a first plurality of sensors having associated sensor specifications; classifying the sensor data into static object data and dynamic object data; performing volumetric scene reconstruction on the classified data to generate a four-dimensional representation of the scene; generating a new sensor dataset of the scene based on sensor data captured by a second plurality of sensors having sensor specifications different from those of the first plurality of sensors; and generating a synthetic dataset by aligning the four-dimensional representation to the new sensor specifications to synthesize the new sensor dataset, wherein in some embodiments, the instructions also cause the processor to update an HD map and control the movement of a vehicle based on the synthetic dataset. Attached Figure Description

[0007] Figure 1 A system for training a neural network according to one embodiment is shown.

[0008] Figure 2 A computer-implemented method for training and utilizing a neural network, according to one embodiment, is shown.

[0009] Figure 3A system-level overview of a sensor data synthesis system designed for online mapping pre-training and sensor generalization, according to one embodiment, is shown.

[0010] Figure 4 A method for synthesizing sensor data for online mapping pre-training and sensor generalization is illustrated according to one embodiment.

[0011] Figure 5 A schematic diagram depicts the interaction between a computer-controlled machine and a control system according to one embodiment.

[0012] Figure 6 A description of an embodiment Figure 5 A schematic diagram of a control system configured to control a vehicle, which may be a partially automated vehicle, a fully automated vehicle, a partially autonomous robot, or a fully autonomous robot. Detailed Implementation

[0013] Embodiments of this disclosure are described herein. However, it is to be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily to scale; some features may be exaggerated or minimized to show detail of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be desired.

[0014] Unless the context clearly specifies otherwise, as used herein, “a,” “an,” and “the” refer to both singular and plural pronouns. By way of example, a “processor” programmed to perform various functions means one processor programmed to perform each of the functions, or more than one processor collectively programmed to perform each of the various functions.

[0015] In the field of autonomous driving, the generation and maintenance of high-definition (HD) maps are crucial for accurate navigation and decision-making. These maps provide a detailed representation of the environment, including road layouts, traffic signs, and other salient elements. However, the dynamic nature of urban and highway environments presents substantial challenges to maintaining the accuracy and relevance of these maps. Construction activities, changes in traffic patterns, and the presence of dynamic objects such as vehicles and pedestrians can quickly render existing maps obsolete, requiring frequent updates. Traditional methods for updating HD maps involve manual labeling and verification, which is both time-consuming and resource-intensive, limiting the scalability for widespread autonomous driving applications.

[0016] Recent advances have sought to address these challenges by generating maps in real-time using camera data, thereby reducing reliance on manually labeled maps. However, these online methods face a significant limitation: a lack of generalization across different sensor configurations. Online map models trained in one sensor setting may not perform optimally when applied to different settings, hindering the adaptability of automated systems across various vehicle platforms. This limitation underscores the need for methods capable of synthesizing data specifically tailored to new or particular sensor configurations, ensuring consistent performance regardless of sensor setup.

[0017] This framework introduces a novel approach for 3D / 4D dynamic scene reconstruction, significantly enhancing the generalization capabilities of vision-driven online map generation architectures. By processing driving data sequences with spatiotemporal continuity, the framework uses multiple cameras and optional LiDAR data to reconstruct both static and dynamic components of a scene. Static components include large and small structures such as buildings, drivable areas, and road signs, while dynamic components encompass moving objects such as vehicles and pedestrians. The framework employs advanced techniques such as Gaussian stitching and diffusion / flow matching to synthesize scenes from new perspectives, adapting to different sensor configurations. This synthetic data regeneration method enables scene synthesis from various angles, facilitating online map model generalization, model pre-training, and training data performance enhancement.

[0018] Machine learning model training and pre-training benefit from the teachings of this disclosure. About Figures 1-2 The following description provides a general overview of machine learning techniques related to methods used to leverage machine learning models, such as those described in this article.

[0019] Figure 1 The illustration depicts a system 100 for training and utilizing machine learning models, such as convolutional neural networks, according to some embodiments. It should be understood that, although this document pertains to... Figure 1 and Figure 2 The example embodiments given in the following paragraphs mention convolutional neural networks, but... Figure 1 and Figure 2 Other embodiments can be applied to any other type of neural network-based or non-neural network-based machine learning model that is developed, trained, fine-tuned, and / or executed for the various applications of audio tracking and interpretation further described herein.

[0020] In some embodiments, system 100 may include an input interface for accessing a training dataset 102 of a convolutional neural network. For example, such as Figure 1 As illustrated, the input interface can be comprised of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or persistent storage interface, such as a hard disk or SSD interface, but it can also be a LAN, local area network, or wide area network interface, such as a Bluetooth, ZigBee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but it can also be an external data storage device, such as a network-accessible data storage device.

[0021] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained version of the model (e.g., a version of a machine learning model that has not yet been trained), which can be accessed by system 100 from data storage device 106. However, it will be understood that training data 102 and the data representation 108 of the pre-trained convolutional neural network may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may be of the type described above for data storage interface 104. In other embodiments, the data representation 108 of the pre-trained convolutional neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may not be explicitly stored on data storage device 106. System 100 may further include a processor subsystem 110, which may be configured to provide an iterative function as an alternative to the layer stack of the convolutional neural network to be trained during operation of system 100. Here, the corresponding layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer (or, for the first layer of the layer stack, the original activation) and a portion of the input of the layer stack as input. The processor subsystem 110 can also be configured to iteratively train and / or fine-tune the convolutional neural network using training data 102 (e.g., thereby generating an updated version of the machine learning model relative to a first “pre-trained” version of the model). Here, the iterations of training performed by the processor subsystem 110 may include a forward propagation portion and a backward (or generative) propagation portion.

[0022] System 100 may further include an output interface for outputting a data representation 112 of a trained convolutional neural network, and this data may also be referred to as trained model data 112. For example, as well as Figure 1 As illustrated, the output interface can be comprised of a data storage interface 104, wherein in these embodiments, the interface is an input / output (“IO”) interface through which trained model data 112 can be stored in a data storage device 106. For example, the data representation 108 defining a “pre-trained” convolutional neural network can be at least partially replaced by the data representation 112 of the trained neural network during or after training, because the parameters of the convolutional neural network (such as the weights, hyperparameters, and other types of parameters) can be adapted to reflect training on the training data 102. This also... Figure 1 The figures are illustrated by reference numerals 108 and 112 relating to the same data records on data storage device 106. In other embodiments, data representation 112 may be stored separately from data representation 108 defining the "pre-trained" convolutional neural network. In some embodiments, the output interface may be separate from data storage interface 104, but it can typically be of the type described above for data storage interface 104.

[0023] Figure 1 The system 100 shown is an example of a system that can be used to train and then execute trained machine learning models (such as those described in this paper).

[0024] Figure 2 The illustration depicts a computer-implemented system for training, pre-training, and / or utilizing models such as convolutional neural networks, according to some embodiments. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206 (and, in some embodiments, a graphics processing unit (GPU)). CPU 206 may be a commercially available processing unit implementing an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0025] In this embodiment, GPUs can be relied upon for training, pre-training, and utilizing models such as convolutional neural networks. GPUs are particularly well-suited for handling the parallel processing demands of training complex machine learning models, offering significant performance advantages over traditional CPUs. By utilizing GPUs, the system can efficiently manage the massive computations required to process high-dimensional data and execute advanced algorithms. The architecture of GPUs allows for the simultaneous execution of multiple operations, which accelerates the training process and enhances the system's ability to handle large datasets. This capability supports rapid iteration and refinement of models, thereby improving accuracy and performance in real-time applications. The integration of GPU resources within the computing system 202 ensures that the system can meet the computational demands of modern machine learning tasks, providing a robust platform for developing and deploying advanced autonomous driving technologies.

[0026] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is deactivated or powered off. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store machine learning model 210 or algorithm, training dataset 212 of machine learning model 210, raw source dataset 214, etc.

[0027] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include wired and / or wireless Ethernet interfaces as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard family. The network interface device 222 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 222 may be further configured to provide a communication interface to an external network 224 or the cloud.

[0028] External network 224 may be referred to as the World Wide Web or the Internet. External network 224 can establish standard communication protocols between computing devices. External network 224 can allow easy exchange of information and data between computing devices and the network. One or more servers 230 can communicate with external network 224.

[0029] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0030] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control input. Examples of input devices may include HMI inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 226. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 226. The display device 226 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with remote HMIs and remote display devices via a network interface device 222.

[0031] System 200 can be implemented using one or more computing systems. While this example depicts a single computing system 202 implementing all the described features, it is intended that various features and functions can be decoupled and implemented through multiple computing units that communicate with each other. The specific system architecture chosen can depend on several factors.

[0032] System 200 can implement machine learning algorithm 210 configured to analyze raw source dataset 214. Raw source dataset 214 may include unprocessed or unprocessed sensor data, which may represent the input dataset of the machine learning system. For example, according to the teachings provided herein, raw source dataset 214 may include data collected by vehicle sensors for constructing a 3D / 4D representation of a scene. In some examples, machine learning algorithm 210 may be a convolutional neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm may be configured to receive audio clips and text-based event descriptions.

[0033] Computer system 200 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a set of previously constructed data used to train machine learning algorithm 210. Machine learning algorithm 210 may use training dataset 212 to learn weighting factors associated with a convolutional neural network algorithm. Training dataset 212 may include a set of source data having the corresponding results or outcomes that machine learning algorithm 210 attempts to replicate via a learning process. In an example, training dataset 212 may include previously stored data from vehicle sensors, where the previously stored data reflects or describes objects (e.g., position, speed, orientation, etc.) detected by the sensors around the vehicle.

[0034] Machine learning algorithm 210 can operate in a learning mode using training dataset 212 as input. Machine learning algorithm 210 can be executed on multiple iterations using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 210 can compare its outputs (e.g., annotations) with those included in training dataset 212. Since training dataset 212 includes the expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212), machine learning algorithm 210 can be executed using data not in training dataset 212. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0035] Machine learning algorithm 210 can be configured to identify specific features in raw source data 214. Raw source data 214 may include multiple instances or input datasets to which annotation results are expected. Machine learning algorithm 210 can be programmed to process raw source data 214 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 214 as predetermined features. Raw source data 214 can be derived from various sources. For example, raw source data 214 can be actual input data collected by a machine learning system.

[0036] In the context of this invention, Figures 1-2 The machine learning system described is an example of a system for processing and utilizing camera data for dynamic scene reconstruction. For example... Figure 2 As shown, the exemplary system allows camera data to be input via an input interface, which is then processed by processor 204 to generate a dynamic reconstruction of the scene. This process involves accessing the original online mapping sensor dataset and the original sensor specifications from storage device 208. Using these datasets, the system can adaptively synthesize new sensor data aligned with the new sensor specifications 312, ensuring accurate representation of the reconstructed scene across different sensor configurations. This capability enhances the generalization and adaptability of the online map generation architecture, allowing for improved performance in autonomous driving applications.

[0037] Figure 3The illustration shows a system-level overview of a sensor data synthesis system designed for online mapping pre-training and sensor generalization according to one embodiment. The system is configured to process sensor datasets, such as a raw online mapping sensor dataset 302, and generate new sensor data aligned with different sensor specifications, including a new sensor specification 312. The system includes several components, including dynamic object modeling 306, static object modeling 308, and a rasterization and diffusion-flow matching module 310, ultimately producing a new online mapping sensor dataset 314.

[0038] The raw online mapped sensor dataset 302 serves as the basic input to the system, comprising sensor data captured by a first set of sensors. This dataset includes detailed information about the scene, providing necessary data for subsequent processing and modeling. This dataset 302 may be referred to as the previously generated sensor dataset associated with the scene. The dataset 302 also includes sensor data captured by a first plurality of sensors, which may include one or more of the following: cameras, LiDAR sensors, radar sensors, ultrasonic sensors, inertial measurement unit (IMU) sensors, etc.

[0039] Accompanying this dataset is the original sensor specification 304, which includes intrinsic and extrinsic parameters for the first plurality of sensors used to capture the raw data. Intrinsic parameters describe the sensor's internal characteristics, such as focal length, optical center, and lens distortion coefficients. For example, focal length determines the camera's field of view, while the optical center indicates the point where light rays converge on the sensor. Lens distortion coefficients account for any wrapping effects caused by the lens, ensuring accurate correction of the captured image for geometric distortion. These intrinsic parameters are used to calibrate the sensors and ensure that the data they capture is accurate and reliable. Extrinsic parameters refer to the spatial positioning and orientation of the sensors relative to a fixed frame of reference, such as the vehicle or platform on which they are mounted. For example, extrinsic parameters can include translation and rotation vectors describing the sensor's position and angle relative to the vehicle coordinate system. This information is used to align and integrate data from multiple sensors, allowing for a coherent and comprehensive representation of the scene. By accurately defining both intrinsic and extrinsic parameters, the original sensor specification 304 enables the system to interpret sensor data with high fidelity, ensuring that the reconstructed scene accurately reflects the real-world environment. This detailed specification allows for tasks such as sensor fusion, where data from different sensors must be combined to create a unified view of the scene and to adapt the data to new sensor configurations during the fusion process.

[0040] Raw sensor data 302 and raw sensor specifications 304 allow for the classification of sensor data into static and dynamic object data. Utilizing intrinsic and extrinsic specifications, the system can distinguish between static and dynamic elements based on their spatial and temporal characteristics. Static object data can be identified by analyzing the consistent spatial attributes and fixed positions of objects that remain unchanged over time, such as buildings, road signs, and drivable areas. In contrast, dynamic object data can be characterized by changes in position, velocity, and orientation, indicating objects exhibiting movement and interaction within the scene, such as vehicles and pedestrians. With the help of detailed sensor specification 304, the system can effectively classify sensor data, ensuring accurate classification and representation of both static and dynamic components in subsequent modeling processes. This classification allows for the construction of a reliable and comprehensive four-dimensional representation of the scene, which serves as the basis for further synthesis and alignment with new sensor configurations.

[0041] Following these ideas, system 300 can employ dynamic object modeling 306, which represents a model responsible for classifying and reconstructing dynamic objects (such as vehicles and pedestrians) within a scene. This model can be trained to function as a tuned machine learning model based on the teachings provided above. The model uses the intrinsic properties of the dynamic objects (such as their shape, size, and color) along with extrinsic parameters (such as their position and orientation relative to sensors) to create a detailed and accurate model of the movement of these dynamic objects. By continuously tracking these parameters, the module can simulate the interaction of objects with their environment and other objects, capturing their dynamic behavior over time. This allows for accurate modeling of complex scenes (such as traffic flow and pedestrian movement), which is crucial for realistic scene reconstruction and autonomous navigation.

[0042] In contrast, static object modeling 308 is a model configured to reconstruct static elements within a scene, such as buildings, road signs, and drivable areas. This model employs advanced algorithms to analyze and interpret the spatial data associated with these static objects, ensuring accurate capture of their geometry, spatial orientation, and surface properties. By maintaining a detailed and consistent representation of these static elements, the model provides a stable reference frame supporting the dynamic components of the scene. This stability aids in overall scene reconstruction because it allows for accurate localization and tracking of dynamic objects relative to the static environment, thereby enhancing the reliability and accuracy of the synthesized scene data. Additionally, static object modeling contributes to the system's ability to cope with congestion and maintain continuity within the scene, essential for applications such as autonomous navigation and real-time mapping.

[0043] The Static Object Modeling 308 and Dynamic Object Modeling 306 components work together to output a four-dimensional representation of the scene. This "four-dimensional" representation refers to a model that incorporates both spatial and temporal dimensions to provide a comprehensive depiction of the environment; this model is rooted in Dynamic Object Modeling 306.

[0044] Then, system 300 performs rasterization and diffusion and flow matching model 310. This can be more generally referred to as the synthetic model at 310, whereby the four-dimensional representation of the scene is aligned with the new sensor dataset. Model 310 is configured to transform the four-dimensional representation of the scene into a format compatible with the new sensor specification 312. In this embodiment, the transformation begins with rasterization, the process of converting volumetric data into a two-dimensional mesh. By mapping complex multi-dimensional scene data onto a simpler 2D mesh format, rasterization facilitates the alignment of the scene with the field of view and resolution of the new sensor setting. This step allows ensuring that the synthetic data is accessible and interpretable by the new sensor, which may have different intrinsic and extrinsic parameters compared to the original sensor specification. The rasterization process can take into account variations in sensor resolution, field of view, and other specifications to ensure accurate representation of the data in the new format.

[0045] Following rasterization, diffusion techniques coupled with flow matching are applied to the rasterized data to enhance its quality and consistency. Diffusion involves smoothing and interpolating the data, which helps reduce artifacts and discontinuities that may occur during the rasterization process. This smoothing allows the integrity of the scene representation to be maintained, ensuring that transitions between different elements are seamless and realistic. Flow matching further refines the process by aligning the flow of data points, ensuring that the dynamic aspects of the scene are accurately captured and represented. Flow matching is a technique used to enhance the accuracy and consistency of data representation by aligning the flow of data points within a scene. In the context of synthesizing sensor data, flow matching ensures that the dynamic aspects of the scene, such as moving objects and changing environmental conditions, are accurately captured and represented. In embodiments, flow matching involves tracking data points over time (e.g., location, orientation, etc.), identifying the paths of objects, and aligning them to create a coherent and continuous representation. This alignment allows for the maintenance of temporal consistency of the scene because it ensures that the movement of objects is represented smoothly without abrupt changes or discontinuities. By aligning the flow of data points, flow matching helps preserve the natural movement and interaction of dynamic elements within the scene, such as vehicles and pedestrians. Additionally, stream matching helps interpolate missing or sparse data points, filling gaps in the dataset to create a more complete and accurate representation of the scene. This process enhances the overall quality of the synthetic dataset, ensuring it accurately reflects the dynamic nature of the environment and is well-suited for real-time applications.

[0046] By aligning the rasterized, diffused, and stream-matched data with the new sensor specification 312, the system ensures that the synthetic dataset accurately reflects the scene sensed by the new sensor setup. This alignment enables the system to adapt to different sensor configurations, enhancing its generalization capabilities and ensuring consistent performance across various platforms. The integration of these advanced techniques within model 310 enables the system to generate high-quality, adaptable sensor data for online mapping applications.

[0047] Finally, the new online mapping sensor dataset 314 represents the system's final output: a synthetic dataset based on the alignment of four-dimensional scene representation with the new sensor specifications. This dataset facilitates improved generalization and adaptability of the online map generation architecture across different sensor settings, enhancing the robustness and performance of autonomous driving systems. "Online" signifies that the system can process and update map data in real time upon receiving new sensor information, allowing for immediate adaptation to changes in environment and sensor configuration without offline processing or manual intervention.

[0048] Figure 4 A method for synthesizing sensor data for online mapping pre-training and sensor generalization is illustrated. This method can be implemented by a system designed to process sensor datasets (such as the original online mapping sensor dataset 302 and the original sensor specification 304) and generate synthetic data aligned with the new sensor specification 312, enhancing the adaptability and performance of autonomous driving systems.

[0049] In step 402, the method involves receiving a previously generated sensor dataset associated with the scene. This dataset includes sensor data captured by a plurality of sensors (e.g., cameras, LiDAR, radar, ultrasound, IMU, etc.), providing detailed information about the scene for subsequent processing and modeling. This sensor data may have already been stored in memory 208, for example, as raw source data 214.

[0050] In step 404, the method includes classifying the sensor data into static object data and dynamic object data. Static object data is associated with objects that remain unchanged over time (such as buildings and road signs), while dynamic object data is associated with objects that exhibit movement (such as vehicles and pedestrians).

[0051] In step 406, the method performs volumetric scene reconstruction on both static and dynamic object data to generate a four-dimensional representation of the scene. This representation incorporates spatial and temporal dimensions, providing a comprehensive depiction of the environment.

[0052] In step 408, the method includes generating a new sensor dataset 408 associated with the scene. This new dataset includes sensor data captured by a second plurality of sensors having a new sensor specification 312 that differs from the sensor specifications of the first plurality of sensors.

[0053] In step 410, the method includes synthesizing a new sensor dataset 408 by aligning a four-dimensional representation of the scene to a new sensor specification 312. This alignment ensures that the synthesized dataset 412 accurately reflects the scene perceived through the new sensor setup.

[0054] In step 412, the method generates a synthetic dataset based on the synthesis process. This dataset facilitates improved generalization and adaptability of online map generation architectures across different sensor settings, enhancing the robustness and performance of autonomous driving systems. Step 412 can be part of step 410 because synthesis can generate synthetic data.

[0055] Once the synthetic dataset is generated in step 412, it can provide several capabilities to enhance the capabilities of the autonomous driving system. The synthetic data can be particularly valuable for online mapping and navigation, providing a real-time updated representation of the environment. This allows autonomous vehicles to adapt to changes such as new obstacles, road conditions, or traffic patterns, ensuring safe and efficient navigation. By aligning the synthetic data with new sensor specifications, the dataset enables sensor generalization, allowing the autonomous driving system to perform consistently across different sensor configurations. This adaptability is essential for vehicles equipped with different sensor settings. Furthermore, the synthetic dataset informs the control algorithms and decision-making processes within the vehicle, providing accurate and up-to-date information about the environment to enhance the safety and efficiency of maneuvering. The synthetic dataset also supports predictive modeling, enabling the anticipation of future environmental conditions, such as the movement of other vehicles or pedestrians, which is crucial for proactive planning and collision avoidance. Additionally, the synthetic data can be used in simulated environments to test and validate autonomous driving algorithms, ensuring the vehicle's system is robust and capable of handling a wide range of conditions.

[0056] Figure 5 The diagram illustrates the structure of machine learning models used for training and using these applications (and other applications). Figure 5A schematic diagram depicting the interaction between a computer-controlled machine 500 and a control system 502 is shown. The computer-controlled machine 500 includes actuators 504 and sensors 506. Actuators 504 may include one or more actuators, and sensors 506 may include one or more sensors. Sensors 506 are configured to sense the condition of the computer-controlled machine 500. Sensors 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of sensors 506 include video, radar, LiDAR, ultrasonic, and motion sensors. In one embodiment, sensor 506 is an optical sensor configured to sense an optical image of the environment near the computer-controlled machine 500.

[0057] The control system 502 is configured to receive sensor signals 508 from a computer-controlled machine 500. As described below, the control system 502 can be further configured to calculate an actuator control command 510 based on the sensor signals and transmit the actuator control command 510 to the actuator 504 of the computer-controlled machine 500.

[0058] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive sensor signals 508 from the sensor 506 and transform these sensor signals into input signals x. In an alternative embodiment, without the receiving unit 512, the sensor signals 508 are received directly as input signals x. Each input signal x may be a portion of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to the image recorded by the sensor 506.

[0059] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify an input signal x into one or more labels using a machine learning algorithm (such as the neural network described above). The classifier 514 is configured to be parameterized by parameters (such as those described above, for example, parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y based on the input signal x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to an actuator 504, which is configured to actuate a computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0060] Upon receiving an actuator control command 510, actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. Actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling actuator 504. In one or more embodiments, instead of an actuator or in addition to an actuator, actuator control command 510 may be used to control a display.

[0061] In another embodiment, instead of the computer-controlled machine 500, the control system 502 may include sensor 506, or in addition to the computer-controlled machine 500 including sensor 506. Instead of the computer-controlled machine 500, the control system 502 may also include actuator 504, or in addition to the computer-controlled machine 500 including actuator 504.

[0062] like Figure 5 As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. One or more embodiments include a classifier 514 (e.g., a machine learning algorithm, such as those described above). Figures 1-3 The described ones can be implemented by a control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0063] Non-volatile storage device 516 may include one or more persistent data storage devices, such as hard disk drives, optical drives, tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of persistently storing information. Processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 522. Memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0064] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions residing in non-volatile storage device 516 that implement one or more machine learning algorithms and / or methods of one or more embodiments. Non-volatile storage device 516 may include one or more operating systems and applications. Non-volatile storage device 516 may store data compiled and / or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL (alone or in combination).

[0065] When executed by processor 520, the computer-executable instructions of non-volatile storage device 516 may cause control system 502 to implement one or more of the machine learning algorithms and / or methods disclosed herein. Non-volatile storage device 516 may also include machine learning data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0066] Program code implementing the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of one or more embodiments. Inherently non-transitory computer-readable storage media can include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or via a network to an external computer or external storage device.

[0067] Computer-readable program instructions stored in a computer-readable medium can be used to instruct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of writing that includes instructions to implement the functions, actions, and / or operations specified in the flowchart or figure. In some alternative embodiments, the functions, actions, and / or operations specified in the flowchart and figure can be reordered, processed sequentially, and / or processed simultaneously according to one or more embodiments. Furthermore, any of the flowchart and / or figure may include more or fewer nodes or blocks than those illustrated according to one or more embodiments.

[0068] A process, method, or algorithm can be implemented, either wholly or in part, using appropriate hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0069] Figure 6A schematic diagram is depicted of a control system 502 configured to control a vehicle 600, which may be at least partially autonomous or at least partially autonomous. Typically, the synthetic data described above can be generated by the vehicle 600 for online mapping purposes. The vehicle 600 includes actuators 504 and sensors 506. Sensors 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). One or more of these specific sensors may be integrated into the vehicle 600. In the context of symbol recognition and processing as described herein, sensor 506 is a camera mounted to or integrated into the vehicle 600.

[0070] The classifier 514 of the control system 502 of vehicle 600 can be configured to detect objects near vehicle 600 based on an input signal x. In such an embodiment, the output signal y may include information characterizing the proximity of the object to vehicle 600. An actuator control command 510 can be determined based on this information. The actuator control command 510 can be used to avoid collisions with the detected objects. The classifier 514 may be configured to classify sensor data into categories such as those associated with dynamic or static objects, among other functions.

[0071] In embodiments where vehicle 600 is at least partially automated, actuator 504 may be implemented in the brakes, propulsion system, engine, transmission system, or steering mechanism of vehicle 600. Actuator control command 510 may be determined to control actuator 504 to prevent vehicle 600 from colliding with detected objects. Detected objects may also be classified according to what classifier 514 deems them most likely to be, such as pedestrians or trees. Actuator control command 510 may be determined based on the classification. In scenarios where hostile attacks may occur, the system may be further trained to better detect objects or identify changes in lighting conditions or angles of sensors or cameras on vehicle 600.

[0072] In other embodiments where vehicle 600 is at least partially autonomous, vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be at least partially autonomous lawnmower or at least partially autonomous cleaning robot. In such embodiments, actuator control command 510 may be determined such that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled to prevent the mobile robot from colliding with identified objects.

[0073] In another embodiment, vehicle 600 is at least partially autonomous in the form of a gardening robot. In such an embodiment, vehicle 600 may use an optical sensor, sensor 506, to determine the state of plants in the environment near vehicle 600. Actuator 504 may be a nozzle configured to spray chemicals. Based on the identified species and / or the identified plant state, actuator control command 510 may be determined to cause actuator 504 to spray an appropriate amount of suitable chemical onto the plant.

[0074] Vehicle 600 may be a partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include washing machines, stoves, ovens, microwave ovens, or dishwashers. In such a vehicle 600, sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case of a washing machine, sensor 506 may detect the state of the clothes inside the washing machine. Actuator control commands 510 may be determined based on the detected state of the clothes.

[0075] The system's output can be synthetic data used to update HD maps, rather than for automatic vehicle control. For example, previously generated data can be a portion of an HD map or can generate an HD map, and new data from a second or more sensors can be used (after synthesis) to update the HD map in real time. This could be beneficial for areas that vehicles are traversing and that experience real-time changes, such as construction, parking availability, detours, etc.

[0076] While exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive and not restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing an advantage or superiority over other embodiments or prior art implementations with respect to one or more desired features, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, any embodiment described as being less desirable than other embodiments or prior art in achieving one or more desired features is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for synthesizing sensor data for online mapping pre-training and sensor generalization, the method comprising: Receive a previously generated sensor dataset associated with the scene, wherein the previously generated sensor dataset includes sensor data captured by a first plurality of sensors having associated sensor specifications; The sensor data is classified into (i) static object data associated with static objects in the scene and (ii) dynamic object data associated with dynamic objects in the scene; Volumetric scene reconstruction is performed on the static object data and the dynamic object data to generate a four-dimensional representation of the scene; Generate a new sensor dataset associated with the scene, wherein the new sensor dataset includes new sensor data captured by a second plurality of sensors, the second plurality of sensors having a new sensor specification that is different from the sensor specification associated with the first plurality of sensors; The new sensor dataset is synthesized by aligning the four-dimensional representation of the scene to the new sensor specification; as well as A synthetic dataset is generated based on the synthesis described above.

2. The method according to claim 1, wherein the synthesis comprises rasterization and diffusion.

3. The method of claim 2, wherein the rasterization comprises converting a four-dimensional representation of the scene into a two-dimensional format aligned with the new sensor specification.

4. The method of claim 3, wherein the diffusion comprises applying diffusion techniques to smooth and interpolate rasterized data.

5. The method according to claim 2, wherein the synthesis further comprises: Stream matching, which includes aligning streams of data points within the scene.

6. The method according to claim 1, further comprising generating a four-dimensional simulation of the scene based on the synthetic dataset.

7. The method of claim 1, wherein the previously generated sensor dataset includes spatiotemporal data.

8. The method of claim 1, wherein the first plurality of sensors comprises a camera, LiDAR, or radar sensor.

9. A system for synthesizing sensor data for online mapping pre-training and sensor generalization, the system comprising: A storage device for storing previously generated sensor datasets associated with a scene, wherein the previously generated sensor datasets include sensor data captured by a first plurality of sensors having associated sensor specifications; A second set of sensors located around the vehicle; as well as The processor is communicatively connected to the second plurality of sensors and is programmed to: Receive the previously generated sensor dataset, The sensor data is processed to determine (i) static object data associated with static objects in the scene and (ii) dynamic object data associated with dynamic objects in the scene. Volumetric scene reconstruction is performed on the static object data and the dynamic object data to generate a four-dimensional representation of the scene. Generate a new sensor dataset associated with the scene, wherein the new sensor dataset includes new sensor data captured by a second plurality of sensors, the second plurality of sensors having a new sensor specification different from the sensor specification associated with the first plurality of sensors. The new sensor dataset is synthesized by aligning the four-dimensional representation of the scene to the new sensor specification to generate a synthetic dataset.

10. The system of claim 9, wherein the synthesis of the new sensor dataset includes rasterization and diffusion.

11. The system of claim 10, wherein the rasterization comprises converting a four-dimensional representation of the scene into a two-dimensional format aligned with the new sensor specification.

12. The system of claim 11, wherein the diffusion includes applying diffusion techniques to smooth and interpolate rasterized data.

13. The system of claim 10, wherein the synthesis of the new sensor dataset further comprises: Stream matching, which includes aligning streams of data points within the scene.

14. The system of claim 9, wherein the processor is further programmed to generate a four-dimensional simulation of the scene based on the synthetic dataset.

15. The system of claim 9, wherein the previously generated sensor dataset includes spatiotemporal data.

16. The system of claim 9, wherein the first plurality of sensors includes a camera, LiDAR, or radar sensor.

17. The system of claim 9, wherein the processor is further programmed to generate an updated HD map based on the synthetic data.

18. A non-transitory computer-readable medium comprising instructions that, when executed by a processor, cause the processor to perform the following steps: Receive a previously generated sensor dataset associated with the scene, wherein the previously generated sensor dataset includes sensor data captured by a first plurality of sensors having associated sensor specifications; The sensor data is classified into (i) static object data associated with static objects in the scene and (ii) dynamic object data associated with dynamic objects in the scene; Volumetric scene reconstruction is performed on the static object data and the dynamic object data to generate a four-dimensional representation of the scene; Generate a new sensor dataset associated with the scene, wherein the new sensor dataset includes new sensor data captured by a second plurality of sensors, the second plurality of sensors having a new sensor specification that is different from the sensor specification associated with the first plurality of sensors; The new sensor dataset is synthesized by aligning the four-dimensional representation of the scene to the new sensor specification; as well as A synthetic dataset is generated based on the synthesis described above.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the processor, further cause the processor to: The HD map is updated based on the synthetic dataset.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by the processor, further cause the processor to: The movement of the vehicle is controlled based on the synthetic dataset.