Technology for representing multimodality sensor data

Abstract

This provides a method for training a machine learning (ML) model to generate an integrated particle-based representation of a scene from multimodality sensor data. [Solution] The method generates an integrated particle-based representation 510 of a scene by receiving sensor data 502, 512, and 522 representing the same scene from a set of optical sensors having various sensor modalities, determines the intersection of rays emanating from the sensor positions with the integrated particle-based representation 510, renders scene observations 508, 518, and 528 based on these intersections, determines sensor-specific loss function values ​​by comparing the rendered observations 508, 518, and 528 with the received sensor data 502, 512, and 522, and trains an ML model by optimizing the combination 520 of the sensor-specific loss functions.

Description

Technical Field

[0001] The present invention relates to techniques for training a machine learning (ML) model for generating an integrated particle-based representation of a scene from multi-modality sensor data, and in particular, to techniques for generating a synthetic (e.g., training, testing and / or validation) sensor dataset, including a method, a computing device, a computer program product and a computer-readable storage medium.

Background Art

[0002] Prior Art In the context of autonomous driving and robotics, different sensor modalities are often used, for example, a plurality of cameras and a small number of LiDAR sensors. Given several types of sensor measurements (cameras, LiDAR, radar) from a real-world scene, the task is to reconstruct an integrated representation, whereby multi-modal sensor data can be generated from the same scene but at new sensor positions and orientations and / or using modified sensor parameters.

[0003] Over the past few years, attempts have been made to solve variations of the above task by approaches from the NeRF family [1]. More recently, Gaussian Splatting [2] has been found to be a faster and more explicit alternative to NeRF. In the vanilla Gaussian Splatting pipeline, a scene is represented as a family of 3D Gaussians with several parameters. Here, taking into account the known intrinsic and extrinsic parameters of a camera, 2D rendering of the scene can be formed by projecting (splatting) Gaussians onto the image plane of the camera. The scene representation is optimized by comparing it with the rendered ground truth image of the scene and adjusting the Gaussian parameters accordingly.

[0004] However, training the vanilla Gaussian splatting pipeline is only possible for cameras and does not work for other sensors such as LiDAR or radar.

[0005] In another attempt, [3] LiDAR point clouds are used to initialize a Gaussian method and for depth monitoring during training, however other information provided by the LiDAR sensor is ignored. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] [1]B. Mildenhall, PP Srinivasan, M. Tancik, JT Barron, R. Ramamoorthi, R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis”, ECCV, 2020 [Non-Patent Document 2] [2] B. Kerbl, G. Kopanas, T. Leimkuehler, G. Drettakis, “3d gaussian splatting for real-time radiance field rendering,” ACM Transactions on Graphics, vol. 42, July 2023. [Non-Patent Document 3] [3] N. Moenne-Loccoz, A. Mirzaei, O. Perel, R. de Lu-tio, JM Esturo, G. State, S. Fidler, N. Sharp, Z. Gojcic, “3d gaussian ray tracing: Fast tracing of particle scenes”, 2024 [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, an object of the present invention is to provide a solution for improving the accuracy and / or validity of scene observation from sensors having multiple modalities. Alternatively or additionally, an object is also to augment synthetic (e.g., training, testing and / or validity confirmation) data for multimodality perceptual tasks. [Means for solving the problem]

[0008] Disclosure of the invention The above objectives are achieved by the methods described in each of the attached independent claims: a method for training a machine learning (ML) model to generate an integrated particle-based representation of a scene from multimodality sensor data; a method for generating a synthetic sensor dataset; two computing devices; a computer program (and / or computer program product); and a computer-readable storage medium. Advantageous aspects, features, and embodiments are described in each dependent claim and in the following description along with their advantages.

[0009] In the following, the solutions according to the present invention will be described in relation to the method and the computing device described in the claims. Features, advantages or alternative embodiments described herein may correspond to subjects described in other claims (e.g., computer programs or computer program products), and conversely, subjects described in other claims may correspond to features, advantages or alternative embodiments described herein. In other words, claims relating to computing devices may be improved by features described or claimed in relation to methods. In this case, the functional features of the method are each embodied by the structural units of the computing device, and conversely, the functional features of the structural units of the computing device are each embodied by the method.

[0010] According to a first embodiment of the method, a (e.g., computer-implemented) method is provided for generating an integrated particle-based representation of a scene from multimodality sensor data. The method can train a machine learning (ML) model for generating the integrated particle-based representation. The method includes receiving sensor data acquired by a set of at least two optical sensors, each having a different sensor modality. The received sensor data represents the same scene. The method further includes generating an integrated particle-based representation of the scene covered by the set of at least two optical sensors. The generation is performed based on the received sensor data. The method further includes determining the intersection of rays emanating from a given optical sensor in the set of at least two optical sensors with the generated integrated particle-based representation. The method further includes rendering an observation of the scene (e.g., an image and / or measurements) (particularly by a renderer associated with the sensor modality of the given optical sensor). The rendering is performed based on the intersection of the determined rays with the integrated particle-based representation. The method further includes determining at least one value of a sensor-specific loss function. Determining at least one value of the sensor-specific loss function involves comparing observations of the rendered scene with observations contained in (and / or encoded) in received sensor data acquired by a given (and / or corresponding) optical sensor. The unified particle-based representation of the scene is optimizable (and / or can be trained on the unified particle-based representation of the scene) by iterating through the steps of determining intersections, rendering observations, determining at least one value of the sensor-specific loss function for each optical sensor in a set of at least two optical sensors, and optimizing the combination of at least one value of the sensor-specific loss function determined for each optical sensor.

[0011] This technology allows ML models to be trained to generate multimodal composite (e.g., training, testing, and / or validation) data for downstream perceptual tasks related to a scene. These downstream perceptual tasks can enable autonomous vehicle driving (e.g., for traffic situations) and / or enable the use of robotics in factory and / or home automation environments (e.g., for indoor or manufacturing scenes). The ML model is trained to generate an integrated particle-based representation (also simply called a particle representation) of the scene using multiple (at least two sets) optical sensors, each with a different modality. The ML model is further trained to reconstruct observations from the integrated particle-based representation for each sensor modality.

[0012] In this specification, observation, in a broad sense, refers to observations based on camera images (e.g., two-dimensional, i.e., 2D), measurements of LiDAR sensor data (e.g., three-dimensional, i.e., 3D), measurements of radar sensor data (e.g., 3D), and / or any combination of sensor data acquired by optical sensors (also referred to as image sensors and / or measurement sensors). Alternatively or additionally, a scene observation may include constructed (e.g., reconstructed) sensor data, where construction (e.g., reconstruction) includes deriving sensor data from an integrated particle-based representation of the scene. Further alternative or additionally, a scene observation based on the intersection of rays emanating from a given optical sensor location may be digital data of the same type as a portion of sensor data acquired using a given optical sensor.

[0013] Rendering can be performed by a sensor modality-specific renderer. In particular, 3D observations (e.g., measurements of LiDAR sensor data) do not need to be projected onto 2D (e.g., explicitly). For example, a camera renderer can be configured to render 2D images. Alternatively or additionally, a LiDAR renderer can be configured to reconstruct LiDAR sensor data (e.g., synthetic and / or 3D) (particularly from an integrated particle-based representation). Alternatively or additionally, rendering of LiDAR data (particularly by a LiDAR renderer) may include sampling rays according to a LiDAR sensor model and then calculating depth and / or intensity for each ray. Further alternative or additionally, a radar renderer may be used to reconstruct radar sensor data (e.g., synthetic and / or 3D) (particularly from an integrated particle-based representation).

[0014] The same scene (also simply referred to as a scene) includes a scene (particularly in the real world), in which sensor data is acquired by optical sensors that can be positioned at different locations within that scene. For example, an autonomous vehicle may have multiple sensors positioned at the front, rear, sides, top, and / or bottom of the vehicle.

[0015] Sensor data can be acquired simultaneously, or at least partially, over the same period of time, in order to represent the scene.

[0016] The scene may include traffic-related scenes in the vicinity of an autonomous vehicle (also referred to as the vehicle itself). The set of optical sensors can be localized at least partially within the vehicle itself. Alternatively or additionally, the scene may include a factory environment in the vicinity of a robot or an environment in the vicinity of a home automation robot. In each of these cases, at least a portion of the set of optical sensors can be localized within the robot.

[0017] By training to generate an integrated particle-based representation and subsequently reconstructing observations for each sensor modality, new multimodal synthetic (e.g., training, testing, and / or validation) data for downstream perceptual tasks can be generated for new sensor configurations (e.g., position and / or orientation) and / or configurations of new sets of optical sensors (e.g., changes in the number of optical sensors per sensor modality). Alternatively or additionally, the new multimodal synthetic data for downstream perceptual tasks may include sensor data (also referred to as sensor datasets) with modified sensor-specific parameters and / or parameters that are modified sensor-independent (also referred to as sensor-unknowable).

[0018] Alternatively or additionally, training techniques can be used to generate a "world model" for closed-loop simulations.

[0019] In closed-loop simulations, a system (including, for example, a computing device in the form of an ML model and / or apparatus) can be modeled with control and environmental feedback (and / or scene feedback). For example, in the case of autonomous driving, this could mean that a car (and / or agent) receives observations (and / or sensor data) from the virtual environment and predicts what to do next. An action is then performed, for example, the car moves somewhere. The environment (and / or scene) also changes, for example, other vehicles also move somewhere. The agent then receives new observations (and / or sensor data), and the process is repeated. The next state of the environment (and / or scene) always depends on the agent's previous actions.

[0020] The multimodality of the optical sensor set allows observations from various sensor modalities (also called sensor types) to mutually enhance each other, resulting in a richer particle-based representation. Alternatively or additionally, this can improve the validity of the integrated particle-based representation, and consequently, the validity of the rendered observations.

[0021] Rendered observations may include a time series of observations (also referred to as a sequence). Said time series may correspond to a time series included in each sensor data set (for example, sensor data from one optical sensor in the set).

[0022] The technique may be computationally efficient by virtue of the particle-based representation and by determining (also referred to as calculating) intersections between rays and the particle-based representation.

[0023] The particle-based representation may include a plurality of (for example, 3D) functions. Each (for example, 3D) function may be strongly localized at (and / or around) a position (including a mean) having a predetermined extent (such a width as covariance, variance and / or half-maximum width). The predetermined extent may be isotropic or anisotropic for a first type of (for example, 3D) function, such as a Gaussian. For example, 100,000 3D Gaussians supplemented with a plurality of parameters can integrally represent a scene observed by a set of optical sensors.

[0024] The integrated particle-based representation may include sensor-non-specific (also referred to as sensor-independent) parameters, such as means and covariances, as well as sensor-specific parameters, such as extrinsic parameters (for example, of a camera) and / or camera extrinsic parameters, colors and / or opacities, and / or LiDAR intensities.

[0025] The integrated particle-based representation may be based on a plurality of copies of a particle kernel function (also referred to as a primitive), wherein said copies include, as parameters, a per-particle position and at least one per-particle covariance value.

[0026] The particle kernel function may include, for example, Gaussians, such as three-dimensional (3D) Gaussians, generalized Gaussians (GG2), kernelized surfaces (SGG2), or cosine wave modulation (CSGG2).

[0027] Performing this method on sensor data acquired using only a camera, using a 3D Gaussian as the particle kernel function, can be described as a generalization of Gaussian splatting. Alternatively or additionally, techniques using sensor data acquired by optical sensors with different sensor modalities can generalize Gaussian splatting by directly determining the interaction of rays using a Gaussian representation.

[0028] The parameters can be optimized simultaneously based on two or more (e.g., sensor-specific) loss functions, depending, for example, the number of tensors (e.g., the covariance matrix), for rendering purposes. The loss functions (also simply called losses) can be summed, for example, L = L_im + L_lid, where L_im is the image-based loss and L_lid is the LiDAR-based loss, and parameter optimization can be performed on the total loss L. In some embodiments, the intrinsic and / or extrinsic parameters of the sensor (e.g., camera) may be fixed and / or assumed to be known. The intrinsic and / or extrinsic parameters may describe the 3D position (and / or orientation) and / or fixed characteristics of each sensor.

[0029] Sensor-independent parameters (e.g., mean and / or scale) and sensor-specific parameters (e.g., opacity and / or color) may have learnable (and / or optimizeable) values. Each particle (and / or 3D Gaussian) can be associated with a set of sensor-independent parameters and a set of sensor-specific parameters. Determining the correct values ​​of these parameters may, precisely, be the essence of the technique for training an ML model to generate an integrated particle-based representation. Generating one or more rays may be unknowable in terms of the forward and / or backward directions. Thus, an emission from the position of an optical sensor may be identical to an incidence at the position of the optical sensor for generating one or more rays. Alternatively or additionally, each ray can be characterized by its direction and at least one point on the ray, e.g., the origin or endpoint of the ray.

[0030] Generating one or more rays may involve ray sampling. Sampling can be performed along three (or four) axes and / or may take direction angles and / or lengths into consideration.

[0031] Each all-sensor observation may include numerous rays (e.g., image pixel rays and / or LiDAR rays). However, it is not necessary for each of these rays to be used in every training iteration.

[0032] Alternatively, a subset of rays used in the optimization step (for example, the current one) can be randomly selected. For example, a random fixed-size subset of pixels from an image can be selected.

[0033] Observations (and / or images and / or measurements) are performed at different time steps, and a time component may be introduced into the sampling. In practice, for example, images (and / or LiDAR scans and / or radar point clouds) can be selected from a specific point in time, and radiation lines can then be sampled from there.

[0034] Each ray may correspond to a single point (e.g., a single point) in the point cloud associated with the optical sensor.

[0035] Rendering of scene observations based on the intersection of radiation lines and integrated particle-based representations can be performed based on a set of rendering parameters.

[0036] Rendering parameters may include intrinsic parameters, extrinsic parameters, sensor-independent parameters, and / or sensor-specific parameters. Alternatively or additionally, rendering parameters may correspond to sensor parameters and / or parameters of particle-based representations (e.g., they may be substantially equivalent to or reciprocal of sensor parameters and / or parameters of particle-based representations).

[0037] Determining the rendering parameter values ​​for the intersection of rays and the unified particle-based representation may involve adding the rendering parameter values ​​for each contributing particle (e.g., up to the maximum value and / or threshold of the parameter). For example, the pixel color and / or opacity for rendering a camera image may be the sum over the colors and / or opacities of the contributing particles. Contributing particles may be those whose position (also called mean) is smaller than the covariance value at which they are farther from the pixel. Optimization of combinations of at least one value of the determined loss function (e.g., at least one value of the sum of the determined sensor-specific loss functions) may be gradient-based.

[0038] Rendering may include differentiable volume rendering, which enables optimization based on the gradient of at least one combination of values ​​of a determined (e.g., specific) loss function.

[0039] The determined combination of at least one value of the loss function (e.g., a specific one) may include combinations of multiple values ​​of the sensor-specific loss function (also referred to as the sensor modality-specific loss function), such as a weighted sum.

[0040] Sensor-specific loss functions can be selected independently for each sensor modality. By combining sensor-specific loss functions, the generation of unified particle-based representations can be improved for all sensor modalities.

[0041] At least two optical sensors, each having a different sensor modality, may include at least two of the following: cameras, LiDAR sensors, and radar sensors. For example, a set of sensors may include multiple cameras and multiple LiDAR sensors.

[0042] By combining different sensor modalities, scene observation (also known as imaging and / or measurement) can be improved. For example, sensor data from LiDAR sensors and / or radar sensors in challenging visibility conditions (e.g., foggy weather or nighttime) can significantly improve knowledge about traffic-related situations.

[0043] Alternatively or additionally, different sensor modalities may be suitable for different distance ranges. For example, a camera can capture scenes from 1m to 200m away from itself. Alternatively or additionally, a LiDAR sensor can capture scenes from 10cm to 500m away from itself, and possibly up to 2000m. Further alternative or additionally, a radar sensor can capture scenes from 10cm to 6m away from itself, and possibly up to 150m. It should be noted that the ranges given exemplarily may depend heavily on the specific optical sensor and / or optical sensor manufacturer, and are not limiting to the technology in the sense that it can be deployed beyond the ranges given exemplarily.

[0044] In one example, the set of optical sensors may include six cameras and two LiDAR sensors, and each sensor dataset for each optical sensor is available for a period of 5 seconds, where this period is the same for all optical sensors. The method may include the step of receiving a set of sensor-specific parameters associated with a given optical sensor.

[0045] Rendering can be performed based on a set of sensor-specific parameters (also simply called sensor parameters). This enables faithful reproduction from an integrated particle-based representation of sensor data acquired by a given optical sensor (and / or scene).

[0046] Sensor parameters for a camera may include extrinsic parameters (also simply called extrinsic parameters) and / or intrinsic parameters (also simply called intrinsic parameters) of the camera. Extrinsic parameters can represent the camera's position in a 3D scene. Alternatively or additionally, intrinsic parameters may represent the camera's optical center and / or focal length. The world point can be transformed into camera coordinates using the extrinsic parameters.

[0047] Sensor calibration can refer to the process of measuring the correspondence between the sensor's output and the data actually measured by the sensor.

[0048] The sensor parameters of a LiDAR sensor may include laser wavelength, detection range, field of view (FOV), ranging accuracy, angular resolution, point velocity, beam count, safety level, output parameters, IP rating, power, supply voltage, laser emission mode (mechanical / solid state), and / or lifetime.

[0049] Sensor parameters for radar sensors may include frequency band and / or modulation.

[0050] Observation rendering can be performed based on a set of rendering parameter values ​​for a given optical sensor. Optionally, at least a subset of rendering parameters may correspond to sensor-specific parameters.

[0051] Sensor-specific parameters (and / or extrinsic and / or intrinsic parameters) may be included in the set of sensor-specific parameters received in relation to a given optical sensor. This ensures that the renderer uses the most appropriate set of parameters to faithfully reproduce the sensor data (and / or scene) based on the unified particle-based representation.

[0052] Rendering parameters may include opacity and / or color for a camera as an optical sensor. Alternatively or additionally, rendering parameters may include intensity for a LiDAR sensor as an optical sensor. Alternatively or additionally, rendering parameters may include radar cross-section (RCS) for a radar sensor as an optical sensor.

[0053] Rendering parameters may include at least sensor-specific parameters. For example, a renderer can be used to render observations of a scene for optical sensors having a selected sensor modality and for a specific selected optical sensor among them. According to one embodiment, a scene can be represented by 3D Gaussians. Gaussian splatting may be a 2D projection of each 3D Gaussian that can be used for rendering according to conventional Gaussian splatting methods. Gaussian splatting (as an example of particles) may be small soft blobs that capture identification parameters for camera observation, including color, position and / or opacity that are blended with each other.

[0054] Techniques for generating an integrated particle-based representation of a scene from multimodality sensor data (e.g., for training an ML model), and / or for generating a synthetic sensor dataset, may utilize Gaussian ray tracing, in which the rays are incident on 3D (in particular, instead of projecting the Gaussian onto the image plane), and intersections between the rays and the Gaussian are found in 3D.

[0055] In one embodiment, the renderer may be a Gaussian splatting renderer. The Gaussian splatting renderer may be a renderer that creates a 3D scene from multiple 2D observations, each captured by a camera, using Gaussian splatting. Gaussian splatting may include using the position of the optical sensor and / or the field of view of the optical sensor as sensor-specific parameters. Gaussian splatting may further include utilizing a Gaussian representation.

[0056] Determining the intersection of a ray with an integrated particle-based representation may include generating one or more rays emanating from the position of a given optical sensor. Alternatively or additionally, determining the intersection of a ray with an integrated particle-based representation may include determining the intersection of each generated ray with the generated integrated particle-based representation. Further alternative or additionally, determining the intersection of a ray with an integrated particle-based representation may include determining a set of rendering parameter values ​​for each determined intersection. The determination here may be based on the integrated particle-based representation generated at the intersection. The set of rendering parameter values ​​may depend on the position of the given optical sensor and / or the sensor modality of the given optical sensor.

[0057] To enable rendering, rendering parameter values ​​can be determined per pixel (e.g., in 2D camera images) and / or per voxel (e.g., in 3D LiDAR or 3D radar measurements). Rendering parameter values ​​can be determined as contributions from particles (and / or Gaussians) intersecting rays (e.g., along the line of sight) from a given optical sensor. A combination of at least one sensor-specific loss function values ​​for training an ML model may include a (e.g., weighted) sum of these values. By combining (e.g., summing) the sensor-specific loss function values, the generation of a unified particle-based representation can be synergistically optimized for each sensor modality included in a set of optical sensors.

[0058] The sensor-specific loss function may include contributions from the independently selected sensor modality-specific loss function.

[0059] Sensor-specific loss functions may include L1 loss function, L2 loss function, structural similarity index (SSIM), and / or mean squared error (MSE). For example, camera-specific loss functions may be L1 loss, L2 loss, SSIM, or MSE. Alternatively or additionally, for LiDAR sensors, L1 loss or L2 loss is selectable. Further alternative or additionally, L1 loss is selectable for radar sensors. In some embodiments, generating an integrated particle-based representation may include determining a particle-based representation of a scene for a portion of the received sensor data acquired by one sensor modality, and merging the determined particle-based representations for each sensor modality into an integrated particle-based representation of all sensor modalities included in the set of optical sensors.

[0060] A unified particle-based representation may include the particle's position and at least one covariance value. The particle's position (also called the mean) allows for parameterization of the position in the scene where the particle contributes to the maximum value.

[0061] Alternatively or additionally, at least one covariance value may include a covariance matrix with independent values ​​for each spatial direction. This allows for the parameterization of particles with anisotropic shapes.

[0062] The mean and at least one covariance value can also be referred to as sensor-independent parameters.

[0063] The unified particle-based representation may include a three-dimensional (3D) Gaussian representation of the scene.

[0064] A 3D Gaussian representation can be a particularly simple, and therefore computationally efficient, representation of a scene. For example, a 3D Gaussian representation can utilize Gaussians that represent each isotropic particle. Sensor data may include time series for each sensor dataset. The time series for each sensor dataset can be at least partially captured over the same period for each optical sensor in a set of optical sensors.

[0065] Each different optical sensor will start and end acquiring observations at slightly different points in time. The integrated particle-based representation can only be generated for the period during which all optical sensors acquired sensor datasets. This allows the temporal evolution of the scene to be represented by a time-dependent particle-based representation.

[0066] In some embodiments, for example, when three different sensor modalities are used, the integrated particle-based representation can only be constructed for subsets (e.g., two) each having a different sensor modality over a certain period. This is especially true when one sensor modality does not acquire sensor datasets during the entire acquisition time of the other sensor modalities.

[0067] The method can be implemented using one or more graphics processing units (GPUs) to efficiently and quickly process sensor datasets from optical sensors.

[0068] According to a second embodiment of the method, a (e.g., computer-implemented) method for generating a composite sensor dataset is provided. The method includes the step of receiving instructions for a (e.g., virtual) set of optical sensors. Each (e.g., virtual) optical sensor in the set is associated with one sensor modality. The set here includes at least two optical sensors, each having a different sensor modality. The method further includes the step of generating (e.g., virtual) unified particle-based representations for all sensor modalities included in the set of optical sensors. The generation here is based on the received instructions for the set of optical sensors. The unified particle-based representations represent the same scene. The method further includes the step of determining the intersection of rays emanating from the location of the optical sensors with the generated unified particle-based representations for each optical sensor in the set of optical sensors. The method further includes the step of generating a composite sensor dataset of the scene in relation to each optical sensor in the set. The generation here is based on the intersection of the determined rays with the unified particle-based representations.

[0069] The generation of a synthetic sensor dataset can be performed by an ML model trained according to the method of the first method embodiment.

[0070] Synthetic sensor datasets can be generated to train downstream perceptual tasks, configured to accept sensor datasets of optical sensors used in training (e.g., with respect to sensor modality and sensor parameters) as input.

[0071] The received instruction for at least one sensor parameter may include sensor positioning. Sensor positioning may include the position and / or orientation of an optical sensor. Sensor positioning can, for example, determine the position of the vehicle (or robot) within a scene.

[0072] Arbitrary instructions can be received through a user interface (UI), such as a graphical user interface (GUI).

[0073] The technology can enable the augmentation of training datasets (and / or test datasets and / or validation datasets) that occur infrequently. Alternatively, or in addition to the above, the technology enables the construction of dynamic scenes. For example, traffic participants can be repositioned (e.g., relative to real-world sensor datasets). In some embodiments, the vehicle itself can be repositioned. In other embodiments that can be combined with the first type of embodiment, other traffic participants (e.g., other vehicles) can be repositioned. Construction of dynamic scenes may include object-centered modeling.

[0074] The technique can use interpolation between different sensor locations, between different object locations (and / or between object orientations), between different values ​​of rendering parameters, and / or between different sensor parameters. Alternatively or additionally, the technique does not need to be configured to perform extrapolation toward untrained regions of parameter space, location, and / or orientation.

[0075] The method may include receiving instructions for at least one sensor parameter (e.g., an extrinsic parameter and / or an intrinsic parameter and / or a sensor-specific parameter) for each optical sensor contained within the received instructions. Alternatively or additionally, the method may also include receiving instructions for the object positions and / or at least one characteristic of the scene.

[0076] To generate an integrated particle-based representation, for example, user input specifying a scene can be received with respect to changes in a predefined scene. For example, at least one characteristic of the scene may include path lighting out or floor plan of a factory environment or home automation environment. At least one characteristic of the scene may, alternatively or additionally, relate to lighting and / or weather conditions.

[0077] The indication of an object's position may include the dynamic position of the object and / or the position of the object at an instantaneous point in time.

[0078] The indication of an object's position may (at least implicitly) include the object's orientation. For example, unless explicitly indicated otherwise, a vehicle can be oriented toward a road lane in the direction of normal traffic flow.

[0079] In some embodiments, the Gaussian representation can be initialized from a LiDAR point cloud.

[0080] Determining the intersection of a ray with an integrated particle-based representation may include the step of generating one or more rays emanating from the location of an optical sensor for each optical sensor in a set of optical sensors. Determining the intersection of a ray with an integrated particle-based representation may, alternatively or additionally, include the step of determining the intersection of each generated ray with the generated integrated particle-based representation. Determining the intersection of a ray with an integrated particle-based representation may, alternatively or additionally, include the step of determining at least one value of a rendering parameter for each optical sensor for each determined intersection of a ray with the generated integrated particle-based representation. The value of at least one rendering parameter can be determined based on the received instructions for the set of optical sensors and based on the received instructions for at least one sensor parameter of an optional optical sensor.

[0081] This technology focuses on generating synthetic (e.g., training data, test data, and / or validation) sets from a multi-sensor modality configuration. In some specific cases, a trained ML model can be used to generate synthetic training datasets for only one sensor modality, for example, only LiDAR sensors or only radar sensors.

[0082] A synthetic dataset (e.g., training, test, and / or validation) generated by the method according to the second method embodiment can be used to train a downstream perceptual task model for performing a perceptual task based on a set of optical sensors having at least two different sensor modalities. Optionally, the perceptual task may include object detection and / or classification. Alternatively or additionally, the perceptual task may include occupation estimation, scene segmentation, object tracking, object velocity estimation, depth estimation, and / or distance estimation. Furthermore, alternatively or additionally, the perceptual task may include time prediction.

[0083] In the first embodiment of the apparatus, a computing apparatus is provided for training an ML model for generating an integrated particle-based representation of a scene from multimodality sensor data. The computing apparatus includes a sensor data receiving interface, which is configured to receive sensor data acquired by at least two sets of optical sensors, each having a different sensor modality. The received sensor data represents the same scene. The computing apparatus further includes a representation generation unit configured to generate an integrated particle-based representation of a scene covered by at least two sets of optical sensors. The generation of the integrated particle-based representation is performed based on the received sensor data. The computing apparatus further includes a determination unit configured to determine the intersection of rays emanating from a given optical sensor location (particularly within the set of at least two optical sensors) with the generated integrated particle-based representation. The computing apparatus further includes at least one renderer (particularly a per-optical sensor modality renderer and / or rendering interface, which may also be expressed as a sensor modality-specific renderer) configured to enable rendering of observations of the scene (particularly per optical sensor modality). Rendering is performed based on the intersection of the determined rays with the integrated particle-based representation. The computing device further includes a loss function (also referred to as an optimizer) configured to determine at least one value of a sensor-specific loss function. Determining at least one value of a sensor-specific loss function involves comparing a rendered observation of the scene with an observation contained in (and / or encoded) sensor data acquired and received by a given (and / or corresponding) optical sensor.The unified particle-based representation of the scene can be optimized (and / or trainable for an ML model to generate the unified particle-based representation of the scene) by iterating through the steps of determining intersections, rendering observations, determining at least one value of a sensor-specific loss function for each optical sensor in a set of at least two optical sensors, and optimizing a combination of the determined at least one value of the sensor-specific loss function for each optical sensor, and further optimizing the at least one value of the sensor-specific loss function determined for each optical sensor.

[0084] Optionally, the computing device according to the first configuration may further include a parameter receiving interface, a ray generation subunit, an intersection determination subunit, and / or a value determination subunit.

[0085] A computing device of the first apparatus embodiment can be configured to perform any one of the steps described in relation to the first method embodiment, or to include any one of the features.

[0086] In a second embodiment of the apparatus, a computing device is provided for generating a composite sensor dataset. The computing device includes a sensor instruction receiving interface configured to receive instructions for a set of optical sensors. Each optical sensor in the set is associated with one sensor modality. The set here includes at least two optical sensors, each having a different sensor modality. The computing device further includes a representation generation unit, which is configured to generate a unified particle-based representation for all sensor modalities included in the set of optical sensors. Generating the unified particle-based representation is based on the received instructions for the set of optical sensors. The unified particle-based representation represents the same scene. The computing device further includes a determination unit configured to determine the intersection of rays emanating from the location of the optical sensors with the generated unified particle-based representation for each optical sensor in the set of optical sensors. The computing device further includes a dataset generation unit, which is configured to generate a composite sensor dataset for the scene associated with each optical sensor in the set. The generation of the composite dataset is based on the intersection of the determined rays with the unified particle-based representation. Optionally, the computing device according to the second configuration may further include a parameter receiving interface, a scene characteristic receiving interface, a ray generation subunit, an intersection determination subunit, and / or a value determination subunit.

[0087] A computing device of the second apparatus embodiment can be configured to perform any one of the steps described in relation to the second method embodiment, or to include any one of the features described therein.

[0088] In another embodiment, a computer program product is provided which includes a programming element that guides a computing device to perform the steps of a method for generating an integrated particle-based representation of a scene from multimodality sensor data according to a first method embodiment (e.g., a method for training an ML model) and / or a method for generating a synthetic sensor dataset according to a second method embodiment when the programming element is loaded into the memory of the computing device. In yet another embodiment, a computer-readable medium is provided which stores a programming element that is readable and executable by a computing device in order to perform the steps of a method for generating an integrated particle-based representation of a scene from multimodality sensor data according to a first method embodiment (e.g., a method for training an ML model) and / or a method for generating a synthetic sensor dataset according to a second method embodiment when the programming element is executed by the computing device. [Brief explanation of the drawing]

[0089] [Figure 1] This is an illustrative flowchart demonstrating a method for generating an integrated particle-based representation of a scene from multimodality sensor data. [Figure 2] This is an illustrative flowchart showing a method for generating a synthetic sensor dataset. [Figure 3] This figure schematically illustrates an exemplary architecture of a first computing device for generating an integrated particle-based representation of a scene from multimodality sensor data. The first computing device can be configured to perform the method shown in Figure 1. [Figure 4] This figure schematically illustrates an exemplary architecture of a second computing device for generating a synthetic sensor dataset. The second computing device can be configured to perform the method shown in Figure 2. [Figure 5] This figure schematically illustrates one embodiment of generating a Gaussian representation based on LiDAR, camera, and radar sensor data, which can be performed according to the method shown in Figure 1. [Figure 6] This diagram schematically illustrates a traffic scene with two different positions for the vehicle. The first vehicle corresponds to actual sensor data, while the second vehicle may be synthetically generated based on a modified integrated particle-based representation, where the sensor positioning can be modified according to the method shown in Figure 2. [Figure 7A] For example, Figure 6 shows a single camera image acquired by the movement of a sensor positioned in relation to a traffic scene. [Figure 7B] For example, Figure 6 shows a single camera image acquired by the movement of a sensor positioned in relation to a traffic scene. [Figure 7C] For example, Figure 6 shows a single camera image acquired by the movement of a sensor positioned in relation to a traffic scene. [Figure 8A] This diagram shows variations of the scene in Figure 7A, including the vehicle's hood in the camera image, with each point corresponding to a single LiDAR beam. [Figure 8B] This is an illustrative screen capture of a 3D visualization, where each point corresponds to a single LiDAR radiation line. [Figure 9A] This figure shows a traffic scene with objects in different positions relative to a set of statically positioned optical sensors. [Figure 9B] This figure shows a traffic scene with objects in different positions relative to a set of statically positioned optical sensors.

[0090] No reference numeral in the claims should be construed as limiting the scope of the claims. [Modes for carrying out the invention]

[0091] Detailed explanation Figure 1 shows an illustrative flowchart of Method 100 (e.g., computer-implemented) for generating an integrated particle-based representation of a scene from multimodality sensor data. Method 100 can train a machine learning (ML) model to generate the integrated particle-based representation.

[0092] Method 100 includes step S102 of receiving sensor data acquired by a set of at least two optical sensors having various sensor modalities. The sensor data acquired by at least two different optical sensors and received in S102 represent the same scene. Method 100 further includes step S104 of generating an integrated particle-based representation of the scene covered by the set of at least two optical sensors. Step S104 of generating the integrated particle-based representation is performed based on the sensor data received in S102. Method 100 further includes step S105 of determining the intersection of rays emanating from a given optical sensor location (particularly within the set of at least two optical sensors) with the integrated particle-based representation generated in S104. Method 100 further includes step S112 of rendering an observation of the scene based on the intersection of the rays determined in S105 with the integrated particle-based representation. The rendering in S112 can be performed using a sensor modality-specific renderer associated with a given optical sensor (e.g., a LiDAR renderer for a LiDAR sensor as a given optical sensor). Method 100 further includes step S114 of determining at least one value of a sensor-specific loss function. Determining at least one value of a sensor-specific loss function S114 includes comparing the observation of the scene rendered in S112 with the observation contained in and / or encoded in sensor data (e.g., a portion thereof) acquired and received in S102 by a given optical sensor. An ML model can be trained to generate an integrated particle-based representation of the scene in S104 (and / or optimize the integrated particle-based representation of the scene) by iterating through the steps of determining the crossover in S105, rendering the observation in S112, and determining at least one value of a sensor-specific loss function for all optical sensors in a set of at least two optical sensors in S114. The combination of at least one value of the sensor-specific loss function for each optical sensor determined in S114 is optimized.

[0093] Method 100 may further include step S103 of receiving a set of sensor-specific parameters associated with a given optical sensor.

[0094] Step S105, which determines the intersection of a ray and an integrated particle-based representation, may include a substep S106 that generates one or more rays emanating from a predetermined optical sensor location. Step S105, which determines the intersection of a ray and an integrated particle-based representation, may further include a substep S108 that determines the intersection of each ray generated in S106 with the integrated particle-based representation generated in S104. Step S105, which determines the intersection of a ray and an integrated particle-based representation, may further include a substep S110 that determines a set of rendering parameter values ​​for each intersection determined in S108. The set of rendering parameter values ​​can be determined in S110 based on the integrated particle-based representation generated in S104 at the intersection. Optionally, the set of rendering parameter values ​​may depend on the location of a predetermined optical sensor and / or the sensor modality of a predetermined optical sensor.

[0095] Figure 2 shows an exemplary flowchart of Method 200 (e.g., computer-implemented) for generating a composite sensor dataset. Method 200 includes step S202 of receiving instructions for a set of optical sensors. Each optical sensor in the set is associated with one sensor modality. Here, the set includes at least two optical sensors, each having a different sensor modality. Method 200 further includes step S208 of generating a unified particle-based representation for all sensor modalities included in the set of optical sensors. Step S208 of generating the unified particle-based representation can be performed based on the instructions for the set of optical sensors received in S202. The unified particle-based representation represents the same scene. Method 200 further includes step S209 of determining, for each optical sensor in the set of optical sensors, the intersection of the rays emanating from the optical sensor's position and the unified particle-based representation generated in S208. Method 200 further includes step S214 of generating a training sensor dataset of the scene for each optical sensor in the set. Step S214, which generates the training sensor dataset, is performed based on the intersection of each ray determined in S209 with the integrated particle-based representation.

[0096] Method 200 may further include step S204 of receiving instructions for at least one sensor parameter for each optical sensor included in the instructions of step S202 received. Alternatively or additionally, Method 200 may further include step S206 of receiving instructions for at least one characteristic of an object position and / or scene.

[0097] Step S209, which determines the intersection of a ray with the integrated particle-based representation, may include a substep S210 that generates one or more rays emanating from the position of each optical sensor in the set of optical sensors. Step S209, which determines the intersection of a ray with the integrated particle-based representation, may further include a substep S212 that determines the intersection of each ray generated in S210 with the integrated particle-based representation generated in S208. Step S209, which determines the intersection of a ray with the integrated particle-based representation, may further include a substep S213 that determines at least one value of a rendering parameter for each optical sensor for each intersection of a ray with the integrated particle-based representation generated in S208 determined in S212. The at least one value of the rendering parameter can be determined based on the instruction for the set of optical sensors received in S202, and optionally based on the instruction for at least one sensor parameter of the optical sensor received in S204.

[0098] Figure 3 schematically shows the architecture of a computing device 300 that generates an integrated particle-based representation of a scene from multimodality sensor data (for example, to train an ML model). The computing device 300 includes a sensor data receiving interface 302, which is configured to receive sensor data acquired by at least two sets of optical sensors, each having a different sensor modality. The sensor data acquired by at least two sets of optical sensors having various sensor modalities represents the same scene. The computing device 300 further includes a representation generation unit 304, which is configured to generate an integrated particle-based representation of the scene covered by the at least two sets of optical sensors based on the received sensor data. The computing device 300 further includes a decision unit 305, which is configured to determine the intersection of rays emanating from a given optical sensor location (particularly within the set of at least two optical sensors) with the generated integrated particle-based representation. The computing device 300 further includes a rendering interface 312 (and / or a set of renderers including at least one rendering interface for each optical sensor modality, also referred to as sensor modality-specific renderers) configured to enable (and / or provide) rendering of scene observations based on the intersection of determined rays and integrated particle-based representations. The computing device 300 further includes a loss function 314 configured to determine at least one value of a sensor-specific loss function. Determining at least one value of a sensor-specific loss function involves comparing the rendered scene observation with an observation contained in and / or encoded in sensor data (e.g., a portion thereof) acquired and received by a given optical sensor.The ML model can be trained to generate a unified particle-based representation of the scene (and / or the unified particle-based representation of the scene can be optimized) by iterating through the steps of determining intersections, rendering observations, and determining at least one value of a sensor-specific loss function for each optical sensor in a set of at least two optical sensors, and further optimizing the combination of at least one value of the sensor-specific loss function determined for each optical sensor.

[0099] Optionally, the computing device 300 may further include a parameter receiving interface 303 configured to receive a set of sensor-specific parameters associated with a given optical sensor.

[0100] The decision unit 305 may include a ray generation subunit 306 configured to generate one or more rays emanating from a predetermined optical sensor position.

[0101] The decision unit 305 may further include an intersection decision subunit 308, which is configured to determine the intersection of each generated ray with the integrated particle-based representation generated in S104. The decision unit 305 may further include a value decision subunit 310, which is configured to determine a set of rendering parameter values ​​based on the integrated particle-based representation generated at the intersection (in particular for each determined intersection). Optionally, the set of rendering parameter values ​​may depend on the position of a given optical sensor and / or the sensor modality of a given optical sensor.

[0102] The computing device 300 can be configured to perform method 100.

[0103] Figure 4 schematically shows the architecture of a computing device 400 that generates a synthetic sensor dataset. The computing device 400 includes a sensor instruction receiving interface 402 configured to receive instructions for a set of optical sensors. Each optical sensor in the set is associated with a single sensor modality. The set here includes at least two optical sensors, each having a different sensor modality. The computing device 400 further includes a representation generation unit 408, which is configured to generate a unified particle-based representation for all sensor modalities included in the set of optical sensors. The generation of the unified particle-based representation is performed based on the received instructions for the set of optical sensors. The unified particle-based representation represents the same scene. The computing device 400 further includes a decision unit 409, which is configured to determine the intersection of rays emanating from the optical sensor locations with the unified particle-based representation generated for each optical sensor in the set of optical sensors. The computing device 400 further includes a dataset generation unit 414, which is configured to generate a composite sensor dataset of the scene for each optical sensor in the set. The composite sensor dataset is generated based on the intersection of the determined rays and the unified particle-based representation.

[0104] Optionally, the computing device 400 may include a parameter receiving interface 404, which is configured to receive instructions for at least one sensor parameter for each optical sensor included in the instructions received in S202.

[0105] Alternatively or additionally, the computing device 400 may include a scene characteristic receiving interface 406 configured to receive instructions for the object positions and / or at least one characteristic of the scene.

[0106] The decision unit 409 may include a ray generation subunit 410 configured to generate one or more rays emanating from the position of each optical sensor in the set of optical sensors. The decision unit 409 may further include a crossing decision subunit 412 configured to determine the intersection of each generated ray with the generated integrated particle-based representation. The decision unit 409 may further include a value decision subunit 413 configured to determine at least one value of a rendering parameter for each optical sensor for each intersection of the determined ray with the generated integrated particle-based representation. The value of at least one rendering parameter can be determined based on the received instructions for the set of optical sensors, and optionally, based on the received instructions for at least one sensor parameter of the optical sensors.

[0107] The computing device 400 can be configured to perform method 200.

[0108] The technique (including, for example, Method 100, Method 200, Computing Device 300 and / or Computing Device 400) can also be described alternatively as multimodal Gaussian splatting for synthetic sensor measurement.

[0109] The technique (including, for example, Method 100, Method 200, Computing Device 300 and / or Computing Device 400) generalizes conventional Gaussian splatting pipelines and optimizes an integrated particle-based representation of a scene (also referred to as an integrated scene model and / or scene representation) by taking into account data from various multimodal sensors (e.g., cameras, LiDAR and / or radar). Observations (and / or images and / or measurements) from these modalities can augment each other to generate a richer representation.

[0110] In the concept of vanilla Gaussian splatting models, an optimized (particularly unified particle-based) representation can be used to render new multimodal observations originating from the same scene but with different sensor orientations, for example.

[0111] The technique (including, for example, Method 100, Method 200, Computing Device 300 and / or Computing Device 400) enables the generalization and optimization of multimodal Gaussian splatting scene representations. These representations can be used in various application scenarios. In a first application scenario, additional multimodal composite (e.g., training, test, and / or validation) datasets (also simply referred to as training data) for perceptual tasks (and / or perceptual models) (particularly downstream) can be generated by rendering the scene from a new position and / or orientation, and / or in a new configuration (e.g., by changing multiple optical sensors having at least one modality). The position and / or orientation may relate to optical sensors, the vehicle or robot, and / or other objects in the scene. Alternatively or additionally, in a second application scenario, the learned integrated particle-based representation can be used as a world model for closed-loop simulations.

[0112] The technique (including, for example, Method 100, Method 200, Computing Device 300 and / or Computing Device 400) may include the creation of a digital twin and / or can be used for the analysis of data acquired from a sensor. The sensor can determine the observation (and / or measurement) of the environment (also referred to as a scene) in the form of a sensor signal (also referred to as a sensor dataset and / or sensor data), which may be given by digital images such as video, radar and / or LiDAR.

[0113] This technology can utilize virtual sensors. Alternatively or additionally, information about elements encoded by the sensor signals (e.g., the scene and / or environment around the vehicle) can be obtained based on the sensor signals (e.g., indirect measurements can be performed based on the sensor signals used for direct measurements).

[0114] The technology (including, for example, Method 100, Method 200, Computing Device 300 and / or Computing Device 400) may relate to the upstream part of an ML toolchain. The technology may be a technology for generating training data for training, and / or a technology for generating test data to check whether the trained ML system can operate safely, and / or a generative model for generating training data or test data, without requiring direct improvement of the ML system available for downstream applications.

[0115] In conventional Gaussian splatting pipelines [2], a scene is represented as a family of three-dimensional Gaussians. Two-dimensional camera images of these scenes are used as observations to optimize the scene representation. Each Gaussian has the following parameters that describe it: mean, covariance, opacity, and color. Traditionally, the Gaussian parameters were optimized directly during optimization toward a state where rendering of the scene representation at the training sensor's position and / or orientation of the training sensor produced an image that was as similar as possible to the training image. Rendering was traditionally performed via computationally efficient splatting operations (and thus naming). For mathematical details of both optimization and rendering, see the original Gaussian splatting paper [2] (including, e.g., Method 100, Method 200, Computing Device 300 and / or Computing Device 400). The technique generalizes conventional Gaussian splatting pipelines for the case of multiple sensor modalities. The technique requires a modification of the rendering mechanism. Typically, the splatting step of projecting the Gaussian onto the image plane is replaced by a direct calculation of the ray-Gaussian intersection. This is essential because splatting only works if a flat 2D image plane exists, and does not work for sensors like LiDAR or radar.

[0116] Figure 5 shows an exemplary embodiment of the overview of a multimodal Gaussian splatting pipeline.

[0117] Given multimodal observations of a scene (e.g., LiDAR measurement 502, camera measurement 512, and radar measurement 522), the objective is to optimize an integrated Gaussian representation 510 that can generate new multimodal observations from the same scene, but for example, at different sensor positions and / or sensor orientations. The scene is represented as a family of 3D Gaussians with several parameters, as schematically shown in reference numeral 510. The list of parameters may include sensor-independent mean and Gaussian covariance matrices, along with several sensor-specific parameters, e.g., camera color and opacity, LiDAR intensity, and / or radar RCS. The goal is to optimize the parameters so that the set of Gaussians faithfully represents the recorded scene. The optimization uses all available observations 502;512;522 from different sensor modalities for sensor-independent parameters, and only observations from the corresponding modality for sensor-specific parameters.

[0118] The training process in Figure 5 is initiated by a set of sensor measurements 502;512;522. Given a set of extrinsic and intrinsic sensor parameters for each measurement 502,512,522, a sensor-specific radiation sampler (e.g., LiDAR radiation sampler 504, camera radiation sampler 514, and radar radiation sampler 524) is used to generate a set of 3D radiation lines. As an example to illustrate the basic formula, a LiDAR sensor is used. A similar basic formula can be applied to any number of modalities. For example, each radiation line within a family of LiDAR radiation lines can be sampled and mapped to a single point in a LiDAR point cloud. The interaction between radiation lines and Gaussians can be determined (e.g., calculated), and thus, for each radiation line, it can be determined which Gaussians affect that radiation line. Mathematically this is a simple operation, but it can be extremely costly. A scene representation 510 may contain hundreds of thousands of Gaussians, and it may become necessary to determine (e.g., calculate) the response intensity of the interaction between all rays and the Gaussians. Therefore, non-trivial approximations may be required for the calculations here. An example of such an approximation is Gaussian ray tracing [3].

[0119] In [3], the LiDAR sensor output is not rendered, in contrast to the techniques including Method 100, Method 200, Computing Device 300 and / or Computing Device 400. Furthermore, in [3], the LiDAR intensity is not estimated, in contrast to the techniques including Method 100, Method 200, Computing Device 300 and / or Computing Device 400, where the LiDAR intensity is one of the sensor-specific particle-based (e.g., Gaussian) parameters.

[0120] For LiDAR rendering 508, camera rendering 518, and / or radar rendering 528, the selected Gaussian properties are aggregated along each radiation line using the sensor-specific LiDAR renderer 506, camera renderer 516, and radar renderer 526, respectively. In this way, one measurement is generated for each radiation line. For example, in the case of LiDAR, each radiation line is traversed starting from the sensor, and intensity values ​​from the Gaussian affecting that radiation line are accumulated. This process terminates when the maximum possible intensity is reached. Note that not all Gaussian properties need to be relevant to all sensors. For example, camera-specific color does not affect the calculation of LiDAR intensity values.

[0121] Next, the rendered measurements for each ray are compared with the actual records to obtain a loss value 520 that guides parameter optimization. In the case of LiDAR, the L1 loss between the rendered intensity and the observed intensity can also be used for each ray. A sensor modality can be selected to generate new sensor measurements using the optimized scene representation 510, and the desired sensor position can be set, for example, manually. The sensor observation can then be formed by the rendering procedure described above.

[0122] Figure 6 schematically shows a traffic situation including the vehicle 602 equipped with a set of sensors 604 and a processor, in particular a GPU 606. Two different positions of the vehicle 602 are shown in the first lane 608. The road includes a second lane (or side parking area) 610 with an object 612.

[0123] The vehicle 602 is shown by a solid line at the first position and by a dashed line at the second position. The first position may be included, for example, in an actual sensor dataset. The second position can be obtained, for example, by changing the position and / or orientation of the set of optical sensors located on the vehicle through manual input via a GUI.

[0124] Figures 7A, 7B, and 7C schematically show RGB observations obtained from datasets acquired by six cameras and two LiDAR sensors over a 5-second sequence.

[0125] From the three screenshots, it can be seen that the player's vehicle is moving along a road in a traffic scene that includes parked vehicles, buildings, and trees (for example, as schematically shown in Figure 6).

[0126] Figures 8A and 8B show variations of the scene in Figure 7A. In Figure 8A, the vehicle's hood is visible. Figure 8B shows a screen capture of a 3D visualization where all points correspond to a single LiDAR radiation line. The placement of these points is based on the XY radiation line coordinates and estimated depth. The evolved traffic scenes in Figures 7B and 7C can be obtained, for example, by evolving a Gaussian representation and then performing camera rendering.

[0127] Figures 9A and 9B show another variation of scene modification. In this example, optical sensors located inside or on the surface of the vehicle remain stationary relative to a static environment including road ray out and houses. Object-centered modeling allows for the repositioning of objects within the scene, such as the repositioning and / or dynamic reconstruction of other traffic participants. For example, a repositioned object may be related to a different time instance and / or dynamic scene.

[0128] This technology can enrich the training (and / or testing and / or validation) of sensor datasets, enable sensor simulations for development and / or testing, and / or provide editable scenarios (e.g., related to traffic or manufacturing sites). This can enable the early development of new sensors and / or new assemblies for vehicles or robots.

[0129] Citation of Prior Art Documents [1]By B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng, "Nerf: Representing scenes as neural radiance fields for view synthesis", ECCV, 2020 [2]By B. Kerbl, G. Kopanas, T. Leimkuehler, G. Drettakis, "3d gaussian splatting for real-time radiance field rendering", ACM Transactions on Graphics, vol. 42, July 2023 [3]By N. Moenne-Loccoz, A. Mirzaei, O. Perel, R. de Lu-tio, J. M. Esturo, G. State, S. Fidler, N. Sharp, Z. Gojcic, "3d gaussian ray tracing: Fast tracing of particle scenes", 2024

Claims

1. A computer-implemented method (100) for generating an integrated particle-based representation of a scene from multimodality sensor data, The method involves the following steps, namely: Step (S102) of receiving sensor data representing the same scene acquired by a set of at least two optical sensors having various sensor modalities, The steps include: (S104) generating an integrated particle-based representation of the scene covered by the set of at least two optical sensors based on the received (S102) sensor data; For a predetermined optical sensor in the set of at least two optical sensors, the step (S105) is to determine the intersection of the rays emanating from the position of the predetermined optical sensor and the generated (S104) integrated particle-based representation, The steps include rendering the observation of the scene based on the determined intersection (S105) between the rays and the integrated particle-based representation using a sensor modality-specific renderer associated with the predetermined optical sensor (S112), The steps include determining at least one value of the sensor-specific loss function by comparing the observation of the scene rendered based on the integrated particle-based representation (S112) with the observation included in the sensor data received by the predetermined optical sensor (S102), (S114), Includes, A method (100) in which the integrated particle-based representation of the scene is optimized by iterating through the steps of determining the intersection (S105), rendering the observation (S112), determining at least one value of the sensor-specific loss function for each optical sensor in the set of at least two optical sensors (S114), and optimizing the combination of at least one value of the sensor-specific loss function determined for each optical sensor (S114).

2. The at least two optical sensors, each having a different sensor modality, are at least two of the following: camera, LiDAR sensor, and, radar sensor Including at least two of the following: The method according to claim 1 (100).

3. The above method (100) further, Step (S103): Receiving a set of sensor-specific parameters for the predetermined optical sensor. including, The method according to claim 1 or 2 (100).

4. The step of rendering the observation (S112) is based on a set of rendering parameter values ​​for the predetermined optical sensor, Optionally, at least a subset of the rendering parameters corresponds to sensor-specific parameters. The method according to any one of claims 1 to 3 (100).

5. The aforementioned rendering parameters are: Opacity and / or color of the camera, The intensity of the LiDAR sensor, and / or, Radar cross-sectional area (RCS) for radar sensors including, The method according to claim 4 (100).

6. The step of determining the intersection between the radiation line and the integrated particle-based representation (S105) is at least one of the following steps, namely, The steps include generating one or more rays emanating from the position of the predetermined optical sensor (S106), For each generated (S106) ray, the step (S108) is to determine the intersection with the generated (S104) integrated particle-based representation, Step (S110) of determining a set of rendering parameter values ​​for each determined intersection (S108) based on the integrated particle-based representation generated at the intersection (S104), wherein optionally, the set of rendering parameter values ​​depends on the position of the predetermined optical sensor and / or the sensor modality of the predetermined optical sensor. Including at least one of the following: The method according to any one of claims 1 to 5 (100).

7. The sensor-specific loss function includes at least one of the L1 loss function, L2 loss function, structural similarity index (SSIM), and mean squared error (MSE). The method according to any one of claims 1 to 6 (100).

8. The aforementioned integrated particle-based representation includes the position of the particle and at least one covariance value. The method according to any one of claims 1 to 7 (100).

9. The aforementioned integrated particle-based representation includes a three-dimensional, i.e., 3D, Gaussian representation of the scene. The method according to any one of claims 1 to 8 (100).

10. A computer-implemented method (200) for generating a synthetic sensor dataset, The method involves the following steps, namely: Step (S202) receiving instructions for a set of optical sensors, wherein each optical sensor in the set is associated with one sensor modality, and the set includes at least two optical sensors, each having a different sensor modality. The steps include: (S208) generating an integrated particle-based representation that represents the same scene for all sensor modalities included in the set of optical sensors, based on the received instruction (S202) of the set of optical sensors; For each optical sensor in the set of optical sensors, the step (S209) is to determine the intersection of the ray emanating from the position of the optical sensor and the generated (S208) integrated particle-based representation, The steps include generating a composite sensor dataset of scenes related to each optical sensor in the set based on the determined intersection (S209) of the radiation ray and the integrated particle-based representation (S214), Method (200), including.

11. The method (200) further includes at least one of the following steps, namely: The steps include receiving an instruction for at least one sensor parameter for each optical sensor included in the received instruction (S202) (S204), Step (S206) of receiving an indication of the object position in the scene and / or an indication of at least one characteristic, Including at least one of the following: The method according to claim 10 (200).

12. The step of determining the intersection of the radiation line and the integrated particle-based representation (S209) is at least one of the following steps, namely, Step (S210) for each optical sensor in the set of optical sensors, generating one or more rays emanating from the position of the optical sensor, For each generated (S210) ray, the step (S212) is to determine the intersection with the generated (S208) integrated particle-based representation, Step (S213) of determining at least one value of rendering parameters for each determined (S212) intersection of a ray and the generated (S208) integrated particle-based representation, wherein the at least one value of rendering parameters is determined based on the received (S202) instructions of the set of optical sensors and optionally the received (S204) instructions of at least one sensor parameter of the optical sensors, Including at least one of the following: The method according to claim 10 or 11 (200).

13. Use of a synthetic dataset produced by the method (200) of any one of claims 10 to 12 for training a downstream perceptual task model for performing a perceptual task based on a set of optical sensors having at least two different sensor modalities, optionally, The aforementioned perceptual task is at least one of the following, namely: Object detection and / or classification, occupancy estimation, Scene segmentation, Object tracking, Object velocity estimation, Depth estimation and / or distance estimation, and, Time prediction Use including at least one of the following.

14. A computing device (300) for generating an integrated particle-based representation of a scene from multimodality sensor data, The computing device (300) is A sensor data receiving interface (302) is configured to receive sensor data representing the same scene, acquired by a set of at least two optical sensors having various sensor modalities, A representation generation unit (304) is configured to generate an integrated particle-based representation of the scene covered by the set of at least two optical sensors based on the received sensor data, A determination unit (305) configured to determine the intersection of a predetermined optical sensor within the set of at least two optical sensors and the generated integrated particle-based representation, A set of sensor modality-specific renderers (312) configured to render the observation of the scene based on the determined intersection of the radiation and the integrated particle-based representation, A loss function (314) configured to determine at least one value of a sensor-specific loss function, wherein determining the at least one value of the sensor-specific loss function includes comparing the rendered observation of the scene with an observation acquired by a predetermined optical sensor, Equipped with, Computing device (300) that optimizes the integrated particle-based representation of the scene by iterating through the steps of determining the intersection, rendering the observation, determining at least one value of the sensor-specific loss function for each optical sensor in the set of at least two optical sensors, and optimizing the combination of the determined at least one value of the sensor-specific loss function for each optical sensor.

15. A computing device (400) for generating a synthetic sensor dataset, The computing device (400) is, A sensor instruction receiving interface (402) configured to receive instructions for a set of optical sensors, wherein each optical sensor in the set is associated with one sensor modality, and the set includes at least two optical sensors, each having a different sensor modality, A representation generation unit (408) is configured to generate an integrated particle-based representation that represents the same scene for all sensor modalities included in the set of optical sensors, based on the received instructions of the set of optical sensors, A determination unit (409) is configured to determine the intersection of the rays emanating from the position of the optical sensor and the generated integrated particle-based representation for each optical sensor in the set of optical sensors, A dataset generation unit (414) is configured to generate a composite sensor dataset of the scene associated with each optical sensor in the set, based on the determined intersection of the radiation lines and the integrated particle-based representation, A computing device (400) equipped with the following.

Images