Apparatus, processing pipeline, computer program, and method for obtaining a representation of an environment

Abstract

Embodiments of the present disclosure relate to an apparatus, processing pipeline, a computer program, and / or a method for obtaining a representation of an environment. The method comprises obtaining image data from a multi-camera system and extracting distinctive features of the environment from the image data. Further, the method suggests obtaining a triplane representation of the environment using the extracted features and obtaining a novel view-point representation of the environment based on the triplane representation.

Description

[0001] Description

[0002] Apparatus, processing pipeline, computer program, and method for obtaining a representation of an environment

[0003] Embodiments of the present disclosure relate to an apparatus, processing pipeline, a computer program, and / or a method for obtaining a representation of an environment. In particular, the present disclosure relates to a concept for reconstructing 3D scenes from 2D images.

[0004] The problem of reconstructing 3D scenes from 2D images is a long-standing challenge in computer vision. Some approaches provide novel view synthesis (NVS) using neural radiance fields (NeRF). NeRFs learn a 3D scene representation from camera images and pose information. Some approaches either focus on reducing the number of images required for training or reconstructing 3D occupancy:

[0005] - Sparse 3D reconstruction (I.):

[0006] NeRFs rely on many RGB images for regularizing the learned scene representation. Many methods attempt to reduce the number of images required to learn the scene.

[0007] - Semantic Occupancy prediction (II.):

[0008] Transforming consistent and reliable 3D representations from 2D RGB image inputs or 2,5D lidar inputs is particularly important within autonomous driving. For this, some approaches proposed to parameterize a voxelized space. However, such approaches do not allow instantaneous high-fidelity 3D reconstructions from multi-camera single timepoint data for large unbounded scenes.

[0009] As well, reconstruction models and occupancy prediction applications may suffer from the following drawbacks:

[0010] Sparse 3D reconstruction models may be limited in the following regards: a) Slow training and rendering since most models rely on differentiable volume rendering, which requires going over each image multiple times during training (with an image of size 800 x 800 and 256 queries per ray, the original NeRF required 168 million network queries to render a full image. b) They require absolute pose information, usually estimated using additional compute-intensive structure-from motion algorithms such as COLMAP. c) They are often limited in their ability to perform inference over occluded or unobserved parts of the scene.

[0011] Semantic Occupancy prediction methods may face the following problems: d) They may be limited in their visual fidelity and spatial resolution. Since those models are not aiming at reconstructing the 3D scene faithful but instead focus on occupancy (whether a point in space has non-zero density) and semantic segmentation (to which of the predefined classes does the density belong), they are far from photorealistic or high-fidelity reconstructions. e) They may have a fixed spatial resolution due to the lack of scene contraction. They model the scene up to a certain distance (e.g., 20 meters radius in all directions) but are unable to capture details or scene parts further away.

[0012] Hence, there may be a demand for improved concept for obtaining a representation of an environment.

[0013] This demand may be satisfied by the subject matter of the appended independent claims. The appended dependent claims disclose exemplary embodiments of the present disclosure.

[0014] For this, the proposed approach may combine attentional mechanisms with differentiable volume rendering for instantaneous and photorealistic 3D reconstructions of large unbounded scenes from multicamera single timepoint data. This may retain or improve advantages of existing methods and may overcome their limitations. Exemplary embodiments of the proposed approach particularly provide the following benefits:

[0015] 1. Fast inference time (related to disadvantage a. of existing models) due to a discrete scene representation.

[0016] 2. No / less reliance on absolute or relative pose information (related to disadvantage b) of existing models). The proposed approach may only need a fixed relative camera pose which is maintained (as it is for example with fixed mounted cameras on a vehicle).

[0017] 3. The proposed approach allows the use of a foundation model to perform generative inference over occluded or otherwise unobserved regions of the scene (related to disadvantage c) of existing models). Performing such an inference is non-trivial, and the heavy lifting here may be done by an attention module that parametrizes a Triplane for representing the environment.

[0018] 4. High visual fidelity (related to disadvantage d) of existing models) resulting in an improvement on visual fidelity metrices such as peak signal-to-noise ratio (PSNR), Structural Similarity Index (SSIM) or Learned Perceptual Image Patch Similarity (LPIPS) which closely correspond to human perception.

[0019] 5. Versatile and flexible encoding structure when using a scene contraction method (related to disadvantages e) of existing models).

[0020] 6. The proposed approach further can be trained to additionally output semantics and depth maps.

[0021] Embodiments of the present disclosure provide a (processing) method for obtaining a representation of an environment. The method comprises obtaining image data from a multi-camera system and extracting distinctive features of the environment from the image data. Further, the method suggests obtaining a triplane representation of the environment using the extracted features and obtaining a novel view point representation of the environment based on the triplane representation. This particularly allows an accurate reconstruction and novel view point representation of the environment from less shots or even only one shot of the multi-camera system. This, e.g., allows to instantaneously reconstruct a 3D representation of the observed environment, i.e. , approximately in real-time. So, the proposed approach may be applied particularly in applications where an instantaneous 2D to 3D environment reconstruction (sparse input views) is desired. The proposed approach, for example, may be applied in parking assistance systems or other automotive applications in order to reconstruct a traffic environment where other approaches may fail to reconstruct quickly and accurately enough.

[0022] The distinctive features may be extracted applying a pretrained convolutional neural network, CNN, or a transformer model to the image data.

[0023] The triplane representation may include geometric and semantic information on the environment. This allows not only to capture the environment’s spatial layout but also a semantic assessment, e.g., object categorization / classification.

[0024] In practice, obtaining the triplane representation may comprise obtaining the triplane representation using a cross-attention mechanism for mapping the features into the triplane representation and a self-attention mechanism for improving the 3D coherence and / accuracy of the triplane representation.

[0025] In practice, obtaining the representation of the environment may comprise applying volume rendering on the triplane representation. In this way, the captured geometric and semantic information on the environment may be synthesized into a cohesive and detailed (3D) representation of the environment. In doing so, novel views / perspectives of the environment may be obtained. For this, the triplane representation may be rendered accordingly for a desired perspective.

[0026] Applying volume rendering on the triplane representation may comprise applying a neural decoder to the triplane representation to obtain the representation of the environment.

[0027] Optionally, obtaining the representation of the environment comprises applying a super-resolution module. In this way, a photorealistic representation of the environment may be obtained. For the proposed approach, a machine-learning-based processing pipeline may be trained accordingly.

[0028] So, embodiments of the present disclosure may provide a method for training a machine-learning-based processing pipeline for obtaining a representation of an environment. The training method comprises separately training a feature extractor together with an attention mechanism for extracting distinctive features of an environment from image data of a multi-camera system, a decoder (also called renderer), and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment. The triplane representation is based on the distinctive features. In this way, the components of the processing pipeline (feature extractor, attention mechanism, decoder, super-resolution module) may be attuned to each other for more accurate and / or reliable results.

[0029] In practice, the separate training of the decoder may comprise obtaining a triplane representation of the environment from the image data, processing the triplane representation using the decoder to obtain a novel view of the environment, and adapting the decoder based on a comparison of the novel view and the image data such that a deviation between the novel view and the image data decreases.

[0030] In some embodiments, the separate training of the feature extractor together with the attention mechanism comprises extracting distinctive features of the environment from the image data using the feature extractor, generating a triplane representation of the environment based on the distinctive features, and adapting the extractor and the attention mechanism based on a comparison of the triplane representation and the image data such that a deviation between the triplane representation and the triplane representation generated by the decoder decreases.

[0031] The separate training of the super-resolution module may comprise obtaining a novel view of the environment from a triplane representation of the environment using the trained decoder, obtaining a representation of the environment based on the novel view using the super-resolution module, and adapting the super-resolution module mechanism based on a comparison of the representation of the environment and the image data such that a deviation between the representation of the environment and the image data decreases.

[0032] Further embodiments of the present disclosure provide a machine-learning-based processing pipeline obtainable by the proposed training method.

[0033] Further embodiments of the present disclosure provide a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out the proposed processing method and / or training method.

[0034] In further embodiments provide an apparatus comprising one or more interfaces for communication and a data processing circuit configured to execute an embodiment of any one of the proposed processing and training method.

[0035] Further, embodiments are now described with reference to the attached drawings. It should be noted that the embodiments illustrated by the referenced drawings show merely optional embodiments as an example and that the scope of the present disclosure is by no means limited to the embodiments presented:

[0036] Brief description of the drawings

[0037] Fig. 1 shows a flow chart schematically illustrating an embodiment of a method obtaining a representation of an environment;

[0038] Fig. 2 shows a flow chart schematically illustrating another exemplary embodiment of the proposed approach for obtaining a representation of an environment;

[0039] Fig. 3 shows a flow chart schematically illustrating a further exemplary embodiment of the proposed approach; Fig. 4 shows a flow chart schematically illustrating how a neural decoder may be trained according to the proposed approach for training a processing pipeline;

[0040] Fig. 5 shows a flow chart schematically illustrating how a triplane representation for training a processing pipeline may be obtained;

[0041] Fig. 6 shows a flow chart schematically illustrating how a feature extractor may be trained according to the proposed approach;

[0042] Fig. 7 shows a flow chart schematically illustrating how a super-resolution model may be trained according to the proposed training approach; and

[0043] Fig. 8 shows a block diagram schematically illustrating an apparatus according to the proposed approach.

[0044] Novel view synthesis refers to the process of generating images of a scene from viewpoints that were not present in the original input. This technique has gained significant attention in computer vision and graphics due to its potential applications in virtual reality, augmented reality, 3D scene reconstruction, and more.

[0045] However, as mentioned above, approaches of novel view synthesis may not provide the sufficient visual fidelity and spatial resolution and / or may be to slow for desired applications, e.g., automotive applications where short processing times are required due to the rapidly changing traffic environment.

[0046] One objective of the present disclosure is, therefore, to provide an improved concept of novel view synthesis which provides faster interference time and / or more visual fidelity / accuracy.

[0047] Embodiments of the present disclosure are now described in more detail with reference to the appended drawings Fig. 1 to Fig. 8. Fig. 1 shows a flow chart schematically illustrating an embodiment of a method 100 obtaining a representation of an environment. Further details and aspects are outlined with reference to further embodiments schematically illustrated in Fig. 2 and 3 where an automotive application of the proposed approach is schematically visualized.

[0048] The method 100 comprises obtaining 110 image data from a multi-camera system. The multi-camera system includes two or more cameras or recordings of the environment. The cameras may be configured to obtain 2D images of the environment. Accordingly, the image data may include 2D images (“input images” 210) of the environment. In exemplary applications, the cameras may have a different perspective of the environment. In practice, the cameras, for example, may be positioned and oriented differently relative to the environment. So, the image data may represent the environment from different perspectives. In automotive or robotic applications, the multi-camera system, for example, includes the surround-view camera system, including multiple cameras arranged over an exterior of the vehicle or a robot. As can be seen from Fig. 3, the input images, e.g., accordingly, include a plurality of images 310 from different (RGB) cameras. So, in practice, the images may be obtained from multiple cameras firmly attached to the vehicle. Such camera system, e.g., includes six (more or less than six) cameras mounted at specific angles (front straight, front left, front right, back left, back right, and back straight).

[0049] Further, method 100 comprises extracting 120 (“feature extraction” 220 of) distinctive (image) features of the environment from the image data. Distinctive features, e.g., are predefined objects or objects of one or more predefined classes. Examples of such distinctive features comprise movable and / or static objects. In automotive or robotic applications, the distinctive features, for example, comprise or correspond to infrastructure objects (e.g., buildings, bridges, traffic lights, roads, and / or the like), persons, and / or vehicles to name just a few.

[0050] The skilled person having benefit from the present disclosure will appreciate that any feature extraction module may be used for extracting the distinctive features. In exemplary implementations of the proposed approach, for example, a pretrained convolutional neural network (CNN) or a transformer model is applied to the image data for extracting the distinctive features. In some embodiments, a combination of ResNet, also referred to herein as “image backbone” 320”, and a Feature Pyramid Network (FPN) 330 or a deformable attention encoder is employed to capture multi-scale features 230 from the image data. This may ensure that crucial information from the image data is effectively distilled and prepared for subsequent processing stages.

[0051] Also, method 100 suggests obtaining 130 a triplane representation of the environment using the extracted features. The triplane representation allows novel views of the environment, which may be desired in various applications where it is desired to present the environment from a different perspective, e.g., different from the multi-camera system’s perspective. Accordingly, the proposed approach, for example, may be applied in the parking assistance system and / or remote-control systems (e.g., for maneuvering a vehicle from remote). Apart from that, the proposed approach may be also applied in entertainment applications, for example, for free viewpoint television.

[0052] For obtaining the triplane representation, a cross-attention mechanism 240 / 340 may be used for mapping the features into the triplane representation 250 / 350, and a self-attention mechanism for improving the 3D coherence (i.e. , that the same extracted features of different images are mapped onto each other at the right position) and / or (spatial) accuracy of the triplane representation. So, it allows the three planes of the triplane to exchange information between and within themselves, which leads to an improved rendering quality later.

[0053] To this end, the cross-attention mechanism establishes a relation between triplane grid indexes and the image features through reference points, which allows updating a triplane embedding with information contained in the image features. Since computing the full attention may be intractable, deformable attention for sparcification may be applied. In this way; the triplane embedding is further refined using cross-attention. The triplane representation includes geometric and semantic information on the environment. The triplane is thus information enriched over multiple processing steps and, when outputted, contains rich semantic and geometric information about the scene at hand. The cross- and self-attention mechanisms have been refined to-dos during the initial differentiable weight updating of the architecture during the training process. The resulting semantic and geometric information stored in the triplane representation can later be fed into the decoder (render) for generating a novel view.

[0054] Further, method 100 comprises obtaining 140 a novel view-point representation of the environment based on the triplane representation. For this, the representation the environment may be rendered from a desired perspective / viewpoint. For the aforementioned automotive or robotic application, e.g., the parking assistance system, a bird's-eye view of the environment may be rendered for the images of the surround-view camera system.

[0055] For obtaining 140 the novel view-point representation, the triplane representation may be processed in a modification step 360.

[0056] In the modification step, in practice, for example, a neural Tenderer including a universal / triplane decoder 260 / 362 and volume / volumetric rendering 270 / 364 may be applied. In this way, a low-resolution novel view 280 / 370 is generated.

[0057] Optionally, obtaining 140 the representation of the environment comprises applying a super-resolution module 290 / 380. In doing so, the (low-resolution) novel view representation is upscaled and may be augmented with image-space convolution, enriched with the multi-scale image features. This allows to add visual and semantic details (and as such improving the visual fidelity) into the image space directly from the input images. In this way, a high-resolution novel view 292 / 390 (having a higher resolution than the low-resolution novel view) is generated. This may be particularly desirable in applications where a more accurate representation of the environment is beneficial.

[0058] In some embodiments, the high-resolution novel view 292 / 390 may be a photorealistic representation of the environment.

[0059] To summarize, embodiments of the proposed approach provide a 3D scene representation of the vehicle’s environment from which (quickly) photorealistic 2D images may be rendered for different viewpoints.

[0060] Compared with other approaches, the proposed approach does not require information on an absolute pose of the cameras, which reduces the complexity of data collection and processing.

[0061] A machine learning-based processing pipeline (architecture) may be complex and lack stability as well as speed when trained end-to-end.

[0062] As laid out in more detail below with reference to Fig. 4 to 7, the present disclosure provides a solution to this problem.

[0063] Embodiments of the proposed approach suggest a method for training a machine-learning-based processing pipeline for obtaining a representation of an environment.

[0064] The method for training (training method) suggests that components of the machine-learning-based processing pipeline are trained separately (not end-to-end).

[0065] So, the method suggests separately training a feature extractor together with an attention mechanism for extracting distinctive features of an environment from image data of a multi-camera system, a (neural) decoder, and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment, wherein the triplane representation is based on the distinctive features.

[0066] Further details of the separate training of the decoder are described below with reference to Fig. 4.

[0067] For this, a triplane representation of the environment from the image data is obtained, as laid out in more detail with reference to Fig. 5.

[0068] Subsequently, novel views are rendered based on the generated triplanes, and the novel views are compared to a ground truth. The ground truth may be obtained from or correspond to the image data, i.e., the respective images for the novel views. For comparability, the novel views may have the same perspective (field of view, FOV) of the environment / scene as the ground truth.

[0069] From the comparison of the novel views and the ground truth, a loss value indicating an accuracy of the novel views is obtained. In practice, e.g., the mean squared error (MSE) loss may be used for this. Alternatively, other loss va I ues / fu notions may be used. The accuracy of the novel views than may be used to assess the accuracy of the generated triplanes. If the accuracy of the generated triplanes (triplane representation) is insufficient, the decoder may be retrained and the triplane generation may be executed iteratively until a sufficient (predefined) accuracy is achieved.

[0070] If appropriate (i.e., if the triplane representation meets a predefined accuracy), the triplane representation is processed using the decoder to obtain a novel view of the environment. In doing so, the decoder assembles information of the triplanes in such a way that a representation of the environment from a novel perspective is obtained for rendering a novel view of the captured environment. So, in a first step, the decoder provides a 3D representation of the environment based on the generated triplanes. In a second step, volumetric rendering is applied with the 3D representation of the environment as input to the volumetric rendering. Novel views resulting from the volumetric rendering are then compared to a ground truth. The ground truth (again) may be obtained from image data of the same perspective as the novel views. For the comparison, (again) a loss value / function (e.g., the mean squared error loss) may be used. Then, the decoder is trained based on the resulting loss value / function. For example, parameters of the decoder are adapted / optimized based on the loss value until the loss value / function indicates a desirable / predefined or an optimal accuracy. In other words, the decoder is adapted based on a comparison of the novel view and the image data such that a deviation between the novel view and the image data decreases.

[0071] Further details of the separate training of the (machine-learning-based) feature extractor together with the attention mechanism are described below with reference to Fig. 6.

[0072] As can be seen, the training of the feature extractor and attention mechanism may comprise multiple steps.

[0073] In the first step, the feature extractor is applied to the image data (“input images”) for extracting distinctive (multi-scale image) features of the environment from the image data.

[0074] The extracted distinctive features are then used as input to the attention mechanism (“2D-to-3D attention mechanism") for generating a triplane representation of the environment based on the distinctive features. The triplane representation generated by the attention mechanism is then compared to the triplane representation generated by the decoder (see Fig. 5).

[0075] The feature extractor and the attention mechanism are, then, adapted based on a comparison of the triplane representation and the image data such that a deviation between the triplane representation and a triplane representation generated by the decoder. For this, the triplane representation generated by the decoder may be reused. For the comparison, e.g., (again) a loss value / function used (e.g., MSE loss value / function). For adapting the feature extractor and the attention mechanism, e.g., model parameters may be adapted according to principles of machine learning. Details of the separate training of the super-resolution module are outlined below with reference to Fig. 7.

[0076] For training the super-resolution module, low-resolution novel views are generated applying the neural decoder and volumetric rendering (see Fig. 4) and the triplane representation generated by the decoder as input to the decoder.

[0077] Then, the super-resolution module generates high-resolution novel views for the low-resolution novel views using the extracted distinctive features (see Fig. 6 and respective description).

[0078] In a further step, a loss value / function (e.g., MSE loss value / function) is obtained from a comparison of the resulting high-resolution novel views with the image data as ground truth. The loss value / function, then may be used to adapt model parameters of the (machine learning based) super resolution module according to principles of machine learning.

[0079] In this way, the super-resolution module mechanism is adapted based on a comparison of the representation of the environment and the image data such that a deviation between the representation of the environment and the image data decreases.

[0080] The separate training of the processing pipeline components provides that less parameters are considered per training of a single component compared to end-to-end training. For this reason, the separate training of the components may lead to more stable and reliable models for the components.

[0081] As the skilled person will appreciate, the proposed training approach may be suitable for various tasks and / or applications. Accordingly, the resulting machine-learning-based processing pipeline may be applied for different tasks and / or for different applications. In practice, the proposed approach may be implemented as a computer-implemented method. For example, embodiments provide a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out an embodiment of the proposed method for obtaining a representation of the environment. Alternatively, or additionally, the computer program may be configured to carry out an embodiment of the proposed training method.

[0082] Then, in an inference stage, for generating the triplane representation, the decoder is frozen and the frozen decoder is applied to the image data for generating the triplane representation for the image data. In practice, a triplane is generated for each image of the image data. The other parts of the architecture once trained also remain fixed during inference time.

[0083] As well the proposed approach may be implemented in an apparatus. Details of such an apparatus are described below with reference to Fig. 8.

[0084] Fig. 8 shows a block diagram schematically illustrating an embodiment of such an apparatus 800. The apparatus comprises one or more interfaces 810 for communication and a data processing circuit 820 configured to execute the proposed method.

[0085] In embodiments, the one or more interfaces 810 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means.

[0086] The data processing circuit 820 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 820, e.g., comprise a memory, microcontroller, field programable gate arrays, one or more central, and / or graphical processing units. To execute the proposed method, the data processing circuit 820 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 820 ora separate memory which is communicatively coupled to the data processing circuit 820.

[0087] In applications, the apparatus 800 may be deployed for representing the environment from a novel perspective, i.e., compared to cameras capturing the environment. For this, the apparatus 800 may be communicatively coupled (via the one or more interfaces 810) to such cameras to receive image data of the environment. In automotive applications, the apparatus 800 may be deployed for parking assistance of systems or autonomous driving systems.

[0088] Alternatively, or additionally, the apparatus 800 may be suitable for training the machine learning based processing pipeline as suggested. For this, existing image data may be used or dedicated image data may be received from one or more cameras.

[0089] The skilled person having benefit from the present disclosure will appreciate that the proposed approach not only applies for image data but also for any kind of sensor data indicative of a geometry or spatial appearance of the environment.

[0090] In the foregoing description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim.

[0091] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the embodiments be limited only by the claims and the equivalents thereof.

Claims

Patent claims1. A method (100) for obtaining a representation of an environment, the method (100) comprising: obtaining (110) image data from a multi-camera system; extracting (120) distinctive features of the environment from the image data; obtaining (130) a triplane representation of the environment using the extracted features; obtaining (140) a novel view point representation of the environment based on the triplane representation.

2. The method (100) of claim 1 , wherein extracting distinctive features comprises extracting the distinctive features applying a pretrained convolutional neural network, CNN, or a transformer model to the image data.

3. The method (100) of claim 1 or 2, wherein the triplane representation includes geometric and semantic information on the environment4. The method (100) of any one of the preceding claims, wherein obtaining the triplane representation comprises obtaining the triplane representation using a cross-attention mechanism for mapping the features into the triplane representation and a self-attention mechanism for improving the 3D coherence and / accuracy of the triplane representation.

5. The method (100) of any one of the preceding claims, wherein obtaining the representation of the environment comprises applying volume rendering on the triplane representation.

6. The method (100) of claim 5, wherein applying volume rendering on the triplane representation comprises applying a neural decoder to the triplane representation to obtain the representation of the environment.

7. The method (100) of any one of the preceding claims, wherein obtaining the representation of the environment comprises applying a super-resolution module.

8. A method for training a machine-learning-based processing pipeline for obtaining a representation of an environment, the method comprising: separately training a feature extractor together with an attention mechanism for extracting distinctive features of an environment from image data of a multi-camera system, a decoder, and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment, wherein the triplane representation is based on the distinctive features.

9. The method of claim 8, wherein the separate training of the decoder comprises: obtaining a triplane representation of the environment from the image data; processing the triplane representation using the decoder to obtain a novel view of the environment; and adapting the decoder based on a comparison of the novel view and the image data such that a deviation between the novel view and the image data decreases.

10. The method of claim 8 or 9, wherein the separate training of the feature extractor together with the attention mechanism comprises:extracting distinctive features of the environment from the image data using the feature extractor; generating a triplane representation of the environment based on the distinctive features; and adapting the extractor and the attention mechanism based on a comparison of the triplane representation and the image data such that a deviation between the triplane representation and a triplane representation generated by the decoder decreases.

11. The method of any one of the claims 8 to 10, wherein the separate training of the super-resolution module comprises: obtaining novel view of the environment from a triplane representation of the environment using the trained decoder; obtaining a representation of the environment based on the novel view using the super-resolution module; and adapting the super-resolution module mechanism based on a comparison of the representation of the environment and the image data such that a deviation between the representation of the environment and the image data decreases.

12. A machine-learning-based processing pipeline obtainable by a method of any one of the claims 8 to 11 .

13. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out a method (100) of any one of the claims 1 to 11 or the machine-learning-based processing pipeline of claim 12.

14. An apparatus (800) comprising: one or more interfaces (810) for communication; and a data processing circuit (820) configured to execute a method of any one of the claims 1 to 11 and / or the machine-learning-based processing pipeline of claim 12.

Images