Device, processing pipeline, computer program and method for obtaining a representation of an environment

Abstract

Embodiments of the present disclosure relate to a device, processing pipeline, computer program, and / or method for obtaining a representation of an environment. The method comprises obtaining image data from a multi-camera system and extracting salient features of the environment from the image data. The method further proposes obtaining a triplane representation of the environment using the extracted features and obtaining a representation of the environment from a new viewpoint based on the triplane representation.

Description

[0001] Embodiments of the present disclosure relate to a device, processing pipeline, computer program, and / or method for obtaining a representation of an environment. In particular, the present disclosure relates to a concept for reconstructing 3D scenes from 2D images.

[0002] The problem of reconstructing 3D scenes from 2D images has long been a challenge in computer vision. Some approaches use neural radiance fields (NeRFs) to provide novel view synthesis (NVS). NeRFs learn a 3D scene representation from camera images and pose information. Some approaches focus either on reducing the number of required images or on reconstructing the 3D occupancy: - Sparse 3D reconstruction (I.): To regularize the learned scene representation, NeRF uses many RGB images. Many methods attempt to reduce the number of images required to learn the scene. - Semantic occupancy prediction (II.): Transforming consistent and reliable 3D representations from 2D RGB input images or 2.5D lidar input data is particularly important for autonomous driving. Some approaches have proposed parameterizing a voxelized space for this purpose. However, such approaches do not allow for instantaneous, high-fidelity 3D representations from data from multiple cameras at single time points for large, unbounded scenes.

[0003] Reconstruction models and occupancy forecasting applications can also have the following disadvantages: Sparse 3D reconstruction models may be limited in the following respects: a) Slow training and rendering, as most models rely on differentiable volume rendering, which requires multiple processing of each image during training (with an image size of 800 x 800 and 256 queries per ray, the original NeRF required 168 million network queries to render a complete image). b) They require absolute position information, which is usually estimated using additional computationally intensive structure-from-motion algorithms such as COLMAP. c) Their ability to infer occupancy prediction methods for hidden or unobserved parts of the scene is often limited.The following difficulties can be observed in semantic occupancy prediction methods: d) They may be limited in their fidelity and spatial resolution. Since these models do not aim to accurately reconstruct the 3D scene, but instead focus on occupancy (whether a point in space has a non-zero density) and semantic segmentation (which of the predefined classes the density belongs to), they are far from photorealistic or high-fidelity reconstructions. e) Since they do not involve scene contraction, they may have a fixed spatial resolution. They model the scene up to a certain distance (e.g., a radius of 20 meters in any direction), but they cannot capture details or parts of the scene located further away.

[0004] Thus, there may be a need for an improved concept for obtaining a representation of an environment.

[0005] This need can be met by the subject matter of the appended independent claims. The appended dependent claims disclose embodiments of the present disclosure.

[0006] To this end, the proposed approach combines attention mechanisms with differentiable volume rendering for delay-free and photorealistic 3D reconstructions of large, unbounded scenes from data from multiple cameras at single time points. This allows the advantages of existing methods to be maintained or improved upon and their limitations to be overcome.

[0007] Examples of implementations of the proposed approach bring the following advantages in particular: 1. Short inference time (relative to disadvantage of existing models) through a discrete scene representation. 2. No / little dependence on absolute or relative position information (related to disadvantage b) of existing models). The proposed approach may only require a fixed relative camera position that is maintained (as is the case, for example, with permanently mounted cameras on a vehicle). 3. With the proposed approach, a foundation model can be used for generative inference in occluded or otherwise unobserved areas of the scene (related to disadvantage c) of existing models). Such inference is not trivial, and the bulk of the work can be performed by an attention module that parameterizes a triplane to represent the environment. 4. High fidelity (related to disadvantage d) of existing models) and consequently an improvement of fidelity metrics such as peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) or learned perceptual image patch similarity (LPIPS), which come close to human perception. 5. Versatile and flexible coding structure using a scene contraction method (related to disadvantage e) of existing models). 6. The proposed approach can be further trained to output semantics and depth maps.

[0008] Embodiments of the present disclosure provide a method (processing method) for obtaining a representation of an environment. The method comprises obtaining image data from a multi-camera system and extracting prominent features of the environment from the image data. Furthermore, the method proposes obtaining a triplane representation of the environment using the extracted features and obtaining a representation of the environment from a new perspective based on the triplane representation. This allows, in particular, an accurate reconstruction as well as a representation of the environment from a new perspective based on fewer images or even just a single image from the multi-camera system. This allows, for example, a 3D representation of the observed environment to be reconstructed without delay, i.e., approximately in real time.The proposed approach can thus be particularly applied to applications where a delay-free reconstruction of the environment from 2D to 3D (sparse input views) is desired. The proposed approach can be used, for example, in parking assistance systems or other automotive applications to reconstruct a traffic environment where other approaches may not achieve a sufficiently fast and accurate reconstruction.

[0009] The salient features can be extracted by applying a pre-trained convolutional neural network (CNN) or a transformer model to the image data.

[0010] The triplane representation can include geometric and semantic information about the environment. This allows not only the spatial layout of the environment to be captured, but also semantic assessment, such as object categorization / classification.

[0011] In practice, obtaining the triplane representation may involve obtaining the triplane representation using a cross-attention mechanism to map the features into the triplane representation and a self-attention mechanism to improve the 3D coherence and accuracy of the triplane representation.

[0012] In practice, obtaining the environment representation may involve applying volume rendering to the triplane representation. This allows the captured geometric and semantic information about the environment to be synthesized into a coherent and detailed (3D) representation of the environment. This allows new views / perspectives of the environment to be obtained. For this purpose, the triplane representation can be rendered accordingly for a desired perspective.

[0013] Applying volume rendering to the triplane representation may involve applying a neural decoder to the triplane representation to preserve the representation of the environment.

[0014] Preserving the environment representation optionally includes applying a super-resolution module. This allows a photorealistic representation of the environment to be obtained.

[0015] For the proposed approach, a machine learning-based processing pipeline can be trained accordingly.

[0016] Thus, embodiments of the present disclosure may provide a method for training a machine learning-based processing pipeline to obtain a representation of an environment. The training method includes separately training a feature extractor together with an attention mechanism for extracting salient features of an environment from image data of a multi-camera system, a decoder (also referred to as a renderer), and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment. The triplane representation is based on the salient features. In this way, the components of the processing pipeline (feature extractor, attention mechanism, decoder, super-resolution module) can be coordinated for more accurate and / or reliable results.

[0017] In practice, separately training the decoder may include obtaining a triplane representation of the environment from the image data, processing the triplane representation using the decoder to obtain a new view of the environment, and adjusting the decoder based on a comparison of the new view and the image data such that a discrepancy between the new view and the image data decreases.

[0018] In some embodiments, separately training the feature extractor together with the attention mechanism comprises extracting salient features of the environment from the image data using the feature extractor, generating a triplane representation of the environment based on the salient features, and adjusting the extractor and the attention mechanism based on a comparison of the triplane representation and the image data such that a deviation between the triplane representation and the triplane representation generated by the decoder decreases.

[0019] Separately training the super-resolution module may include obtaining a new view of the environment from a triplane representation of the environment using the trained decoder, obtaining a representation of the environment based on the new view using the super-resolution module, and adjusting the super-resolution module based on a comparison of the representation of the environment and the image data such that a discrepancy between the representation of the environment and the image data decreases.

[0020] Further embodiments of the present disclosure provide a machine learning-based processing pipeline obtainable with the proposed training method.

[0021] Further embodiments of the present disclosure provide a computer program comprising instructions that, when the computer program is executed by a computer, cause the computer to perform the proposed processing method and / or training method.

[0022] Further embodiments provide a device comprising: one or more interfaces for communication and a data processing circuit configured to carry out an embodiment of the proposed processing method and / or training method.

[0023] Further, embodiments will now be described with reference to the accompanying drawings. It should be noted that the embodiments illustrated by the aforementioned drawings merely show optional embodiments as an example, and that the scope of the present disclosure is by no means limited to the presented embodiments. Short description of the drawings Fig. 1 shows a flowchart schematically illustrating one embodiment of a method for obtaining a representation of an environment; Fig. 2 shows a flowchart schematically illustrating another embodiment of the proposed approach for obtaining a representation of an environment; Fig. 3 shows a flowchart schematically illustrating another embodiment of the proposed approach; Fig. Figure 4 shows a flowchart schematically illustrating how a neural decoder can be trained according to the proposed approach for training a processing pipeline; Fig. 5 shows a flow diagram schematically illustrating how a triplane representation can be obtained for training a processing pipeline; Fig. Figure 6 shows a flowchart schematically illustrating how a feature extractor can be trained according to the proposed approach; Fig. Figure 7 shows a flowchart schematically illustrating how a super-resolution model can be trained according to the proposed training approach; and Fig. Figure 8 shows a block diagram schematically illustrating a device according to the proposed approach.

[0024] New view synthesis refers to the process of generating images of a scene from viewpoints not present in the original input. In computer vision and graphics, this technique has gained considerable attention due to its potential applications in virtual reality, augmented reality, 3D scene reconstruction, and so on.

[0025] However, as previously mentioned, approaches to synthesizing new views may not provide sufficient fidelity and spatial resolution and / or may be too slow for desired applications, e.g., automotive applications, where short processing times are necessary due to the rapidly changing traffic environment.

[0026] An object of the present disclosure is therefore to provide an improved new view synthesis concept that provides faster interference time and / or higher fidelity / accuracy.

[0027] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings Fig. 1 to Fig. 8 described.

[0028] Fig. 1 shows a flowchart schematically illustrating an embodiment of a method 100 for obtaining a representation of an environment. Further details and aspects are presented with reference to further embodiments, which are schematically illustrated in Fig. 2 and Fig. 3, where an automotive application of the proposed approach is schematically shown.

[0029] The method 100 includes obtaining 110 image data from a multi-camera system. The multi-camera system includes two or more cameras or recordings of the environment. The cameras may be configured to obtain 2D images of the environment. The image data may accordingly include 2D images (“input images” 210) of the environment. In exemplary applications, the cameras may have a different view of the environment. In practice, the cameras may, for example, be positioned and oriented differently with respect to the environment. The image data may thus represent the environment from different perspectives. In automotive or robotic applications, the multi-camera system includes, for example, the surround-view camera system, which includes multiple cameras arranged on an exterior surface of the vehicle or a robot. As can be seen from Fig. 3, the input images therefore comprise, for example, a plurality of images 310 from different (RGB) cameras. In practice, the images can be obtained from multiple cameras permanently mounted on the vehicle. Such a camera system comprises, for example, six (more or less than six) cameras mounted at specific angles (straight forward, front left, front right, rear left, rear right, and straight backward).

[0030] The method 100 further includes extracting 120 ("feature extraction" 220) distinctive features (image features) of the environment from the image data. Distinctive features are, for example, predefined objects or objects from one or more predefined classes. Examples of such distinctive features include moving and / or static objects. In automotive or robotic applications, the distinctive features include, for example, infrastructure objects (e.g., buildings, bridges, traffic lights, roads, and / or the like), people, and / or vehicles, to name just a few.

[0031] Those skilled in the art having benefit from the present disclosure will appreciate that any feature extraction module may be used to extract the salient features. In exemplary implementations of the proposed approach, for example, a pre-trained convolutional neural network (CNN) or a transformer model is applied to the image data to extract the salient features. In some embodiments, a combination of ResNet, also referred to herein as the "image backbone" 320, and a Feature Pyramid Network (FPN) 330 or a deformable attention encoder is used to capture multi-scale features 230 from the image data. This can ensure that crucial information is effectively extracted from the image data and prepared for subsequent processing stages.

[0032] The method 100 also proposes obtaining 130 a triplane representation of the environment using the extracted features. The triplane representation allows for new views of the environment, which may be desirable in various applications where the environment should be represented from a different perspective, e.g., different from the perspective of the system with the multiple cameras. The proposed approach can therefore be applied, for example, in parking assistance systems and / or in remote control systems (e.g., for remotely steering a vehicle). Furthermore, the proposed approach can also be applied in entertainment applications, e.g., for free-viewpoint television.

[0033] To obtain the triplane representation, a 240 / 340 cross-attention mechanism can be used to map the features into the 250 / 350 triplane representation, and a self-attention mechanism can be used to improve the 3D coherence (i.e., that the same extracted features from different images are mapped to each other in the correct position) and / or the (spatial) accuracy of the triplane representation. This enables an exchange of information between and within the three levels of the triplane, which subsequently leads to improved rendering quality.

[0034] To this end, the cross-attention mechanism establishes a relationship between triplane grid indices and the image features via reference points, allowing a triplane embedding to be updated with information contained in the image features. Since calculating complete attention can be very difficult, deformable attention can be applied for sparsification. In this way, the triplane embedding is further refined using cross-attention.

[0035] The triplane representation contains geometric and semantic information about the environment. The triplane is thus enriched with information over several processing steps and, upon output, contains extensive semantic and geometric information about the current scene. The cross-attention and self-attention mechanisms were refined during the initial differentiable weight update of the architecture during the training process. The resulting semantic and geometric information stored in the triplane representation can later be fed into the decoder (render) to generate a new view.

[0036] The method 100 further includes obtaining 140 a representation of the surroundings from a new viewing angle based on the triplane representation. For this purpose, the representation of the surroundings can be rendered from a desired perspective / viewing angle. For the aforementioned automotive or robotic application, e.g., the parking assistance system, a bird's-eye view of the surroundings can be rendered for the images from the surround-view camera system.

[0037] To obtain 140 the representation from a new viewpoint, the triplane representation can be processed in a modification step 360.

[0038] In practice, the modification step can, for example, utilize a neural renderer that includes a universal decoder / triplane decoder 260 / 362 and volume rendering / volumetric rendering 270 / 364. This creates a new low-resolution view 280 / 370.

[0039] Optionally, obtaining the environment representation 140 involves applying a 290 / 380 super-resolution module. The representation with the new (low-resolution) view is thus upscaled and can be augmented with an image-space convolution enriched with the multi-scale image features. This allows the image space to be enhanced with visual and semantic details directly from the input images (thus improving fidelity).

[0040] At this distance, a new high-resolution 292 / 390 view is created (which has a higher resolution than the new low-resolution view). This can be particularly desirable in applications where a more precise representation of the environment is beneficial.

[0041] In some embodiments, the new high resolution 292 / 390 view may be a photorealistic representation of the environment.

[0042] In summary, embodiments of the proposed approach provide a 3D scene representation of the vehicle's surroundings, from which photorealistic 2D images for different viewing angles can be rendered (quickly).

[0043] Compared to other approaches, the proposed approach does not require any information about the absolute position of the cameras, thus reducing the complexity of data acquisition and processing.

[0044] A machine learning-based processing pipeline (architecture) can be complex and, when trained end-to-end, may lack both stability and speed.

[0045] As described below with reference to Fig. 4 to 7, the present disclosure provides a solution to this problem.

[0046] Embodiments of the proposed approach propose a method for training a machine learning-based processing pipeline to obtain a representation of an environment.

[0047] The training procedure (training procedure) proposes that components of the machine learning-based processing pipeline are trained separately (not end-to-end).

[0048] The method thus proposes the separate training of a feature extractor together with an attention mechanism for extracting salient features of an environment from image data of a system comprising multiple cameras, a (neural) decoder, and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment, wherein the triplane representation is based on the salient features.

[0049] Further details on the separate training of the decoder are given below with reference to Fig. 4 described.

[0050] For this purpose, a triplane representation of the environment is obtained from the image data, as described in more detail with reference to Fig. 5 is set out.

[0051] New views are then rendered based on the generated triplanes, and the new views are compared to a ground truth. The ground truth can be obtained from the image data, i.e., the respective images for the new views, or can correspond to them. For comparability, the new views can have the same perspective (field of view - FOV) of the environment / scene as the ground truth.

[0052] To compare the new views with the ground truth, a loss value is obtained that indicates the accuracy of the new views. In practice, the loss can be used, for example, as a mean squared error (MSE). Alternatively, other loss values / functions can be used. The accuracy of the new views can then be used to assess the accuracy of the generated triplanes. If the accuracy of the generated triplanes (triplane representation) is insufficient, the decoder can be retrained and the triplane generation can be repeated until a sufficient (predefined) accuracy is achieved.

[0053] Where appropriate (i.e., if the triplane representation meets a predefined accuracy), the triplane representation is processed using the decoder to obtain a new view of the environment. The decoder compiles information from the triplanes to obtain a representation of the environment from a new perspective to render a new view of the captured environment. In a first step, the decoder provides a 3D representation of the environment based on the generated triplanes. In a second step, volumetric rendering is applied, with the 3D representation of the environment as the input to the volumetric rendering. New views resulting from the volumetric rendering are then compared with a ground truth. The ground truth can be (again) obtained from image data from the same perspective as the new views. For the comparison, a loss value / loss function (e.g.,The decoder is then trained based on the resulting loss value / loss function. Decoder parameters are adjusted / optimized based on the loss value until the loss value / loss function indicates a desired / predefined or optimal accuracy. In other words, the decoder is adjusted based on a comparison of the new view and the image data such that any discrepancy between the new view and the image data decreases.

[0054] Further details on the separate training of the (machine learning based) feature extractor together with the attention mechanism are given below with reference to Fig. 6 described.

[0055] As can be seen, the training of the feature extractor and the attention mechanism can involve several steps.

[0056] To extract distinctive (multi-scale image) features of the environment from the image data, the feature extractor is applied to the image data (“input images”) in the first step.

[0057] The extracted salient features are then used as input to the attention mechanism (“2D-to-3D attention mechanism”) to generate a triplane representation of the environment based on the salient features. The triplane representation generated by the attention mechanism is then compared with the triplane representation generated by the decoder (see Fig. 5).

[0058] The feature extractor and the attention mechanism are then adjusted based on a comparison of the triplane representation and the image data such that any discrepancy between the triplane representation and a triplane representation generated by the decoder decreases. The triplane representation generated by the decoder can be used again for this purpose. For example, a loss value / function (e.g., MSE loss value / function) is used for the comparison. To adjust the feature extractor and the attention mechanism, model parameters can be adjusted according to the principles of machine learning.

[0059] Details on the separate training of the super-resolution module are given below with reference to Fig. 7.

[0060] To train the super-resolution module, new low-resolution views are created using the neural decoder and volumetric rendering (see Fig. 4) and the triplane representation generated by the decoder as input to the decoder.

[0061] The super-resolution module then creates new high-resolution views for the new low-resolution views using the extracted salient features (see Fig. 6 and the corresponding description).

[0062] In a further step, a loss value / function (e.g., MSE loss value / function) is obtained by comparing the resulting new high-resolution views with the ground truth image data. This loss value / function can then be used to adjust model parameters of the (machine learning-based) super-resolution module according to machine learning principles.

[0063] In this way, the super-resolution module mechanism is adjusted based on a comparison of the representation of the environment and the image data in such a way that a deviation between the representation of the environment and the image data decreases.

[0064] Training the components of the processing pipeline separately ensures that fewer parameters are considered per training of a single component compared to end-to-end training. Therefore, training the components separately can lead to more stable and reliable models for the components.

[0065] As experts recognize, the proposed training approach may be suitable for various tasks and / or applications. The resulting machine learning-based processing pipeline may therefore be applicable to various tasks and / or applications.

[0066] In practice, the proposed approach can be implemented as a computer-implemented method. For example, embodiments provide a computer program comprising instructions that, when executed by a computer, cause the computer to perform an embodiment of the proposed method for obtaining a representation of the environment. Alternatively or additionally, the computer program can be configured to perform an embodiment of the proposed training method.

[0067] In an inference stage, the decoder is then frozen to generate the triplane representation, and the frozen decoder is applied to the image data to generate the triplane representation. In practice, a triplane is generated for each image of the image data. The other parts of the architecture, once trained, also remain fixed during inference.

[0068] The proposed approach can also be implemented in a device. Details of such a device are described below with reference to Fig. 8 described.

[0069] Fig. Figure 8 shows a block diagram schematically illustrating one embodiment of such a device 800. The device comprises one or more interfaces 810 for communication and a data processing circuit 820 configured to execute the proposed method.

[0070] In embodiments, the one or more interfaces 810 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the implementation of the proposed concept. In practice, the interfaces comprise, for example, pins, wires, antennas, and / or the like. The interfaces may also comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means, as well as signal amplifiers, compressors, and / or any encryption / decryption means.

[0071] The data processing circuit 820 may correspond to or include any type of programmable hardware. For example, examples of the data processing circuit 820 include a memory, a microcontroller, field-programmable gate arrays, one or more central and / or graphics processing units. To carry out the proposed method, the data processing circuit 820 may be configured to access or retrieve a suitable computer program for executing the proposed method from a memory of the data processing circuit 820 or a separate memory communicatively coupled to the data processing circuit 820.

[0072] In applications, device 800 can be used to display the surroundings from a new perspective, ie, compared to cameras that capture the surroundings. For this purpose, device 800 can be communicatively connected (via one or more interfaces 810) to such cameras to receive image data from the surroundings. In automotive applications, device 800 can be used for parking assistance systems or autonomous driving systems.

[0073] Alternatively or additionally, the device 800 may be suitable for training the machine learning-based processing pipeline as proposed. Existing image data may be used for this purpose, or dedicated image data may be received from one or more cameras.

[0074] Those skilled in the art having benefit from the present disclosure will appreciate that the proposed approach applies not only to image data, but also to any type of sensor data that indicates a geometry or spatial appearance of the environment.

[0075] In the foregoing description, it will be appreciated that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure should not be construed as reflecting an intent that the claimed examples require more features than are expressly recited in each claim. Rather, the subject matter may lie in fewer than all of the features of a single disclosed example, as the following claims reflect. Thus, the following claims are hereby incorporated into the specification, with each claim being capable of standing on its own as a separate example.While each claim may stand on its own as a separate example, it is noted that although a dependent claim may refer to a specific combination with one or more other claims in the claims, other examples may also include a combination of the dependent claim with the subject matter of any other dependent claim, or a combination of any feature with other dependent or independent claims. Such combinations are suggested herein unless it is stated that a specific combination is not intended. Furthermore, it is intended that features of one claim may also be included in any other independent claim, even if that claim is not directly made dependent on the independent claim.

[0076] Although specific embodiments have been illustrated and described in this specification, it will be understood by those of ordinary skill in the art that a variety of alternative and / or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present embodiments. This application is intended to cover any adaptations or variations of the embodiments discussed in this specification. Therefore, the embodiments are intended to be limited only by the claims and their equivalents.

Claims

[1] A method (100) for obtaining a representation of an environment, the method (100) comprising: Obtaining (110) image data from a multi-camera system; Extracting (120) prominent features of the environment from the image data; Obtaining (130) a triplane representation of the environment using the extracted features; Obtaining (140) a representation of the environment from a new perspective based on the triplane representation. [2] The method (100) of claim 1, wherein extracting salient features comprises extracting the salient features by applying a pre-trained convolutional neural network, CNN, or a transformer model to the image data. [3] Method (100) according to claim 1 or 2, wherein the triplane representation comprises geometric and semantic information about the environment. [4] The method (100) of any preceding claim, wherein obtaining the triplane representation comprises obtaining the triplane representation using a cross-attention mechanism for mapping the features into the triplane representation and a self-attention mechanism for improving the 3D coherence and accuracy of the triplane representation. [5] The method (100) of any preceding claim, wherein obtaining the representation of the environment comprises applying volume rendering to the triplane representation. [6] The method (100) of claim 5, wherein applying volume rendering to the triplane representation comprises applying a neural decoder to the triplane representation to obtain the representation of the environment. [7] The method (100) of any preceding claim, wherein obtaining the representation of the environment comprises applying a super-resolution module. [8] A method for training a machine learning-based processing pipeline to obtain a representation of an environment, the method comprising: separately training a feature extractor together with an attention mechanism for extracting salient features of an environment from image data of a multi-camera system, a decoder, and a super-resolution module for obtaining the representation of the environment from a triplane representation of the environment, wherein the triplane representation is based on the salient features. [9] The method of claim 8, wherein the separate training of the decoder comprises: Obtaining a triplane representation of the environment from the image data; Processing the triplane representation using the decoder to obtain a new view of the environment and adjusting the decoder based on a comparison of the new view and the image data such that a deviation between the new view and the image data decreases. [10] A method according to claim 8 or 9, wherein the separate training of the feature extractor together with the attention mechanism comprises: Extracting prominent features of the environment from the image data using the feature extractor; Generating a triplane representation of the environment based on the salient features; and adapting the extractor and the attention mechanism based on a comparison of the triplane representation and the image data such that a deviation between the triplane representation and a triplane representation generated by the decoder decreases. [11] A method according to any one of claims 8 to 10, wherein the separate training of the super-resolution module comprises: Obtaining a new view of the environment from a triplane representation of the environment using the trained decoder, Obtaining a representation of the environment based on the new view using the super-resolution module and adjusting the super-resolution module based on a comparison of the representation of the environment and the image data such that a deviation between the representation of the environment and the image data decreases. [12] A machine learning-based processing pipeline obtainable by a method according to any one of claims 8 to 11. [13] A computer program comprising instructions that, when the computer program is executed by a computer, cause the computer to perform a method (100) according to any one of claims 1 to 11 or the machine learning-based processing pipeline according to claim 12. [14] Facility (800), comprising: one or more interfaces (810) for communication; and a data processing circuit (820) configured to execute a method according to any one of claims 1 to 11 and / or the machine learning-based processing pipeline according to claim 12.

Images