Scene generation

Abstract

Examples disclosed herein provide computer-implemented methods and apparatus of generating a realistic front view, FV, of an environment. A method comprises: receiving, as input, a latent floorplan indicative of an environment, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment; obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; and outputting the realistic FV of the environment.

Description

[0001] SCENE GENERATION TECHNICAL FIELD [1] The present disclosure relates to scene generation, and, in particular, generating a realistic front view, FV, of an environment from a representation of a bird’s-eye-view, BEV, of that environment. Examples relate to an apparatus, computer-implemented methods, and machine-readable media having program code stored thereon configured to perform computerised generation of a FV of an environment. BACKGROUND [2] Semantic bird’s-eye-view (BEV) representations are a popular map format, being easily interpretable and editable. For example, the ability to generate and test within traffic scenes with a high degree of control, complexity, variability and fidelity is essential for identifying potential points of failure and maximising predictive power, thus improving systems before deployment. [3] However, synthesising ground-view images from BEVs (for example, obtaining front view, FV, images, or other ground-level views such as side view images, or rear view images, depending on the angle of rotation of the input semantic BEV) is a difficult task. Both the mapping from the BEV to a FV, and synthesising highly photo-realistic imagery of the FV are desirable, thus the synthesis process is preferably capable of simultaneously considering both the geometry and the appearance of the scene. This is a challenging computational problem. Examples disclosed herein aim to provide methods of synthesising accurate photo-realistic ground-view images from BEVs, and solve this problem in the art. SUMMARY [4] According to an aspect, there is provided a computer-implemented method of generating a realistic front view, FV, of an environment, the method comprising: receiving, as input, a latent floorplan indicative of an environment, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment; obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; and outputting the realistic FV of the environment. [5] The method may comprise receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to obtain the latent floorplan. [6] The method may comprise receiving a BEV segmentation map, representational of the semantic BEV of the environment; and encoding the BEV segmentation map to obtain the latent floorplan. [7] The method may comprise receiving, as input, a text caption, the text caption descriptive of a semantic BEV; and encoding the text caption to generate the BEV segmentation map. [8] Obtaining the volumetric representation of the environment using a neural radiance field, NeRF may comprise: using a ray-based point sampler to sample points along one or more rays projected within a volume above the latent floorplan; projecting the points onto the latent floorplan to obtain one or more latent floorplan features; and lifting each latent floorplan feature to obtain a lifted feature and a density each corresponding to the latent floorplan feature, the lifted feature and the density of a plurality of latent floorplan features providing the volumetric representation. [9] Obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment may comprise: integrating the lifted features, weighted by the densities, over each ray, to obtain a FV feature map indicative of the features present in the FV.

[0010] Obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment may comprise: integrating the point depths of the lifted features, weighted by the densities, over each ray, to obtain a FV depth map indicative of the depth of the features present in the FV.

[0011] Obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment may comprise: concatenating the FV feature map with the FV depth map to obtain a concatenated map, and decoding the concatenated map to obtain the semantic FV and a FV instance map.

[0012] The FV instance map may comprise an instance pixel centre map and an instance pixel offset map.

[0013] Obtaining the realistic FV of the environment based on the semantic FV using diffusion may comprise using a diffusion model on the semantic FV, the diffusion model conditional on one or more of: a BEV segmentation map representative of the latent floorplan, an FV segmentation map, an FV instance map, an FV depth map, and a learned latent conditioning.

[0014] The semantic BEV and the semantic FV of the environment may each comprise at least two semantically different feature categories.

[0015] The method may be of generation of a realistic FV of an urban environment, wherein the feature categories comprise one or more of: roads, pedestrian paths, kerbs, trees, shrubs, grass, lampposts, vehicles, fencing, road markings, buildings, bus shelters, sky, empty land, pedestrians, bicycles, bicycle riders, traffic poles, traffic signs, traffic lights, train tracks, tram tracks, drivable paths, and non-drivable paths.

[0016] The method may comprise outputting one or more of: the semantic FV corresponding to the realistic FV; an instance FV indicative of one or more individual instances of objects in the same semantic class, the instance FV corresponding to the realistic FV; and a depth map indicative of an apparent depth of objects in the FV, the depth map corresponding to the realistic FV.

[0017] The method may generate the realistic FV of an urban environment from the semantic BEV of the urban environment.

[0018] In an aspect there is provided a computer-implemented method of training a generative model for generation of a realistic front view, FV, of an environment from a semantic birds- eye-view, BEV, of the environment, the computer-implemented method comprising: a first training phase comprising: receiving, as input, a latent floorplan indicative of an environment, the latent floor plan representational of a semantic BEV of the environment, the latent floor plan associated with a corresponding semantic FV of the environment; obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; and associating the obtained semantic FV of the environment with the semantic BEV of the environment; and a second training phase comprising: receiving, as input, the semantic FV; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; associating the obtained realistic FV of the environment with the semantic FV of the environment; and providing, as output, the realistic FV associated with the semantic FV and the semantic BEV.

[0019] The method may comprise, in the first training phase, receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to obtain the latent floorplan.

[0020] The method may comprise, in the first training phase: receiving a BEV segmentation map, representational of the semantic BEV; and encoding the BEV segmentation map to obtain the latent floorplan.

[0021] The method may comprise, in the first training phase, receiving, as input, a text caption, the text caption descriptive of a semantic BEV; and encoding the text caption to generate the BEV segmentation map.

[0022] The text caption may be obtained by one or more of: manual user entry of a description of the environment to provide the text caption; extraction of the text caption from a description of the environment using a computer-implemented description extraction module; and procedural generation of a text caption alongside generation of the semantic BEV.

[0023] The first training phase may operate on a first set of training data, and the second training phase may operate on a second set of training data, the second set of training data different from the first set of training data.

[0024] The semantic BEV may be obtained by: segmenting a point cloud representative of the environment; obtaining one or more segmented FV of the environment from the segmented point cloud, the one or more segmented FV from different viewpoints; for the one or more segmented FV of the environment, capturing a sparse depth map of the segmented FV and a corresponding aerial view of the segmented point cloud to obtain the semantic BEV.

[0025] In an aspect there is provided an apparatus configured to generate a realistic front view, FV, of an environment, the apparatus comprising: an input module configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment; a neural radiance field, NeRF, module configured to obtain a volumetric representation of the environment based on the latent floorplan; a volume renderer module configured to obtain a semantic FV of the environment based on the volumetric representation of the environment; a diffusion module configured to obtain a realistic FV of the environment based on the semantic FV; and an output module configured to output the realistic FV of the environment.

[0026] In an aspect there is provided an apparatus configured to train a generative model for generation of a realistic front view, FV, of an environment from a semantic birds-eye-view, BEV, of the environment, the apparatus comprising: a first training phase module comprising: BEV input module configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic BEV of the environment, the latent floor plan associated with a corresponding semantic FV of the environment; a neural radiance field, NeRF, module configured to obtain a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; a volume renderer module configured to obtain a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; and a first association module configured to associate the obtained semantic FV of the environment with the semantic BEV of the environment; and a second training phase module comprising: a FV input module configured to receive, as input, the semantic FV; a diffusion module configured to obtain a realistic FV of the environment based on the semantic FV using a diffusion technique; a second association module configured to associate the obtained realistic FV of the environment with the semantic FV of the environment; and an output module configured to provide, as output, association data comprising the realistic FV associated with the semantic FV and the semantic BEV.

[0027] In an aspect there is provided a machine-readable medium having program code stored thereon which, when executed by a computer, causes the computer to perform any method disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Examples are further described hereinafter with reference to the accompanying drawings, in which: Figures 1A-1C illustrate examples of a BEV segmentation map, a semantic FV, and a realistic synthesised colour image, obtained according to examples disclosed herein; Figure 2 shows schematically a computer-implemented method of generating a realistic front view, FV, of an environment, according to examples disclosed herein; Figure 3 shows an example of a computer-implemented method of generation of a semantic FV of an environment for subsequent processing to obtain a realistic FV of the environment, according to examples disclosed herein; Figures 4a-4d show different FV representations of a scene generated according to examples disclosed herein; Figure 5 illustrates schematically a computer-implemented method of training a generative model for generation of a realistic FV of an environment from a semantic BEV of the environment according to examples disclosed herein; Figure 6a shows schematically an apparatus which is configured to generate a realistic FV of an environment, and Figure 6b shows an apparatus configured to train a generative model for generation of a realistic FV of an environment from a semantic BEV of the environment, according to examples disclosed herein; Figure 7 shows a table of values indicative of outputs from obtaining FVs obtained from methods as disclosed herein, and from known methods for comparison; and Figures 8a to 8c represent ground truth images of a scene, and Figures 8d to 8o illustrate qualitative examples of FV image output, for comparison to the ground truth images, across the methods listed in Table 2 of Figure 7, including methods disclosed herein. DETAILED DESCRIPTION

[0029] Semantic bird’s-eye-view (BEV) representations are a popular map format, being easily interpretable and editable. However, synthesising ground-view images from BEVs is a difficult task, as the computational system needs to learn both the mapping from the BEV to a Front View (FV) structure, as well as being capable of synthesising highly photo-realistic imagery. Thus both the geometry and appearance of the scene are to be considered.

[0030] Creating visual content is of interest, especially in the context of autonomous driving, where software-stack validation can be done without exposure to risks in the wild. Two major paradigms have emerged to generate such content: model / physics-based and data-driven simulators.

[0031] Physics-based testing can provide free and perfect Ground-Truth (GT) signals for a variety of tasks and sensors, and high visual consistency, control, and editability make them desirable for scene manipulation. However, there are major drawbacks: assets and scenes require tedious and time-consuming manual creation, rendering through ray-tracing can require high computational resources, and the simulated world suffers from a lack of realism commonly known as the sim-to-real gap. On the other hand, data-driven simulators favour realistic sensor readings over other aspects, exploiting real sensor data to drive the rendering. This approach typically involves generative models that discover and learn the data distribution to generate more data by sampling from it. However these approaches can suffer from low diversity, since they rely on real-world data acquisition, which may not capture the full range of possible combinations of appearance and structure of the world at acquisition time.

[0032] Examples disclosed herein present a factorised, or phased, approach that tackles the problem in two stages: a first stage which learns a BEV to FV transformation in the semantic space via Neural Radiance Fields (NeRFs), and a second stage which leverages a diffusion model, such as a Latent Diffusion Model (LDM), to synthesise images conditional on the output of the first.

[0033] Experiments described below show that this approach can produce realistic images (e.g. colour images (e.g. red, green, blue - RGB; blue, green, red – BGR; RGBA, where A encodes transparency; hue, saturation, value (HSV); hue, saturation, luminance (HSL)), greyscale images, or black-and-white images) with a high perceptual quality that are also well aligned with their corresponding FV ground-truth. Regarding different colour spaces as noted above, the methods disclosed herein can be trained on any type of image / colour encoding, provided there is a step of converting the image back to a required image / colour encoding (e.g. RGB) if needed.

[0034] As the Autonomous Driving (AD) industry targets higher levels of autonomy, testing and validating software stacks is paramount to ensuring the safety of passengers and other traffic participants. The ability to generate and test within traffic scenes with a high degree of control, complexity, variability and fidelity is essential for identifying potential points of failure and maximising predictive power, thus improving systems before deployment. As a result, research in this area has gained popularity in both academia and industry, where there has been much focus on building digital twins that are endowed with the diversity and high-fidelity of the real world they are emulating. In this area, synthetic 2D and 3D worlds have been simulated through either data- or model-based approaches. Data-driven approaches allow high realism but suffer from poor scalability and diversity since they rely on real-world data acquisition, which may not capture the full range of possible scenes or situations. Model-based approaches, instead, support high complexity and diversity but are difficult to scale up as they are computationally expensive and require detailed knowledge of the environment being simulated.

[0035] Examples disclosed herein demonstrate that Neural Radiance Fields (NeRFs) can offer a novel solution to these limitations by modelling a scene as a continuous 3D function, enabling high-quality image rendering with realistic lighting and appearance from any viewpoint. NeRFs may be used as a source of training data for downstream tasks such as reinforcement learning and object detection. NeRFs can be used to encode a (static) scene as a continuous volumetric function of colour and density, parametrised by a differentiable renderer via a Multi-Layer Perceptron (MLP) and reconstruction loss supervision. While the approach is capable of high photo realism, it cannot render unseen environments or rearrange objects as each scene is optimized individually. To overcome this limitation, later approaches condition the model on extra information. Contributions have also been made to focus on semantic information and enabling editability of the objects present in scenes.

[0036] The reprojection of sensor inputs into a 2D ground plane, also known as a bird’s-eye- view (BEV) – that is, aerial to street-view translation, is of great interest in the AD community due to its advantages for planning and navigation problems, though it can be challenging – for example, dynamic objects and street assets such as lampposts or traffic lights are not well handled, and thus editability is limited.

[0037] Starting from BEV representations, for example using multiple cameras and / or a fusion of diverse sensors, is desirable because they embody a unified representation of the traffic scene. BEV semantic maps can provide simple and compact, yet rich representation of traffic scenes, enabling easy visualisation, inspection and a high degree of editability. They may be used in road scene-understanding tasks, for example cross-view object detection, segmentation and synthesis. However, BEV to FV segmentation mapping is challenging due to the lack of rich features from which to learn a correspondence. Moreover, to synthesise a FV photorealistic image directly from BEV semantic maps, the system would need to learn the mapping from BEV to FV structure and to synthesise highly photo-realistic images, thus having to consider both the scene’s geometric structure and appearance simultaneously. NeRFs are thus attractive to solve this problem as they leverage the intrinsic geometry of a ray-based sampler following a simple camera model.

[0038] Examples proposed herein use a top-down approach in which the model is conditioned on BEV segmentation maps to enable visualisation, interpretability, and improved control over the scene contents. As such, examples disclosed herein are able to generate realistic FV RGB images of urban traffic scenes, coupled with well-aligned FV semantic, instance, and depth maps, starting from their semantic BEV projections only – i.e.2D top-down semantic maps, as depicted in Figures 1A to 1C. A BEV segmentation map is shown in Figure 1A, where in this example a “V” shaped road fork has vehicles along the sides, buildings set back from the roads, and there are trees and greenery in between the roads inside the “V”. The cross-view translation problem is split in two steps. In the first step, the BEV segmentation is transformed into a FV segmentation as shown in Figure 1B, wherein the “V” shaped road is now shown as an area at the bottom of the FV, with the vehicles along the edges of the road, and the buildings (an area on the right) set back from the side of the right road fork, and with the trees (above) and greenery (below) are shown above the road horizon, and the sky above the road and trees / greenery. A realistic RGB colour image synthesised from the FV of Figure 1B is output, as shown in Figure 1C, wherein the regions are transformed from semantic regions of a category of object (e.g. road, vehicle, trees, greenery, sky, buildings) are shown as photo- realistic simulated objects.

[0039] In some examples, the semantic BEV and the semantic FV of the environment may each comprise at least two semantically different feature categories. The method may be of generation of a realistic FV of an urban environment. In such an urban environment, the feature categories of the semantic BEV and the semantic FV may comprise one or more of: roads, pedestrian paths, kerbs, trees, shrubs, grass, lampposts, vehicles, fencing, road markings, buildings, bus shelters, sky, empty land, pedestrians, bicycles, bicycle riders, traffic poles, traffic signs, traffic lights, train tracks, tram tracks, drivable paths, and non-drivable paths. Methods disclosed herein may generate the realistic FV of an urban environment from the semantic BEV of the urban environment. Generally the methods disclosed herein work well in environments such as the urban real world, where most of the objects and features sit “on the ground” without being obscured, in a BEV, by objects above them.

[0040] In examples shown herein, a flat 2D BEV representation is used for simplicity and scalability. Incorporating 3D information is possible but uses additional effort and tools in modelling semantic scenarios. By factorising the output realistic image generation from a BEV into separate steps of moving from BEV to FV, and then generating a photorealistic image from the FV, each process step can use computer architectures explicitly designed for these distinct purposes. NeRF are used for generating the semantic FV from the BEV, and Diffusion is used to generate the photorealistic FV from the semantic FV. Additionally, depth, segmentation and instance maps may be generated which are aligned with the output images, which may, for example, be used as training data for downstream tasks.

[0041] In summary, the cross-view image synthesis task in an urban traffic scene context is tackled in an improved way by examples disclosed herein. By providing a staged approach to generate FV RGB images of complex driving scenes from BEV semantic maps, using NeRF for geometric projections of semantic information and diffusion for high-quality image synthesis, computation efficiency is improved and output images are provided of high quality and accuracy with flexibility for editing and adjustment to allow for training data generation e.g. for autonomous vehicle testing. The additional generation of FV segmentation, depth, and instance maps which are well-aligned with generated images is advantageous in providing data for scene generation. The proposed methods have been tested against a set of baselines and different components of the approach have been tested through an ablation study as described below.

[0042] Advantageously, methods disclosed herein may be conditional on BEV segmentation maps, as opposed to street-level sensor recordings, allowing for greater control over the quality and content of the input data provided to the computational methods for generation of the FV views. The methods disclosed herein may also be fully generative, not reconstructive – that is, given the same conditional BEV map or latent floorplan as input, a large number of variations of the same environment can be generated, and any new environment can be generated by varying the conditional BEV map or latent floorplan. Additionally, as opposed to methods relying on noise inputs, by the use of semantic maps and the staged approach of the methods disclosed herein, scenes can be synthesised from a humanly interpretable and controllable map and outputs can be provided through the overall process which is humanly interpretable, rather than, for example, sampling latent codes from a Gaussian distribution, which are not humanly interpretable. The staged approach makes the problem easier to solve computationally, and relaxes the assumptions on the datasets.

[0043] An example use case of examples disclosed here is querying a map view of an environment, such as a map obtained from a public source such as an internet website, or other mapping source, and querying it to identify areas of particular traffic patterns such as locations prone to accidents or high traffic volume. The methods disclosed herein may be used to process their top-down layout (BEV) of such maps and synthesise street-level (FV) scenery with the possibility to manipulate positions of road objects. This provides the ability to train and / or test an AV in such complex situations before deployment in the real world.

[0044] Another example use case is city-level planning. The methods may be used to (re-) synthesise how a street would look after a proposed change, for instance, the construction of a new building.

[0045] Figure 2 illustrates a computer-implemented method 200 of generating a realistic front view, FV, of an environment.

[0046] The method 200 comprises receiving, as input, a latent floorplan indicative of an environment.202. The latent floor plan is representational of a semantic birds-eye-view, BEV, of the environment. In artificial intelligence, 'latent space' refers to a mathematical space of low dimensionality, which maps what a neural network has learnt from training input. Once it has been trained it understands, for example, all images of trees as existing in a specific area, and all images of birds in another. It may be thought of as the overarching understanding of relationships between learned data points that a machine learning system has been able to derive from the information that it gets fed. In this example, therefore, the latent floor plan may be considered to be a mathematical or computer code representation of a human-readable or understandable floor plan (i.e. which can be seen by a human and understood as a spatial map).

[0047] The method 200 comprises obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF 204. As described in more detail below, the NeRF acts to “lift” features from the 2D latent floorplan into a 3D space, to obtain a 3D distribution of points representing the floorplan, and then project the lifted points from a particular point of view (i.e. location and direction of gaze) of a theoretical observer onto a FV feature image to obtain a FV image.

[0048] The method 200 comprises obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment 206. In some examples, a FV depth map may also be obtained. In some examples, an FV instance map may also be obtained. Volume rendering may be considered to mean one or more techniques used to display a 2D projection of a 3D data set such as a discretely sampled data set, e.g. a 3D scalar field. To render a 2D projection of a 3D data set, a point of view (a virtual “camera”) location in space may be selected, and properties of the points in the volume may be defined.

[0049] The method 200 comprises obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique 208. A diffusion technique in the field of machine learning may be considered to be using a diffusion model which is trained on a series of training images and is capable of generating a realistic image from an input prompt, such as a semantic image or text prompt. Diffusion models are a class of probabilistic generative models that turn noise to a representative data sample.

[0050] The method 200 then comprises outputting the realistic FV of the environment 210, for example causing the realistic FV to be displayed on a display screen, or providing a computer- readable output file as output.

[0051] The method 200 may comprise receiving, as input, a text caption 212. The text caption is descriptive of the semantic BEV. For example, the text caption may be “a road at a 5% incline bends gently to the left with trees on either side and cars parked on the left side of the road but not the right side of the road, and there are no buildings”. The text caption may then be encoded 214 to obtain the latent floorplan 202. In this example the encoding 214 converts the human-readable text caption 212 to the latent floorplan. In some examples, the same text caption 220 which is encoded to provide the BEV segmentation floorplan 216 and then obtain a latent floorplan 202 may be directly encoded 214 to obtain the latent floorplan 202 without the intermediate step of obtaining the BEV segmentation floorplan 216. In some examples, a different text caption may be used 212, 220 depending on whether an intermediate BEV segmentation floorplan is obtained 216 prior to obtaining the latent floorplan 202.

[0052] The method 200 may comprise receiving a BEV segmentation map 216, representational of the semantic BEV of the environment; and encoding the BEV segmentation map 218 to obtain the latent floorplan 202. A BEV segmentation map may be considered to be a map or floorplan which is divided into sections according to the type of object in those sections, as in Figure 1A. The BEV segmentation map may be human-readable or understandable, and therefore can be seen by a human and understood as a spatial map. The different regions in the map correspond to different types of objects in the image, such as roads, trees, grass, vehicles, buildings, and people, for example. The different regions may be colour coded for example to aid human understanding of what the representation is showing.

[0053] In some examples, the method 200 may comprise receiving, as input, a text caption 220, the text caption descriptive of a semantic BEV; encoding the text caption 222 to generate the BEV segmentation map 216, and encoding the BEV segmentation map 218 to obtain the latent floorplan 202. This example may be considered a combination of the features described above, of receiving a text caption 220 descriptive of the semantic BEV, encoding the text caption 222 to produce the semantic BEV 216, and encoding that semantic BEV 218 to arrive at the latent floorplan. In this example the first encoding converts the human-readable text caption 220 the BEV segmentation map 216, and a second encoding 218 converts the human- readable BEV segmentation map to the latent floorplan 202. This may be useful to allow a user to visualise the image-based representation of the BEV scene (i.e. the BEV segmentation map) which is generated from the text caption input, before using the BEV segmentation map as input to obtain the latent floorplan and, in turn, the realistic FV.

[0054] Advantageously, in the approaches set out above, a human-readable form of the BEV may be used as input to generate the latent floorplan, which is then processed using the NeRF to obtain a semantic FV, which is then processed using diffusion to obtain the realistic FV. This stepwise approach allows for the separation of each processing stage, thereby allowing for the processing to be tailored to the particular step being performed, rather than one “all- in”, or end-to-end, process attempting to obtain the realistic FV from a segmented BEV or from a text caption without separating out the FV to BEV conversion stage from the diffusion stage.

[0055] Figure 3 illustrates an example of a computer-implemented method 300 for generating a semantic FV of an environment for subsequent processing to obtain a realistic FV of the environment. This may be considered to represent an example first stage of a two stage process, of learning a mapping from BEV segmentation maps 316 to FV segmentation maps 340, 344 (BEV → FV). A second stage comprises obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique, and outputting the realistic FV of the environment (that is, synthesising FV photo-realistic images from FV segmentation maps (FV → FV). This modular (here, two-stage) approach beneficially allows training each stage with different data sources, lessening the difficult requirement for paired BEV segmentation and FV realistic images. Table 1 below summarises the notation used.

[0056] Table 1: Summary of notation used. Terms with the ˆ (hat) symbol below represent the predicted counterpart of the ground truth term. If a letter has the symbol “ ^ “ above it, this refers to prediction i.e. is the predicted / generated depth from the front-view perspective. Whereas, the symbol refers to the ground-truth (real observation that comes from the dataset). Generally, dashed both ways in Figure 3 are used to illustrate training supervision. In this case in Figure 3, shows that the is trained with Cross-Entropy loss applied between the predicted segmentation and ground-truth segmentation as Equation 10. Similarly for , the L1 norm loss is applied between a predicted depth and a ground truth depth as described by Equation 9, and a predicted instance and a ground truth instance map as per Equation 8. The BEV segmentation map therefore does not have a corresponding predicted BEV segmentation map, because there is no reconstruction learning step for this entity in Figure 3. The front view feature does not have a corresponding ground-truth / predicted pair (although it can be in these examples, as an intermediary representation) because we there is not a distinction made of a feature map being ground-truth (coming from the dataset) or being predicted (being generated by the model); a different architecture with loss applied on feature map would require the “^” notation.

[0057] Overall, Figure 3, which may be considered to show BEV → FV semantic lifting, leverages NeRFs 304 to lift features from a 2D latent floorplan 324b into a 3D space 324a and project them onto a FV feature image 334 using a volume renderer 328, which is then decoded 338 into instance 344 and segmentation maps 340 (instance map 344 is the predicted counterpart of the ground truth instance map 346, and segmentation map 340 is the predicted counterpart of the ground truth segmentation map term 342).

[0058] Figure 3 illustrates that 318 encodes a top-down segmentation 316 into a 2D latent floor plan 324b. A neural radiance field 304 produces lifted features densities 326 corresponding to 3D coordinates . A volume renderer 328 further produces FV feature and depth images: 334 330, by aggregating the points along the rays. Finally, FV segmentation and instance maps 340, 344 are produced by a decoder 338.

[0059] The latent floorplan feature map is conditional on an encoded BEV segmentation map 316:

[0061] A ray-based point sampler used to obtain a series of points in 3D space from the latent floorplan feature map 324b. This step may be considered to comprise receiving 302, as input, a latent floorplan 324b indicative of an environment, wherein the latent floor plan 324b is representational of a semantic birds-eye-view, BEV, 316 of the environment. The ray-based point sampler 324, following a simple pinhole camera model in this example, is then used to sample 3D points along rays within a 3D volume above the latent floorplan 324b. The 3D points are projected onto the plane) to yield 2D coordinates, which are then used to sample a per-point feature via bilinear interpolation:

[0062]

[0063] A neural radiance field function 304 is then used to transform each latent floorplan feature , conditioned on its 3D coordinate (with positional applied), yielding a lifted feature

[0065] A key aspect is that the neural radiance field function 304 learns to lift 2D features into a 3D volume of transformed features. In Figure 3, for points that project onto the same of the latent floorplan 324b, the ray sampler 324 will pick the same feature irrespective of the points’ height coordinate . Thus, for each group of points that project to the same location, the neural radiance field function 304 learns to map latent feature and a height a transformed feature and density associated with 3D coordinate .

[0066] Obtaining the volumetric representation of the environment using a neural radiance field, NeRF 304 may be considered to comprise using a ray-based point sampler 324 to sample points along one or more rays projected within a volume above the latent floorplan 324b; projecting the points onto the latent floorplan 324b to obtain one or more latent floorplan features; and lifting each latent floorplan feature to obtain a lifted feature and a density 326 each corresponding to the latent floorplan feature, the lifted feature and the density of a plurality of latent floorplan features providing the volumetric representation.

[0067] A volume renderer 328 is used to integrate the features and densities 326 across each ray obtain final FV feature and depth 334, 330:

[0069] This step may be considered to comprise obtaining a volumetric representation of the environment based on the latent floorplan 302 using a neural radiance field, NeRF 304 and volume renderer 328.

[0070] The feature image 334 is obtained by integrating features weighted by their

[0071]

[0072] That is, obtaining the semantic FV 340 of the environment by volume rendering 328 based on the volumetric representation of the environment may comprise: integrating the lifted features, weighted by the densities 326, over each ray, to obtain a FV feature map 334 indicative of the features present in the FV.

[0073] Similarly the depth image 330 is obtained by integrating point depths (along the ray) weighted by their densities: light transmittance from the near-plane a point , as a function of densities along each ray.

[0076] That is, obtaining the depth FV 330 of the environment by volume rendering 328 based on the volumetric representation of the environment may comprise: integrating the point depths of the lifted features, weighted by the densities 326, over each ray, to obtain a FV depth map 330 indicative of the depth of the features present in the FV.

[0077] The feature image 334 and the depth image 330 are concatenated 336 and fed to a decoder 338 in this example.

[0078] Finally, the decoder 338 decodes the feature image 334 into a one-hot FV segmentation map 340 and a FV instance map 344:

[0080] where the instance map 346 is given by an instance centres and instance pixel offsets map , on which L1 loss is applied:

[0082] Thus, obtaining the semantic FV 340 of the environment by volume rendering 328 based on the volumetric representation of the environment may comprise concatenating 336 the FV feature map 334 with the FV depth map 330 to obtain a concatenated map, and decoding 338 the concatenated map to obtain the semantic FV 340 and a FV instance map 344. The FV instance map 344 may comprise, in some examples, an instance pixel centre map and an instance pixel offset map.

[0083] These steps above may overall be considered to comprise the step of obtaining a semantic FV 340 and FV instance map 344 of the environment by volume rendering 328 based on the volumetric representation of the environment.

[0084] To speed up convergence and stabilize training, a masked L1 loss may be applied on

[0085]

[0086] In an example, the encoder , decoder and neural radiance field function may be trained end-to-end using a combination of losses. For the segmentation map a cross- entropy loss may be used:

[0088] where if the pixel belongs to class in the ground truth segmentation map and 0 otherwise, denotes a weight balancing each class.

[0089] The second stage synthesises at inference time FV photo-realistic images conditional on the predicted FV segmentation 340, instance 344 and depth 330 maps generated by the first stage as shown in Figure 3. At train time, this stage is conditional on the GT FV segmentation 342, GT instance 346 and GT depth 332. Similar to a Latent Diffusion Model (LDM), a diffusion model is used, which is conditional on the maps generated during the first stage:

[0091] where sampled from a normal distribution (or is a noisy version of the encoded input during training), is the number of denoising steps, is the decoder of the variational autoencoder (VAE) used is a learned latent conditioning: , data at train time, or at inference time - the maps generated during the first stage ( , , ). The denoising diffusion objective used to train the model remains unchanged, except for the added conditional term :

[0095] where is the ground truth noise, is the predicted noise and is the encoder of the VAE .

[0096] These diffusion steps may be considered to comprise the steps of obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; and outputting the realistic FV of the environment. In other words, obtaining the realistic FV of the environment based on the semantic FV using diffusion may comprise using a diffusion model on the semantic FV, the diffusion model conditional on one or more of: a BEV segmentation map representative of the latent floorplan, an FV segmentation map, an FV instance map, an FV depth map, and a learned latent conditioning.

[0097] Various FV representations of the scene may be generated and output by the above approach, and an example is shown in Figures 4a-d. Figure 4a shows a semantic FV (which can be considered to show a segmentation) corresponding to the realistic FV. It can be seen that the different semantic classes of object (road, vehicle, grass, pavement, buildings, trees and sky) are each separately categorised and represented in a different colour to visually distinguish them to an observer. Figure 4b shows an instance FV indicative of one or more individual instances of objects in the same semantic class, wherein the instance FV corresponds to the realistic FV. The semantic class in this example is vehicles, and three separate instances of vehicles are identified and shown in different colours, with the remainder of the FV categorised as “non-vehicle” i.e. “not of the same semantic class as the semantic class of the instances of the differentiated objects”. Figure 4c shows a depth map indicative of an apparent depth of objects in the FV, wherein the depth map corresponds to the realistic FV. Figure 4d shows the realistic FV generated from a latent floorplan via the process outlined above, where (4a,4b,4c) are the result of the first stage, and 4d is the result of the second stage, where diffusion model is applied on (4a,4b,4c).

[0098] The above discussion pertains to using a method to generate a realistic FV from a semantic BEV of an environment. Also included in this disclosure is a computer-implemented method of training a generative model for generation of a realistic front view, FV, of an environment from a semantic birds-eye-view, BEV, of the environment, as shown in Figure 5. The computer-implemented method 500 comprises a first training phase 502 and a second training phase 504. The first training phase 502 comprises receiving, as input, a latent floorplan 506 indicative of an environment. The latent floor plan is representational of a semantic birds-eye-view, BEV, of the environment. The latent floor plan is associated with a corresponding semantic FV of the environment. The first training phase 502 comprises obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF 508; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment 510; and associating the obtained semantic FV of the environment with the semantic BEV of the environment 512.

[0099] In some examples, step 510 may also comprise obtaining a FV depth map and / or a FV instance map as well as the semantic FV (i.e. an FV segmentation map). Thus, data output from step 510 may comprise an FV semantic map, as well as one or more of an FV depth map and / or an FV instance map, to provide different modalities (data representations) to allow for finer-grained information to be obtained and used in generating the realistic FV in step 520.

[0100] The second training phase 504 comprises: receiving, as input, the semantic FV 514; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique 516; associating the obtained realistic FV of the environment with the semantic FV of the environment 518; and providing, as output, the realistic FV associated with the semantic FV and the semantic BEV 520. Step 518 may, in examples where the FV depth map and / or FV instance map is / are also obtained in step 510, associate the obtained FV depth and map / or FV instance map with the realistic FV of the environment, and may comprise outputting the FV depth map and / or FV instance map.

[0101] The method 500 may comprise, in the first training phase 502, receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to obtain the latent floorplan. The method 500 may comprise, in the first training phase 502: receiving a BEV segmentation map, representational of the semantic BEV; and encoding the BEV segmentation map to obtain the latent floorplan. The method 500 may comprise, in the first training phase 502, receiving, as input, a text caption, the text caption descriptive of a semantic BEV; and encoding the text caption to generate the BEV segmentation map.

[0102] The text caption used to generate the latent floor plan 506 (either with or without generation of an intermediate BEV segmentation map) may be obtained by one or more of: manual user entry of a description of the environment to provide the text caption; extraction of the text caption from a description of the environment using a computer-implemented description extraction module; and procedural generation of a text caption alongside generation of the semantic BEV.

[0103] The first training phase 502 may operate on a first set of training data, and the second training phase 504 may operate on a second set of training data, wherein the second set of training data may be different from the first set of training data. In this way, the first set of training data may be tuned to the training of the NeRF-based generation of the semantic FV from the latent floorplan, whereas the second set of training data may be tuned to the generation of realistic FV views from semantic FVs by diffusion. By training the model in these two phases 502, 504, training data curated for the different training phases may be used to improve computation efficiency and increase accuracy in comparison to models which use a single training data set to move from a BEV segmentation map to a realistic FV. Both the factorization and the data sources used provide advantages. The use of a single training stage means it can be difficult to learn or model all the tasks at the same time, so by factorising, or phasing, the method of obtaining a realistic FV from a segmented BEV, the tasks may be more easily and more accurately managed. Similarly, a single data source that satisfies all the requirements of the model, can be difficult to obtain, so by using different data sources, there is greater flexibility in the data which can be used as input. Furthermore, once the overall task has been split into factors / phases, the data source assumption can also be relaxed, also allowing many different sources to be used. For example this allows for additional datasets to be used in the second diffusion phase to advantageously obtain more accurate realistic FVs.

[0104] The semantic BEV from which the latent floorplan may be obtained may itself be obtained by: segmenting a point cloud representative of the environment; obtaining one or more segmented FV of the environment from the segmented point cloud, the one or more segmented FV from different viewpoints; and for the one or more segmented FV of the environment, capturing a sparse depth map of the segmented FV and a corresponding aerial view of the segmented point cloud to obtain the semantic BEV.

[0105] This disclosure includes apparatus which are configured to perform the methods disclosed. For example, Figure 6a illustrates schematically an apparatus 600 which is configured to generate a realistic front view, FV, of an environment. The apparatus may be a laptop or desktop computer, a server, a distributed computer processing network, or other computational apparatus capable of running the models discussed herein.

[0106] The apparatus 600 in this example comprises: an input module 606 configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic birds- eye-view, BEV, of the environment; a neural radiance field, NeRF, module 608 configured to obtain a volumetric representation of the environment based on the latent floorplan; a volume renderer module 610 configured to obtain a semantic FV of the environment based on the volumetric representation of the environment; a diffusion module 612 configured to obtain a realistic FV of the environment based on the semantic FV; and an output module 614 configured to output the realistic FV of the environment.

[0107] It will be appreciated that the term “module” may be considered to mean a hardware module such as a dedicated or shared processor or processors, may be considered to mean a software module such as a section of code, or may be a combination of hardware and software, which is configured to perform the described function. While there is one processor 602 illustrated, it will be appreciated that there may be a plurality of processors connected to provide the described functionality in other examples.

[0108] For example, Figure 6b illustrates schematically an apparatus 650 configured to train a generative model for generation of a realistic front view, FV, of an environment from a semantic birds-eye-view, BEV, of the environment. The apparatus 650 comprises a first training phase module 656 and a second training phase module 658. The first training phase module 656 comprises: a BEV input module 660 configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment, the latent floor plan associated with a corresponding semantic FV of the environment; a neural radiance field, NeRF, module 662 configured to obtain a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; a volume renderer module 664 configured to obtain a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; and a first association module 666 configured to associate the obtained semantic FV of the environment with the semantic BEV of the environment.

[0109] The second training phase module 658 comprises: a FV input module 668 configured to receive, as input, the semantic FV; a diffusion module 670 configured to obtain a realistic FV of the environment based on the semantic FV using a diffusion technique; a second association module 672 configured to associate the obtained realistic FV of the environment with the semantic FV of the environment; and an output module 674 configured to provide, as output, association data comprising the realistic FV associated with the semantic FV and the semantic BEV.

[0110] Again, it will be appreciated that the term “module” may be considered to mean a hardware module such as a dedicated or shared processor or processors, may be considered to mean a software module such as a section of code, or may be a combination of hardware and software, which is configured to perform the described function. While the first and second training phase modules are illustrated as being part of (and / or run on) a processor 652, it will be appreciated that there may be a plurality of processors connected to provide the described functionality.

[0111] Also covered by this disclosure is a machine-readable medium having program code stored thereon which, when executed by a computer, causes the computer to perform any method disclosed herein. Such a machine-readable medium may, in other examples, comprise any medium configured to include instructions stored thereon to perform any computer-implemented method disclosed herein. The instructions may be embedded in said one or more electronic processors 602, 652 of the apparatus 600, 650; may be stored in a memory 604, 654, or may be provided as software to be executed in the apparatus 600, 650. The memory 604, 654 may comprise any suitable memory device and may store a variety of data, data structures, and / or instructions thereon. For example, the memory 604, 654 may store instructions for software, firmware, programs, algorithms, scripts, applications that may control or cause suitable apparatus to perform all or part of the methodology described herein. The memory 604, 654 may comprise a computer-readable storage medium (e.g. a non- transitory, non-volatile or non-transient storage medium) that may comprise any mechanism for storing information in a form readable by a machine or electronic processors / computational devices, including, without limitation: a magnetic storage medium (e.g. floppy diskette); optical storage medium (e.g. CD-ROM); magneto optical storage medium; read only memory (ROM); random access memory (RAM); erasable programmable memory (e.g. EPROM and EEPROM); flash memory; or electrical or other types of medium for storing such information / instructions.

[0112] The apparatus 600, 650 may comprise one or more electronic processors 602, 652 (e.g., a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), Boolean logic circuitry, etc.) that is configured to execute electronic instructions. The term “controller,” “control unit,” or “computational device” may be understood to include a single controller, control unit, or computational device, and a plurality of controllers, control units, or computational devices collectively operating to provide the required control functionality.

[0113] The following discussion sets out an experimental setup performed to illustrate the improvements obtained using methods disclosed herein. The short-name “NeRFloors” is used to indicate the methods disclosed herein.

[0114] A. Data

[0115] The KITTI-360 dataset (Y. Liao et al, “Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d,” ArXiv, vol. abs / 2109.13410, 2021) was chosen as it contains diverse and complex urban scenes. Out of the 9 scenes featured in KITTI-360, the first 8 were used for training, while the last scene is split with approximately 20% for model selection and 80% for final testing. To produce the paired data (BEV segmentation and corresponding FV segmentation), the accumulated point clouds provided were used. The segmented accumulated point clouds were loaded to generate the BEV segmentation maps. For each pose associated with an existing FV segmentation map, a sparse depth map and a corresponding aerial view of the segmented point cloud were captured. As the BEV is generated from accumulated LiDAR data over multiple traversals, dynamic objects in motion do not appear in the BEV semantic map, but they do in the FV segmentation map and RGB image (e.g. walking pedestrians and moving cars, while parked static cars remain valid). Thus the dataset was curated by removing inconsistent BEV-FV pairs to prevent the model from “hallucinating” non-existent objects since a goal of the experiment was to synthesise FV data conditional on the contents of the BEV map. This results in approximately 22,300 train observations, 500 observations for model selection and 2200 testing observations.

[0116] For stage 2 of this factorised approach, the provided FV segmentation maps and RGB images from KITTI-360 were used and, additionally, the train set of the Cityscapes dataset (Cordts et al, "The Cityscapes Dataset for Semantic Urban Scene Understanding", ArXiv, vol. abs / 1604.01685, 2016).

[0117] B. Metrics

[0118] The perceptual quality of the generated images were compared using (1) the Fréchet Inception Distance (FID, a measure of the similarity of two datasets of images) and (2) Kernel Inception Distance (KID) at resolutions of 64 × 64 and at 512 × 512. The second set of metrics measures the ability to reproduce scene structure. Where available, the predicted FV segmentation was compared with the ground truth (GT) segmentation map in terms of Mean Intersection Over Union (mIoU) to understand how accurately the scene’s geometry has been projected. Similarly, the Root Mean Squared Error (RMSE) is reported between predicted and GT depths. Finally, mIoU-align (mean-intersection-over-union alignment) is reported, which is defined here as the mIoU between the segmentation maps extracted by an off-the-self segmentation model – from both the GT and predicted FV RGB images. This approach was selected to test the images’ alignment without introducing undesired variance due to the performance of the off-the-shelf segmentation model itself.

[0119] C. Benchmarks

[0120] First the unconditional baseline was tested, where scene reconstruction starts from normally distributed noise input. For this experiment, the dataset was split into sub-sequences of 100 frames each, and each model was trained using the associated poses as input and the RGB images as targets. The camera intrinsics specific to the front-left camera of the KITTI- 360 dataset were used. The size of was kept at to 32×32 with 32 channels was set to 80m. To study the quality of conditional FV output in the absence of a dedicated BEV to FV transformation step, two models were trained to directly from . The SPADE model (“SPADE” – T Park et al, “Semantic image synthesis with spatially-adaptive normalization”, Proc IEEE / CVF conference on computer vision and pattern recognition, 2019, 2337-2346), and state-of-the-art latent diffusion model “LDM” (R. Rombach et al. “High resolution image synthesis with latent diffusion models”, 2021) were chosen to be trained as single-stage models.

[0121] To check the contribution of a conditional BEV to FV transformation step in the single- stage case, the following experiments were performed (with varying outputs and losses) and the size was set to 100 × 100: 1) (Base 1): A model that only outputs ; 2) (Base 2): A model that outputs pairs, where the additional segmentation and cross-entropy (CE) loss supervision acts as a guide for producing better structure; 3) (Base 3): Similar to (Base 2), but with an adversarial loss, since that reconstruction loss may lead to a model learning a blurry average image; 4) (Base 4): A combination of L1, adversarial and CE loss, as this mixture may lead to better results; 5) (SPADE B1), (SPADE B2), (SPADE B3): similar variations as above, but with the “SPADE” backbone for image synthesis.

[0122] Depending on the combination of losses, image reconstruction (L1) loss and / or an

[0123]

[0124] where represents the generator. While single-stage models are lifting and image synthesis simultaneously, the two stage modular approach presented herein has each stage specialised on one task: the first stage performs the BEV to FV semantic map transformation , while at inference time the second stage synthesises the RGB image of the first stage

[0125] In this example, two experiments were performed for the second-stage image synthesis: one with the “SPADE” backbone (labelled “NeRFloors SPADE”) and one with the “LDM” backbone (labelled “NeRFloors LDM”), both having the first-stage, as disclosed herein, in common.

[0126] D. Model components

[0127] The encoder and decoder from “LDM” were repurposed for the encoder-decoder pair initialised and the entire “LDM” from publicly available pre-trained parameters (SD-v-1-4) but the weights of the first layer were extended to support the one- hot encoded input.

[0128] For the second stage of “NeRFloors LDM”, the model was extended from “LDM” with two extra convolutional layers that embed the conditional inputs. The embedded inputs are concatenated with the standard noisy latents and given as input to the “LDM” denoising network. For adversarial supervision in experiments (Base 1) to (Base 4), the “OASIS” discriminator backbone was used (E. Schonfeld et al, “You only need adversarial supervision or semantic image synthesis,” in International Conference on Learning Representations, 2021).

[0129] V. RESULTS

[0130] Table II, included as Figure 7, presents the benchmarking results on the KITTI-360 dataset. While both the NeRFloors method as disclosed herein and LDM achieve the best performances in terms of FID score – and thus perceptual quality – the NeRFloors method shows much better alignment with the GT FV semantic segmentation, as the BEV to FV transformation is explicitly modelled (i.e. separately to the diffusion step). The comparison against single-stage baselines shows that the factorised approach disclosed herein benefits the system’s performance, increasing output image quality and improving semantic alignment.

[0131] Figures 8d to 8o illustrate qualitative examples of FV output across all methods indicated in Table 2 of Figure 7. The ground truth (GT) is included in Figures BEV ), Figure 8b (semantic FV ) and Figure 8c (realistic FV ), for reference. The results shown in Figures 8d-8o are from the following models, with predictions specified using “^” notation: ● Figure 8d – Generative Scene Networks (GSN): ● Figure 8e – SPADE: ● Figure 8f – LDM: ● Figure 8g – Base 1: ● Figure 8h – Base ● Figure 8i – Base ● Figure 8j – Base ● Figure 8k – SPADE B1: ● Figure 8l – SPADE B2: ● Figure ● Figure 8n – NeRFloors SPADE: ● Figure 8o – NeRFloors LDM:

[0132] In terms of perceptual quality, both the single stage LDM (Figure 8f) approach and the factorised model according to the method NeRFloors LDM (Figure 8o) have a significantly lower (better) FID score than the rest of the models, with 41.10 / 68.75 and 42.23 / 65.81, respectively. In contrast, the unconditional model (Figure 8d) has a much lower perceptual quality, with an FID of 187.39.

[0133] The conditional single-stage models Base 1, (Figure 8g) through SPADE B3, (Figure 8m) generally produce low-quality images as opposed to LDM (Figure 8f) with FID ranging from 309.30 to 85.52 and KID following a similar ranking, as these models are tasked to deal with both BEV to FV transformation and image synthesis. However, in the SPADE experiment, (Figure 8e), where the BEV to FV semantic map transformation is not employed, the perceptual quality is much lower than the experiments where the transformation is employed, such as SPADE B1, (Figure 8k) to SPADE B3, (Figure 8m), although they all have the same backbone for image synthesis, highlighting the benefit of the semantic transformation step. Moreover, both the mIoU alignment and perceptual quality is improved within this group of models, as they are forced to reproduce the FV semantic structure .

[0134] In terms of how well the produced RGB images align with the GT RGB images, the mIoU-alignment metric shows that our factorised approach NeRFloors LDM, (Figure 8o) has significantly better alignment (28.11) than both the single-stage LDM (Figure 8f) model (20.55) and all other approaches. The LDM model, Figure 8f, can reproduce the general layout of scenes in terms of road and buildings but cannot accurately synthesise smaller classes in the right location. In contrast, our two-stage approach reproduces the general layout and the local details. The success of the LDM model, Figure 8f, in terms of perceptual quality can be attributed to the ability to leverage the learnt prior (being pre-trained on a very large dataset) however, when compared to the factorised model NeRFloors LDM (Figure 8o) that also employs the BEV to FV transformation, the mIoU alignment is lower (20.55). In terms of mIoU alignment between predicted and GT FV segmentation, the NeRFloors model (Figure 8n, 8o) also performs best (32.06) compared to all other baselines that output .

[0135] In terms of root-mean-squared-error, RMSE, all models approximately the same, which is however unsurprising, as extra experiments were not designed to optimise the output depth in particular. From a qualitative point of view, GSN, Figure 8d, is unable to generate high-quality samples when trained on urban scenes. On the other hand, it is visible in Figure 8f that LDM produces very realistic looking images, but that these are misaligned with the GT semantic layout. In this particular example, the centre of the frame is dominated by a truck, which is however not present in the BEV semantic segmentation. In contrast, the method disclosed herein as shown in Figure 8o produces imagery much better aligned with the input semantic contents and FV semantic map, while also being highly realistic-looking.

[0136] To further support the design choices in the above experiment, ablations were performed on the first and second stage of the factorised approach, with results in Tabs. III and IV respectively, where new features were iteratively added to a base case.

[0137]

[0138]

[0139] A. BEV → FV semantic lifting

[0140] A simple 1-plane representation of the latent floorplan and a tri-plane representation were tested. In panoptic segmentation, panoptic supervision was introduced, as the model might benefit from information related to object boundaries. Since a real-world dataset was area classes are under-represented thus, they are often ignored in the predicted . To overcome this limitation, class weighting was used to improve accuracy for small classes such as poles and traffic signs, which are difficult to resolve in prior art methods. To improve segmentation output, the final decoder was modified to also be conditional on the predicted depth. As large outdoor scenes were sampled, the number of samples per ray was increased from 100 to 200, at a cost of approx. 2× more memory consumption and increase in the training and inference time. Results in Table. III show that the mIoU has benefited from each additional component. However, depth was only slightly advantaged, with the most improvement being brought by a tri-plane representation and panoptic segmentation.

[0141] B. FV → FV image synthesis

[0142] It was also investigated whether additional conditional inputs to the LDM-based image synthesis improves performance. A basic model was taken to start with, that is conditional on semantic segmentation only:

[0143] Instance segmentation was added; and finally depth conditioning applied .

[0144] To isolate stage 2 ablation results from the effects of stage 1 predictions, GT segmentation, instance and depth data were used as inputs rather than predictions from stage 1. The results are shown in Table. IV, where the inclusion of instances did not have a net beneficial effect but the inclusion of depth significantly improved FID and KID.

[0145] In summary, the examples disclosed herein, supported by experiment, provide an improved system and method for synthesising realistic FV imagery from a BEV segmentation map. The examples disclosed herein address the difficult problem of relating the geometry of the BEV to that of the Ground-View and the generation of realistic imagery from FV semantics. Using conditional NeRFs to model and improve the geometric transformation from the BEV to the FV provides improved results, in combination with separately applying diffusion to obtain the realistic FV. LDMs are extended to be conditional on segmentation, instances, and depth information to achieve high-quality image synthesis. Extensive experiments demonstrate that this factorised approach produces more realistic imagery than prior methods and baselines and that, critically, this realism is accompanied by good alignment to output and GT semantic layout. An advantage of this factorised approach is that it relaxes the requirement of paired BEV segmentation and FV RGB images, thus enabling the usage of more diverse sources of input data, such as simple / proto-simulators for the first stage. Since our model is conditional on BEV segmentation, it offers higher interpretability and level of control than previous methods.

[0146] In other examples, viewpoint and temporal consistency may be maintained, in order to produce structure- and appearance-consistent scenes across consecutive frames, which enables a wider range of downstream tasks to be trained or performed on the resulting images, such as object tracking or trajectory prediction. For example, structure consistency may be obtained in the first stage by training the model on multiple views, such as an original pose from the dataset paired with the full (dense) FV segmentation (i.e. the GT FV segmentation that comes with the dataset), and a random pose (obtained for example by a small random translation on X-axis and a random rotation around the Y-axis applied to the original pose) with an associated sparse FV segmentation that is generated from the point cloud. For the second stage, global style consistency is enforced by applying diffusion in an auto-regressive manner, wherein the output colour image from the previous step (and which may include some or all other current input, e.g., segmentation, instance, depth) is given as input to the current step, to generate the current frame.

[0147] During the training phase, the previous GT colour image can be used as input, while at inference time, the previous predicted colour image can be used as input. If M controls how far in the past the method considers, then a very large M (e.g.100th frame in the past) would require, during training, that the current generated frame preserves a style that was present 100 frames ago. A small M (say for example, the 1st frame in the past) would require that the current generated frame preserves a style which was present one frame ago. If M is too large, then the method may be too restrictive, since objects might have been occluded / revealed in the meantime, but a very small M may be too relaxed, since there can be high correlation between consecutive frames. A colour image may be provided as input to a diffusion model as described above, but it may be concatenated with other inputs and fed into extra

[0149] Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example of a generic series of equivalent or similar features. Many modifications and other examples set out herein will come to mind to a person skilled in the art in light of the teachings presented herein. Therefore, it will be understood that the disclosure herein is not to be limited to the specific examples disclosed herein. Moreover, although the description provided herein provides examples in the context of certain combinations of elements, steps and / or functions may be provided by alternative examples without departing from the scope of the appended claims.

Claims

CLAIMS 1. A computer-implemented method of generating a realistic front view, FV, of an environment, the method comprising: receiving, as input, a latent floorplan indicative of an environment, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment; obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; and outputting the realistic FV of the environment.

2. The computer-implemented method of claim 1, comprising: receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to obtain the latent floorplan.

3. The computer-implemented method of claim 1, comprising: receiving a BEV segmentation map, representational of the semantic BEV of the environment; and encoding the BEV segmentation map to obtain the latent floorplan.

4. The computer-implemented method of claim 3, comprising: receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to generate the BEV segmentation map.

5. The computer-implemented method of any preceding claim, wherein obtaining the volumetric representation of the environment using a neural radiance field, NeRF comprises: using a ray-based point sampler to sample points along one or more rays projected within a volume above the latent floorplan; projecting the points onto the latent floorplan to obtain one or more latent floorplan features; andlifting each latent floorplan feature to obtain a lifted feature and a density each corresponding to the latent floorplan feature, the lifted feature and the density of a plurality of latent floorplan features providing the volumetric representation.

6. The computer-implemented method of claim 5, wherein obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment comprises: integrating the lifted features, weighted by the densities, over each ray, to obtain a FV feature map indicative of the features present in the FV.

7. The computer-implemented method of claim 5 or claim 6, wherein obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment comprises: integrating the point depths of the lifted features, weighted by the densities, over each ray, to obtain a FV depth map indicative of the depth of the features present in the FV. 8 The computer-implemented method of claim 7 when dependent on claim 6, wherein obtaining the semantic FV of the environment by volume rendering based on the volumetric representation of the environment comprises: concatenating the FV feature map with the FV depth map to obtain a concatenated map, and decoding the concatenated map to obtain the semantic FV and a FV instance map.

9. The computer-implemented method of claim 8, wherein the FV instance map comprises an instance pixel centre map and an instance pixel offset map.

10. The computer-implemented method of any preceding claim, wherein obtaining the realistic FV of the environment based on the semantic FV using diffusion comprises using a diffusion model on the semantic FV, the diffusion model conditional on one or more of: a BEV segmentation map representative of the latent floorplan, an FV segmentation map, an FV instance map, an FV depth map, and a learned latent conditioning.

11. The computer-implemented method of any preceding claim, wherein the semantic BEV and the semantic FV of the environment each comprise at least two semantically different feature categories.

12. The computer-implemented method of claim 10, wherein the method is of generation of a realistic FV of an urban environment, wherein the feature categories comprise one or more of: roads, pedestrian paths, kerbs, trees, shrubs, grass, lampposts, vehicles, fencing, road markings, buildings, bus shelters, sky, empty land, pedestrians, bicycles, bicycle riders, traffic poles, traffic signs, traffic lights, train tracks, tram tracks, drivable paths, and non-drivable paths.

13. The computer-implemented method of any preceding claim, comprising outputting, one or more of: the semantic FV corresponding to the realistic FV; an instance FV indicative of one or more individual instances of objects in the same semantic class, the instance FV corresponding to the realistic FV; and a depth map indicative of an apparent depth of objects in the FV, the depth map corresponding to the realistic FV.

14. The computer-implemented method of any preceding claim, comprising generating the realistic FV of an urban environment from the semantic BEV of the urban environment.

15. A computer-implemented method of training a generative model for generation of a realistic front view, FV, of an environment from a semantic birds-eye-view, BEV, of the environment, the computer-implemented method comprising: a first training phase comprising: receiving, as input, a latent floorplan indicative of an environment, the latent floor plan representational of a semantic BEV of the environment, the latent floor plan associated with a corresponding semantic FV of the environment; obtaining a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; obtaining a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; and associating the obtained semantic FV of the environment with the semantic BEV of the environment; and a second training phase comprising:receiving, as input, the semantic FV; obtaining a realistic FV of the environment based on the semantic FV using a diffusion technique; associating the obtained realistic FV of the environment with the semantic FV of the environment; and providing, as output, the realistic FV associated with the semantic FV and the semantic BEV.

16. The computer-implemented method of claim 15, comprising, in the first training phase, receiving, as input, a text caption, the text caption descriptive of the semantic BEV; and encoding the text caption to obtain the latent floorplan.

17. The computer-implemented method of claim 15, comprising, in the first training phase: receiving a BEV segmentation map, representational of the semantic BEV; and encoding the BEV segmentation map to obtain the latent floorplan.

18. The computer-implemented method of claim 17, comprising, in the first training phase: receiving, as input, a text caption, the text caption descriptive of a semantic BEV; and encoding the text caption to generate the BEV segmentation map.

19. The computer-implemented method of claim 16 or claim 18, wherein the text caption is obtained by one or more of: manual user entry of a description of the environment to provide the text caption; extraction of the text caption from a description of the environment using a computer- implemented description extraction module; and procedural generation of a text caption alongside generation of the semantic BEV.

20. The computer-implemented method of any of claims 15 to 19, wherein the first training phase operates on a first set of training data, and the second training phase operates on a second set of training data, the second set of training data different from the first set of training data.

21. The computer-implemented method of any of claims 15 to 20, wherein the semantic BEV is obtained by: segmenting a point cloud representative of the environment;obtaining one or more segmented FV of the environment from the segmented point cloud, the one or more segmented FV from different viewpoints; and for the one or more segmented FV of the environment, capturing a sparse depth map of the segmented FV and a corresponding aerial view of the segmented point cloud to obtain the semantic BEV.

22. An apparatus configured to generate a realistic front view, FV, of an environment, the apparatus comprising: an input module configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic birds-eye-view, BEV, of the environment; a neural radiance field, NeRF, module configured to obtain a volumetric representation of the environment based on the latent floorplan; a volume renderer module configured to obtain a semantic FV of the environment based on the volumetric representation of the environment; a diffusion module configured to obtain a realistic FV of the environment based on the semantic FV; and an output module configured to output the realistic FV of the environment.

23. An apparatus configured to train a generative model for generation of a realistic front view, FV, of an environment from a semantic birds-eye-view, BEV, of the environment, the apparatus comprising: a first training phase module comprising: a BEV input module configured to receive, as input, a latent floorplan, the latent floor plan representational of a semantic BEV, of the environment, the latent floor plan associated with a corresponding semantic FV of the environment; a neural radiance field, NeRF, module configured to obtain a volumetric representation of the environment based on the latent floorplan using a neural radiance field, NeRF; a volume renderer module configured to obtain a semantic FV of the environment by volume rendering based on the volumetric representation of the environment; and a first association module configured to associate the obtained semantic FV of the environment with the semantic BEV of the environment; and a second training phase module comprising: a FV input module configured to receive, as input, the semantic FV; a diffusion module configured to obtain a realistic FV of the environment based on the semantic FV using a diffusion technique;a second association module configured to associate the obtained realistic FV of the environment with the semantic FV of the environment; and an output module configured to provide, as output, association data comprising the realistic FV associated with the semantic FV and the semantic BEV.

24. A machine-readable medium having program code stored thereon which, when executed by a computer, causes the computer to perform any of methods 1 to 21.

Images