A hybrid solution for efficient rendering of reconstructed 3D scenes for simulation

Abstract

Described is a computer apparatus (900) for generating a rendered frame (108), the computer apparatus (900) being configured to: obtain one or more 3D scene representations (103) corresponding to the frame, a camera pose (101) corresponding to the frame, and one or more camera characteristics (102) corresponding to the frame; partition the frame into two or more parts (203a, 203b, 203c); render a first part (203a) of the frame using a first rendering method (105a), and render a second part (203b) of the frame using a second rendering method (105b), the first rendering method (105a) being different to the second rendering method (105b); and compose the rendered frame (108) based on first rendered part (106a) of the frame and the second rendered part (106b) of the frame. In this way, more appropriate different rendering methods may be applied for the different parts of the frame.

Description

[0001] A HYBRID SOLUTION FOR EFFICIENT RENDERING OF RECONSTRUCTED 3D SCENES FOR SIMULATION

[0002] FIELD OF THE INVENTION

[0003] This invention relates to an apparatus and method for generating a rendered frame from 3D scene representations.

[0004] BACKGROUND

[0005] Realistic simulation for autonomous driving is a key component to reach a degree of reliability and security sufficient of the autonomous car controller. Simulation of new and challenging scenario needs to be fast and accurate, as close as possible to real scenario. It involves generating high quality frames in an efficient manner (ideally real-time or higher).

[0006] To generate accurate 3D simulation environment, reconstruction techniques (like NeRF or 3DGS) are used on real data to reduce as much as possible the domain gap between synthetic and real data. Such reconstruction algorithm provides high quality reconstruction but my not be as efficient as standard 3D representation (such as triangle meshes) for rendering. Moreover, one reconstruction solution may not be sufficient to faithfully represent any 3D scene and most advance simulators use a combination of 3D representation for different parts of the scene (like foreground objects, background and sky). Using more than one 3D representation in simulation may make the rendering step even more challenging, as some of the 3D representation may not be compatible with some rendering algorithms.

[0007] 3D representation of real environment. A variety of 3D representations exists which are used in reconstruction pipeline, such as:

[0008] • 3D sparse or dense point cloud: the points cloud is usually obtained using Structure from Motion (SfM) technique and Multi View Stereo (MVS) solution applied on images captured by a mobile device (like an autonomous car equipped with cameras). Such representations are versatile but most of the time incomplete as homogeneous and untextured area are hard to reconstruct from images only.

[0009] • 3D triangles mesh: the most common representation for 3D assets and in the computer graphics community. Such representation is compact and can be derived from the previous representation (point cloud) using meshing algorithms. Obtaining an accurate mesh of a complex outdoor environment is still an open challenge, especially in the context of autonomous driving when the data acquired by the vehicle only partially cover the entire scene.

[0010] • Textured mesh: by combining a 3D mesh of an area with posed images, we can obtain a textured mesh where a high-resolution texture is associated to each face of each triangle of the mesh.

[0011] • Radiance field: popularized by NeRF, a radiance field implicitly represents a 3D scene in a continuous manner where in each 3D location of the scene a colour and a density value are learned by a neural network. By accumulating the local information along rays defined by a camera using volume rendering, the final colour and geometry of the scene is generated. Radiance field are a powerful representation for Novel View Synthesis (NVS) as their model within the 3D representation some light effects such as secularity . Radiance fields remain challenging to train on large unbounded area and are costly to use for rendering.

[0012] • 3D gaussians: introduced in 2023 by the paper “3D gaussian splatting” (3DGS), the scene is described by a set of trainable 3D gaussian where each gaussian is associated to a trainable pose, color and density value. These representations are obtained following a similar training strategy as NeRF but are much more efficient to render. Nevertheless, 3DGS can generate important visual artifact is the rendered images is too far away from the training data (extrapolation issue).

[0013] 2D rendering. The following rendering techniques may be used for simulation:

[0014] • Rasterization: the process of projecting 3D primitives (like triangles from a mesh) to the 2D rendering space. Rasterization is supported by material acceleration hardware such as GPU in order to process each pixel of the rendered frame in parallel, being extremely efficient and widely use in computer graphics.

[0015] • Ray marching: the process of sampling point along a ray of the rendered frame. Usually combined with volumetric rendering, ray marching permits to render non-opaque structure and complex lightning effect.

[0016] • Ray tracing: the process of computing the intersection between a ray casted from the rendered frame with the 3D geometry of the scene. Ray tracing is widely used in simulation to render inter-object lightning interaction, shadow casting and other lightning effects.

[0017] • Neural rendering: also known as deferred shading, is not a real 3D to 2D rendering method, rather a decoding step from a 2D buffer frame obtained by one of the aforementioned methods to the final rendered frame. It can be used to increase the quality (or spatial resolution) of a rendered frame at reduced cost, or to add specific visual effect using a neural model.

[0018] In order to simulate faithfully and efficiently novel driving scenario, prior art methods rely on different solutions:

[0019] • Merging all representation in one [Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures; and Bakedsdf: Meshing neural sdfs for real-time view synthesis]: a trained radiance field that can model specular surface effect is compressed into triangles of a mesh for final rendering.

[0020] • Optimizing one representation to be fast to render [HybridNeRF : Efficient Neural Rendering via Adaptive Volumetric Surfaces; 3D Gaussian Splatting for Real-Time Radiance Field Rendering]: using novel representation (3DGS) or optimized ray marching + volumetric rendering technique, the final scene can be rendered at interactive speed (20-200 FPS).

[0021] • Combining two rendering solutions: ray tracing + marching [Adaptive shells for efficient neural radiance field rendering]: using ray tracing on a coarse mesh followed by ray marching and volumetric rendering inside the volume, less samples need to be evaluated to obtain the final rendered colour.

[0022] The aforementioned solutions may have the following limitations:

[0023] • Trade-off between efficiency and precision: method that uses a single 3D representation may suffer from either efficiency or rendering quality issue. Indeed, a complex outdoor scene may not be able to be represented faithfully with only one 3D representation.

[0024] • Scene specific solution: aforementioned solutions using combination of mesh and volumetric rendering are specific to object centric or indoor scene, which may limit their application to simulation of large outdoor urban scene.

[0025] It is desirable to develop an apparatus and method that overcomes the above problems. SUMMARY

[0026] According to a first aspect, there is provided a computer apparatus for generating a rendered frame, the computer apparatus comprising one or more processors and a memory storing in non-transient form data defining program code executable by the one or more processors, wherein the program code is executable by the one or more processors so that the computer apparatus is configured to: obtain one or more 3D scene representations corresponding to the frame, a camera pose corresponding to the frame, and one or more camera characteristics corresponding to the frame; partition the frame into two or more parts; render a first part of the frame using a first rendering method, and render a second part of the frame using a second rendering method, the first rendering method being different to the second rendering method; and compose the rendered frame based on first rendered part of the frame and the second rendered part of the frame. In this way, more appropriate different rendering methods may be applied for the different parts of the frame.

[0027] In some implementations, the apparatus may be configuration to partition the frame into two or more parts by: generating an empty buffer frame in dependence on the one or more 3D scene representations corresponding to the frame, the camera pose corresponding to the frame, and the one or more camera characteristics corresponding to the frame; and partitioning the empty buffer frame into the two or more parts of the frame. In this way, the partitioning may be applied to a frame comprising reduced information, which may be more efficient.

[0028] In some implementations, the apparatus may be configuration to partition the empty buffer frame into the two or more parts of the frame in dependence on a predefined template. In this way, the partitioning may be predefined to be more efficient.

[0029] In some implementations, the apparatus may be configuration to partition the empty buffer frame into the two or more parts of the frame by: rendering an initial rendered frame in dependence the empty buffer frame using an initial rendering method; and partitioning the initial rendered frame into the two or more parts of the frame. In this way, the partitioning may be based on an initial render, which may make the partitioning more accurate.

[0030] In some implementations, the initial rendering method may be faster per frame than one or more of the first rendering method and the second rendering method. In some implementations, the initial rendering method may be mesh rasterization. In this way, the initial rendering may take less time and so be more efficient.

[0031] In some implementations, the initial rendering method may produce lower quality rendering than one or more of the first rendering method and the second rendering method. In some implementations, the initial rendering method may be volumetric rendering. In this way, the initial rendering may take less memory and so be more efficient.

[0032] In some implementations, the apparatus may be configuration to: before generating the initial rendered frame in dependence on the empty buffer frame using the initial rendering method, downscale the empty buffer frame; rendering a downscaled initial rendered frame in dependence on the downscaled empty buffer frame; upscale the downscaled initial rendered frame; and partition the upscaled initial rendered frame into the two or more parts of the frame. In this way, the initial rendering may take less time and less memory and so be more efficient.

[0033] In some implementations, the apparatus may be configuration to partition the initial rendered frame into the two or more parts of the frame in dependence on one or more of: a semantic segmentation according to a pixel class of a temporary rendered semantic map, an occupancy mask obtained from a rasterization of a 3D mesh, and a classification according to a distance to the camera obtained by a temporary generated depth map. In this way, further inputs may be used to control the efficient partitioning. In some implementations, the apparatus may be configuration to partition the initial rendered frame into the two or more parts of the frame in dependence on one or more characteristic of one or more pixels in the respective parts of the frame. In this way, the partitioning may be more efficient for specific types of pixels.

[0034] In some implementations, the one or more characteristics of one or more pixels in the respective parts of the frame comprise one or more of: the pixel renders an area with no mesh; the pixel renders a texture from a mesh; and the pixel renders an untextured area from the mesh. In this way, specific characteristics of the pixel may be used to determine the efficient partitioning.

[0035] In some implementations, the apparatus may be configuration to select the first rendering method and the second rendering method in dependence on the one or more characteristics of one or more pixels in the respective parts of the frame. In this way, the rendering may be more efficient for specific types of pixels.

[0036] In some implementations, the apparatus may be configuration to select the first rendering method and the second rendering method such that: a pixel that renders an area with no mesh is rendered by one or more of: a radiance field rendering method, a ray marching rendering method, and a volumetric rendering method; a pixel renders the texture from the mesh is rendered by one or more of: a textured mesh rendering method, or a rasterization rendering method; and a pixel renders the untextured area from the mesh is rendered by surface rendering method. In this way, specific characteristics of the pixel may be used to determine the efficient rendering.

[0037] In some implementations, the two or more parts of the frame overlap. In this way, an edge blurring post processing may be applied to the boundaries of the parts of the frame.

[0038] In some implementations, the apparatus may be configuration to obtain a plurality of 3D scene representations corresponding to the frame, each of the plurality of 3D scene representations corresponding to the frame being different representations of the same 3D scene. In this way, different representations may be used to further improve the accuracy of the rendering.

[0039] In some implementations, the apparatus may be configuration to post-process the rendered frame. In some implementations, the post-processing may comprise one or more of: application of an image enhancing algorithm, application of a filtering algorithm, neural rendering and / or decoding, and edge blending. In this way, issues with the rendering may be removed and the rendering may be more accurate.

[0040] According to a second aspect, there is provided a method for generating a rendered frame, carried out by a computer apparatus, comprising the steps of: obtaining one or more 3D scene representations corresponding to the frame, a camera pose corresponding to the frame, and one or more camera characteristics corresponding to the frame; partitioning the frame into two or more parts; rendering a first part of the frame using a first rendering method, and rendering a second part of the frame using a second rendering method, the first rendering method being different to the second rendering method; and composing the rendered frame based on first rendered part of the frame and the second rendered part of the frame. In this way, more appropriate different rendering methods may be applied for the different parts of the frame.

[0041] BRIEF DESCRIPTION OF THE FIGURES

[0042] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0043] Figure 1 is a schematic illustration of the procedure of the present system. Figure 2 is a schematic illustration of a partition part of the procedure of the present system.

[0044] Figure 3 is a schematic illustration of a first embodiment of the partition part of the procedure of the present system.

[0045] Figure 4 is a schematic illustration of a second embodiment of the partition part of the procedure of the present system.

[0046] Figure 5 is a schematic illustration of a third embodiment of the partition part of the procedure of the present system.

[0047] Figure 6 is a schematic illustration of a rendering and composition part of the procedure of the present system.

[0048] Figure 7 is a schematic illustration of a first specific implementation of the procedure of the present system.

[0049] Figure 8 shows an example of the computer implemented method of the procedure of the present system.

[0050] Figure 9 shows an example of a computer apparatus configured to perform the method described herein.

[0051] DETAILED DESCRIPTION

[0052] The apparatus and method described herein concern generating a rendered frame from 3D scene representations.

[0053] Embodiments of the present invention may tackle one or more of the problems previously mentioned by: rendering a first part of the frame using a first rendering method, and rendering a second part of the frame using a second rendering method, the first rendering method being different to the second rendering method. In this way, more appropriate different rendering methods may be applied for the different parts of the frame.

[0054] The following terms are used in present disclosure:

[0055] Rendering (2D rendering) - the process of creating a 2D representation of a 3D environment from a specific location and point of view.

[0056] Pose - 6 DoF position and rotation of a camera or object or scene

[0057] Frame - 2D grid, where on each position on the grid a punctual information is stored (such as colour for an image).

[0058] Neural Radiance Field (NeRF) - method based on deep learning for reconstructing a three-dimensional representation of a scene from two-dimensional images.

[0059] 3D Gaussian Splatting (3DGS) - a volume rendering technique that deals with the direct rendering of volume data without converting the data into surface or line primitives.

[0060] Degree of Freedom (DoF).

[0061] Frames per second (FPS).

[0062] Structure from Motion (SIM) - the process of estimating the 3-D structure of a scene from a set of 2-D images.

[0063] Multi- view stereo (MVS) - the task of reconstructing a 3D shape from calibrated overlapping images captured from different viewpoints.

[0064] The present system aims to provide the best rendering quality of a reconstructed 3D environment at a rendering speed compatible with simulation. The present system arms to provide the optimal trade-off between rendering quality and rendering speed.

[0065] The present system may have the following solutions and corresponding advantages: • The present system may use a mixture of 3D scene representations and rendering techniques - Based on the assumption that a complex 3D scene cannot be efficiently and accurately represented by a single 3D representation, this solution allows for high quality rendering at high framerates by combining multiple 3D representations and rendering techniques.

[0066] • The present system may partition the render frame based on predefined criteria - This partitioning may optimise the quality and efficiency of each partitioned part of the final frame.

[0067] • The present system may refine the composed frame after rendering - Frame post-processing may remove rendering / merging artifacts and may improve the overall quality of the rendering.

[0068] The present system may be divided in three main parts: pixel partitioning, parallel rendering and final frame composition.

[0069] The present system may render 2D frames (image, depth map, semantic map or other modalities) of a 3D scene described by a mixture of 3D representations by using different rendering methods in parallel. The frame may be divided at the pixel level and each partition may be rendered using a combination of 3D representations and Tenderers, before the final composition and optional post-processing. An overview of the procedure 100 of the present system is presented in Figure 1. The procedure 100 may be implemented by the computer apparatus 900 shown in Figure 9.

[0070] Inputs

[0071] One or more inputs 101, 102, 103 may be obtained. The apparatus 900 may receive the inputs 101, 102, 103. Given the one or more inputs 101, 102, 103, a rendered frame 108 of one or multiples modalities (colour, semantic, depth, etc . ) may be outputted. The inputs 101, 102, 103 are shown in Figures 1, 2, 4, 5 and 7.

[0072] The input may comprise one or more 3D scene representations 103 corresponding to a frame. The 3D scene representations 103 may represent a 3D scene from which the 2D frame may be generated. The 3D representations 103 may be of a given environment. The 3D representations 103 may be of a given environment that can have been reconstructed by different reconstruction algorithm from various observations. There may be a plurality of 3D scene representations 103 obtained. The plurality of 3D scene representations 103 may be different representations 103 of the same scene. The different representations 103 may be generated through different methods. The 3D representations 103 may be a set of aligned 3D representations 103. Given the set of 3D representations 103 the rendered frame 108 may be outputted.

[0073] The input may comprise a camera pose 101 corresponding to a frame. The camera pose 101 may be the position, alignment and direction of the camera capturing a 3D scene from which the 2D frame may be generated. The camera pose 101 may be the extrinsic parameters of the camera. Given the camera pose 101 the rendered frame 108 may be outputted.

[0074] The input may comprise one or more camera characteristics 102. corresponding to a frame. The camera characteristics 102 may be the characteristics of the camera capturing a 3D scene from which the 2D frame may be generated. The camera characteristics 102 may be the intrinsic parameters of the camera. The characteristics 102 may include focal distance, optical centre, height / width of the frame and optional distortion parameters. There may be a plurality of camera characteristics 102 obtained. The plurality of camera characteristics 102 may be different camera characteristics 102 of the same camera capturing the same scene. Given the camera characteristics 102 the rendered frame 108 may be outputted. Pixel Partitioning

[0075] The frame may be partitioned into two or more parts 202a, 203b, 203c. The apparatus 900 may partition the frame into the parts 202a, 203b, 203c. The parts 202a, 203b, 203c are shown in Figures 2, 3, 4 and 7. Figure 2 shows an overview of the partitioning part of the procedure 100.

[0076] The parts 202a, 203b, 203c of the frame may be distinct. The parts 202a, 203b, 203c of the frame may not overlap. Alternatively, the parts 202a, 203b, 203c of the frame may overlap. Some of the parts 202a, 203b, 203c of the frame may overlap and some may not.

[0077] The pixel partitioning 104 may comprise two steps. The pixel partitioning 104 may comprise generating an empty buffer frame 202 and partitioning 104 the empty buffer frame 202. The empty buffer frame 202 partitioning 104 is shown in Figures 2, 3, 4 and 5.

[0078] The empty buffer frame 202 may be generated in dependence on the one or more 3D scene representations 103, the camera pose 101 and the camera characteristics 102. The empty buffer frame 202 may be an empty buffer for the final frame 108, 110. The empty buffer frame 202 may be an empty buffer for the final frame 108, 110 from the camera pose 101 and camera parameters 102 and computing the ray parameters for each pixel (ray origin and direction).

[0079] The empty buffer frame 202 may be partitioned 104 into the two of more parts 203a, 203b, 203c. The empty buffer frame 202 may be partitioned at a pixel level according to a predefined routine, in a set of N partitioned frames.

[0080] While the generation of the empty frame buffer 202 and the corresponding rays may be achieved relatively easily given the camera pose 101 and intrinsic parameters 102, the partitioning step 104 may not be straightforward. The partitioning may be achieved in different ways:

[0081] Figure 3 is a schematic illustration of a first embodiment of the partition part of the procedure 100. The empty buffer frame 202 may be partitioned 104 into the parts 203a, 203b, 203c in dependent on a template 301. The template 301 may be predefined. The template 301 may be set before the partitioning 104 to partition 104 certain regions of the frame. The template 301 may be set so as to split the frame into different regions based on what is expected to be in those regions. A simple manner to split the empty frame 202 is to define a fixed partitioning of the frame that is independent of the camera pose. For instance, considering a virtual reality application, the rendered frame can be divided in two parts: the centre 203a of the image and its periphery 203b. It is possible to use different rendering method for these two parts 203a, 203b, for instance a high-quality rendering method for the central part 203a and a faster and less precise rendering for the periphery 203b as it may not be the area of interest for the user.

[0082] Figures 4 and 5 are a schematic illustration of second and third embodiments of the partition part of the procedure 100. These embodiments may rely on an initial rendered frame 402 for partitioning 104.

[0083] Segmentation of a temporary fast rendered frame 402 at the camera location may be used. To adapt the rendering methods to the content of the scene, an initial rendering 401, 503 may be performed. To be efficient enough, the temporary frame 402 may need to be generate using a fast rendering solution. Fast rendering method can be a rendering solution that is fast by design (like mesh rasterization) 401 (see Figure 4), a rendering method that is executed in a low quality / fast inference mode (like a volumetric rendering solution but with a low number of samples along the ray) 401 (see Figure 4), or using a downscaled version of the temporary frame 503 (see Figure 5). Then, the temporary rendered frame 402 may be segmented 403, 404, 405 according to some criteria to obtain the final partitioned frame 203.

[0084] Figure 4 is a schematic illustration of a second embodiment of the partition part of the procedure 100.

[0085] An initial rendered frame 402 may be rendered 401 in dependence on the empty buffer frame 202. The empty buffer frame 202 may be rendered 401 into the initial rendered frame 402. The initial rendered frame 402 may be rendered 401 using an initial rendering method 401.

[0086] The initial rendering method 401 may be faster per frame than one or more of the first rendering method 105a and the second rendering method 105b. The initial rendering method 401 may be faster per frame than one or more of the rendering methods 105a, 105b, 105c used for the rendered frame 108. The initial rendering method 401 may be faster per frame than all the rendering methods 105a, 105b, 105c used for the rendered frame 108. The initial rendering method 401 may be mesh rasterization.

[0087] Alternatively, or in addition, the initial rendering method 401 may produce lower quality rendering than one or more of the first rendering method 105a and the second rendering method 105b. The initial rendering method 401 may produce lower quality rendering than one or more of the rendering methods 105a, 105b, 105c used for the rendered frame 108. The initial rendering method 401 may produce lower quality rendering than all the rendering methods 105a, 105b, 105c used for the rendered frame 108. The initial rendering method 401 may be volumetric rendering.

[0088] Figure 5 is a schematic illustration of a third embodiment of the partition part of the procedure 100.

[0089] Before generating the initial rendered frame 402 in dependence on the empty buffer frame 202 using the initial rendering method 503, the empty buffer frame 202 may be downscaled 501. The downscaling 501 may produce a downscaled empty buffer frame 502. The downscaled empty buffer frame 502 may comprise a lower pixel density than the empty buffer frame 402.

[0090] A downscaled initial rendered frame 504 may be generated in dependence on the downscaled empty buffer frame 502. The downscaled empty buffer frame 502 may be rendered 503 to generate the downscaled initial rendered frame 504. An initial rendering method 401 as described herein may provide the rendering 503. The downscaled initial rendered frame 504 may comprise a lower pixel density than if the downscaling was not used (e.g., the temporary frame 402 of the embodiment in Figure 4).

[0091] An upscaled initial rendered frame 402 may be generated in dependence on the downscaled initial rendered frame 504. The downscaled initial rendered frame 504 may be upscaled 505 to generate the upscaled initial rendered frame 402. The upscaled initial rendered frame 402 may provide the initial rendered frame 402.

[0092] The initial rendered frame 402 (either from Figure 4, or the upscaled initial rendered frame 402 from Figure 5) may be partitioned 104 into the two or more parts of the frame 203a, 203b, 203c.

[0093] The initial rendered frame 402 may be partitioned into the two or more parts 203a, 203b, 203c of the frame in dependence on one or more characteristic of one or more pixels in the respective parts of the frame. For example, depending on the characteristics of each pixel, it may be partitioned into a different part 203a, 203b, 203c. Pixels with common characteristics may be grouped into different parts 203a, 203b, 203c. The characteristics of one or more pixels in the respective parts 203a, 203b, 203c of the frame comprise one or more of: the pixel renders an area with no mesh; the pixel renders a texture from a mesh; the pixel renders an untextured area from the mesh. The rendering method 106a, 106b 106c may be selected based on the characteristics of the pixels in the part 203a, 203b, 203c of the frame.

[0094] The partitioning 104 may comprise steps 403, 404 and 405 as shown in Figure 4. The initial rendered frame 402 may be segmented 403. The initial rendered frame 402 may be segmented 404 to form a segmented frame 404. The segmented frame 404 may be pixel assigned 405. The segmented frame 404 may be pixel assigned 405 into the two or more parts 203a, 203b, 203c of the frame.

[0095] A possible implementation of the segmentation 403 could be: semantic segmentation 403 according to the pixel class of a temporary rendered semantic map; an occupancy mask obtained from the rasterization of a 3D mesh; and / or a classification according to the distance to the camera obtained by a temporary generated depth map.

[0096] Parallel Rendering

[0097] The apparatus 900 may render 105 the parts 202a, 203b, 203c of the frame. The rendered parts 106a, 106b, 106c are shown in Figures 1 and 6. Figure 6 shows an overview of the rendering and composition part of the procedure 100.

[0098] Once the empty frame 202 has been partitioned into N partial frames 202a, 203b, 203c, each of the partial frames 202a, 203b, 203c may be rendered using a different rendering method 105a, 105b, 105c in parallel. Each rendering method 105a, 105b, 105c may use one or more 3D representations from the set of representations of the solution. The rendering 105 may be executed in any available hardware component (GPU, CPU, etc.) or even on a remote device.

[0099] A first part 203a of the frame may be rendered using a first rendering method 105a. A second part 203b of the frame may be rendered using a second rendering method 105b. The first rendering method 105a being different to the second rendering method 105b. The first rendering method 105a and the second rendering method 105b may carry out the rendering in different ways. The first rendering method 105a and the second rendering method 105b may involve different steps. The first rendering method 105a and the second rendering method 105b may take different amount of time to execute. The first rendering method 105a and the second rendering method 105b may have a different speed per frame. The first rendering method 105a and the second rendering method 105b may use different inputs. The first rendering method 105a and the second rendering method 105b may produce different outputs.

[0100] Further rendering methods 105c may also be used. Further rendering methods 105c may be used for further parts 203c of the frame. At least two of the rendering methods 105a, 105b, 105c may be different. The remaining methods 105c may be the same as one of the first and second methods 105a, 105b. Alternatively, each of the parts 202a, 203b, 203c of the frame may use different rendering methods 105a, 105b, 105c. There may be the same number of methods 105a, 105b, 105c as the number of parts 202a, 203b, 203c of the frame. The number of rendering methods 105a, 105b, 105c may be varied depending on the requirements for the procedure 100.

[0101] The rendering methods 105a, 105b, 105c may be selected in dependence on one or more characteristics of one or more pixels in the respective parts 202a, 203b, 203c of the frame. In particular, the first rendering method 105a and the second rendering method 105b may be selected in dependence on one or more characteristics of one or more pixels in the respective parts 202a, 203b of the frame. The characteristics of the pixels may be as described herein. Optimal rendering methods 105a, 105b, 105c for the specific characteristics of the pixels may be selected. This may provide faster and / or more accurate rendering. The following are examples of a method 105a, 105b, 105c selected for a specific characteristic of the pixels.

[0102] • A pixel that renders an area with no mesh may be rendered by one or more of: a radiance field rendering method, a ray marching rendering method, and a volumetric rendering method - a more efficient method may be chosen for the lower complexity pixel that renders an area with no mesh;

[0103] • A pixel renders the texture from the mesh is rendered by one or more of: a textured mesh rendering method, or a rasterization rendering method - a more efficient and more accurate method may be chosen for the medium complexity pixel that renders the texture from the mesh; and

[0104] • A pixel renders the untextured area from the mesh is rendered by surface rendering method - a more accurate method may be chosen for the higher complexity pixel that renders the untextured area from the mesh.

[0105] Final Frame Composition

[0106] The apparatus 900 may compose 107 the rendered parts 106a, 106b, 106c of the frame into a composed frame 108. The composed frame 108 is shown in Figures 1 and 6. Figure 6 shows an overview of the rendering and composition part of the procedure 100.

[0107] The rendered frame 108, also known as the composed frame 108, may be based on any and all of the rendered parts 106a, 106b, 106c of the frame. In particular, the rendered frame 108 may be based on first rendered part 106a of the frame and the second rendered part 106b of the frame, when two parts 106a, 106b are rendered.

[0108] Once each partial frame 203 has been rendered, the rendered parts 106a, 106b, 106c are composed back using the partition from the initial step. Put another way, the partitioning 104 is used to re-compose 107 the frame.

[0109] Because of the different rendering methods and potential different 3D representation used during rendering, the rendered frame 108 may exhibit artifact or discontinuity. To improve the final rendering results, an optional post-processing step 109 may be applied to the rendered frame 108. The post-processing step 109 may produce a final frame 110. If post-processing 109 is not used, then the rendered frame 108 may provide the final frame 110.

[0110] The post-processing 109 may comprise one or more of: application of an image enhancing algorithm, application of a filtering algorithm, neural rendering and / or decoding, and edge blending. In the case of edge blending, one or more of the parts 203a, 203b, 203c may overlap. In this way, the overlapping edges may be blended to form a higher quality frame.

[0111] Specific Implementations

[0112] Figure 7 is a schematic illustration of a first specific implementation of the procedure 100 of the present system:

[0113] The apparatus 900 may obtain two 3D representations 103:

[0114] • A textured mesh 701 , which may cover close-range elements of the scene and with some triangles untextured (due to occlusion or bad geometry); and

[0115] • A continuous radiance field 702, which may cover the whole scene (including the sky).

[0116] The initial step of frame partitioning 104 may be implemented as follows:

[0117] • The temporary frame may be generated using the textured mesh 701 with rasterization 401 at the given camera pose 101. Such operation may be executed efficiently on GPU (~400 FPS on our not-optimized implementation). • The temporary frame 402 may be divided into three partial frames 203a, 203b, 203c according to the following criteria:

[0118] 1. The pixel is rendering an area with no mesh 203a (the ray casted from the pixel does not intersect the mesh);

[0119] 2. The pixel is rendering a texture from the mesh 203b; and

[0120] 3. The pixel is rendering a untextured are of the mesh 203c.

[0121] The three partial frames 203a, 203b, 203c may be rendered in parallel using the corresponding rendering technique:

[0122] 1. Using the radiance field, ray marching and volumetric rendering 105a for the part 203a,

[0123] 2. Using the textured mesh and rendered with rasterization 105b for the part 203b,

[0124] 3. Using the radiance field with a single network query at the surface of the mesh (surface rendering) 105c for the part 203c.

[0125] The three partial frames 106a, 106b, 106c may be composed 107 into the final image 108, 110.

[0126] The first implementation may render a frame of a complex outdoor environment at 20 FPS (vs <1 FPS for a pure ray marching + volumetric rendering using radiance field) thanks to the limited usage of the radiance field while maintaining high quality rendering on textured area.

[0127] The following is a description of a second specific implementation of the procedure 100 of the present system.

[0128] A key principle of the present system is the adaptive frame partitioning. As such, the initial step of frame partitioning 104 may be implemented one the following techniques:

[0129] • Semantic-based frame partitioning: using a fast-to-render 3D semantic representation of the scene, the frame partitioning may be achieved using the semantic class of the rendered elements.

[0130] • Distance-based frame partitioning: using the depth map rendered by a fast-to-render 3D representation, the partitioning can be done according to the distance to the camera.

[0131] Figure 8 summarises an example of a method 800. At step 801, the method 800 comprises obtaining one or more 3D scene representations corresponding to the frame, a camera pose corresponding to the frame, and one or more camera characteristics corresponding to the frame. At step 802, the method 800 comprises partitioning the frame into two or more parts. At step 803, the method 800 comprises rendering a first part of the frame using a first rendering method, and rendering a second part of the frame using a second rendering method, the first rendering method being different to the second rendering method. At step 804, the method 800 comprises composing the rendered frame based on first rendered part of the frame and the second rendered part of the frame.

[0132] An example of computer apparatus 900 configured to implement the method 800 is schematically illustrated in Figure 9. The computer apparatus 900 may carry out the procedures 100 illustrated in Figures 1 to 7. The computer apparatus 900 may be implemented on an electronic device, such as a computer, a computer for a vehicle, laptop, tablet, or smart phone. The computer apparatus 900 may be connected to the internet.

[0133] The computer apparatus 900 comprises a processor 901 configured to process the datasets in the manner described herein. For example, the processor 901 may be implemented as a computer program running on a programmable device such as a Central Processing Unit (CPU). The computer apparatus 900 comprises a memory 902 which is arranged to communicate with the processor 901. Memory 902 may be a non-volatile memory. The processor 901 may also comprise a cache (not shown in Figure 9), which may be used to temporarily store data from memory 902. The computer apparatus 900 may comprise more than one processor 901 and more than one memory 902. The memory 902 may store data that is executable by the processor 901. The processor 901 may be configured to operate in accordance with a computer program stored in non-transitory form on a machine- readable storage medium. The computer program may store instructions for causing the processor to perform its methods in 5 the manner described herein. The method steps described herein may be carried out by a computer-readable storage medium.

[0134] The method steps described herein may be carried out by a computer program product.

[0135] The applicant hereby discloses in isolation each individual feature described herein and any combination of two or more such features, to the extent that such features or combinations are capable of being carried out based on the present specification as0 a whole in the light of the common general knowledge of a person skilled in the art, irrespective of whether such features or combinations of features solve any problems disclosed herein, and without limitation to the scope of the claims. The applicant indicates that aspects of the present invention may consist of any such individual feature or combination of features. In view of the foregoing description, it will be evident to a person skilled in the art that various modifications may be made within the scope of the invention.

Claims

CLAIMS1. A computer apparatus (900) for generating a rendered frame (108), the computer apparatus (900) comprising one or more processors (901) and a memory (902) storing in non-transient form data defining program code executable by the one or more processors (901), wherein the program code is executable by the one or more processors (901) so that the computer apparatus (900) is configured to: obtain one or more 3D scene representations (103) corresponding to the frame, a camera pose (101) corresponding to the frame, and one or more camera characteristics (102) corresponding to the frame; partition the frame into two or more parts (203a, 203b, 203c); render a first part (203a) of the frame using a first rendering method (105a), and render a second part (203b) of the frame using a second rendering method(105b), the first rendering method (105a) being differentto the second rendering method (105b); and compose the rendered frame (108) based on first rendered part (106a) of the frame and the second rendered part (106b) of the frame.

2. A computing apparatus (900) according to claim 1, wherein the apparatus (900) is configuration to partition the frame into two or more parts (203a, 203b, 203c) by: generating an empty buffer frame (202) in dependence on the one or more 3D scene representations (103) corresponding to the frame, the camera pose (101) corresponding to the frame, and the one or more camera characteristics (102) corresponding to the frame; and partitioning the empty buffer frame into the two or more parts of the frame.

3. A computing apparatus (900) according to claim 2, wherein the apparatus (900) is configured to partition the empty buffer frame (202) into the two or more parts (203a, 203b, 203c) of the frame in dependence on a predefined template (301).

4. A computing apparatus (900) according to claim 2 or 3, wherein the apparatus (900) is configured to partition the empty buffer frame into the two or more parts (203a, 203b, 203c) of the frame by: rendering an initial rendered frame (402) in dependence the empty buffer frame (202) using an initial rendering method (401, 503); and partitioning the initial rendered frame (402) into the two or more parts (203a, 203b, 203c) of the frame.

5. A computing apparatus (900) according to claim 4, wherein: the initial rendering method (401) is faster per frame than one or more of the first rendering method (105a) and the second rendering method (105b), and optionally wherein the initial rendering method (401) is mesh rasterization; and / or wherein the initial rendering method (401) produces lower quality rendering than one or more of the first rendering method (105a) and the second rendering method (105b), and optionally wherein the initial rendering method (401) is volumetric rendering.

6. A computing apparatus (900) according to claim 4 or 5, wherein the apparatus (900) is further configured to: before generating the initial rendered frame (402) in dependence on the empty buffer frame (202) using the initial rendering method (503), downscale the empty buffer frame (202); render a downscaled initial rendered frame (504) in dependence on the downscaled empty buffer frame (502); upscale the downscaled initial rendered frame (504); and partition the upscaled initial rendered frame (402) into the two or more parts of the frame.

7. A computing apparatus (900) according to any of claims 4 to 6, wherein the apparatus (900) is configured to partition the initial rendered frame (402) into the two or more parts (203a, 203b, 203c) of the frame in dependence on one or more of: a semantic segmentation according to a pixel class of a temporary rendered semantic map, an occupancy mask obtained from a rasterization of a 3D mesh, and a classification according to a distance to the camera obtained by a temporary generated depth map.

8. A computing apparatus (900) according to any of claims 4 to 7, wherein the apparatus (900) is configured to partition the initial rendered frame (402) into the two or more parts (203a, 203b, 203c) of the frame in dependence on one or more characteristic of one or more pixels in the respective parts (203a, 203b, 203c) of the frame.

9. A computing apparatus (900) according to claim 8, wherein the one or more characteristics of one or more pixels in the respective parts (203a, 203b, 203c) of the frame comprise one or more of: the pixel renders an area with no mesh; the pixel renders a texture from a mesh; and the pixel renders an untextured area from the mesh.

10. A computing apparatus (900) according to claim 8 or 9, wherein the apparatus (900) is configured to select the first rendering method (105a) and the second rendering method (105b) in dependence on the one or more characteristics of one or more pixels in the respective parts (203a, 203b, 203c) of the frame.

11. A computing apparatus (900) according to claim 10, wherein the apparatus (900) is configured to select the first rendering method (105a) and the second rendering method (105b) such that: a pixel that renders an area with no mesh is rendered by one or more of: a radiance field rendering method, a ray marching rendering method, and a volumetric rendering method; a pixel renders the texture from the mesh is rendered by one or more of: a textured mesh rendering method, or a rasterization rendering method; and a pixel renders the untextured area from the mesh is rendered by surface rendering method.

12. A computing apparatus (900) according to any preceding claim, wherein two or more parts (203a, 203b, 203c) of the frame overlap.

13. A computing apparatus (900) according to any preceding claim, wherein the apparatus (900) is configured to obtain a plurality of 3D scene representations (103) corresponding to the frame, each of the plurality of 3D scene representations (103) corresponding to the frame being different representations (103) of the same 3D scene.

14. A computing apparatus (900) according to any preceding claim, wherein the apparatus (900) is configured to postprocess (109) the rendered frame (108), and optionally wherein the post-processing (109) comprises one or more of: application of an image enhancing algorithm, application of a filtering algorithm, neural rendering and / or decoding, and edge blending.

15. A method (800) for generating a rendered frame, carried out by a computer apparatus (800), comprising the steps of: obtaining one or more 3D scene representations corresponding to the frame, a camera pose corresponding to the frame, and one or more camera characteristics corresponding to the frame (801); partitioning the frame into two or more parts (802); rendering a first part of the frame using a first rendering method, and rendering a second part of the frame using a second rendering method, the first rendering method being different to the second rendering method (803); andcomposing the rendered frame based on first rendered part of the frame and the second rendered part of the frame(804).15

Images