Apparatus and method for generating light intensity images

Abstract

An apparatus includes a receiver (201, 203) that receives a texture map and a mesh representing a scene from first and second viewpoints. An image generator (205) determines a light intensity image for a third viewpoint based on the received data. A first view converter (207) determines first image positions and depth values ​​in the image for vertices of the first mesh, while a second view converter (209) determines second image positions and depth values ​​for vertices of the second mesh. A first shader (211) determines first light intensity values ​​and first depth values ​​based on the first image positions and depth values, and a second shader (213) determines second light intensity values ​​and second depth values ​​based on the second image positions. A combiner (215) generates an output value as a weighted combination of the first and second light intensity values, wherein the weighting of the light intensity values ​​increases for increasing depth values.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for generating a light intensity image, and in particular to generating a light intensity image for a new viewpoint suitable for accelerated hardware processing based on texture maps and meshes from multiple different viewpoints. Background Art

[0002] Graphics applications based on three-dimensional scenes have become commonplace in many applications, particularly computer graphics. To support fast three-dimensional graphics processing, numerous standards and specifications have been developed. This not only allows for faster design and implementation, as it provides standardized functions and routines for many standard operations (such as viewpoint shifting), but also allows for the development and optimization of specialized hardware graphics engines for these routines. In fact, for many computers, the graphics processing unit (GPU) is now often at least as powerful and important as the central processing unit (CPU).

[0003] An example of a standard for supporting fast graphics processing is the OpenGL specification, which provides an application programming interface (API) with several functions that support graphics processing. This specification is typically used to provide hardware-accelerated graphics processing, where specific routines are implemented by dedicated acceleration hardware in the form of a GPU.

[0004] In most such graphics specifications, the representation of a scene is done by combining multiple texture maps with associated three-dimensional meshes. In fact, in many cases, a particularly efficient approach is to represent image objects, or indeed entire scenes, by polygonal meshes, where a set of polygons are connected by their common edges or angles (vertices), which are given by their three-dimensional positions. The combined three-dimensional polygonal mesh thus provides an efficient model of a three-dimensional object, including possibly a three-dimensional description of the entire image. The polygonal mesh is typically a triangular mesh formed by triangles having common angles given in 3D space.

[0005] As an example, a stereo camera may record an image of a scene from a given viewpoint. For each pixel, a disparity estimation may be performed to estimate the distance to the object represented by the pixel. This may be performed for each pixel, providing a three-dimensional position of x, y, z for each pixel. These positions may then be used as vertices for a triangular (or other element) mesh, where two triangles are formed for each group of 2×2 pixels. Since this may result in a large number of triangles, the process may include combining some of the initial triangles into larger triangles (or more generally into larger polygons in some scenarios). This will reduce the number of triangles, but will also reduce the spatial resolution of the mesh. Therefore, it is typically dependent on depth variations and is performed primarily in flatter areas.

[0006] Each vertex is also associated with a light intensity value from a texture map. The texture map essentially provides the light / color intensity in the scene for the object at the vertex's pixel location. Typically, a light intensity image / texture map is provided along with the mesh, with each vertex containing data representing the vertex's x,y,z position and u,v data identifying the location of a link in the texture map, i.e., it points to the light intensity at that x,y,z position captured in the texture map.

[0007] In such representations, polygon meshes are used to provide information about the three-dimensional geometry of an object, while textures are usually provided as separate data structures. Specifically, textures are usually provided as separate two-dimensional images that can be superimposed on the three-dimensional geometry through processing algorithms.

[0008] The use of triangular meshes is particularly well suited for processing and manipulation by computer graphics algorithms, and many effective software and hardware solutions have been developed and are commercially available. Significant computational efficiencies have been achieved in many systems by processing individual vertices collectively for multiple polygons rather than processing each polygon separately. For example, for a typical triangular mesh, a single vertex is often a common point for several (usually 3-8) triangles. Therefore, processing a single vertex can be applied to a relatively large number of triangles, substantially reducing the number of points in the image or other object being processed.

[0009] As a specific example, many current systems on a chip (SoCs) include a GPU that is highly optimized for processing 3D graphics. For example, the processing of 3D object geometry and 3D object textures is done using two largely separate paths in the so-called OpenGL rendering pipeline (or many other APIs, such as DirectX). 3D sources are presented to the GPU in the form of vertices (usually triangles) and textures, and the hardware of the GPU on the SoC can then process the 3D graphics efficiently. The OpenGL application interface then allows for the setup and control of a virtual perspective camera, which determines how the 3D objects are projected onto a 2D screen. Although OpenGL uses 3D objects as input, the output is typically a 2D image suitable for a normal 2D display.

[0010] However, this approach requires three-dimensional information to be provided by polygonal meshes and associated texture information. While this may be relatively easy to provide in some applications, such as games based on completely computer-generated virtual scenes and environments, it may be less so in other implementations. In particular, applications based on capturing real-world scenes need to convert these into texture and mesh representations. As mentioned earlier, this can be based on stereo images or scene-based image and depth representations. However, while many methods are known for performing this conversion, it is not straightforward and presents several complex issues and challenges.

[0011] A common operation in graphics processing is a viewpoint change, where an image is generated for a viewpoint that is different from the viewpoint of the input texture map and mesh. Graphics APIs usually have functions that perform such viewpoint conversions very efficiently. However, since the input mesh is usually not perfect, such a viewpoint transformation may cause the quality of the shift to be too significant. In addition, the representation of the scene from the viewpoint will usually include multiple occluded elements, where foreground objects occlude elements behind them. These elements may be visible from the new direction, i.e., the viewpoint change can cause de-occlusion. However, for the estimated depth, in this case, the input texture map and mesh will not include any information for these de-occluded parts. Therefore, since the required information is not available, it cannot be optimally represented.

[0012] For these reasons, viewpoint transformations are often based on multiple texture maps and meshes corresponding to different view directions. In fact, to synthesize new (unseen) viewpoints, it is often preferable or even necessary to combine multiple captured meshes from different viewpoints with associated camera images (textures). The main reason for combining data from different viewpoints is to recover objects that are hidden (occluded) in one view but visible in another. This problem is often called viewpoint interpolation.

[0013] However, traditional approaches still tend to be suboptimal.

[0014] For example, one method for generating new viewpoints is to transform meshes originating from different viewpoints into a single global coordinate system and then perform a perspective projection onto the new camera plane. These steps can be done in standard graphics hardware. However, this often does not correctly display hidden surfaces. Specifically, when points are combined at a single pixel, the graphics hardware uses a depth test to select the frontmost point. This method is used to address self-occlusion, where viewpoint shifting can cause image objects to move relative to each other, causing new occlusions to occur, i.e., at the new viewpoint, there may be occlusions for two points that were not occluded at the original viewpoint. However, this can lead to errors or degradation when applied to different images. In practice, depth is often linearly interpolated so that it extends beyond foreground objects (like a halo effect), and the frontmost point often corresponds to an area that may be occluded due to being in close proximity to foreground objects.

[0015] An example of a technique for view interpolation based on depth images is provided in CL Zitnick et al. "High-quality video view interpolation using a layered representation". SIGGRAPH'04 ACM SIGGRAPH 2004, pages 600-608. To achieve high quality, the technique uses a two-layer representation consisting of a main layer and a boundary layer (around depth transitions). These are built using alpha matting (to account for transparency) and both are warped (and blended with other views) during rendering. A disadvantage of this approach is that the mesh needs to be disconnected to generate a two-layer representation. This process requires selecting a threshold for the depth map and erasing triangles of the corresponding mesh at depth discontinuities. This is undesirable because using a threshold can potentially reduce temporal stability in rendering.

[0016] WO2018104102A1 provides an example of a specific method for view interpolation based on a grid and texture map representation. This method is based on combining light intensity information from different viewpoints based on the gradients in the corresponding grids. However, while this method can provide advantageous performance in many embodiments, it is often not optimal in all scenarios and often requires relatively complex operations and high computational resources.

[0017] Hence, an improved method for generating images for different viewpoints would be advantageous, and in particular a method allowing increased flexibility, increased accuracy, reduced complexity, improved computational efficiency, improved compatibility with existing graphics processing methods, improved image quality, improved de-occlusion performance and / or improved performance would be advantageous. Summary of the Invention

[0018] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0019] According to one aspect of the present invention, a method for generating a light intensity image is provided, the method comprising: receiving a first texture map and a first mesh representing a scene from a first viewpoint; receiving a second texture map and a second mesh representing the scene from a second viewpoint; determining a light intensity image representing the scene from a third viewpoint in response to the first texture map, the first mesh, the second texture map, and the second mesh; wherein determining the light intensity image comprises: determining a first image position and a depth value of the first image position corresponding to a vertex of the first mesh in the light intensity image; determining a second image position and a depth value of the second image position corresponding to a vertex of the second mesh in the light intensity image. ; and for each first position of a plurality of positions in the light intensity image: determining a first light intensity value based on the first texture map and the first image position, and determining a first depth value based on the first image position and the first image position depth value; determining a second light intensity value based on the second texture map and the second image position, and determining a second depth value based on the second image position and the second image position depth value; determining the light intensity value at the first position for the light intensity image as a weighted combination of at least the first light intensity value and the second light intensity value; the weighting of the first light intensity value relative to the second light intensity value increases for increasing first depth values.

[0020] In many embodiments, the present invention can provide for the generation of an improved light intensity image for a given viewpoint based on a texture map and meshes from other viewpoints. In many scenarios, this results in an improved representation of areas that are deoccluded relative to one input viewpoint but not to another. This approach can particularly provide improved quality around foreground object edges in many embodiments.

[0021] This method may avoid the generation of grid holes known from other methods and may provide improved depth consistency.

[0022] A particular advantage of the method is that in many embodiments it can be closely supported by standard graphics routines. In many embodiments, many individual operations can be performed by standardized, dedicated and optimized graphics processing hardware. For example, the method can be compatible with standardized graphics processes and can effectively utilize these processes. In fact, the method can be compatible with such standardized methods (such as the OpenGL specification) by limiting non-standard methods to the parts of the rendering / processing pipeline that are allowed to be adjusted by the user. For example, the determination of the first and second light intensity values ​​and / or combinations can be performed as part of, for example, the fragment shader stage of an OpenGL rendering pipeline. The determination of the first / second image position and the corresponding depth value can be performed by a vertex shader, such as the vertex shader of an OpenGL rendering pipeline.

[0023] Furthermore, the method does not require any specialized determination of specific values ​​such as depth gradients and warping / viewpoint shifting, and thus may be suitable for implementations with limited computational resources. For example, the method may be particularly suitable for implementation in portable devices such as smartphones.

[0024] The possibility of using standardized hardware accelerated processing for resource-intensive elements of processing can significantly increase, for example, the speed or performance of the system. In many embodiments, it can reduce complexity and / or resource requirements.

[0025] In some embodiments, the combination may include additional light intensity values ​​derived from the texture map and meshes for other viewpoints.

[0026] The mesh can be a three-dimensional mesh, wherein each vertex is associated with (has a three-dimensional position). Each vertex can be specifically represented by at least a three-dimensional spatial position x, y, z. Each vertex of the mesh can further be linked to a position in a corresponding texture map. For example, for each vertex, a position u, v corresponding to the spatial position x, y, z in the texture map can be stored.

[0027] The mesh may include information related to the depth of the represented object, but is not limited thereto (eg, as described, vertices may be represented by three-dimensional coordinates rather than just depth coordinates).

[0028] Depth can be represented by disparity, for example, and a depth map can be an indication of a disparity map. Higher depth values ​​indicate a greater distance from a reference viewpoint, and thus indicate a depth farther from the viewpoint and closer to the background.

[0029] Increasing the weight of the first light intensity value relative to the second light intensity value for increasing first depth values ​​may be achieved by determining the weight for the first light intensity value as a monotonically increasing function of the first depth value and / or determining the weight for the second light intensity value as a monotonically decreasing function of the first depth value.

[0030] In many embodiments, the first and second texture maps and the first and second meshes are generated based on a capture of a real-life scene.

[0031] The method may allow for high performance and efficiently determined new viewpoint images to be captured for realistic scenes, and is not limited to images provided by, for example, a virtual reality graphics engine.

[0032] According to an optional feature of the invention, the weighting depends on the sign of the difference between the first depth value and the second depth value.

[0033] This may provide a particularly efficient implementation and / or high performance and in particular may in many cases bias the combination towards light intensity values ​​representing non-occluded values ​​rather than occluded values ​​in the input texture map.

[0034] The sign of the difference may indicate whether the first depth value is greater than the second depth value or whether the second depth value is greater than the first depth value. It may not matter how large the difference is.

[0035] According to an optional feature of the invention, the weighting is dependent on the magnitude of the difference measure between the first depth value and the second depth value.

[0036] This may provide a particularly efficient implementation and / or high performance.The magnitude of the difference may indicate how large the difference between the first depth value and the second depth value is, but not which is largest.The magnitude may be an absolute value.

[0037] According to an optional feature of the invention, it is provided that the weighting of the largest of the first and second depth values ​​relative to the lowest of the first and second depth values ​​increases for increasing magnitude of the difference measure.

[0038] In many embodiments, this may provide a particularly efficient implementation with high performance.

[0039] According to an optional feature of the invention, determining the light intensity value for the light intensity image comprises setting the light intensity value to a value corresponding to a highest depth value of a first light intensity value and a second light intensity value.

[0040] In many embodiments, this may provide a particularly efficient implementation with high performance.

[0041] According to an optional feature of the invention, determining corresponding first image positions and first image position depth values ​​in the light intensity image for vertices of the first mesh is performed by a vertex shader.

[0042] In many embodiments, this can provide a particularly efficient implementation with high performance. It can provide particularly efficient operation and can significantly reduce resource requirements and / or increase processing speed. In particular, in many embodiments, it can allow the use of standardized hardware acceleration for resource-critical elements of the process.

[0043] This approach can allow for extremely efficient and low-complexity operation, where the same optimized functionality (eg, GPU) can be reused for resource-critical elements of the processing.

[0044] According to an optional feature of the invention, the step of determining the first light intensity value is performed by a first fragment shader.

[0045] In many embodiments, this can provide a particularly efficient implementation with high performance. It can provide particularly efficient operation and can significantly reduce resource requirements and / or increase processing speed. In particular, in many embodiments, it can allow the use of standardized hardware acceleration for resource-critical elements of the process.

[0046] According to an optional feature of the invention, the step of determining light intensity values ​​of the light intensity image is performed by a second fragment shader.

[0047] In many embodiments, this can provide a particularly efficient implementation with high performance. It can provide particularly efficient operation and can significantly reduce resource requirements and / or increase processing speed. In particular, in many embodiments, it can allow the use of standardized hardware acceleration for resource-critical elements of the process.

[0048] According to an optional feature of the present invention, the second fragment shader is arranged to: receive a second shader texture map comprising multi-component values, the multi-component values ​​comprising at least one light intensity component value and one transparency component value; and generate an output light intensity value by combining light intensity component values ​​from multiple texture maps in response to the transparency component value; and the method further comprises a step of setting the transparency component value of the multi-component value comprising the first light intensity value to the first depth value.

[0049] This approach can allow for extremely efficient and low-complexity operation, where the same optimized functionality (eg, GPU) can be reused for resource-critical elements of the processing.

[0050] The step of setting the transparency component value of a multi-component value including the first light intensity value to the first depth value may be performed before a second fragment shader determines a light intensity value for the light intensity image. The first fragment shader may determine the first light intensity value and the first depth value. The image generator may then generate a texture map including multi-component values, wherein the multi-component values ​​are arranged to include at least the first light intensity value and the first depth value, wherein the first depth value is included in a transparency component field / parameter / value of the multi-component values. The second fragment shader may then process the texture map to generate an output light intensity value by combining light intensity component values ​​from the multiple texture maps in response to a transparency component value (in this case, the first depth value determined by the first fragment shader).

[0051] According to an optional feature of the invention, the weighted combination further comprises a third light intensity value generated from a third mesh and a third texture map.

[0052] This method is well suited to generating light intensity images based on more input grid images for different viewpoints (or indeed multiple grid images from the same viewpoint).

[0053] According to an optional feature of the invention, the weighting of the first light intensity value relative to the second light intensity value depends on first light intensity values ​​in a first neighborhood around the first light intensity value and second light intensity values ​​in a second neighborhood around the second light intensity value.

[0054] This may improve performance in many embodiments.

[0055] The first neighborhood may include light intensity values ​​generated from the first texture map and the first mesh, and the second neighborhood may include light intensity values ​​generated from the second texture map and the second first mesh.

[0056] According to an optional feature of the invention, the weighting further depends on a difference between the first viewpoint and the third viewpoint.

[0057] This may improve performance in many embodiments.

[0058] According to an optional feature of the invention, the weighting of the first light intensity value relative to the second light intensity value further depends on a reliability indication indicating the reliability of the depth estimate used to determine the depth at the first grid, and in the first depth map, the weighting of the first light intensity value relative to the second light intensity value increases for an increased reliability of the depth estimate indicated by the reliability indication.

[0059] This may improve performance in many embodiments.

[0060] According to one aspect of the present invention, a device for generating a light intensity image is provided, the device comprising: a receiver for receiving a first texture map and a first grid representing a scene from a first viewpoint; a receiver for receiving a second texture map and a second grid representing the scene from a second viewpoint; an image generator for generating a light intensity image representing the scene from a third viewpoint in response to the first texture map, the first grid, the second texture map and the second grid; wherein the image generator comprises: a first view converter for determining a first light intensity value for a first position in the light intensity image by a viewpoint conversion based on the first texture map and the first grid; a second view converter for determining a second light intensity value for the first position by a viewpoint conversion based on the second texture map and the second grid; a combiner for determining a light intensity value of the light intensity image for the first position by a weighted combination of the first light intensity value and the second light intensity value; the weight of the weighted combination depending on a first depth gradient at a first grid position in the first grid corresponding to the first position relative to a second depth gradient in a second grid in the second grid corresponding to the first position.

[0061] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Embodiments of the invention are described, by way of example only, with reference to the accompanying drawings, in which:

[0063] Figure 1 illustrates an example of viewpoint transformation for a scene with one background object and a foreground object;

[0064] Figure 2 illustrates an example of elements of an apparatus for generating an image according to some embodiments of the present invention;

[0065] Figure 3 illustrates an example of elements of a method for generating an image according to some embodiments of the present invention;

[0066] Figure 4 illustrates an example of elements of a method for generating an image according to some embodiments of the present invention;

[0067] Figure 5 An example of elements of a method for generating light intensity values ​​for an image according to some embodiments of the invention is illustrated. DETAILED DESCRIPTION

[0068] The following description focuses on an embodiment of the present invention applicable to rendering a two-dimensional image of a mesh generated based on a texture map and a real-life three-dimensional scene captured from different viewpoints (i.e., different camera views). However, it should be understood that the present invention is not limited to this application, but can be applied, for example, to generating images based on computer-generated depth maps and meshes for virtual reality scenes.

[0069] Many 3D image (including video) processing algorithms and processes are based on the use of triangular meshes, as this can provide high computational efficiency in many embodiments. Meshes that provide three-dimensional information and associated texture maps are widely used in many graphics processing systems, for example, particularly in computer graphics. A mesh can provide information about the geometry including depth information (but is of course not limited to providing only depth information, for example, it can generally provide three-dimensional coordinates for each vertex). For simplicity, the combination of a mesh and its associated texture map will also be referred to as a mesh image hereinafter.

[0070] Grid-based graphics processing and image generation are ubiquitous, and most computer graphics cards, mobile phones, and gaming consoles are based on or support one or more grid-based graphics processing methods. Several standards have been developed to provide cross-platform application programming interfaces (APIs) that applications can use to efficiently utilize graphics processing based on specialized hardware, for example. Such methods include the well-known OpenGL or DirectX application programming interfaces, which are widely supported, including on highly specialized and optimized hardware.

[0071] Mesh-image based processing is well suited for flexibly modifying the view direction when rendering a scene, and is particularly well suited for scenes where the viewpoint is not limited to movement in a strictly horizontal line but where free movement of the viewpoint is desired. An efficient way to generate new viewpoints is to transform meshes originating from different viewpoints into a single global coordinate system and then perform perspective projection on the new camera plane. These steps can be done very efficiently using standard optimized graphics hardware, such as hardware based on the OpenGL standard. However, in order to optimize quality and provide additional information for areas that are de-occluded by viewpoint transformation, the generation of rendered images from new viewpoints is preferably based on texture maps and meshes provided for multiple different viewpoints.

[0072] The inventors have realized that this may often lead to suboptimal results with artifacts, and in particular depth distortion and incorrect filling of de-occluded areas.

[0073] This can be done by Figure 1 To illustrate, it illustrates an example with a foreground object in front of a background object. In the example, each point / blob can be considered to correspond to a scene location that can be represented by a vertex, and in general a point can represent a pixel and is actually a vertex of a mesh. Figure 1The diagram illustrates an example of providing grid images for a first viewpoint 1 and a second viewpoint 2. Based on these grid images, a two-dimensional image is generated for a third viewpoint 3 between the first viewpoint 1 and the second viewpoint 2. Thus, in this example, the first and second viewpoints 1 and 2 correspond to the camera viewpoints when capturing the scene, and viewpoint 3 corresponds to the desired viewpoint of the scene. The image is a light intensity image generated for rendering. It can directly reflect the light intensity values ​​of each pixel of the image to be rendered / displayed on a display.

[0074] It should be understood that the light intensity value can be any value indicating the light intensity, and specifically can be, for example, the light intensity for one color channel. For example, the light intensity value can be an R, G, or B value of an RGB representation, or can be, for example, a Y value of a Yuv representation, or can indeed be a U value or a V value of such a Yuv representation. Note that the U and V values ​​can be chromaticity values, and these values ​​also provide information about the light intensity of each color channel when rendering. Therefore, the light intensity value can be the brightness, chromaticity, or actually the chromaticity value of the color representation.

[0075] exist Figure 1 , the first figure shows the locations visible from viewpoint 1 by block (solid) points, and the locations on background objects occluded by foreground objects are shown by white (non-solid) points. The second figure illustrates the corresponding situation from viewpoint 2. Thus, the two first sub-figures show the visibility of scene points for two different camera viewpoints 1 and 2.

[0076] The third sub-figure illustrates the result of combining the meshes and textures from viewpoints 1 and 2 and warping (viewpoint conversion) these meshes and textures to viewpoint 3. The mesh originating from viewpoint 2 will be locally stretched and form gaps (corresponding to the deocclusion of some background objects). After projection to virtual viewpoint 3, the depth of the gaps is linearly interpolated between points with different depths. Thus, effectively, after the view transform from viewpoint 2 to viewpoint 1, the primitives and, in most embodiments, triangles that would form the mesh at the transition between background and foreground in the mesh of viewpoint 2 are stretched.

[0077] Figure 1 The gray circles in the figure represent points originating from viewpoint 2, and these points are linearly interpolated between the known 3D points in the grid when projecting the scene in the camera coordinates of viewpoint 3. Therefore, the gray circles / dots indicate that the depth positions are generated to correspond to positions in the image from viewpoint 3 after the deocclusion caused by the viewpoint transformation from viewpoint 2 to viewpoint 3. These points are closer to viewpoint 3 than the black dots behind them, which represent points on background objects visible from viewpoint 1.

[0078] However, when generating images for viewpoint 2, the typical approach is to select the frontmost position. Therefore, the system will select gray points originating from viewpoint 2, rather than the ideal black points originating from and visible from viewpoint 1. Consequently, the light intensity image values ​​at these points will be generated based on the texture map for viewpoint 2, rather than the texture map for viewpoint 1. This will result in the image for the deoccluded area being determined from the texture map where the corresponding object is occluded, rather than from the texture map where the object is unoccluded. As a result, the quality of the generated image is reduced, and elements of the deoccluded area are often not accurately rendered, despite the fact that appropriate information is available for processing.

[0079] It should be noted that while it is theoretically possible to modify the depth test logic used in the standardized hardware, in practice this can only be done for a given draw call. Therefore, it is necessary to split the mesh into several parts again, with the disadvantages of using thresholds mentioned earlier. The standard depth test is part of the process of resolving self-occlusion, where a mesh from a separate viewpoint can collapse onto itself (self-occlusion), where the depth test is used to resolve the resulting blur.

[0080] In the following, reference will be made to Figure 2 An apparatus and method for generating a light intensity image from multiple meshes and associated texture maps for different viewpoints is described, illustrating some exemplary elements of such an apparatus, and Figure 3-5 Some exemplary elements of such a method are illustrated.

[0081] Figure 2 and Figure 3-5 The apparatus and method aim to alleviate some of the described disadvantages. Specifically, rather than always selecting the foremost point, the system is arranged to generate a light intensity value for a given position in the output image based on a weighted combination of light intensity values ​​from two texture maps (for different viewpoints) with a weight depending on the depth value at that position, and typically the difference between the depth values ​​at that position. The weighting may be such that the weighting of the light intensity values ​​increases as the corresponding point is further back. Thus, the weighting of the light intensity values ​​may increase with depth.

[0082] The apparatus comprises a first receiver 201 which performs step 301, wherein a first mesh image is received, ie a first mesh and a first texture map is received. The first mesh image represents a scene from a given viewpoint.

[0083] The mesh image can be received from any internal or external source. For example, in some embodiments, the mesh image can be received from an external source, such as via a communication system, or specifically via a network (e.g., the Internet). In some embodiments, the mesh image can be received from an internal source that is operable to read a suitable distribution medium, such as a DVD. In some embodiments, the mesh image can be retrieved from a hard drive or local storage, or indeed in some embodiments, the mesh image can be generated locally, such as by a local virtual reality application.

[0084] In many embodiments, the first texture map and the first mesh may be generated from a capture of a real-life scene. The capture may be performed by a suitable set of cameras. For example, a single stereo camera or depth camera may capture the real-life scene and generate an image and a depth ( / disparity) map. In other embodiments, multiple cameras at different positions may capture two-dimensional images, and a depth map may be generated from the captured images, for example by disparity estimation. One of the images may be used as the texture map, and the mesh may be generated from the corresponding depth map. Thus, the mesh image may be an image providing a capture of the real scene, where the texture map represents light intensity values ​​captured by the camera at a given viewpoint and the captured depth information is represented by the mesh. Thus, the mesh image provides limited information about the scene, as it only represents information that can be determined from a given viewpoint (unlike, for example, a detailed geometric model that may be available for, for example, a local virtual reality engine).

[0085] The apparatus further comprises a second receiver 203 which performs step 303 in which it receives a second mesh image, i.e., a second texture map and a second mesh. The second mesh image provides a representation of the same scene, but from a different viewpoint. It will be appreciated that the comments regarding the first mesh image also apply to the second mesh image.

[0086] The first receiver 201 and the second receiver 203 are coupled to an image generator 205 that performs step 305, wherein a light intensity image representing the scene from a third viewpoint is generated in response to the first texture map, the first mesh, the second texture map, and the second mesh; thus, an image for the third viewpoint is generated based on the first and second mesh images for the first and second viewpoints, respectively. The first and second viewpoints may typically be camera viewpoints of the scene, wherein the third viewpoint represents a desired viewpoint. The image is a light intensity image and may specifically correspond to a given viewport and may typically be an image that can be directly presented on a display to provide a view of the scene from the third viewpoint. Specifically, the light intensity image may include a set of light intensity values, each light intensity value indicating the light intensity at a location in the image. Typically, an image may include a set of pixels, each pixel having one or more light intensity values. Thus, each light intensity value may correspond to a pixel value.

[0087] It will be appreciated that in some embodiments, each position (pixel) may be associated with / described by / defined by a single light intensity value. This may be the case, for example, for a monochrome image. In other embodiments, each position (pixel) may be associated with / described by / defined by multiple light intensity values, such as, for example, one light intensity value for each channel of a color representation. Thus, the light intensity values ​​may be considered to be color intensity values, and the light intensity image may be a color intensity image. In the case where multiple pixels are represented by multiple light intensity values, e.g. corresponding to different color channels, the described method may, for example, be applied to each color channel light intensity value separately.

[0088] Thus, the image generator 205 is arranged to generate images for different viewpoints. For example, in some embodiments, the image generator 205 may be provided with an input defining a desired viewpoint, such as a desired viewpoint generated in response to user input. The image generator 205 may then generate an output display image corresponding to the viewpoint based on the received grid image. This image may then be rendered, for example, by a suitable display.

[0089] Will refer to Figure 2 and Figure 4 The operation of the image generator 205 is described in more detail, illustrating an example of the method of the image generator 205.

[0090] Image generator 205 includes a first view converter 207, which performs step 401, wherein a first depth value and a first image position in the intensity image are determined for a vertex of a first mesh. Specifically, for each vertex in the first mesh, i.e., the mesh for the first viewpoint, first view converter 207 determines a corresponding image position in the intensity image. This can be considered to correspond to a projection of the first mesh, and in particular the vertex, onto the image plane of the intensity image, and is therefore effectively a projection from the first viewpoint to the third viewpoint.

[0091] In addition to determining the first image position corresponding to the position in the output image (light intensity image) to be generated, the first view converter 207 also continues to determine a first depth value for the vertex. The first depth value can specifically indicate the depth / distance of the vertex from the third viewpoint. Therefore, for each first image position / vertex, the first view converter 207 determines the depth corresponding to that position from the third viewpoint.

[0092] The first view converter 207 is accordingly arranged to warp the mesh from the first viewpoint to the third viewpoint by determining a position in the image from the third viewpoint for each vertex. Furthermore, a depth value is generated for each of these positions / vertices and an output is thus determined in the form of a plurality of points / positions in the third image and associated depths, wherein each point / position corresponds to a vertex.

[0093] Image generator 205 includes a second view converter 209, which performs step 403, wherein second depth values ​​and second image positions in the intensity image are determined for the vertices of the second mesh. Specifically, for each vertex in the second mesh, i.e., the mesh for the second viewpoint, second view converter 209 determines a corresponding image position in the intensity image. This can be considered to correspond to a projection of the second mesh, and in particular the vertex, onto the image plane of the intensity image, and is therefore effectively a projection from the second viewpoint to the third viewpoint.

[0094] In addition to determining the second image position corresponding to the position in the output image (light intensity image) to be generated, the second view converter 209 also continues to determine the second depth value for the vertex. The second depth value can specifically indicate the depth / distance of the vertex from the third viewpoint. Therefore, for each second image position / vertex, the second view converter 209 determines the depth corresponding to that position from the third viewpoint.

[0095] Thus, the second view transformer 209 performs an equivalent operation to the first view transformer 207, but on a second mesh rather than the first, and is therefore arranged to warp the mesh from the second viewpoint to the third viewpoint by determining a position in the image from the third viewpoint for each vertex in the second style. Furthermore, a depth value is generated for each of these positions / vertices, and an output is thus determined in the form of a plurality of points / positions in the third image and associated depths, where each point / position corresponds to a vertex. It should be noted that the first and second image positions are typically different, as they are based on different meshes, different vertices, and different viewpoint offsets.

[0096] It will be appreciated that different algorithms and methods for warping the grid to determine image position and depth for the third image are known, and any suitable method may be used.

[0097] In many embodiments, the operations of the first view converter 207 and the second view converter 209 are performed by vertex shaders. Vertex shading is a graphics operation used in many different graphics applications, and many algorithms and processes have been developed to perform efficient and very fast vertex shading. In particular, dedicated hardware and processors have been developed for vertex shading. These methods utilize large amounts of parallel processing, thereby providing very fast processing and allowing real-time processing even for very complex scenes and meshes. The first view converter 207 and the second view converter 209 can be specifically implemented as vertex shaders using dedicated hardware accelerated processing.

[0098] Most vertex shading algorithms inherently generate depth values ​​for vertices for the output viewpoint as part of the process of determining image position. Thus, in many embodiments, the described functionality of first view converter 207 and second view converter 209 can be implemented by vertex shaders without significantly increasing the required processing power or complexity.

[0099] The method thus allows very efficient and generally high-speed calculation of image position and depth values, while allowing a high degree of backward compatibility and reuse of existing functionality, in particular reuse of dedicated hardware.

[0100] The image generator 205 is arranged to perform step 405, wherein an output light intensity image for a third viewpoint is generated based on the outputs of the first viewpoint converter 207 and the second viewpoint converter 209. Step 405 specifically comprises performing for each pixel / position Figure 5 The light intensity value is determined for each pixel of the output light intensity image by the steps of:

[0101] The image generator 205 also includes a first shader 211, which is arranged to determine the light intensity values ​​and depth values ​​of multiple positions in the output image to be generated based on the first image position and the first texture map (i.e. based on the output generated from the first view transformer 207), i.e. the light intensity values ​​and depth values ​​in the light intensity image for the third viewpoint.

[0102] The first shader 211 is specifically arranged to generate a light intensity value and a depth value for each pixel position and, therefore, for each pixel, for the image from the third viewpoint based on the first image position and the depth value. Therefore, the first shader 211 may perform step 501 for each position / pixel of the output light intensity image, thereby generating a first intermediate image and a corresponding first intermediate depth map for the third viewpoint based on the first grid image (i.e., based on the first grid and the first texture map).

[0103] The first shader 211 may specifically use an interpolation between the light intensity value and the depth value of the point generated by the first view converter 207 .

[0104] For example, a three-dimensional triangle of a first mesh may be considered. For the three vertices forming the triangle, the first view transformer 207 determines corresponding image positions in the first intermediate image and the first intermediate depth map. Thus, a planar triangle is determined in the first intermediate image, corresponding to the projection of the first mesh triangle onto the image plane for the third viewpoint. Each of the three vertices is associated with a position in the first texture map, thereby defining a triangle in the texture map that corresponds to the three-dimensional triangle in the first mesh. Thus, the same triangle in the texture map is mapped to the planar triangle determined in the first intermediate image, and the first shader 211 may map the texture map accordingly. Figure 3 The polygons are stretched onto flat image triangles, for example by interpolation.

[0105] Similarly, to determine a planar triangle in the first intermediate image, the first image positions of the three vertices also determine a corresponding planar triangle in the first intermediate depth map. In addition, for each vertex, the first view converter 207 has already determined a corresponding depth value, so the depth values ​​of the three vertex positions in the first depth map are known, and the first shader 211 can continue to determine the depth values ​​for different positions and specific pixels within the planar triangle, for example, by interpolating between the vertex depth values.

[0106] The image generator 205 further includes a second shader 213, which is arranged to perform corresponding processing on the output from the second view converter 209. Thus, the second shader 213 can generate a second light intensity value based on the second texture map and a second image position and a second depth value based on the second image position. Thus, the second shader 213 can perform step 503 for each position / pixel of the output light intensity image, thereby generating a second intermediate image and a second intermediate depth map for the third viewpoint based on the second grid image (i.e., based on the second grid and the second texture map provided for the second viewpoint).

[0107] As a further example, the first shader 211 and the second shader 213 may determine the light intensity value by interpolating texture map values ​​of vertices of the polygon where the grid position is located. The interpolation may depend on the distance between the grid position and each vertex.

[0108] For example, in the original mesh before transformation, each vertex may have an associated position in a coordinate system based on the current viewpoint (e.g., x, y, z coordinates are defined for each vertex). In addition, each vertex will point to a specific light intensity value / pixel value in the corresponding texture map, i.e., a set of two-dimensional texture map coordinates (u, v) is also stored for each vertex.

[0109] The viewpoint transformation of the mesh by the first viewpoint transformer 207 / second viewpoint transformer 209 results in a modification of the vertex positions so that the x, y, z values ​​will be transformed into appropriate coordinates in the third viewpoint coordinate system. The point on the mesh corresponding to a given first position in the intermediate light intensity image / depth map will be determined, and the vertices of the polygon within which it falls will be identified. The distance from this point to each vertex will be determined, and for each vertex, the light intensity image of the texture map at the vertex's u, v position will be determined. The light intensity value for the first position is then determined by weighted interpolation of these texture map values, where the weights depend on the distance to the individual vertices. The same method is applied to the depth values ​​of the vertices to generate the depth value for the corresponding first position in the intermediate depth map.

[0110] In many embodiments, the operations of the first shader 211 and the second shader 213 are performed by a fragment shader. Fragment shading is a graphics operation used in many different graphics applications, and many algorithms and processes have been developed to perform efficient and very fast fragment shading. In particular, dedicated hardware and processors for fragment shading have been developed. These methods utilize a large amount of parallel processing, thereby providing very fast processing and allowing real-time processing even for very complex scenes. The first shader 211 and the second shader 213 can be specifically implemented as fragment shaders that use dedicated hardware accelerated processing.

[0111] Thus, the method allows very efficient and typically high-speed generation of a plenoptic intensity image and a depth map for a third viewpoint based on grid images from both the first and second viewpoints.

[0112] Indeed, a particular advantage of this method is that the generation of the first intermediate image, the second intermediate image, the first intermediate depth map, and the second intermediate depth map can be achieved by executing the same algorithm and functionality. In particular, four fragment shading operations can be applied to each fragment shading, implementing the same algorithm, or combined fragment shading can be performed to generate both light intensity values ​​and depth values. This is particularly advantageous because the algorithms and operations performed are algorithms and operations that can use existing functionality and, in particular, optimized and dedicated hardware. In particular, the algorithm is particularly well-suited for parallel processing and is therefore well-suited for hardware-accelerated processing.

[0113] It will be appreciated that the different fragment shading (i.e. for each intermediate image and each depth map) may be performed by separate functions, and in particular by different hardware fragment shaders operating in parallel, or may be performed, for example, by the same function, and in particular by iterations performed by the same hardware fragment shader.

[0114] The first shader 211 and the second shader 213 are coupled to a combiner 215 which is arranged to generate an output light intensity image for a third viewpoint by combining the first intermediate light intensity image and the second light intensity image, combining depth values ​​based on the first intermediate depth map and the second intermediate depth map.

[0115] Specifically, for a given position / pixel in the output light intensity image, the combiner 215 can execute step 503 and generate an output light intensity value for the position / pixel as a weighted combination of the light intensity value at the same position in the first intermediate image and the light intensity value at the same position in the second intermediate image, wherein the weight for at least one of the light intensity values ​​depends on the depth value at the position in at least one of the first intermediate depth map and the second intermediate depth map.

[0116] Furthermore, the combiner 215 is arranged to increase the weight of the light intensity values ​​in the first intermediate image relative to the weight of the light intensity values ​​in the second intermediate image for increasing depth values ​​in the first intermediate depth map. Correspondingly, the combiner 215 may be arranged to increase the weight of the light intensity values ​​in the first intermediate image relative to the weight of the light intensity values ​​in the second intermediate image for decreasing depth values ​​in the second intermediate depth map.

[0117] In many embodiments, the combiner 215 may be further arranged to increase the weight of the light intensity values ​​in the second intermediate image relative to the weight of the light intensity values ​​in the second intermediate image for increasing depth values ​​in the second intermediate depth map. Correspondingly, the combiner 215 may be arranged to increase the weight of the light intensity values ​​in the second intermediate image relative to the weight of the light intensity values ​​in the first intermediate image for decreasing depth values ​​in the first intermediate depth map.

[0118] Therefore, the combiner 215 may be arranged to increase the weight of light intensity values ​​that are further back (i.e., have higher depth values ​​and distances) from the third viewpoint. Thus, the combiner 215 may bias the combination towards light intensity values ​​that are closer to the background, and indeed in some embodiments may perform a selective combination by selecting the light intensity values ​​that are furthest back.

[0119] It should be noted that this approach may be counterintuitive and contrary to the traditional approach of selecting the front-most light intensity value to solve the self-occlusion problem. However, the inventors have realized that when the combination of the outputs from the fragment shaders (such as those of the first shader 211 and the second shader 213) is biased towards the deepest (closest to the background) light intensity, improved performance can be achieved in many embodiments.

[0120] The inventors have particularly appreciated that, in certain circumstances, such a combination of two images generated for a given viewpoint from surrounding viewpoints via a grid image viewpoint transform can provide improved deocclusion performance by increasing the weights of image objects that tend to be unoccluded in the original grid image. For example, in Figure 1 In the specific example, Figure 2 The method of

[0066] may allow the resulting image to the left of the foreground image to be based more on the input image for view 1 than for view 2, and thus more on an image where areas are unoccluded. This method has been found to provide very significant improvements, and in particular provides significantly improved quality in deoccluded areas.

[0121] It should be understood that different weighted combinations may be used by combiner 215 in different embodiments. As a specific example, a weighted linear combination may be used. For example, for each pixel in the output image of the third viewpoint, the following combination of light intensity values: c n , where n refers to the viewpoint, you can execute:

[0122]

[0123] The weights in the previous formula depend on the depth values ​​in the generated intermediate depth values:

[0124] w1=f(d1,d2), w2=f(d1,d2).

[0125] The function f is typically a monotonically increasing function of the depth value for the depth value of the viewpoint and / or a monotonically decreasing function of the depth value for the depth value of the other viewpoint. It will be appreciated that in some embodiments, the function may be different for different weights and / or have different dependencies on the two depth values, and indeed in some embodiments, one or both weights may depend on only one of the depth values. In practice, changing the relative weighting between the first and second viewpoints of the source light intensity value may be achieved by changing a first weight, a second weight, or both the first and second weights, where each weight is a function of the first depth value, the second depth value, or both the first and second depth values.

[0126] In many embodiments, the weighting depends on the magnitude of a difference measure between the first depth value and the second depth value. It will be appreciated that any suitable difference measure may be used. In many embodiments, the difference measure may simply be the difference, and thus the weighting may depend on the magnitude of the difference measure between the first depth value and the second depth value. In some embodiments, the difference measure may be, for example, the difference between the outputs of a monotonic function applied to the depth values, or a monotonic function of the magnitude of such a difference. The magnitude of the difference measure may indicate the degree of difference between the first and second depth values, but not which of these values ​​is largest.

[0127] In many embodiments, the smaller the magnitude, the smaller the difference between the weights of the first light intensity value and the second light intensity value.

[0128] For example, the difference between the weights of the first and second light intensity values ​​may be determined as a monotonic function of the magnitude of the difference between d1 and d2, e.g., normalized to the square of the magnitude between 0 and 1 for a given depth value. For example, the difference between the weights may be determined as:

[0129] w a =α|d1-d2| 2 =α(d1-d2) 2

[0130] Among them, α is chosen to be w a Scaled to belong to [0; 1].

[0131] The weighting of the first light intensity value and the second light intensity value may depend on the sign of the difference between d1 and d2, and thus on whether d1 and d2 are maximum (with the maximum value indicating that the pixel is at a higher depth / farthest back position).

[0132]

[0133] and

[0134]

[0135] As another example, a selection method can be used:

[0136] w1 = 1 and w2 = 0, for d1 > d2 + ∈

[0137] and

[0138] w1 = 0 and w2 = 1, for d1 < d2 + ∈

[0139] and

[0140] for other cases.

[0141] It should be understood that many different methods and formulas can be used to determine the weights according to the preferences and requirements of individual embodiments. It should also be understood that in many embodiments, the weights can depend on the sign of the difference between the depth values (and thus on which is greater / farthest from the back) as well as the magnitude of the difference measure, where the difference in weights increases as the difference measure increases.

[0142] In many embodiments, the weight for the maximum depth value relative to the minimum depth value among the first depth value and the second depth value increases for an increasing magnitude of the difference measure (specifically, the greater the difference (measure) between the two depth values, the greater the difference between the weight for the light intensity value with the maximum depth value and the weight for the light intensity value with the minimum depth value (among the two depth values)). The weight for the light intensity value with the maximum depth value is higher than the weight for the light intensity value with the minimum depth value.

[0143] Thus, the combiner 215 can be arranged to increase the weight for the rearmost light intensity value, i.e., to bias the generated output light intensity value towards the rearmost light intensity value. The greater the depth difference, the greater the difference.

[0144] In many embodiments, this method can provide a very efficient implementation, which results in particularly advantageous image generation.

[0145] In some embodiments, the combiner 215 can be arranged to perform a selection combination, for example, by simply selecting the light intensity value corresponding to the highest depth value. Thus, when generating an output light intensity value for the output light intensity image of the third viewpoint, if the first depth value is higher than the second depth value, the combiner 215 can simply set it to the first light intensity value and set it to the first light intensity value, and if the second depth value is higher than the first depth value, then set it to the first light intensity value (if they are the same, either one of them, or for example, the average).

[0146] As previously mentioned, in many embodiments, the operations of the first view changer 207 and the second view changer 209 may advantageously be performed by a vertex shader, and the operations of the first shader 211 and the second shader 213 may be performed by a first fragment shader.

[0147] Furthermore, in many embodiments, the operation of combiner 215, i.e., the combination of the first and second intermediate images, can also be advantageously performed by a fragment shader. Specifically, many algorithms and processes have been developed to perform efficient and very fast fragment shading to blend two images together, and many of them can be optimized for hardware acceleration using specialized hardware. Combiner 215 can be specifically implemented as a fragment shader that uses specialized hardware-accelerated processing to blend the two images.

[0148] Indeed, a particular advantage of the described approach is that most of the individual operations can be performed using developed graphics processing and dedicated hardware.

[0149] In graphics processing, it is sometimes necessary to represent transparent image objects and elements, and many algorithms have been developed for this purpose. The inventors have recognized that such functionality can be exploited in the current method to provide efficient operation and allow, for example, specialized hardware and algorithms designed for such transparency processing to be re-used for generating new views based on mesh images for other viewpoints.

[0150] In some embodiments, the combiner 215 may utilize a fragment shader that implements a fragment shader algorithm, in which two (or more) texture maps including multi-component values ​​are received and blended together. Specifically, for each uv position / pixel in the texture map, the algorithm is designed to receive multiple values, one of which is a light intensity value and the other is a transparency component value. Typically, the color of each pixel is represented by three color component values, such as RGB color values, i.e., the texture map is a three-channel texture map. In addition, for color channels / components, the algorithm is based on a texture map that includes a fourth channel, i.e., a transparency channel that provides a transparency value, typically represented by α. The fragment shader algorithm can be arranged to generate an output value for a given position by blending the color channels based on the transparency value. This can allow a transparent image to be superimposed on a background image while still allowing the background image to be seen. The degree of transparency for an individual pixel determines how much of the background and foreground are respectively present in the output pixel, i.e., it indicates how visible the background is through the (semi-)transparent foreground.

[0151] In some embodiments, the combiner 215 can be arranged to retrieve depth values ​​from the first shader 211 and the second shader 213 and generate an intermediate multi-component texture map, wherein each pixel includes a multi-component value, the multi-component value including a light intensity value from the intermediate image and a depth value for the corresponding intermediate depth map. Specifically, the depth value is stored by the fragment shader algorithm in a channel allocated for transparency. Thus, for a given position / pixel in the output image, the combiner 215 sets the transparency component value of the multi-component value including the intermediate light intensity value to the intermediate depth value for that position. In some embodiments, the intermediate depth value is not included directly, but a monotonic function is applied, for example, to generate the value stored in the transparency channel. For example, normalization can be applied.

[0152] Therefore, two (or more, for example) four-component / channel intermediate texture maps may be generated and the fragment shader algorithm will then be called with these intermediate texture maps. The fragment shader algorithm treats the depth (derived) value as a transparency value and blends the light intensity value (color channel value) based on the depth (derived) value that the algorithm assumes is a transparency value.

[0153] In this way, existing transparency-based fragment shaders can be "tricked" into performing depth-based blending as described for combiner 215. It allows for efficient reuse of functionality that includes dedicated hardware.

[0154] A very significant advantage of the described method is that in many embodiments, many of the required operations can be performed by hardware-optimized graphics processing. For example, the two view transformers 207, 209, the two shaders 211, 213, and even most of the described processing of the combiner 215 can be performed by a GPU supporting, for example, the OpenGL standard. This can reduce costs, facilitate design and implementation, and generally provide a more efficient implementation with greatly improved processing speed. Furthermore, the method is based on commonly available values ​​and does not require specialized additional processing, such as generating and warping depth gradients. Instead, generating a warp of the intermediate image will generally inherently generate the required depth values ​​as part of the warping.

[0155] As a concrete example, you can implement a GLSL (OpenGL Shading Language) shader program in which the fragment's depth value is written to the fourth element of the variable outputColor:

[0156]

[0157] In this example, the input texture map texture1 and the two-component input value uvCoordinate representing the uv coordinate in the texture map are used to generate a four-component output value outputColor for the uv coordinate.

[0158] The outputColor value is first set to the value of the texture at the given location, i.e., texture(texture1, uvCoordinate). This provides the output color value (e.g., RGB value) for the uv location of the specific texture texture1 uvCoordinate. Additionally, the algorithm sets the fourth component of outputColor , i.e., outputColor[3] , to the value gl_FragCoord.z ​​, which is the depth value inherently generated for the uvCoordinate. Typically, the fourth value is used for the transparency value, but in this example it is overwritten by the depth value.

[0159] The first view converter 207 and the second view converter 209 can generate two intermediate texture maps of four-component values ​​in this way, thereby representing both the intermediate image and the depth map. This can then be mixed by implementing a fragment shader through the combiner 215, for example:

[0160]

[0161] In this example, a blended output value is generated from two textures, texture1 and texture2, for a given uvCoordinate. First, the four component values ​​of the given uv coordinate are assigned to t1 and t2, respectively. Their depth value components are then assigned to d1 and d2. The difference is determined and the weight w1 of the first texture map is determined. The outputColor, representing the blended color value, is then set by calling the function mix(), which performs a weighted linear combination of the two values ​​based on a weight (the weight of the second texture map is inherently set to 1-w1 by the blend / combine operation). In this example, the depth difference is scaled by a gain (10.0 in this case), which can be applied to provide a desired bias toward later light intensity values.

[0162] The above description focuses on a very efficient method for generating an output image based on two input grid images. However, a particularly advantageous feature of this method is that it can be easily extended to a larger number of input grid images in many embodiments. For example, the method can be extended to three grid images, using the same method to generate an intermediate image and depth map for the third grid image, but expanding the combination to include a weighted combination of the three light intensity values, with increasing weights based on the depth values ​​being closer to the end.

[0163] It should be understood that different algorithms may be used to determine the weights of the three (or more) depth values.

[0164] For example, in some embodiments, the weighting of the first light intensity value relative to the second light intensity value depends on the first light intensity value in a first neighborhood around the first light intensity value and the first light intensity value in a second neighborhood around the second light intensity value. The first and second neighborhoods can specifically be in the first and second intermediate images, respectively. Thus, the first neighborhood can include light intensity values ​​generated by the first shader 211 based on the first grid image, and the second neighborhood can include light intensity values ​​generated by the second shader 213 based on the second grid image.

[0165] For example, if the light intensity values ​​in the first and second neighborhoods do not vary much (low variance), the weights may become less dependent (or not dependent at all) on the depth difference. If the variance is, for example, below a threshold, the weights may become equal. For example, the weights may become fixed and equal to 0.5 for each reference image. The motivation for this is that when both reference images have uniform texture, there is likely no real depth transition or difference, and the observed depth difference is due to noise or depth estimation errors.

[0166] In many embodiments, the combined weights may also be responsive to other parameters.

[0167] In many embodiments, the weighting may also depend on the difference between the first viewpoint and the third viewpoint. Specifically, the greater the difference, the lower the weighting. The difference may be determined, for example, as a geometric difference between the x, y, and z coordinates of the viewpoints, or may be determined directly from viewpoint change parameters provided to the image generator 205. This approach may weight smaller viewpoint changes more heavily than larger changes, reflecting the fact that smaller viewpoint changes are associated with higher quality.

[0168] Therefore, in some embodiments, the weights may also depend on a global parameter that measures how different the output viewpoint (the third viewpoint) is from the input viewpoint (e.g., in terms of angular difference). If the angular difference is large, a lower weight may be chosen so that nearby cameras have a larger contribution in the blending operation.

[0169] In some embodiments, the weighting may depend on a reliability indication for the first gradient, wherein the reliability indication indicates the reliability of the depth estimate for determining the depth value, and in particular for determining the depth value of the mesh.

[0170] Many depth cameras or depth estimation techniques provide a reliability indication, which indicates how reliable the generated depth estimate is considered to be. For example, depth sensors, whether stereo pairs or structured light sensors, typically have an associated uncertainty. This uncertainty can come from the disparity estimation process or from the infrared measurement process of the depth sensor. The reliability information can, for example, be provided as a pixel confidence map. In this case, pixels with higher uncertainty can be weighted lower in the combination than pixels with lower uncertainty. For example, the uncertainty can be taken into account when determining the weight map before the viewpoint transformation, or for example the confidence map can be warped to a third viewpoint and the uncertainty taken into account when determining the weights in the third viewpoint domain.

[0171] According to some embodiments, the first texture map and the first mesh are generated from a capture of a real-life scene.

[0172] It should be understood that, for the sake of clarity, the above description has described embodiments of the present invention with reference to different functional circuits, units, and processors. However, it will be apparent that any suitable distribution of functionality between different functional circuits, units, or processors may be used without departing from the present invention. For example, functions shown as being performed by separate processors or controllers may be performed by the same processor. Therefore, references to specific functional units or circuits are merely to be considered as references to suitable devices for providing the described functionality, rather than to indicating a strict logical or physical structure or organization.

[0173] The present invention can be implemented in any suitable form, including hardware, software, firmware or any combination thereof. The present invention can optionally be implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of the embodiments of the present invention can be implemented physically, functionally and logically in any suitable manner. In fact, the functions can be implemented in a single unit, in multiple units or as part of other functional units. Like this, the present invention can be implemented in a single unit, or can be physically and functionally distributed between different units, circuits and processors.

[0174] Although the present invention has been described in conjunction with certain embodiments, it is not intended to limit the invention to the specific forms set forth herein. Rather, the scope of the present invention is limited only by the appended claims. In addition, although it may appear that features have been described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0175] Furthermore, although listed separately, multiple devices, elements, circuits or method steps may be implemented by, for example, a single circuit, unit or processor. In addition, although individual features may be included in different claims, these features may be advantageously combined, and inclusion in different claims does not mean that the combination of features is not feasible and / or disadvantageous. The inclusion of a feature in one category of claims does not imply a limitation to that category, but rather indicates that the feature is equally applicable to other claim categories when appropriate. Furthermore, the order of features in the claims does not imply any specific order in which the features must work, and in particular, the order of the individual steps in a method claim does not imply that the steps must be performed in that order. Rather, the steps may be performed in any suitable order. In addition, singular references do not exclude pluralities. Thus, references to "one", "first", "second", etc. do not exclude pluralities. The reference numerals in the claims are provided merely to clarify the examples and should not be construed as limiting the scope of the claims in any way.

Claims

1. A method for generating a light intensity image, the method comprising: Receiving (301) a first texture map and a first mesh representing a scene from a first viewpoint; receiving (303) a second texture map and a second mesh representing the scene from a second viewpoint; In response to the first texture map, the first mesh, the second texture map, and the second mesh, determining (305) a light intensity image representing the scene from a third viewpoint, wherein determining (305) the light intensity image comprises: determining (401) corresponding first image positions and first image position depth values ​​in the light intensity image for vertices of the first mesh; determining (403) corresponding second image locations and second image location depth values ​​in the light intensity image for vertices of the second mesh; and For each first position in a plurality of positions in the light intensity image: determining (501) a first light intensity value based on the first texture map and the first image position, and determining a first depth value based on the first image position and the first image position depth value; determining (503) a second light intensity value based on the second texture map and the second image position, and determining a second depth value based on the second image position and the second image position depth value; A light intensity value at the first position for the light intensity image is determined (505) as a weighted combination of at least the first light intensity value and the second light intensity value; the weighting of the first light intensity value relative to the second light intensity value increases for increasing first depth values, wherein the weighting depends on the magnitude of a difference measure between the first depth value and the second depth value.

2. The method according to claim 1, wherein The weighting of a maximum depth value of the first depth value and the second depth value relative to a minimum depth value of the first depth value and the second depth value increases for an increasing magnitude of the disparity measure.

3. A method according to any preceding claim, wherein: Determining (405) the light intensity value for the light intensity image includes setting the light intensity value to a value of the first light intensity value and the second light intensity value that corresponds to a maximum depth value.

4. The method according to claim 1 or 2, wherein: Determining (401) corresponding first image positions and first image position depth values ​​in the light intensity image for vertices of the first mesh is performed by a vertex shader.

5. The method according to claim 1 or 2, wherein: The step of determining the first light intensity value is performed by a first fragment shader.

6. The method according to claim 1 or 2, wherein: The step of determining the light intensity values ​​for the light intensity image is performed by a second fragment shader.

7. The method according to claim 6, wherein: The second fragment shader is arranged to: receive a second shader texture map comprising multi-component values, the multi-component values ​​comprising at least one light intensity component value and one transparency component value; and generating an output light intensity value by combining light intensity component values ​​from a plurality of texture maps in response to the transparency component value; And the method further includes the step of setting a transparency component value of a multi-component value including the first light intensity value as the first depth value.

8. The method according to claim 1 or 2, wherein: The weighted combination also includes a third light intensity value generated based on a third mesh and a third texture map.

9. The method according to claim 1 or 2, wherein: The weighting of the first light intensity value relative to the second light intensity value depends on first light intensity values ​​in a first neighborhood around the first light intensity value and second light intensity values ​​in a second neighborhood around the second light intensity value.

10. The method according to claim 1 or 2, wherein: The weighting also depends on the difference between the first viewpoint and the third viewpoint.

11. The method according to claim 1 or 2, wherein: The weighting of the first light intensity value relative to the second light intensity value also depends on a reliability indication indicating the reliability of the depth estimate used to determine the depth at the first grid position in the first grid map, and the weighting of the first light intensity value relative to the second light intensity value increases for an increased reliability indication indicating the depth estimate.

12. The method according to claim 1 or 2, wherein: The weighting depends on the sign of the difference between the first depth value and the second depth value.

13. An apparatus for generating a light intensity image, the apparatus comprising: A first receiver (201) for receiving a first texture map and a first mesh representing a scene from a first viewpoint; a second receiver (203) for receiving a second texture map and a second mesh representing the scene from a second viewpoint; an image generator (205) for determining the light intensity image representing the scene from a third viewpoint in response to the first texture map, the first mesh, the second texture map, and the second mesh; Wherein, the image generator (205) comprises: a first view converter (207) for determining corresponding first image positions and first image position depth values ​​in the light intensity image for vertices of the first mesh; a second view converter (209) for determining corresponding second image positions and second image position depth values ​​in the light intensity image for vertices of the second mesh; a first shader (211) for determining, for each first position of a plurality of positions in the light intensity image, a first light intensity value based on the first texture map and the first image position, and determining a first depth value based on the first image position and the first image position depth value; a second shader (213) for determining, for each first position of a plurality of positions in the light intensity image, a second light intensity value based on the second texture map and the second image position, and determining a second depth value based on the second image position and the second image position depth value; a combiner (215) for determining, for each first position of the plurality of positions in the light intensity image, a light intensity value at the first position for the light intensity image as a weighted combination of at least the first light intensity value and the second light intensity value; the weighting of the first light intensity value relative to the second light intensity value increasing for increasing first depth values.

14. A computer program product adapted to perform the method according to any one of claims 1 to 12.

Images