Data signal comprising representation of three-dimensional scene

Abstract

An apparatus generates a data signal comprising three-dimensional image data providing a representation of a three-dimensional scene. The three-dimensional image data includes at least one image providing visual data of the scene. The data signal further includes a view dependency indication for an image region of the image, where the view dependency indication indicates a degree of change in one or more visual properties of a scene point indicative of the image region according to the viewing direction. The rendering device comprises a receiver (201) that receives the data signal, and a render (203) generates a view image of the scene in accordance with a view gesture from the three-dimensional image data in accordance with a view dependency indication. In particular, mixing contributions from different points of the scene to a given pixel of the view image may depend on the view dependency indication. The view dependency indication may indicate a view dependency of a scene point represented by the image region.

Description

Technical Field

[0001] The present invention relates to data signals comprising a representation of a three-dimensional scene, apparatus and methods for generating such data signals, and apparatus and methods for rendering view images based on such data signals. The present invention particularly, but not exclusively, relates to data signals providing a three-dimensional video signal, e.g. for immersive video. Background Art

[0002] The variety and range of image and video applications has increased substantially in recent years as new services and ways of utilizing and consuming video continue to be developed and introduced.

[0003] For example, one service that is becoming increasingly popular is to provide image sequences in a way that enables the viewer to actively and dynamically interact with the system to change the parameters of the rendering. A very attractive feature in many applications is the ability to change the viewer's effective viewing position and viewing direction, for example allowing the viewer to move around and look around in the scene being rendered.

[0004] Such features may generally allow providing a user with a virtual reality experience. This may allow the user, for example, to move (relatively) freely in a virtual environment and to dynamically change his position and where he is looking. Typically, such virtual reality (VR) applications are based on a three-dimensional model of a scene, wherein the model is dynamically evaluated to provide a specific requested view. This approach is well known from, for example, gaming applications, for example in the category of first-person shooters, for computers and consoles. Other examples include augmented reality (AR) or mixed reality (MR) applications.

[0005] An example of a video service or application that has been proposed is immersive video, where the video is played back on, for example, a VR headset to provide a three-dimensional experience. For immersive video, the viewer has the freedom to look around and move around in the presented scene so that this can be perceived as being viewed from different viewpoints. However, in many typical approaches, the amount of movement is limited to, for example, a relatively small area around a nominal viewpoint, which may typically correspond to the viewpoint from which video capture of the scene has been performed. In such applications, three-dimensional scene information is often provided, which allows high-quality view image synthesis of viewpoints that are relatively close to (one or more) reference viewpoints, but which deteriorates if the viewpoint deviates too much from the reference viewpoint.

[0006] Immersive video may also often be referred to as 6 degrees of freedom (6DoF) or three-dimensional video.MPEG Immersive Video (MIV) is an emerging standard where metadata is used on top of existing video codecs to enable and standardize immersive video.

[0007] A number of different representations have been developed and standardized to allow efficient data description of scenes, to allow generation of view images for different view poses.

[0008] A common approach for representing scenes is called multi-view with depth (MVD) representation and capture. In this approach, a scene is represented by multiple images with associated depth data, where the image representations are typically from different view poses of a limited capture area. Images can actually be captured using a camera suite that includes multiple cameras and a depth sensor.

[0009] Other examples of representations include, for example, point clouds, multi-plane images, multi-sphere images, and densely sampled volumetric representations. Such representations are called volumetric representations, where a point in space is represented by a position and a light property for that position. Other representations may include densely sampled light fields or other so-called light field representations. For a light field representation, a given point in a scene may be associated with different light properties corresponding to different rays (corresponding to different light rays and ray directions) that pass through the scene point.

[0010] However, while such representations may be suitable for many different applications and scenarios, they often do not provide ideal performance, or allow perfect generation of images. They may also result in higher than preferred data rates and / or processing complexity and / or resource requirements in many scenarios.

[0011] Therefore, improved methods for scene representation and processing thereof would be advantageous. In particular, methods allowing improved operation, increased flexibility, improved immersive user experience, reduced complexity, facilitated implementation, improved image quality, improved and / or facilitated rendering, improved and / or facilitated scene representation or processing thereof, and / or improved performance and / or operation would be advantageous. Summary of the invention

[0012] Accordingly, the Invention seeks to mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0013] According to one aspect of the present invention, there is provided an apparatus arranged to generate a data signal describing a three-dimensional scene, the apparatus comprising: a generator for generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; a processor arranged to generate a view dependency indication for an image area of ​​the first image, the view dependency indication indicating the degree of change of one or more visual properties of scene points of the image area according to a viewing direction; and wherein the generator is arranged to include the view dependency indication in the data signal.

[0014] In many embodiments, the invention may provide improved performance and / or operation and / or implementation. It may generally allow improved representation of a scene, which in particular allows more accurate and higher quality view images of the scene to be drawn.

[0015] The method can be applied to many different representations. In particular, it can provide additional information and improved view adjustment for volumetric representations of three-dimensional scenes, since such volumetric formats do not inherently allow for representation of view dependencies.

[0016] The view dependency indication may be a Lambertian indication indicating a Lambertian level of a scene point of the image region, which indicates a level of view dependency of the scene point of the image region.

[0017] The light property value of the scene may be any value indicating the brightness, chromaticity, saturation, lightness, brightness and / or color of a scene point.The view dependency indication for an image region may indicate the degree of Lambertianity of scene points of which the image region comprises light property values.

[0018] The first image may be any two-dimensional data structure that provides light property values ​​for scene points in the scene. The first image may be a frame of a video sequence. The first image may be a projected image of a point cloud, an image corresponding to a viewport of the scene from the capture pose, a multi-plane image that provides light property values ​​for different planes or spheres, an image or video atlas, etc.

[0019] In many embodiments, multiple view dependency indications may be provided to reflect the Lambertian nature of different image regions of the first image and / or image regions of other images. In some embodiments, the view dependency indication may vary spatially and provide an indication of the Lambertian nature of different image regions / scene regions (including, for example, image regions as single image samples / pixels).

[0020] According to an optional feature of the invention, the processor is arranged to generate the view dependency indication from the three dimensional image data.

[0021] In many embodiments, this may provide improved performance and / or operation.In many embodiments, it may provide an efficient and high performance method for determining view dependency indications.

[0022] According to an optional feature of the invention, the processor is arranged to determine light properties of a scene area represented by the image area in different directions; and determine the view dependency indication in response to changes in the light properties in different directions.

[0023] In many embodiments, this may provide improved performance and / or operation.In many embodiments, it may provide an efficient and high performance method for determining view dependency indications.

[0024] In some embodiments the processor may be arranged to generate the view dependency indication in response to a comparison of light output from the image area in different directions.

[0025] According to an optional feature of the invention, the view dependency indication is indicative of a variation of the light radiance according to a direction of a scene point of the image area.

[0026] In many embodiments, this may provide improved performance and / or operation.

[0027] According to an optional feature of the invention, the representation of the three dimensional scene is a multi-view plus depth representation, and the first image is an image of a set of multi-view images of the multi-view plus depth representation.

[0028] The method may provide particularly advantageous operation and performance for multi-view plus depth representation.Using view dependency indication with multi-view plus depth representation may, for example, provide a synergistic effect allowing improved view images of a scene to be rendered.

[0029] According to an optional feature of the invention, the representation of the three dimensional scene is a multi-plane image representation and the first image is a plane of the multi-plane image representation.

[0030] The method may provide particularly advantageous operation and performance for multi-plane image representations.Using a view dependency indication with a multi-plane image representation may, for example, provide synergistic effects for volumetric representations that do not inherently consider or encode any view direction changes.

[0031] According to an optional feature of the invention, the representation of the three dimensional scene is a point cloud representation, and the first image comprises light property values ​​for projecting at least part of the point cloud representation onto the image plane.

[0032] The method may provide particularly advantageous operation and performance for multi-plane image representations.Using a view dependency indication with a multi-plane image representation may, for example, provide synergistic effects for volumetric representations that do not inherently consider or encode any view direction changes.

[0033] The image region may include pixels indicating light radiance for points of the point cloud.The scene points of the image region may be points of the point cloud.

[0034] According to an optional feature of the invention, the representation of the three dimensional scene is a representation comprising at least the first image and projection data indicating a relationship between positions of light property values ​​of the first image and positions of corresponding scene points in the three dimensional scene.

[0035] In many embodiments, this may provide improved performance and / or operation.

[0036] The corresponding scene point of the light property value position is a scene point for which the light property value at the light property value position provides an indication of the light property value of the scene point.The light property value position may be a pixel position in the first image and the light property value may be a pixel value.

[0037] According to an optional feature of the invention, the generator is arranged to receive input 3D image data of the 3D scene and select a subset of the input 3D image data for inclusion in the 3D image data of the data signal, wherein the selection of the subset depends on the view dependency indication.

[0038] In many embodiments, this may provide improved performance and / or operation. In many embodiments, a feature may provide an improved data signal, allowing for higher image quality rendering at a lower data rate. The feature may allow for an improved tradeoff between image data for high quality rendering and the data rate of the data signal.

[0039] In some embodiments, the generator may be arranged to select the subset to have a size that increases monotonically with the degree of view dependency indicated by the view dependency indication.

[0040] According to an optional feature of the invention, the generator is arranged to include the view dependency indication in a chrominance colour channel of an image for the three dimensional image data.

[0041] In many embodiments, this may provide improved performance and / or operation.

[0042] According to an aspect of the present invention, there is provided an apparatus for rendering a view image of a three-dimensional scene, the apparatus comprising: a first receiver arranged to receive a data signal, the data signal comprising: three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; at least one view dependency indication for an image area of ​​the first image, the view dependency indication indicating a degree of change of one or more visual properties of scene points of the image area according to a viewing direction; a renderer arranged to generate a view image for the view pose according to the three-dimensional image data and according to the view dependency indication.

[0043] In many embodiments, the present invention may provide improved performance and / or operation and / or implementation. It may generally allow for improved representation of a scene, thereby allowing for more accurate and higher quality view images of the scene to be drawn specifically.

[0044] According to an optional feature of the invention, the renderer is arranged to generate a value of a pixel of the view image by blending contributions from a plurality of light property values ​​of the three dimensional image data projected onto the position of the pixel in the view image, the blending of contributions from the light property values ​​from the image area being dependent on the view dependency indication.

[0045] This can provide particularly advantageous operation in many scenarios and embodiments. The method can allow improved rendering, and in particular improved / more realistic images. It can generally provide improved perceived image quality, including, for example, less image noise often generated by blending operations.

[0046] The weight of the contribution of light property values ​​from an image region in the blending / fusion of values ​​for generating pixels of a view image depends on the view dependency indication of the image region.

[0047] According to an optional feature of the invention, the three dimensional image data comprises transparency values, and wherein the renderer is arranged to modify at least one transparency value in dependence on the view dependency indication.

[0048] This may provide particularly advantageous operation in many scenarios and embodiments.

[0049] According to an optional feature of the invention, the renderer is further arranged to modify at least one transparency value in dependence on a difference between a light direction of the pixel of the at least one transparency value and a direction from a position in the three dimensional scene represented by the pixel and the view pose.

[0050] This may provide particularly advantageous operation in many scenarios and embodiments.

[0051] According to one aspect of the present invention, there is provided a method for generating a data signal describing a three-dimensional scene (excluding a method for performing such mental behavior), the method comprising: generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; generating a view dependency indication for an image area of ​​the first image, the view dependency indication indicating the degree to which one or more visual properties of scene points of the image area change according to a viewing direction; and including the view dependency indication in the data signal.

[0052] According to one aspect of the present invention, a method for drawing a view image of a three-dimensional scene (excluding a method for performing such mental behavior) is provided, the method comprising: receiving a data signal, the data signal comprising: three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; at least one view dependency indication for an image area of ​​the first image, the view dependency indication indicating the degree of change of one or more visual properties of scene points in the image area according to a viewing direction; receiving a view pose for the three-dimensional scene; and generating a view image for the view pose based on the three-dimensional image data and based on the view dependency indication.

[0053] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which

[0055] Figure 1 illustrates an example of an apparatus for generating a data signal providing a representation of a scene according to some embodiments of the present invention;

[0056] Figure 2 illustrates an example of an apparatus for rendering an image from a data signal providing a representation of a scene according to some embodiments of the present invention; and

[0057] Figure 3 The diagram shows the Figure 1 or Figure 2 An example of a processor of a device. DETAILED DESCRIPTION

[0058] Three-dimensional video capture, distribution and presentation are becoming increasingly popular and desirable in some applications and services. Specific methods are referred to as immersive video, and typically include providing a view of a real-world scene (and often a real-time event) that allows small viewer movements, such as relatively small head movements and rotations. For example, allowing a live video broadcast of, for example, a sporting event to be generated based on a view of a local client after a small head movement of the viewer can provide the impression of a user sitting in the stands watching the sporting event. The user can, for example, look around and will have a natural experience similar to the experience of the audience existing at that position in the stands. Recently, display devices with position tracking and three-dimensional interaction support applications based on three-dimensional capture of real-world scenes have become increasingly popular. Such display devices are very suitable for immersive video applications that provide an enhanced three-dimensional user experience.

[0059] In order to provide such services for real-world scenes, a suitable data representation of the scene must be generated and delivered to the device that presents the view to the end user.

[0060] Typically, scenes are captured from different locations and using different camera capture poses. As a result, the relevance and importance of multi-camera capture and, for example, 6DoF (6 degrees of freedom) processing are increasing rapidly. Applications include live concerts, live sports, and telepresence. The freedom to choose your own viewpoint enriches these applications by increasing the feeling of being above regular video. In addition, immersive scenarios can be envisioned, in which the observer can navigate and interact with the live captured scene. For broadcast applications, this may require real-time depth estimation on the production side and real-time view synthesis at the client device. Both depth estimation and view synthesis introduce errors, and these errors depend on the implementation details of the algorithm.

[0061] In the art, the terms placement and posture are used as general terms for position and / or direction / orientation. For example, the combination of the position and direction / orientation of an object, camera, head or view can be referred to as a posture or placement. Therefore, a placement or posture indication can include six values / components / degrees of freedom, wherein each value / component generally describes an individual property of the position / position or orientation / orientation of the corresponding object. Of course, in many cases, a placement or posture can be considered or represented with fewer components, for example, if one or more components are considered to be fixed or unrelated (for example, if all objects are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the posture of the object). In the following, the term posture is used to refer to a position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom). The term posture can be replaced by the term placement. The term posture can be replaced by the term position and / or orientation. The term posture can be replaced by the term position and orientation (if the posture provides information on both position and orientation), by the term position (if the posture provides information on (possibly only) position) or by the orientation (if the posture provides information on (possibly only) orientation).

[0062] Figure 1 An example of a data signal generating apparatus arranged to generate a data signal comprising a representation of a three-dimensional scene is illustrated. Figure 2 An example of a rendering device for rendering a view image of a three-dimensional scene based on a data signal comprising a representation of the scene is illustrated. Specifically, in this example, a signal generator device generates a data signal, which is provided to Figure 2 The drawing device proceeds to the drawing device based on Figure 1 The signal generating device generates scene representation data of the signal to draw the view image of the scene. The method will be described with reference to this example.

[0063] The data signal generating apparatus comprises an image data source 101, which is arranged to provide three-dimensional image data providing a representation of a three-dimensional scene. The image data source 101 may, for example, comprise a storage device or memory from which the three-dimensional image data is stored and from which the three-dimensional image data may be retrieved. In other embodiments, the image data source 101 may receive or retrieve the three-dimensional image data from an external source (or other internal source). In many embodiments, the image data source 101 may, for example, be received from a video capture system comprising a camera that captures a real-world scene (e.g., in real time).

[0064] The three-dimensional image data provides a suitable representation of the scene using a suitable image or generally video format / representation. In some embodiments, the image data source 101 may receive image data from different sources and / or in different formats, and thereby generate the three-dimensional image data, for example, by converting or processing the received scene data. For example, an image and depth may be received from a remote camera, and may be processed to generate a video representation according to a given format. In some embodiments, the image data source 101 may be arranged to generate the three-dimensional image data by evaluating a model of the scene.

[0065] The three-dimensional image data is provided according to a suitable three-dimensional data format / representation. The representation may be, for example, a multi-view and depth, multi-layer (multi-plane, multi-sphere), mesh model and / or point cloud representation. Furthermore, in the described examples, the three-dimensional image data is in particular video data and may therefore include a temporal component.

[0066] The three-dimensional image data is provided according to a representation comprising one or more images. In many embodiments where the three-dimensional image data represents video data, the three-dimensional image data may comprise a time series of images. An image may be any two-dimensional structure of light property values ​​of scene points of a scene.

[0067] An image may, for example, comprise a view image of a scene, a frame of a video signal, a video atlas, a picture, a coded picture, a decoded picture, etc.

[0068] The light property values ​​may be, for example, brightness, color, chroma, lightness, chromaticity and / or luminance values. The light property values ​​of a scene point may provide an indication of light radiating from the scene point in at least one direction. In some representations, each scene point may be associated with a plurality of light property values. A scene point represented by a light property value of 3D image data may also be referred to as a scene sample, an image sample, or just a sample.

[0069] The image data source 101 is coupled to a generator 103, which is arranged to generate a data signal, such as a bitstream, comprising three-dimensional image data. The data signal / bitstream is typically a (three-dimensional) video data signal / bitstream. The generator 103 may specifically generate a bitstream according to a suitable transmission format and comprising three-dimensional image data.

[0070] The generator 103 is coupled to a transmitter 105 which is arranged to send the data signal to a remote source. The transmitter 105 may for example comprise a network interface to allow the data signal to be sent to a suitable destination via a network. The network may be or may comprise the Internet.

[0071] The rendering device comprises a first receiver 201, which is arranged to receive the data signal from the data signal generating device. The receiver may be any suitable function for receiving three-dimensional image data according to the preferences and requirements of individual embodiments. The first receiver 201 may specifically include a network interface allowing the image data source 101 to be received via a network (such as (or including) the Internet).

[0072] The first receiver 201 is arranged to extract three-dimensional image data from the received data signal.The first receiver 201 is coupled to a renderer 203, which is arranged to generate view images for different view gestures from the three-dimensional image data.

[0073] Figure 2 The rendering apparatus also includes a second receiver 205, which is arranged to receive a view gesture for a viewer / user (and in particular, a view gesture in a three-dimensional scene). The view gesture represents the position and / or orientation of the viewer viewing the scene, and it may in particular provide a gesture of a view of the scene that should be generated.

[0074] It will be appreciated that many different methods for determining and providing a view pose are known, and any suitable method may be used. For example, the second receiver 205 may be arranged to receive pose data from a VR headset worn by a user, from an eye tracker, etc. In some embodiments, a relative view pose may be determined (e.g., a change from an initial pose may be determined), and this may be relative to a reference pose (e.g., a camera pose or the center of a capture pose area).

[0075] The second receiver 205 is coupled to the renderer 203, which is arranged to generate a view frame / image based on the three-dimensional image / video data, wherein the view image is generated to represent the view of the three-dimensional scene from the view gesture. Therefore, the renderer 203 can specifically generate a video stream of view images / frames of the three-dimensional scene based on the received three-dimensional image data and the view gesture. In the following, the operation of the renderer 203 will be described with reference to the generation of a single image. However, it should be understood that in many embodiments, the image can be part of an image sequence, and specifically can be a frame of a video sequence. In fact, the described method can be applied to generate multiple frames / images and often all frames / images of an output video sequence.

[0076] It should be understood that a stereoscopic video sequence can often be generated that includes a video sequence for the right eye and a video sequence for the left eye. Thus, if the image is presented to the user, for example, via an AR / VR headset, it appears as if the three-dimensional scene is seen from the viewing posture. In another example, the image is presented to the user using a tablet computer with a tilt sensor, and a monoscopic video sequence can be generated. In yet another example, multiple images are woven or tiled for presentation on an autostereoscopic display.

[0077] The renderer 203 may perform appropriate image synthesis / generation operations to generate a view image of a particular representation / format of the three-dimensional image data (or indeed in some embodiments the three-dimensional image data may be converted into a different format to generate the view image).

[0078] For example, the renderer 203 may be typically arranged to perform view shifting or projection of the received multi-view images based on the depth information for a multi-view+depth representation. This will typically include techniques such as shifting pixels (changing pixel positions to reflect the appropriate disparity corresponding to the disparity change), de-occlusion (typically based on padding from other images), combining pixels from different images, etc., as known to those skilled in the art.

[0079] It will be appreciated that many algorithms and methods are known for synthesizing images from different three-dimensional image data formats and representations, and that renderer 203 may use any suitable method.

[0080] The image synthesis device can thus generate a view image / video of the scene. In addition, since the view pose can change dynamically corresponding to the user moving around in the scene, the view of the scene can be continuously updated to reflect changes in the view pose.

[0081] In the method, the three-dimensional image data is further supplemented by including a view dependency indication indicating the extent to which one or more visual properties of the image region vary depending on the viewing direction.

[0082] The view dependency indication may specifically be a Lambertian indication indicating a Lambertian level of at least one image area of ​​at least one of the images of the three-dimensional image data. The view dependency / Lambertian indication may indicate a degree of view dependency of a visual property of an image area. The view dependency / Lambertian indication may indicate a degree / level of variation of one or more visual properties / light properties for the image area according to a view direction. The view dependency / Lambertian indication may indicate a variation of one or more visual properties of the image area according to a viewing direction. The view dependency / Lambertian indication may indicate a degree / level of variation of light radiation and / or reflection of the image area according to a viewing direction. The visual property may specifically be a brightness and / or chromaticity property of the image area.

[0083] In the following, the term Lambertian indicator will be primarily used, but it should be understood that this term can be replaced by the term view-dependent indicator. For the sake of brevity, references to view-dependent / Lambertian indicators will simply be referred to as the term Lambertian indicator. Similarly, it should be understood that references to Lambertian or Lambertian can be replaced by the terms view-dependent and view-dependent as appropriate. Therefore, for the sake of brevity, the following text will use the term Lambertian to include the term view-dependent. It should also be understood that the term view-dependent includes the terms view-direction-dependent or viewing-angle-dependent.

[0084] The data signal generating device accordingly comprises a Lambert determiner 107, which is arranged to generate a Lambert indication. The Lambert determiner 107 is coupled to a data generator 103, which is fed with the Lambert indication and is arranged to include it in the data signal / bit stream. The data generator 103 may specifically encode the Lambert indication and add it to a dedicated field of the bit stream.

[0085] A Lambertian indication may indicate a variation in light output (from a scene surface / element / object / area represented by the image area) as a function of direction with respect to the scene area represented by the image area.

[0086] Lambertian reflection is the reflection of light provided by an ideal diffuse reflecting surface. The apparent brightness of a Lambertian surface to an observer is the same regardless of the viewing angle of the observer. A completely Lambertian surface will provide the same visual impression regardless of the viewing direction, such as a completely matte white surface that reflects light in all directions. Similarly, a non-collimated light source provides the same light in all directions and will therefore have the same visual appearance in all directions. For a non-collimated light source, the light has no visual dependence on the viewing direction / angle. For the purposes of this document, it does not matter if the observed light is the result of reflection or emission and therefore the non-collimated light source is actually Lambertian. The term Lambertian includes non-collimated light sources and is therefore not limited to Lambertian reflection, but also includes Lambertian or non-collimated light (source) radiation / emission.

[0087] For many practical reflective surfaces and light sources, there is a degree of viewing direction sensitivity, and some changes in light can result in changes in the viewer's direction. Therefore, there is some view dependency for this non-Lambertian effect. The degree of viewing angle dependency can vary significantly. For example, the smoother a white wall is, the less Lambertian it is, and the more visually dependent the visual impression of the wall will be. Other view-dependent effects (also called non-Lambertian effects) can include:

[0088] Smooth surfaces such as marble and wood.

[0089] Hard specular reflection.

[0090] Metal surface.

[0091] Differences between views due to polarization, glare, etc.

[0092] The data signal generating means is arranged to generate a Lambertian indication indicative of a variation in light output as a function of direction of a scene area represented by the image area.

[0093] In some embodiments, the Lambertian indication may be received from a given source, or may be generated, for example, by manual input, such as by a person selecting a particular image region and manually assigning a Lambertian level to the region. In other embodiments, information may be provided on objects in the scene, such as the materials of different objects in the scene, and corresponding Lambertian levels may be assigned based on the objects.

[0094] In some embodiments, the Lambertian determiner 107 is arranged to generate a Lambertian indication in response to the three-dimensional image data, and in particular in many embodiments, it can generate a Lambertian indication based on a comparison of light output from the image area in different directions. In many embodiments, the Lambertian determiner 107 can be arranged to determine the light radiation in different directions of the scene area represented by the image area. It can then determine the Lambertian indication based on the change in light radiation in different directions. Light from the same scene area in different directions can be compared and a similarity measure can be evaluated. For example, differences in brightness, color, luminance and / or chromaticity can be determined. The higher the determined difference, the less Lambertian the corresponding image area is. If the light is considered to be the same for all different directions, then the (one or more) image areas can be considered to be completely Lambertian.

[0095] Light in different directions may be determined in different ways in different embodiments and indeed for different representations of the scene used for the three-dimensional image data. For example, for a multi-view plus depth representation, pixel values ​​corresponding to the same scene point in different multi-view images may be compared. For a representation comprising multiple point clouds, such as obtained by using one LiDAR for each point cloud, scene points of the multiple point clouds may be registered and compared. For point clouds captured using a photogrammetry room with multiple cameras and light sources, the appearance of scene points in multiple cameras and / or with multiple light conditions may be compared.

[0096] As a specific example, the image sample for which the Lambertian property is calculated is associated with a plurality of other image samples corresponding (roughly) to the same scene point. From this set of image samples, statistics are calculated based on the color values ​​of these image samples. Suitable statistics are variance, standard deviation or value range (max-min), e.g. higher values ​​indicating more appearance variation. The Lambertian property is then determined by applying a mapping to the statistics such that higher variations result in lower Lambertian properties. To calculate the Lambertian property of an image region, the Lambertian properties of a plurality of image samples may be averaged.

[0097] In many embodiments, the image region may be an image region where scene points represented by the image region are part of the same surface in the scene. However, it will be appreciated that this is not required. For example, in some embodiments, the image may be divided into predetermined image regions / segments, wherein a view dependency / Lambertian indicator is generated for each region / segment. In this case, the view dependency indicator may simply reflect, for example, the maximum or average viewing angle dependency of the scene points of the image region. Although this approach may generally introduce some inaccuracies when compared to the ideal case and potentially reduce the rendering that may be achieved, in many practical applications it will achieve acceptable performance and image quality.

[0098] Each pixel of the image area may provide at least one light property value for a scene point. The Lambertian indication may indicate the degree to which the light from the scene point depends on the direction from which it is viewed (i.e., the direction from the viewing pose to the scene point). A scene point may often correspond to a point on a surface in the scene, and the Lambertian indication may indicate the degree of Lambertianity of the surface.

[0099] In some embodiments, the Lambertian indication may be a simple binary indication indicating whether the image region represents a Lambertian surface / point or a non-Lambertian surface / point. The Lambertian indication of a given image region of a given image may provide an indication of whether the visual information provided by the (one or more) light property values / pixel values ​​of the image region is view-dependent or view-independent.

[0100] However, in many embodiments, a more detailed indication may be provided and the Lambertian indication may reflect more levels / degrees of view dependence of an image region. For example, the Lambertian indication may be a two-bit indication indicating whether an image region represents a surface such as:

[0101] Perfectly Lambertian (for example, a white wall).

[0102] Somewhat non-Lambertian (like a marble base).

[0103] Quite non-Lambertian (e.g. fire, smoke).

[0104] Completely non-Lambertian (e.g., hard specular reflection).

[0105] In other embodiments, more levels may be used, and in such cases, the Lambertian indication may be no less than a 4-bit, 6-bit, or 8-bit value.

[0106] In some applications, only a single Lambertian indication for a single image region of a single image may be included in the bitstream. However, in many embodiments, multiple Lambertian indications may be included.

[0107] In many embodiments, Lambertian indications for multiple image regions of an image may be included. Indeed, in some applications, an image may be divided into multiple image regions, and a Lambertian indication may be provided for each image region, indicating the level / degree of Lambertianity for that image region. Each element of an image may have an associated Lambertian indication.

[0108] In some applications, the image region(s) may be predetermined and fixed. For example, the image may be divided into small rectangles, and a Lambertian indication may be provided for each rectangle. In fact, in some cases, each region may include a single pixel, and thus a Lambertian indication may be provided for each pixel of the image in some embodiments.

[0109] The bitstream may be generated to include spatially varying Lambertian indications reflecting the Lambertian properties of a surface represented by different portions of an image. In many embodiments, a Lambertian map may be generated that includes the Lambertian indications corresponding to / mapped to the image. For example, a two-dimensional map of the Lambertian indications may be provided that has the same resolution as the image. In many embodiments, the resolution of the Lambertian map may be lower than the resolution of the image.

[0110] In some embodiments, the image can be divided into image regions by a dynamic and adaptive process that can depend on the data of the image. For example, segmentation can be performed based on the image data to divide the image into regions with similar visual properties. A Lambertian indication can then be provided for each region or for a subset of regions. In some embodiments, segmentation can be performed to detect and identify objects with a Lambertian indication provided on an object-by-object basis.

[0111] Furthermore, in many embodiments, a Lambertian indication or a Lambertian indication map may be provided not only for a single image, but may be applied to two, more, or all images used to represent a three-dimensional scene at a given point in time.

[0112] In addition, for a video sequence, a Lambertian indication or a Lambertian indication map may be provided for an image at different times. In some embodiments, one or more Lambertian indication maps may be provided for each image / frame of a video sequence. In this case, the Lambertian indication may be determined, for example, taking into account temporal aspects and correlations across different moments. Similarly, the encoding of the Lambertian indication may include, for example, relative encoding. For example, a Lambertian indication map for one image / frame may be encoded as a relative value relative to a Lambertian indication map for a corresponding previous image / frame.

[0113] The renderer 203 is arranged to perform image generation in response to the Lambertian indication. Specifically, the generation of pixel values ​​for the view image may include, for some pixels, consideration of multiple scene points and light property values ​​(i.e., multiple samples) from the three-dimensional image data. For example, when multiple samples are projected to the same pixel position in the view image, the pixel value may be determined by combining contributions from multiple light property values. In many applications, the renderer may perform a blending operation to combine multiple operations.

[0114] When such a combination includes samples / light property values ​​from an image area that provides a Lambertian indication, the combination may be dependent on the Lambertian indication. Specifically, in many cases, the determination of a pixel value of a view image may include a combination / merging / mixing of multiple light property values ​​of one or more samples of three-dimensional image data (including samples from different images). For example, when a scene point including a light property value of the three-dimensional image data is projected to an image position of the view image, multiple scene points may be projected to the same pixel position. Multiple scene points of the three-dimensional image data may provide visual information of the image position. Multiple pixel values / visual data points may then be combined to generate a pixel value of the view image.

[0115] The combination may include taking into account the depth of the corresponding scene points. For example, if the scene points are at very different depths, the frontmost scene point may be selected and its pixel value may be used for the view image pixel value. However, if the depth values ​​are close together, all pixel values ​​and scene points may be correlated, and the view image pixel value may be generated as a mixture including contributions from multiple scene points. In many cases, multiple scene points and light property values ​​may actually represent the same surface. For example, for a multi-view plus depth representation, the scene point / pixel value may represent the same surface seen from different viewpoints.

[0116] The combination / merging / mixing of multiple pixel values ​​may also depend on the Lambertian indication. For example, when the Lambertian indication indicates that one or more pixel values ​​are associated with a very high degree of Lambertianity, then the combination may be a selection combination, wherein the pixel values ​​are selected from the preferred image, for example. In contrast, if the Lambertian indication indicates a low degree of Lambertianity, then the combination may include contributions from all pixel values. For example, the weighting of each contribution may depend on the proximity of the corresponding viewing direction to the viewing direction of the multi-view image. In many embodiments, this approach may provide improved view image quality.

[0117] In some cases, the renderer 203 may be arranged to switch between selection blending and combination blending according to the Lambertian indication.In some embodiments, the renderer may be arranged to determine blending weights of contributions from scene points of the image region to pixels being blended according to the Lambertian indication.

[0118] Thus, in many embodiments, the renderer 203 may be arranged to generate light property values ​​for pixels of the view image by blending contributions from different light property values ​​of the three-dimensional image data projected to the position of the pixel in the viewport of the view image. For example, the projection of a scene point may result in a given position in the viewport of the image for a given view pose (and this at a given (pixel) position in the view image). Light property values ​​of scene points projected to the same pixel may be blended together to generate an image / pixel value for that pixel in the view image. Such blending may cause the contribution of image values ​​within a given image region to depend on the Lambertian indication of that image region.

[0119] In many embodiments, the renderer 203 may be arranged to take into account transparency values ​​when synthesizing (one or more) view images. In many applications, an image may be associated with a transparency / alpha map, which includes a transparency / alpha value indicating the transparency of the associated pixel. When combining / mixing different light property values, the mixture may be weighted based on the contribution of the transparency value to the different light property values. For example, for a transparency value of 1 - corresponding to full opacity - if it is the frontmost pixel, the contribution of the light property value may be 100%. For a transparency value of 0 - corresponding to full transparency - the contribution of the light property value may be 0%. For transparency values ​​between 0 and 1 - corresponding to partial transparency - the contribution may be between 0% and 100%. Such a mixture may allow the representation of partially transparent objects, such as frosted glass or some plastics.

[0120] In an embodiment where a transparency / alpha value is provided, the renderer 203 may be arranged to modify one or more of the transparency values ​​in response to (one or more) Lambertian indications. For example, for a given scene point and light property value of the three-dimensional image data, a Lambertian indication may be used to modify the transparency value of the light property value.

[0121] As a specific example, a transparency value can be used to perform the above blending / selection. For pixel values ​​at substantially the same depth and where Lambertianity is high, a pixel can be selected by setting the transparency value to indicate no transparency, while for pixel values ​​with less Lambertianity, it can be set to a lower level to allow blending between different values.

[0122] As another example, a more gradual modification of transparency based on the degree of Lambertianity may be accomplished by, for example, keeping transparency unchanged for full Lambertian and increasing transparency (ie, decreasing opacity) based on having both lower Lambertian values ​​and larger differences between the original and virtual view poses.

[0123] exist Figure 2In the example of , the renderer 203 accordingly includes an alpha processor 207 that receives the Lambertian indication and the transparency value. It can then proceed to modify the transparency value based on the Lambertian indication. The alpha processor 207 is coupled to a mixer 209, which is arranged to mix multiple light property values ​​to generate pixel values ​​of a view image, wherein the mixing depends on the transparency value. The mixer 209 may also include functionality for performing a related projection operation to determine a sample of the three-dimensional image data to include in the mixing for a given pixel of the view image. The mixer is also coupled to a view image generator 211, which generates a view image by combining individual pixel values.

[0124] As a specific example that may be applicable to many types of representations (e.g., including multi-view, video-based point clouds, meshes, etc.), the blending of image values / samples at similar depths in the viewport of the view image can use a transparency (α) value that can be modified according to the Lambertian indication. For example, if a Lambertian indication is provided but no transparency is indicated (fully opaque), the output of the blending operation can be a single opaque value (R, G, B, α=1). This can be achieved by scaling the contributing α values. Otherwise, the transparency / α values ​​can also be blended, for example using a saturated addition operation (to constrain α to the [0,1] range).

[0125] In many embodiments, if only a single sample / image value is projected to a given position in the view image viewport, then the Lambertian indication may have little effect or may not actually be considered. However, when multiple samples (close enough) are projected to the same scene point, the contributions may generally be blended to advantage. In many embodiments, the renderer 203 may be able to distinguish scene points along the same ray (one in front of the other), and may generally blend only scene points of similar depth.

[0126] A specific example of a possible method for modifying the transparency value according to Lambertian indication is as follows:

[0127]

[0128] in:

[0129] x is an integer or fractional sample coordinate in the scene.

[0130] α0(x) is the value of the transparency component or the transparency value automatically calculated based on the texture and geometry components (drawn in two layers). When there is no transparency information, α0(x) = 1

[0131] L(x) is the value of the Lambertian indicator at position x.

[0132] β is a measure of the difference in ray direction between a sample and a viewport. A given sample of three-dimensional image data may provide a light property value for the scene. Such a value is typically explicitly or implicitly associated with a ray direction. For example, for an image of a multi-view representation, the ray direction is from the scene point of the sample to the capture point. For many images, the ray direction may be perpendicular to the image plane. For some representations, such as point clouds, multiple light property values ​​may be provided for each point, and each of these values ​​may be associated with a ray direction. β may be a value that reflects the difference between the ray direction of the light property value / sample and the ray direction from the corresponding scene point to the view pose of the view image.

[0133] For some constants c1 and c2 is an example equation for the Lambertian indication of the effect of the modification on the transparency value. An alternative could be a piecewise linear curve of predetermined or transmitted coefficients. Also, the exponent could be replaced by another, e.g. non-linear function, such as a scaled sigmoid function.

[0134] There are multiple ways to encode and derive the ray disparity value β, and this may generally depend on the chosen volume representation. The ray disparity value β always depends on the viewport / view pose of the view image. Typically, the ray disparity value β may be a positive angle, i.e. a value related to this, such as b=sinβ.

[0135] For perspective and / or orthographic projection representations, such as MPEG Immersive Video (MIV), the ray disparity value β can be determined as the angle between two vectors, where the first vector is from the base point of the source view to the scene point, and the second vector is from the viewport / view pose (the base point of the virtual camera generating the view image) to the scene point.

[0136] For some orthogonal projections, such as some examples of MIV or point cloud representations, the ray disparity value β can be determined as the angle between two vectors, where the first vector is the normal vector to the orthogonal plane and the second vector is from the viewport / view pose (the base point of the virtual camera that generates the view image) to the scene point.

[0137] In some embodiments, additional metadata may be provided for the three-dimensional image data, which indicates the ray direction of a sample of the three-dimensional image data. Such metadata may, for example, include a table with source view positions or direction vectors. The metadata for the image region may then include an identity or index into the table. The ray disparity value β may then be determined based on the ray direction of a given sample relative to the direction to the view pose.

[0138] The method may provide advantageous operation and performance in many scenarios and applications. In particular, the method may provide an improved view image to be generated in many scenarios. The method may result in a lower bit rate at a given visual quality level of the view image. The method may result in a lower pixel rate at a given visual quality level of the view image. The method may result in a lower drawing complexity in terms of instruction count or battery consumption at a given quality level of the view image.

[0139] The method also allows for efficient implementation and operation. In many cases, the method can be implemented with low complexity and relatively low computational overhead. In addition, many existing devices and systems require relatively few modifications, and therefore a high degree of backward compatibility and simple modifications can generally be achieved.

[0140] The method may be adapted to a range of different representations, such as multi-view and depth representations / formats as described above.

[0141] The described method can be very suitable for the representation of a scene, which is a representation of a scene comprising one or generally multiple images with pixels indicating light properties / light radiation intensity and also comprising projection data indicating the relationship between the pixels of the images and the spatial positions in the three-dimensional scene. Thus, a representation can be used, in which spatial / geometric information relating image pixels to scene points is implicitly and / or explicitly provided. Such data can be effectively used to determine suitable rendering operations, such as blending and determination of transparency values ​​as described above in particular. The resulting transparency value can then be used to perform the blending operation.

[0142] The method may be particularly suitable for volumetric representations, such as multi-plane image representations or point cloud representations. A volumetric representation may specifically be a representation that describes the geometry and properties (e.g. color, reflectivity, material) of one or more objects within the volume. A problem with pure volumetric representations is that they typically do not represent view dependent / non-Lambertian effects, and often cannot represent view dependent / non-Lambertian effects. The described method allows the communication and rendering of such effects also for volumetric representations.

[0143] In many embodiments, a multi-plane image representation may be used, and images including image regions that provide Lambertian indications may be planes of the multi-plane image representation. In particular, each plane of the multi-plane image will typically have only a few image regions that are not completely translucent. These image regions of one or more planes may be packed together into an atlas, and the atlas may have an image (or video) component for each attribute of the multi-plane image. Typically, the attributes are color and transparency. Typically, only specific areas in the scene are non-Lambertian, and only some image regions correspond to these scene areas. These specific image regions may be packed multiple times, with different indications of the original view pose. Thus, this approach enables the transmission of non-Lambertian effects with a moderate increase in the size of the atlas.

[0144] In many embodiments, a point cloud representation may be used, and the image comprising the image region providing a Lambertian indication may be a (partial) projection of the point cloud representation onto the image plane. The image may be a map comprising patches of pixels indicating the light radiation for the points of the point cloud. In particular, the map encoding the point cloud will typically have only a few patches corresponding to non-Lambertian scene regions. These specific patches may be packed into the map multiple times, with different indications of the original view pose. Thus, this approach enables the transmission of non-Lambertian effects with a modest increase in the size of the map.

[0145] In some embodiments, the image data source 203 may be arranged to receive input three-dimensional image data of a scene, and then proceed to select a subset of the input for inclusion in the data signal / bitstream. The image data source 203 may be arranged to select the subset based on a Lambertian indication. In particular, for a region of a scene indicating relatively high Lambertianity and therefore low view dependence, the amount of candidate data representing the region of the scene in the bitstream may be significantly reduced relative to the case where the region is indicated as having a low level of Lambertianity. In some embodiments, the image data source 203 may be arranged to select a subset having a size that increases monotonically with the degree of view dependence indicated by the Lambertian indication. The greater the view dependence, the larger the size of the subset.

[0146] As a specific example, for a multi-view plus depth representation, a given surface may be represented by different images corresponding to different views / capture positions. If the surface is indicated as fully Lambertian, data may be included from only a single multi-view image. If the surface is indicated as the opposite, data may be included from all received multi-view images. For Lambertianities between these extremes, data may be included from only some of the multi-view images, for example.

[0147] As another example, for specular highlighting and specularity, all available (or in some cases, a (large) subset) of multi-view images may be included, and a low Lambertian indication will cause the renderer to render the viewport primarily using images that are closest in view pose to the viewport.

[0148] The Lambertian indication may be included in the data signal in any suitable manner. In many embodiments, the Lambertian indication may be included as metadata in a dedicated field of the data signal. For example, a Lambertian indication map may be included for each image represented.

[0149] In some embodiments, the generator 103 may be arranged to include the Lambertian indication(s) in the chrominance color channels of the image for the three-dimensional image data. In some embodiments, the Lambertian indications may be packed into the channels of the video component. This approach may provide an efficient method for delivering the Lambertian indications.

[0150] For example, the described approach may be well suited for representations according to the Visual Volumetric Video Based Coding (V3C) standard that is being developed.

[0151] For such applications, the current data format can be modified to include a Lambertian indication in the video component data field. For V3C, the data format and packing_information(j) syntax structure can be found in ISO / IEC DIS23090-5(2E):2021 "V3C". The packing information explains how to pack multiple video components into a single video bitstream. The described method can be modified as indicated below to enable packing of multiple components in the same location but in different channels.

[0152] In the case of video other than 4:4:4, the video component that needs to have the highest resolution can be in the first component, and the other video components can be in subsequent channels. For example:

[0153] α-component --> 0 (Y / brightness)

[0154] Lambert Indicator - Component --> 1 (Cb / Blue)

[0155] Geometry Component --> 2 (Cr / Red)

[0156] The proposed syntax change would leave this choice to the encoder.

[0157] The described syntax can be modified to include the following:

[0158]

[0159] The complete syntax is:

[0160]

[0161]

[0162] The purpose of the vps_region_dimension_offset_enabled_flag is to maintain backward compatibility. It may for example be added to the V3C parameter set as indicated below. If more tags are introduced, then there may be a vps_v3c3e_flag instead, which in turn enables the vps_region_dimension_offset_enabled_flag to be present at another bitstream position.

[0163]

[0164]

[0165]

[0166] The data signal generating means and the drawing means may be implemented in particular in a suitably programmed processor. Examples of suitable processors are provided below.

[0167] Figure 3 is a block diagram illustrating an example processor 300 according to an embodiment of the present disclosure. The processor 300 may be used to implement Figure 1 A data signal generating device or Figure 2 The processor 300 may be any suitable processor type, including but not limited to a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable array (FPGA) (where the FPGA has been programmed to form a processor), a graphics processing unit (GPU), an application specific circuit (ASIC) (where the ASIC has been designed to form a processor), or a combination thereof.

[0168] Processor 300 may include one or more cores 302. Core 302 may include one or more arithmetic logic units (ALUs) 304. In some embodiments, core 302 may include a floating point logic unit (FPLU) 306 and / or a digital signal processing unit (DSPU) 308 in addition to or instead of ALU 304.

[0169] Processor 300 may include one or more registers 312 communicatively coupled to core 302. Registers 312 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, registers 312 may be implemented using static memory. Registers may provide data, instructions, and addresses to core 302.

[0170] In some embodiments, processor 300 may include one or more levels of cache memory 310 communicatively coupled to core 302. Cache memory 310 may provide computer readable instructions to core 302 for execution. Cache memory 310 may provide data for processing by core 302. In some embodiments, computer readable instructions may have been provided to cache memory 310 by a local memory (e.g., a local memory attached to external bus 316). Cache memory 310 may be implemented with any suitable cache memory type, such as metal oxide semiconductor (MOS) memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), and / or any other suitable memory technology.

[0171] Processor 300 may include controller 314, which may control inputs to processor 300 and / or data from processor 300 to other processors and / or components included in the system. The controller 314 may control the data paths in the ALU 304, FPLU 306, and / or DSPU 308. The controller 314 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 314 may be implemented as independent gates, FPGAs, ASICs, or any other suitable technology.

[0172] Registers 312 and cache 310 may communicate with controller 314 and core 302 via internal connections 320A, 320B, 320C, and 320D. External connections may be implemented as buses, multiplexers, crossbars, and / or any other suitable connection technology.

[0173] Input and output of processor 300 may be provided via bus 316, which may include one or more conductors. Bus 316 may be communicatively coupled to one or more components of processor 300, such as controller 314, cache 310, and / or registers 312. Bus 316 may be coupled to one or more components of the system, such as previously mentioned components BBB and CCC.

[0174] The bus 316 may be coupled to one or more external memories. The external memory may include a read-only memory (ROM) 332. The ROM 332 may be a masked ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include a random access memory (RAM) 333. The RAM 333 may be a static RAM, a battery-backed static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include an electrically erasable programmable read-only memory (EEPROM) 335. The external memory may include a flash memory 334. The external memory may include a magnetic storage device such as a disk 336. In some embodiments, the external memory may be included in the system.

[0175] The present invention may be implemented in any suitable form, including hardware, software, firmware or any combination of these. The present invention may optionally be implemented at least in part as computer software running on one or more data processors and / or digital signal processors. The elements and components of an embodiment of the present invention may be implemented physically, functionally and logically in any suitable manner. In fact, the function may be implemented in a single unit, in multiple units or as part of other functional units. In this way, the present invention may be implemented in a single unit, or may be physically and functionally distributed between different units, circuits and processors.

[0176] According to standard terminology in the art, the term pixel may be used to refer to properties associated with a pixel, such as light intensity, depth, position of the portion / element of a scene represented by the pixel. For example, the depth of a pixel or pixel depth may be understood to refer to the depth of an object represented by the pixel. Similarly, the brightness of a pixel or pixel brightness may be understood to refer to the brightness of an object represented by the pixel.

[0177] Although the present invention has been described in conjunction with some embodiments, it is not intended to be limited to the specific form set forth herein. On the contrary, the scope of the present invention is limited only by the appended claims. In addition, although it may appear that features are described in conjunction with specific embodiments, those skilled in the art will recognize that various features of the described embodiments can be combined according to the present invention. In the claims, the term includes does not exclude the presence of other elements or steps. The (one or more) methods of the claim are one / multiple methods excluding one / multiple methods for performing mental acts in this way. The term image area can be (appropriately) replaced by the term image fragment, image portion, image area or continuous pixel set of the first image. In many embodiments, the image area can include no more than 95%, 90%, 70%, 50%, 30%, 10% or 5% of the pixels / area of ​​the complete first image.

[0178] In addition, although listed individually, multiple modules, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although individual features may be included in different claims, these features may be advantageously combined, and inclusion in different claims does not mean that the combination of features is not feasible and / or advantageous. Moreover, including a feature in one category of a claim does not imply a limitation on the category, but rather indicates that the feature is equally applicable to other claim categories when appropriate. In addition, the order of features in the claims does not mean that the features must be processed in any specific order, and in particular, the order of individual steps in a method claim does not mean that the steps must be performed in that order. On the contrary, the steps can be performed in any suitable order. In addition, singular references do not exclude multiples. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiples. The figure marks in the claims are provided only as clarification examples and should not be interpreted as limiting the scope of the claims in any way.

[0179] In general, examples of apparatus and methods for generating data signals are indicated by the following embodiments.

[0180] Example:

[0181] Embodiment 1: A device arranged to generate a data signal describing a three-dimensional scene, the device comprising:

[0182] A generator (101) for generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene;

[0183] a processor (107) arranged to generate a Lambertian indication for an image region of the first image, the Lambertian indication being indicative of a level of Lambertianity of scene points of the image region; and wherein

[0184] The generator (103) is arranged to include a Lambertian indication in the data signal.

[0185] Embodiment 2: The apparatus according to embodiment 1, wherein the processor (107) is arranged to generate a Lambertian indication based on the three-dimensional image data.

[0186] Embodiment 3. An apparatus according to embodiment 1 or 2, wherein the processor (107) is arranged to determine light properties of a scene area represented by the image area in different directions; and determine the Lambertian indication in response to changes in the light properties for the different directions.

[0187] Embodiment 4. The apparatus according to any preceding embodiment, wherein the Lambertian indication indicates a variation of the light radiance according to the direction of the scene point of the image area.

[0188] Embodiment 5: The apparatus according to any of the preceding embodiments, wherein the representation of the three-dimensional scene is a multi-view plus depth representation, and the first image is an image in a multi-view image set of the multi-view plus depth representation.

[0189] Embodiment 6. An apparatus according to any preceding embodiment, wherein the representation of the three-dimensional scene is a multi-plane image representation, and the first image is a plane of the multi-plane image representation.

[0190] Embodiment 7. An apparatus according to any preceding embodiment, wherein the representation of the three-dimensional scene is a point cloud representation, and the first image includes light property values ​​for projecting at least part of the point cloud representation onto an image plane.

[0191] Embodiment 8. An apparatus according to any of the preceding embodiments, wherein the representation of the three-dimensional scene is a representation of projection data including at least the first image and an indication of a relationship between positions of light property values ​​of the first image and positions of corresponding scene points in the three-dimensional scene.

[0192] Embodiment 9. An apparatus according to any preceding embodiment, wherein the generator (101) is arranged to receive input three-dimensional image data of a three-dimensional scene and select a subset of the input three-dimensional image data to be included in the three-dimensional image data of the data signal, wherein the selection of the subset depends on a Lambertian indication.

[0193] Embodiment 10. The apparatus according to any preceding embodiment, wherein the generator (101) is arranged to include the Lambertian indication in a chrominance color channel of an image for the three-dimensional image data.

[0194] Embodiment 11: A device for drawing a view image of a three-dimensional scene, the device comprising:

[0195] A first receiver (201) is arranged to receive a data signal, the data signal comprising:

[0196] three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene;

[0197] at least one Lambertian indication for an image area of ​​the first image, the Lambertian indication being indicative of a level of Lambertianness of the image area;

[0198] a receiver (205) arranged to receive a view gesture for the three-dimensional scene;

[0199] A renderer (203) is arranged to generate a view image for the view gesture based on the three-dimensional image data and based on the Lambertian indication.

[0200] Embodiment 12. An apparatus according to embodiment 11, wherein the renderer (203) is arranged to generate a value of a pixel of the view image by mixing contributions from multiple optical property values ​​of three-dimensional image data at positions of pixels projected into the view image, the mixing of contributions from optical property values ​​from image areas depending on a Lambertian indication.

[0201] Embodiment 13. An apparatus according to embodiment 11 or 12, wherein the three-dimensional image data comprises transparency values, and wherein the renderer (203) is arranged to modify at least one transparency value according to a Lambertian indication.

[0202] Embodiment 14. An apparatus according to embodiment 13, wherein the renderer (203) is further arranged to modify the at least one transparency value based on a difference between a light direction of a pixel of the at least one transparency value and a direction from a position in the three-dimensional scene represented by the pixel and the view posture.

[0203] Embodiment 15: A method for generating a data signal describing a three-dimensional scene, the method comprising:

[0204] generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene;

[0205] generating a Lambertian indication for an image region of the first image, the Lambertian indication indicating a level of Lambertianity for scene points of the image region; and

[0206] The Lambertian indication is included in the data signal.

[0207] Embodiment 16: A method for drawing a view image of a three-dimensional scene, the method comprising:

[0208] receiving a data signal, wherein the data signal comprises:

[0209] three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene;

[0210] at least one Lambertian indication for an image area of ​​the first image, the Lambertian indication being indicative of a level of Lambertianness of the image area;

[0211] receiving a view pose for the three-dimensional scene;

[0212] A view image for the view gesture is generated based on the three-dimensional image data and based on the Lambertian indication.

[0213] Embodiment 17. A data signal providing a representation of a three-dimensional scene, the data signal comprising: three-dimensional image data providing a representation of the three-dimensional scene, including at least a first image, the three-dimensional image data providing visual information of the scene; at least one Lambertian indication at an image region of the first image, the Lambertian indication indicating a Lambertian level of the image region.

[0214] More particularly, the invention is defined by the appended claims.

Claims

1. An apparatus arranged to generate a data signal describing a three-dimensional scene, the apparatus comprising: A generator (101) for generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; a processor (107) arranged to generate a view dependency indication for an image area of ​​the first image, the view dependency indication indicating the extent to which one or more visual properties of scene points of the image area vary as a function of viewing direction; and wherein The generator (103) is arranged to include the view dependency indication in the data signal.

2. The device according to claim 1, wherein: The processor (107) is arranged to: determine light properties in different directions for a scene area represented by the image area; And determining the view dependency indication in response to a change in the light property for the different directions.

3. A device according to any preceding claim, wherein: The view dependency indication indicates a variation of radiance according to direction of the scene point for the image area.

4. A device according to any preceding claim, wherein: The representation of the three-dimensional scene is a multi-view plus depth representation, and the first image is an image of a set of multi-view images of the multi-view plus depth representation.

5. A device according to any preceding claim, wherein: The representation of the three-dimensional scene is a multi-plane image representation, and the first image is a plane of the multi-plane image representation.

6. An apparatus according to any preceding claim, wherein: The representation of the three-dimensional scene is a point cloud representation, and the first image comprises light property values ​​for projecting at least part of the point cloud representation onto an image plane.

7. A device according to any preceding claim, wherein: The representation of the three-dimensional scene is a representation comprising at least the first image and projection data indicating a relationship between positions of light property values ​​of the first image and positions of corresponding scene points in the three-dimensional scene.

8. An apparatus according to any preceding claim, wherein: The generator (101) is arranged to receive input 3D image data for the 3D scene and select a subset of the input 3D image data for inclusion in the 3D image data of the data signal, wherein the selection of the subset depends on the view dependency indication.

9. An apparatus according to any preceding claim, wherein: The generator (101) is arranged to include the view dependency indication in a chrominance color channel of an image for the three-dimensional image data.

10. A device for rendering a view image of a three-dimensional scene, the device comprising: A first receiver (201) is arranged to receive a data signal, the data signal comprising: three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; at least one view dependency indication for an image area of ​​the first image, the view dependency indication indicating a degree to which one or more visual properties of scene points of the image area vary as a function of a viewing direction; a receiver (205) arranged to receive a view gesture for the three-dimensional scene; A renderer (203) is arranged to generate a view image for the view gesture based on the three-dimensional image data and based on the view dependency indication.

11. The device according to claim 10, wherein: The renderer (203) is arranged to generate a value for a pixel of the view image by blending contributions from a plurality of light property values ​​of the three-dimensional image data projected onto the position of the pixel in the view image, the blending of contributions from the light property values ​​of the image area being dependent on the view dependency indication.

12. A method for generating a data signal describing a three-dimensional scene, the method comprising: generating the data signal to include three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; generating a view dependency indication for an image area of ​​the first image, the view dependency indication indicating an extent to which one or more visual properties of scene points of the image area vary as a function of viewing direction; as well as The view dependency indication is included in the data signal.

13. A method for drawing a view image of a three-dimensional scene, the method comprising: receiving a data signal, wherein the data signal comprises: three-dimensional image data providing a representation of the three-dimensional scene, the three-dimensional image data comprising at least a first image, the first image comprising light property values ​​for scene points in the scene; at least one view dependency indication for an image area of ​​the first image, the view dependency indication indicating a degree to which one or more visual properties of scene points of the image area vary as a function of a viewing direction; receiving a view pose for the three-dimensional scene; A view image for the view pose is generated based on the three-dimensional image data and based on the view dependency indication.

14. A computer program product comprising computer program code means adapted to perform all the steps of claim 12 or 13 when said program is run on a computer.

15. A data signal providing a representation of a three-dimensional scene, the data signal comprising: three-dimensional image data providing a representation of a three-dimensional scene, said three-dimensional image data including at least a first image providing visual information of said scene; At least one view dependency indication for an image area of ​​the first image, the view dependency indication indicating to what extent one or more visual properties of scene points of the image area vary as a function of a viewing direction.

Images