Image synthesis system and method therefor

Abstract

The image synthesis device comprises a first receiver 201 for receiving three-dimensional image data describing at least a portion of a three-dimensional scene, and a second receiver 203 for receiving a view pose for a viewer. An image region circuit 207 identifies at least a first image region in the three-dimensional image data, and a depth circuit 209 identifies a depth indication for the first image region from depth data of the three-dimensional image data. A region circuit 211 identifies a first region for the first image region. A view synthesis circuit 205 generates a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from the view pose. The view synthesis circuit 205 is configured to adapt a transparency for the first image region in the view image depending on the depth indication and a distance between the view pose and the first region.

Description

[Technical field]

[0001] The present invention relates to image synthesis systems, and in particular, but not exclusively, to an image synthesis device supporting view synthesis for immersive video applications. [Background technology]

[0002] The variety and range of image and video applications has increased significantly in recent years, as new services and techniques for using and consuming video continue to be developed and introduced.

[0003] For example, one service that is gaining popularity is the presentation of image sequences in a manner that allows the viewer to actively and dynamically interact with the system to change the parameters of the rendering. A very attractive feature in many applications is the ability to change the viewer's effective viewing position and direction, e.g., allowing the viewer to move around and look around in the scene being presented.

[0004] Such features may allow a particularly virtual reality experience to be provided to the user, which allows the user to, for example, move around (relatively) freely in the virtual environment and dynamically change his / her position and where he / she is looking. Typically, such virtual reality (VR) applications are based on a three-dimensional model of the scene, with the model being dynamically evaluated to provide a specific requested view. This approach is well known, for example, in gaming applications for computers and consoles, for example in the category of first-person shooters, other examples include augmented reality (AR) or mixed reality (MR) applications.

[0005] An example of a proposed video service or application is immersive video, where the video is played, for example, in a VR headset, to provide a three-dimensional experience. In the case of immersive video, the viewer can freely look around and move around in the presented scene, which is then perceived as being seen from different viewpoints. However, in many typical approaches, the amount of movement is limited to a relatively small area around a nominal viewpoint, for example, which typically corresponds to the viewpoint from which the video capture of the scene was performed. In such applications, three-dimensional scene information is often provided that allows high-quality view image synthesis for viewpoints relatively close to the reference viewpoint, although this deteriorates if the viewpoint deviates too much from the reference viewpoint.

[0006] Immersive video is often referred to as Six Degrees of Freedom (6DoF) or 3DoF+ video. MPEG Immersive Video (MIV) [1] is an emerging standard in which metadata is used on top of existing video codecs to enable and standardize immersive video.

[0007] A problem with immersive video is that the view space, the 3d space in which the viewer has a sufficient quality 6DoF experience, is limited. As the viewer moves outside the view space, the degradation and errors introduced by synthesizing the view images become increasingly large, resulting in an unacceptable user experience. Errors, artifacts, and inaccuracies in the generated view images occur especially because the provided 3D video data does not provide enough information (e.g. de-occlusion data) for view synthesis.

[0008] For example, immersive video data is provided in the form of a multi-view with depth (MVD) representation of a scene. The scene is captured by many spatially distinct cameras, and the captured images are provided together with a depth map. However, as the viewpoints become more and more different from the reference viewpoints from which the MVD data is captured, the likelihood that such a representation will not contain sufficient image data for the areas to be de-occluded increases significantly. Thus, as the viewer moves away from the nominal position, image portions that must be de-occluded to the new viewpoint, but are missing from the source view, cannot be directly synthesized from image data describing such image portions. Furthermore, incomplete depth maps introduce distortions when performing view synthesis, especially as part of view warping, which is an integral part of the synthesis process. The further the synthesized viewpoint is from the original camera viewpoint, the more severe the distortions in the synthesized view. Thus, as the user moves out of the viewing space, the quality of the rendered view images deteriorates, and the quality is typically unacceptable even for relatively small movements outside the viewing space.

[0009] To solve this fundamental problem, the 5th Working Draft of the MPEG Immersive Video (MIV) standard ISO / IEC JTC1 SC29 WG11 (MPEG) N19212 includes a proposal for handling such movements outside the viewing space. The standard proposes different actions and modes to be implemented when the viewer moves outside the viewing space.

[0010] [Table 1]

[0011] However, while these approaches provide desirable performance in some scenarios, they tend not to be ideal for all applications and services. These approaches are particularly relatively complex or impractical, and often result in a less than optimal user experience. In particular, the VHM_RENDER and VHM_EXTRAP modes result in a distorted view but keep the viewer oriented, whereas the VHM_FADE, VHM_RESET, VHM_STRETCH and VHM_ROTATE modes prevent distortion but at best break the immersion or even make the viewer feel disoriented.

[0012] Accordingly, improved approaches would be beneficial, particularly approaches that allow for improved operation, improved flexibility, an improved and more immersive user experience, reduced complexity, smoothed implementation, improved synthetic image quality, improved rendering, increased (possibly virtual) freedom of movement for the user, improved user experience, and / or improved performance and / or operation. Summary of the Invention [Problem to be solved by the invention]

[0013] SUMMARY OF THE DISCLOSURE Accordingly, the invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination. [Means for solving the problem]

[0014] According to one aspect of the invention, there is provided an image synthesis apparatus comprising: a first receiver configured to receive three-dimensional image data describing at least a portion of a three-dimensional scene; an image region circuit configured to identify at least a first image region in the three-dimensional image data; a depth circuit configured to identify a depth indication for the first image region from depth data of the three-dimensional image data for the first image region; a region circuit configured to identify the first region for the first image region; a second receiver configured to receive a view pose for a viewer; and a view synthesis circuit configured to generate a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from a view pose, the view synthesis circuit configured to adapt a transparency of the first image region in the view image depending on the depth indication and a distance between the view pose and the first region, the view synthesis circuit configured to increase the transparency as the distance between the view pose and the first region increases and as the depth indication indicates less depth for the first image region.

[0015] The present invention provides an improved user experience in many embodiments and scenarios. The present invention realizes an improved trade-off between image quality and freedom of movement, for example for AR, VR and / or MR applications. The approach often provides a more immersive user experience and is well suited for immersive video applications. The approach reduces the perception of quality degradation, for example reducing the risk that significant artifacts or errors in the view image will make the experience perceived as artificial or incomplete. The approach provides the user with an improved experience, for example with consistent and stable movement in the scene.

[0016] This approach enables improved AR / VR / MR applications, for example based on limited capture of a scene.

[0017] The transparency is semi-transparency. The first region is a set of view poses for which the three-dimensional image data is indicated to be sufficient for image synthesis. Such indication is in response to a synthesis quality criterion being met, the synthesis quality criterion including a requirement that a quality measure for an image representation of the first image region is higher than a threshold, and the image representation is generated (by the view synthesis circuitry) from the received three-dimensional data. The view synthesis circuitry is configured to identify quality measures for the image representations generated from the three-dimensional image data for different view poses. The first region is generated to include view poses for which the quality measure is higher than a threshold.

[0018] 3D image data is a complete or partial description of a 3D scene. Pose is a position and / or orientation.

[0019] The 3D image data includes a multi-view image set. The 3D image data includes depth information, e.g., a depth map, for one or more images. The 3D image data includes multiple images of a scene for different view poses. The 3D image data includes a multi-view-plus-depth (MVD) representation of a scene.

[0020] The image region corresponds to an image object. The term first image region is in some embodiments replaced by the term first image object or first scene object. The first image region is in some embodiments a pixel. The term first image region is in some embodiments replaced by the term first pixel.

[0021] Any suitable distance or difference measure may be used to determine the distance, ie, any suitable distance measure may be used for the distance between the view pose and the visibility region.

[0022] The image regions are specifically generated to correspond to regions of the input image that do not correspond to background regions. The first image regions are those that do not contain background pixels. The first image regions are image regions that represent foreground objects of the scene. Foreground objects are objects that are not background objects.

[0023] According to an optional feature of the invention, the view synthesis circuitry is configured to generate a view image that includes a fully transparent image region when a distance between the view pose and the first region is greater than a threshold.

[0024] This provides beneficial, and typically more efficient, operation, resulting in an improved user experience in many scenarios. In particular, it typically results in foreground objects becoming invisible if the view pose deviates too far from the viewing area. In particular, making foreground objects disappear, for example, rather than being presented at a significantly lower quality, provides a more intuitive experience for many users in many scenarios.

[0025] The threshold depends on the depth indication, and in some embodiments is zero.

[0026] According to an optional feature of the invention, the view synthesis circuitry is configured to generate a view image that includes image regions that are not fully transparent if the distance is not greater than a threshold.

[0027] This provides beneficial, and typically more efficient, operation and / or results in an improved user experience in many scenarios.

[0028] According to an optional feature of the invention, the view synthesis circuitry is configured to generate a view image that includes an opaque image region if the distance is not greater than a threshold.

[0029] This provides beneficial, and typically more efficient, operation and / or results in an improved user experience in many scenarios, for example, it is beneficial in many embodiments for foreground objects to be perceived as either completely present (completely opaque) or completely invisible / absent (completely transparent).

[0030] According to an optional feature of the invention, the image synthesis further comprises an image region circuit that identifies a second region relative to the first image region, and the view synthesis circuit is configured to generate a view image including an image region that is opaque when the view pose is within the second region, partially transparent when the view pose is outside the second region and within the first region, and fully transparent when the view pose is outside the first region.

[0031] This provides an improved user experience in many embodiments: the approach presents foreground objects, for example, as perceived to be fully present / opaque when the view pose is close enough to the capture pose, fully absent / transparent when the view pose is too far from the capture pose, and with gradually increasing transparency between these regions.

[0032] The second viewing area is inside / surrounded by the first area.

[0033] According to an optional feature of the invention, the first region depends on the depth indication.

[0034] This provides beneficial operation and / or an improved user experience in many embodiments.

[0035] In some embodiments, the image region circuitry is configured to adapt at least one of a shape and a size of the first viewing region in response to the depth indicia.

[0036] According to an optional feature of the invention, the first region depends on the geometric complexity of the image region.

[0037] This provides beneficial operation and / or an improved user experience in many embodiments.

[0038] In some embodiments, the image region circuitry is configured to adapt at least one of a shape and a size of the first viewing region in response to a measure of complexity of the shape.

[0039] In some embodiments, the image region circuitry is configured to adapt at least one of a shape and a size of the first viewing region in response to a parallax variation measure for the image region.

[0040] The disparity variation measure indicates the variation in disparity for pixels in an image region for a given viewpoint shift.

[0041] According to an optional feature of the invention, the first region depends on the view shift / pose change sensitivity for the image region.

[0042] This provides beneficial operation and / or an improved user experience in many embodiments.

[0043] According to an optional feature of the invention, the first region depends on an amount of de-occlusion data for the first image region included in the three-dimensional image data.

[0044] This provides beneficial operation and / or an improved user experience in many embodiments.

[0045] According to an optional feature of the invention, the function for determining transparency as a function of distance includes a hysteresis related to changes in viewing pose.

[0046] This provides beneficial operation and / or an improved user experience in many embodiments.

[0047] According to an optional feature of the invention, the three-dimensional image data further includes an indication of an image area for at least one of the input images of the three-dimensional image, and the image area circuit is configured to identify the first image area in response to the image area indication.

[0048] This provides beneficial operation and / or an improved user experience in many embodiments.The approach reduces complexity and / or computational burden in many embodiments.

[0049] According to an optional feature of the invention, the three-dimensional image data further includes an indication of a given region for at least one input image of the three-dimensional image, and the region circuit is configured to identify the first region in response to the indication of the given region.

[0050] This provides beneficial operation and / or an improved user experience in many embodiments.The approach reduces complexity and / or computational burden in many embodiments.

[0051] According to an optional feature of the invention, the view synthesis circuit is configured to select from a plurality of candidate pixel values ​​derived from different images of the multi-view image for at least a first pixel of the view image, the view synthesis circuit being configured to select the rearmost pixel if the distance is greater than a threshold and to select the frontmost pixel if the distance is less than the threshold, the rearmost pixel being associated with a depth value indicative of a depth furthest from the view pose and the frontmost pixel being associated with a depth value indicative of a depth closest to the view pose.

[0052] This provides beneficial operation and / or an improved user experience in many embodiments.

[0053] It allows particularly efficient and less complicated operation.

[0054] According to one aspect of the invention, there is an image signal including three-dimensional image data describing at least a portion of a three-dimensional scene and a data field indicating whether rendering of the three-dimensional image data should include depth indication for the image region and adapting transparency for an image region of the three-dimensional image data in the rendered image depending on the distance between the view pose for the rendered image and a reference region for the image region.

[0055] According to an optional feature of the invention, the image signal comprises at least one of an indication of an image region and a reference region.

[0056] According to one aspect of the invention, there is provided an image signal device arranged to generate an image signal as described above.

[0057] According to one aspect of the invention, there is provided a method of image synthesis comprising receiving three-dimensional image data describing at least a portion of a three-dimensional scene, identifying at least a first image region in the three-dimensional image data, identifying depth indications for the first image region from depth data in the three-dimensional image data for the first image region, identifying a first region for the first image region, receiving a view pose for a viewer, and generating a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from the view pose, wherein generating the view image comprises adapting a transparency for the first image region in the view image in response to the depth indications and a distance between the view pose and the first region, the transparency increasing as the distance between the view pose and the first region increases and as the depth indication indicates less depth for the first image region.

[0058] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0059] Embodiments of the present invention will now be described, by way of example only, with reference to the drawings in which: [Brief description of the drawings]

[0060] [Figure 1] FIG. 1 illustrates an example of image and depth capture of a 3D object. [Diagram 2] FIG. 2 illustrates example elements of an image synthesis device according to some embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0061] Three-dimensional video capture, distribution, and presentation are becoming increasingly common and desirable for several applications and services. A particular approach is known as immersive video, which typically involves providing views of real-world scenes and often real-time events that allow for small viewer movements, e.g., relatively small head movements and rotations. For example, a real-time video broadcast of, e.g., a sporting event, that allows for local client-based generation of views that follow the viewer's small head movements, provides the impression that the user is sitting in the stands while watching the sporting event. The user can, e.g., look around and have a natural experience similar to that of a spectator present at that location in the stands. Recently, display devices that include position tracking and 3D interaction support applications based on 3D capture of real-world scenes have become popular. Such display devices are well suited for immersive video applications that provide an advanced three-dimensional user experience.

[0062] To provide such services for real-world scenes, the scene is typically captured from different positions, with different camera capture poses used. As a result, the relevance and importance of multi-camera capture and, for example, 6DoF (six degrees of freedom) processing is rapidly increasing. Applications include live concerts, live sports, and telepresence. The freedom to choose one's own viewpoint improves these applications by enhancing the sense of realism beyond normal video. Furthermore, immersive scenarios can be considered, where the observer navigates and interacts with the live-captured scene. For broadcast applications, this requires real-time depth estimation at the production side and real-time view synthesis at the client device. Both depth estimation and view synthesis induce errors, which depend on the implementation details of the algorithms.

[0063] In the art, the terms configuration and pose are used as general terms for position and / or orientation. For example, a combination of position and orientation of an object, camera, head, or view is called a pose or configuration. A configuration or pose indication therefore includes six values / components / degrees of freedom, each value / component typically describing an individual characteristic of the configuration / position or orientation / direction of the corresponding object. Of course, in many situations, a configuration or pose can be considered or represented using fewer components, for example when one or more components are considered fixed or irrelevant (for example, when all objects are considered to be at the same height and horizontally oriented, four components provide a complete representation of the pose of the objects). In the following, the term pose is used to represent a position and / or orientation represented by one to six values ​​(corresponding to the maximum possible degrees of freedom). The term pose may be replaced by the term configuration. The term pose may be replaced by the term position and / or orientation. The term pose may be replaced by the terms position and orientation (if the pose provides both position and orientation information), by the term position (if the pose provides position (or possibly only position) information), or by orientation (if the pose provides orientation (or possibly only orientation) information).

[0064] An often used approach to represent a scene is known as multi-view with depth (MVD) representation and capture. In such an approach, a scene is represented by multiple images with associated depth data, where the images represent different view poses, typically from a limited capture area. The images are in practice captured by using a camera rig with multiple cameras and depth sensors.

[0065] An example of such a capture system is shown in Fig. 1. The figure shows a scene to be captured, including a scene object 101 in front of a background 103. Multiple capture cameras 105 are located in a capture area 105. The result of the capture is a representation of the 3D scene by means of multi-view images and depth representations, i.e. images and depths provided for multiple capture poses. The multi-view images and depth representations thus provide a description of the 3D scene from a capture zone. The data representing the 3D scene thus provides a representation of the 3D scene from a capture zone from which the visual data provide a description of the 3D scene.

[0066] The MVD representation is used to perform view synthesis, in which a view image of a scene from a given view pose can be generated. The view pose requires a view shift of an image of the MVD representation to the view pose so that an image of the view of the scene from the view pose can be generated and presented to a user. The view shift and synthesis is based on depth data, for example, along with a disparity shift between positions in the MVD image and the view pose image according to the depth of the corresponding object in the scene.

[0067] The quality of the generated view images depends on the image and depth information available to the view synthesis operation. The quality of the generated view images further depends on the amount of view shift required.

[0068] For example, view shifting typically results in de-occlusion of parts of an image that are not visible in, for example, the main image used for the view shift. If another image captures the de-occluded elements, such holes are filled with data from the other image, but it is also typically possible that the de-occluded image parts for the new viewpoint are still missing from other source views. In this case, view synthesis requires estimating the data based on, for example, surrounding data. The de-occlusion process is inherently prone to being a process that introduces inaccuracies, artifacts, and errors. Moreover, this tends to increase with the amount of view shift, and in particular, the probability of missing data (holes) during view synthesis increases with increasing distance from the image capture pose.

[0069] Another source of possible distortion is imperfect depth information. Often, depth information is provided by a depth map, and the depth values ​​are generated by depth estimation (e.g., by disparity estimation between source images) or by imperfect measurement (e.g., ranging), and therefore the depth values ​​contain errors and inaccuracies. View shift is based on depth information, and imperfect depth information leads to errors or inaccuracies in the synthesized image. The farther the synthesized viewpoint is from the original camera viewpoint, the more severe the distortion in the synthesized target view image.

[0070] Thus, as the view pose moves further and further from the capture pose, the quality of the synthesized image tends to degrade: if the view pose is far enough away from the capture pose, the image quality degrades to an unacceptable level and a subpar user experience is experienced.

[0071] Many different approaches have been proposed to address these issues, but they tend to be suboptimal, in particular tending to undesirably restrict user movement or introduce undesirable user effects. Figure 2 illustrates a view synthesis device / system that offers performance and an approach that may achieve a more desirable user experience in many scenarios.

[0072] 2 shows an example of an image synthesizer used to generate view images, for example for an immersive video experience. The image synthesizer comprises a first receiver 201 configured to receive 3D image data describing at least a part of a 3D scene. The 3D image data notably describes a real-world scene captured by cameras at different positions.

[0073] In many embodiments, the 3D image data includes a multi-view image, and thus includes multiple (simultaneous) images of a scene from different viewpoints. In many embodiments, the 3D image data is in the form of an image and depth map representation, and one image and an associated depth map are provided. The following description focuses on embodiments in which the 3D image data is a multi-view plus depth representation that includes at least two images from different viewpoints, at least one of the images includes an associated depth map. It is understood that if the received data is a multi-view data representation that does not include an explicit depth map, for example, the depth map may be generated using a suitable depth estimation algorithm, in particular a disparity estimation approach using different images of the multi-view representation.

[0074] Thus, in a particular example, the first receiver 201 receives MVD image data describing a 3D scene using a plurality of images and a depth map, also referred to below as source images and source depth maps, it being understood that for a video experience a temporal sequence of such 3D images is provided.

[0075] The image synthesis system further comprises a second receiver 203 configured to receive a view pose for a viewer (and in particular in a three-dimensional scene). The view pose represents a position and / or orientation from which the viewer views the scene, and in particular provides a pose of a subject from which a view of the scene has to be generated. It will be appreciated that many different approaches for identifying and providing a view pose are known and that any suitable approach may be used. For example, the second receiver 203 is configured to receive pose data from an eye tracker, from a VR headset worn by a user, etc.

[0076] The first receiver and the second receiver may be implemented in any suitable manner and may receive data from any suitable source, including local memory, a network connection, a wireless connection, a data medium, or the like.

[0077] The receiver is implemented as one or more integrated circuits, e.g., an application specific integrated circuit (ASIC). In some embodiments, the receiver is implemented as one or more programmed processing units, e.g., firmware or software running on a suitable processor, e.g., a central processing unit, digital signal processing unit, or microcontroller. In such embodiments, it is understood that the processing unit includes on-board or external memory, clock driving circuits, interface circuits, user interface circuits, etc. Such circuits may be further implemented as part of the processing unit, as an integrated circuit, and / or as discrete electronic circuits.

[0078] The first receiver 201 and the second receiver 203 are coupled to a view synthesis circuit 205 configured to generate at least one view image from the received three-dimensional image data, the view image being generated to represent a view of the three-dimensional scene from a view pose. Thus, the view synthesis circuit 205 generates a view image for the 3D scene from the received image data and the view pose.

[0079] It will be appreciated that in many cases a stereo image / image object is generated that includes a view image / object for a right eye and a view image / object for a left eye, such that when the view images are presented to a user, for example via an AR / VR headset, the 3D scene appears as it is observed from the view pose.

[0080] Thus, the view synthesis circuitry 205 is typically configured to perform view shifting of the multi-view image based on depth, which typically includes techniques such as pixel shifting (changing pixel positions to reflect appropriate disparity corresponding to disparity changes), de-occlusion (typically based on infilling from other images), combining pixels from different images, etc., as will be understood by those skilled in the art.

[0081] It will be appreciated that many algorithms and approaches are known for synthesizing images, and that any suitable approach may be used by the view synthesis circuitry 205.

[0082] Thus, the image synthesizer generates view images for a 3D scene. Moreover, when the view pose changes dynamically in response to a user moving around in the scene, the view of the 3D scene is continuously updated to reflect the change in the view pose. For static scenes, the same source view images are used to generate the output view images, whereas for video applications, different source images are used to generate different view images, e.g., a new set of source images and depths are received for each output image. Thus, the processing is frame-based. In the following, static scenes are considered for clarity and brevity of explanation. However, it will be understood that the approach is equally applicable to dynamic scenes by simply generating output view images over a given period / frame based on the source images and depths received over that period / frame.

[0083] The view synthesis circuitry 205 is configured to generate views of the scene and scene objects to be from different angles relative to lateral movement of the view pose. When the view pose is changed such that the view pose is in a different direction / orientation, the view synthesis circuitry 205 is configured to generate views of the three-dimensional scene objects to be from different angles. Thus, as the view pose changes, the scene objects may be perceived as static and with a fixed orientation in the scene. The viewer effectively moves and views the objects from different directions.

[0084] The view synthesis circuitry 205 may be implemented in any suitable manner, including as one or more integrated circuits, such as, for example, an application specific integrated circuit (ASIC). In some embodiments, the receiver is implemented as one or more programmed processing units, such as, for example, firmware or software running on a suitable processor, such as a central processing unit, digital signal processing unit, or microcontroller. In such embodiments, it will be understood that the processing unit includes on-board or external memory, clock driving circuits, interface circuits, user interface circuits, and the like. Such circuits may be further implemented as part of the processing unit, as integrated circuits, and / or as discrete electronic circuits.

[0085] As mentioned above, a problem with view synthesis is that the more the view pose for which a view is synthesized differs from the capture pose of the provided scene image data, the lower the quality becomes. In fact, if the view pose moves too far from the capture pose, the generated images will contain significant artifacts and errors and will be unacceptable.

[0086] The apparatus of Fig. 2 provides features and embodies an approach to solving and mitigating such problems. In particular, the view synthesis circuit 205 is configured to identify a first region for an image region in the 3D image and to adapt the transparency of that image region depending on the distance between the view pose and the viewing region. The first region is referred to below as a / first region, or more frequently as a / (first) viewing region.

[0087] The view synthesis circuit 205 adapts, for example, the transparency of an object depending on how close the view pose is to the viewing region, and in particular, the view synthesis circuit 205 increases transparency as the distance of the view pose relative to the viewing region increases. As a particular example, if the viewer moves such that the view pose is too far from the viewing region / capture pose, one or more of the foreground objects are rendered to be completely transparent. In such an example, if the view pose moves too far from the capture pose, the foreground objects may, for example, become invisible and disappear from the scene, rather than being rendered / presented with significant errors and artifacts.

[0088] The adaptation of transparency to an image region further depends on the depth marking for the image region, in particular increasing transparency for shallower depths. Thus, transparency is adapted based on several considerations, in particular depending on both the depth of the image region and the distance between the view pose and the viewing region.

[0089] This provides an improved user experience in many scenarios and applications compared to presenting significantly degraded foreground objects. The present approach reflects the inventors' realization that improved performance can be achieved by processing regions / objects at different depths with different morphologies, and in particular that more frontal regions / objects tend to exhibit substantially more degraded quality than more rearward regions / objects (and especially than the background).

[0090] The view synthesis circuit 205 further comprises an image region circuit 207 configured to identify one or more image regions in the 3D image, and in particular an image region of one of the images in the multi-view image representation. The image region is identified, for example, to correspond to a scene object or a part of a scene object. In some embodiments, the image region is identified as a relatively small region, for example an area of ​​less than, for example, 10,000, 1000, 100, or even 10 pixels. Indeed, in some embodiments, the image region is only one pixel.

[0091] Image regions are objects, especially scene objects.

[0092] Different approaches may be used to identify the one or more image regions. For example, in some embodiments, each pixel is considered to be a separate image region. In other embodiments, for example, the input image is tiled into different tiles, each tile being an image region. For example, a predefined tiling is implemented, such that each image region corresponds to a predefined image region.

[0093] However, in many embodiments, dynamic identification of image regions is performed. For example, the image is segmented into a number of image segments that are believed to correspond to scene objects or parts thereof. For example, the segmentation result is a function of image properties, such as pixel color and brightness. Thus, image regions that have similar visual properties and are therefore likely to be part of the same object are generated. Segmentation is alternatively or additionally based on detecting transitions in the image and using such transitions as indications of boundaries between image regions.

[0094] In many embodiments, the identification of image regions is alternatively or additionally based on consideration of depth maps / depth information. For example, image regions may be formed such that they have similar depths, thus increasing the likelihood that they belong to the same scene object, further considering depth uniformity to additionally or alternatively take visual uniformity into account. Similarly, depth transitions may be identified and used to find edges of image regions.

[0095] In some embodiments, scene objects are detected and image regions corresponding to the objects are identified.

[0096] It will be appreciated that many different approaches and algorithms for identifying image regions, and in particular for object detection / estimation and / or image segmentation, are known and therefore any suitable approach may be used.

[0097] In the above examples, the image regions are generated based on the 3D image. In some embodiments, the image regions are identified based on received metadata describing the image regions. For example, the 3D image is received in a bitstream that further includes metadata identifying one or more image regions. For example, metadata is received for each pixel or block of pixels (e.g., for each macroblock) that identifies whether the pixel is a background pixel or a foreground pixel. The image regions are then identified as regions proximate to the foreground pixels.

[0098] The image region circuit 207 is coupled to a depth indication circuit 209 that is configured to determine a depth indication for each image region. The depth indication indicates the depth of the image region.

[0099] A depth indication for an image region is any indication or value that reflects a depth characteristic for an image region, and in particular any indication that reflects the depth of an image region.

[0100] It will be understood that any suitable function or algorithm for identifying such depth indicators from the depth data of the three-dimensional image data may be used, and that the particular identification and depth indicators used will depend on the particular characteristics and requirements of each individual embodiment.

[0101] In many embodiments, a depth indication for an image region is specified as a depth value that is provided as a function of the depth values ​​for the image region, the depth values ​​being included in the depth data of the three-dimensional image data, the function and relationship between the depth indication and the depth values ​​of the depth data of the three-dimensional image data depending on the particular embodiment.

[0102] The depth indication is determined, for example, by considering all depth values ​​for pixels in an image region and determining the depth indication as, for example, the average depth, median depth, maximum depth, or minimum depth for the pixels in the image region. In some embodiments, the depth indication is simply a binary value, or an indication of the depth interval to which the image region belongs. For example, the depth indication is simply an indication of whether the corresponding image region is background or foreground. Of course, many other options are possible and beneficial and may be used to provide the desired effect and performance for a particular embodiment. Indeed, the above is merely exemplary and many other options for generating an indication of the depth of an image region are possible and may be used without detracting from the invention.

[0103] The input source receiver 401 and the depth indication circuit 209 are coupled to a view synthesis circuit 205 which is provided with data describing the identified visibility region and associated depth indications.

[0104] The view synthesiser further comprises a view region circuit 211 configured to identify viewing regions for the image regions. In some embodiments where multiple image regions are identified / generated, the view region circuit 211 is configured to generate a viewing region common to all or some of the image regions. In other embodiments, individual viewing regions are generated for each individual image region. Thus, different image regions are linked to the same viewing region or to different viewing regions.

[0105] The visible / first area is a nominal or reference area for the image area. The reference or nominal area is identified for the image area as one for which a criterion is met. The exact criterion depends on the particular embodiment. In many embodiments, the criterion is, for example, a geometric criterion, and the visible / nominal / reference area is identified as an area for which a geometric criterion associated with the image area and / or capture area for the first image area is met. For example, 3D image data provides image data representing a view of a three-dimensional scene from one or more capture areas and / or points. The visible area is identified as an area for which a geometric criterion associated with the capture area / point is met. The visible area is identified in particular as an area for which a closeness criterion associated with the capture area / point is met.

[0106] The visibility region for an image region is the region of poses for which it is believed that the image region can be synthesized / view-shifted with a given minimum quality, in particular the pose set for which the representation provides data that allows a view image to be generated that includes the image region of sufficiently high quality. Thus, for view poses that fall within the visibility region for the visibility region, it is believed possible to generate a view image of sufficient quality for the image region. For view poses outside the visibility region, it is believed that it is not guaranteed that a view image of sufficient quality for the image region can be generated.

[0107] The exact selection / identification / characterization of the visibility region (typically represented by its boundary, contour, or edge) will of course depend on the particular preferences and requirements of each individual embodiment. For example, in some embodiments, the visibility region is identified to correspond directly to the capture zone, i.e., it is the zone spanned by the capture pose. In many embodiments, the visibility region is identified to include poses where a measure of distance between the pose and the nearest capture pose satisfies a criterion.

[0108] A visibility region is identified in some embodiments as being a region for which a proximity criterion related to the capture region for the three-dimensional image data is satisfied, the exact proximity requirement depending on the requirements and preferences of a particular embodiment.

[0109] In some embodiments, the visibility regions are identified as regions for which an image quality measure for the synthesis of image regions is higher than a threshold. The image quality measure used depends on the particular desirability of the implementation. For example, in some embodiments, the quality measure is identified as a function of the magnitude of the view shift required to perform the synthesis from the received 3D image data and / or as an estimate of how much de-occlusion has to be compensated for, for example by interpolation. In some embodiments, the visibility regions are static, in particular the same for all image regions. In other embodiments, the visibility regions are dynamically identified depending on the characteristics of the image regions. In this case, different image regions contain different visibility regions, and a visibility region is identified specifically for each image region.

[0110] In many embodiments, the viewing area is R N A pose is defined as a subset of a pose in space, where N is the number of dimensions considered. In many embodiments, e.g., particularly many 6DoF applications, N is equal to 6, typically corresponding to three coordinates / dimensions indicating position and three coordinates indicating orientation ( / direction / rotation). In some embodiments, N is less than 6, corresponding to some dimensions not being considered (and specifically ignored or considered fixed).

[0111] In some embodiments, only the position dimension or coordinates are considered, and in some embodiments, only the orientation dimension is considered, but in many embodiments, at least one position dimension and one orientation dimension are considered.

[0112] The viewing area is typically at least two-dimensional and includes poses in which at least two coordinates / dimensions have different values. In many embodiments, the viewing area is at least three-dimensional and includes poses in which at least three coordinates / dimensions have different values. The viewing area is typically at least a two- or three-dimensional zone. The viewing area typically includes poses that vary in at least two dimensions.

[0113] In many embodiments, the visibility region includes poses with different orientations, and therefore often has a non-zero extent for at least one orientation coordinate / dimension.

[0114] In most embodiments, the visibility region has extent in at least one orientation dimension and at least one position dimension, and thus in most embodiments both position and orientation are taken into account by the system.

[0115] In many embodiments, the visibility region is identified simply as a region of a pose where a predefined distance to a reference or preferred viewing pose is less than a given threshold. In other embodiments, the distance is measured relative to a given capture region. As described below, in some embodiments, more complex considerations are applied, with the visibility region depending on many different parameters, etc. In general, however, it will be appreciated that any suitable approach for identifying a visibility region for an image region may be used, and that the approach is not limited to any particular manner of identifying a visibility region.

[0116] In many embodiments, the visibility region for a given image region is identified as a region where it is likely that high quality synthesis of the given image region can be achieved from the received 3D image data, but this is not required and other approaches may be used. For example, the visibility region is identified as a region where it is desirable to bias the user towards that region. For example, in the case of a game or virtual reality application, it is desirable to bias the user towards a particular position or region. Such an approach may be used, for example, to bias the user towards a position directly in front of a virtual object, even though this object in the 3D image data is represented by image data captured substantially from one or both sides of the object. Thus, adaptive transparency is used to bias the user towards a position that does not provide optimal synthesis quality, but is preferred for other purposes, including purposes unrelated to synthesis quality / process.

[0117] In many embodiments, the view region circuit 211 is configured to identify the first region in response to at least one capture pose for the three-dimensional image data. The view region circuit 211 is particularly configured to identify the first region in response to a closeness criterion being satisfied for one or more capture poses for the three-dimensional image data. For example, the first region is identified as a region for which a closeness criterion is satisfied for at least one capture pose for the three-dimensional image data.

[0118] In many embodiments, the first region is a capture region with reference to which three-dimensional image data is provided.

[0119] The three-dimensional image data provides three-dimensional image data linked to a capture / reference pose. The capture / reference pose is a pose from which at least some of the three-dimensional image data is perceived / captured. The first region is identified as a location near the capture / reference pose (e.g., when a closeness criterion is met, e.g., the distance from a pose in the first region to the reference / capture pose is less than a given distance).

[0120] In some examples, more than one reference / capture pose is provided for the three-dimensional image data, in such cases, identifying the visibility region comprises selecting one, more, or all of the reference / capture poses and identifying the visibility region as a region of poses for which a closeness criterion to at least one of the selected capture / reference poses is satisfied.

[0121] The first (visible) region and the first image region do not overlap in many embodiments, and in many embodiments no pose (and possibly no position) falls within both the first (visible) zone and the first image zone.

[0122] The viewing area may be specified as any reference or nominal area that provides a basis for adaptive transparency according to a particular preference desired. The first viewing area is a preferred viewing area that indicates a preferred area for the view pose.

[0123] In some embodiments, the received data includes an indication of the visibility area or parameters used to identify the visibility area. Thus, the received data includes data describing or allowing the nominal and / or reference area to be identified. This nominal / reference area is then used as a reference for the described adaptive transparency to provide a desired effect.

[0124] For example, the 3D image data may be generated by MVD capture as described above, and together with the image and the depth map, an indication of the capture area, or directly the viewing area, may be included in the 3D image data.

[0125] The view synthesis circuit 205 is configured to generate an image of the scene for a view pose (particularly a stereo image collection for a VR headset) based on the received 3D images, and thus in a particular example based on the MVD images and depth.

[0126] However, the view synthesis circuitry 205 is further configured to perform adaptive rendering of the image region based on the association between the view pose and the visibility region for the image region, and in particular to adapt the transparency of the image region based on the association between the view pose and the visibility region for the image region.

[0127] In particular, the view synthesis circuitry 205 is configured to adapt the transparency / semi-transparency of an image region in the view image depending on the depth indication for the image region and the distance between the view pose and the visibility region. The view synthesis circuitry 205 is configured to adapt the transparency to be more transparent as the distance between the view pose and the first region increases and as the depth indication indicates less depth for the first image region.

[0128] For example, the transparency is specified as a function of the depth indicator and the distance between the view pose and the viewing region. The function is monotonically with respect to the depth indicator, and in particular, monotonically increasing as the depth indicator exhibits decreasing depth. Depth is considered to increase towards the background. The function is also a monotonically increasing function of the distance between the view pose and the first region.

[0129] In the following, the approach is described with reference to one image region, called the first image region, but it is understood that the approach is repeated for many more, and typically all, identified image regions. It is further understood that in some embodiments, a transparency is specified to be common to multiple image objects.

[0130] As a particular example, in some embodiments, the view synthesis circuitry 205 is configured to provide increased transparency the greater the distance between the view pose and the visibility region. For example, when the view pose is within the visibility region, an image region is rendered fully opaque, but is made more transparent as the view pose moves further outside the visibility region, until at a given distance the image region is rendered fully transparent; i.e., for view poses farther from the visibility region, the image objects represented by the image region become invisible and the image background is shown rather than the image region being shown.

[0131] Thus, in such an example, when applied to an immersive video application, a view pose beyond the visibility region results in all image regions being invisible and completely transparent, such that only the background of the immersive video scene is presented. In such an example, the foreground objects may be replaced by a background, which may be generated, for example, from a different image of the MVD representation, if available, or by inpainting, if the data is not available. Such an approach may result in an extreme form of de-occlusion, by making the foreground objects completely transparent. This requires or is based on the expectation that the occluded data is available (made available) from the 3D image data, or is generated on the fly (e.g., inpainted from the surrounding area).

[0132] Such an approach effectively broadens the primary viewing space, where the scene is presented / rendered in full with high quality, along with a secondary viewing space where only the background is shown. In fact, the inventors have noticed that the degradation of image quality for the background and more distant objects tends to be less than that for the closer foreground objects, so the perceived quality of the secondary viewing space still remains high. Thus, instead of rendering increasingly lower quality foreground objects as the viewer moves further and further away from the viewing area, these become invisible, but the background and thus the overall scene is still visible. The user is provided with an experience where the lower quality rendered images closer to the viewer disappear, but the scene is still maintained as a whole and is still of sufficient quality. Although such an experience feels unnatural to the user for some applications and scenarios, in many embodiments and applications it provides a substantially more beneficial and often intuitive user experience. When the user notices, for example, that the foreground objects start to disappear, he or she intuitively realizes that the user has moved away and starts to move back towards the viewing area. Furthermore, in some situations, the user moves away from the viewing area precisely to be able to look around the foreground object, i.e. to be able to see the object or the background behind the object. In such cases, it is a highly desirable experience for the foreground object to become transparent, allowing the user to see through the foreground object. Furthermore, in contrast to other proposed approaches to solve the problem of degraded quality when the user moves too far away from the desired viewing area, the present approach allows the user to still experience consistency in the perception of the user's position in the scene, e.g., to be able to navigate to a more desirable position. The approach provides a more immersive experience in many scenarios.

[0133] The view synthesis circuit 205 is configured to determine a transparency for a first image region and to generate a view image including the first image region with the determined transparency. Thus, the view synthesis circuit 205 is configured to generate a view image including the first image region generated with a determined transparency depending on a depth indication and a distance between the view pose and the first region. The view synthesis circuit 205 adapts the transparency of the first image region in the view image by determining an (adapted) transparency and applying the (adapted) transparency to the first image region. The (adapted) transparency is determined depending on a depth indication and a distance between the view pose and the visibility region. The adapted transparency is determined in particular as an alpha value for objects / components / portions in the first image region, and the view synthesis circuit 205 is configured to generate the view image using the alpha values ​​for these objects / components / portions. It is understood that algorithms for generating a view image based on such transparency / alpha values ​​are known to those skilled in the art and therefore will not be described in more detail herein.

[0134] Thus, for example, the transparency of an object depends on different distances. In many embodiments, depending on the depth indication results in depending on the distance from the view pose to the object, i.e., the object distance (to the view pose) is used in specifying the transparency. In addition, the distance from the view pose to the viewing area is used to adapt the transparency of the object. Thus, the view pose change distance reflecting the change of the view pose (relative to the viewing area) is also taken into account. For example, for a viewing area corresponding to a capture area, the transparency is adapted according to both the object distance and the view pose change distance. Such consideration provides a significantly improved effect.

[0135] In particular, the adaptation is performed such that the transparency / semi-transparency increases with decreasing object depth / object distance and increases with increasing distance between the view pose and the visibility region, i.e., increasing the view pose change distance.

[0136] Different considerations include non-separable and / or non-linear and / or multiplicative effects. For example, an adaptation along the following lines may be applied:

[0137] [Table 2]

[0138] The dependence of transparency on depth marking and viewpose distance (to the viewing area) is inseparable in many embodiments and is non-linear and / or multiplicative.

[0139] The adaptation is a constructive combination of depth indication and viewing pose distance. In particular, let us denote by A the distance between the view pose and the viewing region, and let us denote by B the depth indication, and let us say that a function f(A,B) for the transparency of the first image region has the following property: For any B, there is a non-negative correlation between f(A,B) and A. For any A, there is a non-negative correlation between f(A,B) and B. For some B, there is a positive correlation between f(A,B) and A. For some A, there is a positive correlation between f(A,B) and B.

[0140] In the above example, the image region is identified based on the received 3D image or metadata. In this example, the image region is referenced to the 3D input image, in particular a region in one of the images used for view synthesis, e.g. the closest MVD image. In other embodiments, the image region is referenced, e.g., to the output image. For example, for an image object or area in the input 3D input image, a corresponding area in the output image is identified taking into account the depth of the object or area. For example, a required parallax shift for depth is identified and an image region in the output image corresponding to the image object in one or more input images is identified. Typically, the image region is occupied by the frontmost pixel calculated by translation from the different images (as this is what a person sees along its line of sight), but in the current approach the transparency of one or more of the pixels of this image region is adapted based on the distance between the view pose and the viewing region. In particular, for a given pixel in the image region of the output view image, the transparency of the frontmost pixel value (or only one pixel in case only one image provides the pixel after parallax shift) is pixel dependent.

[0141] The image region is an image region in the input image of the 3D input image. The image region is an image region in the input image of the 3D multiview input image. The image region is an image region in the input image of the 3D input image that provides the front-most pixel for the image region in the synthesized output-view image. The image region is an image region in the input image that corresponds to a given pixel region in the synthesized output-view image.

[0142] The transparency of a pixel is specifically an alpha value, and thus the alpha value for at least one pixel depends on the distance between the view pose and the visibility region. The transparency for a pixel value reflects the degree to which further back (including background) scene objects are visible. In particular, for a pixel in the output view image, the pixel value is generated as a combination of the frontmost pixel value and the further back pixel value generated from the 3D input image (typically by parallax shifting). The further back pixel value is generated from the 3D input image (typically by parallax shifting or by infilling). The further back pixel value is a background pixel.

[0143] As a particular example, the output-view image is generated by finding, for each pixel in the output-view image, a corresponding source pixel in each of the view input images. A source pixel in a given input image is identified as a pixel that results in a source pixel at the location of the output pixel after a parallax shift caused by a viewpoint shift from the source image to the output-view image. For some source images, no such pixel exists (e.g., such a pixel is a deoccluded pixel), and thus the view synthesis circuit 205 identifies a smaller number of source pixels, not more, than the number of source images. Each of the source pixels is further associated with a depth. Conventionally, the source pixel with the lowest depth, i.e., closest to the source camera, is selected because this source pixel corresponds to the foremost object and thus is the one seen by the viewer from the view pose in the view direction represented by the pixel. However, in an example of the current approach, the view synthesis circuit 205 proceeds to use this approach if the view pose falls within the visibility region (or within a threshold distance of this); otherwise, the view synthesis circuit 205 proceeds to select the source pixel that is furthest towards the rear, i.e., furthest from the view pose. Typically, this pixel is a background pixel. Thus, this effectively renders the object represented by the frontmost pixel completely transparent or invisible, and instead of presenting this object, the background is presented. In such an approach, an image region in the source image is identified as the pixel that is at the location of a given output pixel after view shifting / warping.

[0144] It is understood that further considerations are included: for example, if all of a set of source pixels correspond to non-background objects (e.g., their distance is less than a threshold), then none of the source pixels are determined to be suitable for the output image, and instead, suitable values ​​are found, for example, by infilling from adjacent background pixels.

[0145] Thus, in some examples, the view synthesis circuit 205 is configured to select from a plurality of candidate pixel values ​​from the image region of the input multiview image for at least a first pixel of the output image. In this example, the selection is based on a depth for the pixel, which depends on whether the distance between the view pose and the first visibility region is less than a threshold. If the distance is less than the threshold, the view synthesis circuit 205 selects the frontmost pixel, otherwise the view synthesis circuit 205 selects the rearmost pixel.

[0146] The rearmost pixel is the pixel associated with a depth value indicating a depth that is closest to the rear and / or furthest from the view pose. The frontmost pixel is the pixel associated with a depth value indicating a depth that is closest to the front / closest to the view pose. The rearmost pixel is the pixel furthest from the view pose. The rearmost pixel is the pixel closest to the view pose.

[0147] Such an approach provides a highly effective implementation in which modifications to existing techniques result in less complex algorithms where foreground objects become invisible / disappear when the view pose moves too far away from the viewing area.

[0148] In some embodiments, the view synthesis circuitry 205 is configured to generate a view image that includes fully transparent image regions when the distance between the view pose and the visibility region (hereinafter referred to briefly as the viewing distance) is longer than a given threshold, which may be zero. Thus, in such cases, the view synthesis circuitry 205 renders the view image such that foreground objects are completely invisible / non-existent when the viewing distance is sufficiently long.

[0149] Similarly, in some embodiments, the view synthesis circuitry (205) may additionally or alternatively be configured to generate view images that include opaque image regions when the viewing distance is not greater than a threshold value, and thus in such cases, the view synthesis circuitry 205 renders view images with foreground objects fully visible / present when the viewing distance is sufficiently short.

[0150] This approach is combined such that the foreground object is either completely present or completely absent (completely opaque or completely transparent) depending on whether the viewing distance is greater than a threshold.

[0151] This can be a very desirable effect in some embodiments, providing a clearer indication to the user than, for example, if the user has moved too far from a preferred pose and must move back towards the viewing area.

[0152] In many embodiments, the view synthesis circuitry 205 is configured to generate a view image to render a first image region with transparency applied, the transparency being determined as a function of both a depth indication for the first image region and a distance between the view pose and the first region.

[0153] When the transparency is not completely opaque, the view image is generated to include contributions from other visual elements for the first image region. A pixel light intensity value for a pixel of the view image representing the first image region is generated as a weighted combination of the contribution from at least one visual element of the first image region and the contribution from at least one other visual element. The other visual element is in particular an object (including the scene background) behind the first image region from the view pose. However, in some scenarios, the at least one other visual element is possibly an element that does not directly correspond to the scene, for example a specific visual characteristic (e.g. a black or gray background). The weighting of the contribution from the visual element of the first image region is greater the lower the transparency. The weighting of the contribution from the visual element not belonging to the first image region is greater the higher the transparency.

[0154] Thus, the view image is generated such that the higher the transparency, the more the first image region is visible through it. Typically, increasing the transparency provides the effect that the first image region in the view image is "see-through" to a greater extent, such that the scene behind the first image region is partially visible. Thus, typically, increasing the transparency allows scene objects behind the first image region to become increasingly visible through the first image region. In particular, in many embodiments, the background of the scene becomes increasingly visible through the first image region.

[0155] In some embodiments, transparency is generated by including visual contributions from elements that are not part of the scene, but instead are elements that have fixed or consistent visual characteristics, such as, for example, a uniform color or a predefined texture.

[0156] Thus, in many embodiments, the greater the transparency, the more the first image region becomes see-through and therefore the objects in the first image region fade. In most embodiments, therefore, visibility of the scene behind the first image region becomes visible and therefore the objects in the first image region fade to reveal the scene behind them.

[0157] The view synthesis circuit 205 is configured to apply the (adapted) transparency by increasing the (relative) contribution from the first image region to the (light intensity pixel value for) pixel in the view image corresponding to the first image region the lower the transparency.

[0158] Alternatively, or in addition, the view synthesis circuit 205 is configured to apply the (adapted) transparency by increasing the (relative) contribution from elements that are not of the first image region to (the light intensity pixel values ​​for) pixels in the view image corresponding to the first image region the higher the transparency.

[0159] In many embodiments, the view synthesis circuit 205 is configured to adapt the transparency of a first image region in the view image such that the greater the transparency, the more visible part of the three-dimensional scene behind the first image region becomes.

[0160] The view synthesis circuit 205 is configured in many embodiments to adapt the transparency of a first image region in the view image such that the greater the transparency, the greater the contribution of the portion of the three-dimensional scene behind the first image region to the view image. In some embodiments, hysteresis is included so that the threshold is adapted depending on whether the view distance is currently greater than or less than the threshold. Thus, to change an opaque object to transparent, the user is required to move to increase the view distance from less than a given first value to greater than the given first value, and to change a transparent object to opaque, the user is required to move to decrease the view distance from greater than a given second value to less than a given second value, the first value being greater than the second value. Such an approach prevents a ping-pong effect in which foreground objects flash between being perceived as present and not present.

[0161] Thus, in some embodiments, transparency as a function of distance has a hysteresis related to changes in viewing pose.

[0162] The hysteresis is a velocity independent hysteresis. The transparency as a function of distance is a hysteresis loop. The transparency value depends on the direction of the change in distance. In some embodiments, the view synthesis circuit 205 is configured to generate view images with more gradual changes in transparency of image regions, and especially of foreground objects. For example, in some embodiments, the transparency (often represented by an alpha value in the field) is gradually increased with increasing view distance. The transparency is a monotonically increasing function of the view distance.

[0163] Such an approach to generate partially transparent objects is combined with a binary approach. For example, instead of simply identifying one viewing region, two viewing regions are identified, one inside the other. In such an embodiment, a view image is generated that includes an image region that is opaque when the view pose is inside the inner viewing region, and completely transparent when the view pose is outside the outer viewing region. For viewer poses between the two regions, the transparency is gradually increased as a monotonically increasing function of the distance to the inner viewing region (or similarly gradually decreased as a monotonically increasing function of the distance to the outer viewing region). Such an approach provides a gradual user experience, where objects do not immediately appear or disappear as the viewer moves, but rather transition gradually through an intermediate region. In such an approach, objects appear / disappear gradually, which, for example, can be alleviated if viewing discomfort is experienced due to the effect.

[0164] In some embodiments, a view pose that exceeds the viewing area by more than a given amount results in only the background of the immersive video scene being visible, thus maintaining the viewer's immersion. To do so, foreground objects are replaced by the background if available, or inpainted if not.

[0165] The present approach extends the primary viewing region, where the scene is rendered / synthesized in its entirety, with a secondary viewing region, where only the background is rendered / synthesized. The secondary viewing region is larger than the primary viewing region, but is more limited since the depth map is involved in view synthesis.

[0166] Thus, in some embodiments, if the viewing distance is greater than a threshold, the scene is, for example, no longer presented, e.g., one of the prior art modes discussed above is applied in this situation.

[0167] In the above examples, the approach has been described primarily with reference to one visibility region, but as described, the approach may be applied separately to different image regions. Different visibility regions are identified for different image regions. For example, depending on the depth, the image regions are categorized into a set of predefined categories, each of which is associated with an image region of a different size.

[0168] The approach is particularly implemented such that, when the view pose is beyond the primary viewing space, foreground objects are replaced by the background when available, and inpainted when not available. The size of the inpainting region of missing data can be large. This depends on the size of the foreground object and the availability of background information in other views. In some embodiments, only foreground objects with substantial background available from other views are removed, i.e. the transparency depends on whether data is available for de-occlusion. Such foreground objects are typically the smallest objects and the most forward / frontmost. The inpainted region leads to the perception of some blurring of the background. However, such blurring is insignificant or tolerable, and is typically constant in time. It has been found that even if some blurring of the background occurs, any visual distortion is perceived relatively unobtrusive compared to existing approaches.

[0169] In many embodiments, the viewing area dynamically depends on different parameters, in particular parameters that affect the quality of the synthesis process. For example, the more data provided in the 3D input image, the better the quality of the view image that can be synthesized for a given view pose, and therefore the less quality degradation. In some embodiments, the viewing area circuit 211 is configured to adapt the viewing area, and in particular to adapt at least one of the size and shape of the viewing area, depending on the quality affecting parameters.

[0170] In many embodiments, the visibility region for an image region depends on the view-shift / view-pose change sensitivity for the image region. The view-shift / view-pose change sensitivity for an image region reflects how sensitive the image region is to distortions introduced by performing view-shift / view-pose change synthesis. The viewpoint change sensitivity for an image region indicates the sensitivity of the image region to changes in the view pose. This can be used to improve performance. For example, a relatively complex object that is relatively close to the camera will have a smaller visibility region than a relatively distant flat object.

[0171] In some embodiments, the view region circuitry 211 is configured to adapt a viewing region for an image region / object in response to a depth indication for the image region / object. In particular, the view region circuitry 211 is configured to adapt at least one of a shape and a size of the viewing region for an image region in response to a depth indication for the viewing region.

[0172] In many embodiments, the size of the image region is increased as the depth indication indicates that the image region is further back. For example, the visibility region for an object that is relatively closer to the view pose is smaller than the visibility region for an object that is relatively farther from the view pose. Thus, the closer an object is to the foreground, the smaller the visibility region and therefore the less movement of the view pose is required before the foreground object becomes invisible.

[0173] Typically, the closer an object is to the viewer, the greater the quality degradation; therefore, by adapting the viewing area to the depth markings, a more gradual user experience can be achieved in which the transparency of the object is more flexibly adapted to reflect the quality degradation.

[0174] In some embodiments, the size and / or shape of the view region, and in particular the visibility region, for a given image region depends on the complexity of the shape of the image region, hi some embodiments, the view region circuit 211 is configured to adapt at least one of the shape and size of the first visibility region depending on a measure of the complexity of the shape.

[0175] The view-shift portion of view synthesis tends to introduce less distortion for image regions / objects with simple shapes compared to more complex ones. For example, simple shapes have more consistency between adjacent pixels and tend to be less deoccluded compared to complex shapes. Thus, the higher the complexity of the shape, the larger the size of the visibility region for that image region.

[0176] Shape complexity is determined according to different measures in different embodiments, for example, algebraic complexity, such as how many sides a visibility region is represented by, the angles between such sides, etc.

[0177] In some embodiments, the viewing area for an image region depends on a disparity variation measure for the image region. The view synthesis circuit 205 is configured to adapt at least one of a shape and a size of the viewing area for the image region depending on the disparity variation measure for the image region. The disparity variation measure indicates the variation of disparity for pixels of the image region for a given viewpoint shift. The disparity variation measure is in particular a depth variation measure for the image region.

[0178] The view region circuit 211 is configured to determine the viewing region to be smaller, for example, for large amounts of disparity or depth variation in an image region. If there is a large variation in depth and therefore disparity is required when implementing a viewpoint shift, there is a higher probability of introducing distortion or inaccuracies, which may result in, for example, greater de-occlusion. Thus, the larger the amount of disparity or depth variation for a given image region, the smaller the viewing region will be, and therefore the smaller the view pose will be before the image region begins to become transparent.

[0179] The view region circuit 211 is configured to determine a visibility region for an image region based on the depth quality of the depth information provided for the visibility region. The depth quality of an object may be an indication of how well the object can be reprojected from a first (real) camera view to a second (real) camera view. For example, a floor surface (with less complex shape) is likely to have high depth quality. Depth quality is relatively easy to determine in many embodiments. For example, a view shift of an input image of the MVD representation to a position of another input of the MVD representation is performed based on the depth data for the input image. The result is compared with the corresponding data in the input image of the MVD representation for the image region, and the depth quality is based on this comparison. The closer the synthesized image is to the input image, the higher the depth quality.

[0180] The viewing region for an image region depends on the amount of de-occlusion data for the image region contained in the three-dimensional image data. In many embodiments, the view region circuit 211 is configured to adapt at least one of the shape and size of the viewing region for an image region depending on how much de-occlusion data is available for the image region in the received 3D data.

[0181] For example, if the received 3D image data includes another image that views the image area from a capture pose at a significantly different angle, this provides substantially additional data that allows for improved de-occlusion. The more de-occlusion data available, the larger the viewing area can be. This reflects that the more de-occlusion data there is available, the less degradation is likely to occur from view shift.

[0182] The amount of de-occlusion data for an image region in the input images of the multi-view representation is determined, for example, by performing a view shift of all the different view images of the representation to the capture pose for the input image. The data, and especially the depth, for the image region determined by such view synthesis is then compared with the original image region. The larger the difference, the more de-occlusion data is considered to be present, since the difference reflects that the different images have captured different objects on the line of sight from the current input image pose.

[0183] In some embodiments, the more de-occlusion data available, the smaller the visibility region is created, reflecting that the more de-occlusion data available, the easier it is to generate an accurate view of the background, and therefore the higher the quality of the presented scene after removal of foreground image regions / objects.

[0184] Indeed, if de-occlusion data is not available, further degradation to the rendering occurs, since de-occlusion requires inpainting, making the foreground object invisible. This depends, for example, on the size of the foreground object and on the availability of background information in other views. In some embodiments, only foreground objects with substantial background available from other views are removed. Thus, if de-occlusion data is available to composite the background in the absence of foreground objects, small visibility regions are identified, whereas very large (possibly infinite) visibility regions are generated when de-occlusion data is not available. These are typically the smallest objects and are furthest forward.

[0185] The above example of how the visibility area is adapted also applies to the dependence of transparency on viewing distance, i.e. the function therefore also depends on any of the parameters mentioned above to affect the visibility area specification.

[0186] In some embodiments, the described view synthesis device performs operations to identify image regions and, for example, to divide the received data into foreground and background image regions. Similarly, in the description so far, operations are performed to identify visibility regions for different image regions. However, in some embodiments, the received input data includes data describing image regions and / or visibility regions.

[0187] For example, the view synthesis apparatus of Figure 2 is a decoder-based implementation, where the input data is received from an encoder. In addition to providing 3D image data, the image data stream includes further data describing image regions for at least one of the input images.

[0188] For example, the received 3D image data is for a given input image (e.g., of a multi-view representation) that includes an image region map indicating, for each pixel, whether the pixel is a foreground pixel or a background pixel. In other embodiments, the 3D image data indicates, for example, for each non-background pixel, the identity of the image region to which the pixel belongs.

[0189] In such an embodiment, the image region circuit 207 is configured to identify an image region in response to the received data indication. For example, the image region circuit 207 considers each foreground pixel to be an image region. As another example, the image region circuit 207 groups a set of contiguous foreground pixels into an image region. If the received data includes an identification of an image region, the image region circuit 207 groups pixels provided with the same identification into an image region.

[0190] In some embodiments, the received 3D data includes an indication of the visibility region that must be applied: the visibility region is a fixed visibility region that must be applied to all image regions / objects, or for example different visibility regions are defined for different image regions or for different properties associated with the image regions.

[0191] In such a case, the view region circuit 211 determines the viewing region in response to the received indication of the viewing region, for example, the view region circuit 211 simply uses the viewing region defined in the received data.

[0192] The advantage of using a data stream containing such information is that it significantly reduces the complexity and resource demands at the decoder side. This is important for example for embodiments where data is distributed to many decoders and thus centralized operation reduces the overall resource demands and provides a consistent experience for different users. Typically, more information and / or options for control are also available at the encoder side. For example manual identification of visibility areas or image regions is practical.

[0193] In many embodiments, an image signal device, e.g. an encoder, is configured to generate an image signal that includes 3D image data and further includes a data field / flag indicating whether the described approach for rendering should be applied or not.

[0194] Thus, the image signal device generates an image signal including three-dimensional image data describing at least a portion of a three-dimensional scene, a depth indication for the image region, and a data field indicating whether rendering of the three-dimensional image data should include adapting transparency for an image region of the three-dimensional image data in the rendered image depending on the distance between the view pose for the rendered image and the viewing area for the image region.

[0195] As a specific example, the described approach is added as an additional operational mode to the list of possible modes provided in the 5th Working Draft of the MPEG Immersive Video (MIV) standard ISO / IEC JTC1 SC29 WG11 (MPEG) N19212, which contains proposals for handling such motion outside the viewing space. For example, mode ID bits for unassigned values ​​(e.g., between 7 and 63) may be used to indicate that the described approach of making one or more foreground objects transparent is used.

[0196] As described, in some embodiments, image region identification is based on processing at the synthesizer (decoder) or at the image signal device (encoder). Doing so at the decoder makes decoding more (computationally) expensive. Doing so at the encoder is more feasible but requires information about background regions to be transmitted to the decoder. The preferred tradeoff depends on the embodiment.

[0197] In the following a particular approach is described that is based on a binary separation of the image into foreground (FG) and background (BG) and using image regions corresponding to FG regions / pixels. In a particular example, the segmentation into FG and BG is performed on the encoder side.

[0198] The approach follows the following steps:

[0199] 1. Compute a dense FG / BG segmentation for each source view. As a result, next to the color and depth attributes, each pixel has an FG or BG label.

[0200] 2.a) The MIV "entity" extension is used to transmit the FG / BG segmentation map to the decoder. To do so, the MIV encoder receives as a further input a binary entity map containing the obtained per-pixel FG / BG segmentation. The resulting bitstream thus contains metadata identifying an entity ID (e.g. a "background" label) for each rectangular texture atlas patch, and the modification of that label to pixel level via "occupancy". This allows the decoder to reconstruct the segmentation map. b) Alternatively, a new "background" flag will be added to the standard specifically for this purpose.

[0201] 3. A second viewing space for background visibility is put into the bitstream metadata. The MIV standard currently does not support multiple viewing spaces. However, it allows a "guard_band_size" to be specified for a (primary) viewing space. This effectively obtains a secondary viewing space of the same shape as the primary viewing space (visibility region) but larger. Alternatively, a modification to the MIV standard must be used to allow multiple viewing spaces, or a non-standardized approach must be chosen.

[0202] At the decoder, dense FG / BG labels are reconstructed from the decoded bitstream and attached to the vertices of rendering primitives (e.g. triangles). When using e.g. OpenGL for view synthesis, the labels are attached to a "texture" and can be sampled by the vertex shader. Optionally, the vertex shader can attach segmentation labels as attributes to the vertices. When the viewer moves beyond the view space boundary, all vertices with FG labels are directly discarded by setting their output values ​​outside the valid clip space, or the attached segmentation labels are used later to discard them there (geometry and fragment shaders have direct means to discard primitives). Discarding foreground objects in the view synthesis process is likely to increase the size of missing data. The process of inpainting missing data is already available in the normal decoding process and will not be described in detail.

[0203] Different approaches are used to segment the image into foreground and background (FG / BG segmentation). At the core of the process is a connectivity measure, i.e. a measure that reflects how connected adjacent pixels are. In one example, the world space distance (meters) is used for this purpose. Each pixel has a world space (x,y,z) coordinate through the use of a depth map. When two adjacent pixels have a distance below a certain threshold (e.g. 2cm depending on the depth map quality), they are considered to be connected. Distinct objects are defined, which are clusters (regions) of pixels that are connected to themselves or only to the floor surface.

[0204] The following steps are used to perform the FG / BG segmentation.

[0205] 1. Find the floor surface. In this embodiment, it is assumed that the z-component (height) of the selected world coordinate system is orthogonal to the floor surface. If this is not the case, further steps are performed to make it so. For all pixels in the image, the minimum value of the z-component (height) is found (the "z-floor"). To do so robustly, for example, the average of the 1st percentile of the minimum z-values ​​is taken. All pixels in the image with z-values ​​near the "z-floor" are labeled. A threshold (possibly the same as the connectivity threshold) is used for this purpose.

[0206] 2. A connected component analysis is performed on the unlabeled pixels in the image.

[0207] 3. Find regions of "available hidden layers", meaning regions of foreground pixels for which background data is available from other source views. To find these for a particular source view, the depth map of that source view is composited from all other available source views. Only here, using inverse z-buffering (OpenGL: glDepthFunc(GL_TRUE)), significant priority is given to the background in the compositing process. In the case of normal view compositing, priority is given to the foreground. By using inverse z-buffering, the composited result contains all available background. It is a distorted image where many foreground objects have disappeared or are eroded by the background. By slicing (thresholding) the difference between the original depth map and the inversely composited one, only used for analysis, foreground regions containing available hidden layers are identified through a binary pixel map.

[0208] 4. Components that contain significant hidden layer portions are classified as "foreground". Significance is then determined by dividing the area of ​​the connected components that contain "available hidden layers" by the total area of ​​the component. The larger the percentage, the more background is occluded by the component indicating it is a foreground object. Optionally, the "foreground" classification may be appended with its component number to distinguish them from other foreground objects.

[0209] 5. Unclassified pixels are classified as "background".

[0210] The invention may be implemented in any suitable form including hardware, software, firmware or any combination of these. The invention may optionally be implemented at least partly as computer software running on one or more data processors and / or digital signal processors. Elements and components of embodiments of the invention may be physically, functionally and logically implemented in any suitable way. Indeed, functionality may be implemented in one unit, in several units or as part of other functional units. Thus, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

[0211] In accordance with standard terminology in the art, the term pixel is used to denote pixel-related properties such as light intensity, depth, position of the part / element of the scene represented by the pixel. For example, the depth of a pixel, or pixel depth, is understood to represent the depth of the object represented by that pixel. Similarly, the brightness of a pixel, or pixel brightness, is understood to represent the brightness of the object represented by that pixel.

[0212] Although the present invention has been described in connection with certain embodiments, it is not intended that the present invention be limited to the specific form described herein. Rather, the scope of the present invention is limited only by the appended claims. Moreover, although functions may appear to be described in connection with certain embodiments, those skilled in the art will recognize that various features of the described embodiments may be combined according to the present invention. In the claims, the term "comprise" does not exclude the presence of other elements or steps.

[0213] Furthermore, even if listed independently, a plurality of means, elements, circuits, or method steps may be implemented by, for example, one circuit, unit, or processor. Moreover, although individual features may be included in different claims, they may in some cases be beneficially combined, and their inclusion in different claims does not imply that the combination of features is not feasible and / or beneficial. Moreover, the inclusion of a feature in a category of claims does not imply limitation to this category, but rather indicates that the feature is equally applicable to other claim categories, as appropriate. Moreover, the order of features in the claims does not imply any particular order in which the features must be performed, and in particular the order of individual steps in a method claim does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. In addition, a singular reference does not exclude a plurality. Thus, expressions such as "first", "second", etc. do not exclude a plurality. Reference signs in the claims are provided merely as examples for clarity and are not to be construed as limiting the scope of the claims in any manner.

[0214] Generally, examples of an image synthesis device, an image signal, a method of image synthesis, and a computer program for implementing the method are illustrated by the following embodiments.

[0215] Embodiment Embodiment 1. A first receiver (201) configured to receive three-dimensional image data describing at least a portion of a three-dimensional scene; an image region circuit (207) configured to identify at least a first image region in the three-dimensional data; a depth circuit (209) configured to identify a depth indication for the first image region from depth data of the three-dimensional image data; a viewing area circuit (211) configured to identify a first viewing area for a first image area; a second receiver (203) configured to receive a view pose for a viewer; a view synthesis circuit (205) configured to generate a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from a view pose, the view synthesis circuit (205) configured to adapt a transparency for a first image region in the view image in response to a depth indication and a distance between the view pose and the first viewing region; An image synthesis device comprising:

[0216] Embodiment 2. The view synthesis circuit (205) is configured to generate a view image that includes a completely transparent image region if the distance is greater than a threshold value. An image synthesis device as described in embodiment 1.

[0217]

[0023] Embodiment 3. The view synthesis circuit (205) is configured to generate a view image including image regions that are not completely transparent if the distance is not greater than a threshold value; An image synthesis device according to embodiment 2.

[0218] Embodiment 4. The view synthesis circuit (205) is configured to generate a view image including an opaque image region if the distance is not greater than the threshold. An image synthesis device according to embodiment 2.

[0219] Embodiment 5. The apparatus further includes an image region circuit (207) that identifies a second viewing region relative to the first image region, and a view synthesis circuit (205) configured to generate a view image including image regions that are opaque when the view pose is within the second viewing region, partially transparent when the view pose is outside the second viewing region and within the first viewing region, and fully transparent when the view pose is outside the first viewing region. 2. An image synthesis device according to any one of the preceding embodiments.

[0220] Embodiment 6. The first viewing area is dependent on the depth marking; 2. An image synthesis device according to any one of the preceding embodiments.

[0221] Embodiment 7. The first viewing area depends on the complexity of the shape of the image area. 2. An image synthesis device according to any one of the preceding embodiments.

[0222] Embodiment 8. The first viewing area depends on the view shift sensitivity to the image area; 2. An image synthesis device according to any one of the preceding embodiments.

[0223] Embodiment 9. The first visibility region depends on the amount of de-occlusion data for the first image region included in the three-dimensional image data. 2. An image synthesis device according to any one of the preceding embodiments.

[0224] Embodiment 10. The function for determining transparency as a function of distance includes a hysteresis associated with changes in viewing pose. 2. An image synthesis device according to any one of the preceding embodiments.

[0225] Embodiment 11. The three-dimensional image data further includes an indication of an image area for at least one of the input images of the three-dimensional image, and the image area circuit (207) is configured to identify the first image area according to the indication of the image area; 2. An image synthesis device according to any one of the preceding embodiments.

[0226] Embodiment 12. The three-dimensional image data further includes an indication of a viewing area for at least one of the input images of the three-dimensional image, and the viewing area circuit (211) is configured to identify the first viewing area according to the indication of the viewing area; 2. An image synthesis device according to any one of the preceding embodiments.

[0227] Embodiment 13. A view synthesis circuit (205) is configured to select from a plurality of candidate pixel values ​​derived from different images of the multi-view image for at least a first pixel of the view image, the view synthesis circuit (205) being configured to select the rearmost pixel if the distance is greater than a threshold, and to select the frontmost pixel if the distance is less than the threshold. 2. An image synthesis system according to any one of the preceding embodiments.

[0228] Embodiment 14. Three-dimensional image data describing at least a portion of a three-dimensional scene; a data field indicating whether the rendering of the three-dimensional image data must include adapting the transparency for an image region of the three-dimensional image data in the rendered image depending on a depth indication for the image region and a distance between a view pose for the rendered image and a viewing area for the image region; , an image signal.

[0229] Embodiment 15. Further comprising at least one of an indication of an image area and a viewing area. 15. An image signal as described in embodiment 14.

[0230] Embodiment 16. An image signal device configured to generate an image signal according to embodiment 14 or embodiment 15.

[0231] Embodiment 17. A method of image synthesis, comprising: Receiving three-dimensional image data describing at least a portion of a three-dimensional scene; Identifying at least a first image region in the three-dimensional data; identifying a depth indication for the first image region from depth data of the three-dimensional image data; Identifying a first viewing area for a first image area; receiving a view pose for a viewer; generating a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from a view pose, said generating the view image comprising adapting a transparency for a first image region in the view image in response to a depth marking and a distance between the view pose and the first viewing region; The method comprising:

[0232] 18. A computer program product comprising computer program code means adapted to perform all the steps as recited in embodiment 17 when said program is executed on a computer.

[0233] The invention is defined in more detail in the accompanying claims.

Claims

1. a first receiver for receiving three-dimensional image data describing at least a portion of a three-dimensional scene; an image region circuit for identifying at least a first image region in the three-dimensional image data; a depth circuit for determining a depth indication for the first image region from depth data of the three-dimensional image data for the first image region; a region circuit for identifying a first region for the first image region; a second receiver for receiving a view pose for a viewer; a view synthesis circuit for generating a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from the view pose, the view synthesis circuit adapting a transparency of the first image region in the view image in response to the depth indication and a distance between the view pose and the first region, the view synthesis circuit increasing transparency as the distance between the view pose and the first region increases and as the depth indication indicates a decreasing depth relative to the first image region; An image synthesis device comprising:

2. The view synthesis circuit generates the view image including a completely transparent image region when the distance between the view pose and the first region is greater than a threshold. The image synthesis device according to claim 1 .

3. the view synthesis circuitry generates the view image including the image region that is not fully transparent if the distance between the view pose and the first region is not greater than the threshold. The image synthesis device according to claim 2 .

4. The view synthesis circuit generates the view image including the image region that is opaque if the distance between the view pose and the first region is not greater than the threshold. The image synthesis device according to claim 2 .

5. the image synthesis device further comprising an image region circuit that identifies a second region relative to the first image region, the view synthesis circuit generating the view image including image regions that are opaque when the view pose is within the second region, partially transparent when the view pose is outside the second region and within the first region, and fully transparent when the view pose is outside the first region. The image synthesis device according to any one of claims 1 to 4.

6. The first region is dependent on the depth indication. The image synthesis device according to any one of claims 1 to 5.

7. The first region depends on the complexity of the shape of the image region. The image synthesis device according to any one of claims 1 to 6.

8. the first region being dependent on a view pose change sensitivity to an image region; The image synthesis device according to any one of claims 1 to 7.

9. The first region depends on an amount of de-occlusion data for the first image region included in the three-dimensional image data. The image synthesis device according to any one of claims 1 to 8.

10. The method of claim 1, wherein a function for determining the transparency as a function of the distance between the view pose and the first region includes hysteresis associated with changes in viewing pose. The image synthesis device according to any one of claims 1 to 9.

11. the three-dimensional image data further includes an indication of an image area for at least one three-dimensional image of the three-dimensional image data, the image area circuit identifying the first image area in response to the indication of the image area; The image synthesis device according to any one of claims 1 to 10.

12. the three-dimensional image data further includes an indication of a given region for at least one image of the three-dimensional image data, and the region circuit identifies the first region in response to the indication of the given region. The image synthesis device according to any one of claims 1 to 11.

13. the view synthesis circuitry selects from a plurality of candidate pixel values ​​derived from different images of the multi-view image of the three-dimensional image data for at least a first pixel of the view image, the view synthesis circuitry selecting a rearmost pixel when the distance between the view pose and the first region is greater than a threshold, and selecting a frontmost pixel when the distance between the view pose and the first region is less than the threshold, the rearmost pixel being associated with a depth value indicative of a depth furthest from the view pose and the frontmost pixel being associated with a depth value indicative of a depth closest to the view pose; The image synthesis device according to any one of claims 1 to 12.

14. three-dimensional image data describing at least a portion of a three-dimensional scene; a data field indicating whether the rendering of the three-dimensional image data must include adapting the transparency for an image region of the three-dimensional image data in the rendered image depending on a depth indication for the image region and a distance between a view pose for the rendered image and a reference region for the image region; An image signal device for generating an image signal.

15. The image signal further comprising at least one of an indication of the image region and the reference region.

15. An image signal device according to claim 14.

16. 1. A method of image synthesis, the method comprising: receiving three-dimensional image data describing at least a portion of a three-dimensional scene; identifying at least a first image region in the three-dimensional image data; determining a depth indication for the first image region from depth data of the three-dimensional image data for the first image region; identifying a first region for the first image region; receiving a view pose for a viewer; generating a view image from the three-dimensional image data, the view image representing a view of the three-dimensional scene from the view pose, the generating of the view image comprising adapting a transparency for the first image region in the view image in response to the depth indication and a distance between the view pose and the first region, the transparency increasing as the distance between the view pose and the first region increases and as the depth indication indicates a decreasing depth relative to the first image region; The method comprising:

17. The program comprises computer program code means for carrying out all the steps of the method according to claim 16 when the program is executed on a computer. Computer program.

Images