Apparatus and method for generating image signal

Abstract

An image source (407) provides an image divided into segments of different sizes, wherein only a subset of the segments include image data. A metadata generator (409) generates metadata structured according to a tree data structure, wherein each node is linked to a segment of the image. Each node is a branch node or a leaf node, wherein the branch node links a parent node to a child node, wherein the child node is linked to a segment that is a subdivision of the parent node, and wherein the leaf node has no child nodes. A leaf node is either an unused leaf node linked to a segment in which the first image does not include image data, or a used leaf node linked to a segment in which the first image includes image data. The metadata indicates whether each node is a branch node, a used leaf node, or an unused leaf node. An image signal generator (405) generates an image signal, wherein the image signal includes the image data of the first image and the metadata.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for generating an image signal and in particular, but not exclusively, to generating and / or processing an image signal comprising a plurality of images of the same scene. Background Art

[0002] The variety and range of image and video applications has increased significantly in recent years, with the continuous development and introduction of new services and ways to utilize and consume video.

[0003] For example, one increasingly popular service provides image sequences in such a way that the viewer can actively and dynamically interact with the system to change the parameters of the rendering. In many applications, a very attractive feature is the ability to change the viewer's effective viewing position and viewing direction, for example allowing the viewer to move and "look around" within the rendered scene.

[0004] Such a feature can in particular allow to provide a user with a virtual reality experience. This can allow the user, for example, to move (relatively) freely in a virtual environment and to dynamically change his position and where he is looking. Typically, such virtual reality applications are based on a three-dimensional model of a scene, wherein the model is being dynamically evaluated to provide a specifically requested view. This approach is well known from, for example, gaming applications for computers and consoles (e.g. in the first-person shooter genre).

[0005] Particularly for virtual reality applications, it is also desirable that the images presented are three-dimensional. Indeed, in order to optimize the viewer's immersion, it is often preferred that the presented scene experienced by the patient be three-dimensional. Indeed, a virtual reality experience should preferably allow the user to select his / her own position, camera viewpoint, and time of day relative to the virtual world.

[0006] An inherent limitation of many virtual reality applications is that they are based on predetermined models of existing scenes, and often on artificial models of the virtual world (e.g. for gaming applications).

[0007] However, it is desirable to provide a virtual reality experience that allows a user to experience a real-world capture. Such applications include, for example, applications that allow a user to freely change the position and viewing direction in a representation of a real-world scene. However, in many cases, it is difficult to meet the requirements for supporting such degrees of freedom, for example, particularly when the real-world scene (particularly a real-time scene) is also dynamically changing. For example, it is usually impractical or infeasible to provide enough data to allow a user to freely move and change the viewing direction for a scene corresponding to a live, real-time broadcast of, for example, a sporting event. Therefore, applications and experiences in which the user has limited freedom in movement and / or viewing direction are receiving increasing attention. For example, a sporting event may be broadcasted in which a user can freely rotate his head within a range of 180°, but can only move his head by a relatively small amount. Such limited movement greatly reduces the requirements for the data that need to be provided.

[0008] For most applications that allow local image rendering for different viewpoints of a scene, a key issue is how to represent such scenes and in particular how to efficiently generate, distribute and process data representing the real-world scene so that end-user devices are provided with enough data to locally generate viewing images of the real-world (and often real-time) scene. Generating a model of the real-world scene is often not feasible or practical, and in particular not the case when the service is supporting a scene that is dynamically changing (e.g., a real-time event).

[0009] In many systems, a scene can be represented by images that have been captured by a suitable capture device (e.g., a camera). For example, the cameras can be arranged in a given configuration (e.g., in a row), where each camera captures the scene from a given capture pose. Images from different positions can provide representations of different parts of the scene. For example, a background object can be occluded by a foreground object from some capture positions, but not by a foreground object from other capture positions, so information about the background object can be present in some captured images but not in other captured images.

[0010] In many practical systems, the captured image can be supplemented by depth information (e.g., z-values ​​or disparity values ​​provided for each pixel in the associated depth map). Such an image+depth representation can be considered a 3D image. The use of image+depth information provided for a range of viewpoints can have many applications, and for many scenes provides an advantageous scene representation that allows local generation of viewing images. The image+depth information can be sent to a rendering device, which can dynamically generate a viewing image for the user's current viewing position and viewing direction.

[0011] However, while in many embodiments an image representation of a scene including multiple images from different capture positions can provide desired performance and operation, it also tends to require a high data rate for transmitting the image representation to the rendering device. In practice, it is often not feasible to directly distribute all captured viewing images because the pixel rate (and therefore the data rate) is too high. It is also wasteful to directly send all images because it includes sending a large amount of redundant data. For example, the front of a foreground object can be seen from multiple capture positions, so visual information for that object will be included in multiple captured images to be sent.

[0012] However, the problem of how to reduce the required data rate is a complex problem that is difficult to solve. It has been proposed to identify and omit certain redundant data and then generate the image signal without the redundant data. However, although this may reduce the data rate, how to specifically achieve this and optimize the image quality, data rate, complexity, resource requirements, etc. as much as possible is a difficult challenge.

[0013] In practice, a problem often encountered when transmitting potentially redundant data / information is not only how to select or reduce data to reduce redundancy, but also specifically how to indicate which data being transmitted is relevant and which data being transmitted is not relevant. For example, for transmitting image data for several images including at least partially redundant data, it is not only a challenge to determine which data to transmit and how to transmit the data effectively, but also how to effectively indicate which data is valid / appropriate / relevant and which data is not valid / appropriate / relevant. For example, for partial images, it is a challenge to indicate which portions include valid image data and which portions do not include valid image data.

[0014] Hence, improved methods would be advantageous. In particular, methods for generating and / or processing image signals representing a scene from images seen from different views, which methods allow improved operation, increased flexibility, improved virtual reality experience, reduced data rates, increased efficiency, ease of distribution, reduced complexity, ease of implementation, increased image quality and / or improved performance and / or operation would be advantageous. Summary of the invention

[0015] Accordingly, the Invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination.

[0016] According to one aspect of the present invention, there is provided an apparatus for generating an image signal, the apparatus comprising: an image source (407) for providing a first image, the first image being divided into a group of segments of different sizes, the first image comprising only image data in a segment subset of the group of segments, the image data of the segments in the segment subset being image data of segments from a group of viewing source images, the group of viewing source images representing a scene seen from a viewpoint; a metadata generator (409) for generating metadata indicating image data content of the first image, the metadata being structured according to a tree data structure, wherein each node is linked to a segment of the first image; each node is a branch node point or leaf node, a branch node is the parent node linking a parent node to at least one child node, each child node of the parent node is linked to a child segment generated by subdividing the segment of the parent node, a leaf node has no child nodes and is linked to a segment in the set of segments, a leaf node is either an unused leaf node linked to a segment in which the first image does not include image data, or a used leaf node linked to a segment in which the first image includes image data, and the metadata indicates whether each node is a branch node, a used leaf node or an unused leaf node; and an image signal generator, which is used to generate an image signal, the image signal including the image data of the first image and the metadata.

[0017] The invention may provide an improved scene representation.In many embodiments, a more efficient scene representation can be provided which, for example, allows a given quality to be achieved for a reduced data rate.

[0018] In many embodiments, the method can provide an improved image signal having a scene representation suitable for flexible, efficient and high performance local generation of viewing images for different viewing positions / postures. In many embodiments, it can allow for improved perceived image quality and / or reduced data rates.

[0019] In many embodiments and scenarios, the method may allow for particularly efficient and / or low-complexity generation of image signals.

[0020] The image may be a 3D image including depth information, for example a 2D image or a texture map having an associated depth image / depth map. Selecting a set of selected images from a set of candidate images in response to the prediction quality may include selecting an image to be included in the set of selected images that has a lowest prediction quality metric in the set of candidate images.

[0021] The set of segments includes segments containing image data for different viewing source images. The set of segments may include: at least one segment including image data from a first viewing source image, and at least one segment including image data from a second (different) viewing source image from the set of viewing source images.

[0022] The image data of the first image is structured according to the tree-like data structure.

[0023] The set of segments may be segments of a partial image of the viewing source image. The partial image may include only a subset of pixel values ​​corresponding to the viewing source image.

[0024] In many embodiments, the image signal may include an indication of a viewing source image origin for at least one segment of the subset of segments. The indication of the viewing source image origin may indicate a viewing source image from which image data for at least one segment of a set of viewing source images originates and / or may indicate an indication of a position in a viewing source image of the set of viewing source images of image data for at least one segment.

[0025] According to an optional feature of the invention, the metadata further comprises an indication of the viewing source image of a plurality of viewing source images for the image data of the fragment linked to the leaf node of use.

[0026] This may provide an effective image signal, thereby facilitating the generation of original viewing source images or partial representations of these original viewing source images.

[0027] According to an optional feature of the invention, the metadata further comprises an indication of a position difference between a position in the first image of a first segment linked to a used leaf node and a position in the viewing source image of a corresponding segment of the viewing source image.

[0028] This may provide an effective combination of different viewing source images into the first image while allowing the original viewing source image or parts thereof to be recreated from the image signal. The indication may specifically be an indication of a translation or movement of the segment from a position in the first image to a position in the viewing source image.

[0029] According to an optional feature of the invention, the image signal includes at least some of the metadata as a data stream, the data stream comprising an ordered sequence of node data blocks for at least multiple nodes of the data tree structure, each node data block comprising data indicating a set of attributes for the node, the set of attributes including an indication of whether the node is a branch node or a leaf node.

[0030] In many embodiments this may provide for a particularly efficient transmission of metadata and image data.In some embodiments, the ordered sequence may be predetermined and known in advance by any receiver of the image signal.

[0031] According to an optional feature of the invention, the ordered sequence comprises a plurality of fragment attribute indications, each fragment attribute indication indicating an attribute of at least one fragment attribute applicable to a subsequent node data block.

[0032] This may provide for efficient encoding of metadata. The attribute may be applied to subsequent node data blocks until a fragment attribute indication is received indicating a different value for the attribute. The fragment attribute indication may be part of a node data block or may be included, for example, as separate data between node data blocks. The attribute may also be applied to a node data block in which a fragment attribute indication is received.

[0033] In some embodiments, the segment attribute indication indicates at least one of a viewing source image indication and a viewing source image position indication for a subsequent node data block.

[0034] According to an optional feature of the invention, the image data comprises an indication of the ordered sequence.

[0035] In many embodiments, this may provide advantageous operation and / or performance.

[0036] According to an optional feature of the invention, the subdivision of the fragments of the branch node is selected according to one of a set of allowable subdivisions, and the metadata comprises data for the branch node indicating a subdivision of the set of allowable subdivisions for the branch node.

[0037] This may provide a particularly efficient image signal.In some embodiments, a set of permissible subdivisions may be predetermined.

[0038] According to an optional feature of the invention, the metadata comprises an indication of the set of permissible subdivisions.

[0039] This may allow the use of efficient communication and efficient segmentation. This may in particular allow flexible segmentation adjustment.

[0040] According to an optional feature of the invention, the set of view source images comprises a plurality of view source images representing the scene as seen from different viewpoints.

[0041] According to an optional feature of the invention, a root node of the data tree structure is linked to a segment corresponding to the entire first image.

[0042] According to an optional feature of the invention, the metadata comprises an indication of a minimum segment size, and wherein the sizes of the segments in the set of segments are multiples of the minimum segment size.

[0043] This may provide an advantageous partitioning suitable for representation by a tree-like data structure.

[0044] The outline of the fragments that can be created by tiling using the fragments having the minimum fragment size can be drawn. Therefore, all fragments can be divided into a plurality of fragments having a size corresponding to the possible minimum fragment.

[0045] According to an optional feature of the present invention, the device also includes: a segmenter (401) for generating a segmented partial image for each viewing source image of at least some of the viewing source images in the group of viewing source images, the segmented partial image for the viewing source image comprising a plurality of segments having contour lines, the contour lines being selected from a set of predetermined contour lines, and wherein the segments are divided into used segments in which at least some pixels include image data from the viewing source image and unused segments in which no pixels include image data from the viewing source image; and a combiner (403) for generating at least some of the segments of the group of different sizes by including used segments from the at least some of the viewing source images in the first image and not including unused segments from the at least some of the viewing source images in the first image, thereby generating the first image.

[0046] According to an optional feature of the present invention, the segmenter (401) is arranged to generate a first segmented partial image for a first viewing source image by: dividing the pixels of the first viewing source image into a first group of pixels in which pixel data is to be included in the image data and a second group of pixels in which pixel data is not to be included in the image data; and tiling the first viewing source image into segments having contour lines, wherein the contour lines are selected from the set of predetermined contour lines, and the tiling is such that unused segments are generated by fitting the predetermined contour lines to areas of pixels belonging to the second group of pixels.

[0047] According to another aspect of the present invention, there is provided an apparatus for processing an image signal, the apparatus comprising: a receiver (303) for receiving an image signal, the image signal comprising: image data of a first image, the first image being divided into a group of segments of different sizes, the first image comprising only image data in a segment subset of the group of segments, the image data of the segments in the segment subset being image data of segments from a group of viewing source images, the group of viewing source images representing a scene seen from a viewpoint; metadata indicating the image data of the first image, the image data being structured according to a tree data structure, wherein each node is linked to a node in the first graph. a fragment of an image; each node is a branch node or a leaf node, a branch node is the parent node linking a parent node to at least one child node, each child node of the parent node is linked to a child fragment generated by subdividing the fragment of the parent node, a leaf node has no child nodes and is linked to a fragment in the set of fragments, a leaf node is either an unused leaf node linked to a fragment in which the first image does not include image data, or a used leaf node linked to a fragment in which the first image includes image data, and the metadata indicates whether the node is a branch node, a used leaf node or an unused leaf node; a renderer, which is used to draw a synthetic viewing image based on the image signal.

[0048] In some embodiments, the renderer is arranged to: extract image data for a set of viewing source images from the first image in response to the metadata; and render the composite viewing image in response to the extracted image data for the set of viewing source images.

[0049] In some embodiments, the apparatus may include a viewing source generator configured to generate a partial image of a first viewing source image of the group of viewing source images by filling an area of ​​the partial image with a segment of image data obtained based on metadata. The renderer may render a synthesized viewing image based on the partial image.

[0050] According to another aspect of the present invention, a method for generating an image signal is provided, the method comprising: providing a first image, the first image being divided into a group of segments of different sizes, the first image comprising only image data in a segment subset of the group of segments, the image data of the segments in the segment subset being image data of segments from a group of viewing source images, the group of viewing source images representing a scene seen from a viewpoint; generating metadata indicating image data content of the first image, the metadata being structured according to a tree data structure, wherein each node is linked to a segment of the first image; each node is a branch node or a leaf node, and the branch node A point is a parent node linking a parent node to at least one child node, each child node of the parent node is linked to a child segment generated by subdividing the segment of the parent node, a leaf node has no child nodes and is linked to a segment in the set of segments, a leaf node is either an unused leaf node linked to a segment in which the first image does not include image data, or a used leaf node linked to a segment in which the first image includes image data, and the metadata indicates whether each node is a branch node, a used leaf node, or an unused leaf node; and an image signal is generated, the image signal including the image data of the first image and the metadata.

[0051] According to another aspect of the present invention, a method for processing an image signal is provided, the method comprising: receiving an image signal, the image signal comprising: image data of a first image, the first image being divided into a group of segments of different sizes, the first image comprising only image data in a segment subset of the group of segments, the image data of the segments in the segment subset being image data of segments from a group of viewing source images, the group of viewing source images representing a scene seen from a viewpoint; metadata indicating the image data of the first image, the image data being structured according to a tree data structure, wherein each node is linked to a segment of the first image ; each node is a branch node or a leaf node, a branch node is the parent node linking a parent node to at least one child node, each child node of the parent node is linked to a child segment generated by subdividing the segment of the parent node, a leaf node has no child nodes and is linked to a segment in the set of segments, a leaf node is either an unused leaf node linked to a segment in which the first image does not include image data, or a used leaf node linked to a segment in which the first image includes image data, and the metadata indicates whether the node is a branch node, a used leaf node or an unused leaf node; and a synthetic viewing image is drawn based on the image signal.

[0052] These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0054] Figure 1 An example of an apparatus for providing a virtual reality experience is illustrated;

[0055] Figure 2 illustrates an example of elements of an image signal transmitter according to some embodiments of the present invention;

[0056] Figure 3 illustrates examples of elements of an image signal receiver according to some embodiments of the present invention;

[0057] Figure 4 illustrates an example of elements of an image generator for an image signal transmitter according to some embodiments of the present invention;

[0058] Figure 5 illustrates an example of a combined partial image according to some embodiments of the present invention;

[0059] Figure 6 illustrates an example of an enlarged combined partial image according to some embodiments of the present invention; and

[0060] Figure 7 illustrates an example of a tree data structure according to some embodiments of the present invention; and

[0061] Figure 8 An example of subdivision of fragments according to some embodiments of the present invention is illustrated. DETAILED DESCRIPTION

[0062] Virtual experiences that allow users to move around in a virtual world are becoming increasingly popular, and services are being developed to meet such demand. However, providing effective virtual reality services is very challenging, especially when the experience is based on a capture of the real-world environment rather than a completely virtually generated artificial world.

[0063] In many virtual reality applications, a viewer gesture input is determined that reflects the pose of a virtual viewer in a scene. The virtual reality device / system / application then generates one or more images corresponding to the viewer scene view and viewport corresponding to the viewer gesture.

[0064] Typically, virtual reality applications generate three-dimensional outputs in the form of separate viewing images for the left and right eyes. These three-dimensional outputs can then be presented to the user by suitable modules (e.g., typically separate left-eye and right-eye displays of a VR headset). In other embodiments, the images may be presented, for example, on an autostereoscopic display (in which case a larger number of viewing images may be generated for the viewer's posture), or indeed in some embodiments, only a single two-dimensional image may be generated (e.g., using a conventional two-dimensional display).

[0065] In different applications, viewer gesture input can be determined in different ways. In many embodiments, the physical movement of the user can be tracked directly. For example, a camera viewing the user area can detect and track the user's head (or even eyes). In many embodiments, the user can wear a VR head-mounted device, which can be tracked by external modules and / or internal modules. For example, the head-mounted device may include an accelerometer and a gyroscope that provide information about the movement and rotation of the head-mounted device, thereby providing information about the movement and rotation of the head. In some examples, the VR head-mounted device can send a signal or include a (e.g., visual) identifier that enables an external sensor to determine the movement of the VR head-mounted device.

[0066] In some systems, viewer gestures may be provided by manual means (e.g., by the user manually controlling a joystick or similar manual input.) For example, a user may manually control the direction in which the virtual viewer is looking by controlling a first analog joystick with one hand, and manually moving a second analog joystick with the other hand.

[0067] In some applications, a combination of manual and automatic methods can be used to generate the input viewer pose. For example, a head mounted device can track the orientation of the head, and the user can use a joystick to control the viewer's movement / position in the scene.

[0068] The generation of the image is based on a suitable representation of the virtual world / environment / scene. In some applications, a complete three-dimensional model for the scene can be provided, and the view of the scene seen from a specific viewer posture can be determined by evaluating the model. In other systems, the scene can be represented by image data corresponding to views captured from different capture postures, and specifically by multiple source images with associated depths, wherein each image represents the scene seen from a different viewpoint. In such a method, viewing images for other postures other than this (one or more) capture postures can be generated by three-dimensional image processing, for example, by using a viewing shift algorithm to achieve this. In a system that describes / references a scene by viewing data stored for discrete viewpoints / positions / postures, these discrete viewpoints / positions / postures may also be referred to as anchor viewpoints / positions / postures. Typically, when a real-world environment has been captured by capturing images from different points / positions / postures, these capture points / positions / postures are also anchor points / positions / postures.

[0069] Typical VR applications accordingly provide (at least) an image corresponding to a scene viewport for a current viewer pose, wherein the image is dynamically updated to reflect changes in the viewer pose, and wherein the image is generated based on data representing a virtual scene / environment / world.

[0070] In this field, the terms placement and attitude are used as common terms for position and / or direction / orientation. For example, the combination of the position and direction / orientation of a target, camera, head or view can be referred to as attitude or placement. Therefore, a placement or attitude indication can include six values / components / degrees of freedom, wherein each value / component generally describes the individual attributes of the position / position or orientation / direction of the corresponding target. Of course, in many cases, placement or attitude can be considered or represented using fewer components, for example, if one or more components are considered to be fixed or unrelated (for example, if all targets are considered to be at the same height and have a horizontal orientation, four components can provide a complete representation of the attitude of the target). In the following, the term attitude is used to refer to a position and / or orientation that can be represented by one to six values ​​(corresponding to the maximum possible degrees of freedom).

[0071] Many VR applications are based on gestures with the greatest degrees of freedom (i.e., three degrees of freedom each in position and orientation for a total of six degrees of freedom). Thus, a gesture may be represented by a set or vector of six values ​​representing the six degrees of freedom, and thus a gesture vector may provide a three-dimensional position and / or a three-dimensional orientation indication. However, it should be understood that in other embodiments, a gesture may be represented by fewer values.

[0072] The posture may be at least one of an orientation and a position. The posture value may indicate at least one of an orientation value and a position value.

[0073] Systems or entities based on providing the most freedom to the viewer are often referred to as having 6 degrees of freedom (6DoF).Many systems and entities only provide orientation or position, and these are often referred to as having 3 degrees of freedom (3DoF).

[0074] In some systems, VR applications can be provided locally to viewers, for example, by a standalone device that receives scene data from a remote device / server (which is independent of a specific viewer pose for a local viewer) and then locally generates a viewing image for a specific current view for the local viewer. Thus, in many applications, particularly for broadcast services, a source can send scene data in the form of an image (including video) representation of a scene that is independent of the viewer pose. For example, an image representation comprising multiple captured viewing images and associated depth maps can be received. Each client can then locally synthesize a viewing image corresponding to the current viewer pose.

[0075] A particular application of particular interest is to support a limited amount of movement, such that the rendered view is updated to follow small movements and rotations corresponding to a substantially static viewer, resulting in only small head movements and head rotations. For example, a seated viewer can turn his head and move his head slightly, with the rendered view / image being adjusted to follow these posture changes. Such an approach can provide a (e.g., video experience) with a high sense of immersion. For example, a viewer watching a sporting event may feel as if he is at a specific location in the arena.

[0076] Such limited freedom applications have the following advantages: an improved experience is provided while not requiring an accurate representation of the scene seen from many different positions, thereby greatly reducing the capture requirements. Similarly, the amount of data that needs to be provided to the renderer can be greatly reduced. In fact, in many scenarios, only the image and usually the depth data for a single viewpoint need to be provided to the local renderer, which can generate the desired view based on these images and depth data. In order to support head rotation, it is usually desired that a large area of ​​the view seen from the viewpoint is represented by the provided data, and preferably, the entire surface of the view sphere centered on the viewpoint is covered by the provided image and depth data.

[0077] In particular, the method may be well suited for applications that require data to be transferred from a source to a destination over a communication channel with limited bandwidth, such as broadcast applications or client-server applications.

[0078] Figure 1Such an example of a VR system is illustrated, wherein a remote VR client device 101 is in contact with a VR server 103, for example via a network 105 (e.g., the Internet). The server 103 may be arranged to support a potentially large number of client devices 101 simultaneously.

[0079] The VR server 103 may support a cast experience, for example, by sending image data and depth for multiple viewpoints, wherein the client device is then arranged to process this information to locally synthesize a viewing image corresponding to the current pose.

[0080] In order to provide efficient distribution, it is desirable to keep the data rate as low as possible for a given image quality, and this may specifically involve seeking to reduce the amount of redundant data generated.

[0081] Figure 2 An example of an apparatus for generating an image signal comprising a representation of a plurality of images of a scene seen from different viewing postures (anchor postures) is illustrated. The apparatus will also be referred to as an image signal transmitter 200. The image signal transmitter 200 may, for example, be included in Figure 1 In the VR server 103.

[0082] Figure 3 An example of an apparatus for presenting a viewing image based on a received image signal comprising a representation of a plurality of images of a scene is illustrated. The apparatus may specifically receive a plurality of images of a scene. Figure 2 The device generates a data signal and continues to process the data signal to draw an image for a specific viewing posture. Figure 3 The device will also be referred to as the image signal receiver 300. The image signal receiver 300 may be included in, for example Figure 1 In the client device 101.

[0083] The image signal transmitter 200 includes an image source receiver 201, which is arranged to receive multiple source images of a scene. The source images may represent views of the scene seen from different capture postures. For example, the source images may include images seen from a row of equally spaced capture postures.

[0084] In many embodiments, the source image may be a 3D image that includes a 2D image with associated depth information. The 2D image may specifically be a viewing image for a scene viewport as seen from a corresponding capture pose, and the 2D image may be accompanied by a depth image or depth map of depth values ​​for each of the pixels of the 2D image. The 2D image may be a texture map.

[0085] The depth value may be, for example, a disparity value or a distance value (e.g., indicated by a z coordinate). In some embodiments, the source image may be a 3D image in the form of a texture map with an associated 3D mesh. In some embodiments, such a texture map and mesh representation may be converted by the image source receiver into an image+depth representation before being further processed by the image signal transmitter 200.

[0086] Accordingly, the image source receiver 201 receives a plurality of source images that characterize and represent scenes seen from different postures. Such a set of source images will allow the use of algorithms such as viewing shifts known to the skilled person to generate viewing images for other postures. Accordingly, the image signal transmitter 200 is arranged to generate an image signal that includes image data for the source images and sends the data to the remote device for local rendering. However, directly sending all source images will require an unfeasibly high data rate and will also include a large amount of redundant information.

[0087] The image signal transmitter 200 is arranged to reduce the data rate by generating partial images from which redundant information has been removed. The partial images are then combined and encoded to generate an image signal.

[0088] The image source receiver 201 is coupled to a pixel selector 203, which is arranged to generate a set of images from a source image, wherein at least one of the images in the set of images is a partial image. The pixel selector 203 may generate the set of partial images by generating a partial version of one, or typically most (or even all) of the source images by selecting a subset of pixels in the source image to be included in the image signal. A partial image will also be referred to as a cropped image, and selecting a subset of pixels of an image to generate a partial image thereof will also be referred to as a cropped image.

[0089] Thus, the output of the pixel selector 203 may be a set of cropped images or partial images corresponding to the source images, but wherein one or typically most of the images are partial versions of the corresponding source images. The pixel selector 203 seeks to select the subset of pixels for inclusion in the partial images such that the overall redundancy of the generated set of partial images is reduced or preferably minimized. The generated partial images may also include one or more of the original source images that have not been cropped.

[0090] The pixel selector 203 is coupled to an image generator 205, which is fed with the set of partial images. The image generator 205 is arranged to combine the partial images into a combined image, wherein each image may represent one or more partial images in the partial images. In particular, the combined image may include pixels originating from a plurality of partial images. For example, pixels from a partial image may be inserted into an unused region of another partial image. The image generator 205 can be viewed as packing the partial images into a more densely packed combined image.

[0091] Image generator 205 generates fewer but less sparse images. This set of combined images is fed to encoder 207, which in turn performs encoding of the images. Since image generator 205 has reduced the number of images to be sent, more efficient encoding is generally achieved. In addition, partial images can generally be packed into combined images, making the resulting image very suitable for encoding.

[0092] A particular advantage of this approach is that it enables scene representations obtained from partial images in a manner that allows encoding of conventional images and videos to be performed by the encoder 207. For example, in many embodiments, a coding format such as High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2 or Versatile Video Coding (VVC) developed by the Joint Video Research Team (JVET), may be used.

[0093] Figure 2 The image signal transmitter 200 may accordingly provide an efficient method for generating an effective image signal representing a scene as seen from a range of viewing postures.

[0094] The image signal is sent to an image signal receiver 300, which comprises a decoder 301 arranged to receive the image signal and decode the image signal to generate a set of combined signals, which are fed to the encoder 207. Thus, the decoder 301 may be arranged to perform decoding according to the standard used by the encoder 207, for example by performing HEVC or VVC decoding.

[0095] It should be understood that the image signal transmitter 200 and the image signal receiver 300 also include the functions required for transmitting the image signal, including the functions required for encoding, modulating, sending, receiving the image signal, etc. It should be understood that such functions will depend on the preferences and requirements of each embodiment, and such techniques will be known to those skilled in the art, so for the sake of clarity and brevity, such techniques will not be further discussed herein.

[0096] The decoder 301 is coupled to an image retriever 303, which is arranged to retrieve partial images from the received combined image. Accordingly, the image retriever 303 may perform the inverse function of the image generator 205 in order to divide the pixels of the combined image into individual partial images. Thus, while the combined image typically includes pixels corresponding to a plurality of different viewpoints or poses, the partial images are generated such that each image includes pixels originating from only one source image (and therefore corresponding to only one viewing pose).

[0097] In many embodiments, the image signal also includes one or more complete source images, and the output of the image retriever 303 accordingly provides a set of images corresponding to the source images (and therefore representing the scene as seen from a range of viewing postures), wherein one or more of the images are only partial images. However, the missing portions in a given partial image typically correspond to redundant information that can be obtained from other images in the set of output images from the image retriever 303.

[0098] The image is fed to the first renderer 305, which is arranged to generate a new viewing image based on a set of received partial images (and any complete images). These images correspond to the original source images. It should be understood that any suitable algorithm can be used to draw the viewing image corresponding to a given viewer posture. For example, the first renderer 305 can be arranged to first generate the viewing image based on the received complete source image. This usually results in multiple holes caused by occlusion due to viewpoint changes. Data from the partial images can then be used to fill in such holes. It should be understood that those skilled in the art will be aware of many different algorithms and methods for synthesizing views for specific purposes based on images from other viewpoints and any suitable algorithm can be implemented by the first renderer 305.

[0099] In some embodiments, the image signal receiver 300 may include a second renderer 307, which is arranged to synthesize the viewing image directly from the received combined image. In many embodiments, the image signal receiver 300 will either include the first renderer 305 and the image retriever 303, or will include the second renderer 307. It should be understood that the second renderer 307 can use any suitable method to present the viewing image for a given viewer posture.

[0100] The first renderer and the second renderer may use the same synthesis method and parameters, which may be advantageous because it may increase the value of the prediction.

[0101] It should be understood that any suitable method or approach may be used to generate a partial image. For example, in some embodiments, the pixel selector 203 may iteratively generate a set of partial images by selecting a new input image in each iteration, for which the pixel selector 203 then generates a partial image that is added to the set of partial images. In each iteration, the image may be selected, for example, randomly or in the order of distance from the center position of the captured gesture. The pixel selector 203 may then attempt to predict the selected image based on the previously selected partial image.

[0102] The predicted image for a first candidate image in a set of candidate images obtained based on a first included image in a set of included images can be an image for the viewing posture of the first candidate image generated by view synthesis based on the first included image. Since the first included image and the first candidate image correspond to source images for different viewing postures, the view synthesis includes a viewing posture shift and typically includes a viewing position shift. The view synthesis can be a viewing shift image synthesis. Therefore, the predicted image for the first candidate image obtained based on the first included image can be an image that reflects how well the viewport of the viewing posture from the candidate image can be predicted / estimated based on the first included image.

[0103] The prediction of the first image based on the second image can be specifically based on the view synthesis of the second image (and the viewing posture of the second image) to the image at the viewing posture of the first image. Therefore, the prediction operation of predicting the first image based on the second image can be a viewing posture shift of the second image from the viewing posture associated with it to the viewing posture of the first image.

[0104] It should be understood that different methods and algorithms may be used for view synthesis and prediction in different embodiments. In many embodiments, a view synthesis / prediction algorithm may be used that takes as input a synthetic viewing gesture for which a synthetic image is to be generated and a plurality of input images, each of the plurality of input images being associated with a different viewing gesture. The view synthesis algorithm may then generate a synthetic image for the viewing gesture based on the input image, which may typically include a texture map and depth.

[0105] Many such algorithms are known, and any suitable algorithm may be used without detracting from the invention. As an example of such an approach, an intermediate synthetic / predicted image may first be generated for each input image. This may be achieved, for example, by first generating a mesh for the input image based on a depth map of the image. The mesh may then be warped / shifted from the viewing pose of the input image to the synthetic viewing pose based on geometric calculations. The vertices of the resulting mesh may then be projected onto the intermediate synthetic / predicted image, and a texture map may be superimposed on the image. For example, such a process may be implemented using vertex processing and fragment shaders, such as known from a standard graphics pipeline.

[0106] In this way, an intermediate synthesized / predicted image for a synthesized viewing posture (hereinafter simply referred to as an intermediate predicted image) may be generated for each of the input images.

[0107] The intermediate prediction images may then be combined together, for example, by weighted combination / summing or by selection combination. For example, in some embodiments, each pixel of the synthesized / predicted image for the synthesized viewing pose may be generated by selecting a pixel from the frontmost intermediate prediction image, or may be generated by a weighted sum of corresponding pixel values ​​for all intermediate prediction images, where the weight for a given intermediate prediction image depends on the depth determined for that pixel. The combination operation is also referred to as a blending operation.

[0108] Thus, the pixel selector 203 can generate a prediction of the image selected in a given iteration based on the partial image selected in the previous iteration. It can compare the predicted image with the selected image and identify each pixel that failed to be predicted accurately enough, for example by determining whether a difference metric for the corresponding pixel in the predicted image and the selected image is above a threshold. The partial image can then be generated to include only these pixels, so the partial image does not include pixels that could be predicted accurately enough by the previously selected partial image. The partial image is then added to the set of partial images, and the pixel selector 205 proceeds to the next iteration, where a new image is selected.

[0109] In some embodiments, the image may be selected as the image that is least accurately predicted based on previously selected partial images.

[0110] In many embodiments, the pixel selector 203 can therefore generate a partial image that indicates, for each of the viewing source images, a group of pixels in which pixel value data / image data should be included in the image signal and a group of pixels in which pixel value data / image data should not be included in the image signal. The partial image for a given source image can simply indicate the two subgroups by including pixel values ​​of pixels belonging to the former group and not including pixel values ​​of pixels belonging to the latter group (rather than, for example, including a predetermined constant pixel value (e.g., zero)). It should be understood that in many embodiments, pixels are included in the partial image, or not based on an assessment of their ability to improve the prediction of views for other viewpoints, but any suitable method can be used, and the method described for effectively packaging and transmitting such partial images does not depend on any particular method for generating partial images. It should also be understood that references to pixel values ​​include references to multiple values ​​for a given pixel. For example, a pixel value can be a vector including multiple values ​​(e.g., values ​​for different color channels, transparency, and / or depth).

[0111] Figure 4 An example of some elements of the image generator 205 is illustrated. The image generator 205 is fed with partial images and then combines these partial images into fewer combined images. The combination is based on segmentation of the images and combining the segmented partial images into one or more segmented combined images. The segmentation and combination are based on hierarchical segmentation and combination using segments of different sizes and contours, where larger segments are subdivided into smaller segments. Accordingly, the segmentation and combination of the partial images is to generate a hierarchy and metadata in the form of a tree-like data structure to describe the content of the (one or more) combined images.

[0112] The image generator 205 comprises a segmenter 401 which receives the partial images from the pixel selector 203. The segmenter 401 is arranged to generate a segmented partial image for each of the partial images, and thus a segmented original viewing source image for each of the original viewing source images. The segmenter 401 may in particular divide the partial images into segments having contour lines selected from a set of predetermined contour lines.

[0113] Thus, a set of segment contours may be selected, and the segmentation of the portion of the image may be based on the constraint that the segment contour must be selected from one of these contours. The segment contour may correspond to the size and shape of the segment. For example, the segment may be constrained to be rectangular, and the contour may be defined by a width and a height (vertical and horizontal segment dimensions, for example measured in pixels).

[0114] The segment contours may be hierarchically related such that smaller segments are subdivisions of larger segments. For example, the set of segment contours may include one or more segments having a maximum size. For example, the maximum segment contour may be a rectangular segment having a given pixel dimension. The maximum segment contour may specifically be equal to the entire image.

[0115] The next level of segment contours can then be selected as a subdivision of the largest contour, for example by using a given subdivision (e.g., dividing the segment in half by a horizontal partition). In some embodiments, multiple subdivisions can be used, for example, another set of contours can be generated by dividing the segment in half by a vertical partition. The next level of segment contours can be generated by using the same (or different) subdivisions, etc. Thus, the set of possible segment contours can include contours of different sizes and possibly different shapes.

[0116] The segmenter 401 may then divide the partial image into segments having contour lines selected from a set of predetermined / possible contour lines. The segments are generated so that some segments include pixels that provide image data (and may also include pixels that do not provide image data), and some segments only include pixels that do not provide image data. Thus, image segments that do not include any image data and image segments that include image data are generated. Image segments that do not include image data (unused segments) may then be discarded, while image segments that do include image data (used segments) will be included in one of the combined images.

[0117] Therefore, the segmenter can tile the partial image by dividing the image into used fragments and unused fragments by adapting the predetermined contour line to the area of ​​pixels belonging to the second group of pixels. For example, the segmenter 401 can start with a fragment corresponding to the entire image. The segmenter can then divide the fragment into four sub-fragments. For each of the four sub-fragments, the segmenter 401 can determine whether the fragment includes any used pixels (wherein the partial image includes pixels of image data). If not, the fragment will not be further processed, and the fragment will be designated as an unused fragment. However, if the fragment does include any used pixels, the process of subdividing the fragment is repeated. The process can be repeated until the fragment including the used pixels reaches the minimum size allowed by the fragment. The image is now divided into unused fragments and a generally large number of minimum size fragments including used pixels, which have a maximum size for a given fragment shape for the area that adapts the unused pixels. The segmenter 401 can now then combine the minimum size fragments with used pixels into larger fragments, for example, if all subdivisions of the larger fragment include used pixels, this is achieved by iteratively combining smaller fragments into larger fragments. This will create a larger set of used fragments, so the process will end up with a set of used fragments and a set of unused fragments.

[0118] In some embodiments, multiple subdivisions may be made for a given segment, and segmenter 401 may select among these subdivisions using given criteria, such as selecting the subdivision that maximizes the difference between the concentrations of used pixels in different segments (i.e., is in a state of maximum equilibrium).

[0119] It should be understood that many other methods or algorithms may be used to generate hierarchically segmented partial images.

[0120] The segmenter 401 may then generate segmented partial images for all viewing source images, and these segmented partial images are then fed to the combiner 403, which is arranged to generate one or more combined partial images. The following description will focus on generating a single combined partial image, but it should be understood that in many embodiments, multiple partial images may be generated.

[0121] The combiner 403 may specifically include generating a single combined image for the used segments of multiple partial images. For example, in some embodiments, the combiner 403 may start by selecting a segmented partial image (e.g., a segmented partial image in which the used segments cover the largest area) as the initial combined partial image. Then, the combiner 403 may select a second segmented partial image, and then fill in the unused segments of the combined partial image with the used segments of the second segmented partial image. For example, the combiner 403 may sequentially select each used segment in the second segmented partial image, and identify the unused segments in the combined partial image with the same contour line for each of the used segments in these used segments. Then (e.g., by copying pixel values) the used segments of the second segmented partial image are moved to the unused segments in the combined partial image, after which they are designated as used segments. Then, the combiner 403 may then select a third segmented partial image, and then include the used segments of the third segmented partial image in the unused segments of the combined partial image. The method may be repeated for the segmented partial images until all of them have been processed or until no suitable unused segments are available in a combined partial image, in which case a new combined partial image may be started.

[0122] It will be appreciated that in many embodiments more complex algorithms may be used which seek to better optimize the combination of segments of the segmented partial image, including for example combining or dividing segments to provide a better fit.

[0123] Accordingly, the combined partial image includes used segments from a plurality of partial images. Some of the segments may be positioned in the combined partial image at the same positions as they were positioned in the segmented partial images, while other segments may have been moved to fit unused segments of the combined partial image.

[0124] Therefore, the combiner 403 may store, for each used segment, the origin of the segment or source data indicating the origin or source of the segment. In particular, the origin data may indicate the segmented partial image from which the segment was selected and may typically also indicate data reflecting the difference between the position of the segment in the combined partial image and the position in the segmented partial image.

[0125] Thus, the combiner 403 may generate a combined partial image which is segmented into segments of different sizes and wherein a segment comprises pixel values ​​from different segmented partial images. Figure 5 An example of a combined partial image is illustrated. Figure 5 An example of a combined partial image is illustrated, and Figure 6 Pictured Figure 5 A close up view of a segment of a combined partial image.

[0126] Specific examples of algorithms that may be used in generating combined partial images are as follows:

[0127] 1. The maximum node size is set to the highest power of two that is at most the width or height of the smallest frame (1024 for 1080p).

[0128] 2. Generate a set of scale-invariant node types from the smallest node size to the largest node size using powers of two, e.g., (8, 8), (16, 8), (8, 16), (16, 16), (32, 8), etc., with the same subdivisions at different scales, e.g., splitting a block in half horizontally or vertically.

[0129] 3. List all source views and packed viewing frame sizes.

[0130] 4. In increasing order of frame size:

[0131] a. Maintain a first list of rectangular boxes and insert the boxes for the entire frame.

[0132] b. Maintain a second list of boxes starting with empty.

[0133] c. Iterate until the first list is empty,

[0134] i. Get a rectangle from the list,

[0135] ii. Find the largest node type that can fit in the rectangle,

[0136] iii. Tile the rectangle and add the box to the second list.

[0137] iv. Divide any remaining pixels into rectangles and add to the first list.

[0138] d. Create a node type for a frame size that includes a single subdivision according to the boxes in the second list.

[0139] The generated combined partial images are fed to an image signal generator 405 which is fed with the combined partial image(s). Thus, the segmenter 401 and the combiner 403 form an image source 407 which provides one or more combined partial images, wherein each of the combined partial images is divided into segments of different sizes and wherein some of the segments are used and comprise image data from the viewing source image, while other segments are not used and do not comprise image data from the viewing source image.

[0140] The image signal generator 405 is arranged to generate an image signal comprising the combined partial images and may in particular comprise / correspond to the encoder 207 .

[0141] However, in addition to the image data, the image generator 205 comprises a metadata generator 409 which is arranged to generate metadata which is fed to the image signal generator 405 and also included in the image signal.

[0142] The metadata is generated to indicate the image data content of the combined partial image(s), and may in particular indicate the segmentation results of the combined partial image and the relationship between individual segments in the combined partial image and individual segments in the segmented partial images.

[0143] The image data and metadata are structured according to a tree-like data structure reflecting the segmentation results of the combined partial images. Figure 7 An example of a tree data structure is illustrated. The metadata particularly includes a tree data structure in which each node is linked to a segment of a combined partial image. The segment linked to a given segment can either be one of the defined used segments or one of the defined unused segments, or can be a combination of multiple segments, i.e., a node can be linked to a segment in a combined partial image whose contour and position are equal to the contour and position of one of the used combined partial images or the unused combined partial images formed in the image, or a node can be linked to the contour and position of a combination of multiple adjacent segments. Each node can be linked to a segment in the following sense, i.e., each node is linked to an area corresponding to a segment or a combination of multiple adjacent segments in a set of segments generated by the combiner 403 and included in the combined partial image.

[0144] The nodes of a tree data structure are either branch nodes or leaf nodes.

[0145] A branch node is a parent node having one or usually multiple child nodes. The child nodes of a branch node represent sub-segments of the segment of the branch node obtained by subdividing the segment of the branch node.

[0146] Thus, for example, a branch node may be linked to a segment of, for example, 100×200 pixels at a given position in the combined partial image. The branch node may be linked to a subdivision that divides the segment into four equal segments, and thus the segment may be subdivided into four corner segments of size 25×50 pixels. For each of these segments, the tree-like data structure may include child nodes of the branch node.

[0147] Thus, a branch node represents a partition of a segment into smaller segments.

[0148] The leaf nodes have no child nodes and thus correspond to segments of the combined partial image that are not further subdivided. Accordingly, the leaf nodes may correspond to used segments and unused segments generated by the combiner 403 .

[0149] For instance, in the above example, if the four 25×50 pixel fragments were used fragments or unused fragments generated by combiner 403 , they would not be further subdivided and therefore have no child nodes.

[0150] However, if, for example, one of these 25×50 pixel fragments covers, for example, a used fragment and an unused fragment, then the node for that fragment will not be a leaf node but a branch node with two child nodes, one child node corresponding to the leaf node for the used fragment and one child node corresponding to the leaf node for the unused fragment.

[0151] While branch nodes correspond to segments that are further subdivided in the combined partial image and do not directly match a used segment or an unused segment, leaf nodes are not subdivided but are directly linked to a used segment or an unused segment.

[0152] Therefore, the leaf node leaf is either an unused leaf node linked to a segment in which the first image does not include image data (i.e., linked to an unused segment), or a used leaf node linked to a segment in which the first image includes image data (i.e., linked to a used segment of the combined partial image).

[0153] Accordingly, the tree-like data structure has a structure and configuration that directly matches the combined partial image and the segmentation for the combined partial image. The tree-like data structure includes leaf nodes for each generated used segment and unused segment in the combined partial image. The tree-like data structure also includes a number of branch nodes that define the segmentation and in particular also define subdivisions of the segments that are performed to efficiently pack segments from different partial images together in the combined partial image.

[0154] The metadata generator 409 generates metadata describing the tree data structure, and in particular generates metadata indicating whether each node is a branch node, a used leaf node, or an unused leaf node. The metadata is fed to the image signal generator 405 and included in the image signal.

[0155] In many embodiments, the root node of the tree data structure may be linked to the segment corresponding to the entire combined partial image. This may provide an efficient method having one tree data structure providing information for the entire image. Thus, in many embodiments, one tree data structure may be provided for each combined partial image.

[0156] The tree data structure describes the segmentation of the combined partial image and the assignment of the segments as used segments or unused segments. In addition, in many embodiments, the tree data structure may include additional information for the segments. In particular, the metadata for each leaf node may include data indicating one or more attributes for the corresponding segment.

[0157] It will be appreciated that in different embodiments, different data may be included. For example, for a used fragment, an indication of the proportion of pixels within the fragment that are used pixels may be provided, ie, whether it is a sparsely packed fragment or a densely packed fragment.

[0158] However, in most embodiments, the metadata includes an indication of a leaf node for at least some of the uses of the viewing source image and a partial image that is the origin of the used fragments connected to the leaf node. In many embodiments, the metadata also includes an indication of the position difference (including, for example, rotation / orientation) between the position of the fragment in the combined partial image and the position in the original viewing source image / partial image. For example, translation information for the used fragment may be included. This approach can provide very efficient communication of the contents of the combined partial image and how these contents can be unpacked to provide the original partial image in this way.

[0159] In some embodiments, only explicit provenance data may be included for some of the used segments / leaf nodes. For example, the absence of explicit provenance data may indicate a default provenance for the combined partial image and for the base partial image of the segment at the same location. As a more advanced example, multiple predictions are generated and the metadata either includes provenance data or a selection of one of the predictions.

[0160] Indeed, while it is generally preferred to include at least some provenance data in the metadata / image signal for at least one segment in order to provide increased flexibility and adaptability, in other embodiments the image signal and metadata may not include any provenance data at all.

[0161] In some embodiments, the provenance data may be provided separately from the image signal, for example, by other hand segments. Typically for, for example, video frames, the image segmentation may change much more slowly than the individual image frame content, and the same image segmentation may be used for multiple frames (potentially for a very large number of frames). In such an example, the image signal including the video frame may be transmitted via a high-speed communication channel, while the provenance data may be provided in a completely separate signal and transmitted via a different communication channel.

[0162] In some embodiments, the image signal transmitter may not send any origin / source indication data anyway. For example, position or position difference data is generally only applicable to embodiments in which fragments can be moved from partial images to a combined image. In some embodiments, generating a combined image based on partial images can result in the fragments not being moved, so the use of position origin information is not required or appropriate. Although such an approach may result in less efficient packaging of fragments in a combined image, it may be advantageous in some applications because it may result in reduced complexity and improved processing convenience. For example, it may be suitable for applications in which the most relevant scene information tends to be present in different and specific areas of different partial images.

[0163] Furthermore, in some embodiments, no indication of the source / origin image is required for all viewing source images, or indeed, for some embodiments, no indication of the source / origin image is required for any viewing source image. For example, a predetermined relationship between the position or origin of the segments in the viewing source image and the position or order of the segments in the combined image(s) may be applied, and this relationship may be applied independently and separately by both the image signal transmitter and the image signal receiver without requiring image signals or metadata to describe the relationship. In some such cases, there is no need to transmit origin data at all.

[0164] As a specific example, a fixed area of ​​the combined image may be allocated to each of the partial images, e.g., an area corresponding to 50% of the combined image may be allocated to the first image, an area corresponding to 25% of the combined image may be allocated to the second image, an area corresponding to 15% of the combined image may be allocated to the third image, and an area corresponding to 10% of the combined image may be allocated to the fourth image. The pixel selection / fragment generation may follow the requirement that the number of selected pixels / generated fragments must be lower than the proportion of the combined image allocated to that image by a suitable margin (to allow for incomplete packing). The packing of fragments into the combined image may then follow the allocation of fragments into the areas allocated for the fragments of the partial image. The overall representation of the packing of the combined image may be the described hierarchical representation, and due to the known constraints on the fragment positions, no further image provenance data will be required.

[0165] In some embodiments, various combinations of requirements, constraints, and methods may be utilized. For example, the combined image may be larger than the partial images to allow for more flexible packing, thereby allowing for more packing constraints to be imposed (e.g., packing may be such that the positions of the fragments are maintained and each partial / source image is constrained to a specific area assigned to that image).

[0166] In some embodiments, specific known or assumed image characteristics may be used to determine an appropriate predetermined relationship between the position or sequence of segments in the source image and the position or sequence of segments in the combined image(s). For example, if a scene is considered static in a given transmission, where changes occur primarily in one area of ​​the scene (e.g., a stage in a theater), then a specific region corresponding to that region may be identified in the different viewing source images and used to determine a predetermined relationship known at both the image signal transmitter and the image signal receiver, e.g., identified regions in different partial images may be assigned to different regions in the combined image in a predetermined / known relationship.

[0167] Therefore, in most embodiments, it is preferred to include provenance data indicating the viewing image source and / or the location in the viewing image source for one or more of the fragments of the combined image, in particular for at least one fragment represented by a leaf node used. In many embodiments, this approach will allow for a more flexible and efficient approach. However, while this will often be advantageous, it is by no means necessary, required or required in all embodiments.

[0168] In some embodiments, there may be only one possible segment subdivision. For example, a rectangular segment may always be subdivided into four subsegments generated by divisions along the vertical centerline and the horizontal centerline. In such a case, the metadata may not include additional information about the subdivision for a given branch node.

[0169] However, in many embodiments, segmentation of the portion of the image may allow the use of different subdivisions, and in particular, the subdivision may be selected from a set of allowed subdivisions. In such a case, for one or more of the branch nodes, the metadata may include an indication of the subdivision applied to the segment of the branch node to generate the segment of the child node.

[0170] In particular, the segmentation may be based on a subdivision selected from a set of permissible subdivisions, e.g., splitting in half by a vertical center split, splitting in half by a horizontal center split, splitting in quarters by a vertical center split and a horizontal center split, etc. Then, for a given branch node, the metadata may include data indicating which of the set of permissible subdivisions has been used for the fragments of the branch node.

[0171] It will be appreciated that more complex subdivisions may be used and may depend on the properties of the segments which may have different shapes and contours. Figure 8 As shown, a (40, 40) pixel fragment / node can be split into two (40, 20) fragments / nodes or two (10, 40) fragments / nodes and one (20, 40) fragment / node. Accordingly, a 40×40 fragment / node has two subdivisions. A 40×20 fragment / node may have only a single subdivision, for example split into two 20×20 fragments / nodes. Thus, the set of permissible subdivisions may also depend on the specific fragment and in particular on the contour (size and / or shape).

[0172] In the described example, the system uses fragments that can be recursively subdivided until a minimum node size / fragment. Thus, in the example, the fragments may all have contours that can be generated as a combination of fragments of the minimum size. In particular, any given fragment can be tiled by fragments of the minimum size. The fragment can be viewed as being generated by combining basic building blocks corresponding to one (or possibly more) fragments of the minimum size. Such an approach may have many advantages, including:

[0173] Improve block alignment using video encoders (e.g., HEVC coded block trees)

[0174] Reduce the amount of metadata required when the segmentation can be represented with less data

[0175] In many embodiments, the metadata generator 409 may also be arranged to generate configuration information which may be sent to the image signal receiver 300 for use in unpacking the combined partial images.

[0176] In particular, in many embodiments, the image signal transmitter 200 may generate restrictive parameters for segmentation and a tree data structure for combined partial images.In many embodiments, metadata may be generated to include an indication of the smallest segment that forms the basis of all other segments.

[0177] As another example, in many embodiments, metadata may be generated to include an indication of what subdivisions are possible for a node / segment. For example, one or more sets of allowable subdivisions may be defined by providing outline data (e.g., shapes) to the segment being divided and the subsegments to be formed for a particular subdivision. Each subdivision may be assigned an identifier that may then be used when indicating a particular subdivision for a branch node.

[0178] Any suitable methods and algorithms may be used to implement the representation, encoding and structuring of metadata when generating data for an image signal.

[0179] In many embodiments, the metadata describing the tree-like data structure can be advantageously generated as a data stream having an ordered sequence of node data blocks for some and typically all of the nodes of the tree-like data structure. For example, the order in which the tree of the tree-like data structure is parsed can be predetermined, and the node data blocks can be sequentially provided in that order.

[0180] Each node data block may include data related to the node. For example, a node data block for a branch node may indicate the segmentation applied. A node data block for a leaf node may indicate whether it is a used leaf node or an unused leaf node, and a node data block for a used leaf node may also indicate the origin of the fragment (e.g., if it is not a default image / position), for example, by providing multiple origin partial images / viewing source images and an indication of the position of the fragment in the origin partial image.

[0181] For example, the data stream may begin with a node data block for a root node corresponding to the entire image. This may indicate a subdivision, thereby indicating a plurality of child nodes. The root node data block may be followed by a node data block for the first child node. If this is a branch node, the node data block may indicate a subdivision. The next node data block may then be used for the first child node of the node. This may continue until a node data block is included for the first encountered leaf node. The sequence may then proceed to the next child node of the parent node of the first leaf node, etc. Therefore, in this example, the node data blocks are arranged in the following order: the tree is parsed essentially in the vertical direction, wherein horizontal shifts occur only when a vertical path has been completed. In other embodiments, the sorting may focus on horizontal parsing. For example, node data blocks may be provided for all nodes at a given hierarchical level, and then node data blocks may be provided for all nodes at the next hierarchical level, etc.

[0182] In some embodiments, the metadata generator 409 may be arranged to also include an indication of the order of the sequence of node data blocks in the metadata, for example as part of the initial configuration data. This may allow a flexible and adaptive approach and may, for example, allow the image signal transmitter 200 to adjust the order according to the specific characteristics of the tree data structure.

[0183] In some embodiments, when a structured sequence of node data blocks is used, the image signal receiver 300 may be arranged to insert fragment attribute indications between node data blocks or include fragment attribute indications in node data blocks. Such fragment attribute indications may provide indications of attributes associated with fragments of nodes of later node data blocks, for example until a new fragment attribute indication is received. Thus, instead of defining specific attributes in each node data block, a common indication may be included that is applicable until a replacement / update indication is received. Fragment attribute indications may be provided in a node data block, i.e., a data value in a node data block may be considered as a fragment attribute indication to be applied before or after the node data block (i.e., it may be applied to nodes of the current node data block or only to (one or more) later nodes).

[0184] Thus, a given fragment attribute indication will apply to subsequent node data blocks (as well as the current node data block), and typically until another fragment attribute indication is received.

[0185] The fragment attribute indication may specifically indicate a viewing source image indication and / or a viewing source image position indication for a subsequent node data block. For example, a fragment attribute indication may be included, which indicates that the subsequent node data block relates to a fragment belonging to, for example, partial image No. 3, and which node data block is offset by a translation vector by, for example, a given number of horizontal pixels and vertical pixels.

[0186] Such an approach can provide efficient data representation.

[0187] Thus, in some embodiments, the described system may use a block tree structure to pack partial views. This approach may be well suited for parallel processing. It may also allow efficient representation of metadata and may provide blocks suitable for subsequent block-based image coding (e.g., HEVC).

[0188] A specific method for encoding a tree data structure may be as follows:

[0189] 1. Initialize by defining the following coded segment attribute indication: View number = 0, translation = (0, 0)

[0190] 2. For each node: Encoding: used leaves | unused leaves | branches (ternary or 2 bits)

[0191] a. When the leaf is used:

[0192] i. Coded view number changed? (1 digit)

[0193] ii. Has the code shift changed? (1 bit)

[0194] iii. When the view number changes: Encode the view number (unit 8 or unit 16)

[0195] iv. When the translation changes: Encoded translation (unit 16 pairs)

[0196] b. When it is a branch:

[0197] i. Coding subdivision

[0198] The translation (x, y) is divisible by the smallest node width and height.

[0199] Each node data block may include a node code, which can be represented as a 4-bit code:

[0200] This might result in the following Node code:

[0201] Code describe <![CDATA[00002(0)]]> Subdivision 0. <![CDATA[00012(1)]]> Segment 1 … … <![CDATA[10102(10)]]> Segment 10. <![CDATA[10112(11)]]> Unused leaves <![CDATA[11002(12)]]> Leaves w / o attributes <![CDATA[11012(13)]]> Leaves with viewing attributes <![CDATA[11102(14)]]> Leaves with translation properties <![CDATA[11112(15)]]> Leaves with these two properties

[0202] The encoding of the metadata may specifically be arithmetic coding, for example, context-adaptive binary arithmetic coding (CABAC) coding.

[0203] It will be appreciated that the method may be used with any suitable image encoder, or in examples where the images correspond to frames of a video sequence, with a video encoder.

[0204] However, advantageous performance can be achieved by aligning segments and nodes with the coded block tree of the video codec. Such (multi-level) block alignment can reduce texture and depth bitrates. Furthermore, metadata bitrate can be reduced by deriving predictions for node subdivisions and / or provenance data from the structure of the coded block tree (if the client device has access to this information).

[0205] Accordingly, the image signal receiver 300 may receive an image signal including a segmented combined partial image and a tree data structure defining how to generate the segmented combined partial image from the original partial image. The image retriever 303 may then regenerate the original partial image by extracting relevant segments from the received segmented combined partial image based on the metadata and locating the relevant segments in the partial image as indicated. Accordingly, the image retriever 303 may then provide the original partial image to the first renderer 305 for presentation.

[0206] Thus, the image retriever 303 can traverse the tree data structure in the appropriate order, copying each decoded block / segment to the indicated partial image. The size and position of each segment is known from the tree data structure, and the position in the partial image is known due to the translation / position metadata.

[0207] In other embodiments, for example, the second renderer 307 may directly parse the tree data structure and render each used leaf node / used fragment directly from the combined partial image without first generating a partial view. This is generally more memory efficient.

[0208] Thus, the image signal receiver 300 may be arranged to extract image data for a set of viewing source images in response to the metadata. Each of the viewing source images may correspond to a given viewpoint of the scene, and thus the extracted image data may be associated with the viewpoint. Thus, the renderer may extract image data for different viewpoints based on the metadata. The image data for different viewpoints may be extracted, for example, by considering a hierarchical tree structure as described above.

[0209] The extracted image data for this set of viewing source images can therefore be a partial image and can correspond to the original partial image. In some embodiments, the partial image can be completely recreated. In other embodiments, only the required or desired specific image data can be extracted.

[0210] Then, the first renderer 305 or the second renderer 307 can synthesize an image for a given viewpoint based on the extracted image data for different viewpoints. It should be understood that many algorithms are known for synthesizing an image for a given viewpoint based on image data from other viewpoints, and any suitable algorithm can be used. For example, as mentioned above, an image can be synthesized based on one (e.g., primary) viewing source image, and image data from other viewing source images can be used to fill in occlusion holes in the generated image.

[0211] It should be understood that, for the sake of clarity, the above description refers to different functional circuits, units and processors to describe embodiments of the present invention. However, it should be understood that any suitable distribution of functions between different functional circuits, units or processors may be used without departing from the present invention. For example, functions illustrated as being performed by a separate processor or controller may be performed by the same processor or controller. Therefore, references to specific functional units or circuits are only considered as references to suitable modules for providing the described functions, rather than indicating a strict logical or physical structure or organization.

[0212] The present invention can be implemented in any suitable form including any combination of hardware, software, firmware or these items.Optionally, the present invention can be implemented at least in part as computer software running on one or more data processors and / or digital signal processors.The elements and components of an embodiment of the present invention can be implemented physically, functionally and logically in any suitable manner.In fact, the function can be implemented in a single unit, in multiple units, or be implemented as the part of other functional units.Just for this reason, the present invention can be implemented in a single unit, or can be distributed between different units, circuits and processors physically and functionally.

[0213] Although the present invention has been described in conjunction with some embodiments, the present invention is not intended to be limited to the specific forms set forth herein. On the contrary, the scope of the present invention is limited only by the claims. In addition, although features may be described in conjunction with specific embodiments, those skilled in the art will recognize that the various features of the described embodiments may be combined according to the present invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

[0214] In addition, although listed separately, multiple modules, elements, circuits or method steps can be implemented by, for example, a single circuit, unit or processor. In addition, although individual features can be included in different claims, these features may also be advantageously combined, and the inclusion of features in different claims does not mean that the combination of features is not feasible and / or advantageous. Moreover, the inclusion of features in a type of claim does not mean that the feature is limited to this type, but indicates that the feature is equally applicable to other types of claims where appropriate. In addition, the order of features in the claims does not mean that the features must work in any particular order, and in particular, the order of the various steps in the method claims does not mean that the steps must be performed in this order. On the contrary, the steps can be performed in any suitable order. In addition, singular references do not exclude multiple. Therefore, references to "one", "one", "first", "second", etc. do not exclude multiple. The figure marks provided in the claims are only used to clarify the examples and should not be interpreted as limiting the scope of the claims in any way.

Claims

1. A device for generating an image signal, the device comprising: An image source (407) for providing a first image, the first image being divided into a set of segments of different sizes, the first image comprising only image data in a segment subset of the set of segments, the image data of the segments in the segment subset being image data of segments from a set of viewing source images, the set of viewing source images representing a scene seen from a viewpoint; A metadata generator (409) is used to generate metadata indicating the image data content of the first image, wherein the metadata is structured according to the following items: Tree data structure, wherein each node is linked to a segment of the first image; Each node is either a branch node or a leaf node. A branch node is the parent node that links a parent node to at least one child node, each child node of the parent node being linked to a child segment generated by subdividing the segment of the parent node, The leaf node has no child nodes and is linked to a fragment in the set of fragments, the leaf node is either an unused leaf node linked to a fragment in which the first image does not include image data, or a used leaf node linked to a fragment in which the first image includes image data, and the metadata indicates whether each node is a branch node, a used leaf node, or an unused leaf node; and An image signal generator (405) is used to generate an image signal, wherein the image signal comprises the image data of the first image and the metadata.

2. The device according to claim 1, wherein: The metadata also includes an indication of the viewing source images of a plurality of viewing source images for the image data of at least some segments linked to the leaf nodes used.

3. The device according to claim 1, wherein: The metadata also includes an indication of a difference in position between a position in the first image of a first segment linked to a used leaf node and a position in the view source image of a corresponding segment of the view source image.

4. The device according to any one of claims 1 to 3, wherein: The image signal includes at least some of the metadata as a data stream, the data stream comprising an ordered sequence of node data blocks for at least multiple nodes of the tree data structure, each node data block comprising data indicating a set of attributes for the node, the set of attributes including an indication of whether the node is a branch node or a leaf node.

5. The device according to claim 4, wherein: The ordered sequence comprises a plurality of fragment attribute indications, each fragment attribute indication indicating an attribute of at least one fragment attribute applicable to a subsequent node data block.

6. The device according to claim 4, wherein: The image data includes an indication of the ordered sequence.

7. The device according to any one of claims 1 to 3, wherein: The subdivision of the fragment of the branch node is selected according to one of a set of allowable subdivisions, and the metadata includes data for the branch node indicating a subdivision in the set of allowable subdivisions for the branch node.

8. The device according to claim 7, wherein: The metadata includes an indication of the set of permissible subdivisions.

9. The device according to any one of claims 1 to 3, wherein: The set of viewing source images includes a plurality of viewing source images representing the scene seen from different viewpoints.

10. The device according to any one of claims 1 to 3, wherein: The root node of the tree data structure is linked to the segment corresponding to the entire first image.

11. The device according to any one of claims 1 to 3, wherein: The metadata comprises an indication of a minimum segment size, and wherein the size of the segments in the set of segments is a multiple of the minimum segment size.

12. The apparatus according to any one of claims 1 to 3, further comprising: a segmenter (401) for generating a segmented partial image for each viewing source image of at least some of the viewing source images in the set of viewing source images, the segmented partial image for the viewing source image comprising a plurality of segments having contour lines, the contour lines being selected from a set of predetermined contour lines, and wherein the segments are divided into used segments in which at least some pixels comprise image data from the viewing source image and unused segments in which no pixels comprise image data from the viewing source image; and A combiner (403) for generating at least some of the set of segments of different sizes by including used segments from the at least some viewing source images in the first image and excluding unused segments from the at least some viewing source images in the first image, thereby generating the first image.

13. The device of claim 12, wherein: The segmenter (401) is arranged to generate a first segmented partial image for a first viewing source image by: dividing the pixels of the first viewing source image into a first group of pixels where pixel data is to be included in the image data and a second group of pixels where pixel data is not to be included in the image data; and The first viewing source image is tiled into segments having contours selected from the set of predetermined contours, the tiling being such that unused segments are generated by fitting the predetermined contours to regions of pixels belonging to the second set of pixels.

14. A device for processing an image signal, the device comprising: A receiver, configured to receive an image signal, wherein the image signal comprises: image data of a first image, the first image being divided into a set of segments of different sizes, the first image comprising only image data in a subset of segments of the set of segments, the image data of the segments in the subset of segments being image data of segments from a set of view source images, the set of view source images representing a scene as seen from a viewpoint; metadata indicating the image data of the first image, the image data being structured according to: Tree data structure, wherein each node is linked to a segment of the first image; Each node is either a branch node or a leaf node. A branch node is the parent node that links a parent node to at least one child node, each child node of the parent node being linked to a child segment generated by subdividing the segment of the parent node, The leaf node has no child nodes and is linked to a fragment in the set of fragments, the leaf node is either an unused leaf node linked to a fragment in which the first image does not include image data, or a used leaf node linked to a fragment in which the first image includes image data, and the metadata indicates whether the node is a branch node, a used leaf node, or an unused leaf node; A renderer (305, 307) is used to render a synthesized viewing image according to the image signal.

15. A method for generating an image signal, the method comprising: providing a first image, the first image being divided into a set of segments of different sizes, the first image comprising only image data in a segment subset of the set of segments, the image data of the segments in the segment subset being image data of segments from a set of viewing source images, the set of viewing source images representing a scene seen from a viewpoint; Metadata indicative of image data content of the first image is generated, the metadata being structured according to: Tree data structure, wherein each node is linked to a segment of the first image; Each node is either a branch node or a leaf node. A branch node is the parent node that links a parent node to at least one child node, each child node of the parent node being linked to a child segment generated by subdividing the segment of the parent node, The leaf node has no child nodes and is linked to a fragment in the set of fragments, the leaf node is either an unused leaf node linked to a fragment in which the first image does not include image data, or a used leaf node linked to a fragment in which the first image includes image data, and the metadata indicates whether each node is a branch node, a used leaf node, or an unused leaf node; and An image signal is generated, the image signal comprising the image data of the first image and the metadata.

16. A method for processing an image signal, the method comprising: receiving an image signal, wherein the image signal comprises: image data of a first image, the first image being divided into a set of segments of different sizes, the first image comprising only image data in a subset of segments of the set of segments, the image data of the segments in the subset of segments being image data of segments from a set of view source images, the set of view source images representing a scene as seen from a viewpoint; metadata indicating the image data of the first image, the image data being structured according to: Tree data structure, wherein each node is linked to a segment of the first image; Each node is either a branch node or a leaf node. A branch node is the parent node that links a parent node to at least one child node, each child node of the parent node being linked to a child segment generated by subdividing the segment of the parent node, The leaf node has no child nodes and is linked to a fragment in the set of fragments, the leaf node is either an unused leaf node linked to a fragment in which the first image does not include image data, or a used leaf node linked to a fragment in which the first image includes image data, and the metadata indicates whether the node is a branch node, a used leaf node, or an unused leaf node; and A synthesized viewing image is drawn based on the image signal.

17. A computer program product comprising computer program code means adapted to perform all the steps of any of claims 15 and 16 when said program is run on a computer.

Images