Device and method for generating an image signal

Abstract

A method for generating image signals from multiple images of a scene performs iterations including the step (505) of generating predicted images for a set of candidate images from a set of (previously) included images. A set of selected images is selected from the set of candidate images in response to the prediction quality (509). For each selected image, a subset of pixels is determined in response to the pixel prediction quality for the pixels of the image (511), and a set of partial images corresponding to the selected images is generated by selecting the subset of pixels (511) (513). The selected images are removed from the set of candidates, and the partial images are added to the set of included images. This approach can provide low complexity and low resource selection for image data representing the scene.

Description

Technology Field

[0001] The present invention relates to an apparatus and method for generating an image signal, and in particular, to the generation and / or processing of an image signal comprising, but not exclusively, multiple images of the same scene. Background Technology

[0002] The diversity and range of image and video applications have increased substantially in recent years, and new services and methods for using and consuming video are continuously being developed and introduced.

[0003] For example, one service that is becoming increasingly popular is providing image sequences in a way that allows the viewer to actively and dynamically interact with the system to change rendering parameters. A very attractive feature in many applications is the ability to change the viewer's effective viewing position and direction, such as allowing the viewer to move and "look around" within the presented scene.

[0004] Such features specifically enable virtual reality experiences to be provided to the user. This allows the user to move (relatively) freely within the virtual environment and dynamically change their position and where they are looking. Typically, such virtual reality applications are based on a three-dimensional model of a scene, which is dynamically evaluated to provide a specific requested view. This approach is well known for computers and consoles, for example, from game applications such as the category of first-person shooters.

[0005] In particular, for virtual reality applications, it is also desirable that the presented image be a three-dimensional image. In fact, to optimize viewer immersion, it is typically desirable for the user to experience the presented scene as a three-dimensional scene. In fact, the virtual reality experience should preferably allow the user to select their own position, camera viewpoint, and temporal moment within the virtual world.

[0006] Many virtual reality applications are inherently limited in that they are based on the existence of a predetermined model of the scene and, for example, on an artificial model of the virtual world, as is the case with gaming applications.

[0007] However, it is desirable to provide virtual reality experiences that allow users to experience real-world captures. Such applications include, for example, those that allow users to freely change their position and view direction within a representation of a real-world scene. However, the requirements for supporting this freedom are difficult to meet in many situations, particularly when the real-world scene is also dynamically changing and, specifically, a real-time scene. For example, providing sufficient data to allow a user to freely move and change their view direction regarding a scene corresponding to, for instance, a live broadcast of a sports event is typically impractical or unfeasible. Consequently, applications and experiences in which users have limited freedom of movement and / or view directions are receiving increasing attention. For example, a sports event may be broadcast where the user can freely rotate their head within a 180° range but move their head only by a relatively small amount. This limited movement can reduce the requirements for the data that actually needs to be provided.

[0008] MAUGEY THOMAS et al. " Reference View Selection in DI BR-Based Multiview Coding", IEEE TRANSACTIONS ON IMAGE PROCESSING, IEEE SERVICE CENTER, PISCATAWAY, NJ, US, vol. 25, no. 4, 1 April 2016 (2016-04-01), pages 1808-1819, XP011602607 In [the document], an algorithm for the optimal selection of reference views in multi-view coding systems was presented. It is explained that augmented reality and interactive navigation in 3D scenes, multi-view video, and other recently emerged multimedia applications require large sets of images, and therefore larger data volumes and increased resources compared to traditional video services. The significant increase in the number of images in multi-view systems gives rise to new challenges in data representation and transmission to provide high-quality experiences in resource-constrained environments. To reduce data size, different multi-view video compression strategies have recently been proposed. Most of these utilize the concepts of reference or principal views used to estimate other images when there is a high correlation in the dataset. Based on metrics measuring similarity between views, optimization for the positioning of reference views is performed to minimize both the distortion of view reconstruction and the coding rate cost.

[0009] A critical issue for most applications that allow for the local rendering of images for different viewpoints of a scene is how to represent these scenes—specifically, how to effectively generate, distribute, and process data representing real-world scenes so that sufficient data is provided to the end-user device to locally generate view images of real-world and often real-time scenes. Typically, creating a model of a real-world scene is not feasible or practical, especially when the service supports dynamically changing scenes such as real-time events.

[0010] In many systems, a scene can be represented by images captured by appropriate capture devices, such as cameras. For example, cameras may be arranged in a given configuration, such as in rows, and each camera captures the scene from a given capture pose. Images from different locations can provide representations of different parts of the scene. For example, a background object may be occluded by a foreground object from some capture locations but not from others, and thus information regarding the background object may be present in some captured images but not in others.

[0011] In many real-world systems, captured images can be supplemented by depth information, such as z-values ​​or disparity values ​​provided for each pixel within an associated depth map. This image-depth representation can be considered a 3D image. Utilizing image-depth information provided for a range of viewpoints can have many applications and can provide an advantageous representation of a scene that enables the local generation of view images for many scenes. The image-depth information can be transmitted to a rendering device capable of dynamically generating view images for the user's current view position and view direction.

[0012] However, while an image representation of a scene containing multiple images from different capture locations can provide desirable performance and operation in many embodiments, it also tends to require high data rates for communicating the image representation to a rendering device. In practice, directly distributing all captured view images is often impractical because the pixel rate, and thus the data rate, is too high. Directly transmitting all images is also wasteful because it involves transmitting a large amount of redundant data. For example, the front of a foreground object may be seen from multiple capture locations, and thus visual information about the object will be included in the multiple captured images to be transmitted.

[0013] However, the problem of how to reduce the required data rate is a complex problem that is difficult to solve. It has been proposed to identify and omit some redundant data and then generate image signals without this redundant data. However, while this can reduce the data rate, it is a difficult task to specifically achieve this so that image quality, data rate, complexity, resource requirements, etc., are optimized as much as possible.

[0014] Therefore, an improved approach will be advantageous. In particular, an approach for generating and / or processing image signals representing a scene by images from different views will be advantageous, enabling improved operation, increased flexibility, improved virtual reality experience, reduced data rates, increased efficiency, easy distribution, reduced complexity, easy implementation, increased image quality, and / or improved performance and / or operation.

[0015] Accordingly, the present invention aims to preferably mitigate, reduce, or eliminate one or more of the aforementioned disadvantages, either alone or in any combination.

[0016] According to a first aspect of the present invention, a method for generating an image signal is provided, the method comprising the steps of: receiving a set of candidate images including a plurality of images of a scene, wherein at least some of the images in the set of candidate images are for different view poses of the scene; selecting at least one image of the set of candidate images and initializing the set of included images to include at least one image; generating prediction images for the images of the set of candidate images from the images of the set of included images; determining a prediction quality measure for each image of the set of candidate images, wherein the prediction quality measure for the first image of the set of candidate images represents the difference between the first image and the first prediction image for the first image; selecting a set of selected images from the set of candidate images in response to the prediction quality measures; and for each image of the set of selected images, determining a subset of pixels in response to pixel prediction quality measures for the pixels of the image. The method comprises the steps of: generating a set of partial images corresponding to a set of selected images, wherein the partial image for a given image in the set of selected images comprises only pixels of a subset of pixels for the given image; including the set of partial images in a set of included images; and repeatedly performing the steps of removing a set of selected images from a set of candidate images; and generating an image signal including image data of the images within the set of included images.

[0017] The present invention can provide an improved representation of a scene. In many embodiments, a more efficient representation of a scene can be provided, for example, enabling a given quality to be achieved by a reduced data rate.

[0018] This approach can provide an improved image signal having a scene representation suitable for flexible, efficient, and high-performance local generation of view images for different view positions / poses in many embodiments. In many embodiments, it can enable improved perceived image quality and / or a reduced data rate.

[0019] This approach can enable the generation of image signals with particularly efficient and / or low complexity in many embodiments and scenarios.

[0020] The images may be 3D images containing depth information, such as 2D images or texture maps having specifically associated depth images / maps. The step of selecting a set of selected images from a set of candidate images in response to prediction qualities may include the step of selecting images from a set of candidate images having the lowest prediction quality measure to be included in the set of selected images.

[0021] According to an optional feature of the present invention, the step of determining the prediction quality for a first image comprises: determining a plurality of pixels of a first image such that pixel values ​​for corresponding pixels of the first image and the first prediction image satisfy a similarity criterion; and determining the prediction quality for the first image in response to the characteristics of the plurality of pixels.

[0022] This can provide a particularly advantageous approach in many embodiments.

[0023] According to an optional feature of the present invention, the characteristic is the number of pixels, and the prediction quality is a monotonically increasing function of the number of pixels within a plurality of pixels.

[0024] According to an optional feature of the present invention, the step of determining a subset of pixels includes the step of determining a subset of pixels for a first image from pixels of a first image that are not included in a plurality of pixels for the image.

[0025] According to an optional feature of the present invention, the step of determining a subset of pixels for a first selected image of a set of selected images comprises: a step of determining an error change representing a change in the difference between a pixel of at least one candidate image and a prediction pixel for a prediction of a pixel based on a set of included images and a first selected image for a prediction of a pixel based only on a set of included images, for at least one candidate image of a set of candidate images excluding the set of selected images, for pixels of at least one candidate image; a step of generating an error change image for a first selected image, wherein the error change image comprises pixel values ​​determined from the error changes of at least one candidate image by a view pose shift from the view pose of at least one candidate image to the view pose of the first selected image; and a step of selecting a subset of pixels in response to the error change image.

[0026] This can provide a particularly advantageous approach in many embodiments. In particular, it can provide an efficient approach for selecting pixels for partial images based on how well the inclusion of pixels can improve the prediction of other images.

[0027] The step of selecting a subset of pixels in response to an error change image may be, for example, by selecting pixels that indicate an improvement where the error change value exceeds a threshold.

[0028] According to an optional feature of the present invention, the step of determining a subset of pixels for a first selected image comprises: determining error changes for a plurality of candidate images of a set of candidate images; and generating an error change image by combining error change values ​​determined from different candidate images for the same pixel of the first selected image.

[0029] This can provide improved performance by efficiently considering how much the predictions of multiple candidate images can be improved by including specific pixels.

[0030] According to an optional feature of the present invention, the step of generating prediction images for a set of candidate images comprises: for a first candidate image of a set of candidate images, providing intermediate prediction images of the first candidate image for each of the images in the set of included images; and generating a first prediction image by combining the intermediate prediction images.

[0031] This can enable more efficient predictive operation in many embodiments.

[0032] According to an optional feature of the present invention, the iteration further comprises the step of storing intermediate prediction images; and the step of providing intermediate prediction images comprises the step of retrieving stored intermediate prediction images for images of a set of included images in which intermediate prediction images are generated and stored in a previous iteration; and the step of predicting intermediate prediction images for any image of a set of reference images in which intermediate prediction images are not stored in a previous iteration.

[0033] This can provide highly efficient operation in many embodiments. This approach enables incremental prediction, which can substantially reduce the amount of necessary prediction operations.

[0034] According to an optional feature of the present invention, the combination is a weighted combination, and the weight for the pixel depends on the depth of the pixel.

[0035] According to an optional feature of the present invention, iterations are stopped when the lowest prediction quality exceeds a threshold.

[0036] In other embodiments, iterations may be stopped, for example, when the set of candidate images is empty.

[0037] According to an optional feature of the present invention, a subset of selected images includes a plurality of selected images in at least one iteration.

[0038] According to an optional feature of the present invention, the images are frames of a video sequence, and the method includes the step of using an array of sets of images included for a plurality of frames.

[0039] This can provide improved temporal consistency in many embodiments.

[0040] According to an optional feature of the present invention, the method further comprises the step of generating a mask corresponding to a subset of pixels and modifying the subset of pixels in response to the application of a spatial filter to the mask.

[0041] According to another aspect of the present invention, an apparatus for generating an image signal is provided, wherein the apparatus is a receiver for receiving a set of candidate images including a plurality of images of a scene, wherein at least some of the images in the set of candidate images are for different view poses of the scene; a reference selector for selecting at least one image of the set of candidate images and initializing a set of included images to include at least one image; an iterator for generating prediction images for the images of the set of candidate images from the images of the set of included images; a step of determining a prediction quality measure for each image of the set of candidate images, wherein the prediction quality measure for a first image of the set of candidate images represents the difference between the first image and the first prediction image for the first image; a step of selecting a set of selected images from the set of candidate images in response to the prediction quality measures; and a step of determining a subset of pixels in response to pixel prediction quality measures for the pixels of the image for each image of the set of selected images. The method comprises: a step of generating a set of partial images corresponding to a set of selected images, wherein the partial image for a given image in the set of selected images comprises only pixels of a subset of pixels for the given image; a step of including the set of partial images in a set of included images; and a step of removing the set of selected images from a set of candidate images, wherein the method comprises an iterator for repeatedly performing these steps; and an output circuit for generating an image signal including image data of the images in the set of included images.

[0042] These and other aspects, features, and advantages of the present invention will be apparent from the embodiments(s) described below and will be explained with reference thereto. Brief explanation of the drawing

[0043] Embodiments of the present invention will be described merely as examples with reference to the drawings. Figure 1 illustrates an example of an array for providing a virtual reality experience. FIG. 2 illustrates examples of elements of an image signal transmitter according to some embodiments of the present invention. FIG. 3 illustrates examples of elements of an image signal receiver according to some embodiments of the present invention. FIG. 4 illustrates examples of repeater elements for an image signal transmitter according to some embodiments of the present invention. FIG. 5 illustrates an example of a method for generating image data for an image signal according to some embodiments of the present invention. Specific details for implementing the invention

[0044] Virtual experiences that allow users to move around in virtual worlds are becoming increasingly popular, and services are being developed to meet this demand. However, providing efficient virtual reality services is very difficult, especially when the experience must be based on capturing real-world environments rather than a completely virtual artificial world.

[0045] In many virtual reality applications, a viewer pose input is determined that reflects the pose of the virtual viewer within the scene. Subsequently, the virtual reality device / system / application generates one or more images corresponding to the views and viewports of the scene for the viewer corresponding to the viewer pose.

[0046] Typically, a virtual reality application generates a three-dimensional output in the form of separate view images for the left and right eyes. These can then be presented to a user by suitable means, typically such as the individual left and right eye displays of a VR headset. In other embodiments, the images may be presented, for example, on an autostereoscopic display (in which case multiple view images may be generated for the viewer pose), or in some embodiments, only a single two-dimensional image may be generated (e.g., using a conventional two-dimensional display).

[0047] Viewer pose input can be determined in different ways in different applications. In many embodiments, the user's physical movement can be tracked directly. For example, a camera scanning the user area can detect and track the user's head (or even eyes). In many embodiments, the user may wear a VR headset that can be tracked by external and / or internal means. For example, the headset may include accelerometers and gyroscopes that provide information regarding the headset and, accordingly, the movement and rotation of the head. In some examples, the VR headset may include identifiers (e.g., visual) that can transmit signals or allow external sensors to determine the movement of the VR headset.

[0048] In some systems, the viewer pose may be provided by manual means, for example, by a user manually controlling a joystick or similar manual input. For example, a user may manually move the virtual viewer in the scene by controlling a first analog joystick with one hand, and manually control the direction the virtual viewer is looking by manually moving a second analog joystick with the other hand.

[0049] In some applications, a combination of manual and automated approaches can be used to generate input viewer poses. For example, a headset can track head orientation, and the viewer's movement / position within the scene can be controlled by a user using a joystick.

[0050] The generation of images is based on a suitable representation of the virtual world / environment / scene. In some applications, a full 3D model may be provided for the scene, and views of the scene from a specific viewer pose may be determined by evaluating this model. In other systems, the scene may be represented by image data corresponding to views captured from different capture poses, specifically by multiple source images having associated depths, each image representing the scene from a different viewpoint. In such approaches, view images for poses different from the capture pose(s) may be generated by 3D image processing, for example, using view-shifting algorithms. In systems where the scene is described / referenced by view data stored for distinct viewpoints / locations / poses, these may also be referred to as anchor viewpoints / locations / poses. Typically, when a real-world environment is captured by capturing images from different points / locations / poses, these capture points / locations / poses are also anchor points / locations / poses.

[0051] Therefore, a typical VR application provides (at least) images corresponding to viewports of the scene for the current viewer pose, the images are dynamically updated to reflect changes in the viewer pose, and the images are generated based on data representing the virtual scene / environment / world.

[0052] In this field, the terms placement and pose are used as common terms for position and / or orientation. For example, a combination of the position and orientation of an object, camera, head, or view may be referred to as a pose or placement. Thus, a placement or pose representation may include six values / components / degrees of freedom, each value / component typically describing an individual characteristic of the position or orientation of the corresponding object. Of course, in many situations, for example, where one or more components are considered fixed or irrelevant, the placement or pose may be expressed with fewer components or considered to have fewer components (e.g., where all objects are considered to be at the same height and have a horizontal orientation, four components may provide a complete representation of the object's pose). Below, the term pose is used to refer to a position and / or orientation that can be expressed by one to six values ​​(corresponding to the maximum possible degrees of freedom).

[0053] Many VR applications are based on poses that have the maximum degrees of freedom, namely, three degrees of freedom each for position and orientation, resulting in a total of six degrees of freedom. Thus, a pose can be represented by a vector or set of six values ​​representing the six degrees of freedom, and thus the pose vector can provide a three-dimensional position and / or three-dimensional orientation indication. However, it will be known that in other embodiments, a pose can be represented by fewer values.

[0054] The pose may be at least one of orientation and position. The pose value may represent at least one of an orientation value and a position value.

[0055] A system or entity based on providing the maximum degrees of freedom for a viewer is typically referred to as having 6 degrees of freedom (6DoF). Many systems and entities provide only orientation or position, and are known to typically have 3 degrees of freedom (3DoF).

[0056] In some systems, a VR application may be provided locally to a viewer by a standalone device that receives scene data (independent of a specific viewer pose for the local viewer), for example, from a remote device / server, and then locally generates view images for specific current views of the local viewer. Thus, in many applications, particularly for broadcast services, a source may transmit scene data in the form of an image representation (including video) of the scene independent of the viewer pose. For example, an image representation including multiple captured view images and associated depth maps may be received. Subsequently, individual clients may locally synthesize view images corresponding to the current viewer pose.

[0057] A specific application attracting particular attention is one where limited motion is supported, allowing presented views to update to follow small movements and rotations corresponding to a practically static viewer who is only making small head movements and rotations. For example, a seated viewer may turn and slightly move their head, and the presented views / images adapt to follow these changes in pose. Such an approach can provide a highly immersive, for instance, video experience. For example, a viewer watching a sports event may feel as though they are in a specific location within the stadium.

[0058] Such limited-free applications have the advantage of providing an improved experience by not requiring an accurate representation of the scene from many different locations, thereby substantially reducing capture requirements. Similarly, the amount of data that needs to be provided to the renderer can be substantially reduced. In fact, in many scenarios, only an image and typically depth data for a single viewpoint need to be provided, and the local renderer can generate the desired views from this. To support head rotations, it is typically required that a large area of ​​the view from the viewpoint be represented by the provided data, and preferably, that the entire surface of the view sphere centered at the viewpoint be covered by the provided image and depth data.

[0059] Specifically, this approach can be very suitable for applications where data needs to be communicated from a source to a destination over a band-limited communication channel, such as in the case of broadcasting or client-server applications.

[0060] FIG. 1 illustrates such an example of a VR system in which a remote VR client device (101) communicates with a VR server (103) via a network (105), such as the Internet. The server (103) may potentially be arranged to support multiple client devices (101) simultaneously.

[0061] The VR server (103) can support a broadcast experience by, for example, transmitting image data and depth for multiple view points, and client devices are then arranged to process this information and locally synthesize view images corresponding to the current pose.

[0062] To provide efficient distribution, it is desirable to keep the data rate as low as possible for a given image quality, and thus may specifically include attempts to reduce the amount of redundant data generated.

[0063] FIG. 2 illustrates an example of a device for generating an image signal comprising representations of multiple images of a scene from different view poses (anchor poses). The device will also be referred to as an image signal transmitter (200). The image signal transmitter (200) may be included, for example, in the VR server (103) of FIG. 1.

[0064] FIG. 3 illustrates an example of a device for rendering view images based on a received image signal including a representation of multiple images of a scene. Specifically, the device may receive a data signal generated by the device of FIG. 2 and proceed to process it to render images for specific view poses. The device of FIG. 3 will also be referred to as an image signal receiver (300). The image signal receiver (300) may be included, for example, in the client device (101) of FIG. 1.

[0065] The image signal transmitter (200) includes an image source receiver (201) arranged to receive a plurality of source images of a scene. The source images may represent views of the scene from different capture poses. For example, the source images may include images from rows of equidistant capture poses.

[0066] In many embodiments, the source images may be 3D images including 2D images having associated depth information. Specifically, the 2D images may be view images of viewports of a scene from a corresponding capture pose, and the 2D images may be accompanied by a depth image or map including depth values ​​for each pixel of the 2D images. The 2D images may be texture maps.

[0067] The depth values ​​may be, for example, disparity values ​​or distance values ​​indicated by, for example, z-coordinates. In some embodiments, the source image may be a 3D image in the form of a texture map having an associated 3D mesh. In some embodiments, such texture maps and mesh representations may be converted into image plus depth representations by an image source receiver before further processing by an image signal transmitter (200).

[0068] Accordingly, the image source receiver (201) receives a plurality of source images characterizing and representing scenes from different poses. A set of such source images will enable view images to be generated for different poses using algorithms such as view shifting, which will be known to those skilled in the art. Accordingly, the image signal transmitter (200) is arranged to generate an image signal containing image data for the source images and to transmit this data to a remote device for local rendering. However, directly transmitting all source images would require an impractically high data rate and would contain a large amount of redundant information.

[0069] An image signal transmitter (200) is arranged to reduce the data rate by generating partial images from which redundant information has been removed. The partial images are subsequently combined and encoded to generate an image signal.

[0070] An image source receiver (201) is coupled to a pixel selector (203) arranged to generate a set of images from source images, at least one of the set of images being a partial image. The pixel selector (203) may generate a set of partial images by generating one or typically most (or even all) of partial versions of the source images by selecting a subset of pixels within the source image to be included in the image signal. A partial image will also be referred to as a pruned image, and generating its partial image by selecting a subset of pixels of an image will also be referred to as pruning the image.

[0071] Accordingly, the output of the pixel selector (203) may be a set of pruned or partial images corresponding to source images, but one or most of the images are partial versions of the corresponding source image. The pixel selector (203) attempts to select a subset of these pixels to be included in the partial image so that the overall redundancy of the set of generated partial images is reduced or, preferably, minimized. The generated partial images may also include one or more unpruned original source images.

[0072] A pixel selector (203) is coupled to an image generator (205) that receives a set of partial images. The image generator (205) is arranged to combine the partial images into combined images, each image may represent one or more of the partial images. Specifically, the combined image may include pixels derived from multiple partial images. For example, pixels from one partial image may be inserted into unused areas of another partial image. The image generator (205) may be considered to pack the partial images into more densely packed combined images.

[0073] The image generator (205) generates fewer but less sparse images. A set of combined images is fed to an encoder (207) that proceeds to perform encoding of the images. Since the image generator (205) has reduced the number of images to be transmitted, typically more efficient encoding is achieved. Additionally, packing partial images into combined images can typically be performed so that the resulting images are highly suitable for encoding.

[0074] A specific advantage of this approach is that the representation of a scene by partial images is achieved in a manner that allows conventional image and video encoding approaches to be performed by an encoder (207). For example, in many embodiments, encoding formats such as HEVC (High Efficiency Video Coding), also known as H.265 and MPEG-H Part 2, or VVC (Versatile Video Coding), developed by the Joint Video Exploration Team (JVET), may be used.

[0075] Accordingly, the image signal transmitter (200) of FIG. 2 can provide an efficient approach for generating an efficient image signal representing a scene from a range of view poses.

[0076] The image signal is transmitted to an image signal receiver (300) comprising a decoder (301) arranged to receive the image signal and decode it to generate a set of combined images supplied to an encoder (207). Accordingly, the decoder (301) may be arranged to perform decoding according to the standard used by the encoder (207), such as by performing HEVC or VVC decoding, for example.

[0077] It will be understood that the image signal transmitter (200) and the image signal receiver (300) further include necessary functions for communicating image signals, such as functions for encoding, modulating, transmitting, receiving, etc. of the image signal. It will be understood that such functions will depend on the preferences and requirements of individual embodiments, and that such technologies will be known to those skilled in the art and, therefore, for the sake of clarity and brevity, will not be further discussed in this specification.

[0078] The decoder (301) is coupled to an image searcher (303) arranged to search for partial images from the received combined images. Thus, the image searcher (303) can perform the inverse function of the image generator (205) to divide the pixels of the combined image into individual partial images. Thus, the combined image typically contains pixels corresponding to a plurality of different view points or poses, whereas the partial images are generated such that each image originates from only one source image and therefore contains pixels corresponding to only one view pose.

[0079] In many embodiments, the image signal also includes one or more complete source images, and the output of the image scanner (303) accordingly provides a set of images corresponding to the source images, thus representing a scene from a range of view poses, and one or more of the images are only partial images. However, the missing part from a given partial image typically corresponds to redundant information available from other images in the output set of images from the image scanner (303).

[0080] The images are supplied to a first renderer (305) arranged to generate new view images based on the received set-off partial images (and any full images). These images correspond to the original source images. It will be known that any suitable algorithm may be used to render a view image corresponding to a given viewer pose. For example, the first renderer (305) may be arranged to first generate a view image based on the received full source image. This may typically result in multiple holes caused by occlusion due to the changed view point. Such holes may then be filled using data from the partial images. Those skilled in the art will know many different algorithms and approaches for synthesizing views for specific purposes based on images from different view points, and will know that any suitable algorithm may be implemented by the first renderer (305).

[0081] In some embodiments, the image signal receiver (300) may include a second renderer (307) arranged to synthesize view images directly from the received combined images. In many embodiments, the image signal receiver (300) will include a first renderer (305) and an image scanner (303), or a second renderer (307). It will be known that the second renderer (307) may use any suitable approach for rendering view images for a given viewer pose.

[0082] The first and second renderers can use the same composition method and parameters, which can be advantageous because it can increase the predicted value.

[0083] As with the approach of FIGS. 2 and 3, a particularly critical issue of the partial image-based approach is how the partial images are generated and how to select which pixels to include in the partial images. FIG. 4 illustrates examples of elements of a possible implementation of a pixel selector (203) that can provide a particularly advantageous and efficient generation of a set of partial images from a set of source images in many embodiments and scenarios. FIG. 5 illustrates an example of a method of operation of an image signal transmitter (200) that particularly highlights the operation of the pixel selector (203) of FIG. 4. However, it will be understood that the approach of FIGS. 4 and 5 may be applicable to the image signal transmitter (200) and image signal receiver (300) of FIGS. 2 and 3 and other embodiments.

[0084] The approach of the pixel selector (203) is specifically based on the iterative processing of input source images. A set of candidate images is generated that includes images that have not yet been processed (and are initialized including all source images). Then, the process proceeds iteratively to select one or more of the candidate images and add them to a set of images that are pruned and included. Thus, candidate images are iteratively selected, pruned, and then moved to a secondary identifier. This approach can provide an efficient method for selecting how to prun images, specifically the order in which to do so to provide a selected set of pruned or partial images that can provide a more accurate and complete representation of the scene for use in locally generating view images.

[0085] As previously mentioned, the image source receiver (201) receives a set of images containing multiple images of a scene, at least some, typically all of which are for different view poses of the scene. The images may specifically be 3D images including 2D image / texture maps and associated depth maps.

[0086] The pixel selector (203) includes an input receiver (401) that performs the step (501) of receiving a set of source images from an image source receiver (201). Accordingly, the input receiver (401) receives a set of source images, which are typically texture maps and depth maps for a range of view poses. The input receiver (401) discloses a set of candidate images (the set will also be referred to simply as a candidate set for brevity).

[0087] The input receiver (401) is coupled to a reference selector (403) that performs the step (503) of initializing the set of included images by selecting one or more of the candidate images and including them in the set of included images. The set of included images includes images (typically partial images) of candidate images in which image data is included in the image signal generated by the image signal transmitter (200). The set of included images is typically initialized in step (503) by selecting one candidate image as the included image.

[0088] The selected image is typically included as a full image, and thus all pixels of the selected image(s) are included. The selected image can typically provide more scene information than any other included image (since these are typically partial images), and can be regarded not only as an initial reference image for image composition by a remote renderer, but also as a reference for selecting other included images and determining which pixels of the corresponding candidate images should be selected for the partial image.

[0089] It will be understood that different approaches and criteria may be used to select initial reference images for a set of included images. In some embodiments, the reference selector (403) may simply select a candidate image corresponding to the most central image. In other embodiments, the reference image may be selected based on an analysis of the associated depth map. For example, to select an image containing as much background as possible, a candidate image having the maximum number of pixels with a depth above a given threshold may be selected.

[0090] In some embodiments, the reference image(s) may be based on viewport similarity, for example, with respect to the viewports of different candidate images. For example, viewport similarity may be calculated by sampling a set of pixels from one view (i.e., one source image / capture pose), projecting them onto another view, and then determining the similarity between this composite image and the original image. The metric may be symmetrical by repeating the process in the reverse direction.

[0091] As an example of a similarity measure, the difference between a composite pixel value and an original pixel value can be determined and averaged across the image / viewport. As another example, the number of pixels with an error below a given threshold can be counted and the number of such pixels can be used as a similarity measure.

[0092] The process can be performed on all other candidate images for a given candidate image, and similarity measures can be combined into a single similarity measure for the candidate image. The process can be further repeated for all (or some) of the candidate images, and finally, the candidate image with the highest overall similarity measure can be selected.

[0093] Therefore, the reference image can be selected as an image that allows for the best average prediction / view synthesis of other candidate images.

[0094] The selector (403) is coupled to an iterator (405) arranged to iteratively expand the set of included images to include partial versions of candidate images. In each iteration, a subset of candidate images, typically one, is selected to be included in the set of included images. For each selected candidate image, a partial image is generated and added to the set of included images, and the selected candidate image is removed from the set of candidate images.

[0095] In some embodiments, the iterator (405) continues to repeat the process until all candidate images are selected. In other embodiments, the iteration process may potentially be stopped before all candidate images are selected.

[0096] The iterative process may cause the order in which candidate images are selected to vary according to the characteristics of the images. Since partial images depend on previously selected images (and partial images already included in the set of included images), a different order may result in different partial images. The iterative process attempts to select candidate images in an order that can reduce the data rate and, specifically, generate partial images suitable for efficient image coding by the encoder (207). In many embodiments, the iterative process may attempt to reduce the number of pixels included in the partial images.

[0097] Iteration begins at step (505), where the iterator (405) generates prediction images for a set of candidate images from images of a set of included images. The prediction image for a first candidate image of a set of candidate images from a first included image from a set of included images may be an image of a view pose of a first candidate image generated by view synthesis from a first included image. Since the first included image and the first candidate image correspond to source images for different view poses, view synthesis includes view pose shifting and typically view position shifting. View synthesis may be view shift image synthesis. Thus, the prediction image for a first candidate image from a first included image may be an image reflecting how well the viewport from the view pose of the candidate image can be predicted / estimated from the first included image.

[0098] The prediction of the first image from the second image may specifically be a view synthesis of images at the view pose of the first image based on the second image (and its view pose). Accordingly, the prediction operation for predicting the first image from the second image may be a view pose shift of the second image from the associated view pose to the view pose of the first image.

[0099] It will be understood that different methods and algorithms for view synthesis and prediction may be used in different embodiments. In many embodiments, a view synthesis / prediction algorithm may be used that takes as input a synthetic view pose on which a synthetic image is to be generated, and a plurality of input images, each associated with a different view pose. Subsequently, the view synthesis algorithm may generate a synthetic image for this view pose based on input images, which may typically include texture maps and depth quanta.

[0100] A number of such algorithms are known, and any suitable algorithm may be used without departing from the present invention. As an example of such an approach, intermediate composite / predicted images may first be generated for each input image. This can be achieved, for example, by first generating a mesh for the input image based on the depth map of the image. Subsequently, the mesh may be warped / shifted from the view pose of the input image to a composite view pose based on geometric calculations. Subsequently, the vertices of the resulting mesh may be projected onto the intermediate composite / predicted image, and a texture map may be overlaid on this image. Such a process may be implemented, for example, using vertex processing and fragment shaders known from standard graphics pipelines.

[0101] In this way, an intermediate composite / predicted image for a composite view pose (hereinafter referred to simply as an intermediate predicted image) can be generated for each of the input images.

[0102] Next, the intermediate prediction images may be combined together, for example, by weighted combination / summation or by selective combination. For example, in some embodiments, each pixel of the composite / prediction image for a composite view pose may be generated by selecting a pixel from the most forward intermediate prediction image, or the pixel may be generated by a weighted summation of corresponding pixel values ​​for all intermediate prediction images, wherein the weight for a given intermediate prediction image depends on the depth determined for that pixel. The combination operation is also known as the blending operation.

[0103] In step (505), the iterator (405) may, accordingly, proceed to generate a prediction image for a first candidate image based on the images in the set of included images, that is, for each of the already included images. Thus, the prediction may be based on a plurality of images in which some images are partial images (typically apart from the first iteration).

[0104] The view compositer may proceed, for example, to generate intermediate images by view-shifting each included image to the view pose of the first candidate image. Then, a prediction image for the first candidate image may be generated by blending these intermediate prediction images. Since some of the images are partial images containing image data for only a subset of pixels, the blending / combining for at least some pixels will include only pixels from a subset of intermediate prediction images (i.e., the intermediate prediction images may also be partial images).

[0105] Accordingly, for a first candidate image, a first predicted image is generated based on the included images, that is, based on the images already present in the set of included images. Thus, this predicted image can reflect how well the first candidate image can be predicted from image data already selected to become part of the image signal.

[0106] The process can be repeated for all candidate images, and thus a prediction image can be generated for each candidate image, and thus step (505) generates a prediction image for each candidate image based on already selected included images.

[0107] After that, the iterator (405) proceeds to step (507), where the prediction quality is determined for each of the candidate images, and the prediction quality measure for a given candidate image is a measure representing the difference between the candidate image and the predicted image for the candidate image.

[0108] Accordingly, the prediction quality measure is a measure that can indicate the improved prediction quality of a candidate image from a set of images that (already) contain increasing values. The prediction quality measure can be determined by directly comparing the prediction image with the candidate image. For example, the prediction quality measure can be determined as the mean or cumulative error between corresponding (same location) pixel values ​​of the candidate image and the prediction image. As another example, prediction quality can be determined as the number of pixels where the difference between the pixel values ​​of the candidate image and the prediction image exceeds a threshold. It will be understood that other prediction quality measures may be used in other embodiments.

[0109] Accordingly, at the end of step (507), a prediction quality measure reflecting the quality of the prediction of the candidate image from the set of included images is determined for each candidate image.

[0110] Step (507) is followed by Step (509), where the iterator (405) proceeds to select a set of selected images from a set of candidate images in response to the prediction quality. Specifically, the iterator (405) may select a predetermined number of candidate images as candidate images having the lowest prediction quality measure, that is, a predetermined number of candidate images that are predicted to be the worst by the already included images are selected.

[0111] Next, the iterator (405) proceeds to determine a pruned or partial image for the selected image(s), and the selected image(s) are then moved from a set of candidate images. In many embodiments, only a single candidate image is selected in each iteration, and for clarity and brevity, the following description will focus on this approach, but it will be understood that the reference to the single candidate image selected can be extended to a subset including multiple images selected.

[0112] Step (509) is followed by Step (511), where the iterator (405) generates a partial image corresponding to the selected image, and the partial image includes only a subset of the pixels of the selected image.

[0113] Specifically, the iterator (405) may proceed to select a subset of pixels of the selected candidate image. It will be known that different approaches and algorithms may be used to select which pixels to include and which pixels to exclude in the subset. The subset of pixels is selected in response to pixel prediction qualities, and specifically, a measure or indication of pixel prediction quality for each pixel may be determined. The pixel prediction quality measure may indicate how well the pixel can typically be predicted from other included images, and / or may indicate the improvement in prediction of other images that can be achieved by including the pixel in the subset. The iterator (405) may select pixels to be included in or excluded from the subset based on whether the pixel prediction quality measure meets the criteria.

[0114] In many embodiments, the selection of pixels may be based on the difference between the pixel values ​​of the predicted image and the selected image. For example, if the difference between the predicted value and the actual value in the candidate image exceeds a threshold, pixels may be included in the subset, otherwise they may not be included.

[0115] As further explained below, much more complex approaches may be used, including evaluating how the prediction of other images can be improved by including pixels within a partial image in some embodiments.

[0116] In some embodiments, a first pixel of a candidate image may be selected for a subset in response to a determination that the scale of the difference between the predicted value for the first pixel in the predicted image and the value of the pixel in the candidate image exceeds a threshold.

[0117] In some embodiments, a prediction improvement measure indicating the improvement of the prediction of (one or more) pixels of other candidate images by including the pixel in the prediction may be determined for each pixel. Then, a subset may be selected in response to the prediction improvement measure, for example, by selecting pixels where the prediction improvement measure is above a threshold.

[0118] Accordingly, a partial image containing image data for only a subset of pixels is generated for the candidate image. The partial image may be generated by discarding a complementary subset of pixels within the candidate image. Discarding pixels may be achieved by not including information of pixel values ​​in the partial image, and, for example, by setting corresponding pixel values ​​to a fixed value (e.g., 0) or a value unrelated to the pixel values ​​of the candidate image. An approach to discarding pixels is to not include them in the combined images generated by the image generator (205). Specifically, the generation of the partial image may be considered to correspond to the generation of a pruning mask that can be overlaid with the candidate image and the partial image containing only the pixels not masked by the mask.

[0119] Step (511) is followed by Step (513), where a partial image (or partial images if more than one candidate image is selected) is added to a set of images. Step (513) is followed by Step (515), where the selected image(s) are removed from the set of candidate images.

[0120] Therefore, the iteration can specifically evaluate a set of included images to find one or more images, and one or more images are then moved to a set of included images after being pruned to generate partial images.

[0121] Step (517) follows Step (515), where it is evaluated whether additional iterations need to be performed. If so, the process returns to Step (505) to initiate a new iteration. Otherwise, Step (519) follows Step (517), where an image signal is generated by an image generator (205) and an encoder (207) to include a set of included images.

[0122] As previously mentioned, in some embodiments, iterations may be performed until the set of candidate images is empty, that is, until all candidate images are selected.

[0123] In other embodiments, the approach may proceed until the prediction errors for all remaining candidate images are below a given threshold. In this case, all views can be predicted sufficiently accurately, and it may be considered that no further image data is needed. In such embodiments, the decision on whether to perform another iteration may be made after the step (505) in which prediction quality measures are determined. Specifically, iterations may be stopped when the lowest prediction quality measure exceeds a given threshold.

[0124] The selection of candidate image(s) and a subset of pixels included in a partial image is performed in response to consideration of relationships with other, specifically, already included images, and in response to consideration of the predictive impact of the selections.

[0125] This approach has been found to tend to produce a set of embedded images that are very suitable for encoding and transmission. This approach has been found to substantially reduce the data rate for a given desired image quality. At the same time, the iterative approach allows for a particularly efficient and low-complexity implementation with low computational requirements.

[0126] In some embodiments, the determination of the prediction quality measurement in step (507) may be made by selecting and counting the number of pixels where the prediction error is less than a given threshold as mentioned.

[0127] Specifically, the iterator (405) can determine multiple pixels of the first candidate image such that the pixel values ​​for the first candidate image and the corresponding pixels of the prediction image for the candidate image satisfy a similarity criterion. Thus, sufficiently well-predicted pixels can be determined, and a prediction quality measure for the first candidate image can be determined in response to the characteristics of the multiple pixels, for example, as a monotonically increasing function of the number of pixels within the multiple pixels. Thus, the prediction quality measure can increase as the number of sufficiently well-predicted pixels increases.

[0128] Therefore, this approach is based on selecting a subset of pixels that satisfy the similarity criteria. In some embodiments, the subset of pixels determined in step (513) may be determined based on the subset determined in step (507).

[0129] Specifically, a subset of pixels included in a partial image can be generated from pixels of a candidate image that are not included in multiple pixels satisfying similarity requirements. Accordingly, the same considerations, criteria, and approaches can be used to determine the predictive quality measures used to select candidate images and to generate a subset / prune mask for these images. This not only reduces complexity and computational resource usage since the same operation is used for both purposes in many embodiments, but also provides an improved selection because, in many embodiments, the selection is directly related to the amount of image data to be included in the image signal as a result of the selection.

[0130] For example, the image signal transmitter (200) proceeds to generate a prune mask for each of the candidate images, thereby dividing the candidate images into a first subset of pixels that satisfy similarity criteria and a second subset that does not satisfy similarity criteria. The number of pixels in the first subset is determined and can be used as a prediction quality measure. Specifically, the candidate image has a minimum prediction quality measure corresponding to the minimum number of pixels in the first subset and the maximum number of pixels in the second subset. Then, the partial image is generated by including only the pixels of the second subset and discarding the pixels of the first subset, for example, by setting its pixel values ​​to a fixed value (e.g., 0).

[0131] In some embodiments, the image signal transmitter (200) may select pixels for a partial image by considering the prediction improvement / error change (i.e., the change in error between the predicted pixel value and the accurate value) for the remaining candidate images resulting from including individual pixels in step (511). Specifically, for a given first pixel within the selected image, the image signal transmitter (200) may determine the prediction quality improvement for pixels of other candidate images by including the first pixel in the prediction process. Then, the first pixel is included in the generated partial image if this prediction quality improvement exceeds a threshold.

[0132] Improvement in forecast quality can be specifically determined by the following process.

[0133] First predictions are generated for each remaining candidate image from the set of selected images and included images, that is, predictions are generated for the candidate images by further considering the image selected in step (509). Thus, in contrast to the prediction in step (505), step (513) generates a new prediction for each candidate image based also on the image selected in step (509). For example, for the first remaining candidate image, the iterator (405) generates a prediction image based on all images in the set of included images from previous iterations and the selected image from this iteration.

[0134] For the pixels of this first remaining candidate image, the iterator (405) then proceeds to determine an error change representing the difference between the pixel value and the prediction value determined based on the selected image for the prediction value determined based on the set of included images plus only the set of included images.

[0135] Accordingly, for each pixel of the first remaining candidate image, a first prediction error may be determined for a prediction including the selected image, and a second prediction error may be determined for a prediction not including the selected image, i.e., a prediction previously made in step (505). The difference between these reflects the prediction improvement achieved by including the selected image in the prediction. In the case of the Yuv color representation, the error change (by including the selected image) may be determined for each pixel, for example, as follows:

[0136]

[0137] Here, the index cond refers to the predicted image, and the index ref refers to the original candidate image, and the index before and afterrefers to the prediction error for a prediction that does not consider the selected image and a prediction that considers the predicted image, respectively.

[0138] Accordingly, an error change is determined for each pixel of the first remaining candidate image. This can form a candidate error change image, and subsequently, the candidate error change image is view-shifted from the view pose of the first remaining candidate image to the view pose of the selected image. As a result, the selected error change image is determined by the view pose shift from the view pose of the first remaining candidate image to the view pose of the selected image, where pixel values ​​reflect the change in prediction error occurring in the prediction candidate image by including that pixel of the selected image in the prediction of the first remaining candidate image. Thus, a direct measure of the gain on the prediction of the first remaining candidate image is determined by including individual pixels in the partial image of the selected image.

[0139] Accordingly, the iterator (405) can proceed to select all pixels that indicate that the error change image will be reduced by more than a given threshold.

[0140] However, in many embodiments, the selection of pixels for a partial image is based not only on considering one remaining candidate image, but on considering many of them, typically all of them. In many embodiments, error change images shifted to the view pose of the selected image may be generated for all remaining candidate images.

[0141] In some embodiments, these may then be considered separately by including pixels in the partial image, for example, if any of the generated error change images indicate that an improvement exceeding a given threshold can be achieved. However, in many embodiments, the combined error change image may be generated by combining error change values ​​determined for different candidate images for the same pixel of the selected image. As a simple example, the error change values ​​for different candidate images may simply be added together or averaged after being shifted to the view pose of the selected image. Then, the iterator (405) may proceed to select pixels of the selected image in which the combined error change values ​​indicate an error reduction above a given threshold within the partial image, that is, pixels that provide the greatest overall prediction improvement to the remaining candidate images are selected.

[0142] In many embodiments, the combined error change image may include an error change image generated for the selected image itself. In practice, at this stage, the selected image is only predicted (not fully included), and thus the selected image may also be advantageously included when calculating in the combined error change image, thereby representing the prediction improvement achieved for the selected image itself. For example, this may be a case where a pixel is included in a partial image because the pixel is required for the selected image itself but not for any of the remaining candidate images.

[0143] As described above, the prediction / view synthesis performed (specifically in step (505)) can be performed by generating individual intermediate prediction images for each of the images in the set of images included in many embodiments, that is, one intermediate image can be generated for each pair of candidate images and included images. These intermediate prediction images are then combined / mixed into a single prediction image for the first candidate image.

[0144] In many embodiments, a synthesizer may be used that is arranged to perform all of these operations and simply receives as input a set of included images (and associated view poses) and a view pose in which a synthesized / predicted image will be generated. This can generate a synthesized / predicted image for a given candidate image from these inputs. The synthesizer may be implemented as an integral function and algorithm, such as a software subroutine, function, or object, which does not allow access to intermediate results or operations. Thus, the generation and blending of intermediate images may not be accessible to other functions, processes, or operations. Therefore, in embodiments using such a synthesizer, prediction in step (505), for example, may require that the entire synthesis be performed for all candidate images using all included images in each iteration.

[0145] However, in some embodiments, a synthesizer that allows access to intermediate prediction images and mixing may be used. This may be, for example, when a synthesis function is specifically developed for the purpose of implementing the method of FIG. 5.

[0146] In such an embodiment, the iterator (405) may be arranged to additionally store intermediate images generated in one iteration and to retrieve the stored intermediate images in a subsequent iteration.

[0147] Accordingly, in these embodiments, intermediate prediction images may be provided for blending by retrieving an image generated and stored in a previous iteration if one such image is available, and for generating a new intermediate prediction image by view shifting only if no such image is stored from a previous iteration. Thus, if an intermediate prediction image has been generated previously, it is reused, and a new view shift operation is not required for the pair of candidates and included images.

[0148] In the method of FIG. 5, new images are predicted for candidate images based on a set of included images that are progressively grown / expanded in each iteration. Thus, in each iteration, the generation of a new prediction for a given remaining candidate image is performed based on the same included images as in the previous iteration plus any images included by the previous iteration. Thus, for all images except those selected in the previous iteration, intermediate prediction images have already been generated and stored. Therefore, new intermediate prediction images are generated only for the images included in the previous iteration, and in many embodiments, only one new intermediate prediction image needs to be generated for each remaining candidate image (e.g., replacing an old intermediate prediction image because it is no longer needed).

[0149] Therefore, this method is highly suitable for incremental prediction approaches and can be used to provide a highly efficient implementation with a very large reduction in the processing required.

[0150] It will be known that different approaches for blending pixels of intermediate prediction images may be used in different embodiments, and specifically, different functions and parameters may be used to generate weights.

[0151] In many embodiments, the blend may be a weighted blend in which the weights for a given pixel depend on the depth associated with the pixel. For example, the blend may be a selection blend in which the depths for all pixels of the intermediate prediction images for a given pixel location are compared, and the pixel of the intermediate prediction image having the farthest forward depth value is selected. In many embodiments, more progressive weighting of different pixels based on weights may be used. For example, an exponential relationship between depth and weight may often be used.

[0152] In other embodiments, other parameters may be considered alternatively or additionally. For example, the amount of stretching (geometric distortion of basic primitives) required to perform a view shift may be considered, and weights may reduce the amount of stretching and geometric distortion of the required primitives.

[0153] In many embodiments, only a single image is selected to be included in the set of images included in each iteration. However, in some embodiments, the subset of selected images may advantageously include multiple images in each iteration, and thus multiple images may be moved to the set of images included in each iteration.

[0154] This may slightly increase the data rate in some scenarios, but in some embodiments, it can substantially reduce computational requirements and, specifically, the number of predictions required to execute the entire method. It can provide substantially reduced computational resource usage, particularly in embodiments where incremental prediction cannot be used, because, for example, there is no access to intermediate prediction images or mixing operations.

[0155] In many embodiments, the selection of pixels for a partial image may include processing of a portion of an initial subset determined, for example, based on pixel prediction characteristics. This processing may specifically consider the spatial characteristics or relationships of the pixels.

[0156] As a specific example, the iterator (405) may generate a mask corresponding to a subset of pixels, for instance, a binary image may be generated indicating whether each value should be included in the partial image. As another example, an image reflecting pixel prediction quality values ​​(e.g., a combined error change image) may be generated. Since this image reflects the possibility or benefit of including values ​​in the partial image, it may be a soft mask and thus represents the pixels included in the partial image along with a threshold value.

[0157] Rather than using the mask directly as originally generated, the iterator (405) may be arranged to process the mask first. Specifically, a spatial filter, typically a spatial low-pass filter, may be applied to the mask. Thus, changes to the mask may cause modifications to a subset of pixels selected for the partial image.

[0158] Such filtering can be very useful for avoiding very thin structures in partial images. An example of a spatial filter that can be applied is box blur with a small kernel.

[0159] In some embodiments, the images being processed may be frames of a video sequence. In such cases, the order of the set of included images determined for one frame may be applied to multiple frames. For example, for an intra-encoded frame, the described approach may be used to generate a sequential set of included images. Subsequently, the same arrangement / order / sequence of images may be used for all inter-coded frames up to the next intra-coded frame. Thus, in some embodiments, the method may be executed only on intra-frames, and the arrangement determined in this operation may be reused for all subsequent intra-frames.

[0160] Pruning masks can be reused for some frames, for example, by generating pruning masks at lower frame rates, or by reusing pruning masks in cases where there is an indication of low image motion. Alternatively, pruning masks can be generated for all inter-coded frames using an array determined for intra-coded frames.

[0161] While it may be particularly advantageous to update the array when changing scenes, such events will also typically trigger additional intra-frames, which may subsequently lead to the determination of a new array.

[0162] This approach can provide improved consistency across frames. While the overall data rate or quality degradation may be slightly higher than when the method is applied to each frame, this is typically a desirable trade-off for achieving more consistent visual results, as temporal variations tend to be very visible to the viewer.

[0163] It will be understood that the foregoing description refers to embodiments of the invention with reference to different functional circuits, units, and processors for the sake of clarity. However, it will be apparent that any suitable distribution of functions among different functional circuits, units, or processors may be used without departing from the invention. For example, functions exemplified to be performed by separate processors or controllers may be performed by the same processor or controller. Accordingly, references to specific functional units or circuits should be regarded as references to suitable means for providing the described functions, rather than strictly indicating a logical or physical structure or organization.

[0164] The present invention may be implemented in any suitable form, including hardware, software, firmware, or any combination thereof. The present invention may be implemented at least partially as computer software running on one or more data processors and / or digital signal processors. The elements and components of embodiments of the present invention may be implemented physically, functionally, and logically in any suitable manner. In practice, functions may be implemented in a single unit, in multiple units, or as part of other functional units. Therefore, the present invention may be implemented in a single unit or may be physically and functionally distributed among different units, circuits, and processors.

[0165] Although the present invention has been described in relation to some embodiments, the invention is not intended to be limited to the specific forms described herein. Rather, the scope of the invention is limited only by the appended claims. Furthermore, while features may appear to be described in relation to specific embodiments, those skilled in the art will recognize that various features of the described embodiments may be combined according to the invention. In the claims, the term 'comprising' does not exclude the presence of other elements or steps.

[0166] Additionally, although enumerated individually, multiple means, elements, circuits, or method steps may be implemented, for example, by a single circuit, unit, or processor. Also, while individual features may be included in different claims, they may be combined advantageously, and inclusion in different claims does not imply that a combination of features is not feasible or / or advantageous. Furthermore, the inclusion of a specific feature in claims of one category does not imply a limitation to that category, but rather indicates that the feature may be equally applicable to other claim categories as appropriate. Also, the order of features in claims does not imply any specific order in which the features must be applied, and in particular, the order of individual steps in method claims does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. Also, singular references do not exclude the plural. Thus, references to the indefinite article (a, an), "first," "second," etc., do not exclude the plural. Drawing symbols in the claims are provided merely as examples for clarification and should not be interpreted as limiting the scope of the claims in any way.

[0167] The present invention Generally It can be summarized as a method for generating an image signal, and the method is,

[0168] A step (501) of receiving a set of candidate images including multiple images of a scene, wherein at least some of the images in the set of candidate images are for different view poses of the scene;

[0169] A step (503) of selecting at least one image from a set of candidate images and initializing a set of included images to include at least one image;

[0170] A step (505) of generating predicted images for a set of candidate images from images of a set of included images;

[0171] A step (507) of determining a prediction quality scale for each image of a set of candidate images, wherein the prediction quality scale for a first image of a set of candidate images represents the difference between the first image and a first prediction image for the first image;

[0172] Step (509) of selecting a set of selected images from a set of candidate images in response to prediction quality.

[0173] For each image in the set of selected images, a step (511) of determining a subset of pixels in response to pixel prediction qualities for the pixels of the image;

[0174] A step (513) of generating a set of partial images corresponding to a set of selected images, wherein the partial image for a given image in the set of selected images comprises only the pixels of a subset of pixels for the given image;

[0175] Step (515) of including a set of partial images in a set of included images; and

[0176] Step (515) of removing a set of selected images from a set of candidate images

[0177] A step of repeatedly performing;

[0178] It includes the step (519) of generating an image signal containing image data of images within a set of included images.

[0179] This method may be further limited as defined in paragraphs 2 through 13 of the attached claims.

[0180] The present invention GenerallyIt may be further summarized as a computer program product, and the computer program product includes means of computer program code configured to perform all steps of any of the summarized methods listed above that generate an image signal when said program is executed on a computer.

[0181] The present invention Generally It can also be further summarized as a device for generating an image signal, and the device,

[0182] A receiver (401) for receiving a set of candidate images (501) including multiple images of a scene, wherein at least some of the images in the set of candidate images are for different view poses of the scene;

[0183] A reference selector (403) for selecting at least one image from a set of candidate images and initializing a set of included images (503) to include at least one image;

[0184] As a repeater (405),

[0185] A step (505) of generating predicted images for a set of candidate images from images of a set of included images;

[0186] A step (507) of determining a prediction quality scale for each image of a set of candidate images, wherein the prediction quality scale for a first image of a set of candidate images represents the difference between the first image and a first prediction image for the first image;

[0187] Step (509) of selecting a set of selected images from a set of candidate images in response to prediction quality.

[0188] For each image in the set of selected images, a step (511) of determining a subset of pixels in response to pixel prediction qualities for the pixels of the image;

[0189] A step (513) of generating a set of partial images corresponding to a set of selected images, wherein the partial image for a given image in the set of selected images comprises only the pixels of a subset of pixels for the given image;

[0190] Step (515) of including a set of partial images in a set of included images; and

[0191] Step (515) of removing a set of selected images from a set of candidate images

[0192] A repeater (405) for repeatedly performing the following;

[0193] It includes an output circuit (205, 207) for generating an image signal (519) that includes image data of images within a set of included images.

[0194] The present invention Specifically It may be summarized as limited to paragraphs 1 through 15 of the attached claims.

Claims

Claim 1 A method for generating an image signal, comprising: receiving a set of candidate images including a plurality of images of a scene (501), wherein at least some of the images in the set of candidate images are for different view poses of the scene; selecting at least one image of the set of candidate images and initializing a set of included images to include the at least one image (503); generating prediction images for the images of the set of candidate images from the images of the set of included images (505); determining a prediction quality measure for each image of the set of candidate images (507), wherein the prediction quality measure for the first image of the set of candidate images represents the difference between the first image and the first prediction image for the first image (507); and selecting a set of selected images from the set of candidate images in response to the prediction quality measures (509), wherein the selected images are the images already included in the set of selected images A step of selecting a set of selected images having a minimum prediction quality measure (509); a step of determining a subset of pixels for each image of the set of selected images in response to pixel prediction quality measures for the pixels of the image (511); a step of generating a set of partial images corresponding to the set of selected images (513), wherein the partial image for a given image of the set of selected images comprises only the pixels of the subset of pixels for the given image (513); and a step of including the set of partial images in the set of included images (515).A method comprising: repeatedly performing the step (515) of removing the set of selected images from the set of candidate images; and generating the image signal (519) including image data of the images within the set of included images. Claim 2 A method according to claim 1, wherein the step (507) of determining the prediction quality measure for the first image comprises: determining a plurality of pixels of the first image such that pixel values ​​for corresponding pixels of the first image and the first prediction image satisfy a similarity criterion; and determining the prediction quality measure for the first image in response to the characteristics of the plurality of pixels. Claim 3 A method according to paragraph 2, wherein the characteristic is the number of pixels, and the prediction quality measure is a monotonically increasing function of the number of pixels within the plurality of pixels. Claim 4 In paragraph 3, the step of determining a subset of the pixels (511) comprises determining a subset of the pixels for the first image from the pixels of the first image that are not included in the plurality of pixels for the image. Claim 5 In any one of claims 1 to 4, the step (511) of determining a subset of pixels for a first selected image of a set of selected images comprises: a step of determining an error change representing a change in the difference between a pixel of the at least one candidate image and a prediction pixel for a prediction of the pixel based on the set of included images and a prediction of the pixel based only on the set of included images, for at least one candidate image of the set of candidate images excluding the set of selected images, for pixels of the at least one candidate image; a step of generating an error change image for the first selected image, wherein the error change image comprises pixel values ​​determined from the error changes of the at least one candidate image by a view pose shift from the view pose of the at least one candidate image to the view pose of the first selected image; and a step of selecting a subset of pixels in response to the error change image. Claim 6 A method according to claim 5, wherein the step of determining a subset of pixels for the first selected image comprises: determining error changes for a plurality of candidate images of the set of candidate images; and generating an error change image by combining error change values ​​determined from different candidate images for the same pixel of the first selected image. Claim 7 A method according to any one of claims 1 to 4, wherein the step (505) of generating prediction images for a set of candidate images comprises: for a first candidate image of the set of candidate images, providing intermediate prediction images of the first candidate image for each of the images of the set of included images; and generating the first prediction image by combining the intermediate prediction images. Claim 8 In claim 7, the iteration further comprises the step of storing intermediate prediction images; the step of providing intermediate prediction images comprises the step of retrieving stored intermediate prediction images for images of the set of included images in which the intermediate prediction images were generated and stored in a previous iteration; and the step of predicting intermediate prediction images for any image of the set of reference images in which the intermediate prediction images were not stored in a previous iteration. Claim 9 In claim 7, the combination is a weighted combination, and the weight for the pixel depends on the depth of the pixel. Claim 10 A method according to any one of claims 1 to 4, wherein the iterations are stopped when the lowest prediction quality measure exceeds a threshold. Claim 11 A method according to any one of claims 1 to 4, wherein the subset of the selected images comprises a plurality of selected images in at least one iteration. Claim 12 A method according to any one of claims 1 to 4, wherein the images are frames of a video sequence, and the method comprises the step of using an array of sets of the included images for a plurality of frames. Claim 13 A method according to any one of claims 1 to 4, further comprising the step of generating a mask corresponding to a subset of pixels and modifying the subset of pixels in response to the application of a spatial filter to the mask. Claim 14 A device for generating an image signal, comprising: a receiver (401) for receiving (501) a set of candidate images including a plurality of images of a scene, wherein at least some of the images in the set of candidate images are for different view poses of the scene; a reference selector (403) for selecting at least one image of the set of candidate images and initializing (503) a set of included images to include the at least one image; an iterator (405) for generating prediction images for the images of the set of candidate images from the images of the set of included images (505); a step (507) for determining a prediction quality measure for each image of the set of candidate images, wherein the prediction quality measure for the first image of the set of candidate images represents the difference between the first image and the first prediction image for the first image; a step (507) for selecting a set of selected images from the set of candidate images in response to the prediction quality measures (509), wherein the selected images are the selected images A step of selecting a set of selected images having a minimum prediction quality measure for the images already included in the set (509); a step of determining a subset of pixels for each image in the set of selected images in response to pixel prediction quality for the pixels of the image (511); a step of generating a set of partial images corresponding to the set of selected images (513), wherein the partial image for a given image in the set of selected images includes only the pixels of the subset of pixels for the given image (513); a step of including the set of partial images in the set of included images (515);A device comprising: an iterator (405) for repeatedly performing the step (515) of removing the set of selected images from the set of candidate images; and an output circuit (205, 207) for generating the image signal (519) including image data of the images within the set of included images. Claim 15 A recording medium having a computer program recorded thereon, comprising a computer program code means configured to perform all of the steps of any one of claims 1 to 4 when the program is executed on a computer.

Images