Method and apparatus for encoding and decoding one or more views of a scene

Abstract

A method is provided for encoding and decoding image or video data including two or more views 10 of a scene. The encoding method includes obtaining (11) for each of the two or more views, a respective block segmentation mask 12 for the view and block image data 13 for the view. The method further includes generating (14) at least one packed frame 40 including the two or more block segmentation masks and the block image data for the two or more views, and encoding (15) the at least one packed frame into at least one bitstream 16. Each view is divided into blocks 30 of pixels, and the block segmentation mask indicates which blocks of pixels belong to a region of interest 31 within the view. The block image data includes the blocks of pixels that belong to the region of interest. Corresponding encoders, decoders, and bitstreams are also provided.

Description

[Technical Field]

[0001] The present invention relates to the encoding of image or video data for one or more views of a scene, and in particular to a method and apparatus for encoding and decoding video sequences captured from multiple viewpoints for virtual reality (VR) or immersive video applications. [Background technology]

[0002] Virtual reality can be a highly immersive way to view images or videos of a scene. When using virtual reality to view captured images or videos of a scene, multiple cameras are typically required to capture many views of the scene from various angles to allow the viewer to move around within the virtual reality scene. The more views captured from different angles, the more freedom the viewer has to move within the virtual reality scene, allowing for a more accurate rendered view of the scene. However, increasing the number of views captured increases the amount of data that must be processed and transmitted. Due to limited bandwidth, this can degrade the image or video quality of the virtual reality scene experienced by the viewer, as the data must be more highly compressed.

[0003] Multiple views of a scene are often encoded with metadata that indicates to a decoder how to reconstruct the original view. Efficient encoding often requires computationally intensive decision steps that delay the transmission of data to the viewer, resulting in latency. There can be a trade-off between efficiency (in terms of bit rate or pixel rate for a given bandwidth) and latency. For live streaming video, latency is of particular concern because viewers want to experience the virtual reality scene without delay, especially in interactive streaming scenarios such as video conferencing. Summary of the Invention [Problem to be solved by the invention]

[0004] In terms of computational effort and data rate (bandwidth), it is desirable to efficiently encode and decode one or more views of a scene. [Means for solving the problem]

[0005] The invention is defined by the claims. According to one aspect of the invention there is provided a method for encoding image or video data as claimed in claim 1.

[0006] For each view, the block segmentation mask indicates the location of blocks of pixels that belong to the region of interest. In any given view, there may be more than one region of interest.

[0007] Embodiments of the present method can facilitate simple, low-latency encoding of multi-view video. The block segmentation mask can, in effect, provide implicit metadata that allows a decoder to quickly and easily reconstruct one or more views from at least one packed frame. Meanwhile, pixel rates can be reduced because only a portion of each view (i.e., the region of interest) is coded / transmitted.

[0008] In some embodiments, the at least one packed frame may be a single packed frame, and the at least one bitstream may be a single bitstream.

[0009] For each block of pixels in a view, there may be a corresponding pixel in the block segmentation mask that indicates whether the block of pixels belongs to the region of interest. Thus, there may be as many pixels in the block segmentation mask as there are blocks of pixels in each view. In some embodiments, there may be more than one pixel in the block segmentation mask that corresponds to a block of pixels in a view. For example, a block of pixels in a view may have a corresponding block of pixels in the block segmentation mask that indicates whether the block of pixels in the view belongs to the region of interest. The blocks of pixels in the block segmentation mask may be smaller than the blocks of pixels in the view.

[0010] In each block segmentation mask, each pixel may contain a pixel value that indicates whether the corresponding block of pixels is part of the region of interest. The pixel value may be a luminance value or another pixel value such as a chrominance value, a depth value or a transparency value.

[0011] Pixel values ​​used to indicate blocks belonging to the region of interest may be separated from pixel values ​​used to indicate blocks not in the region of interest by an unused level. This unused level may create robustness against slight deviations in pixel values ​​that may be introduced by applying conventional lossy video compression techniques to packed frames. If the resulting ranges of pixel values ​​remain distinct and separable, it may be possible to reconstruct the block segmentation map without error at the decoder, despite such deviations.

[0012] In some embodiments, there may be two or more regions of interest. Pixel values ​​in the block segmentation mask may serve as indices for the regions of interest. For example, a first region of interest may be labeled with a first non-zero pixel value in the block segmentation mask, and a second region of interest may be labeled with a second non-zero pixel value in the block segmentation mask. Blocks that do not belong to a region of interest may be labeled with a pixel value of zero.

[0013] The region of interest may include or consist of foreground objects, and the remainder of the view may include or consist of background.

[0014] The block image data may consist of only image data for blocks of pixels that belong to the region of interest.

[0015] The image or video data includes multiple views.

[0016] For each view, blocks of pixels belonging to the region of interest may be packed into the packed frame in a sequence based on the scan order of the respective block segmentation mask.

[0017] Thus, a packed frame may comprise a first block segmentation mask associated with a first view followed by a second block segmentation mask associated with a second view. The block image data may comprise one or more blocks of pixels of the first view followed by one or more blocks of pixels of the second view. Third, fourth and further views may be added in the same manner.

[0018] The scan order of the block segmentation mask may be a raster scan order (ie, scan across the rows before scanning along the columns).

[0019] Obtaining the block segmentation mask may include segmenting each view to generate the block segmentation mask. For example, if the views are captured against a chromakey background such as a green screen, the segmentation may include classifying pixels as foreground or background by color separation (color keying). Any block of pixels that contains a predetermined minimum number of foreground pixels may be marked as foreground in the block segmentation map.

[0020] The at least one packed frame may comprise a packed frame having a first continuous portion comprising block segmentation masks of two or more views and a second continuous portion comprising block image data of two or more views.

[0021] Two or more block segmentation masks may be packed adjacently within the packed frame, such that each block segmentation mask is adjacent to at least one other block segmentation mask. The first portion may be the top-left most portion of the packed frame.

[0022] The at least one bitstream may consist of one bitstream. A first portion may be encoded to appear in the one bitstream before a second portion. This may facilitate fast and efficient decoding, and the decoder may receive and optionally decode a block segmentation mask before receiving and optionally decoding the block image data. Thus, when the block image data is received / decoded, the decoder will already have the information necessary to position the blocks of pixels within the reconstructed view.

[0023] In other embodiments, the at least one packed frame may include two or more packed frames. A first packed frame may include two or more block segmentation masks. A second packed frame may include block image data. This allows the block segmentation masks to be coded separately from the block image data, for example, using different video compression parameters. A first bitstream may be coded to include the first packed frame, and a second bitstream may be coded to include the second packed frame.

[0024] The blocks of pixels in the block image data of a view may all be the same size within that view, or optionally, the blocks may all be the same size in the block image data of all views. In the former case, the block size is constant within a view (or within each view individually). This can facilitate efficient packing and simple encoding and decoding. In the latter case, the block size is constant across all views. This can also facilitate efficient packing and simple encoding and decoding.

[0025] The block image data for each different view may be packed into at least one packed frame in a block-interleaved arrangement, with the first block of pixels from a first view consecutively followed by the first block of pixels from a second view, followed by the first block of pixels from a third view, and so on, until the first block of pixels from each view has been packed into the packed frame. This pattern of taking a single block of pixels from each view at a time and interleaving the blocks adjacent to each other in the packed frame may be repeated for subsequent blocks from each view, as long as there are enough blocks in each view to maintain this pattern.

[0026] This approach can yield good video compression efficiency, especially when the content and location of the regions of interest are similar across different views. In this case, the content of adjacent blocks of pixels interleaved in a packed frame is likely to be similar. Standard video compression algorithms can exploit this spatial consistency to reduce the bit rate required to encode the blocks and / or to increase compression quality for a given bit rate.

[0027] The block image data for each different view may be packed into at least one packed frame in a row-interleaved configuration, with blocks of pixels from the first row of a first view consecutively followed by blocks of pixels from the first row of a second view. This may be followed by blocks of pixels from the first row of a third view, and so on, until the first row of blocks from each view has been packed. In this manner, the pattern of taking an entire row of blocks from each view at a time and interleaving these rows adjacent to each other in the packed frame may continue as long as there are enough rows in each view to maintain the pattern. Again, this approach may result in good video compression efficiency for scenes containing typical visual content.

[0028] Encoding the at least one packed frame into the at least one bitstream optionally includes using a video compression algorithm, optionally using a standard video compression algorithm such as H.264 or HEVC. This can facilitate at least partial backward compatibility with existing encoding and decoding hardware and software and can avoid the need for proliferation of different video compression standards specifically to support multi-view video.

[0029] The method may include selecting a quality factor of the video compression algorithm such that at least the block segmentation mask is reconstructable without error from the at least one bitstream, and / or selecting a number of quantization levels used in the video compression algorithm such that at least the block segmentation mask is reconstructable without error from the at least one bitstream. Optionally, the method may include jointly selecting the number of quantization levels and the quality factor used in the video compression algorithm to ensure that the block segmentation mask is reconstructable without error from the at least one bitstream.

[0030] The at least one packed frame may comprise two packed frames. The first packed frame may comprise two or more block segmentation masks. The second packed frame may include block image data. The first packed frame may be encoded using lossless video compression. The second packed frame may be encoded using lossy video compression.

[0031] The method may comprise quantizing the block segmentation mask to a first number of quantization levels and quantizing the block image data to a second number of quantization levels, the first number being different from the second number. By independently setting the quantization parameters, the method can ensure that the block segmentation mask can be reconstructed without error after video compression while avoiding unnecessary bandwidth waste on the block image data. For example, the block image data may be quantized in a manner that introduces some error into the reconstructed block image data at the decoder. In this way, the block segmentation mask can be effectively compressed losslessly, while the block image data is compressed lossily.

[0032] At least one packed frame may comprise a third portion, the third portion comprising depth data for two or more views, and / or at least one packed frame may comprise a fourth portion, the fourth portion comprising transparency data for two or more views. The depth data may be included in at least one packed frame at a lower spatial resolution than the block image data. For example, the depth data may be provided in smaller blocks and / or a reduced number of blocks.

[0033] A decoding method as claimed in claim 10 is also provided.

[0034] The block image data consists of only blocks of pixels that belong to the region of interest. All of the two or more views may be reconstructed. The method may further include compositing the region of interest onto a background scene to synthesize a complete view.

[0035] Any of the methods summarized or claimed above may be used in live streaming multi-view video, which would be well suited to the needs of live video streaming, as its simplicity facilitates low latency at both the encoder and decoder ends.

[0036] There is also provided a computer program as claimed in claim 12. The computer program may be stored on a computer readable medium, optionally a non-transitory computer readable medium.

[0037] An encoder according to claim 13 and a decoder according to claim 14 are also provided.

[0038] There is also provided a bitstream as claimed in claim 15. The block segmentation mask can be packed into the packed frame before the block image data. The block segmentation mask can be decoded by a decoder before the block image data. [Brief explanation of the drawings]

[0039] For a better understanding of the present invention and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1] FIG. 2 is a block diagram of an encoder according to an embodiment of the present invention; [Figure 2] 2 is a flowchart illustrating an encoding method performed by the encoder of FIG. 1; [Figure 3A] 1 is a diagram of a scene divided into blocks of pixels, according to one embodiment. [Figure 3B] FIG. 3B shows a block segmentation mask (zoomed in) for the view of FIG. 3A. [Figure 3C] FIG. 3B shows block image data for the view of FIG. 3A. [Figure 4] FIG. 1 illustrates a packed frame according to one embodiment. [Figure 5A] FIG. 1 illustrates a packed frame according to one embodiment. [Figure 5B] FIG. 1 illustrates a packed frame according to one embodiment. [Figure 6] FIG. 1 illustrates a packing method according to one embodiment. [Figure 7] FIG. 1 illustrates a packing method according to one embodiment. [Figure 8] FIG. 2 illustrates an encoding method according to one embodiment. [Figure 9] FIG. 2 is a block diagram of a decoder according to one embodiment. [Figure 10] 10 illustrates a decoding method performed by the decoder of FIG. 9. [Figure 11] FIG. 1 illustrates a reconstruction method according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0040] The present invention will now be described with reference to the drawings.

[0041] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0042] 1 is a block diagram of an encoder 100 according to one embodiment. The encoder 100 includes an input 110 configured to receive two or more block segmentation masks 12 and block image data 13. The input 100 is connected to a packing unit 120 configured to generate packed frames 40. The packing unit 120 is connected to a video encoder 130 configured to encode the packed frames 40 into at least one bitstream 16.

[0043] A method performed by the encoder 100 will now be described with reference to Figures 2, 3, 4, and 6. Figure 2 is a flowchart illustrating the method. In an embodiment described in detail below, three views 10 of a scene are captured. Figure 3A shows one of the views 10 of the scene. The view 10 is uniformly divided into blocks 30 of pixels. Each of the other views (not shown) is also uniformly divided into blocks of pixels. The blocks 30 of pixels in a view 10 are all the same size in this example. Each view 10 includes a foreground region and a background region. As is common in the art, a green screen can be used as the background. An example of this is a view 10 that includes an object placed in front of a green screen. Using the green screen as the background allows each view 10 to be segmented by chroma keying. Pixels that match the color of the background screen (green in this case) are identified as background pixels.

[0044] FIG. 3B shows an expanded view of the block segmentation mask 12 of FIG. 10 in FIG. 3A. Chroma keying is used to distinguish between foreground and background regions, referred to herein as regions of interest 31. The block segmentation mask 12 is a map of the view 10 and indicates which blocks of pixels 30 in the view 10 belong to the regions of interest 31. Each block of pixels 30 in the view 10 is represented by a pixel in the block segmentation mask 12, making the block segmentation mask 12 a low-resolution segmentation mask for the view 10.

[0045] The block segmentation mask 12 indicates whether a block 30 of pixels in the view 10 belongs to the region of interest 31 by setting the pixel value of each pixel in the block segmentation mask 12 to a first or second value. In the case of segmentation by chroma keying, a block may be determined to belong to the region of interest if it contains at least one foreground pixel (i.e., at least one pixel of a color other than green).

[0046] In Figure 3(b), the pixel values ​​of the block segmentation mask are luminance values. Light areas 32 of the block segmentation mask 12 indicate blocks 30 of pixels in the view 10 that belong to the region of interest 31. Dark areas 33 of the block segmentation mask 12 indicate blocks 30 of pixels in the view 10 that do not belong to the region of interest 31. Figure 3C shows the blocked image data 13 for the view 10 of Figure 3A, as indicated by the block segmentation mask 12 of Figure 3B. The blocked image data 13 includes only blocks 30 of pixels that belong to the region of interest 31.

[0047] Referring now to the flowchart of Fig. 2, in step 11, an input unit 110 obtains, for each of the views 10, respective block segmentation masks 12 and block image data 13. In this example, the block segmentation masks 12 and block image data 13 are created using chroma keying. The block segmentation masks 12 and block image data 13 are passed to a packing unit 120.

[0048] In step 14, packing unit 120 generates a packed frame 40 including block segmentation masks 12 and block image data 13. Figure 4 shows packed frame 40. Packing unit 120 packs block segmentation masks 12 consecutively in a first portion 41 of packed frame 40. First portion 41 of packed frame 40 is the first portion that is coded and later decoded.

[0049] Packing unit 120 packs block image data 13 into second portion 42 of packed frame 40. Block image data 13 is also packed contiguously so as to leave no space between blocks of pixels 30 or between blocks of pixels 30 and block segmentation mask 12.

[0050] Another possible configuration of the packed frame is shown in FIG. 5 and described below.

[0051] Referring now to Figure 6, the packing method used to generate the packed frame 40 shown in Figure 4 will be described. Block segmentation masks 60, 61, and 62 indicate regions of interest in first, second, and third views 600, 610, and 620, respectively.

[0052] First, the block segmentation masks 60, 61, 62 are packed into a row in the first portion of the packed frame 40. The first block segmentation mask 60 is packed into the leftmost position of the row. The second block segmentation mask 61 is packed next to the first block segmentation mask 60, and the third block segmentation mask 62 is packed next to the second block segmentation mask 61. The order of the block segmentation masks 60, 61, 62 within the row from left to right defines the sequence.

[0053] Next, the block image data from each of the three views 600, 610, and 620 is packed into a second portion of the packed frame 40 using block interleaving. The packed frame 40 is packed with blocks of pixels in raster scan order. First, packing unit 120 packs the first pixel block from the first view 600 into the packed frame 40. Then, packing unit 120 packs the first pixel block from the second view 610 into the packed frame 40. Then, packing unit 120 packs the first pixel block from the third view 620 into the packed frame 40. This cycle then repeats, packing the second pixel block from the first view 600, and so on, until there are not enough pixels from each view to continue the sequence. In this way, the blocks of pixels in the packed frame 40 are interleaved such that no block of pixels immediately precedes or follows another block of pixels from the same view unless there are no longer enough blocks of pixels to maintain the interleaving. The sequence of blocks of pixels in the packed frame 40 is the same as the sequence defined by the order of the block segmentation masks 60, 61, 62 in the packed frame.

[0054] Similar to how block image data is packed into frame 40 in raster scan order, blocks of pixels from views 600, 610, 620 are selected to pack in raster scan order, as shown by Figure 6. Blocks of pixels are selected from each view 600, 610, 620 for packing in rows from top to bottom of view 600, 610, 620. Figure 6 shows that the top left-most block of pixels from view 600 is packed first from the view, as it is the first block in the region of interest according to raster scan order. The next block to be packed from view 600 is the second block in raster scan order, immediately to the right of the first block.

[0055] FIG. 7 shows another method of packing blocks of pixels, called row interleaving, which is described below.

[0056] After packing, in step 14, the packed frames 40 are passed to a video encoder 130. The video encoder 130 encodes the packed frames 40 into a bitstream 16. Encoding the packed frames includes compressing (15) the packed frames. Compressing (15) the packed frames 40 reduces the amount of data that must be encoded and transmitted into the bitstream 16. The compression 15 is lossy in this example. Any standard video compression codec can be used. In this example, the High Efficiency Video Coding (HEVC) compression standard is used.

[0057] Compressing (15) packed frames 40 can reduce the amount of data to be coded and transmitted, particularly due to the manner in which packed frames 40 are packed. When the regions of interest 31 of each view 10 contain similar visual content, such as in multi-view images or videos of a scene, the block image data 13 from each view 10 is likely to be similar in visual content. More specifically, due to the block interleaving packing method, adjacent blocks of pixels in a packed frame are likely to be blocks that correspond to similar portions of the region of interest seen from different views. These blocks of pixels are likely to be most similar in terms of visual content. Existing video compression algorithms, such as HEVC, can exploit this similarity (redundancy). Thus, by using block or row interleaving and compressing packed frames using video compression algorithms, embodiments of the present invention can reduce the amount of data that needs to be transmitted while utilizing existing video compression software and / or hardware.

[0058] Additional steps that may be performed by the encoder in some embodiments are shown in FIG. 8 and described below.

[0059] 9 is a block diagram of a decoder 200 according to one embodiment. The decoder 200 comprises an input section 210 configured to receive a bitstream 16. The input section 210 is connected to a video decoder 220 configured to decode the bitstream 16. The video decoder 220 is connected to a reconstruction unit 230 configured to reconstruct one or more views 93 of a scene. It is assumed that the bitstream 16 received by the decoder 200 is generated by the encoder 100 in the embodiment described above.

[0060] The method performed by the decoder 200 will now be described with reference to Figure 10. At step 90, the input unit 210 receives the bitstream 16. The bitstream 16 includes packed frames 40. The bitstream 16 is passed to the video decoder 220 and decoded at step 91. The decoding includes decompression (using HEVC) and retrieval of the packed frames 40. After decoding, the packed frames are passed to a reconstruction unit 230.

[0061] In step 92, reconstruction unit 230 reconstructs three views 93 of the scene. First, for simplicity, the reconstruction will be described with reference to reconstructing only the first view 690. The reconstruction method shown in Figure 11 includes creating a blank first reconstructed view 690. Reconstruction unit 230 inserts blocks of pixels of the first view into first reconstructed view 690 at foreground locations indicated by the first view's block segmentation mask 60.

[0062] The block segmentation mask 60 acts as a map that indicates the locations of foreground blocks (indicated by straight arrows) in the reconstructed view 690. Both the encoder and decoder use the same predefined scan order to scan the block segmentation mask and block image data. When the decoder finds the first foreground pixel in the block segmentation mask, it inserts the first decoded block of pixels (indicated by curved arrows) at the location in the reconstructed view 690 indicated by the pixel in the block segmentation mask. When the decoder finds the next foreground pixel in the block segmentation mask 60, it inserts the next decoded block of pixels for that view until all of the foreground blocks of pixels for that view are placed in their correct locations in the reconstructed view 690.

[0063] This method is the reverse of the packing process performed in the encoder and is applied to reconstruct all views 93 as well.

[0064] The decoder 200 does not require additional metadata to know where to find and locate the associated blocks of pixels. Instead, the sequence of the block segmentation masks 12 and blocks of pixels 30 within the packed frame serves as implicit metadata, indicating to the decoder 200 which block segmentation masks 12 are associated with which blocks of pixels. Avoiding the need to include additional metadata in the packed frame 40 can help reduce the amount of data that must be encoded. Additionally, reconstruction using a block segmentation map is relatively simple to implement and does not involve complex or time-consuming operations that can cause significant additional latency.

[0065] The foreground blocks (i.e., block image data of the region of interest) can be overlaid by the decoder onto any desired background (e.g., a computer graphics background, or an image or video from a different video stream). In some embodiments, multiple reconstructed views 93 can be used to create new virtual views that have a different perspective than any of the views originally captured by the camera.

[0066] Next, we will discuss possible alternative configurations of a packed frame 40, as shown in Figures 5A and 5B. Figure 5A shows a packed frame 40 having a third portion 50 in addition to first and second portions (not shown). The third portion is at the bottom of the packed frame and is the last portion of the frame to be encoded and decoded. The third portion 50 includes block depth data 51 for two or more views 10. The block depth data 51 are packed contiguously into the packed frame 40. The block depth data 51 may be included in at least one packed frame 40 at a lower spatial resolution than the block image data 13. For example, the block depth data 51 may be provided in smaller blocks and / or a fewer number of blocks, since the human visual system is less sensitive to depth errors than errors in image data.

[0067] FIG. 5B shows another packed frame 40 having a fourth portion 52 in addition to first, second, and third portions (not shown). The fourth portion 52 includes block transparency data 53 for two or more views 10. The block transparency data 53 is packed contiguously within the fourth portion 52. The transparency data may be determined during segmentation of the two or more views and stored in the alpha channel 10. A block of pixels at the boundary between a region of interest 31 and the background likely contains both some (green) background pixels and some (non-green) foreground pixels. The block transparency data for a block of pixels specifies, for each pixel in the block, how transparent that pixel is. This may be determined by how green it is in the case of chroma keying, or by any other suitable pixel-level segmentation. Transparent pixels can be treated as background pixels during reconstruction. This helps prevent green (background) pixels incorporated into the block image data for the region of interest 31 from appearing as green halos around objects in the reconstructed image 93. Partially transparent pixels (pixels that contain some green, but not completely green) may have color values ​​averaged by alpha blending. If transparency data 53 is not included in the packed frame, in some embodiments, decoder 200 may itself perform chroma keying on blocks of pixels to determine the transparency data. That is, during reconstruction of view 93, decoder 200 may check each block of pixels 31 (especially surrounding blocks) for green pixels to determine block transparency data for the reconstructed view 93.

[0068] If the packed frame includes block depth data 51 or block transparency data 53, the block depth and block transparency data may be packed into the third and fourth portions of the packed frame in the same pattern as the block image data is packed into the second portion of the packed frame. This means that when a decoder scans the packed frame 40 and reconstructs a view 93, it can apply the block depth / transparency data 51, 53 to the reconstructed view using the same backward block interleaving method that was used to generate the reconstructed view 93.

[0069] In some embodiments, the block interleaving method described with reference to FIG. 6 can be modified to interleave multiple blocks from each view, for example, taking two, three, or more blocks at a time as the interleaving unit.

[0070] 7 shows an example of a packing method based on row interleaving. Row interleaving operates similarly to block interleaving, except that instead of packing one block of pixels from each view at a time, packing unit 120 packs rows of blocks from each view 700, 710, 720. In some embodiments, this principle is extended further for all views 10, such as packing all blocks 30 of pixels from a first view 10 consecutively, followed by all blocks 30 from a second view 10. Decoder 200 always reconstructs view 93 using the inverse of the process used to pack packed frame 40.

[0071] Row interleaving may be more advantageous than block interleaving when it is desired to decode and reconstruct only a subset of views 10. With row interleaving, the prediction of a block of pixels 30 (at the decoder) is more likely to be based on other blocks of pixels 30 from the same view 10, because compression algorithms often predict based on adjacent blocks, and entire rows of blocks 30 are packed as one contiguous group. This can help avoid the need to decode blocks of pixels 30 from additional views 10 when it is desired to decode a given view 10. In a similar manner, packed frames 40 are preferably packed so that all blocks of pixels 30 from a first view 10 are arranged contiguously, followed by all blocks 30 from a second view 10, and so on for all views 10.

[0072] We now turn to an embodiment of the encoding method shown in Figure 8. Figure 8 is a flow chart illustrating an encoding method including quantization and compression. Steps 11, 14 and 15 are substantially identical to the corresponding steps in the embodiment of Figure 2 above.

[0073] In step 11, the encoder 100 obtains a block segmentation mask 12 and block image data 13 from two or more views 10 of a scene.

[0074] In step 80, the encoder 100 selects the number of quantization levels for the block segmentation mask 12. To reduce the amount of data that needs to be transmitted, it may be beneficial to quantize the block segmentation mask 12. As an example, consider a block segmentation mask 12 that uses luminance values ​​as pixel values. First and second values ​​may be used to distinguish between foreground and background, for example, by setting all foreground pixels to 255 and all background pixels to 0. Eight bits are required to encode the values ​​0 and 255. To reduce the amount of data, the pixel values ​​may be quantized (81) to a smaller number of levels. Instead of using 0 and 255 as the first and second values, for example, 0 and 127 may be used, which require only seven bits to represent. In the most extreme reduction, each pixel may have a luminance value of either 0 or 1, represented by only a single bit.

[0075] In step 81, the selected quantization parameter is applied to block segmentation mask 12. In step 14, packing unit 120 generates packed frame 40. In step 82, the encoder selects a quality factor for video compression. The higher the quality factor, the less data is lost due to lossy compression.

[0076] The packed frames are then encoded into a bitstream 16, which includes compressing 15 the packed frames using an HEVC codec (according to a selected quality factor). The decoder 200 is preferably able to accurately reconstruct the block segmentation mask 12 from the compressed packed frames 40; otherwise, the decoder 200 would not be able to accurately reconstruct the view 93. When selecting quantization levels for the block segmentation mask 12, it may be useful to maintain some unused levels between the first and second pixel values. This may help make the method robust to slight deviations in pixel values ​​that may be introduced by lossy video compression.

[0077] To ensure that the block segmentation mask 12 can be accurately reconstructed by the decoder 200, the block segmentation mask 12 must be compressed either by lossless compression or by lossy compression with a minimum quality factor. For example, a binary segmentation mask with values ​​0 and 1 can be compressed by lossless compression or by lossy compression with a certain minimum quality so that none of the pixels change value. Alternatively, a segmentation mask using values ​​0 and 255 can be quantized and compressed by lossy compression. Lossy compression can allow values ​​to deviate from their initial levels, provided that the quantization is robust to these deviations and there is no reconstruction error. There are various ways to ensure this. In one example, quantization and quality factor parameters can be selected so that the probability of error is low or zero over a large training dataset of views. Alternatively, the parameters can be optimized online by the encoder for a given set of views to be coded. To this end, the encoder 100 may include a local decoder 200 that decodes the packed frame 40 and reconstructs the block segmentation mask 12 prior to transmission of the bitstream 16. The encoder 100 may check whether an accurate reconstruction has been achieved by comparing the decoded block segmentation mask from the local decoder with the original block segmentation mask. The encoder 100 may iteratively quantize the block segmentation mask 12, generate a packed frame 40, compress, encode, decode the packed frame 40, compare the reconstructed block segmentation mask with the original, and change the quantization and / or compression conditions each time to achieve an accurately reconstructed block segmentation mask with the minimum amount of data. The optimal solution may then be applied to the packed frame 40, and the resulting bitstream 16 may be transmitted.

[0078] The encoding and decoding methods of Figures 2, 8, and 10, and the encoder 100 and decoder 200 of Figures 1 and 9, may be implemented in hardware or software, or a combination of both (e.g., as firmware running on a hardware device). To the extent that an embodiment is implemented partially or entirely in software, the functional steps illustrated in the process flowcharts may be performed by appropriately programmed physical computing devices, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process, and its individual component steps illustrated in the flowcharts, may be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code configured to cause one or more physical computing devices to perform an encoding or decoding method as described above when the program is executed on the one or more physical computing devices.

[0079] Various modifications to the above examples are possible. For example, in the above examples, a block segmentation map was provided to the encoder. In some embodiments, the encoder may include a segmentation unit configured to receive the view 10 and configured to use chroma keying (or another segmentation algorithm) to generate the block segmentation mask 12.

[0080] In some embodiments, the first and second pixel values ​​used in the block segmentation mask 12 may be chrominance values, depth values, or transparency values. If the block segmentation mask 12 includes two regions of interest, the first and second pixel values ​​may be used to indicate the first and second regions of interest, respectively, and the third pixel value may be used to indicate a background region. For example, a value of 0 may indicate the background, and values ​​of 128 and 255 may indicate the first and second regions of interest (e.g., foreground objects). Widely spaced values ​​such as these may be suitable for lossy compression. Alternatively, values ​​1 and 2 may be used for the regions of interest, and value 0 may be used for the background. Adjacent values ​​such as these may be suitable for lossless compression, without the risk of introducing deviations in the values.

[0081] In some embodiments, packing unit 120 may generate two or more packed frames. The first packed frame may include two or more block segmentation masks 12. The second packed frame may include block image data 13. The two or more packed frames are passed to video encoder 130. Video encoder 130 may encode a first bitstream comprising the first packed frame and a second bitstream comprising the second packed frame. Compression of the first bitstream may be lossless, and compression of the second bitstream may be lossy.

[0082] It is not necessary for the blocks of pixels to be the same size in all views. In some embodiments, the blocks of pixels in some or all of the views may be different sizes. Compression and decompression of packed frames may be performed using any suitable algorithm known in the art, such as H.264 or various MPEG codecs.

[0083] In some embodiments, the block segmentation mask 12 and the block image data 13 need not be packed consecutively. For example, a block segmentation mask for a first view can be immediately followed by the block image data for the first view, which can be followed by the block segmentation mask for a second view and the block image data for the second view, etc.

[0084] The storage media may include volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. Various storage media may be installed or transportable within a computing device so that one or more programs stored on the storage media can be loaded into a processor.

[0085] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the description, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where a computer program is described above, the computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but it may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. When the term "adapted for" is used in the claims or the description, it has the same meaning as the term "configured to." Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A method for encoding image or video data having two or more views of a scene, comprising the steps of: obtaining a respective view block segmentation mask and view block image data for each of the two or more views; generating at least one packed frame including two or more of the block segmentation masks and the block image data of the two or more views; encoding the at least one packed frame into at least one bitstream. Each view is divided into blocks of pixels, each block of pixels in each of the views is represented by a pixel in that view's respective block segmentation mask; the block segmentation mask indicates which blocks of pixels belong to a region of interest of the view; the region of interest is only a portion of the view; A method wherein the block image data consists only of blocks of pixels that belong to the region of interest.

2. 2. The method of claim 1 , wherein the at least one packed frame comprises a packed frame having a first neighboring portion and a second neighboring portion, the first neighboring portion comprising the block segmentation masks of the two or more views, and the second neighboring portion comprising the block image data of the two or more views.

3. 3. A method according to claim 1, wherein the blocks of pixels in the block image data of a view are all the same size within that view, and optionally the blocks of pixels are all the same size in the block image data of all views.

4. 4. The method of claim 1, wherein the blocked image data for each different view is packed into the at least one packed frame in a block interleaved configuration, where a first block of pixels of a first view is followed consecutively by a first block of pixels of a second view.

5. 4. The method of claim 1, wherein the blocked image data of each different view is packed into the at least one packed frame in a row-interleaved configuration, where a block of pixels of a first row of a first view is followed consecutively by a block of pixels of a first row of a second view.

6. 6. The method of claim 1, wherein encoding the at least one packed frame into the at least one bitstream uses a video compression algorithm, optionally using a standard video compression algorithm such as H.264 or HEVC.

7. selecting a quality factor of the video compression algorithm such that at least the block segmentation mask is reconstructable without errors from the at least one bitstream; and / or determining a number of quantization levels to be used in the video compression algorithm such that at least the block segmentation mask can be reconstructed without error from the at least one bitstream; 7. The method of claim 6, comprising:

8. 8. The method of claim 1, further comprising the steps of quantizing the block segmentation mask to a first number of quantization levels and quantizing the block image data to a second number of quantization levels, the first number being different from the second number.

9. the at least one packed frame having a third portion, the third portion having depth data for the two or more views; and / or the at least one packed frame having a fourth portion, the fourth portion having transparency data for the two or more views; 9. The method according to any one of claims 1 to 8.

10. 1. A method for decoding image or video data having two or more views of a scene, comprising the steps of: receiving at least one bitstream in which at least one packed frame including a block segmentation mask for each of the two or more views and block image data for the view is encoded; Each view is divided into blocks of pixels, for each view, the block image data consists only of blocks of pixels that belong to a region of interest in that view, and the block segmentation mask indicates the locations of those blocks of pixels in that view; each block of pixels in each of the views is represented by a pixel in that view's respective block segmentation mask; the region of interest is only a portion of the view; The method further comprises: decoding the at least one bitstream to obtain the at least one packed frame; reconstructing at least one view of the two or more views by arranging the block image data of the at least one view according to the positions indicated by the block segmentation mask for the at least one view; A method having the following.

11. The method of any one of claims 1 to 10, for use in live streaming multiview video.

12. A computer program product which, when executed by a computer, causes the computer to carry out the method of any one of claims 1 to 11.

13. 1. An encoder configured to encode image or video data having two or more views of a scene, comprising: an input configured to obtain, for each of the two or more views, a respective view block segmentation mask and view block image data; Each view is divided into blocks of pixels, each block of pixels in each of the views is represented by a pixel in that view's respective block segmentation mask; the block segmentation mask indicates which blocks of pixels belong to a region of interest of the view; the region of interest is only a portion of the view; an input unit, wherein the block image data consists of only blocks of pixels belonging to the region of interest; a packing unit configured to generate at least one packed frame including two or more of the block segmentation masks and the block image data of the two or more views; a video encoder configured to encode the at least one packed frame into at least one bitstream; An encoder having:

14. 1. A decoder configured to decode image or video data having two or more views of a scene, comprising: an input unit configured to receive at least one bitstream in which at least one packed frame including, for each of the two or more views, a block segmentation mask of the view and block image data of the view, Each view is divided into blocks of pixels, each block of pixels in each of the views is represented by a pixel in that view's respective block segmentation mask; for each view, the block image data consists only of blocks of pixels that belong to a region of interest in that view, and the block segmentation mask indicates the locations of those blocks of pixels in that view; an input unit, wherein the region of interest is only a portion of the view; a video decoder configured to decode the at least one bitstream to obtain the at least one packed frame; a reconstruction unit configured to reconstruct at least one view of the two or more views by arranging the block image data of the at least one view according to the positions indicated by the block segmentation mask for the at least one view; A decoder having:

Images