A coding scheme for immersive video using asymmetric downsampling and machine learning
By downsampling and filtering individual views with metadata-assisted decoding and machine learning, the method addresses visible artifacts in immersive video encoding, enhancing quality and efficiency.
Patent Information
- Application Number
- JP2022517762
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2019-09-24
- Filing Date
- 2020-09-18
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2040-09-18
AI Technical Summary
Existing immersive video encoding methods using texture atlases with pruning and patch-based approaches suffer from visible artifacts, particularly at patch edges and specular highlights, due to misalignment with natural content edges and changing highlight positions.
The method involves downsampling and optionally filtering individual source views to reduce redundancy, encoding them as whole views, and using metadata to assist decoding, without patch-based pruning, and employing machine learning for optimal reconstruction filters.
This approach reduces coding artifacts and improves quality and efficiency by aligning downsampling with content similarity and using machine learning to minimize reconstruction errors, resulting in high-quality immersive video rendering.
Smart Images

Figure 0007577740000001 
Figure 0007577740000002 
Figure 0007577740000003
Abstract
Description
[Technical field]
[0001] The present invention relates to video coding, and more particularly to a method and apparatus for encoding and decoding immersive video. [Background technology]
[0002] Immersive video, also known as six degrees of freedom (6DoF) video, is a video of a three-dimensional (3D) scene that allows a view of the scene to be reconstructed for a viewpoint that changes position and orientation. It is an extension of three degrees of freedom (3DoF) video, which allows a view to be reconstructed for a viewpoint of any orientation but only a fixed point in space. In 3DoF, the degrees of freedom are angular, namely pitch, roll and yaw. 3DoF video supports head rotation. In other words, a user consuming video content can look in any direction in a scene, but cannot move to another location in the scene. 6DoF video supports head rotation and also supports the selection of the location in the scene from which the scene is viewed.
[0003] Producing 6DoF video requires multiple cameras to record a scene. Each camera produces image data (often referred to in this context as texture data) and corresponding depth data. For each pixel, the depth data represents the depth at which the corresponding image pixel data is observed by a given camera. Each of the multiple cameras provides a respective view of the scene. In many applications, it may not be practical or efficient to transmit all of the texture and depth data for all of the views.
[0004] To reduce redundancy between views, it has been proposed to prune views for each frame of a video stream and pack these views into a "texture atlas". This approach aims to reduce or eliminate overlap between multiple views, thereby improving efficiency. The non-overlapping parts of different views that remain after pruning are sometimes called "patches". An example of this approach is described in "High-quality streamable free-viewpoint video" by Alvaro Collet et al., ACM Trans. Graphics (SIGGRAPH), 34(4), 2015. Summary of the Invention [Problem to be solved by the invention]
[0005] It would be desirable to improve the quality and coding efficiency of immersive videos. As mentioned above, the approach of generating texture atlases using pruning has the problem that the edges of the patches may cause visible artifacts in the reconstructed view rendered after decoding. This is especially true when the edges of the patches are not aligned with natural edges in the content, for example, when the patches are selected to be rectangular. Rectangular patches may cause zigzag boundaries as visible artifacts in the decoded output. For these coding methods, specular highlights cause particular difficulties, since the position of the highlights changes from view to view, even if the position of the object does not change. When a view of an object is reconstructed from multiple patches taken from multiple views, the change between views may be particularly noticeable near the highlights, which may be visible in the decoded output. [Means for solving the problem]
[0006] The invention is defined by the claims.
[0007] According to an example embodiment of the present invention, there is provided a method for encoding immersive video, the method comprising: - receiving source video data having a plurality of source views, each of the source views including texture data and depth data; - processing the source video data to generate processed source video data, the processing comprising downsampling one or more of the source views; - encoding the processed source video data to generate a video bitstream and a metadata bitstream; wherein the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views.
[0008] According to an embodiment of this method, the need to prune views and generate patches can be avoided: instead of pruning source views into patches, each source view can be encoded as a whole, although one or more of these source views may be subsampled to reduce redundancy and thus reduce the data rate.
[0009] The texture data may include at least luma (i.e., brightness or intensity) information. It may also include chroma (i.e., color) information. The depth data may include normalized depth data, e.g., normalized depth values calculated as 1 / Z, where Z is the depth from a given reference point or plane. For each view, the texture and / or depth data may include a 2D array of pixels.
[0010] The processing step comprises downsampling one or more of the source views. Each source view typically comprises a two-dimensional array of pixels. Downsampling means that the size of this array is reduced (in one or both dimensions) by reducing the resolution (in each dimension) of that view.
[0011] The processing step may further include filtering in addition to downsampling. For example, the processing may include anti-aliasing filtering followed by downsampling. The metadata may describe the downsampling configuration, the filtering configuration, or both the filtering and downsampling configuration.
[0012] Filtering and downsampling the one or more source views may include filtering and downsampling the texture data or the depth data, or both. The downsampling may be described by a scaling factor, for example downsampling by a factor of 2, 4, or 8 in each dimension. Note that the texture data may be downsampled by the same factor as the depth data, or by a different factor.
[0013] The encoding of the processed source video data may include lossy compression of the texture and / or depth data.
[0014] The metadata may further include camera parameters associated with the multiple source views. Generally, camera parameters may be specified to be independent of image resolution. For example, camera parameters may specify focal length relative to sensor width, or normalized image coordinates may be used. As another example (e.g., if focal length is specified relative to pixel pitch, also known as "pixel units"), camera parameters may refer to full resolution source images or downsampled images. Typically, image size is known from the video bitstream, but may also be part of the metadata. Image size may also be part of the set of camera parameters.
[0015] In the metadata, the downsampling factor can be implicitly specified by defining two different image resolutions (one in the video bitstream and one in the camera parameters), and the decoder can understand by design / specification that it needs to calculate the scaling factor by comparing these resolutions.
[0016] Typically, the encoder outputs metadata that allows the decoder to determine the amount of downsampling that has been performed.
[0017] The method may further comprise analysing the source video data to select a configuration for the processing, the processing including downsampling (and optionally filtering) the one or more source views in accordance with the selected configuration.
[0018] This allows the downsampling (and optionally filtering) to be adapted to the content of the source video data. The analysis may, for example, be based on camera parameters associated with the multiple source views. Alternatively or additionally, the analysis may be based on similarities (i.e. redundancy) of visual content between views.
[0019] In some embodiments, the analysis is performed prior to processing of the source video data, while in other embodiments the analysis includes or depends on the processing (filtering and downsampling).
[0020] The analyzing step optionally includes the steps of analyzing similarity of visual content between the views; assigning a first downsampling factor to the first view; assigning a second downsampling factor to the second view; and assigning a third downsampling factor to the third view, where the second downsampling factor is greater than the third downsampling factor if the similarity between the first view and the second view is greater than the similarity between the first view and the third view.
[0021] The analyzing step may include the steps of obtaining at least one reference view based on the source video data; filtering and downsampling the one or more of the source views according to a plurality of different configurations; and for each of the plurality of different configurations, encoding corresponding processed source video data to generate a candidate video bitstream, decoding one or more of the plurality of source views from the candidate video bitstream, reconstructing at least one test view corresponding to the at least one reference view from the decoded one or more source views, and comparing the reconstructed test view with the reference view, wherein the analyzing step further includes the step of selecting the configuration based on a result of the comparison, and the method further includes the step of selecting a candidate video bitstream associated with the selected configuration as the video bitstream.
[0022] The at least one reference view may include one of the source views. In some embodiments, obtaining the at least one reference view may include synthesizing a reference view from the source video data. The synthesized reference view may correspond to a different viewport than the source view.
[0023] In some embodiments, the analyzing step comprises selecting a configuration in which a reconstructed test view best matches the reference view, in other words, the configuration may be selected based on reducing or minimizing an error between a reference view and a reconstructed test view.
[0024] The method may further comprise the step of designing a reconstruction filter using a machine learning algorithm, where the reconstruction filter can be used by a decoder in upsampling and filtering one or more decoded views, and where the metadata further comprises a description of the reconstruction filter.
[0025] In some embodiments, the machine learning algorithm is a convolutional neural network (CNN), optionally using transfer learning.
[0026] The description of the reconstruction filter may include a set of coefficients (weights) for a CNN, which may be partial (e.g., describing only some layers of the CNN) or complete.
[0027] Designing the reconstruction filter may include training filter coefficients of the reconstruction filter using the machine learning algorithm to reduce a reconstruction error, which may include a difference between a reconstructed test view and a reference view.
[0028] The method may include the steps of obtaining at least one reference view based on the source video data, decoding one or more of the plurality of source views from the video bitstream, and determining a test view to be reconstructed from the decoded one or more source views corresponding to the reference view, and the step of designing the reconstruction filter includes the step of training filter coefficients of the reconstruction filter using the machine learning algorithm to reduce an error between the test view and the reference view.
[0029] The machine learning algorithm may be trained to minimize the error between the test view and the reference view.
[0030] The test view and the reference view may correspond to each other in the sense that they have at least the same viewport parameters.
[0031] In some cases, the reference view may simply be one of the source views, and the test view may be a reconstruction of the source view (in which case synthesizing the reference view may comprise simply copying the source view), while in other cases the reference view may be a synthesized view that is different from any of the source views.
[0032] A method for decoding immersive video is also disclosed, the method comprising: - receiving a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, and the metadata bitstream including metadata to assist in decoding the video bitstream; - obtaining viewport parameters of at least one target view to be rendered; - decoding one or more of the plurality of source views from the video bitstream; - reconstructing the at least one target view from the decoded one or more source views based on the obtained viewport parameters; having the metadata describes a downsampling configuration performed in encoding the one or more source views; The reconstructing step includes upsampling and filtering at least one of the decoded source views or a derivative thereof based on the described configuration.
[0033] The viewport parameters of a view may include a position and an orientation that define the view. These parameters may further include a field of view.
[0034] A derivative of the at least one decoded source view may be a synthesized target view derived from the source view (eg, by projecting / warping the source view according to target viewport parameters).
[0035] The decoding method may comprise the steps of: decoding at least two of the plurality of source views from the video bitstream; and for each of the at least two decoded source views, synthesizing a synthetic target view based on the viewport parameters; wherein the step of reconstructing the at least one target view includes steps of upsampling, filtering and combining (combining) the synthetic target views according to a configuration described in the metadata.
[0036] Synthesizing each synthetic target view may include projecting or warping the corresponding decoded source view according to viewport parameters of said target view.
[0037] The at least two decoded source views may have been downsampled by different factors (1x, 2x, 4x, ...) in the encoder. They may have different sizes / resolutions.
[0038] The upsampling, filtering and combining steps can be performed in separate steps or all together, for example they can be performed all together by a neural network such as a CNN.
[0039] The synthetic target views may have different scales / resolutions, i.e. they may require upsampling by various factors.
[0040] The upsampling and filtering can take various synthetic target views at different scales / resolutions and upsample and filter them to reconstruct the at least one target view. The upsampling, filtering and combining can include using a neural network to process the synthetic target views as input and generate the at least one target view as output.
[0041] The metadata may include a description of a reconstruction filter to be used in the upsampling and filtering steps, and the upsampling and filtering steps may include using the described reconstruction filter.
[0042] The description may identify one of a predetermined set of filters. Alternatively or additionally, the description may include filter coefficients.
[0043] The filter description may include a partial or complete set of weights for a neural network, and the upsampling and filtering may include processing the decoded source view or views using a neural network applying such weights.
[0044] The video bitstream can be received from a remote encoder, the method further comprising transmitting viewport parameters to the encoder.
[0045] The transmitted viewport parameters may be the viewport parameters of the at least one target view, or a prediction of a target view to be rendered in the future. Transmitting an indication or a prediction of a target view upstream may facilitate more efficient coding at the encoder, since the encoder can adjust the encoding to the expected target view.
[0046] There is also provided a computer program comprising computer code which, when executed on a processing system, causes the processing system to carry out the methods summarized above.
[0047] The computer program may be stored in a computer-readable storage medium, which may be a non-transitory storage medium.
[0048] Also provided is a video encoder configured to encode an immersive video, the video encoder comprising: - an input configured to receive source video data having a plurality of source views, each of the source views including texture data and depth data; - a video processor configured to process the source video data to generate processed source video data, the processing including one or more downsamplings of the source views; - an encoder configured to encode the processed source video data to generate a video bitstream and a metadata bitstream; and - an output unit configured to output the video bitstream and the metadata bitstream; wherein the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views.
[0049] The video encoder may further comprise a video analyzer configured to analyze the source video data to select a configuration of the video processor, the video processor configured to filter and downsample the one or more source views in accordance with the selected configuration.
[0050] The video encoder may further comprise a reference view generator and a reference decoder, wherein the reference view generator is configured to generate at least one reference view based on the source video data, the video processor is configured to filter and downsample the one or more of the source views according to a plurality of different configurations, for each of the plurality of different configurations, the encoder is configured to encode corresponding processed source video data to generate a candidate video bitstream, the reference decoder is configured to decode one or more of the plurality of source views from the candidate video bitstream, reconstruct at least one test view corresponding to the at least one reference view from the decoded one or more source views, and compare the reconstructed test view with the reference view, the analyzer is configured to select the configuration based on a result of the comparison, and the output unit is configured to output the candidate video bitstream associated with the selected configuration as the video bitstream.
[0051] The reference view generator may be configured to select one of the source views as the at least one reference view and / or may be configured to synthesize a reference view from the source video data.
[0052] The video encoder may alternatively or additionally comprise a reference view generator, a reference decoder and a trainer, in which case the reference view generator is configured to generate at least one reference view based on the source video data; the reference decoder is configured to decode one or more of the plurality of source views from the candidate video bitstream; The trainer: - defining a test view to be reconstructed from the decoded one or more source views, which corresponds to the reference view; and - using a machine learning algorithm to design a reconstruction filter to be used by the decoder in upsampling and filtering one or more decoded views; It is configured as follows: Here, the reconstruction filter is designed to reduce the error between the test view and the reference view.
[0053] Further provided is a video decoder configured to decode immersive video, the video decoder comprising: - a bitstream input configured to receive a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, and the metadata bitstream including metadata for assisting in decoding the video bitstream; - a control input configured to receive viewport parameters of at least one target view to be rendered; - a decoder configured to decode one or more of the plurality of source views from the video bitstream; - a reconstruction processor configured to reconstruct the at least one target view from the decoded one or more source views based on the obtained viewport parameters, the reconstruction comprising upsampling and filtering of at least one of the decoded one or more source views or derivatives thereof based on the metadata; has.
[0054] The metadata may include a description of a reconstruction filter to be used in the upsampling and filtering, while the reconstruction processor may be configured to upsample and filter at least one of the decoded one or more source views using the described reconstruction filter.
[0055] The reconstruction processor may include a synthesizer configured to synthesize a synthetic target view from each of the one or more source views; and an upscaler configured to upsample, filter, and combine the synthetic target view.
[0056] Further disclosed is a bitstream for encoding an immersive video, the bitstream comprising: - a video bitstream including, for each of a plurality of source views, encoded texture data and encoded depth data; and - a metadata bitstream comprising metadata for assisting a decoder in decoding said video bitstream; wherein the metadata describes a downsampling configuration performed on the one or more source views in a process of encoding the video bitstream.
[0057] In particular, the metadata may include a description of reconstruction filters that should be used by the decoder in upsampling and filtering one or more decoded views.
[0058] The immersive video bitstream can be encoded and decoded using the methods summarized above. The bitstream can be embodied on a computer-readable medium or as a signal modulated onto an electromagnetic carrier wave.
[0059] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.
[0060] For a better understanding of the present invention and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Brief description of the drawings]
[0061] [Figure 1] FIG. 1 is a flowchart of a method for encoding 6DoF video according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram of a 6DoF encoder according to one embodiment. [Diagram 3] FIG. 3 is a flowchart of a method for encoding 6DoF video according to a second embodiment. [Figure 4] FIG. 4 is a flowchart of a method for encoding 6DoF video according to the third embodiment. [Diagram 5] FIG. 5 is a flowchart illustrating a method for decoding 6DoF video according to the fourth embodiment. [Figure 6] FIG. 6 is a block diagram of a 6DoF decoder according to one embodiment. [Figure 7] FIG. 7 is a flowchart illustrating a method for decoding 6DoF video according to the fifth embodiment. [Figure 8] FIG. 8 is a flowchart illustrating a method for decoding 6DoF video according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0062] The present invention will now be described with reference to the drawings.
[0063] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the devices, systems and methods of the present invention will become better understood from the following description, appended claims and accompanying drawings. It should be understood that the figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to denote the same or similar parts.
[0064] A method for encoding and decoding immersive video is disclosed. In the encoding method, source video data including multiple source views is encoded into a video bitstream. At least one of the source views is filtered and downsampled before encoding. A metadata bitstream associated with the video bitstream includes metadata describing the filtering and downsampling configurations to assist a decoder in decoding the video bitstream. The use of downsampled views is believed to help reduce coding artifacts compared to patch-based encoding approaches. Encoders and decoders for immersive video, and immersive video bitstreams are also provided.
[0065] Embodiments of the present invention may be suitable for implementing portions of technical standards such as MPEG-I Part 12 Immersive Video. Where possible, terminology used herein is selected to be consistent with terminology used in MPEG-I Part 12. Nevertheless, it is understood that the scope of the present invention is not limited to MPEG-I Part 12 or any other technical standards.
[0066] It may be useful to provide the following definitions / explanations:
[0067] "3D scene" refers to visual content in a global reference frame.
[0068] An "atlas" is a collection of patches from one or more view representations after a packing process into image pairs that include a texture component image and a corresponding depth component image.
[0069] An "atlas component" is a texture or depth component of an atlas.
[0070] The "camera parameters" define the projection used to generate a view representation from a 3D scene.
[0071] "Pruning" is the process of identifying and extracting occluded regions across views to create patches.
[0072] A "renderer" is an embodiment of a process that creates a viewport or omnidirectional view from a 3D scene representation that corresponds to a viewing position and orientation.
[0073] A "source view" is the source video material before encoding that corresponds to the format of a view representation, which may have been obtained by capturing a 3D scene with a real camera or by projecting it onto a surface using source camera parameters with a virtual camera.
[0074] A "target view" is defined as either a perspective viewport or an omnidirectional view at a desired viewing position and orientation.
[0075] A "view representation" comprises a 2D sample array of texture components and corresponding depth components, and represents the projection of the 3D scene onto a surface using the camera parameters.
[0076] "Viewport" means the projection of a texture onto a flat display of an omnidirectional or 3D image or video field of view, suitable for display and viewing by a user at a particular viewing position and orientation.
[0077] A machine learning algorithm is a self-training algorithm that processes input data to generate or predict output data. In some embodiments of the invention, the input data comprises one or more views decoded from a bitstream, and the output data comprises a prediction / reconstruction of a target view.
[0078] Machine learning algorithms suitable for use in the present invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms such as logistic regression, support vector machines or naive Bayes models are also suitable alternatives.
[0079] The structure of an artificial neural network (or simply neural network) is inspired by the human brain. A neural network is made up of layers, each layer having multiple neurons. Each neuron has a mathematical operation. In particular, each neuron may contain different weighted combinations of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weightings). In the process of processing input data, the mathematical operation of each neuron is performed on the input data to generate a numerical output, and the output of each layer in the neural network is fed (e.g., sequentially) to one or more other layers. The last layer provides the output.
[0080] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining a training data set including training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to correct the machine learning algorithm. Such a process can be repeated until the error converges and the predicted output data entries are sufficiently similar to the training output data entries (e.g., ±1%). This is commonly known as a supervised learning approach.
[0081] For example, if a machine learning algorithm is formed from a neural network, the mathematical operations (weights) of each neuron may be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation algorithms, etc.
[0082] In an embodiment, the training input data entries may correspond, for example, to decoded views from a test video sequence, or to synthetic views reconstructed from such decoded views, and the training output data entries may correspond to original source views (before encoding) or to reference views directly synthesized from the original source views.
[0083] A Convolutional Neural Network (CNN, or ConvNet) is a class of deep neural network that is most commonly applied to analyzing visual images. A CNN is a regularized version of a multi-layer perceptron.
[0084] Fig. 1 shows a flow chart of a method for encoding an immersive video according to a first embodiment. Fig. 2 shows a block diagram of a video encoder 300 capable of implementing the method of Fig. 1.
[0085] In step 110, an input (receiver 310) of the encoder 300 receives source video data. The source video data comprises a number of source views, each of which includes texture and depth data. The receiver 310 is coupled to a video analyzer 320. In step 120, the video analyzer analyzes the source video data to determine the best configuration for the subsequent filtering and downsampling steps. This analysis will be described in more detail below.
[0086] The video analyzer 320 is coupled to a video processor 330, which is also coupled to a receiver 310. The video processor receives the source video data from the receiver 310 and receives from the video analyzer information regarding the configuration determined by the analyzer. In step 130, the video processor filters at least one of the source views and in step 140, the video processor downsamples the filtered source view. The filtering and downsampling are performed according to the configuration determined by the video analyzer.
[0087] The output of the video processor is passed to an encoder 340, which in step 150 encodes the processed source video data to generate a video bitstream, the video bitstream comprising encoded texture data and encoded depth data. The encoder 340 also generates a metadata bitstream containing metadata to assist a decoder in decoding the video bitstream. In particular, the metadata bitstream contains a description of the filtering and downsampling performed on the one or more source views.
[0088] Both the video bitstream and the metadata bitstream are output via an output of the video encoder 300. These bitstreams can be sent to a decoder or can be stored in a machine-readable storage medium for later reading and decoding. Downsampling at least one of the source views achieves compression (bitrate reduction). At the same time, the method of the first embodiment does not rely on patch-based packing of source views into a texture atlas, so that artifacts that may be related to patch edges are avoided.
[0089] Figure 3 shows a flow chart of a method for encoding immersive video according to a second embodiment. Steps similar to those in the embodiment of Figure 1 are given similar reference numbers. The method of Figure 3 differs from the method of Figure 1 mainly in the use of a reference decoder in the video encoder.
[0090] As shown in Fig. 2, the video encoder 300 comprises a reference view generator 375. The video encoder also comprises a reference decoder with a decoder 350, a selector 360 and a synthesizer 370. The video encoder 300 further comprises a trainer 380 coupled to the reference view generator 375 and to an output of the reference decoder, in particular to an output of the synthesizer 370. In step 122, the reference view generator 375 generates at least one reference view from the source video data. The reference view may be one of the source views or a synthesized view. The selected reference view is determined by test viewport parameters input to the video encoder 300.
[0091] The analyzing step 120a, filtering step 130a and downsampling step 140a are similar to the corresponding steps 120, 130 and 140 of Fig. 1, except that in this embodiment the filtering and downsampling are performed according to a number of different configurations. The objective is to determine the best configuration for filtering and downsampling a particular content of the source video data. The reference view is used as a reference or "gold standard" for determining the best configuration.
[0092] For each of the filtering and downsampling configurations, the encoder 340 encodes the corresponding processed source video data to generate a candidate bitstream (see step 150a). The candidate bitstreams are passed from the encoder 340 to the reference decoder via a switch 390. In step 124, the reference decoder decodes one or more of the source views from each candidate video bitstream. The decoding (step 124) is performed by a decoder 350. The relevant source view or views are selected by a selector 360 according to the test viewport. From the selected decoded source views, a synthesizer 370 reconstructs a test view according to the test viewport (step 126). Note that if the test viewport corresponds directly to one of the source views, the synthesizer simply passes the (decoded) source view to the trainer 380. Similarly, the reference view in this case will be the original source view. In general, however, the test viewport will be different from any source view. In this case, the synthesizer 370 reconstructs a test view corresponding to the test viewport from one or more decoded source views (step 126). Similarly, the reference view generator 375 will have synthesized a reference view from the original source views.
[0093] In effect, the reference decoder replicates the decoding process that would be performed by an actual decoder of the video bitstream. The output of the synthesizer 370 is therefore a reconstructed video stream that is identical to what can be rendered at the far-end decoder. In step 128, the trainer 380 compares the reconstructed test view with the reference view. The result of the comparison is passed to the analyzer 320, which selects the filtering and downsampling configuration that resulted in the smallest error between the reconstructed test view and the reference view. The candidate video bitstream associated with this configuration is then selected as the video bitstream to be output via the switch 390.
[0094] In this way, the analysis at the encoder can try different configurations (e.g. different filter kernels and different downsampling rates, etc.) and select the configuration that gives the best performance (in terms of minimum coding error for a given bit rate) at the decoder.
[0095] FIG. 4 is another flow chart showing a method of encoding according to the third embodiment. This method is similar to the second embodiment in that it utilizes a reference decoder in the encoding loop. However, it differs from the second embodiment in that the trainer 380 designs, in step 129, a reconstruction filter to be used by the decoder in upsampling and filtering the decoded view. This can be achieved using machine learning techniques. In a similar manner to the second embodiment, a reference view and a corresponding test view are obtained. In step 129, the trainer uses a machine learning algorithm to train the filter coefficients of the reconstruction filter so that the resulting reconstruction filter minimizes the error between the test view and the reference view. In other words, across frames of video data and / or across pixels of each frame, the machine learning algorithm learns the set of coefficients that results in the best reconstruction performance. At each iteration of the machine learning algorithm (e.g., for each frame of the video), the synthesizer 370 applies the current set of filter coefficients derived so far by the machine learning algorithm. As the machine learning algorithm optimizes the coefficients (e.g., across successive frames), the reconstructed test view becomes a better approximation of the reference view.
[0096] Note that unlike the second embodiment, the method of the third embodiment does not necessarily require encoding multiple candidate video bitstreams. It is the decoding that is optimized, not (necessarily) the encoding. Once the source video data is encoded into a video bitstream and the training of the reconstruction filter is completed, the encoder 340 embeds a description of the reconstruction filter into the metadata bitstream, which the decoder uses to form the reconstruction filter and to configure the upsampling and filtering in the decoding process.
[0097] Fig. 5 is a flow chart of a method for decoding an immersive video according to a fourth embodiment. Fig. 6 shows a block diagram of a corresponding immersive video decoder in which the decoding method can be implemented according to an embodiment. The immersive video decoder 400 comprises a bitstream input (receiver 410); a control input (selector 450); a decoder 420; and a reconstruction processor with a synthesizer 430 and an upscaler 440. The video decoder further comprises a selector 450.
[0098] In step 210, the bitstream input receives the video bitstream and the metadata bitstream generated by the video encoder 300 as described above. The selector 450 receives viewport parameters of at least one target view to be rendered. In step 220, the decoder 420 decodes one or more of the source views (which may be selected by the selector 450 based on the target viewport parameters) from the video bitstream. If the decoder 420 has knowledge or prediction of the target viewport, it can selectively decode the video bitstream accordingly, i.e., first decode the source views most relevant to the target viewport, and optionally skip decoding source views not relevant to the target viewport.
[0099] The synthesizer 430 and the upscaler 440 reconstruct the at least one target view from the decoded source view(s) based on the target viewport parameters. The reconstruction depends on the metadata bitstream describing the filtering and downsampling configuration performed in the video encoder 300. The reconstruction includes a step 230 of upsampling and a step 240 of filtering the one or more decoded source views according to the configuration described in the metadata bitstream. For example, if a particular source view was downsampled by a factor of 2 in the video encoder 300, the upsampling step 230 can upsample this view by a factor of 2 after decoding. The reconstructed target view can be output, for example, to a frame buffer for display.
[0100] By filtering and downsampling one or more of the source views at the encoder and informing the decoder, for example, which views have been downsampled and by what downsampling factor, the bitrate of the video stream can be reduced while allowing the decoder to achieve high quality reconstruction results. In some embodiments, the metadata bitstream includes a description of the reconstruction filters designed by the encoder for use in the decoder. In this case, filtering 240 the decoded source views may comprise filtering using the reconstruction filters defined in the metadata bitstream.
[0101] Fig. 7 shows a variant of the method of Fig. 5 according to a fifth embodiment. In this variant, the target viewport parameters are transmitted upstream towards the encoder in step 250. At the encoder, knowledge of the target viewport can be used to optimise the encoding, for example by allocating higher bitrates to views that are more relevant for reconstructing the target view. According to another example, the encoder can encode each view with multiple scaling factors and based on the viewport the analyser can decide which bitstream to send to the client. (The decision can be changed only once in a period.) Advantageously, a view similar to the viewport is transmitted in high resolution.
[0102] Fig. 8 is a flow chart of a method of decoding according to the sixth embodiment. In step 220a, the decoder 420 decodes at least two source views from a video bitstream. In step 225, the synthesizer 430 synthesizes a synthetic target view from each of the decoded source views based on the viewport parameters of the target viewport. This step may include warping each decoded source view to the target viewport to generate a corresponding synthetic target view. In the example shown in Fig. 8, two synthetic target views are upsampled and filtered separately, one is upsampled in step 232 and filtered in step 242, and the other is upsampled in step 234 and filtered in step 244. The upsampling and filtering may be performed by the upscaler 440. However, this is not necessary and in fact the functions of the upscaler 440 and the synthesizer 430 can be shared or combined. In step 260, the two synthetic target views are combined. This step may involve averaging or interpolating between different synthetic target views. Again, in this embodiment it is assumed that the combining step 260 is performed by the upscaler 440, although this is not required.
[0103] In some embodiments, the upsampling steps 230, 232, 234; the filtering steps 240, 242, 244; and the combining step 260 can be integrated into a single process. This may be especially true if the reconstruction filter is designed using a machine learning algorithm. For example, the reconstruction filter may have a set of weights / coefficients for a neural network. The neural network may be configured to reconstruct a target view from multiple synthetic target views, whereby the upsampling, filtering and combining steps are all performed together by the neural network, which takes the synthetic target views (at different scales / resolutions) as input and outputs a reconstructed target view.
[0104] Referring again to FIG. 2, the components of the video encoder 300 and their operation in various embodiments are described in further detail.
[0105] The source video data (marked "Source Material" in FIG. 2) may be 6DoF source material consisting of luma, chroma and depth components. Camera parameters may be provided that allow points to be reprojected from one view to another. For example, it may be possible to synthesize one view from another and compare similarities and differences due to reflections and occlusions.
[0106] Depth can be represented as a dense depth map (optionally aligned with texture data), or it can be a 3D mesh that associates 2D coordinates to a texture frame. The texture data can include luma and chroma, e.g., YUV or RGB.
[0107] The 6DoF source material may be the result of an acquisition and estimation process or the result of a post-production process. The source video data may include a series of frames (6DoF video). The source may also include other information such as audio data.
[0108] The receiver 310 can receive all the material at once (offline processing), segments of frames at a time, or one frame at a time (online). The receiver can buffer information and provide frames to the analyzer in segments.
[0109] The analyzer 320 can perform calculations to identify views suitable for downscaling and any other filtering. The analyzer 320 can be subject to constraints such as, but not limited to, constraints on bitrate or pixel rate, or a list of possible filtering operations. Another input to the analyzer can be a decoder profile that includes the maximum number of views and / or total number of pixels that the decoder can decode. The output of the analyzer is 1) configurations for further processing, and 2) metadata to be sent to the decoder.
[0110] The list of filtering operations can be, for example, {none, downscale 2x, downscale 4x, skip, etc.}, but can also include other filtering operations, as will be further explained below. Some views can be transmitted at a lower temporal resolution, for example 15 fps instead of 120 fps.
[0111] The analyzer can look only at the camera configuration (especially their relative position in scene space) to identify the appropriate filtering. For example, the most central forward-viewing camera can be kept at full resolution (no filter) and the surrounding cameras are clustered. From each cluster, the central camera is downscaled by 2x. In this case, the remaining cameras in each cluster can be downscaled by 4x. This pattern can continue to have 8x, 16x (and so on) downsampling for some cameras. As another example, the lowest level views are skipped. Such 8x or 16x downsampled thumbnail views do not retain much edge information, but do provide information about viewpoint-dependent color shifts and moving (semi-)specular highlights. According to some embodiments of the present invention, the trained filter can fuse the exact edges of the high-resolution view with the low-resolution neighboring views.
[0112] The source views may contain different perspectives, positions and projections (e.g., fisheye, perspective, etc.), and comparing cameras is not straightforward. Here, the concept of viewport similarity can help. A subset of points (at typical depth values) from a first viewport is reprojected onto a second viewport, and the number of points that match points in the second viewport is counted. Rays can be weighted by angle or resolution. Various methods for calculating viewport similarity are known in the art.
[0113] In some embodiments, the analyzer can perform a point reprojection or view synthesis process for each source view to form a map of inter-view redundancy. Based on this map, the most useful view is selected to have the highest quality. From the remaining views, the view with the most information not present in the high quality view can be downscaled by 2x, and all other views can be 4x. Some views can be identified as (almost) completely redundant, and the analyzer can decide to skip such views in the subsequent encoding process.
[0114] In some embodiments, the analyzer performs a multi-pass analysis, where a 6DoF reference decoder is in the loop with the 6DoF encoder, and the analyzer receives information about the actual quality of a test viewport (which may coincide with the source view). Based on this, the analyzer can modify the configuration for the next pass. The number of passes can be determined in advance, or can be based on a correlation between expected and actual quality determined by running a 6DoF decoder as part of the 6DoF encoder. This approach can be adopted, for example, in the embodiment of FIG. 4.
[0115] The video processor 330 receives the source material and is composed of the analyzer 320. All processing may be performed per view or per view component. For example, the texture of a particular view may be left unfiltered, but the depth may be downsampled by a factor of 4x. At least one view may be scaled somewhat.
[0116] When the aim is to have a number of pixels corresponding to 2-4 source views, but the number of input views could be 10-100, downsampling by 2x is not sufficient. However, if one view was to be kept at full resolution and all other views were to be downsampled by 8x, it is unlikely that there would be enough high resolution information available to render views near the periphery of the visual space. Instead, it is expected that there could be multiple downsampling options, e.g. 2x, 4x, 8x, 16x, etc.
[0117] The video processor may be configured to distort the view such that the processed view has relatively high resolution in some regions (typically regions determined to be more important, e.g., less redundancy), while having relatively low resolution in other regions (typically regions determined to be less important, e.g., because of similarity to other views). The entire view will still fit within a given rectangle, which will typically be smaller than that of the original (full resolution) source view. Note that this variable resolution process, unlike patch-based pruning / packing, does not introduce edging artifacts.
[0118] The video processor may be configured to synthesize a virtual view, such as an omnidirectional central view. In general, a decoder would not know whether an encoded view is real or virtual. It would be useful to flag this in the metadata. In this case, the flag could be used by the decoder's reconstruction filter, which can be configured to prefer a physical view for example over object edges.
[0119] The video processor can be configured to remove certain spatial resolution. For example, low-resolution information of a high-resolution view can be removed due to the presence of a nearby low-resolution view. Another example is to use downsampling to preserve more of the high-frequency information at the expense of intentionally introducing aliasing. For example, a view can be band-pass or high-pass filtered to remove low frequencies and then downsampled. The high-frequency information is aliased but not lost because it is not mixed with the low-frequency information.
[0120] A view can be filtered in multiple ways, resulting in multiple filtered views for a single source view, for example a view can be band-pass filtered in multiple ways (forming a filter bank).
[0121] Typically, the encoder 340 uses 2D video compression techniques in its core to compress multiple views at different resolutions. The encoder can encode the views as separate bitstreams transmitted as a multicast stream, for example HEVC (H.265) or multi-layer HEVC (H.265 Annex F). Alternatively or additionally, the encoder can encode views of the same viewpoint with different filtering using a scalable video codec, such as Scalable HEVC (H.265 Annex H). As a further option, the encoder can apply a multi-view codec. Such an encoder can use inter-view prediction. An example of such a codec is MV-HEVC (H.265 Annex G). As yet another option, the encoder itself can perform inter-view prediction using depth maps. An example of such a codec is 3D-HEVC (H.265 Annex I). Such a codec can be improved by using a 6DoF reference decoder according to an embodiment of the invention to predict the views.
[0122] The encoder 340 can pack several views together into a single frame (atlas) to make better use of the decoding capabilities. For example, too many decoders can cause synchronization problems. Such atlases typically separate views by guard bands and align views across coding units.
[0123] The encoder can use different quantization parameters (QP) for each view and / or perform bitrate allocation based on target bitrates for each region in each of the views. For example, if a region in a view is determined to have little impact on the final rendering quality, this region can be encoded with fewer bits / pixel on average.
[0124] The video encoder 300 may include components of a reference decoder: a decoder 350, a selector 360, and a synthesizer 370. This reference decoder or loop filter may render test viewports that are compared to reference views. The reference views may be source views or may be views synthesized directly from a source.
[0125] The reason for using decoded views is that the reconstruction filters can be trained to compensate for the loss of quality due to lossy compression of the source views. Also, with uncompressed views, the analyzer 320 may determine that a particular view is redundant because it can be accurately predicted. However, it may be determined that the view needs to be encoded (even if not at full resolution) because depth / texture fidelity would be lost if a decoded version of that view were used.
[0126] The reconstructed test and reference views are used as training inputs to train or update the filter coefficients of the reconstruction filter, which in some embodiments are transmitted to the 6DoF decoder 400 in a metadata bitstream.
[0127] Nevertheless, online training / updating of filter coefficients is not mandatory. In some embodiments, all filters are trained offline (and / or designed / tuned by experts). In this case, there would be no Trainer 380 in the video encoder 300. Instead, the errors (per-view or spatial) between the test and reference views are sent to the Analyzer 320, which uses these error maps to improve the configuration, including bitrate allocation. Such error maps may be based on measures such as PSNR, VIF, VMAF, etc.
[0128] An example of the third embodiment (see FIG. 4) utilizes a convolutional neural network with transfer learning. The use of transfer learning can provide significant advantages. Transfer learning trains parts of the neural network offline, while updating the training of other parts online. For example, the last few layers of the neural network are trained online. In this way, less training data is needed to adapt the neural network to the specific content, and only a portion of the neural network coefficients need to be transmitted to the video decoder 400, thus reducing the metadata bitrate.
[0129] Offline, i.e., outside the encoding loop, multiple convolutional neural networks (CNNs) can be pre-trained for a given set of multi-view prediction scenarios. For example, a CNN is trained that uses two reference views for prediction, where one reference view has the original resolution and the other has half the original resolution. A second CNN is trained with three reference views for prediction, where the third reference view is a quarter of the original resolution. For each multi-view prediction scenario, there is a pre-trained CNN. The encoder can compare each scenario and select the best one. This configuration selection is added as metadata. Note that the texture and depth maps do not need to have the same resolution, i.e., they can be filtered and downsampled differently.
[0130] A pre-trained CNN can be improved during encoding using transfer learning to better model a given video, a group of frames, or a single frame. To do this, the weights of most of the filters (usually the early layers of the network) are kept fixed, while the weights of the late filters are adjusted. These weights are specifically adapted to model the characteristics of the video being encoded. Thus, the metadata will contain a reference to the network used, plus the video-specific weights for the layers that are modified. This allows for a significantly lower metadata bitrate without compromising quality.
[0131] Referring again to FIG. 6, the components of the video decoder 400, and the operation of such components in various embodiments, will now be described in further detail.
[0132] The 6DoF decoder 400 receives a 6DoF bitstream (or a portion of such a bitstream). In some embodiments, the receiver 410 receives all the encoded information, while in other embodiments, a prediction of the target viewport is used to determine what subset of the base bitstream is needed, and the server streams only the subset of information to the client (the video decoder 400) via client / server communication. This represents an extension of adaptive streaming previously used for 2D video.
[0133] In some embodiments, the decoder 420 simply decodes all the (base) bitstream received. However, it may be desirable to apply adaptive decoding. Since the latency between the decoder 420 and the display is much smaller than between the 6DoF encoder 300 and the 6DoF decoder 400, the decoder has access to a more accurate target viewport prediction. Therefore, the decoder 420 may determine that it only needs to decode a portion of the received data. This may help reduce the complexity and / or power usage of the client device (6DoF decoder 400).
[0134] The synthesizer 430 takes the decoded views and the target viewport and predicts the target view using view synthesis. In one embodiment, the target viewport may be defined by the orientation of the virtual reality (VR) headset. In the case of a stereo VR headset, there may be one target viewport per eye. In another embodiment, the target viewport may be calculated using an inertial measurement unit (IMU) of a handheld device, such as a smartphone or tablet computing device. In this case, the viewport depends on the orientation of the device. This approach can be used, for example, for "tablet-based augmented reality (AR)".
[0135] As mentioned above, the depth and texture information may be from different views, in which case the depth is projected first and then the texture is retrieved.
[0136] In the view synthesis process, it may happen that the target view is missing some information (disocclusion) or has a lower resolution of some information (stretching). The synthesizer 430 can output a quality measure such as a confidence map, or more specific maps such as a resolution map, a ray angle map (= a map of angles between points, source and target viewports), a stretching map, etc. This additional information is then input to the upscaler 440 and, optionally, the trainer 380 of the 6DoF encoder 300.
[0137] The view synthesis process may perform upscaling along with the view synthesis, or may generate synthesized views at similar resolution as the input views. In the latter case, upscaling is left to upscaler 440.
[0138] The upscaler 440 typically comprises image filters. It performs the upscaling (if this has not already been performed by the synthesizer 430). It can compensate for missing frequencies. It typically performs some blending between the views.
[0139] The upscaler is often a trained filter that converts the synthesized source views into a single image per viewport, using coefficients that are pre-determined or received in the metadata bitstream.
[0140] The analyzer 320 in the 6DoF encoder 300 may determine that a view's texture is needed, but that depth information from a nearby view is sufficient. In that case, the view's depth component may be omitted. In other situations, the analyzer 320 may determine that a view's depth component is needed to accurately model the foreground-background transition, but that the nearby view's texture is sufficient to retexture the depth map in the 6DoF decoder 400. In both cases, the upscaler 440 is designed to be able to accommodate views with missing components. In extreme cases, there is only one depth map or mesh (sometimes called a geometric proxy) and multiple textures, with at least one of these textures having a lower resolution than at least one of the others.
[0141] As mentioned in the description of the receiver 410 and the decoder 420, it is expected that only a portion of the available data is used in the 6DoF decoder 400. Views closer to a target viewport carry more information about the target viewport than views further away.
[0142] The complexity of the training filter can be reduced by only allowing a certain number of views as input. For example, a CNN can be trained to take in one full view, one downscaled by 2x, and one downscaled by 4x. There can be multiple networks depending on which of these scalings is closest to the target viewport, the second closest, etc.
[0143] In this case, even if more views are temporarily available, the selector 450 will still select a fixed number of views. The 6DoF encoder 300 can take advantage of this property to train more unique filters.
[0144] The codec can define what subset of views is active in a given situation. This can be an algorithm implemented in the selector 450 (e.g. a rule like "always take the 3 closest views"), or it can be something that the analyzer 320 decides and sends as metadata. The metadata can take, for example, any of the following forms: Specify a subset of grid points in 3D space; Divide the 3D space into cells, each of which has the subset as an attribute; Algorithms (selection rules) with parameters sent as metadata.
[0145] Note that while most of the above embodiments have used full-view filtering and downsampling as an alternative to patch-based pruning, these techniques can be combined in some embodiments. For example, the 6DoF encoder can implement both techniques and choose which of them achieves the best coding efficiency when applied to a given video sequence.
[0146] The encoding and decoding methods of Figures 1, 3, 4, 5, 7 and 8, as well as the encoders and decoders of Figures 2 and 6, may be implemented in hardware or software or a mixture of both (e.g., firmware running on a hardware device). To the extent that an embodiment is implemented partially or entirely in software, the functional steps illustrated in the process flow charts may be performed by one or more appropriately programmed physical computing devices, such as one or more central processing units (CPUs), graphics processing units (GPUs) or neural network accelerators (NNAs). Each process (and its individual constituent steps illustrated in the flow charts) may be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code configured to cause one or more physical computing devices to perform the encoding or decoding methods described above when executed on the one or more physical computing devices.
[0147] The storage medium may include volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The various storage media may be fixed or transportable within a computing device, allowing one or more programs stored therein to be loaded into a processor.
[0148] Metadata according to an embodiment may be stored on a storage medium. A bitstream according to an embodiment may be stored on the same storage medium or on a different storage medium. The metadata may be embedded in the bitstream, but this is not required. Similarly, the metadata and / or the bitstream (with or separate from the metadata in the bitstream) may be transmitted as a signal modulated onto an electromagnetic carrier wave. The signal may be defined according to a standard for digital communications. The carrier wave may be an optical carrier wave, a radio frequency wave, a millimeter wave, or a short-range wireless communication wave. It may be wired or wireless.
[0149] Insofar as an embodiment is implemented partially or entirely in hardware, the blocks shown in the block diagrams of Figures 2 and 6 may be separate physical components or logical parts of a single physical component, or may all be implemented in an integrated manner in one physical component. The functionality of a block shown in the figures may be divided into multiple parts in the implementation, or the functionality of multiple blocks shown in the figures may be combined into a single element in the implementation. Hardware components suitable for use in the embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more blocks may be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuits for performing other functions.
[0150] Overall, examples of methods for encoding and decoding data, computer programs implementing these methods, video encoders and decoders, and bitstreams are given in the following embodiments.
[0151] [Embodiment] 1. A method for encoding immersive video, the method comprising: - receiving (110) source video data having a plurality of source views, each of which includes texture data and depth data; - processing the source video data to generate processed source video data, the processing comprising downsampling (140) one or more of the source views; and - encoding (150) the processed source video data to generate a video bitstream and a metadata bitstream; having - the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views. 2. The method of embodiment 1, further comprising a step of analyzing (120) the source video data to select a configuration for the processing, the processing comprising a step of filtering and downsampling the one or more source views in accordance with the selected configuration. 3. The method of embodiment 2, wherein the analyzing step (120) comprises: - analyzing visual content similarity between said views; - assigning a first downsampling factor to a first view; - assigning a second downsampling factor to the second view; and - assigning a third downsampling factor to a third view; having the second downsampling factor is greater than the third downsampling factor if a similarity between the first view and the second view is greater than a similarity between the first view and the third view. 4. The method of embodiment 2 or embodiment 3, wherein the analyzing step comprises: - obtaining at least one reference view based on the source video data; - filtering (130) and downsampling (140) said one or more of said source views according to a number of different configurations; - for each of said plurality of different configurations, encoding (150) the corresponding processed source video data to generate a candidate video bitstream; Decoding (124) one or more of the plurality of source views from the candidate video bitstreams; reconstructing (126) at least one test view from the decoded one or more source views, the test view corresponding to the at least one reference view; comparing (128) the reconstructed test view with the reference view; Steps and; having - the analyzing step (120a) further comprises selecting the configuration based on a result of the comparison; - the method further comprises the step of selecting, as said video bitstream, said candidate video bitstream associated with said selected configuration. 5. A method according to any one of the preceding embodiments, further comprising the step of designing the reconstruction filter using a machine learning algorithm (129), the reconstruction filter is usable by a decoder for upsampling and filtering one or more decoded views; The metadata further comprises a description of the reconstruction filter. 6. The method of embodiment 5, comprising: - obtaining (122) at least one reference view based on said source video data; - decoding (124) one or more of the plurality of source views from the video bitstream; and - defining a test view corresponding to the reference view, the test view being reconstructed (126) from the decoded source view or views; having - designing the reconstruction filter comprises training filter coefficients of the reconstruction filter using the machine learning algorithm, thereby reducing an error between the test view and the reference view. 7. A method of decoding immersive video, comprising: - receiving (210) a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, and the metadata bitstream including metadata to assist in decoding the video bitstream; - obtaining viewport parameters of at least one target view to be rendered; - decoding (220) one or more of the plurality of source views from the video bitstream; and - reconstructing the at least one target view from the decoded one or more source views based on the obtained viewport parameters; having - the metadata describes a downsampling configuration performed in encoding the one or more source views; - the reconstructing step comprises upsampling (230) and filtering (240) at least one of the decoded source views or a derivative thereof based on the described configuration. 8. The method of embodiment 7, comprising: - decoding (220a) at least two of the plurality of source views from the video bitstream; and - for each of the at least two decoded source views, synthesizing (225) a synthetic target view based on the viewport parameters; having - the step of reconstructing at least one target view comprises the steps of upsampling (232, 234), filtering (242, 244) and combining (260) the synthetic target view according to the configuration described in the metadata. 9. The method of embodiment 7 or embodiment 8, wherein the metadata includes a description of a reconstruction filter to be used in the upsampling and filtering steps, and the upsampling (230, 232, 234) and filtering (240, 242) steps include using the described reconstruction filter. 10. The method according to any one of embodiments 7 to 9, wherein the video bitstream is received from a remote encoder, the method further comprising the step of transmitting (250) viewport parameters to the encoder. 11. A computer program comprising computer code which, when executed on a processing system, causes the processing system to perform the method of any one of embodiments 1 to 10. 12. A video encoder (300) configured to encode immersive video, comprising: - an input (310) configured to receive source video data having a plurality of source views, each of which includes texture data and depth data; - a video processor (330) configured to process the source video data to generate processed source video data, the processing including one or more downsamplings of the source views; - an encoder (340) configured to encode the processed source video data to generate a video bitstream and a metadata bitstream; and - an output unit configured to output the video bitstream and the metadata bitstream; having - the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views. 13. The video encoder of embodiment 12, further comprising a video analyzer (320) configured to analyze the source video data and select a configuration for the video processor (330), the video processor being configured to filter and downsample the one or more source views according to the selected configuration. 14. A video decoder (400) configured to decode immersive video, comprising: - a bitstream input unit (410) configured to receive a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, and the metadata bitstream having metadata for assisting in decoding the video bitstream; - a control input configured to receive viewport parameters of at least one target view to be rendered; - a decoder (420) configured to decode one or more of the plurality of source views from the video bitstream; and - a reconstruction processor (430, 440) configured to reconstruct the at least one target view from the decoded source view or views based on the obtained viewport parameters, the reconstruction comprising upsampling and filtering of at least one of the decoded source views or derivatives thereof based on the metadata; has. 15. A bitstream encoding immersive video comprising: - a video bitstream including encoded texture data and encoded depth data for each of a plurality of source views; and - a metadata bitstream comprising metadata for assisting a decoder in decoding said video bitstream; having - the metadata describes a downsampling configuration performed on the one or more source views in the process of encoding the video bitstream.
[0152] More specifically, the invention is defined by the appended claims.
[0153] Variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from an examination of the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the singular does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. In the case of the computer program described above, the program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, provided together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems. It should be noted that when the term "adapted" is used in the claims or the detailed description, the term "adapted" is intended to be equivalent to the term "configured". Reference signs in the claims should not be interpreted as limiting the scope.
Claims
1. 1. A method of encoding an immersive video, comprising: receiving source video data having a plurality of source views including at least three source views, each of the source views including texture data and depth data; analyzing the source video data to select a configuration for processing the source video data; processing the source video data to generate processed source video data, the processing comprising downsampling one or more of the plurality of source views in accordance with the selected configuration; and encoding the processed source video data for the multiple source views to generate a video bitstream and a metadata bitstream. having the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views; The analyzing step includes: analyzing visual content similarity between the source views; assigning a first downsampling factor to a first view; assigning a second downsampling factor to the second view; and Assigning a third downsampling factor to a third view. having if a similarity in visual content between the first view and the second view is greater than a similarity in visual content between the first view and the third view, the second downsampling factor is greater than the third downsampling factor. method
2. The analyzing step includes: obtaining at least one reference view based on the source video data; filtering and downsampling the one or more of the source views according to a number of different configurations; For each of the plurality of different configurations, encoding the corresponding processed source video data to generate a candidate video bitstream; decoding one or more of the plurality of source views from the candidate video bitstreams; reconstructing at least one test view from the decoded one or more source views, the test view corresponding to the at least one reference view; comparing the reconstructed test view with the reference view; Steps and having The analyzing step further includes selecting the configuration based on a result of the comparison; the method further comprising the step of selecting as the video bitstream the candidate video bitstream associated with the selected configuration; The method of claim 1.
3. and designing the reconstruction filter using a machine learning algorithm. the reconstruction filter is used by a decoder for upsampling and filtering one or more decoded views; the metadata further comprising a description of the reconstruction filter; The method according to claim 1 or claim 2.
4. obtaining at least one reference view based on the source video data; decoding one or more of the plurality of source views from the video bitstream; and defining a test view corresponding to the reference view, the test view being reconstructed from the decoded one or more source views. having designing the reconstruction filter includes training filter coefficients of the reconstruction filter using the machine learning algorithm, thereby reducing an error between the test view and the reference view. The method according to claim 3.
5. 1. A method for decoding immersive video, the method comprising: receiving a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, the plurality of source views including at least three source views, and the metadata bitstream including metadata to assist in decoding the video bitstream; obtaining viewport parameters of at least one target view to be rendered; decoding at least two of the plurality of source views from the video bitstream; and reconstructing the at least one target view from the at least two decoded source views based on the obtained viewport parameters. having the metadata describes a downsampling configuration performed in encoding the one or more source views; The method includes synthesizing, for each of the at least two decoded source views, a synthesized target view based on the viewport parameters; reconstructing the at least one target view comprises upsampling and filtering at least one of the synthetic target views based on a configuration described in the metadata, and combining the synthetic target views. method.
6. The method of claim 5 , wherein the metadata includes a description of a reconstruction filter to be used in the upsampling and filtering steps, and the upsampling and filtering steps include using the described reconstruction filter.
7. The method according to claim 5 or 6, wherein the video bitstream is received from a remote encoder, the method further comprising the step of transmitting viewport parameters to the remote encoder.
8. A computer program comprising computer code which, when executed on a processing system, causes the processing system to carry out a method according to any one of claims 1 to 7.
9. 1. A video encoder for encoding immersive video, comprising: an input for receiving source video data having a plurality of source views, including at least three source views, each of the source views including texture data and depth data; Video Analyzer, Video processor, Encoder and Output section having the video analyzer analyzing the source video data to select a configuration for the video processor; the video processor processes the source video data to generate processed source video data, where the processing includes downsampling one or more of the plurality of source views, the video processor downsampling the one or more source views in accordance with the selected configuration; the encoder encodes the processed source video data for the multiple source views to generate a video bitstream and a metadata bitstream; the output unit outputs the video bitstream and the metadata bitstream; the video bitstream includes encoded texture data and encoded depth data, and the metadata bitstream includes metadata for assisting a decoder in decoding the video bitstream, the metadata describing a configuration of processing performed on the one or more source views; The video analyzer comprises: Analyzing visual content similarity between the source views; assigning a first downsampling factor to a first view; assigning a second downsampling factor to the second view; and assigning a third downsampling factor to the third view; the second downsampling factor is greater than the third downsampling factor if a similarity in visual content between the first view and the second view is greater than a similarity in visual content between the first view and the third view. Video encoder.
10. 1. A video decoder for decoding immersive video, the video decoder comprising: a bitstream input for receiving a video bitstream and a metadata bitstream, the video bitstream including encoded texture data and encoded depth data for each of a plurality of source views, the plurality of source views including at least three source views, the metadata bitstream including metadata for assisting in decoding the video bitstream; a control input for receiving viewport parameters of at least one target view to be rendered; a decoder for decoding at least two of the plurality of source views from the video bitstream; and a reconstruction processor for reconstructing the at least one target view from the at least two decoded source views based on the obtained viewport parameters. having the reconstruction processor comprising: a synthesizer for synthesizing a synthetic target view from each of the at least two decoded source views; and an upscaler for upsampling and filtering at least one of the synthetic target views and combining the synthetic target views. Video decoder.
Citation Information
Patent Citations
Image transmission method, image transmission apparatus, image sending apparatus, image receiving apparatus, image sending program, and image receiving program
JP2013030898A
Frame packing for video coding
JP2016105631A
Transcoding video
US20180242017A1
