Video data encoding method

Abstract

To provide a method and an apparatus for encoding and decoding immersive video that improves the quality and coding efficiency of immersive video.SOLUTION: An encoding method encodes source video data including one or more source views into a video bitstream, nonlinearly filtering and downsampling depth data of at least one of the source views before encoding, and upsampling and nonlinearly filtering the decoded depth data after decoding.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to video encoding, and more particularly to a method and apparatus for encoding and decoding immersive video. [Background technology]

[0002] Immersive video, also known as six degrees of freedom (6DoF) video, is a video of a three-dimensional (3D) scene that allows a view of the scene to be reconstructed for a viewpoint whose position and orientation change. It represents a further development of three degrees of freedom (3DoF) video, allowing the view to be reconstructed for a viewpoint with any orientation but only at some fixed point in space. In 3DoF, the degrees of freedom are angles, namely pitch, roll and yaw. 3DoF video supports head rotation, in other words, a user consuming video content can look in any direction in the scene, but cannot move to a different location in the scene. 6DoF video supports head rotation and also supports the selection of a position in the scene from which the scene is observed.

[0003] To generate 6DoF video, multiple cameras are required to record a scene. Each camera generates image data (often referred to in this context as texture data) and corresponding depth data. For each pixel, the depth data represents the depth at which the corresponding image pixel data is observed by a given camera. Each of the multiple cameras provides a respective view of the scene. In many applications, it may not be practical or efficient to transmit all of the texture and depth data for every view.

[0004] To reduce redundancy between views, it has been proposed to prune views and pack them into a "texture atlas" for each frame of a video stream. This approach attempts to reduce or eliminate overlap between multiple views, thereby improving efficiency. The non-overlapping parts of each view that remain after pruning are sometimes called "patches". An example of this approach is given in Alvaro Collet et al, "High-quality streamable free-viewpoint video", ACM Trans. Graphics (SIGGRAPH), 34(4), 2015. Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to improve the quality and coding efficiency of immersive videos. As mentioned above, approaches that use pruning (i.e., filtering out redundant texture patches) to generate texture atlases can help reduce pixel rates. However, pruning of views often requires detailed analysis, which is not error-free and may result in quality degradation for end users. Therefore, a robust and simple method for reducing pixel rates is needed. [Means for solving the problem]

[0006] The invention is defined by the claims.

[0007] According to an example embodiment of the present invention, there is provided a method for encoding video data comprising one or more source views, each source view comprising a texture map and a depth map, the method comprising: receiving video data; processing the depth map of at least one source view to generate a processed depth map, Nonlinear filtering, and A step including downsampling; and encoding the processed depth map and the texture map of the at least one source view to generate a video bitstream.

[0008] Preferably, at least part of the non-linear filtering is performed before downsampling.

[0009] The inventors have found that nonlinear filtering of the depth map before downsampling helps to avoid, reduce or mitigate errors introduced by downsampling. In particular, nonlinear filtering can help to prevent small or thin foreground objects from disappearing partially or entirely from the depth map due to downsampling. It has been found that linear filtering is preferable in this respect since it can introduce intermediate depth values ​​at the boundary between the foreground object and the background. This makes it difficult for the decoder to distinguish between object boundaries and large depth gradients.

[0010] The video data may include 6DoF immersive video.

[0011] The non-linear filtering may include expanding a region of at least one foreground object in the depth map.

[0012] Upscaling foreground objects before downsampling helps to ensure that the foreground objects survive the downsampling process better, i.e., are better preserved in the processed depth map.

[0013] Foreground objects can be identified as localized groups of pixels with relatively small depths, and the background can be identified as pixels with relatively large depths. Pixels around the foreground objects can be locally distinguished from the background, for example, by applying a threshold to the depth values ​​in the depth map.

[0014] The nonlinear filtering may include morphological filtering, especially grayscale morphological filtering, for example a maximum filter, a minimum filter, or another order filter. If the depth map includes depth levels with special meaning, for example if depth level zero indicates a non-valid depth, such depth levels should preferably be considered as foreground, regardless of their actual value. In that way, these levels are preferably preserved after subsampling. As a result, their regions can also be expanded.

[0015] The non-linear filtering may include applying a filter designed using a machine learning algorithm.

[0016] The machine learning algorithm may be trained to reduce or minimize the reconstruction error of the reconstructed depth map after the processed depth map is encoded and decoded.

[0017] The trained filter can similarly help to preserve foreground objects in the processed (downsampled) depth map.

[0018] The method may further include designing a filter using a machine learning algorithm, the filter designed to reduce a reconstruction error in a reconstructed depth map after the processed depth map is encoded and decoded, and the nonlinear filtering includes applying the designed filter.

[0019] The non-linear filtering may involve processing by a neural network, and designing the filter may involve training the neural network.

[0020] The nonlinear filtering may be performed by a neural network having multiple layers, and the downsampling may be performed between two of the layers.

[0021] The downsampling may be performed by a max-pooling (or min-pooling) layer of a neural network.

[0022] The method may include processing the depth map according to a plurality of sets of processing parameters to generate a respective plurality of processed depth maps, the method further including selecting a set of processing parameters that reduces a reconstruction error of a reconstructed depth map after each processed depth map is encoded and decoded, and generating a metadata bitstream identifying the selected set of parameters.

[0023] This allows the parameters to be optimized for a particular application or for a particular video sequence.

[0024] The processing parameters may include a definition of the non-linear filtering and / or the downsampling to be performed. Alternatively or additionally, the processing parameters may include a definition of the processing operations to be performed in the decoder when reconstructing the depth map.

[0025] For each set of processing parameters, the method may include generating a respective processed depth map in accordance with the set of processing parameters, encoding the processed depth map to generate an encoded depth map, decoding the encoded depth map, reconstructing the depth map from the decoded depth map, and comparing the reconstructed depth map with the depth map of at least one source view to determine a reconstruction error.

[0026] According to another aspect, there is provided a method for decoding video data having one or more source views, the method comprising: receiving a video bitstream having an encoded depth map and an encoded texture map for at least one source view; decoding the encoded depth map to generate a decoded depth map; decoding the encoded texture map to generate a decoded texture map; and processing the decoded depth map to generate a reconstructed depth map, the processing comprising: Upsampling, and Includes non-linear filtering.

[0027] The method may further include detecting that the decoded depth map has a lower resolution than the decoded texture map prior to processing the decoded depth map to generate a reconstructed depth map.

[0028] In some encoding schemes, depth maps may be downsampled only in certain cases, or only for certain views. By comparing the resolution of the decoded depth map with the resolution of the decoded texture map, the decoding method can determine whether downsampling was applied at the encoder. This can avoid the need for metadata in the metadata bitstream to signal which depth maps were downsampled and the degree to which they were downsampled. (In this example, we assume that the texture maps are encoded at full resolution.)

[0029] To generate the reconstructed depth map, the decoded depth map may be upsampled to the same resolution as the decoded texture map.

[0030] Preferably, the non-linear filtering in the decoding method is adapted to compensate for the effects of the non-linear filtering applied in the encoding method.

[0031] The nonlinear filtering may include reducing a region of the at least one foreground object in the depth map, which may be appropriate if the nonlinear filtering during encoding includes increasing a region of the at least one foreground object.

[0032] The non-linear filtering may include morphological filtering, particularly grayscale morphological filtering, for example a maximum filter, a minimum filter or another ordinal filter.

[0033] The nonlinear filtering during decoding preferably compensates or reverses the effect of the nonlinear filtering during encoding. For example, if the nonlinear filtering during encoding includes a maximum filter (grayscale dilation), the nonlinear filtering during decoding can include a minimum filter (grayscale erosion), and vice versa. If the depth map includes depth levels with special meaning, such as depth levels where depth level zero indicates a non-valid depth, such depth levels should preferably be considered as foreground regardless of their actual value.

[0034] Preferably, at least part of the non-linear filtering is performed after upsampling. Optionally, all of the non-linear filtering is performed after upsampling.

[0035] The processing of the decoded depth map may be based at least in part on the decoded texture map. The inventors have recognized that the texture map contains useful information that helps to reconstruct the depth map. In particular, if the boundaries of the foreground objects are altered by non-linear filtering during encoding, analysis of the texture map can help to compensate or reverse this alteration.

[0036] The method may include upsampling the decoded depth map, identifying surrounding pixels of at least one foreground object in the upsampled depth map, determining whether the surrounding pixels are more similar to the foreground object or more similar to the background based on the decoded texture map, and applying nonlinear filtering only to the surrounding pixels determined to be more similar to the background.

[0037] In this way, the texture map is used to help identify pixels that have been transformed from background to foreground as a result of nonlinear filtering during encoding. Nonlinear filtering during decoding helps to return these identified pixels to being part of the background.

[0038] The non-linear filtering may include smoothing edges of at least one foreground object.

[0039] The smoothing may include identifying peripheral pixels of at least one foreground object in the upsampled depth map; for each peripheral pixel, analyzing the number and / or arrangement of foreground and background pixels in a neighborhood around the peripheral pixel; based on the results of the analysis, identifying peripheral pixels that are outside the projection from the object to the background; and applying nonlinear filtering to only the identified peripheral pixels.

[0040] The analysis may include counting the number of nearby background pixels, where a surrounding pixel is identified as an outlier from the object if the number of nearby background pixels is above a predefined threshold.

[0041] Alternatively or in addition, the analysis may include identifying spatial patterns of nearby foreground and background pixels, and a surrounding pixel is identified as an outlier if the spatial pattern of its neighborhood matches one or more predetermined spatial patterns.

[0042] The method may further include receiving a metadata bitstream associated with the video bitstream, the metadata bitstream specifying a set of parameters, and the method may optionally include processing the decoded depth map in accordance with the specified set of parameters.

[0043] The processing parameters may include a definition of the non-linear filtering and / or an upsampling definition to be performed.

[0044] The non-linear filtering may include applying a filter designed using a machine learning algorithm.

[0045] The machine learning algorithm may be trained to reduce or minimize the reconstruction error of the reconstructed depth map after the processed depth map is encoded and decoded.

[0046] The filters can be defined in a metadata bitstream associated with a video bitstream.

[0047] Also provided is a computer program comprising computer code which, when executed on a processing system, causes the processing system to carry out the method summarized above.

[0048] The computer program can be stored in a computer readable storage medium, which may be a non-transitory storage medium.

[0049] According to another aspect, there is provided a video encoder configured to encode video data including one or more source views, each source view having a texture map and a depth map, the video encoder comprising: an input configured to receive video data; 1. A video processor configured to process a depth map of at least one source view to generate a processed depth map, the processing comprising: Nonlinear filtering, and a video processor, including downsampling; an encoder configured to encode the texture map of the at least one source view and the processed depth map to generate a video bitstream; and an output configured to output a video bitstream.

[0050] According to yet another aspect, there is provided a video decoder configured to decode video data comprising one or more source views, the video decoder comprising:

[0051] a bitstream input configured to receive a video bitstream, the video bitstream including an encoded depth map and an encoded texture map for at least one source view; a first decoder configured to decode the encoded depth map from the video bitstream to generate a decoded depth map; a second decoder configured to decode the encoded texture map from the video bitstream to generate a decoded texture map; a reconstruction processor configured to process the decoded depth map to generate a reconstructed depth map, the processing comprising: Upsampling, and a reconstruction processor including non-linear filtering; and an output configured to output the reconstructed depth map.

[0052] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief description of the drawings]

[0053] For a better understanding of the present invention and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1] FIG. 1 illustrates an example of encoding and decoding immersive video using existing video codecs. [Diagram 2]4 is a flow chart illustrating a method for encoding video data according to one embodiment. [Diagram 3] 1 is a block diagram of a video encoder according to one embodiment. [Figure 4] 4 is a flow chart illustrating a method for encoding video data according to a further embodiment; [Diagram 5] 4 is a flow chart illustrating a method for decoding video data according to one embodiment. [Figure 6] 2 is a block diagram of a video decoder according to one embodiment. [Figure 7] 4 illustrates a method for selectively applying non-linear filtering to certain pixels in a decoding method according to one embodiment. [Figure 8] 4 is a flow chart illustrating a method for decoding video data according to a further embodiment; [Figure 9] FIG. 1 illustrates the use of neural network processing to encode and decode video data, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] The present invention will now be described with reference to the drawings.

[0055] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, the appended claims and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0056] A method for encoding and decoding immersive video is disclosed. In the encoding method, source video data including one or more source views are encoded into a video bitstream. Depth data of at least one of the source views is nonlinearly filtered and downsampled before encoding. Downsampling the depth map helps to reduce the amount of data transmitted, and therefore helps to reduce the bitrate. However, the inventors have found that simply downsampling can cause thin or small foreground objects, such as cables, to disappear from the downsampled depth map. An embodiment of the present invention aims to mitigate this effect and preserve small and thin objects in the depth map.

[0057] Embodiments of the present invention are suitable for implementing portions of technical standards such as ISO / IEC 23090-12 MPEG-I Part 12 Immersive Video. Wherever possible, terminology used herein is selected to be consistent with terminology used in MPEG-I Part 12. Nevertheless, it will be understood that the scope of the present invention is not limited to MPEG-I Part 12 or any other technical standards.

[0058] It is useful to state the following definitions / explanations:

[0059] "3D scene" refers to visual content in a global reference frame.

[0060] An "atlas" is a collection of patches into image pairs containing texture component images and corresponding depth component images from one or more view representations after a packing process.

[0061] An "atlas component" is a texture or depth component of an atlas.

[0062] The "camera parameters" define the projection used to generate a view representation from a 3D scene.

[0063] "Pruning" is the process of identifying and extracting occluded regions across views, resulting in patches.

[0064] A "renderer" is an embodiment of a process that creates a viewport or omni-directional view from a 3D scene representation that corresponds to some viewing position and orientation.

[0065] A "source view" is the source video material before encoding, corresponding to a format of a view representation that can be obtained by capturing a 3D scene with a real camera, or by projecting onto a surface with a virtual camera using the source camera parameters.

[0066] A "target view" is defined as either a perspective viewport or an omnidirectional view at a desired viewing position and orientation.

[0067] A "view representation" consists of a 2D sample array of a texture component and a corresponding depth component, which represents the projection of the 3D scene onto a surface using the camera parameters.

[0068] A machine learning algorithm is any self-learning algorithm that processes input data to generate or predict output data. In some embodiments of the invention, the input data includes one or more views decoded from a bitstream and the output data includes a prediction / reconstruction of a target view.

[0069] Suitable machine learning algorithms for use in the present invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms such as logistic regression, support vector machines, or naive Bayes models are suitable alternatives.

[0070] The structure of an artificial neural network (or simply neural network) is inspired by the human brain. A neural network consists of layers, each layer containing multiple neurons. Each neuron contains a mathematical operation. In particular, each neuron may contain different weighted combinations of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weightings). In the process of processing input data, the mathematical operation of each neuron is performed on the input data to generate a numerical output, and the output of each layer in the neural network is fed (e.g., sequentially) to one or more other layers. The final layer provides the output.

[0071] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining a training data set that includes training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to correct the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is commonly known as a supervised learning technique.

[0072] For example, if a machine learning algorithm is formed from a neural network, the mathematical operations (weights) of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation algorithms, etc.

[0073] Convolutional Neural Networks (CNNs or ConvNets) are a class of deep neural networks that are most commonly applied to the analysis of visual images. CNNs are regularized versions of multi-layer perceptrons.

[0074] FIG. 1 shows in simplified form a system for encoding and decoding immersive video. An array of cameras 10 is used to capture multiple views of a scene. Each camera captures a conventional image of its front view (herein referred to as a texture map) and a depth map. The set of views, including texture and depth data, are provided to an encoder 300. The encoder encodes both the texture and depth data into a conventional video bitstream, in this case a High Efficiency Video Coding (HEVC) bitstream. This is accompanied by a metadata bitstream to inform the decoder 400 of the meaning of each part of the video bitstream. For example, the metadata tells the decoder which parts of the video bitstream correspond to the texture map and which parts correspond to the depth map. Depending on the complexity and flexibility of the encoding scheme, more or less metadata is needed. For example, a very simple scheme can very strictly dictate the structure of the bitstream and require little or no metadata to unpack it at the decoder side. If the bitstream has many optional possibilities, more metadata is needed.

[0075] The decoder 400 decodes the encoded views (texture and depth). It passes the decoded views to a compositor 500. The compositor 500 is coupled to a display device, such as a virtual reality headset 550. The headset 550 requests the compositor 500 to synthesize and render a particular view of a 3D scene using the decoded views according to the current position and orientation of the headset 550.

[0076] An advantage of the system shown in Figure 1 is that the texture and depth data can be encoded and decoded using conventional 2D video codecs. However, a disadvantage is that there is a large amount of data to encode, transport, and decode. It is therefore desirable to reduce the data rate while compromising the quality of the reconstructed views as little as possible.

[0077] Fig. 2 illustrates an encoding method according to a first embodiment. Fig. 3 illustrates a video encoder that can be configured to execute the method of Fig. 2. The video encoder comprises an input unit 310 configured to receive video data. A video processor 320 is coupled to the input unit and configured to receive the depth map received by the input unit. An encoder 330 is configured to receive the processed depth map from the video processor 320. An output unit 370 is configured to receive and output the video bitstream generated by the encoder 330. The video encoder 300 also includes a depth decoder 340, a reconstruction processor 350 and an optimizer 360. These components are explained in more detail in connection with a second embodiment of the encoding method described below with reference to Fig. 4.

[0078] 2 and 3, the method of the first embodiment starts with the input unit 310 receiving video data including a texture map and a depth map in step 110. In steps 120 and 130, the video processor 320 processes the depth map to generate a processed depth map. This processing includes non-linear filtering of the depth map in step 120 and downsampling of the filtered depth map in step 130. In step 140, the encoder 330 encodes the processed depth map and the texture map to generate a video bitstream. The generated video bitstream is then output via the output unit 370.

[0079] The source views received at the input 310 may be views captured by the array of cameras 10. However, this is not required and the source views need not be identical to the views captured by the cameras. Some or all of the source views received at the input 310 may be synthesized or otherwise processed source views. The number of source views received at the input 310 may be more or less than the number of views captured by the array of cameras 10.

[0080] In the embodiment of Fig. 2, nonlinear filtering 120 and downsampling 130 are combined in a single step. A "max pooling 2x2" downscaling filter is used. This means that each pixel of the processed depth map takes the maximum pixel value of its 2x2 neighborhood of four pixels in the original input depth map. This choice of nonlinear filtering and downsampling is derived from two insights: The result of downscaling should not include intermediate, i.e. "intermediate" depth levels, which are generated, for example, when a linear filter is used. The inventors have recognized that intermediate depth levels often produce erroneous results after view synthesis at the decoder side. Faint foreground objects represented in the depth map should be preserved; otherwise, for example, relatively faint objects will disappear into the background. Note that the assumption is made (the disparity rule) that foreground, i.e., nearby objects, are coded as high (light) levels, and background, i.e., distant objects, are coded as low (dark) levels. Alternatively, the "min pooling 2x2" downscaler has the same effect when using the z-coordinate coding rule (where z-coordinate increases with distance from the lens).

[0081] This processing operation effectively increases the size of all local foreground objects, thus preserving small, thin objects. However, the decoder preferably knows what operation has been applied, since it preferably shrinks all objects to undo the bias introduced and realign the depth map with the texture.

[0082] According to this embodiment, the memory requirement of the video decoder is reduced. The original pixel rate was 1Y + 0.5CrCb + 1D, where Y = luminance channel, CrCb = chrominance channel, and D = depth channel. According to this example, with 4x (2x2) downsampling, the pixel rate becomes 1Y + 0.5CrCb + 0.25D. As a result, a pixel rate reduction of 30% can be achieved. Most practical video decoders are 4:2:0 and do not include monochrome mode. In this case, a pixel reduction of 37.5% is achieved.

[0083] FIG. 4 is a flow chart showing an encoding method according to a second embodiment. The method starts similarly to the method of FIG. 2, where in step 110 the input section 310 of the video encoder receives a source view. In steps 120a and 130a the video processor 320 processes the depth map according to a number of sets of processing parameters to generate a respective number of processed depth maps, each corresponding to a set of processing parameters. In this embodiment, the system aims to test each of these depth maps to determine which one produces the best quality at the decoder side. Each of the processed depth maps is encoded by the encoder 330 in step 140a. In step 154 ​​the depth decoder 340 decodes each of the encoded depth maps. The decoded depth maps are passed to the reconstruction processor 350. In step 156 the reconstruction processor 350 reconstructs the depth map from the decoded depth maps. Next, in step 158, the optimizer 360 compares each reconstructed depth map with the original depth map of the source view to determine a reconstruction error. The reconstruction error quantifies the difference between the original and reconstructed depth maps. Based on the results of the comparison, the optimizer 360 selects a set of parameters that leads to a reconstructed depth map with the smallest reconstruction error. This set of parameters is selected for use in generating a video bitstream. The output unit 370 outputs the video bitstream corresponding to the selected parameter set.

[0084] It should be noted that the operation of the depth decoder 340 and the reconstruction processor 350 is explained in more detail below with reference to the decoding method (see Figures 5 to 8).

[0085] In effect, the video encoder 300 implements an in-loop decoder to allow prediction of how the bitstream will be decoded at the remote decoder. The video encoder 300 selects the set of parameters that gives the best performance at the remote decoder (in terms of minimizing the reconstruction error for a given target bitrate or pixel rate). The optimization can be performed iteratively, as suggested by the flowchart of FIG. 4, with the parameters of the nonlinear filtering 120a and / or the downsampling 130a being updated at each iteration after the comparison 158 by the optimizer 360. Alternatively, the video decoder can test multiple fixed parameter sets, which can be done serially or in parallel. For example, in a highly parallel implementation, there may be N encoders (and decoders) in the video encoder 300, each of which is configured to test one set of parameters for encoding the depth map. This can increase the number of parameter sets that can be tested in the available time, at the expense of an increase in the complexity and / or size of the encoder 300.

[0086] The parameters tested may include parameters of the nonlinear filtering 120a, parameters of the downsampling 130a, or both. For example, the system may experiment with downsampling by various factors in one or both dimensions. Similarly, the system may experiment with different nonlinear filters. For example, instead of a maximum filter (which assigns each pixel the maximum value in the local neighborhood), other types of ordinal filters may be used. For example, the nonlinear filter may analyze a local neighborhood around a given pixel and assign the pixel the second highest value in the neighborhood. This can provide a similar effect to a maximum filter while helping to avoid sensitivity to a single outlier. The kernel size of the nonlinear filter is another parameter that can be altered.

[0087] Parameters for processing at the video decoder may also be included in the parameter set (described in more detail below). In this way, the video encoder can select a set of parameters for both encoding and decoding that helps optimize quality versus bit rate / pixel rate. The optimization may be performed for a given scene, or for a given video sequence, or more generally, over a training set of diverse scenes and video sequences. Thus, the best set of parameters may vary from sequence to sequence, bit rate to bit rate, and / or allowed pixel rate.

[0088] Parameters that are useful or necessary for a video decoder to properly decode a video bitstream may be embedded in a metadata bitstream associated with the video bitstream, which may be transmitted / transferred to the video decoder together with or separately from the video bitstream.

[0089] Figure 5 is a flow chart of a method for decoding video data according to an embodiment. Figure 6 is a block diagram of a corresponding video decoder 400. The video decoder 400 comprises an input unit 410, a texture decoder 424, a depth decoder 426, a reconstruction processor 450, and an output unit 470. The input unit 410 is coupled to the texture decoder 424 and the depth decoder 426. The reconstruction processor 450 is configured to receive a decoded texture map from the texture decoder 424 and a decoded depth map from the depth decoder 426. The reconstruction processor 450 is configured to provide a reconstructed depth map to the output unit 470.

[0090] The method of Fig. 5 begins at step 210, where an input unit 410 receives a video bitstream and optionally a metadata bitstream. At step 224, a texture decoder 424 decodes a texture map from the video bitstream. At step 226, a depth decoder 426 decodes a depth map from the video bitstream. At steps 230 and 240, a reconstruction processor 450 processes the decoded depth map to generate a reconstructed depth map. This processing includes upsampling 230 and nonlinear filtering 240. The processing, and in particular the nonlinear filtering 240, may also depend on the content of the decoded texture map, as will be explained in more detail below.

[0091] An example of the method of FIG. 5 will now be described in more detail with reference to FIG. 8. In this embodiment, the upsampling 230 includes nearest neighbor upsampling, where each pixel in a block of 2x2 pixels in the upsampled depth map is assigned the value of one of the pixels from the decoded depth map. This "nearest neighbor 2x2" upscaler scales the depth map to its original size. Similar to the max pooling operation in the encoder, this procedure in the decoder avoids the generation of intermediate depth levels. The characteristics of the upscaled depth map compared to the original depth map in the encoder are predictable in advance, and the "max pooling" downscale filter tends to enlarge the area of ​​foreground objects. Thus, some depth pixels in the upsampled depth map are foreground pixels that should rather be background, but there are generally no background depth pixels that should rather be foreground. In other words, after upscaling, objects may be too large, but generally not too small.

[0092] In this embodiment, to undo the bias (foreground objects that have increased in size), the nonlinear filtering 240 of the upscaled depth map includes a color-adaptive conditional erosion filter (steps 242, 244 and 240a in FIG. 8). The erosion part (minimum operator) ensures that the size of the object is reduced, and the color adaptation ensures that the depth edge ends at the correct spatial location, i.e., the transition in the full-scale texture map indicates where the edge should be. Due to the nonlinear way in which the erosion filter works (i.e., pixels are eroded or not eroded), the resulting object edges may be noisy. Adjacent edge pixels may give different results in terms of "eroded or non-eroded" classification for minimally different inputs. Such noise adversely affects the smoothness of the object edges. The inventors have recognized that such smoothness is a key requirement for view synthesis results of sufficient perceptual quality. As a result, the nonlinear filtering 240 also includes a contour smoothness filter (step 250) to smooth the edges in the depth map.

[0093] The nonlinear filtering 240 according to the present embodiment will now be described in more detail. Figure 7 shows a small zoomed-in region of the upsampled depth map representing the filter kernel before the nonlinear filtering 240. The grey squares indicate foreground pixels and the black squares indicate background pixels. The surrounding pixels of the foreground object are labelled as X. These are pixels that may represent the extension / expansion region of the foreground object resulting from the nonlinear filtering in the encoder. In other words, there is uncertainty as to whether the surrounding pixel X is truly a foreground pixel or a background pixel.

[0094] The steps taken to perform adaptive erosion are as follows: 1. Find local foreground edges, i.e., peripheral pixels of the foreground object (marked with X in FIG. 7). This can be done by applying a local threshold to distinguish foreground pixels from background pixels. Peripheral pixels are then identified as foreground pixels that are adjacent to background pixels (in the four connected sense in this example). This is performed by the reconstruction processor 450 in step 242. The depth map can contain packed regions from multiple camera views for efficiency. Edges on the boundaries of such regions are ignored as they do not indicate the edge of an object. 2. For the identified edge pixels (e.g., the central pixels in the 5x5 kernel in Fig. 7), determine the average foreground and average background texture colors within the 5x5 kernel. This is done based on the "confident" pixels only (marked with dots ●), in other words, the calculation of the average foreground and average background textures excludes the uncertain edge pixels X. They also exclude pixels from the neighboring patch regions that may apply to, for example, other camera views. 3. Similarity to the foreground, i.e., foreground confidence

number

[0095] As mentioned above, this process can be noisy and result in jagged edges in the depth map. The steps taken to smooth object edges appearing in the depth map are as follows: 1. Find local foreground edges, i.e., the surrounding pixels of the foreground object (such as the pixels marked with X in Figure 7). 2. For these edge pixels (e.g., the central pixel in Figure 7), count the number of background pixels within a 3x3 kernel around the pixel of interest. 3. Mark all edge pixels where count > threshold. 4. Erode all marked pixels, i.e. take the minimum in a local (eg 3x3) neighborhood. This step is performed by the reconstruction processor 450 in step 250.

[0096] This smoothing tends to convert outlying or protruding foreground pixels into background pixels.

[0097] In the above example, the method used the number of background pixels in a 3x3 kernel to identify whether a given pixel was an outer peripheral pixel that protruded from a foreground object. Other methods can also be used. For example, instead of or in addition to counting the number of pixels, the location of foreground and background pixels in the kernel can be analyzed. If the background pixels are all on one side of the pixel in question, then it is more likely to be a foreground pixel. On the other hand, if the background pixels are all spread out around the pixel in question, then this pixel may be an outlier or noise, and is more likely to be an actual background pixel.

[0098] Pixels within the kernel can be binary classified as foreground or background. A binary flag encodes this for each pixel, with a logical "1" indicating background and a "0" indicating foreground. The neighborhood (i.e., a pixel within the kernel) can then be described by an n-bit binary number, where n is the number of pixels within the kernel that surround the pixel of interest. One exemplary way to construct the binary number is as shown in the table below. [Table 1] In this example, b = b7b6b5b4b3b2b1b0 = 101001012 = 165. (Note that the algorithm described above with reference to Figure 5 corresponds to counting the number of non-zero bits in b, which is 4.)

[0099] Training involves counting how often the pixel of interest (the central pixel of the kernel) is foreground or background for each value of b. Assuming equal costs for false alarms and false negatives, a pixel is determined to be foreground if it is more likely (in the training set) to be a foreground pixel than a background pixel, and vice versa.

[0100] The decoder implementation constructs b and fetches the answer (pixel of interest is foreground or pixel of interest is background) from a look-up table (LUT).

[0101] Approaches that nonlinearly filter the depth map at both the encoder and decoder (e.g., dilation and erosion, respectively, as described above) are counterintuitive, since one would typically expect them to remove information from the depth map. However, the inventors surprisingly find that the smaller depth maps produced by nonlinear downsampling techniques can be encoded (using conventional video codecs) with higher quality for a given bitrate. This quality gain exceeds the loss in reconstruction, and thus the net effect is to improve end-to-end quality while reducing the pixel rate.

[0102] As described above with reference to Figures 3 and 4, it is possible to implement a decoder inside the video encoder to optimize the non-linear filtering and downsampling parameters and thereby reduce the reconstruction error. In this case, the depth decoder 340 of the video encoder 300 is substantially identical to the depth decoder 426 of the video decoder 400, and the reconstruction processor 350 of the video encoder 300 is substantially identical to the reconstruction processor 450 of the video decoder 400. Each of these components performs substantially identical processing.

[0103] As described above, if the parameters of the nonlinear filtering and downsampling in the video encoder are selected to reduce the reconstruction error, the selected parameters may be signaled in the metadata bitstream input to the video decoder. The reconstruction processor 450 can use the parameters signaled in the metadata bitstream to assist in accurate reconstruction of the depth map. The parameters of the reconstruction process can include, but are not limited to, an upsampling factor in one or both dimensions, a kernel size for identifying the surrounding pixels of the foreground object, a kernel size for erosion, a type of nonlinear filtering to be applied (e.g., whether to use a minimum filter or another type of filter), a kernel size for identifying the foreground pixels to smooth, and a kernel size for smoothing.

[0104] An alternative embodiment will now be described with reference to FIG. 9. In this embodiment, instead of hand-coded nonlinear filters for the encoder and decoder, a neural network architecture is used. The neural network is partitioned to model the depth downscale and upscale operations. The network is trained end-to-end to learn both how to optimally downscale and how to optimally upscale. However, during deployment (i.e. for encoding and decoding real sequences), the first part is used before the video encoder and the second part is used after the video decoder. Thus, the first part provides the nonlinear filtering 120 for the encoding method and the second part provides the nonlinear filtering 240 for the decoding method.

[0105] The network parameters (weights) of the second part of the network may be transmitted as metadata along with the bitstream. Note that different sets of neural network parameters may be created corresponding to different coding settings (different downscale factors, different target bitrates, etc.). This means that the upscaling filter for the depth map will behave optimally for a given bitrate of the texture map. This can increase performance since texture coding artifacts change luminance and chroma characteristics, especially at object boundaries, and this change results in different weights of the depth upscaling neural network.

[0106] 9 shows an exemplary architecture of this embodiment, where the neural network is a convolutional neural network (CNN). The symbols in the figure have the following meaning: I = Input 3 channel full resolution texture map TIFF2025060994000004.tif75= Decoded full resolution texture map D = Input 1 channel full resolution depth map D down= Downscaled depth map TIFF2025060994000005.tif512= Downscaled and decoded depth map C k = convolution with k×k kernel P k = Downscale factor k U k = upscaling factor k

[0107] Each vertical black bar in the figure represents a tensor of input or intermediate data, in other words, input data to a layer of the neural network. The dimensions of each tensor are described by a triplet (p, w, h), where w and h are the width and height of the image, respectively, and p is the number of planes or channels of data. Thus, the input texture map has dimensions (3, w, h) (3 planes corresponding to the 3 color channels). The downsampled depth map has dimensions (1, w / 2, h / 2).

[0108] Downscaling P k may include a downscaled average of coefficients k, or a max-pooling (or min-pooling) operation with kernel size k. The downscaled average operation may introduce some intermediate values, but later layers of the neural network can fix this (e.g., based on texture information).

[0109] In the training phase, the decoded depth map TIFF2025060994000006.tif512 is not used. Instead, the uncompressed downscaled depth map D downis used. The reason for this is that the training phase of the neural net requires the computation of derivatives, which is not possible for non-linear video encoder functions. This approximation is likely to be valid in practice, especially for higher qualities (higher bitrates). During the inference phase (i.e., while processing real video data), the uncompressed downscaled depth map D down Obviously, it is not available to the video decoder. Therefore, the decoded depth map TIFF2025060994000007.tif512 is used, and the decoded full resolution texture map Note also that TIFF2025060994000008.tif75 is used in the training and inference phases: it is not the data processed by the neural network, but rather helper information, so there is no need to calculate derivatives.

[0110] The second part of the network (after video decoding) typically contains only a few convolutional layers due to the complexity constraints that may be present on the client device.

[0111] Essential for using deep learning approaches is the availability of training data. In this case, these can be easily obtained. Uncompressed texture images and full resolution depth maps are used at the input side before video encoding. The second part of the network uses the decoded texture and downscaled depth maps (through the first half of the network as input for training) and the error is evaluated against the ground truth full resolution depth map, which was also used as input. Thus, in essence, patches from the high resolution source depth map serve as both input and output to the neural network. Thus, the network has some aspects of both the autoencoder and UNet architectures. However, the proposed architecture is not just a combination of these approaches. For example, the decoded texture map enters the second part of the network as helper data to optimally reconstruct the high resolution depth map.

[0112] In the example shown in FIG. 9, the input to the neural network in the video encoder 300 consists of a texture map I and a depth map D. Downsampling P2 is performed between the other two layers of the neural network. Before downsampling there are three neural network layers and after downsampling there are two layers. The output of the part of the neural network in the video encoder 300 is a downscaled depth map D. down This is encoded by the encoder 320 in step 140.

[0113] The encoded depth map is transferred in the video bitstream to the video decoder 400. It is decoded by the depth decoder 426 in step 226. This results in a downscaled decoded depth map. TIFF2025060994000009.tif512 is produced, which is upsampled (U2) for use in a portion of the neural network in video decoder 400. The other input to this portion of the neural network is the decoded full resolution texture map TIFF2025060994000010.tif75, which is generated by the texture decoder 424. This second part of the neural network has three layers: it produces as output a reconstructed estimate of the depth map D, which is compared with the original depth map D to produce a resulting error e Generates TIFF2025060994000011.tif74.

[0114] As is evident from the above, the neural network processing may be performed by the video processor 320 in the video encoder 300 and by the reconstruction processor 450 in the video decoder 400. In the illustrated example, the nonlinear filtering 120 and the downsampling 130 are performed in an integrated manner by part of the neural network in the video encoder 300. In the video decoder 400, the upsampling 230 is performed separately, before the nonlinear filtering 240 performed by the neural network.

[0115] It will be understood that the configuration of neural network layers shown in Figure 9 is non-limiting and may be modified in other embodiments. In this example, the network produces a 2x2 downsampled depth map. Of course, different scaling factors may be used.

[0116] In some embodiments described above, reference was made to maximum filtering, maximum pooling, dilation or similar operations in the encoder. It will be appreciated that these embodiments assume that depth is coded as 1 / d (or other similar inverse proportional relationship), where d is the distance from the camera. With this assumption, high values ​​in the depth map indicate foreground objects and low values ​​in the depth map indicate background. Thus, by applying a maximum or dilation type operation, the method tends to expand the foreground objects. The corresponding inverse process in the decoder may be to apply a minimum or erosion type operation.

[0117] Of course, in other embodiments, depth may be coded as d or log d (or another variable that has a direct correlation with d). That is, foreground objects are represented by low values ​​of d and background is represented by high values ​​of d. In such embodiments, minimum filtering, minimum pooling, erosion, or similar operations may be performed at the encoder. Again, this tends to expand the foreground objects, which is the goal. The corresponding inverse process may be to apply a maximum or dilation type operation at the decoder.

[0118] The encoding and decoding methods of Figures 2, 4, 5, 8 and 9, and the encoders and decoders of Figures 3 and 6 may be implemented in hardware or software, or a mixture of both (e.g., as firmware executed on a hardware device). To the extent that an embodiment is implemented partially or entirely in software, the functional steps illustrated in the process flow charts may be performed by appropriately programmed physical computing devices, such as one or more central processing units (CPUs), graphics processing units (GPUs), or neural network accelerators (NNAs). Each process, and its individual component steps illustrated in the flow charts, may be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code configured to cause one or more physical computing devices to perform an encoding or decoding method as described above when the program is executed on one or more physical computing devices.

[0119] The storage medium may include volatile and non-volatile computer memories such as RAM, PROM, EPROM, and EEPROM. Various storage media may be installed in a mobile computing device or may be transportable such that one or more programs stored on the storage medium may be read by a processor.

[0120] The metadata according to an embodiment may be stored on a storage medium. The bitstream according to an embodiment may be stored on the same storage medium or on a different storage medium. The metadata may be embedded in the bitstream, but this is not required. Similarly, the metadata and / or the bitstream (with the metadata in the bitstream or separate therefrom) may be transmitted as a signal modulated onto an electromagnetic carrier wave. The signal may be defined according to a standard for digital communication. The carrier wave may be an optical carrier wave, a radio frequency wave, a millimeter wave, a short-range communication wave. It may be wired or wireless.

[0121] To the extent that an embodiment is implemented partially or wholly in hardware, the blocks shown in the block diagrams of Figures 3 and 6 may be separate physical components, or logical subdivisions of a single physical component, or may all be implemented in an integrated manner in one physical component. The functionality of a block shown in the figures may be split into multiple components in an implementation, or the functionality of multiple blocks shown in the figures may be combined into a single component in an implementation. For example, although Figure 6 shows the texture decoder 424 and the depth decoder 46 as separate components, their functionality may be provided by a single integrated decoder component.

[0122] In general, examples of methods for encoding and decoding data, computer programs for implementing these methods, and video encoders and decoders are illustrated by the following embodiments.

[0123] Embodiments: 1. A method for encoding video data including one or more source views, each source view including a texture map and a depth map, the method comprising: Receiving video data (110); processing a depth map of at least one source view to generate a processed depth map, Nonlinear filtering (120), and A step including downsampling (130); and encoding (140) the processed depth map and texture map of the at least one source view to generate a video bitstream. 2. The method of embodiment 1, wherein the nonlinear filtering includes expanding a region of at least one foreground object in the depth map. 3. The method of embodiment 1 or embodiment 2, wherein the non-linear filtering comprises applying a filter designed using a machine learning algorithm. 4. A method according to any one of the preceding embodiments, wherein the nonlinear filtering is performed by a neural network including multiple layers, and the downsampling is performed between two of the layers. 5. The method includes processing (120a, 130a) the depth map according to a plurality of sets of processing parameters to generate a respective plurality of processed depth maps, the method further comprising: selecting a set of processing parameters that reduces a reconstruction error of a reconstructed depth map after each processed depth map is encoded and decoded; 13. A method as in any one of the preceding embodiments, comprising generating a metadata bitstream identifying the selected parameter set. 6. A method for decoding video data including one or more source views, the method comprising: receiving a video bitstream having an encoded depth map and an encoded texture map for at least one source view (210); decoding (226) the encoded depth map to generate a decoded depth map; decoding (224) the encoded texture map to generate a decoded texture map; processing the decoded depth map to generate a reconstructed depth map, the processing comprising: Upsampling (230) and and non-linear filtering (240). 7. The method of embodiment 6, further comprising the step of detecting that the decoded depth map has a lower resolution than the decoded texture map prior to the step of processing the decoded depth map to generate the reconstructed depth map. 8. The method of embodiment 6 or embodiment 7, wherein the nonlinear filtering includes reducing a region of at least one foreground object in the depth map. 9. A method according to any one of embodiments 6 to 8, wherein the processing of the decoded depth map is based at least in part on the decoded texture map. 10. The method of any one of embodiments 6 to 9, comprising: upsampling the decoded depth map (230); identifying peripheral pixels of at least one foreground object in the upsampled depth map (242); determining whether the surrounding pixel is more similar to the foreground object or to the background based on the decoded texture map (244); and applying the non-linear filtering (240a) only to the surrounding pixels determined to be more similar to the background. 11. The method according to any one of the embodiments, wherein the non-linear filtering includes smoothing (250) edges of at least one foreground object. 12. A method according to any one of embodiments 6 to 11, further comprising a step of receiving a metadata bitstream that identifies a set of parameters associated with the video bitstream, the method further including processing the decoded depth map according to the identified set of parameters. 13. A computer program comprising computer code which, when executed on a processing system, causes the processing system to carry out any one of the embodiments of embodiments 1 to 12. 14. A video encoder (300) configured to encode video data including one or more source views, each source view including a texture map and a depth map, the video encoder comprising: an input (310) configured to receive video data; a video processor (320) configured to process a depth map of at least one source view to generate a processed depth map, said processing comprising: Nonlinear filtering (120), and a video processor including downsampling (130); an encoder (330) configured to encode the texture map and the processed depth map of at least one source view to generate a video bitstream; An output (360) configured to output a video bitstream. 15. A video decoder (400) configured to decode video data including one or more source views, the video decoder comprising: a bitstream input unit (410) configured to receive a video bitstream, the video bitstream including an encoded depth map and an encoded texture map for at least one source view; a first decoder (426) configured to decode the encoded depth map from the video bitstream to generate a decoded depth map; a second decoder (424) configured to decode the encoded texture map from the video bitstream to generate a decoded texture map; a reconstruction processor (450) configured to process the decoded depth map to generate a reconstructed depth map, said processing comprising: Upsampling(230) a reconstruction processor including nonlinear filtering (240); An output (470) configured to output the reconstructed depth map.

[0124] Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more of the blocks may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0125] More particularly, the present invention is defined by the following claims.

[0126] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where a computer program is described above, the computer program can be stored or distributed on a suitable medium, for example an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, for example via the Internet or other wired or wireless telecommunication systems. When the term "adapted for" is used in the claims or the description, the term "adapted for" is meant to be equivalent to the term "configured for". Any reference signs in the claims should not be interpreted as limiting the scope.

Claims

1. 1. A method for encoding video data having one or more source views, each source view including a texture map and a depth map, comprising: receiving the video data; - processing the depth map of at least one source view to generate a processed depth map, the processing comprising non-linear filtering and downsampling; encoding the processed depth map and the texture map of the at least one source view to generate a video bitstream; The method comprising:

2. The method of claim 1 , wherein the nonlinear filtering comprises expanding a region of at least one foreground object in the depth map.

3. The method of claim 1 or 2, wherein the non-linear filtering comprises applying a filter designed using a machine learning algorithm.

4. The method according to claim 1 , wherein the non-linear filtering is performed by a neural network having multiple layers and the downsampling is performed between two of the layers.

5. processing the depth map according to a plurality of sets of processing parameters to generate a respective plurality of processed depth maps; The method further comprises: selecting the set of processing parameters that reduces a reconstruction error of a reconstructed depth map after each of the processed depth maps is encoded and decoded; generating a metadata bitstream identifying the selected set of parameters; The method according to any one of claims 1 to 4, comprising the steps of:

6. 1. A method for decoding video data having one or more source views, comprising the steps of: receiving a video bitstream having an encoded depth map and an encoded texture map of at least one source view; decoding the encoded depth map to generate a decoded depth map; decoding the encoded texture map to generate a decoded texture map; processing the decoded depth map to generate a reconstructed depth map, the processing including upsampling and non-linear filtering. method.

7. 7. The method of claim 6, further comprising detecting that the decoded depth map has a lower resolution than the decoded texture map prior to processing the decoded depth map to generate the reconstructed depth map.

8. The method of claim 6 or 7, wherein the non-linear filtering comprises reducing a region of at least one foreground object in the depth map.

9. The method of claim 6 , wherein the processing of the decoded depth map is based at least in part on the decoded texture map.

10. upsampling the decoded depth map; identifying peripheral pixels of at least one foreground object in the upsampled depth map; determining whether the surrounding pixel is more similar to the foreground object or to a background based on the decoded texture map; and applying non-linear filtering to only those surrounding pixels determined to be more similar to the background.

11. The method according to claim 6 , wherein the non-linear filtering comprises smoothing of edges of at least one foreground object.

12. receiving a metadata bitstream associated with the video bitstream, the metadata bitstream identifying a set of parameters; The method according to any one of claims 6 to 11, further comprising processing the decoded depth map according to the determined set of parameters.

13. A computer program comprising computer code when executed by a processing system to cause the processing system to carry out a method according to any one of claims 1 to 12.

14. 1. A video encoder configured to encode video data having one or more source views, each source view including a texture map and a depth map, the video encoder comprising: an input configured to receive the video data; a video processor configured to process the depth map of at least one source view to generate a processed depth map, the processing comprising non-linear filtering and downsampling; an encoder configured to encode the texture map and the processed depth map of the at least one source view to generate a video bitstream; and an output unit configured to output the video bitstream. Video encoder.

15. The video encoder of claim 14 , wherein the non-linear filtering comprises expanding a region of at least one foreground object in the depth map.

16. 1. A video decoder configured to decode video data having one or more source views, comprising: a bitstream input configured to receive a video bitstream, the video bitstream including an encoded depth map and an encoded texture map for at least one source view; a first decoder configured to decode the encoded depth map from the video bitstream to generate a decoded depth map; a second decoder configured to decode the encoded texture map from the video bitstream to generate a decoded texture map; a reconstruction processor configured to process the decoded depth map to generate a reconstructed depth map, the processing comprising upsampling and non-linear filtering; an output unit configured to output the reconstructed depth map; 1. A video decoder comprising:

Images