Encoding scheme for depth data

Abstract

Methods of encoding and decoding depth data are disclosed. In the encoding method, both depth values and occupancy data are encoded into a depth map. The method adjusts how the depth values and occupancy data are converted into map values in the depth map. For example, it can adaptively select a threshold, above or below which all values represent unoccupied pixels. By adjusting how depth and occupancy are encoded based on analysis of the depth values, the method can enable more efficient encoding and transmission of depth data and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used by a corresponding decoding method to decode the map values. An encoder and decoder are also provided for depth data, as well as a corresponding bitstream that includes a depth map and its associated metadata.

Description

TECHNICAL FIELD

[0001] The present application relates to the encoding of depth data. It particularly relates to methods and apparatus for encoding and decoding immersive video. BACKGROUND

[0002] Immersive video, also known as six degrees of freedom (6DoF) video, is video of a three-dimensional (3D) scene that allows reconstruction of scene views for viewpoints that vary in position and orientation. It represents an evolution of three degrees of freedom (3DoF) video, which allows reconstruction of views for viewpoints with arbitrary orientation but only at a fixed point in space. In 3DoF, the degrees of freedom are angular - i.e. pitch, roll and yaw. 3DoF video supports head rotation - in other words, a user consuming the video content can look in any direction in the scene but cannot move to a different place in the scene. 6DoF video supports head rotation and, in addition, the selection of a position in the scene from which the scene is viewed.

[0003] To generate 6DoF video, multiple cameras are required to record the scene. Each camera generates image data (commonly referred to as texture data in this context) and corresponding depth data. For each pixel, the depth data represents the depth at which the corresponding image pixel data is observed. Each of the multiple cameras provides a respective view of the scene.

[0004] To reduce redundancy between the views, it has been proposed to prune the views and pack them into "texture atlases" for each frame of the video stream. This approach attempts to reduce or eliminate overlapping portions among the multiple views and thereby improve efficiency. The non-overlapping portions of the different views that remain after pruning can be referred to as "patches". An example of this approach is described in Alvaro Collet et al., "High-quality streamable free-viewpoint video", ACM Trans. Graphics (SIGGRAPH), 34(4), 2015. SUMMARY

[0005] It would be desirable to efficiently encode depth data. Related to depth data is occupancy data, which indicates whether a given pixel in a given view is occupied by a patch. There is valid depth data for a patch. There is no valid depth data for unoccupied pixels. One approach would be to encode the occupancy map separately from the depth data, but this would require transmission of an additional data structure. Another approach is to encode occupancy information into the depth data - for example, by reserving a particular depth value to signal that a pixel is unoccupied. However, the depth data can be compressed for transmission. Both the compression and the subsequent transmission can introduce errors in the decoded depth data. If the occupancy data is embedded with the depth data, compression or transmission errors can also corrupt the occupancy data, which can ultimately result in noticeable or disturbing visual artifacts in the decoded and rendered immersive video. It would be desirable to reduce or avoid such artifacts when efficiently encoding depth and occupancy data.

[0006] The invention is defined by the claims.

[0007] According to examples in accordance with aspects of the invention, there is provided a method of encoding depth data, the method comprising:

[0008] receiving source data comprising at least depth values;

[0009] defining a depth map, the depth map comprising an array of map values, each map value taking one of a plurality of levels;

[0010] analyzing the depth values to determine how to encode the depth values into the plurality of levels;

[0011] defining a first subset of the plurality of levels based on a result of the analysis, the first subset being for representing depth data of an occupied portion of the depth map;

[0012] defining a second subset of the plurality of levels based on the result of the analysis, the second subset being for representing an unoccupied portion of the depth map;

[0013] for the unoccupied portion of the depth map, assigning one or more map values in accordance with the defined second subset of the plurality of levels;

[0014] for the occupied portion of the depth map, converting the depth values to map values in the defined first subset of the plurality of levels, thereby encoding the depth values in the depth map; and

[0015] generating a bitstream, the bitstream comprising the depth map, and metadata comprising a definition of the first subset and the second subset.

[0016] This method allows for the encoding of occupancy in the depth map. The inventors have recognized that encoding occupancy and depth together works better if the encoding can be adapted to the content of the depth map. If the first and second subsets are fixed, the encoding can be suboptimal. By analyzing the depth values ​​and selecting the first and second subsets based on the analysis results, this method can more effectively utilize the bits available for the depth map.

[0017] The source data can be source video data. The source data may also include occupancy data indicating the occupied and / or unoccupied portions of the depth map.

[0018] The definitions of the first and second subsets depend on the content of the depth data. In other words, the selection of subsets is adapted to the content of the source data.

[0019] The method may also include compressing the depth map using a video compression algorithm, optionally using a standardized video compression algorithm such as High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2. The bitstream may include the compressed depth map.

[0020] Defining a subset may include selecting at least one threshold level.

[0021] Defining the first subset and the second subset may include selecting a first threshold level among multiple levels, wherein one of the subsets includes all levels greater than the first threshold level.

[0022] Optionally, other subsets include all levels less than the first threshold level. Metadata may include a definition of the first threshold level. This may be explicit (i.e., metadata may include the first threshold level) or implicit (e.g., metadata may define a transformation function implicitly involving the first threshold).

[0023] The first threshold level can be chosen as a power of 2.

[0024] In other words, the first threshold level was chosen to be 2. n , where n is a non-negative integer. When graph values ​​are encoded as binary numbers, this allows occupancy to be determined by examining a subset of the bits in the binary graph value.

[0025] The method may further include defining a third subset of the plurality of levels as a protection level that will not be used in the depth map, wherein the metadata also includes a definition of the third subset.

[0026] The third subset is different from the first and second subsets. That is, the level of the third subset is neither in the first subset nor in the second subset.

[0027] The introduction of a protection level can also enhance the robustness to errors introduced by compression or transmission. The protection level can comprise a range of levels between the first subset and the second subset. The protection level is not used to represent the depth data, and is not used to represent the unoccupied part of the depth map. At the decoder, any instance of a map value equal to one of the protection levels can be detected as an error and optionally corrected.

[0028] The third subset can be defined by a second threshold level. For example, the third subset can be defined to comprise levels between the first threshold and the second threshold. The metadata can comprise a definition of the second threshold level. The definition can be direct / explicit (e.g. the metadata can comprise the second threshold level) or indirect / implicit.

[0029] Analyzing the depth data can comprise determining a dynamic range of the depth data.

[0030] The method can further comprise measuring or predicting the extent of errors in the decoded depth data that would be caused by encoding the depth values in the depth map in a given way.

[0031] The depth data can comprise normalized disparity values. The normalized disparity values occupy the range [0, 1], where 0 represents infinite depth and 1 represents the minimum depth. The use of normalized disparity values can facilitate better allocation of bits to depth values. For example, suppose that depths are measured in meters. The difference between 1 m and 2 m is visually more significant than the difference between 10 m and 11 m (even though the absolute depth difference of 1 m is the same in both cases). Normalized disparity helps to capture this relative significance: in the first case, the difference in normalized disparity is (1 / 1 - 1 / 2) = 0.5, while in the second case, the difference in normalized disparity is (1 / 10 - 1 / 11) = 0.009.

[0032] Converting the depth values to map values can comprise using a piecewise linear function, and the metadata can further comprise a definition of the piecewise linear function.

[0033] The source data can be video data comprising a plurality of source views, each source view comprising texture values and depth values. In other words, the depth data encoding method as outlined above can be applied in a method of encoding immersive video.

[0034] There is also provided a method of decoding depth data, the method comprising:

[0035] receiving a bitstream comprising a depth map and associated metadata,

[0036] the depth map comprising an array of map values,

[0037] each map value taking one of a plurality of levels,

[0038] The metadata defines a first subset of the plurality of levels, the first subset representing depth data for an occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing an unoccupied portion of the depth map;

[0039] decoding the depth map includes converting map values in the first subset of values to depth values; and

[0040] outputting the depth values.

[0041] Decoding the depth map optionally further includes generating an occupancy map by identifying map values in the second subset of values.

[0042] The depth map in the bitstream can have been compressed using a video compression algorithm, optionally using a standardized video compression algorithm. The method can include decompressing the depth map according to the video compression algorithm prior to decoding the depth map.

[0043] At least one of the first subset and the second subset can be defined by a first threshold level intermediate the plurality of levels, wherein one of the subsets includes all levels greater than the first threshold level. The other subset can include all levels less than the first threshold level.

[0044] The metadata can further include a definition of a piecewise linear function, and converting the map values to depth values can include converting using the piecewise linear function.

[0045] The metadata can further include a definition of a third subset of the plurality of levels, the third subset being guard levels not used in the depth map, the method further including detecting any map values in the third subset of values prior to decoding the depth map.

[0046] Any values in the third subset in the received depth map are erroneous. They can have been introduced by compression or transmission errors. The method can include skipping or correcting these values when decoding the depth map. In some embodiments, map values close to values in the first subset (i.e., close to valid values) can be corrected by changing them to the closest valid value.

[0047] The depth values can be depth data for a video data including a plurality of source views, and the method can further include reconstructing at least one of the plurality of source views.

[0048] The metadata can comprise a negative normalised disparity value and a positive normalised disparity value. This can provide a way of implicitly defining the first and second subsets - particularly when a linear (or piecewise linear) conversion function between depth values and map values is used. The negative normalised disparity value can define the x-axis intercept of the conversion function (or segment of the conversion function). This negative normalised disparity value can be assigned to a horizontal 0 in the map values. The positive normalised disparity value can define the maximum normalised disparity (minimum depth) to be encoded (or the maximum normalised disparity that is to be encoded using that segment of the conversion function). This positive normalised disparity value is assigned to a specified horizontal (e.g. the maximum horizontal in the case of a single linear conversion function). This implicitly defines a first subset of horizontals for representing depth data (those corresponding to positive depth values), and implicitly defines a second subset of horizontals for representing unoccupied portions of the depth map (those corresponding to "negative" depth values). Since true normalised disparity values cannot be negative, all depth values will be converted to horizontals in the first subset.

[0049] A computer program comprising computer code for causing a processing system to implement a method as outlined above is also disclosed. The computer program can be stored on a computer-readable storage medium. This can be a non-transitory storage medium.

[0050] An encoder for depth data is also provided, configured to encode depth values into a depth map comprising an array of map values, each map value taking one of a plurality of horizontals, the encoder comprising:

[0051] an input configured to receive source data comprising at least depth values;

[0052] an analyser configured to:

[0053] analyse the depth values to determine how to encode the depth values into the plurality of horizontals,

[0054] define a first subset of the plurality of horizontals based on the result of the analysis, the first subset for representing depth data of an occupied portion of the depth map;

[0055] define a second subset of the plurality of horizontals based on the result of the analysis, the second subset for representing an unoccupied portion of the depth map; and

[0056] a depth value encoder configured to:

[0057] assign one or more map values according to the defined second subset of the plurality of horizontals for the unoccupied portion of the depth map;

[0058] for the occupied part of the depth map, converting the depth values to map values in a defined first subset of the plurality of levels, thereby encoding the depth values in the depth map; and

[0059] generating a bitstream comprising the depth map and comprising metadata defining the first subset and the second subset.

[0060] There is also provided a decoder for depth data, the decoder comprising:

[0061] an input configured to receive a bitstream comprising a depth map and associated metadata,

[0062] the depth map comprising an array of map values,

[0063] each map value taking one of a plurality of levels,

[0064] the metadata defining a first subset of the plurality of levels, the first subset representing depth data for an occupied part of the depth map, and the metadata defining a second subset of the plurality of levels, the second subset representing an unoccupied part of the depth map;

[0065] a depth value decoder configured to decode the depth map, including converting map values in the first subset of values to depth values; and

[0066] an output configured to output the converted depth values.

[0067] According to yet another aspect, there is provided a bitstream comprising a depth map and associated metadata,

[0068] the depth map comprising an array of map values, each map value taking one of a plurality of levels,

[0069] the metadata defining a first subset of the plurality of levels, the first subset representing depth data for an occupied part of the depth map, and the metadata defining a second subset of the plurality of levels, the second subset representing an unoccupied part of the depth map.

[0070] The bitstream can be encoded and decoded using the methods as outlined above. It can be embodied on a computer readable medium or as a signal modulated onto an electromagnetic carrier.

[0071] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0072] For a better understanding of the present application, and to show more clearly how it can be carried into effect, reference will now be made, by way of example, to the accompanying drawings in which:

[0073] Figure 1 is a flowchart of a method of encoding depth data according to a first embodiment of the application;

[0074] Figure 2 is a block diagram of an encoder of depth data according to an embodiment, the encoder being configured to perform Figure 1 the illustrated method;

[0075] Figure 3 is a flowchart of a method of decoding depth data according to a second embodiment of the application;

[0076] Figure 4 is a block diagram of a decoder of depth data according to an embodiment, the decoder being configured to perform Figure 3 the illustrated method; and

[0077] Figure 5 shows an exemplary linear conversion function for converting between depth values and map values;

[0078] Figure 6 shows another exemplary linear conversion function;

[0079] Figure 7 shows a linear conversion function with a protection level;

[0080] Figure 8 illustrates an alternative way for defining a linear conversion function;

[0081] Figure 9 shows a piecewise linear conversion function; and

[0082] Figure 10 shows a piecewise linear conversion function with a protection level. DETAILED DESCRIPTION

[0083] The present application will be described with reference to the accompanying drawings.

[0084] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of apparatuses, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the present application. These and other features, aspects, and advantages of the apparatuses, systems and methods of the present application will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the drawings are only schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings and that these are used to indicate the same or similar components from drawing to drawing.

[0085] Methods of encoding and decoding depth data are disclosed. In the encoding method, both depth values and occupancy data are encoded into a depth map. The method adjusts how the depth values and occupancy data are converted into map values in the depth map. For example, it can adaptively select a threshold, above or below which all values represent unoccupied pixels. By adjusting how depth and occupancy are encoded based on an analysis of the depth values, the method can enable more efficient encoding and transmission of depth data and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used by a corresponding decoding method to decode the map values. An encoder and decoder are also provided for depth data, as well as a corresponding bitstream, including a depth map and its associated metadata.

[0086] It would be desirable to compress depth data using known image and video compression algorithms. It would be particularly desirable to be able to compress depth data using standardized algorithms. Suitable hardware and software for encoding and decoding according to standardized codecs is widely available, and is often highly optimized in terms of both speed, quality and power consumption. However, most video compression is lossy in order to achieve a bit rate that is practical for transmission and storage. Thus, it is not generally guaranteed that a depth map compressed using video compression techniques will be perfectly reconstructed at the decoder. Errors will be introduced by the compression and potentially also by both transmission and / or storage of the bitstream.

[0087] One basic method of combining encoding of depth values and occupancy data in a depth map (not according to the invention) would be to set a fixed threshold to distinguish between unoccupied pixels and valid depth values. For example, it can be desirable to encode depth data using HEVC Main 10 Level 5.2, which means a maximum bit depth of 10. This means that 1024 levels (from 0 to 1023) are available for encoding depth and occupancy data. The encoding scheme for depth data can define that all levels from 0 to 63 indicate unoccupied pixels. Only levels 64 to 1023 are used for encoding depth values. This means that more than 6% of the available range is used to encode occupancy. This can work well for some content, but is inefficient for other content. It is difficult to choose a single fixed threshold that works well for all types of content.

[0088] Figure 1 An encoding method according to a first embodiment of the invention is illustrated. Figure 2 is an encoder for performing Figure 1 the method of FIG. 1.

[0089] The encoder 300 comprises an input section 310; an analyzer 320; a depth value encoder 330; and an output section 340. In step 110, the input section 310 receives source data comprising depth values. In the present embodiment, the source data is immersive video data comprising a plurality of source views. Each source view comprises texture values and depth values. The encoding of the texture values is outside the scope of the present application and is not discussed further here.

[0090] In step 120, the depth value encoder 330 defines a depth map comprising an array of map values. Each map value takes one of a plurality of levels. For example, if the maximum bit depth is 10, there will be 1024 levels.

[0091] The input section 310 is coupled to the analyzer 320. In step 130, the analyzer analyzes the depth values to determine how best to encode the depth values into the plurality of levels. In the present embodiment, map values below a threshold value will be used to represent unoccupied pixels, and map values above the threshold value will be used to encode the depth values. Thus, the task of the analyzer is to select a threshold value based on the analysis of the depth values (step 140). Further details on how to select the threshold value will be discussed later below. For now, note that the threshold value (T) can be chosen to be a power of two (T = 2 n ). This can be advantageous because it can allow a simple check at the decoder to determine whether a given map value is above or below the threshold value. Rather than comparing the map value to a particular threshold value, the decoder can simply check the most significant bits (MSBs) of the map value. For example, if the threshold value T = 256 = 2 8 , then the decoder can check the two most significant bits of the 10-bit representation. If these two bits are 0, then the value is below the threshold; otherwise, if either bit is 1, then the value is above the threshold.

[0092] In step 150, the depth value encoder 330 populates the depth map. For unoccupied pixels, the depth map is populated with one or more map values below the selected threshold value. For each occupied pixel, the depth value encoder 330 converts the depth value to a corresponding map value that lies above the threshold value.

[0093] The depth value encoder 330 provides the populated depth map containing the encoded map values to the output section 340. At the same time, the analyzer 320 provides metadata to the output section 340. The metadata comprises information defining how the depth values were encoded. In particular, the metadata comprises information about the threshold value that was selected. The metadata can also comprise information about the mapping of the depth values to map values in the range above the threshold value. However, this can not be necessary in some embodiments because the mapping can be explicitly defined in the encoding scheme. For example, all depth values can be normalized disparity values in the range [0, 1], and the mapping can be a linear mapping that maps values above the threshold value.

[0094] The output section 340 generates and outputs a bitstream comprising at least the depth map. It also outputs metadata, either as part of the same bitstream or separately from the bitstream.

[0095] Figure 3 is a flowchart illustrating a method of decoding an encoded depth map according to a second embodiment of the application. Figure 4 is a schematic block diagram of a decoder for performing the method of Figure 3

[0096] The decoder 400 comprises an input section 410; a depth value decoder 420; and an output section 430. Optionally, it can also comprise a renderer 440.

[0097] In step 210, the input section 410 receives a bitstream comprising a depth map. It also receives metadata describing the bitstream. The metadata can be embedded in the bitstream or can be separate. The depth map in this example is a depth map created according to the method of Figure 1 described above. Note that the depth map input to the decoder 400 will typically be a version of the depth map output by the encoder 300 that has subsequently been subject to lossy video compression (and possibly error-prone communication over a transmission channel).

[0098] In step 220, the depth value decoder 420 decodes the depth map. This involves identifying map values above a threshold and converting them back to depth values. As mentioned above, the threshold is included in the metadata. An appropriate conversion function can be pre-agreed between the encoder and the decoder (e.g. defined as part of a standardised encoding scheme). Alternatively, if not pre-defined / agreed, the conversion function can be embedded in the metadata and the decoder can extract it from the metadata.

[0099] The depth value decoder 420 provides the decoded depth values to the output section 430. The output section 430 outputs the depth values (step 230). The depth value decoder can also output an occupancy map indicating the pixels in the depth map for which the map values are below the threshold.

[0100] If the decoder 400 comprises the optional renderer 440, the depth value decoder 420 can provide the decoded depth values to the renderer, which reconstructs one or more views from the depth data. In this case, the renderer 440 can provide the reconstructed views to the output section 430, and the output section 430 can output the reconstructed views (e.g. to a frame buffer).

[0101] There are various ways in which map values can be dynamically allocated for encoding depth (and, correspondingly, occupancy) data. Some of these ways will now be discussed in more detail - together with the corresponding analysis to be performed by the analyser 320.

[0102] ​In some embodiments, analyzing the depth values includes determining the dynamic range of the depth values. If the dynamic range is small (that is, if the depth values are all around the same value and the differences between them are not significant, then a small number of bits can be used to encode the occupancy. For example, if all the cameras are close enough to the object and the dynamic range of the depth map is not critical, then one bit can be used to encode the occupancy map, i.e., for a 10-bit depth map, the threshold level would be T = 512 = 2 9 This still leaves 512 levels for encoding the depth data, which can be sufficient in close-up situations.

[0103] When a patch, view, frame, or video is determined to have full occupancy, then the threshold can be set to 0, which indicates that all pixels are occupied. This maximizes the number of levels available for encoding the depth data.

[0104] In some embodiments, the method can include measuring or predicting the extent of visible error in the decoded depth data that would result from encoding the depth values in a particular manner. For example, the analyzer 320 can study the camera parameters associated with the source view in order to determine how to encode the depth values. If two cameras have a wide angular separation (e.g., 90°), then a depth error in one view will be easily noticeable as shifted left or right in the other view. In these cases, it would be advantageous to encode the depth values as accurately as possible. On the other hand, if two cameras have a small angular separation (e.g., 5°), then a depth value error is less likely to be perceived.

[0105] In some embodiments, the analysis can include encoding the depth values, compressing the depth values, decompressing and decoding the depth values, and synthesizing a test view from the decoded depth values. The synthesized test view can be compared to a reference view derived from the original source data to produce an error / difference image. This can be repeated for different configurations of the first and second subsets of levels. The configuration that results in the least error / difference can be selected for encoding the depth values.

[0106] The depth values can be stored as normalized disparity (1 / Z), where the near depth and far depth correspond to the highest and lowest depth levels, respectively. One model assumes that the selected threshold level corresponds to the far depth, and 1023 corresponds to the near depth (for 10-bit data).

[0107] When specifying the depth range, there are multiple ways to specify the occupancy encoding.

[0108] Figure 5 An example of encoding depth values as map values through a linear function is illustrated. For a given depth value on the x-axis, the function defines a corresponding map value on the y-axis. It is assumed that the depth values are provided as normalized disparity; thus, the depth values range from 1 / Z maxto 1 / Z min . The maximum value of the normalized disparity is 1. In the example of Figure 5 , it is assumed that the maximum depth is infinite (Z = ∞); thus, the minimum value of the normalized disparity is 0. The y-intercept of the line gives the threshold level T. Values of the map below T will be used to represent unoccupied pixels. The maximum depth value (1 / Z min ) will be converted to the maximum map value (1023, in the case of 10-bit encoding). The map values for depth values between 0 and 1 / Z min can be obtained by linear interpolation. The conversion function can be defined in the metadata in various ways. For example, the metadata can include the threshold T and the minimum depth value Z min . The maximum and minimum map values can be implicitly known from the bit depth (1023 and 0, respectively, for 10-bit); thus, it can not be necessary to explicitly encode these in the metadata.

[0109] Figure 6 Another example linear conversion function is shown. Here, the maximum depth Z max is less than infinite; thus, there is a non-zero 1 / Z max . The linear conversion function runs from the coordinate (1 / Z max , T) to the coordinate (1 / Z min , 1023). Again, map values below T are used to represent unoccupied regions. This conversion function can be defined in the metadata by three parameters: the threshold T; the minimum depth value Z min ; and the maximum depth value Z max .

[0110] Potential problems can arise if compression or transmission errors are introduced in normalized disparity values very close to 1 / Z max . The map value can exceed the threshold T due to the error, which means that a pixel at a far depth is incorrectly replaced with an unoccupied pixel. Figure 7 A conversion function is shown that attempts to prevent this by introducing a guard level between the level used to encode depth data and the level used to indicate unoccupied pixels. The lowest map value used to encode depth values is given by a first threshold T. But only map values below a second threshold T G are used to represent unoccupied pixels. This can also be seen as a guard band introduced in the depth values, between 1 / Z max and 1 / Z G . It can be described in the metadata in any way. In other words, an additional threshold is defined in the metadata, which can be a threshold on the map values or a threshold on the depth values.

[0111] Figure 8 Another way of defining a linear conversion function is illustrated. Here, the metadata includes the positive depth value 1 / Zmin and negative depth values -d. Negative depth values have no real physical meaning (as normalized disparity is always non-negative). They are used as implicit definition of a threshold T (at the y-axis intercept) below which map values are used to indicate unoccupied pixels. Figure 8 The example of FIG. 7 does not protect levels, but these levels can be introduced by specifying additional thresholds, similar to the approach taken in Figure 7 .

[0112] Figure 9 A piecewise linear conversion function is shown. The piecewise linear function has three segments, which can be defined in the metadata by the coordinates of their endpoints. The map value defining the y-axis coordinate of the first endpoint implicitly defines the threshold T. In other words, the (x, y) coordinates of the first endpoint are (1 / Z max , T). The first linear segment has a relatively shallow slope, which means that relatively few map values are assigned to depth values in this range. The middle linear segment has a steeper slope, which indicates that relatively more map values are assigned to depth values in this range. The third linear segment again has a relatively shallow slope, to which fewer levels are assigned. Such a profile can be appropriate, for example, when most depth values are clustered in the middle distance. In this case, errors in pixels very close to the camera or very far away can be less disruptive - that is, more easily accepted by the observer.

[0113] Figure 10 Another piecewise linear conversion function is shown, with protected levels. The protected levels can be understood (and potentially encoded in the metadata) as vertical steps in the piecewise linear function. The protected levels are map values in the range from T G to T. Unoccupied pixels are indicated only by map values smaller than T G .

[0114] Figures 5-10 The examples in FIGS. 6-8 are not mutually exclusive, and conversion functions can be provided that combine attributes of several of these examples. For example, one or more piecewise linear function segments (like one in FIG. 7) can be defined by nominally negative depth values, as shown in FIG. 8. A third subset of levels composed of protected levels can be introduced in any of the examples. Figure 9 Figure 8

[0115] ​​In each of the above examples, the second horizontal subset (indicating unoccupied pixels) is separated from the first horizontal subset (indicating valid depth values) by one or two thresholds, with the first horizontal subset being higher than the second horizontal subset. It will be appreciated that this is not essential. In other embodiments, the levels can be allocated differently. For example, the analysis in step 130 can reveal that the depth data comprises a cluster of depth values close to the camera and a cluster of depth values very far from the camera, with no pixels having depth values of intermediate distances. In this case, it can be meaningful to allocate a set of levels in the middle of the range of graph values to refer to unoccupied pixels. Such a range can be defined by a start threshold and an end threshold. In the case of encoding using a piecewise linear function, these thresholds can be implicit in the coordinates of the end points of the linear pieces.

[0116] Although the above examples have used piecewise linear functions, of course other functions can be used to convert between depth values and graph values are possible. Such functions can include (but are not limited to): quadratic functions, higher order polynomial functions, exponential functions and logarithmic functions. These functions can be used in their entirety, or can be used piecewise, in combination with other piecewise functions.

[0117] Embodiments of the invention rely on the use of metadata describing the encoding process when decoding the graph values. As the metadata is important to the decoding process, it can be beneficial if the metadata is encoded with an extra error detection or error correction code. Suitable codes are known in the field of communication theory.

[0118] Figure 1 and the encoding and decoding methods of Figure 3 and Figure 2 and Figure 4 The encoder and decoder of may be implemented in hardware or software or a mix of both (e.g. as firmware running on a hardware device). To the extent that embodiments are partly or entirely implemented in software, the functional steps illustrated in the process flow diagrams can be performed by a suitably programmed physical computing device, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process - and the individual constituent steps of each process illustrated in the flow diagrams - can be performed by the same or different computing devices. According to embodiments, a computer-readable storage medium stores a computer program comprising computer program code configured to, when the program is run on one or more physical computing devices, cause the one or more physical computing devices to perform an encoding or decoding method as described above.

[0119] The storage medium can include volatile and nonvolatile computer memory such as RAM, PROMs, EPROMs, and EEPROMs. The various storage media can be fixed in place or transportable, such that the one or more programs stored thereon can be loaded into a processing device.

[0120] The metadata according to embodiments can be stored on a storage medium. The bitstream according to embodiments can be stored on the same storage medium or on a different storage medium. The metadata can be embedded in the bitstream, but this is not required. Likewise, the metadata and / or the bitstream (with the metadata in the bitstream or separate from it) can be transmitted as a signal modulated onto an electromagnetic carrier. The signal can be defined according to a standard for digital communication. The carrier can be an optical carrier, a radio frequency wave, a millimeter wave, or a near-field communication wave. It can be wired or wireless.

[0121] To the extent that embodiments are implemented partly in hardware, embodiments can be implemented with a hardware component such as a processor of a computer system or any other hardware equivalent. Figure 2 And Figure 4 The blocks shown in the block diagrams of FIGS. 1-3 can be either separate physical components or logical subdivisions of a single physical component, or all can be implemented in an integrated manner in a single physical component. The functionality of one block shown in the block diagrams can be divided among multiple components in an implementation, or the functionality of multiple blocks shown in the block diagrams can be combined in a single component in an implementation. Hardware components suitable for implementing the embodiments of the application include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more blocks can be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0122] Variations of the disclosed embodiments can become apparent to those of ordinary skill in the art upon reading the foregoing disclosure and it is intended that such variations would be encompassed by this application. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. A computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state storage medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. If the term "adapted to" is used in the claims and / or the description, it should be noted that the term "adapted to" is used in the sense of "configured to", unless otherwise indicated. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A method of encoding depth data, the method comprising: receiving (110) source data comprising at least depth values; defining (120) a depth map comprising an array of map values, each map value taking one of a plurality of levels; analyzing (130) the depth values to determine how to encode the depth values into the plurality of levels; defining a first subset of the plurality of levels based on a result of the analysis, the first subset being for representing depth data of an occupied portion of the depth map; defining a second subset of the plurality of levels based on the result of the analysis, the second subset being for representing an unoccupied portion of the depth map; assigning one or more map values according to the defined second subset of the plurality of levels for the unoccupied portion of the depth map; encoding (150) the depth values in the depth map by converting the depth values to map values in the defined first subset of the plurality of levels for the occupied portion of the depth map; and generating a bitstream comprising the depth map and metadata comprising definitions of the first and second subsets. defining the first and second subsets comprises selecting (140) a threshold level among the plurality of levels, 2. The method of claim 1, wherein, wherein one of the subsets comprises all levels greater than the threshold level. the metadata further comprises a definition of a third subset.

3. The method of claim 1 or claim 2, further comprising defining a third subset of the plurality of levels as a protection level, the protection level to not be used in the depth map, wherein, analyzing the depth values comprises determining a dynamic range of the depth values.

4. The method of claim 1 or claim 2, wherein, converting the depth values to map values comprises converting using a piecewise linear function, and wherein the metadata further comprises a definition of the piecewise linear function.

5. The method of claim 1 or claim 2, wherein, 6. A method of decoding depth data, the method comprising: receiving (210) a bitstream comprising a depth map and associated metadata, the depth map comprising an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data of an occupied portion of the depth map and defining a second subset of the plurality of levels representing an unoccupied portion of the depth map, the first and second subsets being defined based on an analysis of depth values of source data; decoding (220) the depth map, including converting map values in the first subset of values to depth values; and outputting (230) the depth values. at least one of the first and second subsets is defined by a first threshold level among the plurality of levels, 7. The method of claim 6, wherein, wherein one of the subsets comprises all levels greater than the first threshold level. the metadata further comprises a definition of a piecewise linear function, and wherein converting the map values to depth values comprises converting using the piecewise linear function.

8. The method of claim 6 or claim 7, wherein, the metadata further comprises a definition of a third subset of the plurality of levels, the third subset being unused guard levels in the depth map, 9. The method of claim 6 or claim 7, wherein, the method further comprising detecting any map values in the third subset of values prior to decoding the depth map. ​ 10. The method of claim 6 or claim 7, wherein, The depth values are depth data of video data comprising a plurality of source views, the method further comprising reconstructing at least one of the plurality of source views.

11. The method of claim 6 or claim 7, wherein, The metadata comprises negative and positive normalized disparity values.

12. A computer program product comprising computer code for causing a processing system to implement a method according to any one of claims 1 to 11 when said computer code is run on the processing system.

13. An encoder (300) for depth data, configured to encode depth values into a depth map comprising an array of map values, each map value taking one of a plurality of levels, the encoder comprising: an input (310) configured to receive source data comprising at least depth values; an analyzer (320) configured to: analyze the depth values to determine how to encode the depth values into the plurality of levels, define a first subset of the plurality of levels based on a result of the analysis, the first subset being used to represent depth data of an occupied portion of the depth map; define a second subset of the plurality of levels based on the result of the analysis, the second subset being used to represent an unoccupied portion of the depth map; and a depth value encoder (330) configured to: assign one or more map values according to the defined second subset of the plurality of levels for the unoccupied portion of the depth map; convert the depth values into map values of the defined first subset of the plurality of levels for the occupied portion of the depth map, thereby encoding the depth values in the depth map; and generate a bitstream comprising the depth map and metadata comprising definitions of the first and second subsets.

14. A decoder for depth data, the decoder comprising: an input (410) configured to receive a bitstream comprising a depth map and associated metadata, the depth map comprising an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data of an occupied portion of the depth map and defining a second subset of the plurality of levels representing an unoccupied portion of the depth map, the first and second subsets being defined based on an analysis of depth values of source data; a depth value decoder (420) configured to decode the depth map, including converting map values of the first subset of values into depth values; and an output (430) configured to output the converted depth values. ​

Images