Coding scheme for depth data

Abstract

To reduce or avoid artifacts while encoding the depth and occupancy data efficiently.SOLUTION: Methods of encoding and decoding depth data are disclosed. In an encoding method, depth values and occupancy data are both encoded into a depth map. The method adapts how the depth values and occupancy data are converted to map values in the depth map. It may adaptively select a threshold, above or below which all values represent unoccupied pixels. By adapting how the depth and occupancy are encoded, based on analysis of the depth values, the method can enable more effective encoding and transmission of the depth data and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used by a corresponding decoding method, to decode the map values.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to encoding depth data, and more particularly to a method and apparatus for encoding and decoding immersive video. [Background technology]

[0002] Immersive video, also known as six degrees of freedom (6DoF) video, is a video of a three-dimensional (3D) scene that allows a view of the scene to be reconstructed for a viewpoint with varying position and orientation. It represents an evolution of three degrees of freedom (3DoF) video, which allows the view to be reconstructed for a viewpoint with any orientation, but only at a fixed point in space. In 3DoF, the degrees of freedom are angles: pitch, roll, and yaw. 3DoF video supports head rotation; in other words, a user consuming video content can look in any direction within a scene, but cannot move to a different location within the scene. 6DoF video supports head rotation and also supports the selection of a location within the scene from which the scene is observed.

[0003] To generate 6DoF video, multiple cameras are required to record a scene. Each camera generates image data (often called texture data in this context) and corresponding depth data. For each pixel, the depth data represents the depth at which the corresponding image pixel data is observed. Each of the multiple cameras provides a respective view of the scene.

[0004] To reduce redundancy between views, it has been proposed to prune views and pack them into a "texture atlas" for each frame of a video stream. This approach attempts to reduce or eliminate overlap between multiple views, thereby improving efficiency. The non-overlapping parts of different views that remain after pruning are sometimes called "patches." An example of this approach is described in Alvaro Collet et al., "High-quality streamable free-viewpoint video," ACM Trans. Graphics (SIGGRAPH), 34(4), 2015. Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to efficiently encode depth data. Related to depth data is occupancy data, which indicates whether a given pixel in a given view is occupied by a patch. Valid depth data exists for patches. Valid depth data does not exist for unoccupied pixels. One approach is to encode the occupancy map separately from the depth data, but this requires transmission of an additional data structure. Another approach is to encode the occupancy information embedded in the depth data, for example, by reserving specific depth values ​​to signal that a pixel is unoccupied. However, the depth data may be compressed for transmission. Both the compression and subsequent transmission can introduce errors into the decoded depth data. If the occupancy data is embedded together with the depth data, compression or transmission errors can also corrupt the occupancy data, which can ultimately lead to noticeable or disturbing visual artifacts in the decoded and rendered immersive video. It is desirable to reduce or avoid such artifacts while efficiently encoding the depth and occupancy data.

[0006] The invention is defined by the claims. [Means for solving the problem]

[0007] According to an example according to an aspect of the present invention, there is provided a method for encoding depth data, the method comprising: receiving source data including at least depth values; defining a depth map including an array of map values, each map value taking one of a plurality of levels; analyzing the depth values ​​to determine how to encode the depth values ​​into the plurality of levels; defining a first subset of the plurality of levels to represent depth data for occupied portions of the depth map based on results of the analysis; defining a second subset of the plurality of levels to represent unoccupied portions of the depth map based on results of the analysis; assigning one or more map values ​​from the defined second subset of the plurality of levels to unoccupied portions of the depth map; converting depth values ​​to map values ​​of the defined first subset of the plurality of levels for occupied portions of the depth map, thereby encoding the depth values ​​in the depth map; and generating a bitstream having the depth map and metadata including definitions of the first and second subsets.

[0008] This method allows occupancy to be coded into the depth map. The inventors recognized that jointly coding occupancy and depth works better if the coding can adapt to the content of the depth map. If the first and second subsets are fixed, the coding may be suboptimal. By analyzing the depth values ​​and selecting the first and second subsets according to the results of the analysis, the method makes more effective use of the bits available for the depth map.

[0009] The source data may be source video data. The source data may further include occupancy data indicating occupied and / or unoccupied portions of the depth map.

[0010] The definition of the first and second subsets depends on the content of the depth data, i.e. the selection of the subsets adapts to the content of the source data.

[0011] The method may further include compressing the depth map using a video compression algorithm, optionally a standardized video compression algorithm such as High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2. The bitstream may include the compressed depth map.

[0012] Defining the subset can include selecting at least one threshold level.

[0013] Defining the first and second subsets can include selecting a first threshold level from among a plurality of levels, one of the subsets including all levels greater than the first threshold level.

[0014] Optionally, the other subset includes all levels below the first threshold level. The metadata can include a definition of the first threshold level, either explicitly (i.e., the metadata can include the first threshold level) or implicitly (e.g., the metadata can define a transformation function in which the first threshold level is implicitly involved).

[0015] The first threshold level may be selected to be a power of two.

[0016] In other words, the first threshold level is 2. n where n is a non-negative integer. When the map values ​​are encoded as binary numbers, this may allow the occupancy to be determined by examining a subset of the bits of the binary map value.

[0017] The method may further include defining a third subset of the plurality of levels as guard levels not used in the depth map, the metadata further including a definition of the third subset.

[0018] The third subset is distinct from the first and second subsets, i.e., the levels of the third subset are not present in either the first or second subset.

[0019] Introducing guard levels may further enhance robustness against errors introduced by compression or transmission. The guard levels may include a range of levels between the first and second subsets. The guard levels are not used to represent depth data, nor are they used to represent unoccupied portions of the depth map. At the decoder, any instance of a map value equal to one of the guard levels may be detected as an error and, optionally, corrected.

[0020] The third subset may be defined by a second threshold level. For example, the third subset may be defined as including levels between the first and second thresholds. The metadata may include a definition of the second threshold level. This definition may be direct / explicit (e.g., the metadata may include the second threshold level) or indirect / implicit.

[0021] Analyzing the depth data can include determining a dynamic range of the depth data.

[0022] The method may further include measuring or predicting the degree of error in the decoded depth data caused by encoding the depth values ​​in the depth map in a given manner.

[0023] Depth data can include normalized disparity values. Normalized disparity values ​​occupy the range [0,1], where 0 represents infinite depth and 1 represents minimum depth. Using normalized disparity values ​​can facilitate better allocation of bits to depth values. For example, assume depth is measured in meters. The difference between 1 m and 2 m is more visually significant than the difference between 10 m and 11 m (even though the absolute depth difference of 1 m is the same in both cases). Normalized disparity helps capture this relative significance: in the first case, the normalized disparity difference is (1 / 1 - 1 / 2) = 0.5, and in the second case, the normalized disparity difference is (1 / 10 - 1 / 11) = 0.009.

[0024] Converting the depth values ​​to map values ​​may include converting using a piecewise linear function, and the metadata may further include a definition of the piecewise linear function.

[0025] The source data can be video data including multiple source views, each of which includes texture and depth values, i.e., the above method for encoding depth data can be applied to a method for encoding immersive video.

[0026] A method of decoding depth data is also provided, the method comprising the steps of receiving a depth map and associated metadata, the depth map having an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for occupied portions of the depth map, and the metadata defining a second subset of the plurality of levels representing depth values ​​for unoccupied portions of the depth map; decoding the depth map, the step including converting the map values ​​in the first subset of values ​​to depth values; and outputting the depth values.

[0027] Decoding the depth map optionally further includes generating an occupancy map by identifying map values ​​that are within the second subset of values.

[0028] The depth map in the bitstream may be compressed using a video compression algorithm, optionally using a standardized video compression algorithm, and the method may include decompressing the depth map according to the video compression algorithm before decoding the depth map.

[0029] At least one of the first and second subsets may be defined by a first threshold level of the plurality of levels, with one of the subsets including all levels greater than the first threshold level and the other subset including all levels less than the first threshold level.

[0030] The metadata may further include a definition of a piecewise linear function, and converting the map values ​​to depth values ​​may include converting using the piecewise linear function.

[0031] The metadata may further include a definition of a third subset of the plurality of levels that are guard levels not used in the depth map, and the method further includes detecting any map values ​​within the third subset of values ​​before decoding the depth map.

[0032] In the received depth map, any values ​​in the third subset are erroneous. They may have been introduced by compression or transmission errors. The method may include skipping or correcting these values ​​when decoding the depth map. In some embodiments, map values ​​that are close to values ​​in the first subset (i.e., close to a valid value) may be corrected by changing them to the nearest valid value.

[0033] The depth value may be depth data for video data that includes a plurality of source views, and the method may further include reconstructing at least one of the plurality of source views.

[0034] The metadata can include negative and positive normalized disparity values. This can provide one way of implicitly defining the first and second subsets, especially when using a linear (or piecewise linear) transformation function between depth values ​​and map values. The negative normalized disparity value can define the x-axis intercept of the transformation function (or part of the transformation function). This negative normalized disparity value may be assigned to level 0 of the map values. The positive normalized disparity value can define the maximum normalized disparity (minimum depth) to be encoded (or the maximum normalized disparity encoded in this part of the transformation function). This positive normalized disparity value is assigned to a particular level (e.g., the maximum level in the case of a single linear transformation function). This implicitly defines a first subset of levels (those corresponding to positive depth values) for representing depth data and a second subset of levels (those corresponding to "negative" depth values) for representing unoccupied portions of the depth map. Since actual normalized disparity values ​​cannot be negative, all of the depth values ​​are converted to levels in the first subset.

[0035] Also disclosed is a computer program comprising computer code that, when executed on a processing system, causes the processing system to perform the methods summarized above. The computer program can be stored in a computer-readable storage medium, which may be a non-transitory storage medium.

[0036] Also provided is an encoder for depth data configured to encode depth values ​​into a depth map having an array of map values, each map value taking on one of a plurality of levels, the encoder having an input configured to receive source data including at least the depth values; an analyzer configured to analyze the depth values ​​to determine how to encode the depth values ​​into the plurality of levels, and based on results of the analysis, define a first subset of the plurality of levels for representing depth data for occupied portions of the depth map; and based on results of the analysis, define a second subset of the plurality of levels for representing unoccupied portions of the depth map; and a depth value encoder configured to assign one or more map values ​​from the defined second subset of the plurality of levels for unoccupied portions of the depth map, and to convert depth values ​​to map values ​​in the defined first subset of the plurality of levels for occupied portions of the depth map, thereby encoding the depth values ​​in the depth map and generating a bitstream including metadata including the depth map and definitions of the first and second subsets.

[0037] Further provided is a decoder for depth data, the decoder having an input configured to receive a bitstream including a depth map and associated metadata, the depth map having an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for occupied portions of the depth map, and the metadata defining a second subset of the plurality of levels representing unoccupied portions of the depth map; a depth value decoder configured to decode the depth map to convert map values ​​in the first subset of values ​​to depth values; and an output configured to output the converted depth values.

[0038] According to yet another aspect, a bitstream is provided that includes a depth map and associated metadata, the depth map including an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for an occupied portion of the depth map, and the metadata defining a second subset of the plurality of levels representing an unoccupied portion of the depth map.

[0039] The bitstream can be encoded and decoded using the methods summarized above, which can be embodied on a computer-readable medium or as a signal modulated onto an electromagnetic carrier wave.

[0040] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0041] For a better understanding of the present invention and to show more clearly how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings in which: [Figure 1] 3 is a flowchart of a method for encoding depth data according to a first embodiment of the present invention; [Figure 2] 2 is a block diagram of an encoder for depth data according to one embodiment configured to perform the method shown in FIG. 1; [Figure 3] 6 is a flowchart illustrating a method for decoding depth data according to a second embodiment of the present invention. [Figure 4] 4 is a block diagram of a decoder for depth data according to an embodiment, configured to perform the method shown in FIG. 3; [Figure 5] 4A and 4B illustrate example linear transformation functions for converting between depth values ​​and map values. [Figure 6] FIG. 10 illustrates another exemplary linear transformation function. [Figure 7] FIG. 10 illustrates a linear transformation function with guard levels. [Figure 8]FIG. 10 illustrates another way to define a linear transformation function. [Figure 9] FIG. 10 is a diagram illustrating a piecewise linear transformation function. [Figure 10] FIG. 1 illustrates a piecewise linear transformation function with guard levels. DETAILED DESCRIPTION OF THE INVENTION

[0042] The present invention will now be described with reference to the drawings.

[0043] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will become better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0044] A method for encoding and decoding depth data is disclosed. In the encoding method, both depth values ​​and occupancy data are encoded into a depth map. The method adapts how the depth values ​​and occupancy data are converted to map values ​​in the depth map. For example, it can adaptively select a threshold above or below which all values ​​represent unoccupied pixels. By adapting how the depth and occupancy are encoded based on an analysis of the depth values, the method can enable more efficient encoding and transmission of the depth and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used to decode the map values ​​in a corresponding decoding method. Also provided are an encoder and decoder for the depth data, and a corresponding bitstream comprising the depth map and its associated metadata.

[0045] It is desirable to compress the depth data using known image and video compression algorithms. It is particularly desirable to be able to compress the depth data using standardized algorithms. Suitable hardware and software for encoding and decoding according to standardized codecs are widely available and are often highly optimized for both speed, quality, and power consumption. However, most video compression is lossy in order to achieve practical bit rates for transmission and storage. Therefore, it is generally not possible to guarantee that a depth map compressed using video compression techniques will be perfectly reconstructed at the decoder. Errors are introduced both by the compression and by the transmission and / or storage of the bitstream.

[0046] One basic approach to combining the coding of depth values ​​and occupancy data in a depth map (not according to the present invention) is to set a fixed threshold to distinguish between unoccupied pixels and valid depth values. For example, it may be desirable to code depth data using HEVC main 10 level 5.2, which implies a maximum bit depth of 10. This means that 1024 levels (0 to 1023) are available for coding depth and occupancy data. A depth data coding scheme may define that all levels from 0 to 63 indicate unoccupied pixels. Only levels 64 to 1023 are used to code depth values. This means that more than 6% of the available range is given up for coding occupancy. While this may be appropriate for some content, it may be inefficient for other content. It is difficult to select a single fixed threshold that is generally suitable for all types of content.

[0047] Fig. 1 shows an encoding method according to a first embodiment of the present invention, and Fig. 2 is a schematic block diagram of an encoder for carrying out the method of Fig. 1.

[0048] The encoder 300 includes an input 310, an analyzer 320, a depth value encoder 330, and an output 340. In step 110, the input 310 receives source data including depth values. In this embodiment, the source data is immersive video data including multiple source views. Each source view includes texture values ​​and depth values. The encoding of the texture values ​​is outside the scope of the present invention and will not be described further here.

[0049] In step 120, the depth value encoder 330 defines a depth map that includes an array of map values, each of which can take one of multiple levels, e.g., for a maximum bit depth of 10, there are 1024 levels.

[0050] The input 310 is coupled to an analyzer 320. In step 130, the analyzer analyzes the depth values ​​to determine the best way to encode the depth values ​​into multiple levels. In this embodiment, map values ​​below the threshold are used to represent unoccupied pixels, and map values ​​above the threshold are used to encode the depth values. Thus, the task of the analyzer is to select a threshold based on the analysis of the depth values ​​(step 140). More details on how to select the threshold are provided below. Here, the threshold (T) is a power of two (T=2 n ). This can be advantageous as it can allow a simple check at the decoder to establish whether a given map value is above or below the threshold. Rather than comparing the map value to a particular threshold, the decoder can simply check the most significant bit (MSB) of the map value. For example, if the threshold is T=256=2, then 8 , the decoder can check the most significant 2 bits of the 10-bit representation. If both of these bits are 0, the value is below the threshold. Otherwise, if either bit is 1, the value is above the threshold.

[0051] In step 150, the depth value encoder 330 populates the depth map. For unoccupied pixels, the depth map is populated with one or more map values ​​below a selected threshold. For each occupied pixel, the depth value encoder 330 converts the depth value to a respective map value above the threshold.

[0052] The depth value encoder 330 provides a populated depth map including the encoded map values ​​to the output 340. Meanwhile, the analyzer 320 provides metadata to the output 340. The metadata includes information defining how the depth values ​​are encoded. In particular, the metadata includes information about the selected threshold. The metadata may also include information about the mapping of depth values ​​to map values ​​in the range above the threshold. However, this may not be necessary in some embodiments, as the mapping may be explicitly defined in the encoding scheme. For example, all depth values ​​may be normalized disparity values ​​in the range [0, 1], and the mapping may be a linear mapping to map values ​​above the threshold.

[0053] The output 340 generates and outputs a bitstream that includes at least the depth map, and also outputs metadata, either as part of the same bitstream or separately from the bitstream.

[0054] Figure 3 is a flow chart illustrating a method for decoding an encoded depth map according to a second embodiment of the present invention, and Figure 4 is a schematic block diagram of a decoder for performing the method of Figure 3.

[0055] The decoder 400 comprises an input 410, a depth value decoder 420, and an output 430. Optionally, it may also include a renderer 440.

[0056] In step 210, input 410 receives a bitstream containing a depth map. The input also receives metadata describing the bitstream. The metadata may be embedded in the bitstream or separate. The depth map in this example was created according to the method of FIG. 1 described above. Note that the depth map input to decoder 400 is typically a version of the depth map output by encoder 300, which has subsequently undergone lossy video compression (and possibly error-prone communication over a transmission channel).

[0057] In step 220, the depth value decoder 420 decodes the depth map. This involves identifying map values ​​that exceed a threshold and converting them back to depth values. As mentioned above, the threshold is included in the metadata. An appropriate conversion function may be pre-agreed between the encoder and decoder (e.g., defined as part of a standardized encoding scheme). Alternatively, if not pre-defined / agreed upon, the conversion function may be embedded in the metadata and extracted by the decoder from the metadata.

[0058] The depth value decoder 420 provides the decoded depth values ​​to an output 430, which outputs the depth values ​​(step 230). The depth value decoder may also output an occupancy map indicating pixels in the depth map whose map values ​​were below a threshold.

[0059] If the decoder 400 includes an optional renderer 440, the depth value decoder 420 can provide the decoded depth values ​​to the renderer, which reconstructs one or more views from the depth data. In this case, the renderer 430 can provide the reconstructed views to the output 430, which can output the reconstructed views (e.g., to a frame buffer).

[0060] There are various ways in which map values ​​can be dynamically assigned to encode depth (and respective occupancy) data, some of which will now be described in more detail, along with the corresponding analysis performed by analyzer 320.

[0061] In some embodiments, analyzing the depth values ​​includes determining the dynamic range of the depth values. If the dynamic range is small (i.e., the depth values ​​are all around the same value and the differences between them are not significant), a small number of bits can be used to encode the occupancy. For example, if all cameras are close enough to the object that the dynamic range of the depth map is not important, one bit can be used to encode the occupancy map. That is, for a 10-bit depth map, the threshold level is T=512=2 9 This still leaves 512 levels for encoding depth data, which may be sufficient in close-up scenarios.

[0062] If a patch, view, frame, or video is determined to have full occupancy, the threshold can be set to 0, indicating that all pixels are occupied, which maximizes the number of levels available for encoding depth data.

[0063] In some embodiments, the method may include measuring or predicting the degree of visible error in the decoded depth data caused by encoding the depth values ​​in a particular manner. For example, the analyzer 320 may examine camera parameters associated with the source view to determine how to encode the depth values. If the two cameras are angularly spaced apart (e.g., 90°), a depth error in one view may be readily apparent as a shift to the left or right in the other view. In these situations, it may be advantageous to encode the depth values ​​as accurately as possible. On the other hand, if the two cameras are angularly spaced apart (e.g., 5°), errors in the depth values ​​may be much less noticeable.

[0064] In some embodiments, the analysis can include encoding the depth values, compressing the depth values, expanding and decoding the depth values, and synthesizing a test view from the decoded depth values. The synthesized test view can be compared to a reference view derived from the original source data to generate an error / difference image. This can be repeated for different configurations of the first and second subsets of levels. The configuration that results in the smallest error / difference can be selected for encoding the depth values.

[0065] Depth values ​​can be stored as normalized disparities (1 / Z), with near and far depths corresponding to the highest and lowest depth levels, respectively. One model assumes that a selected threshold level corresponds to near depth, with 1023 corresponding to near depth (for 10-bit data).

[0066] When specifying a depth range, there are various ways to specify the occupation coding.

[0067] Figure 5 shows an example of depth values ​​encoded into map values ​​by a linear function. For a given depth value on the x-axis, the function defines the corresponding map value on the y-axis. Since the depth values ​​are assumed to be provided as normalized disparities, the depth values ​​are limited to a minimum of 1 / Z max to 1 / Z of the maximum value min In the example of Figure 5, the maximum depth is assumed to be infinity (Z=∞), and therefore the minimum normalized disparity is 0. The y-axis intercept of the line gives the threshold level. Map values ​​below T are used to represent unoccupied pixels. The maximum depth value (1 / Z min ) is converted to the maximum map value (1023 for 10-bit encoding). minThe map values ​​for depth values ​​of can be obtained by linear interpolation. The transformation function can be defined in the metadata in various ways. For example, the metadata can be min The maximum and minimum map values ​​can be implicitly known from the bit depth (e.g., 1023 and 0, respectively, for 10 bits), so there would be no need to explicitly encode these in the metadata.

[0068] FIG. 6 shows another example linear transformation function, where the maximum depth Z max is less than infinity, so 1 / Z is nonzero max This linear transformation function is expressed as the coordinate (1 / Z max , T) to coordinate (1 / Z min , 1023). Again, map values ​​less than T are used to represent unoccupied areas. This transformation function is specified in the metadata as a threshold T, a minimum depth value Z min , maximum depth value Z max It can be defined by three parameters:

[0069] 1 / Z max Potential problems can arise when compression or transmission errors occur at normalized disparities very close to . The map value may pass the threshold T as a result of the error, meaning that a pixel at a nearby depth is mistakenly replaced with an unoccupied pixel. Figure 7 shows a transformation function that attempts to prevent this by introducing a guard level between the levels used to encode depth data and the levels used to indicate unoccupied pixels. The minimum map value used to encode depth values ​​is specified by the first threshold T, but the second threshold T G Only map values ​​less than 1 / Z are used to represent unoccupied pixels. max to 1 / Z GThis can be seen as introducing a guard band in the depth value between the two. Either way can be described in the metadata. In other words, one additional threshold is defined in the metadata, which can be a map value threshold or a depth value threshold.

[0070] Figure 8 shows another way to define a linear transformation function, where the metadata is positive depth values ​​1 / Z min , and a negative depth value -d. Negative depth values ​​have no real physical meaning (since normalized disparity is not necessarily negative). It serves as an implicit definition of a threshold T (at the y-axis intercept), below which map values ​​are used to indicate unoccupied pixels. The example in Figure 8 does not have a guard level, but this can be introduced by specifying an additional threshold, similar to the approach taken in Figure 7.

[0071] Figure 9 shows a piecewise linear transformation function. A piecewise linear function has three segments, which may be defined in the metadata by the coordinates of their endpoints. The map value defining the y-axis coordinate of the first endpoint implicitly defines a threshold T. That is, the (x, y) coordinate of the first endpoint is (1 / Z max , T). The first linear segment has a relatively shallow slope, meaning that relatively few map values ​​are assigned to depth values ​​in this range. The middle linear segment has a steeper slope, indicating that relatively more map values ​​are assigned to depth values ​​in this range. The third linear segment also has a shallower slope, meaning that fewer levels are assigned to that map value. Such a profile may be appropriate, for example, if most of the depth values ​​are clustered at intermediate distances. In this case, errors in pixels very close to or very far from the camera are less of a problem, i.e., more tolerable to the viewer.

[0072] Figure 10 shows another piecewise linear transformation function with guard levels. The guard levels can be understood as vertical steps in the piecewise linear function (and can potentially be encoded in the metadata). The guard levels are G The map values ​​range from T to T in steps, and unoccupied pixels are G It is only indicated by a smaller map value.

[0073] The examples of Figures 5 to 10 are not mutually exclusive, and transformation functions can be provided that combine properties of some of these examples. For example, one or more sections of a piecewise linear function (such as in Figure 9) can be defined by conceptual negative depth values, as in Figure 8. A third subset of levels consisting of guard levels may be introduced in any example.

[0074] In each of the above examples, the second subset of levels (indicating unoccupied pixels) is separated from the first subset of levels (indicating valid depth values) by one or two thresholds, with the first subset of levels being higher than the second subset of levels. It will be understood that this is not required. In other embodiments, levels may be assigned differently. For example, the analysis of step 130 may reveal that the depth data consists of a cluster of depth values ​​close to the camera and a cluster of depth values ​​very far from the camera, with no pixels with depth values ​​at intermediate distances. In this case, it may make sense to assign one set of levels in the middle of the range of map values ​​to indicate unoccupied pixels. Such a range can be defined by a start threshold and an end threshold. In the case of encoding using a piecewise linear function, these thresholds may be implicit in the coordinates of the end points of the linear segment.

[0075] While the above examples use piecewise linear functions, it is of course possible to use other functions to convert between depth values ​​and map values. Such functions can include, but are not limited to, quadratic functions, higher order polynomial functions, exponential functions, and logarithmic functions. These functions can be used in their entirety or piecewise or in combination with other piecewise functions.

[0076] Embodiments of the present invention rely on the use of metadata that describes the encoding process when decoding map values. Because the metadata is important to the decoding process, it is beneficial if the metadata is encoded with additional error detection or error correction codes. Suitable codes are known in the field of communications theory.

[0077] The encoding and decoding methods of Figures 1 and 3 and the encoders and decoders of Figures 2 and 4 may be implemented in hardware or software, or a mixture of both (e.g., as firmware executing on a hardware device). To the extent that an embodiment is implemented partially or entirely in software, the functional steps illustrated in the process flowcharts may be performed by appropriately programmed physical computing devices, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process, and its individual component steps illustrated in the flowcharts, may be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code configured to cause one or more physical computing devices to perform an encoding or decoding method as described above when the program is executed on the one or more physical computing devices.

[0078] The storage medium may include volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. Various storage media may be installed in a mobile computing device or may be transportable such that one or more programs stored on the storage medium may be read by a processor.

[0079] Metadata according to an embodiment may be stored on a storage medium. A bitstream according to an embodiment may be stored on the same storage medium or a different storage medium. The metadata may be embedded in the bitstream, but this is not required. Similarly, the metadata and / or the bitstream (with the metadata in the bitstream or separate from it) may be transmitted as a signal modulated onto an electromagnetic carrier wave. The signal may be defined according to a standard for digital communication. The carrier wave may be an optical carrier wave, a radio frequency wave, a millimeter wave, or a short-range communication wave. It may be wired or wireless.

[0080] To the extent that an embodiment is implemented partially or entirely in hardware, the blocks shown in the block diagrams of Figures 2 and 4 may be separate physical components, logical subdivisions of a single physical component, or all integrated into one physical component. The functionality of a block shown in the figures may be split among multiple components in implementation, or the functionality of multiple blocks shown in the figures may be combined into a single component in implementation. Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). One or more blocks may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0081] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. Where a computer program is described above, this computer program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet, or a wired or wireless telephone communication system. When the term "adapted for" is used in the claims or the description, it has the same meaning as the term "configured to." Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. 1. A method for encoding depth data, comprising: receiving source data having at least a depth value; defining a depth map having an array of map values, each map value taking one of a plurality of levels; analyzing the depth values ​​to determine how to encode the depth values ​​into the plurality of levels; defining a first subset of the plurality of levels for representing depth data of the depth map occupancy based on results of the analysis; defining a second subset of the plurality of levels to represent the unoccupied portion of the depth map based on the results of the analysis; assigning one or more map values ​​from the second defined subset of the plurality of levels to the unoccupied portion of the depth map; converting the depth values ​​to map values ​​in the first defined subset of the plurality of levels for the occupied portion of the depth map, thereby encoding the depth values ​​into the depth map; generating a bitstream comprising the depth map and metadata comprising a definition of the first and second subsets.

2. 2. The method of claim 1, wherein the step of defining the first and second subsets includes selecting a threshold level from the plurality of levels, one of the subsets having all levels above the threshold level.

3. 3. The method of claim 1, further comprising defining a third subset of the plurality of levels as guard levels that are not used in the depth map, the metadata further comprising a definition of the third subset.

4. The method of claim 1 , wherein the step of analysing the depth values ​​comprises determining a dynamic range of the depth values.

5. 5. The method of claim 1, wherein the step of converting the depth values ​​to map values ​​comprises converting using a piecewise linear function, and the metadata further comprises a definition of the piecewise linear function.

6. 1. A method for decoding depth data, comprising: receiving a bitstream having a depth map and associated metadata, the depth map having an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for occupied portions of the depth map, and the metadata defining a second subset of the plurality of levels representing unoccupied portions of the depth map; decoding the depth map, comprising converting map values ​​in the first subset of values ​​to depth values; and outputting the depth value.

7. 7. The method of claim 6, wherein at least one of the first and second subsets is defined by a first threshold value of the plurality of levels, one of the subsets having all levels above the first threshold value.

8. The method of claim 6 or 7, wherein the metadata further comprises a definition of a piecewise linear function, and wherein converting the map values ​​to depth values ​​comprises converting with the piecewise linear function.

9. 9. The method of claim 6, wherein the metadata further comprises a definition of a third subset of the plurality of levels that are guard bands not used in the depth map, and further comprising detecting map values ​​in the third subset of values ​​before decoding the depth map.

10. 10. The method according to claim 6, wherein the depth values ​​are depth data of video data having multiple source views, and further comprising the step of reconstructing at least one of the multiple source views.

11. The method of claim 1 , wherein the metadata comprises negative and positive normalized disparity values.

12. A computer program which, when executed by a processing system, causes the processing system to carry out the method of any one of claims 1 to 11.

13. 1. An encoder for depth data configured to encode depth values ​​into a depth map having an array of map values, each map value taking one of a plurality of levels, comprising: an input configured to receive source data having at least a depth value; an analyzer configured to analyze the depth values ​​to determine how to encode the depth values ​​into the plurality of levels, and based on results of the analysis, define a first subset of the plurality of levels to represent depth data for occupied portions of the depth map, and based on results of the analysis, define a second subset of the plurality of levels to represent unoccupied portions of the depth map; assigning one or more map values ​​from the second defined subset of the plurality of levels to the unoccupied portion of the depth map; for the occupied portion of the depth map, converting the depth values ​​to map values ​​in the defined first subset of the plurality of levels, thereby encoding the depth values ​​into the depth map; a depth value encoder configured to generate a bitstream comprising the depth map and metadata comprising a definition of the first and second subsets; An encoder having:

14. 1. A decoder for depth data, comprising: an input configured to receive a bitstream having a depth map and associated metadata, the depth map having an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for occupied portions of the depth map, and the metadata defining a second subset of the plurality of levels representing unoccupied portions of the depth map; a depth value decoder configured to convert map values ​​in the first subset of values ​​to depth values ​​and decode the depth map; an output unit configured to output the converted depth values; A decoder having:

15. 1. A bitstream having a depth map and associated metadata, the depth map having an array of map values, each map value taking one of a plurality of levels, the metadata defining a first subset of the plurality of levels representing depth data for occupied portions of the depth map, and the metadata defining a second subset of the plurality of levels representing unoccupied portions of the depth map.

Images