Coding scheme for depth data

Abstract

Methods of encoding and decoding depth data are disclosed. In an encoding method, both depth values and occupancy data are encoded into a depth map. The method adjusts how the depth values and occupancy data are converted into map values in the depth map. For example, it may adaptively select a threshold, all values above or below which represent unoccupied pixels. By adjusting how to encode depth and occupancy based on analysis of depth values, the method can achieve more efficient encoding and transmission of depth data and occupancy data. The encoding method outputs metadata defining adaptive encoding. The metadata may be used by a corresponding decoding method to decode the graph values. An encoder and a decoder are also provided for depth data, as well as corresponding bitstreams that include the depth map and its associated metadata.

Description

Technical Field

[0001] This invention relates to the encoding of depth data. In particular, it relates to methods and apparatus for encoding and decoding immersive video. Background Technology

[0002] Immersive video, also known as six degrees of freedom (6DoF) video, is video of a three-dimensional (3D) scene that allows for the reconstruction of a view of the scene for a changing viewpoint in position and orientation. It represents an evolution of three degrees of freedom (3DoF) video, which allows for the reconstruction of a viewpoint with arbitrary orientation, but only at a fixed point in space. In 3DoF, the degrees of freedom are angular—that is, pitch, roll, and yaw. 3DoF video supports head rotation—in other words, a user consuming the video content can look in any direction within the scene, but cannot move to different locations within the scene. 6DoF video supports head rotation and, in addition, allows for selection of the location within the scene being viewed.

[0003] To generate 6DoF video, multiple cameras are needed to record the scene. Each camera generates image data (often referred to as texture data in this context) and corresponding depth data. For each pixel, the depth data represents the depth of the corresponding image pixel data observed. Each of the multiple cameras provides its respective view of the scene.

[0004] To reduce redundancy between views, it has been suggested to trim views for each frame of the video stream and pack them into a “texture atlas.” This approach attempts to reduce or eliminate overlapping portions between multiple views, thereby improving efficiency. The non-overlapping portions of different views retained after trimming can be called “patches.” An example of this approach is described in “High-quality streamable free-viewpoint video” by Alvaro Collet et al. (ACM Trans. Graphics (SIGGRAPH), 34(4), 2015). Summary of the Invention

[0005] The goal is to efficiently encode depth data. Related to depth data is occupancy data, which indicates whether a given pixel in a given view is occupied by a patch. Valid depth data exists for patches. No valid depth data exists for unoccupied pixels. One approach would be to encode the occupancy map separately from the depth data, but this would require transmitting additional data structures. Another approach is to encode the occupancy information embedded in the depth data—for example, by preserving a specific depth value to signal that a pixel is not occupied. However, depth data can be compressed for transmission. Both compression and subsequent transmission can introduce errors into the decoded depth data. If occupancy data is embedded along with depth data, compression or transmission errors can also corrupt the occupancy data, which can ultimately lead to noticeable or disturbing visual artifacts in the decoded and rendered immersive video. The goal is to reduce or avoid such artifacts when efficiently encoding both depth and occupancy data.

[0006] This invention is defined by the claims.

[0007] According to an example of an aspect of the present invention, a method for encoding depth data is provided, the method comprising:

[0008] Receive source data that includes at least depth values;

[0009] Define a depth map, which comprises an array of map values, each map value taking one of a plurality of levels;

[0010] Analyze the depth values ​​to determine how to encode the depth values ​​into the multiple levels;

[0011] Based on the results of the analysis, a first subset of the multiple levels is defined, the first subset representing the depth data of the occupied portion of the depth map;

[0012] Based on the results of the analysis, a second subset of the multiple levels is defined, the second subset representing the unoccupied portion of the depth map;

[0013] For the unoccupied portion of the depth map, one or more map values ​​are assigned according to a second subset of the definitions of the plurality of levels;

[0014] For the occupied portion of the depth map, the depth value is converted into a map value within a first subset of the definitions of the plurality of levels, thereby encoding the depth value in the depth map; and

[0015] Generate a bitstream, the bitstream including the depth map, and metadata including definitions of the first subset and the second subset.

[0016] This method allows for the encoding of occupancy in the depth map. The inventors have recognized that encoding occupancy and depth together works better if the encoding can be adapted to the content of the depth map. If the first and second subsets are fixed, the encoding can be suboptimal. By analyzing the depth values ​​and selecting the first and second subsets based on the analysis results, this method can more effectively utilize the bits available for the depth map.

[0017] The source data can be source video data. The source data may also include occupancy data indicating the occupied and / or unoccupied portions of the depth map.

[0018] The definitions of the first and second subsets depend on the content of the depth data. In other words, the selection of subsets is adapted to the content of the source data.

[0019] The method may also include compressing the depth map using a video compression algorithm, optionally using a standardized video compression algorithm such as High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2. The bitstream may include the compressed depth map.

[0020] Defining a subset may include selecting at least one threshold level.

[0021] Defining the first subset and the second subset may include selecting a first threshold level among multiple levels, wherein one of the subsets includes all levels greater than the first threshold level.

[0022] Optionally, other subsets include all levels less than the first threshold level. Metadata may include a definition of the first threshold level. This may be explicit (i.e., metadata may include the first threshold level) or implicit (e.g., metadata may define a transformation function implicitly involving the first threshold).

[0023] The first threshold level can be chosen as a power of 2.

[0024] In other words, the first threshold level was chosen to be 2. n , where n is a non-negative integer. When graph values ​​are encoded as binary numbers, this allows occupancy to be determined by examining a subset of the bits in the binary graph value.

[0025] The method may further include defining a third subset of the plurality of levels as a protection level that will not be used in the depth map, wherein the metadata also includes a definition of the third subset.

[0026] The third subset is different from the first and second subsets. That is, the level of the third subset is neither in the first subset nor in the second subset.

[0027] Introducing protection levels can also enhance robustness to errors introduced by compression or transmission. Protection levels can include a range of levels between a first subset and a second subset. Protection levels are not used to represent depth data and are not used to represent unoccupied portions of the depth map. At the decoder, any instance of a map value equal to one of the protection levels can be detected as erroneous and optionally corrected.

[0028] The third subset can be defined by a second threshold level. For example, the third subset can be defined as including the level between the first and second thresholds. Metadata can include a definition of the second threshold level. This definition can be direct / explicit (e.g., metadata can include the second threshold level) or indirect / implicit.

[0029] Analyzing depth data can include determining the dynamic range of the depth data.

[0030] The method may also include measuring or predicting the extent of error in the decoded depth data caused by encoding depth values ​​in a depth map in a given manner.

[0031] Depth data can include normalized disparity values. Normalized disparity values ​​occupy the range [0,1], where 0 represents infinite depth and 1 represents minimum depth. Using normalized disparity values ​​facilitates better bit allocation for depth values. For example, suppose depth is measured in meters. The difference between 1 m and 2 m is visually more significant than the difference between 10 m and 11 m (even though the absolute depth difference of 1 m is the same in both cases). Normalized disparity helps capture this relative significance: in the first case, the difference in normalized disparity is (1 / 1 - 1 / 2) = 0.5, while in the second case, the difference is (1 / 10 - 1 / 11) = 0.009.

[0032] Converting depth values ​​to map values ​​can include using a piecewise linear function for the conversion, and the metadata can also include the definition of the piecewise linear function.

[0033] The source data can be video data comprising multiple source views, each including texture and depth values. In other words, the depth data encoding method outlined above can be applied to methods for encoding immersive video.

[0034] A method for decoding depth data is also provided, the method comprising:

[0035] Receive a bitstream including a depth map and associated metadata.

[0036] The depth map comprises an array of map values.

[0037] Each map value takes one of multiple levels.

[0038] The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map;

[0039] Decoding the depth map includes converting map values ​​from the first subset of values ​​into depth values; and

[0040] Output the depth value.

[0041] Decoding the depth map may optionally also include generating a occupancy map by identifying map values ​​in a second subset of the values.

[0042] The depth map in the bitstream can be compressed using a video compression algorithm, optionally using a standardized video compression algorithm. The method may include decompressing the depth map according to the video compression algorithm before decoding the depth map.

[0043] At least one of the first and second subsets can be defined by a first threshold level intermediate among multiple levels, wherein one subset includes all levels greater than the first threshold level. The other subset may include all levels less than the first threshold level.

[0044] Metadata may also include the definition of a piecewise linear function, and converting graph values ​​to depth values ​​may include using a piecewise linear function for the conversion.

[0045] The metadata may also include definitions of a third subset of multiple levels, which are protection levels not used in the depth map. The method also includes detecting any map values ​​in the third subset of values ​​before decoding the depth map.

[0046] In the received depth map, any values ​​in the third subset are erroneous. They may be introduced by compression or transmission errors. The method may include skipping or correcting these values ​​when decoding the depth map. In some embodiments, map values ​​that are close to values ​​in the first subset (i.e., close to valid values) can be corrected by changing them to the closest valid values.

[0047] The depth value can be depth data that includes video data from multiple source views, and the method can also include reconstructing at least one of the multiple source views.

[0048] Metadata can include negative and positive normalized disparity values. This provides a way to implicitly define a first and second subset—especially when using a linear (or piecewise linear) transformation function between depth and map values. A negative normalized disparity value can define the x-intercept of the transformation function (or a segment of the transformation function). This negative normalized disparity value can be assigned to level 0 in the map values. A positive normalized disparity value can define the maximum normalized disparity (minimum depth) to be encoded (or the maximum normalized disparity to be encoded using that segment of the transformation function). This positive normalized disparity value is assigned to a specified level (e.g., the maximum level in the case of a single linear transformation function). This implicitly defines a first subset of levels used to represent depth data (corresponding to those with positive depth values) and a second subset of levels used to represent the unoccupied portions of the depth map (corresponding to those with "negative" depth values). Since true normalized disparity values ​​cannot be negative, all depth values ​​are converted to levels within the first subset.

[0049] A computer program comprising computer code is also disclosed, which, when run on a processing system, causes the processing system to implement the methods outlined above. The computer program may be stored on a computer-readable storage medium. This may be a non-transitory storage medium.

[0050] An encoder for depth data is also provided, configured to encode depth values ​​into a depth map comprising an array of map values, each map value taking one of a plurality of levels, the encoder comprising:

[0051] The input unit is configured to receive source data that includes at least depth values.

[0052] The analyzer is configured as follows:

[0053] The depth values ​​are analyzed to determine how to encode them into the multiple levels.

[0054] Based on the results of the analysis, a first subset of the multiple levels is defined, the first subset representing the depth data of the occupied portion of the depth map;

[0055] Based on the results of the analysis, a second subset of the multiple levels is defined, the second subset representing the unoccupied portion of the depth map; and

[0056] The depth value encoder is configured as follows:

[0057] For the unoccupied portion of the depth map, one or more map values ​​are assigned according to a second subset of the definitions of the plurality of levels;

[0058] For the occupied portion of the depth map, the depth value is converted into a map value within a first subset of the definitions of the plurality of levels, thereby encoding the depth value in the depth map; and

[0059] Generate a bitstream, the bitstream including the depth map, and metadata including definitions of the first subset and the second subset.

[0060] A decoder for depth data is also provided, the decoder comprising:

[0061] The input section is configured to receive a bitstream including a depth map and associated metadata.

[0062] The depth map comprises an array of map values.

[0063] Each map value takes one of multiple levels.

[0064] The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map;

[0065] A depth value decoder, configured to decode the depth map, including converting map values ​​from a first subset of values ​​into depth values; and

[0066] The output section is configured to output the converted depth value.

[0067] According to another aspect, a bitstream including a depth map and associated metadata is provided.

[0068] The depth map comprises an array of map values, each map value representing one of multiple levels.

[0069] The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map.

[0070] Bitstreams can be encoded and decoded using the methods outlined above. They can be represented on computer-readable media or as signals modulated onto an electromagnetic carrier.

[0071] These and other aspects of the invention will become apparent from and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0072] To better understand the invention, and to more clearly illustrate how it can be carried out, reference will now be made only by way of example to the accompanying drawings, in which:

[0073] Figure 1 This is a flowchart of a method for encoding depth data according to a first embodiment of the present invention;

[0074] Figure 2 This is a block diagram of an encoder for depth data according to an embodiment, the encoder being configured to perform... Figure 1 The method illustrated;

[0075] Figure 3 This is a flowchart illustrating a method for decoding depth data according to a second embodiment of the present invention;

[0076] Figure 4 This is a block diagram of a depth data decoder according to an embodiment, the decoder being configured to perform... Figure 3 The method illustrated; and

[0077] Figure 5 An exemplary linear transformation function for converting between depth values ​​and map values ​​is shown;

[0078] Figure 6 Another exemplary linear transformation function is shown;

[0079] Figure 7 A linear transformation function with protection level is shown;

[0080] Figure 8 The diagram illustrates the alternative methods for defining linear transformation functions;

[0081] Figure 9 The piecewise linear transformation function is shown; and

[0082] Figure 10 A piecewise linear transformation function with a protection level is shown. Detailed Implementation

[0083] The invention will be described with reference to the accompanying drawings.

[0084] It should be understood that the detailed descriptions and specific examples, while indicating exemplary embodiments of the apparatuses, systems, and methods, are intended for illustrative purposes only and not to limit the scope of the invention. These and other features, aspects, and advantages of the apparatuses, systems, and methods of the present invention will become better understood from the following description, claims, and drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used in all drawings to indicate the same or similar parts.

[0085] Methods for encoding and decoding depth data are disclosed. In the encoding method, both depth values ​​and occupancy data are encoded into a depth map. The method adjusts how the depth values ​​and occupancy data are converted into map values ​​in the depth map. For example, it can adaptively select a threshold, where all values ​​above or below the threshold represent unoccupied pixels. By adjusting how depth and occupancy are encoded based on analysis of the depth values, the method enables more efficient encoding and transmission of depth data and occupancy data. The encoding method outputs metadata defining the adaptive encoding. This metadata can be used by a corresponding decoding method to decode the map values. Encoders and decoders are also provided for the depth data, as well as the corresponding bitstream, including the depth map and its associated metadata.

[0086] It will be desirable to compress depth data using known image and video compression algorithms. In particular, it will be desirable to be able to compress depth data using standardized algorithms. Suitable hardware and software for encoding and decoding according to standardized codecs are widely available and often highly optimized for both speed and quality, as well as power consumption. However, most video compression is lossy in order to achieve a practical bitrate for transmission and storage. Therefore, it is generally not guaranteed that a depth map compressed using video compression techniques will be perfectly reconstructed at the decoder. Errors will be introduced by compression and potentially also by the transmission and / or storage of the bitstream.

[0087] A basic approach (not according to this invention) to encode depth values ​​and occupancy data in a depth map would be to set a fixed threshold to distinguish between unoccupied pixels and valid depth values. For example, it might be desirable to encode depth data using HEVC Main 10Level 5.2, meaning a maximum bit depth of 10. This means 1024 levels (from 0 to 1023) are available for encoding depth and occupancy data. The depth data encoding scheme could define all levels from 0 to 63 as indicating unoccupied pixels. Only levels 64 to 1023 are used to encode depth values. This means that more than 6% of the available range is used to encode occupancy. This works for some content but is inefficient for others. It is difficult to choose a single fixed threshold that is universally applicable to all types of content.

[0088] Figure 1 The illustration shows an encoding method according to a first embodiment of the present invention. Figure 2 It is used for execution Figure 1 A schematic block diagram of the encoder method.

[0089] The encoder 300 includes an input unit 310; an analyzer 320; a depth encoder 330; and an output unit 340. In step 110, the input unit 310 receives source data including depth values. In this embodiment, the source data is immersive video data including multiple source views. Each source view includes texture values ​​and depth values. The encoding of texture values ​​is outside the scope of this invention and will not be discussed further here.

[0090] In step 120, the depth value encoder 330 defines a depth map comprising an array of map values. Each map value uses one of multiple levels. For example, if the maximum bit depth is 10, there will be 1024 levels.

[0091] Input unit 310 is coupled to analyzer 320. In step 130, the analyzer analyzes the depth values ​​to determine how best to encode the depth values ​​into multiple levels. In this embodiment, map values ​​below a threshold are used to represent unoccupied pixels, and map values ​​above a threshold are used to encode the depth values. Therefore, the analyzer's task is to select a threshold based on the analysis of the depth values ​​(step 140). Further details regarding how the threshold is selected will be discussed later. For now, note that the threshold (T) can be selected as a power of 2 (T=2). n This can be advantageous because it allows for a simple check at the decoder to determine whether a given graph value is above or below a threshold. Instead of comparing the graph value to a specific threshold, the decoder can simply check the most significant bit (MSB) of the graph value. For example, if the threshold T = 256 = 2... 8 The decoder can then examine the two most significant bits of the 10-bit representation. If both bits are 0, the value is below the threshold; otherwise, if either bit is 1, the value is above the threshold.

[0092] In step 150, the depth value encoder 330 fills the depth map. For unoccupied pixels, the depth map is filled with one or more map values ​​below a selected threshold. For each occupied pixel, the depth value encoder 330 converts the depth value into a corresponding map value above the threshold.

[0093] The depth encoder 330 provides the output unit 340 with a filled depth map containing the encoded map values. Simultaneously, the analyzer 320 provides metadata to the output unit 340. The metadata includes information defining how the depth values ​​are encoded. Specifically, the metadata includes information about a selected threshold. The metadata may also include information about mapping the depth values ​​to map values ​​in a range above the threshold. However, this may not be necessary in some embodiments, as the mapping can be explicitly defined in the encoding scheme. For example, all depth values ​​could be normalized disparity values ​​in the range [0,1], and the mapping could be a linear mapping of values ​​above the threshold.

[0094] Output unit 340 generates and outputs a bitstream that includes at least a depth map. It also outputs metadata, either as part of the same bitstream or separately from it.

[0095] Figure 3 This is a flowchart illustrating a method for decoding and encoding a depth map according to a second embodiment of the present invention. Figure 4 It is used for execution Figure 3 A schematic block diagram of the decoder for the method.

[0096] The decoder 400 includes an input unit 410; a depth value decoder 420; and an output unit 430. Optionally, it may also include a plotter 440.

[0097] In step 210, the input unit 410 receives a bitstream including a depth map. The input unit also receives metadata describing the bitstream. The metadata may be embedded in the bitstream or it may be separate. The depth map in this example is based on the above... Figure 1 The depth map is created using this method. It should be noted that the depth map input to the decoder 400 will typically be a version of the depth map output by the encoder 300, which has subsequently undergone lossy video compression (and may have been subject to error-prone communication over the transmission channel).

[0098] In step 220, the depth value decoder 420 decodes the depth map. This involves identifying map values ​​above a threshold and converting them back to depth values. As mentioned above, the threshold is included in the metadata. An appropriate transformation function can be pre-agreed upon between the encoder and decoder (e.g., defined as part of a normalized encoding scheme). Alternatively, if not pre-defined / agreed upon, the transformation function can be embedded in the metadata, and the decoder can extract it from the metadata.

[0099] The depth value decoder 420 provides the decoded depth value to the output unit 430. The output unit 430 outputs the depth value (step 230). The depth value decoder can also output a occupancy map, which indicates pixels in the depth map whose map value is below a threshold.

[0100] If decoder 400 includes an optional renderer 440, then depth value decoder 420 can provide the decoded depth values ​​to the renderer, which reconstructs one or more views from the depth data. In this case, renderer 430 can provide the reconstructed view to output unit 430, and output unit 430 can output the reconstructed view (e.g., to a frame buffer).

[0101] There are various methods, among which map values ​​can be dynamically allocated to encode depth (and corresponding occupancy) data. Some of these methods will now be discussed in more detail—along with the corresponding analysis to be performed by analyzer 320.

[0102] In some embodiments, analyzing depth values ​​includes determining the dynamic range of the depth values. If the dynamic range is small (that is, if all depth values ​​are around the same value and the differences between them are not significant), then a small number of bits can be used to encode the occupancy rate. For example, if all cameras are close enough to the object and the dynamic range of the depth map is not critical, then one bit can be used to encode the occupancy rate map; that is, for a 10-bit depth map, the threshold level would be T=512=2. 9 This still leaves 512 levels for encoding depth data, which may be sufficient in close-up scenarios.

[0103] When a patch, view, frame, or video is determined to have full occupancy, the threshold can be set to 0, indicating that all pixels are occupied. This maximizes the number of levels available for encoding depth data.

[0104] In some embodiments, the method may include measuring or predicting the extent of visible error in the decoded depth data caused by encoding depth values ​​in a particular manner. For example, analyzer 320 may examine camera parameters associated with a source view to determine how to encode depth values. If the two cameras have a wide angular spacing (e.g., 90°), then a depth error in one view will be readily apparent as a left or right shift in the other view. In these cases, it would be advantageous to encode the depth values ​​as accurately as possible. On the other hand, if the two cameras have a small angular spacing (e.g., 5°), then the error in the depth values ​​is less likely to be perceived.

[0105] In some embodiments, the analysis may include encoded depth values, compressed depth values, decompressed and decoded depth values, and a synthesized test view from the decoded depth values. The synthesized test view can be compared with a reference view derived from the original source data to produce an error / difference image. This can be repeated for different configurations of a first subset and a second subset of levels. The configuration that results in the minimum error / difference can be selected for the encoded depth values.

[0106] Depth values ​​can be stored as normalized disparity (1 / Z), where near depth and far depth correspond to the highest and lowest depth levels, respectively. One model assumes that the chosen threshold level corresponds to far depth, and 1023 corresponds to near depth (for 10-bit data).

[0107] When specifying a depth range, there are several ways to specify the occupancy encoding.

[0108] Figure 5 The illustration shows an example of depth values ​​encoded as map values ​​using a linear function. For a given depth value on the x-axis, the function defines the corresponding map value on the y-axis. It is assumed that the depth values ​​are provided as normalized disparity; therefore, the depth values ​​range from 1 / Z. maxThe minimum value to 1 / Z min The maximum value. Figure 5 In the example, the maximum depth is assumed to be infinite (Z=∞); therefore, the minimum normalized disparity is 0. The y-intercept of the line gives the threshold level T. Map values ​​below T will be used to represent unoccupied pixels. Maximum depth value (1 / Z) min This will be converted to the maximum map value (1023, in the case of 10-bit encoding). 0 and 1 / Z min The depth values ​​between these points can be obtained through linear interpolation. Transformation functions can be defined in the metadata in various ways. For example, the metadata may include a threshold T and a minimum depth value Z. min The maximum and minimum map values ​​can be implicitly known from the bit depth (e.g., 1023 and 0 for 10 bits); therefore, it may not be necessary to explicitly encode these in the metadata.

[0109] Figure 6 Another exemplary linear transformation function is shown. Here, the maximum depth Z max Less than infinity; therefore, there exists a non-zero 1 / Z. max The linear transformation function transforms the coordinates (1 / Z) max (T) runs to coordinates (1 / Z) min (1023). Again, map values ​​below T are used to represent unoccupied areas. This transformation function can be defined in the metadata using three parameters: the threshold T; the minimum depth value Z. min ; and the maximum depth value Z max .

[0110] If it is very close to 1 / Z max Introducing compression or transmission errors into the normalized disparity value can lead to potential problems. The image value may exceed a threshold T due to these errors, meaning that pixels at greater depths are incorrectly replaced by unoccupied pixels. Figure 7 The transformation function is shown, which attempts to prevent this by introducing a protection level between the level used to encode depth data and the level used to indicate unoccupied pixels. The minimum map value used to encode the depth value is given by a first threshold T. However, only values ​​below a second threshold T are allowed. G The image value is used to represent unoccupied pixels. This can also be seen as introducing a guard band in the depth values, between 1 / Z. max With 1 / Z G Between. It can be described in the metadata in any way. In other words, an additional threshold is defined in the metadata, which can be a threshold for graph values ​​or a threshold for depth values.

[0111] Figure 8 The diagram illustrates another way to define a linear transformation function. Here, the metadata includes the positive depth value 1 / Z.min And the negative depth value -d. The negative depth value has no real physical meaning (because normalized disparity is always non-negative). It is used as an implicit definition of a threshold T (at the y-intercept), below which map values ​​are used to indicate unoccupied pixels. Figure 8 The example does not have a protection level, but these levels can be introduced by specifying additional thresholds, similar to... Figure 7 The method used in [the context].

[0112] Figure 9 The piecewise linear transformation function is shown. The piecewise linear function has three segments, which can be defined in the metadata by the coordinates of their endpoints. The plot value defining the y-axis coordinate of the first endpoint implicitly defines the threshold T. In other words, the (x, y) coordinate of the first endpoint is (1 / Z). max The first linear segment has a relatively shallow slope, meaning relatively few map values ​​are assigned to depth values ​​within that range. The middle linear segment has a steeper slope, indicating that relatively more map values ​​are assigned to depth values ​​within that range. The third linear segment again has a shallow slope, with fewer levels assigned to its map values. Such a profile can be appropriate, for example, when most depth values ​​are clustered in the middle distance. In this case, errors in pixels very close to the camera or very far away may be less detrimental—that is, more acceptable to the observer.

[0113] Figure 10 Another piecewise linear transformation function is shown, with a guard level. The guard level can be understood (and potentially encoded in metadata) as the vertical step in the piecewise linear function. The guard level is a graph value within a range of steps, from T... G Up to T. Unoccupied pixels are only those less than T. G The value is indicated by the graph.

[0114] Figure 5-10 The examples in the examples are not mutually exclusive, and transformation functions that combine several properties from these examples can be provided. For example, one or more piecewise linear function segments (such as...) Figure 9 One of them (can be defined by a nominal negative depth value, such as) Figure 8 As shown. A third subset of protection levels can be introduced in any of the examples.

[0115] In each of the examples above, the second level subset (indicating unoccupied pixels) is separated from the first level subset (indicating valid depth values) by one or two thresholds, where the first level subset is higher than the second level subset. It will be understood that this is not necessary. In other embodiments, levels can be assigned in different ways. For example, the analysis in step 130 may reveal that the depth data includes clusters of depth values ​​close to the camera and clusters of depth values ​​very far from the camera, with no pixels having depth values ​​at intermediate distances. In this case, it could be meaningful to assign a set of levels in the middle of the range of map values ​​to refer to unoccupied pixels. Such a range can be defined by start and end thresholds. When encoding using a piecewise linear function, these thresholds can be implicitly contained in the coordinates of the endpoints of the linear segments.

[0116] While the example above has used a piecewise linear function, it is certainly possible for other functions to be used to convert between depth and graph values. Such functions can include (but are not limited to): quadratic functions, higher-order polynomial functions, exponential functions, and logarithmic functions. These functions can be used as a whole or piecewise in combination with other piecewise functions.

[0117] Embodiments of the present invention rely on the use of metadata describing the encoding process when decoding graph values. Since metadata is important to the decoding process, it may be beneficial if the metadata is encoded with additional error detection or error correction codes. Suitable codes are known in the field of communication theory.

[0118] Figure 1 and Figure 3 Encoding and decoding methods and Figure 2 and Figure 4 The encoder and decoder can be implemented in hardware, software, or a combination of both (e.g., as firmware running on a hardware device). The functional steps illustrated in the process flowchart can be performed by a suitably programmed physical computing device, such as one or more central processing units (CPUs) or graphics processing units (GPUs), to the extent that the embodiments are implemented partially or entirely in software. Each process—and its individual constituent steps illustrated in the flowchart—can be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program including computer program code configured to cause one or more physical computing devices to perform the encoding or decoding methods described above when the program is run on one or more physical computing devices.

[0119] Storage media can include volatile and non-volatile computer memories, such as RAM, PROM, EPROM, and EEPROM. Various storage media can be fixed within a computing device or can be transportable, allowing one or more programs stored thereon to be loaded into a processor.

[0120] Metadata according to the embodiments can be stored on a storage medium. Bitstreams according to the embodiments can be stored on the same storage medium or different storage media. Metadata can be embedded in the bitstream, but this is not required. Similarly, metadata and / or the bitstream (metadata within or separate from the bitstream) can be transmitted as a signal modulated onto an electromagnetic carrier. This signal can be defined according to standards used for digital communication. The carrier can be an optical carrier, radio frequency wave, millimeter wave, or near-field communication wave. It can be wired or wireless.

[0121] In terms of the extent to which the embodiments are implemented in hardware, either partially or entirely, Figure 2 and Figure 4 The blocks shown in the block diagram can be individual physical components, logical subdivisions of a single physical component, or all integrated into a single physical component. The functionality of one block shown in the figures can be divided among multiple components in an implementation, or the functionality of multiple blocks shown in the figures can be combined in a single component. Hardware components suitable for embodiments of the present invention include, but are not limited to, conventional microprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). One or more blocks can be implemented as a combination of dedicated hardware performing some functions and one or more programmable microprocessors and associated circuitry performing other functions.

[0122] By studying the accompanying drawings, disclosure, and appended claims, those skilled in the art can understand and implement variations of the disclosed embodiments in practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality. A single processor or other unit can perform the functions of several items listed in the claims. Although specific measures are listed in mutually different dependent claims, this does not indicate that combinations of these measures cannot be advantageously used. If a computer program has been discussed above, the computer program can be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but the computer program can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. If the term "suitable" is used in the claims or description, it should be noted that the term "suitable" is intended to be equivalent to the term "configured as." Any reference numerals in the claims should not be construed as limiting the scope.

Claims

1. A method for encoding depth data, the method comprising: Receive (110) source data that includes at least depth values; Define (120) a depth map, which comprises an array of map values, each map value taking one of a plurality of levels; Analyze (130) the depth values ​​to determine how to encode the depth values ​​into the plurality of levels; Based on the results of the analysis, a first subset of the plurality of levels is defined (140), the first subset being used to represent the depth data of the occupied portion of the depth map; Based on the results of the analysis, a second subset of the plurality of levels is defined (140), the second subset representing the unoccupied portion of the depth map; For the unoccupied portion of the depth map, one or more map values ​​are assigned according to a second subset of the definitions of the plurality of levels; For the occupied portion of the depth map, the depth value is converted into a map value in a first subset of the definitions of the plurality of levels, thereby encoding the depth value in the depth map (150). and Generate a bitstream, the bitstream including the depth map, and metadata including definitions of the first subset and the second subset.

2. The method according to claim 1, wherein, Defining the first subset and the second subset includes selecting a threshold level (140) among the plurality of levels. Wherein, a subset of the subset includes all levels greater than the threshold level.

3. The method according to claim 1 or claim 2, further comprising defining a third subset of the plurality of levels as protection levels, wherein the protection levels will not be used in the depth map, wherein, The metadata also includes a definition of the third subset.

4. The method according to any one of the preceding claims, wherein, Analyzing the depth value includes determining the dynamic range of the depth value.

5. The method according to any one of the preceding claims, wherein, Converting the depth value to a map value involves using a piecewise linear function for the conversion, and the metadata also includes a definition of the piecewise linear function.

6. A method for decoding depth data, the method comprising: Receive (210) a bitstream including a depth map and associated metadata, The depth map comprises an array of map values. Each map value takes one of multiple levels. The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map; Decoding the depth map (220) includes converting map values ​​from the first subset of values ​​into depth values; and Output the depth value described in (230).

7. The method according to claim 6, wherein, At least one subset of the first subset and the second subset is defined by a first threshold level intermediate among the plurality of levels. Wherein, a subset of the subset includes all levels greater than the first threshold level.

8. The method according to claim 6 or claim 7, wherein, The metadata also includes a definition of a piecewise linear function, and wherein converting the graph values ​​to depth values ​​includes performing the conversion using the piecewise linear function.

9. The method according to any one of claims 6 to 8, wherein, The metadata also includes a definition of a third subset of the multiple levels, which are protection levels that are not used in the depth map. The method also includes detecting any map values ​​in the third subset of values ​​before decoding the depth map.

10. The method according to any one of claims 6 to 9, wherein, The depth value is depth data that includes video data from multiple source views, and the method further includes reconstructing at least one of the multiple source views.

11. The method according to any one of the preceding claims, wherein, The metadata includes negative normalized disparity values ​​and positive normalized disparity values.

12. A computer program comprising computer code, the computer code being configured to cause the processing system to perform the method according to any one of claims 1 to 11 when the program is run on the processing system.

13. An encoder (300) for depth data, configured to encode depth values ​​into a depth map comprising an array of map values, each map value taking one of a plurality of levels, the encoder comprising: The input unit (310) is configured to receive source data that includes at least a depth value; The analyzer (320) is configured as follows: The depth values ​​are analyzed to determine how to encode them into the multiple levels. Based on the results of the analysis, a first subset of the multiple levels is defined, the first subset representing the depth data of the occupied portion of the depth map; Based on the results of the analysis, a second subset of the multiple levels is defined, the second subset representing the unoccupied portion of the depth map; as well as The depth value encoder (330) is configured as follows: For the unoccupied portion of the depth map, one or more map values ​​are assigned according to a second subset of the definitions of the plurality of levels; For the occupied portion of the depth map, the depth value is converted into a map value in a first subset of the definitions of the plurality of levels, thereby encoding the depth value in the depth map; and Generate a bitstream, the bitstream including the depth map, and metadata including definitions of the first subset and the second subset.

14. A decoder for depth data, the decoder comprising: The input unit (410) is configured to receive a bitstream including a depth map and associated metadata. The depth map comprises an array of map values. Each map value takes one of multiple levels. The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map; A depth value decoder (420) is configured to decode the depth map, including converting map values ​​in the first subset of values ​​into depth values; as well as The output unit (430) is configured to output the converted depth value.

15. A bitstream comprising a depth map and associated metadata, The depth map comprises an array of map values, each map value representing one of multiple levels. The metadata defines a first subset of the plurality of levels, the first subset representing the depth data of the occupied portion of the depth map, and the metadata defines a second subset of the plurality of levels, the second subset representing the unoccupied portion of the depth map.

Images