Depth difference instead of motion vectors

Abstract

An input texture image is received. A depth map is determined for the input texture image. Each depth map in the depth maps includes depth values ​​for pixels represented by a corresponding input texture image in the input texture image. A depth difference map is generated from the depth maps. The depth difference maps include at least one depth difference map generated from two consecutive depth maps in the depth maps. A video signal is output that is encoded with a compressed version of the input texture image and the depth difference map. The video signal causes a receiving device of the video signal to generate a display image from the compressed version of the input texture image and the depth difference map for rendering on an image display.

Description

[Technical field]

[0001] [Related Applications] This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 305,651 (Reference No. D21073USP1), filed February 1, 2022, and European Patent Application No. 22155825.7, filed February 9, 2022, the contents of each of which are incorporated herein by reference in their entirety.

[0002] [Technical field] The present invention relates generally to image processing operations. More specifically, embodiments of the present disclosure relate to video codecs. [Background technology]

[0003] Texture images and depth images can be coded into the video signal to support 3D or multi-view video streaming and rendering of the receiving device of the video signal. The texture images can be compressed, for example using inter- and intra-prediction, to remove inter- and intra-frame data redundancy from the encoded texture image data in the video signal, thus reducing the total amount of texture image data. Similarly, the depth data can be compressed to remove data redundancy from the encoded depth data in the video signal, thus reducing the total amount of depth image data.

[0004] Needless to say, the resulting amount of encoded video and depth data to support a seamless high-quality experience in 3D or multi-view video streaming and rendering can still be enormous in many operational scenarios.

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Thus, unless otherwise noted, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, problems identified with one or more approaches should not be assumed to have been recognized in any prior art under this section unless specifically indicated. [Brief description of the drawings]

[0006] The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference symbols represent similar elements and in which:

[0007] [Figure 1A] 1 shows an example of an image and depth difference data server. [Figure 1B] 2 shows an example of a downstream receiving device. [Figure 1C] 1 shows examples of image and depth difference data servers in different configurations.

[0008] [Figure 2A] Two example depth maps are shown. [Figure 2B] 2 shows two example depth difference maps.

[0009] [Figure 3A] Two examples of Group of Pictures are shown. [Figure 3B] Two examples of Group of Pictures are shown.

[0010] [Figure 4A] 1 shows an exemplary process flow. [Figure 4B] 1 shows an exemplary process flow.

[0011] [Diagram 5]1 illustrates an exemplary hardware platform on which the computers or computing devices described herein may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Exemplary embodiments related to supporting 2D, 3D or multi-view video coding and rendering using depth differences instead of motion vectors are described herein. Throughout the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are not described in exhaustive detail in order to avoid obscuring, obscuring or obscuring the present invention.

[0013] Exemplary embodiments are described according to the following outline. 1. Overview 2. Image and depth difference data server 3. Depth Map and Depth Difference Map 4. Deriving motion parameters 5. Exemplary Process Flow 6. Implementation mechanism – hardware overview 7. Equivalents, extensions, substitutions, etc.

[0014] 1. Overview This overview provides a basic description of some aspects of exemplary embodiments of the present invention. It is noted that this overview is not an extensive or exhaustive summary of the exemplary embodiments. Moreover, it is not intended to be understood as identifying any particular key aspects or elements of the exemplary embodiments, or as representing any scope of the exemplary embodiments in particular, or the present invention in general. This overview merely presents some concepts related to the exemplary embodiments in a summary and simplified form, and should be understood as merely a conceptual prelude to the more detailed description of the exemplary embodiments that follows. Although separate embodiments have been described herein, any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0015] To support two-dimensional (2D), three-dimensional (3D) or multi-view video streaming and rendering, the video signal may include a compressed version of a texture image and a compressed version of a depth image corresponding to the texture image. The texture image may be represented in a color space with multiple color channels, such as RGB color space, YCbCr color space, etc. The texture image may include an array of texture pixel values ​​(e.g., a two-dimensional array of image frames, etc.), each of which may include multiple component codeword values ​​in a color channel of the color space in which the texture image is represented.

[0016] Each depth image encoded in the video signal may include multiple depth values ​​for some or all of the pixels in the array of pixels represented in each texture image.

[0017] The texture images may be divided into scenes or groups of pictures, where the texture information within a scene or group of pictures may be relatively stable in time compared to the texture information across different scenes or different groups of pictures.

[0018] To generate a compressed version of a texture image in a video signal, predictive coding, including but not limited to inter-predictive coding, can be used to remove data redundancy between temporally adjacent texture images within a scene or group of pictures.

[0019] In some approaches that do not implement techniques as described herein, motion vectors may be generated or estimated from texture image content as indicated by texture pixel values ​​in a texture image coding operation. For example, the intensity, luma, chroma or color values ​​of texture pixel values ​​of pixels in two temporally adjacent texture images, which may be referred to as a first texture image and a second texture image, may be analyzed by a texture encoding operation in a video encoder to determine whether a first image / video block in the first texture image corresponds to a second image / video block in the second texture image.

[0020] If the texture pixel values ​​of pixels in the first and second image / video blocks from the first and two images share sufficient similarity in visual characteristics, as indicated by the intensity, luma, chroma or color value of the texture pixel values, a motion vector may be determined using the position of the pixel in the image / video block of the texture image (e.g., a combination of row and column) and the frame index of the texture image (each of which logically represents a point in time).

[0021] For example, given a texture pixel value of a pixel in a first image / video block in a first texture image, a texture pixel value of a pixel in a second image / video block in a second texture image may be inferred, predicted, or estimated based at least in part on a motion vector. A texture residual value, such as a texture pixel value prediction error or motion compensated difference information, may be calculated in a video encoding operation performed by a video encoder as a difference between a predicted texture pixel value of a pixel of the second image / video block in the second texture image and the (actual, non-predicted) texture pixel value of the pixel of the second image / video block in the second texture image.

[0022] As a result, the (actual, non-predicted) texture pixel values ​​of the pixels of the second image / video block in the second texture image are not directly encoded by the video encoder in the compressed version of the texture image, but rather some or all of the residual texture values, motion vectors, and reference information of the second image / video block in the second texture image for referencing the first image / video block in the first texture image are encoded into the compressed version of the texture image.

[0023] Thus, under those approaches that do not implement techniques such as those described herein, motion vectors generated or estimated from the texture pixel values ​​of a texture image are encoded into a compressed version of the texture image within the video signal. Similar or different compression methods can be implemented to compress the depth image into a compressed version of the depth image encoded within the video signal along with a compressed version of the texture image that includes the motion vectors.

[0024] In contrast, under techniques such as those described herein, the motion parameters used to generate the encoded texture image do not need to be transmitted together with the encoded texture image in the video signal. Rather, the motion parameters used to generate the encoded texture image can be generated or regenerated from a depth difference map generated by techniques such as those described herein by an upstream device, such as a video encoder that encodes the encoded image into a video signal, and reproduced by a downstream device receiving the video signal.

[0025] Furthermore, the depth information of the pixels represented by the coded texture image in the video signal can be generated or regenerated from the same depth difference map, both by the upstream device and by the receiving downstream device. As a result, a relatively large saving in bit rate can be obtained or achieved under the techniques as described herein, and the coded texture image can be carried together with the depth difference map in the video signal free from carrying the motion parameters. Furthermore, optionally or alternatively, further compression efficiency can be obtained or achieved by encoding a depth difference map with a relatively small numerical value comparable to the data size of the residual in the coded texture image, instead of directly encoding a depth map with a relatively large numerical value in the video signal. Furthermore, optionally or alternatively, processing accuracy and efficiency can be obtained, achieved or guaranteed under the techniques as described herein, by freeing the upstream and downstream devices from performing some or all of the image processing operations related to optimizing the motion parameters or the prediction-related operation parameters. Meanwhile, these techniques also enable a wide range of available video codecs to continue to perform MCP or motion vector prediction using motion parameters that can be derived indirectly from the depth difference map, rather than directly from the coded motion parameters in the video signal.

[0026] Exemplary embodiments described herein relate to encoding image content. A sequence of input texture images is received. A sequence of depth maps is determined for the sequence of input texture images. Each depth map in the sequence of depth maps includes depth values ​​for a number of pixels represented by a corresponding input texture image in the sequence of input texture images. A sequence of depth difference maps is generated from the sequence of depth maps. The depth difference maps include at least one depth difference map generated from two consecutive depth maps in the sequence of depth maps. A video signal is output that is encoded with the compressed versions of the input texture images and the sequence of depth difference maps. The video signal causes a receiving device of the video signal to generate a sequence of display images from the compressed versions of the sequence of input texture images and the sequence of depth difference maps for rendering on an image display.

[0027] Example embodiments described herein relate to decoding image content. A sequence of encoded images is obtained from a video signal. The sequence of encoded images represents a compressed version of a sequence of input texture images. The sequence of input texture images is used by an upstream device to generate a sequence of encoded images that are encoded into the video signal. A sequence of depth difference maps is obtained from the video signal. The sequence of depth difference maps has been generated by the upstream device from a sequence of depth maps determined for the sequence of input texture images. A sequence of display images derived from the sequence of encoded images and the sequence of depth difference maps is rendered on an image display.

[0028] In some exemplary embodiments, the mechanisms described herein form part of a media processing system, including but not limited to a cloud-based server, a mobile device, a virtual reality system, an augmented reality system, a head mounted display device, a helmet mounted display device, a cave-type system, a wall-sized display, a video game device, a display device, a media player, a media server, a media production system, a camera system, a home system, a communication device, a video processing system, a video codec system, a studio system, a streaming server, a cloud-based content services system, a handheld device, a gaming machine, a television, a cinema display, a laptop computer, a notebook computer, a tablet computer, a cellular wireless telephone, an e-book reader, a point of sale terminal, a desktop computer, a computer workstation, a computer server, a computer kiosk, or various other types of terminals and media processing units.

[0029] Various modifications to the preferred embodiments and generic principles and features described herein will be readily apparent to those skilled in the art, and thus the scope of the claims is not intended to be limited to the implementations shown in this disclosure, but is to be accorded the broadest scope consistent with the principles and features described herein.

[0030] 2. Image and depth difference data server 1A illustrates an exemplary image and depth difference data server 100, such as a video encoder device, a video streaming server, or an upstream device, including an input image receiver 104, an image and depth analyzer 106, an image MV and depth difference generator 108, an image and depth difference data encoder 110, etc. Some or all of the components of the image and depth difference server (100) may be implemented by one or more devices, modules, units, etc., in software, hardware, a combination of software and hardware, etc.

[0031] The input image receiver (104) includes software, hardware, a combination of software and hardware, etc., configured to receive or access a sequence of input (e.g., 2D, single-view, stereoscopic, multi-view, etc.) texture images 102 generated from an internal or external texture image source. The sequence of input texture images (102) depicts a visual scene at a sequence of time points covering a time period (e.g., a media program, etc.). Each input texture image in the sequence of input texture images (102) may be indexed by a respective unique frame index or index value representing a respective logical time instance / point, corresponding to a respective time point in the sequence of time points, and including an individual pixel value for each pixel in the array of pixels in the image frame. The individual pixel values ​​may include multiple component codewords for multiple color channels of a color space in which each input texture image in the sequence of input texture images (102) is represented.

[0032] For purposes of implementing predictive coding to reduce data redundancy in the video signal 112, the image and depth difference data server (100) may divide the sequence of (e.g., contiguous, sequential, mutually exclusive, etc.) input texture images (102) into (e.g., contiguous, sequential, mutually exclusive, etc.) GOPs, each GOP in the sequence of GOPs representing a group or collection of consecutive texture images that are encoded in the video signal (112) output by the image and depth difference data server (100).

[0033] Each GOP described herein includes or begins with an I-frame or image (e.g., an intra-coded picture, key frame, or i-frame, etc.) that is coded independently of all other texture images represented in the GOP or video signal (112). The GOP further includes one or more of texture images represented by P-frames, texture images represented by B-frames, other non-I-frames, etc. A P-frame (or predictively coded image) described herein includes, for example, residual or motion-compensated difference information that references at least one previously decoded (reference) texture image. A B-frame (or bi-predictively coded picture) described herein includes residual or motion-compensated difference information that references multiple (e.g., two) previously decoded (reference) texture images.

[0034] The image and depth analyzer (106) includes software, hardware, a combination of software and hardware, etc. configured to perform texture image analysis on a sequence of input texture images (102) and determine or estimate depths of pixels represented in the input texture images. For example, input texture images in a GOP and / or its adjacent GOPs in a sequence of GOPs may be analyzed to determine or estimate depths of pixels represented in the input texture images, or depths of a selected small number of pixels (e.g., every few frames, every I-frame, etc.) of the input texture images in the GOP and / or its adjacent GOPs.

[0035] In some operational scenarios, the depth of a pixel of an input texture image may be determined or estimated based at least in part on extracted information regarding blur (e.g., circle of confusion or COC, etc.) or deblurring, focus or defocus, stereo vision, and / or motion. The information used to determine or estimate the depth may be extracted from some or all of the texture pixel values ​​of the input texture image and / or texture pixel values ​​of temporally or spatially adjacent images represented in the sequence of input texture images (102).

[0036] A sequence of depth maps can be generated by the image and depth analyzer (106) for a sequence of input texture images (102) or a sequence of GOPs. In one example, the sequence of depth maps has a one-to-one correspondence with the sequence of input texture images (102). Each depth map in the sequence of depth maps corresponds to or specifies an individual depth for some or all of the pixels in a respective input texture image in the sequence of input texture images (102).

[0037] In another example, the sequence of depth maps does not have a one-to-one correspondence with the sequence of input texture images (102). Each depth map in the sequence of depth maps corresponds to or specifies individual depths of some or all of the pixels in a sequence of selective input texture images in the sequence of input texture images (102). The sequence of selective input texture images may be composed of specifically selected input texture images from the sequence of input texture images (102), e.g., represented as a sequence of I-frames (e.g., contiguous, sequential, etc.) in a sequence of GOPs encoded in the video signal (112). Thus, in this example, each I-frame in the sequence of I-frames represents a respective input texture image in the sequence of selective input texture images and corresponds to a respective GOP (e.g., a starting picture / image therein, etc.) in the sequence of GOPs. And each depth map in the sequence of depth maps corresponds to or includes individual depths of some or all of the pixels in a sequence of I-frames in the sequence of I-frames.

[0038] The image MV and depth difference generator (108) includes software, hardware, a combination of software and hardware, etc. configured to use a sequence of depth maps and / or a sequence of input texture images (102) to determine or identify one or more pixel sets of adjacent and co-moving pixels in each depth map in the sequence of depth maps.

[0039] Because the depth map specifies the depth of pixels at the same pixel location in the input texture image to which the depth map corresponds, pixel sets of adjacent and co-moving pixels determined or identified in the depth map have a one-to-one correspondence with one or more pixel sets of adjacent and co-moving pixels at the same pixel location in the corresponding input texture image. The one or more pixel sets, i.e., pixels having a specified depth in the depth map and a specified texture pixel value in the input texture image to which the depth map corresponds, can each represent one or more image features depicted in the input texture image and the depth map. Each pixel set in the one or more pixel sets can visually represent a respective image feature in the one or more image features when, for example, the input texture image and the depth map are used to generate a display image that is rendered on an image display.

[0040] A visual scene depicted in one or more (e.g., consecutive, sequential, etc.) GOPs in a sequence of GOPs described herein may include some or all of foreground image features visually representing foreground visual objects, background image features representing background visual objects, etc. A visual object or image feature described herein may refer to a visually perceptible form whose (1) spatial contour or boundary may be a visually perceptible irregular shape that is visually defined, outlined, or depicted by soft or hard edges (outermost boundary pixels) of the visual object or image feature, whose (2) texture may be defined or represented by texture pixel values ​​of pixels covered by the visual object or image feature, and whose (3) depth may be defined or represented by depth values ​​of pixels measured along a depth axis. The depth axis may be the z-axis of a three-dimensional Cartesian coordinate system having a reference point or origin coincident with a virtual or real camera that captures or generates a texture image including the visual object or image feature. Examples of image features depicting a visual object or at least a portion thereof may include, but are not limited to, some or all of a human character, an animated character or object, a vehicle, a road, a tree, a building, an architectural structure, an artistic object, an automobile, an airplane, a train, etc.

[0041] A visual object, or each of one or more image features depicting a visual object in an input texture image and / or a depth map corresponding to an input texture image, is not a regularly shaped logically defined image / video block (e.g., a slice, a video coding unit such as a quadtree node, a coding tree unit or CTU, etc.), such as a rectangle or square of pixels used in video coding (e.g., based on quadtrees, etc.), or a collection of regularly shaped image / video blocks (or slices, video coding units, quadtree nodes, CTUs, etc.) used in video coding. The visual object, its edges, and the texture of the visual object may be visually recognized, depicted, and distinguished from other image features depicted in the texture image by a viewer to whom the input texture image and / or depth map are rendered. In contrast, logical data units such as logically defined image / video blocks, slices, coding units, quadtree nodes, CTUs, etc. cannot be visually recognized, depicted, or distinguished by a viewer.

[0042] A visual object as described herein may be visually represented or depicted in a texture image belonging to a GOP by a single image feature or by multiple image features, each image feature of the image feature representing the visual object may be visually represented or depicted by a texture pixel value of a pixel in a corresponding pixel set of adjacent (or spatially adjacent) co-moving pixels in the input texture image and / or by a depth of the same pixel in the same pixel set in a depth map corresponding to the input texture image.

[0043] For example, a visual object may visually represent a human character, but one or more image features, such as a first image feature, a second image feature, a third image feature, etc., may visually represent portions of the visual object, such as a head, a right hand, a left hand, etc., of the human character, respectively. The first image feature visually representing the head of the human character in the texture image and / or corresponding depth map may include a first pixel set of adjacent (or spatially adjacent) co-moving pixels whose pixel values ​​in the texture image provide texture information for the head of the human character. However, if the pixels used to visually depict or represent the head are no longer co-moving (e.g., the human character has a visual expression on its face), then those pixels depicting the head of the human character may be determined or identified as being in different pixel sets, each of which includes adjacent (or spatially adjacent) co-moving pixels.

[0044] In some operating scenarios, similarity or dissimilarity of texture pixel values ​​of pixels in or between two or more successive input texture images in a GOP and / or preceding / subsequent GOPs, e.g., similar color values / patterns in spatially adjacent pixels, may be used to determine or identify pixel sets each including pixels of co-moving pixels or non-co-moving pixels in successive texture images. Additionally, optionally or alternatively, in some operating scenarios, mutually adjacent depth values ​​of depth values ​​of pixels in or between two or more successive depth maps in a GOP and / or preceding / subsequent GOPs, e.g., similar depth values ​​or relatively smoothly transitioning depth values ​​in spatially adjacent pixels, may be used to determine or identify pixel sets.

[0045] Once a set of adjacent (or spatially adjacent) co-moving pixels is identified in the depth map corresponding to the input texture image, a set of adjacent (or spatially adjacent) co-moving pixels is also identified in the input texture image within the GOP, since the depth pixels represented in the depth map have a one-to-one correspondence with the corresponding texture pixels represented in the input texture image. For example, a depth pixel in a pixel array in the depth map has the same row / column index or array position as a texture pixel in a corresponding pixel array in the texture image. Conversely, once a set of adjacent (or spatially adjacent) co-moving pixels is identified in the input texture map, a set of adjacent (or spatially adjacent) co-moving pixels is also identified in the depth map corresponding to the input texture image.

[0046] In some operating scenarios, a set of pixels is determined or identified (e.g., explicitly, etc.) in an I-frame that starts a GOP or in a depth map corresponding to the I-frame. A set of pixels in other types of frames in the GOP may not be determined or identified explicitly, but may be estimated from the spatial or planar motion or displacement of the set of pixels that is determined or identified explicitly in the I-frame. For example, a first depth map for a first I-frame that starts a first GOP and a second depth map for a second I-frame that starts a second GOP that follows (e.g., directly, etc.) the first GOP may be used to determine the spatial or planar motion or displacement between corresponding sets of pixels shared in the first and second GOPs. These motions or displacements may be used to calculate or estimate (e.g., via interpolation, extrapolation, etc.) the locations of corresponding sets of pixels for other types of frames in the first or second GOP.

[0047] By way of example, but not limitation, the image MV and depth difference generator (108) can determine, for each pixel set in one or more pixel sets determined or identified in or between two temporally adjacent input texture images or two corresponding temporally adjacent depth maps, a planar displacement of pixels in the pixel set between the two temporally adjacent texture images (or the two temporally adjacent depth maps). In one example, the two temporally adjacent texture images (or the two temporally adjacent depth maps) may be entirely within a single GOP. In another example, one of the two temporally adjacent texture images (or one of the two temporally adjacent depth maps) may be a first I-frame that starts a first GOP, while the other of the two temporally adjacent texture images (or the other of the two temporally adjacent depth maps) may be a second I-frame that starts a second GOP that follows (e.g., directly, etc.) the first GOP.

[0048] The planar displacement of a set of pixels may represent the (two-dimensional) change in position of the set of pixels in an image plane perpendicular to the depth or z-axis within or between two time-adjacent texture images (or two time-adjacent depth maps), where the image plane may represent the screen or canvas of an image display onto which the texture images / depth maps are rendered. The image plane may be represented as an xy plane along which the pixels in the texture images or depth maps are arranged or rendered, while the depth of the pixels may be represented along the z-axis perpendicular to the xy plane.

[0049] Because the pixels in the pixel set move together (e.g., with the same planar velocity, etc.), the image MV and depth difference generator (108) can obtain or calculate the planar displacement of the pixels in the pixel set as a (vector) difference (e.g., row difference, column difference, a combination of row difference and column difference, etc.) between a first planar (row and column) position of a representative or group position (e.g., symmetric position, eccentric position, average pixel position of some or all pixels, etc.) of the pixel set in a first texture image of the two time-adjacent texture images (or a corresponding first depth map of the two time-adjacent depth maps) and a second planar (row and column) position of the same representative or group position of the same pixel set in a second texture image of the two time-adjacent texture images (or a corresponding second depth map of the two time-adjacent depth maps). For example, a first planar location of the representative or group location may be calculated as a first average row location generated by averaging all row locations in a first texture image (or first depth map) of the pixels in the pixel set, and a first average column location generated by all column locations in the first texture image (or first depth map) of the pixels in the pixel set. Similarly, a second planar location of the representative or group location may be calculated as a second average row location generated by averaging all row locations in a second texture image (or second depth map) of the pixels in the pixel set, and a second average column location generated by all column locations in the second texture image (or second depth map) of the pixels in the pixel set. To maintain accuracy of the motion compensation prediction or interpolation, fractional or floating point sample / pixel location values ​​(e.g., quarter sample accuracy, etc.) may be used to represent some or all of the representative or group location / location, average row location, average column location, planar displacement, etc.

[0050] Once the planar displacement is determined for a set of pixels between two adjacent images or depth maps, the rate of planar displacement or velocity can be calculated for the set of pixels as the planar displacement divided by the difference between the frame indexes of the two time-adjacent images or depth maps, since the frame index represents a (logical) time instance of each of the two time-adjacent images or depth maps.

[0051] 3. Depth Map and Depth Difference Map 2A shows two exemplary depth maps constructed from depths (or depth values) for pixels in two corresponding input texture images in a sequence of input texture images. For purposes of example only, a first depth map 202-1 is constructed from a first depth for a first pixel in the first input texture image, while a second depth map 202-2 is constructed from a second depth for a second pixel in the second input texture image.

[0052] In some operating scenarios, the first input texture image and the second input texture image may belong to the same GOP, such as two temporally adjacent input texture images or two input texture images separated by one or more intervening input texture images. In some operating scenarios, the first input texture image and the second input texture image may belong to different GOPs in a sequence of input texture images, such as two I-frames starting two temporally adjacent GOPs.

[0053] 2A, the (depth) pixels represented in each of the depth maps (202-1 and 202-2) or the (texture) pixels represented in the corresponding input texture images may be arranged in a particular two-dimensional (2D) array pattern corresponding to a particular spatial resolution supported by the sequence of input texture images. The pixel sets 212 may be identified in the first depth map (202-1) and the second depth map (202-2) by, for example, similarity or dissimilarity of texture color or pattern between spatially adjacent pixels, mutually adjacent or relatively smoothly varying depth between spatially adjacent pixels, etc.

[0054] The set of pixels (212) may be at a first spatial location indicated by a first combination of rows and columns in the first depth map (202-1) or first input texture image, and the set of pixels (212) may be at a second (displaced) spatial location indicated by a second combination of rows and columns in the second depth map (202-2) or second input texture image.

[0055] By way of example and not limitation, a set of pixels (212) in a first depth map (202-1) or first input texture image may be composed of pixels (or previously mentioned first pixels) whose texture values ​​in the first input texture image and depths in the first depth map (202-1) indicate a particular image feature such as a human eye. Similarly, a set of pixels (212) in a second depth map (202-2) or second input texture image may be composed of pixels (or previously mentioned second pixels) whose texture values ​​in the second input texture image and depths in the second depth map (202-2) indicate the same particular image feature or human eye.

[0056] A first pixel in the pixel set (212) of the first depth map (202-1) has a one-to-one correspondence with a second pixel in the pixel set (212) of the second depth map (202-1), but the first pixel and the second pixel indicate the same specific image feature at two different time instances / points. For example, the specific image feature may include a specific visually recognizable image feature such as a corner of an eye. The specific visually recognizable image feature or the corner of an eye may be indicated by a first pixel at a first pixel location (x, y, z) in the first depth map (202-1) or a first input texture image corresponding to the first depth map (202-1). Similarly, the same specific visually recognizable image feature or the corner of an eye may be indicated by a second pixel at a second pixel location (x', y', z') in the second depth map (202-2) or a second input texture image corresponding to the second depth map (202-2). The second pixel in the second depth map (202-1) has a one-to-one correspondence with the first pixel in the first depth map (202-1) because the first and second pixels indicate the same specific visually recognizable image feature or corner of the eye, even at two different time instances / points.

[0057] A planar displacement of the pixel set (212) from a first time instance / point in the first depth map (202-1) to a second time instance / point in the second depth map (202-2) may be determined or represented as the (planar) difference (shown as a combination of Δx and Δy) between some or all pairs of corresponding pixels in the pixel set (212) that have a one-to-one correspondence. For example, the planar displacement may be determined or represented as the (planar) difference Δx and Δy between pairs of first and second pixels (x, y, z) and (x', y', z') that have a one-to-one correspondence, as follows:

number

[0058] A depth difference between a first and second pixel in a pair of pixels having one-to-one correspondence in the first depth map (202-1) and the second depth map (202-2) may be determined or expressed as the difference between the depths of the first and second pixels along the depth direction or z-axis with the reference view / camera position as the origin of an xyz coordinate system. For example, a depth difference Δz between a first pixel (x, y, z) in the set of pixels (212) of the first depth map (202-1) and a second pixel (x', y', z') in the set of pixels (2120) of the second depth map (202-2) may be determined between depths z and z' as follows:

number

[0059] The image MV and depth difference generator (108) of FIG. 1A can store planar displacements and depth differences derived from the sequence of depth maps and / or the sequence of input texture images in a sequence of depth difference maps.

[0060] 2B shows two exemplary depth difference maps (or depth diff maps) composed of depth differences (or depth value differences) as well as planar differences of pixels in two corresponding input texture images in a sequence of input texture images. For purposes of example only, a first depth difference map 222-1 is composed of a first depth difference and a first planar displacement for a first pixel in a first input texture image, and a second depth difference map 222-2 is composed of a second depth difference and a second planar displacement for a second pixel in a second input texture image.

[0061] The first depth difference map (222-1) may be generated to store a first depth difference and a first planar displacement generated from the first depth map (202-1) and the second depth map (202-2) and / or generated from the first input texture image and the second input texture image to which the first depth map (202-1) and the second depth map (202-2) respectively correspond. The second depth difference map (222-2) may be generated to store a second depth difference and a second planar displacement generated from the second depth map (202-2) and a third depth map (not shown) subsequent to the second depth map (202-2) and / or generated from the second input texture image and the third input texture image to which the second depth map (202-2) and the third depth map respectively correspond.

[0062] As shown in FIG. 2B, the (depth difference) pixels represented in each of the depth difference maps (222-1 and 222-2) may be arranged in the same manner as in the corresponding input texture image, such as in a particular two-dimensional (2D) array pattern corresponding to a particular spatial resolution supported by the sequence of input texture images.

[0063] Since the pixel set (212) is composed of pixels of co-moving pixels, a planar displacement calculated from one or more representative image feature portions of a particular image feature represented in the pixel set (212) can represent a global planar displacement for all pixels in the pixel set (212). Thus, in some implementations, this planar displacement does not need to be repeated by each and every pixel in the pixel set (212). In these implementations, for data compression efficiency, only a single planar displacement can be used to represent the same planar displacement of all pixels in the same pixel set of co-moving pixels, such as the pixel set (202). For example, in the first depth difference map (222-1), a planar displacement represented by a combination of Δx and Δy can be used to represent the same planar displacement of all pixels in the first depth difference map (222-1) or the pixel set (212) of the first input texture image, although these pixels in the first depth difference map (222-1) may have different depth differences (or different Δz values) between them. Similarly, in the second depth difference map (222-2), the planar displacement represented by a combination of Δx and Δy can be used to represent the same planar displacement of all pixels in the second depth difference map (222-2) or the pixel set (212) of the second input texture image, but these pixels in the second depth difference map (222-2) may have different depth differences (or different Δz values) between them. As a result, although a depth difference map as described herein includes both planar displacements and depth differences, the total amount or amount of data of the depth difference map after compression / encoding can be comparable to the total amount or amount of data of the corresponding depth map after compression / encoding. Since difference values ​​tend to be smaller in value than non-difference values, data redundancy, and therefore compression efficiency, is more likely to be based on or achieved by difference values ​​in the depth difference map than by non-difference depth values ​​in the depth difference map.

[0064] The image and depth difference encoder (110) of FIG. 1A includes software, hardware, a combination of software and hardware, etc., configured to process and encode a sequence of input texture images (102) and a sequence of depth difference maps into a video signal (112).

[0065] As mentioned above, a sequence of (e.g., contiguous, sequential, etc.) input texture images (102) or their encoded / compressed versions may be divided, for example along the playback time direction, into a sequence of (e.g., contiguous, sequential, mutually exclusive, etc.) GOPs, each of which includes a group of pictures that is a group of coded images (e.g., intra-predicted images, inter-predicted images, etc.) derived from or corresponding to a respective group of input texture images in the sequence of input texture images.

[0066] Each GOP in the sequence of GOPs begins with a respective I-frame (or intra-predicted image) in the sequence of I-frames and ends with the frame before the next I-frame that starts or begins the next GOP in the sequence of GOPs (or an encoded / compressed version of the corresponding input texture image in the sequence of input texture images).

[0067] 4. Deriving motion parameters 3A and 3B show two exemplary GOPs, a first GOP 302-1 and a second (or next) GOP 302-2 immediately following the first GOP (302-1). As shown in FIG. 3A, the first GOP (302-1) includes a group of coded images corresponding to a first group of input texture images in a sequence of input texture images (102), beginning with a first I-frame 304-1, followed by a combination of zero or more first P-frames (e.g., 306-1, etc.) and / or zero or more first B-frames (e.g., 308-1, etc.), until the next GOP (after the first GOP (302-1)) or the frame immediately preceding the second I-frame (after the first GOP (304-1)) that starts the second GOP (302-2). As shown in FIG. 3B, the second GOP (302-2) includes a group of coded images corresponding to the second group of input texture images that follow the first group of input texture images in the sequence of input texture images (102), beginning with a second I-frame (304-2), followed by a combination of zero or more second P-frames (e.g., 306-2, etc.) and / or zero or more second B-frames (e.g., 308-2, etc.), up to the frame immediately preceding the third I-frame, which is the next I-frame (after the second GOP (304-2)) that starts the next GOP (after the second GOP (302-2)).

[0068] Each coded image in a GOP can be generated by the image and depth difference encoder (110) of FIG. 1A from each input texture image in the group of input texture images corresponding to the GOP. A quadtree structure (e.g., nested quadtrees, etc.) can be adaptively determined or used to partition or divide a coded image into multiple rectangular image blocks of various sizes or levels of subdivision. An image block as described herein can refer to an image macroblock, an image block, an image sub-block, a coding tree unit, a coding tree block, a coding tree node, etc. The quadtree structure can be specified or defined according to one or more normative or informative passages described in a standard-based or proprietary video coding standard (e.g., H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc.).

[0069] A quadtree structure for the coded images of a GOP may be selected using an adaptive quadtree selection algorithm based on one or more quadtree structure selection factors or tradeoffs thereof. Exemplary quadtree structure selection factors may relate to some or all of, but are not necessarily limited to, maximizing compression or coding efficiency, reducing bitrate usage, minimizing coding or prediction errors, reducing visual artifacts, minimizing transmission and / or processing delays, etc.

[0070] The image blocks constituting the coded image may include two types of blocks: predictive blocks and transform blocks. The predictive blocks described herein may include samples generated from intra-prediction of previously decoded samples in the coded image, or samples generated from inter- or motion-compensated prediction (MCP) of previously decoded samples of reference image blocks in the reference image. The transform blocks described herein may include non-predictive sample values ​​or samples having transform coefficients (such as digital cosine transform or DCT coefficients). The difference or residual between the predicted sample values ​​from inter- or intra-prediction in the predictive blocks of the coded image and the sample values ​​derived from the corresponding image blocks of the respective input texture images may be coded into the video signal (112) in the residual image blocks of the residual image corresponding to the predictive blocks of the coded image.

[0071] The prediction block or inter-prediction block generated by MCP or motion vector prediction can apply motion compensation prediction using one or more sets of motion parameters, each of which can include a reference image / frame index and a motion vector. Thus, for an image (e.g., P frame, B frame, etc.) having a large number of inter-prediction blocks (e.g., tens of thousands per image) located in different spatial regions of the image, a large number of sets of motion parameters representing the motion vector fields in these different spatial regions of the image are used to perform image processing operations for MCP or motion prediction.

[0072] Under other MCP or inter-prediction approaches that do not implement the techniques described herein, a large set of motion parameters is encoded or conveyed in the video signal, for example using lossless data compression, solely for encoding the texture image in the video signal. In addition, a depth map of the texture image is also encoded in the video signal to provide depth information for pixels represented in the texture image. As a result, a relatively large bit rate is used to convey both the motion parameters and the depth information of the texture image in the video signal.

[0073] In contrast, the motion parameters used to generate the coded texture image do not need to be transmitted together with the coded texture image in the video signal (e.g., 112 in FIG. 1A). Rather, the motion parameters used to generate the coded texture image can be generated or regenerated from a depth difference map as described herein by an upstream device, such as a video encoder that encodes the coded image into the video signal (112), and also by a downstream device receiving the video signal (112). Furthermore, depth information for pixels represented by the coded texture image in the video signal (112) can be generated or regenerated from the same depth difference map, both by the upstream device and by the downstream device receiving the video signal (112). As a result, relatively large savings in bit rate can be obtained or achieved under the techniques as described herein, allowing the coded texture image to be carried together with a depth difference map in the video signal (112) freed from carrying motion parameters. Further, optionally or alternatively, further compression efficiency may be obtained or achieved by encoding a depth difference map with a relatively small value comparable to the data size of the residual in the encoded texture image, instead of directly encoding a depth map with a relatively large value into the video signal (112). Further, optionally or alternatively, processing accuracy and efficiency may be obtained, achieved or guaranteed under the techniques as described herein, by relieving upstream and downstream devices from performing some or all of the image processing operations related to optimizing the motion parameters or prediction-related operation parameters. Meanwhile, these techniques also enable a wide range of available video codecs to continue to perform MCP or motion vector prediction using motion parameters that may be derived indirectly from the depth difference map, rather than directly from the coded motion parameters in the video signal (112).

[0074] As shown in FIG. 3A, the first GOP (302-1) includes a first coded image represented by a first P frame 306-1 and a second coded image represented by a first B frame 308-1. The first P frame (306-1) may include a first plurality of image blocks coded according to a first quadtree structure for the first P frame (306-1), and the first B frame (308-1) may include a second plurality of image blocks coded according to a second quadtree structure for the first B frame (308-1). The first plurality of image blocks in the first P frame (306-1) may include a first inter-predicted image block whose samples are predicted based on a part or all of a first motion vector field identified by a first MV 308-1, and the second plurality of image blocks in the first B frame (308-1) may include a second inter-predicted image block whose samples are predicted based on a part or all of a second motion vector field identified by a second MV 308-2.

[0075] 3B, the second GOP (302-2) includes a third encoded image represented by a second P frame 306-2, a fourth encoded image represented by a second B frame 308-2, etc. The second P frame (306-2) may include a third plurality of image blocks encoded according to a third quadtree structure for the second P frame (306-2), and the second B frame (308-2) may include a fourth plurality of image blocks encoded according to a fourth quadtree structure for the second B frame (308-2). The third plurality of image blocks in the second P frame (306-2) may include a first inter-predicted image block whose samples are predicted based on part or all of a third motion vector field identified by a third MV 308-3, and the fourth plurality of image blocks in the second B frame (308-2) may include a fourth inter-predicted image block whose samples are predicted based on part or all of a fourth motion vector field identified by a fourth MV 30842.

[0076] Under the technique described herein, a set of motion parameters such as a motion vector and / or a reference image / frame index can be generated from the depth difference map. For example, the first MV (308-1) and the second MV (308-2) as shown in FIG. 3A may be generated from one or both of the first depth difference map (222-1) generated for the first I-frame (304-1) of the first GOP (302-1) and the second depth difference map (222-2) generated for the second I-frame (304-2) of the second GOP (302-2). Similarly, the third MV (308-3) and the fourth MV (308-4) as shown in FIG. 3B may be generated from one or both of the third depth difference map (222-3) and the second depth difference map (222-2) generated for the third I-frame 304-3 of the third GOP (not shown) following the second GOP (302-2).

[0077] In some operating scenarios, the image and depth difference encoder (110) of FIG. 1A can determine a displaced pixel set of adjacent and co-moving pixels in a first P frame (306-1) using a pixel set of adjacent and co-moving pixels and a planar displacement of the pixel set, as shown in a first depth difference map (222-1) for a first I frame (304-1).

[0078] In response to determining the set of displaced pixels in the first P frame (306-1), the image and depth difference encoder (110) of FIG. 1A may further determine whether each of some or all of the image blocks of the first plurality of image blocks, or sample values ​​therein, can be predicted from decoded image blocks, or decoded sample values ​​therein, in the pixel set of the first I frame (304-1).

[0079] For purposes of example only, the image and depth difference encoder (110) of Figure 1A determines that a particular image block in the first plurality of image blocks, or sample values ​​therein, can be predicted from a particular decoded image block in a pixel set (212) of a first I-frame (304-1), or decoded sample values ​​therein, as shown in Figure 2B. Because the pixel set (212) has a planar displacement (Δx, Δy), the particular image block in the pixel set (212) also has the same planar displacement (Δx, Δy).

[0080] The planar displacement (Δx, Δy) of a particular image block in the pixel set (212) represents the spatial position difference between a first spatial position of a representative pixel (e.g., the top leftmost pixel) of the particular image block at a first time point indicated by a first frame index of the first I-frame (304-1) and a second spatial position of the same representative pixel (e.g., the top leftmost pixel) of the particular image block at a second time point indicated by a second frame index of the second I-frame (304-2).

[0081] The image and depth difference encoder (110) of Figure 1A can use the planar displacements (Δx, Δy) of particular image blocks shown in the first depth difference map (222-1) as motion vectors in the MV (308-1) relative to the first P frame (306-1). Additionally, optionally or alternatively, an interpolated version of the planar displacements (Δx, Δy) can be derived, specifically for the first P frame (306-1), as motion vectors in the first MV (308-1) relative to the first P frame (306-1). The interpolated version of the planar displacement specific to the first P frame (306-1) is obtained by interpolating based on at least some or all of the first frame index of the first I frame (304-1) in the first GOP (302-1), the second frame index of the second I frame (304-2) in the second GOP (302-2), the frame index of the first P frame (306-1) in the first GOP (302-1), etc.

[0082] 1A can use the planar displacement (Δx, Δy) of a particular image block shown in the first depth difference map (222-1) as a motion vector in the second MV (308-2) relative to the first B frame (308-1). Additionally, optionally or alternatively, an interpolated version of the planar displacement (Δx, Δy) can be derived, specifically for the first B frame (308-1), as a motion vector in the MV (308-2) relative to the first B frame (308-1). The interpolated version of the planar displacement specific to the 1st B frame (308-1) is obtained by interpolating based on at least some or all of the first frame index of the 1st I frame (304-1) in the 1st GOP (302-1), the second frame index of the second I frame (304-2) in the 2nd GOP (302-2), the frame index of the 1st B frame (308-1) in the 1st GOP (302-1), etc.

[0083] In some operating scenarios, to support video codecs (e.g., legacy, already deployed, etc.) that operate using or relying on motion vectors or motion parameters defined in a particular standard-based or proprietary video coding specification, the image and depth difference encoder (110) of FIG. 1A can generate these motion vectors or motion parameters in a particular format or representation defined in the particular video coding specification from the planar displacements of a set of pixels in the depth difference map, enabling these video codecs to perform MCP or motion vector prediction operations in dependence on the motion vectors or motion parameters of these formats or representations. Additionally, and optionally or alternatively, the image and depth difference encoder (110) of FIG. 1A can generate a video signal using a video coding syntax, and include in the video signal motion vectors and / or motion parameters generated or derived from the planar displacements of a set of pixels in the depth difference map.

[0084] In some operating scenarios, to support video codecs that do not rely on (e.g., updated, modified, extended, etc.) motion vectors or motion parameters defined in a particular standard-based or proprietary video coding specification, the image and depth difference encoder (110) of FIG. 1A does not generate these motion vectors or motion parameters in a particular format or representation defined in a particular video coding specification from the planar displacements of a set of pixels in a depth difference map. These video codecs can directly or indirectly use the planar displacements of a set of pixels in a depth difference map to perform MCP or motion vector prediction operations and do not need to generate motion vectors or motion parameters in a particular format or representation defined in a coding specification. In one example, some or all video codecs can directly use the planar displacements of a depth difference map to perform MCP or motion vector prediction operations. In another example, some or all video codecs can use the planar displacements of a depth difference map to generate motion vectors or motion parameters in an internal or informal format (e.g., with a precision greater than integer precision as specified in a standard, etc.) and use these motion vectors or motion parameters in an internal or informal format to perform MCP or motion vector prediction operations.

[0085] Further, optionally or alternatively, the image and depth difference encoder (110) of FIG. 1A can generate a video signal such as 112 of FIG. 1A using a video coding syntax and not include some or all of the motion vectors and / or motion parameters generated or derived from the planar displacement of a set of pixels in the depth difference map in the video signal (112).

[0086] As used herein, video content within a video signal (or stream) as described herein may include, but is not necessarily limited to, any of audiovisual programs, movies, video programs, television broadcasts, computer games, augmented reality (AR) content, virtual reality (VR) content, automotive entertainment content, and the like.

[0087] As used herein, an "image and depth difference data server" may refer to one or more upstream devices, video encoders, and / or video streaming servers that prepare and stream video content to one or more video streaming clients, such as video decoders, for rendering at least a portion of the video content on one or more displays. The displays on which the video content is rendered may be part of or may work in conjunction with the one or more video streaming clients.

[0088] Examples of image and depth difference data servers include, but are not limited to, cloud-based image and depth difference data servers located remotely from the video streaming client, video streaming servers connected to the video streaming client via a local wired or wireless network, VR devices, AR devices, automotive entertainment devices, digital media devices, digital media receivers, set-top boxes, game consoles (e.g., Xbox), general-purpose personal computers, tablets, and dedicated digital media receivers such as AppleTV and Roku boxes.

[0089] The image and depth difference data server (100) can be used to support AR applications, VR applications, 360 degree video applications, stereoscopic video applications, real-time video applications, near real-time video applications, non-real-time omnidirectional video applications, automotive entertainment, helmet mounted display applications, head mounted display applications, games, 2D display applications, 3D display applications, multi-perspective display applications, and the like.

[0090] 1B illustrates an exemplary downstream receiving device, such as a video client device 150, including an image and depth difference receiver 114, an image MV generator 116, a target display image renderer 118, a target image display 120, etc. Some or all of the components of the video client device (150) may be implemented by one or more devices, modules, units, etc., in software, hardware, a combination of software and hardware, etc.

[0091] Exemplary video client devices described herein may include, but are not limited to, any of the following: large screen image displays, home entertainment systems, set-top boxes and / or audiovisual devices operating in conjunction with image displays, mobile computing devices handheld by a user / viewer (e.g., in a spatially stable or changing relationship with the user / viewer's eyes, etc.), wearable devices that include or operate in conjunction with image displays, computing devices that include or operate in conjunction with head-mounted or head-up displays, etc.

[0092] The video streaming client (150) or the image and depth difference receiver (114) therein includes software, hardware, a combination of software and hardware, etc., configured to receive a video signal or bitstream (e.g., 112, etc.) that has been encoded with a sequence of encoded images (e.g., contiguous, sequential, logically ordered by frame index, etc.) and a sequence of depth difference maps (e.g., contiguous, sequential, logically ordered by frame index, etc.) that correspond to the sequence of encoded images. The sequence of encoded images may include a sequence of GOPs or have been partitioned by an upstream device (e.g., 100 in FIG. 1A) into a sequence of GOPs, with each GOP beginning with an I-frame (or intra-prediction coded image) in a sequence of I-frames, followed by one or more P-frames or B-frames (or one or more inter-prediction coded images).

[0093] In some operating scenarios, each depth difference map in the sequence of depth difference maps corresponds to a respective coded image in the sequence of coded images and has been generated by an upstream device (100). In some operating scenarios, each depth difference map in the sequence of depth difference maps corresponds to a respective I-frame in the sequence of I-frames and has been generated by an upstream device.

[0094] In some operation scenarios, the upstream device (100) may split the sequence of depth difference maps into multiple (e.g., consecutive, sequential, mutually exclusive, etc.) subsequences of depth difference maps. For example, the multiple subsequences of depth difference maps may have a one-to-one mapping relationship with the multiple (e.g., consecutive, sequential, mutually exclusive, etc.) subsequences of GOPs split from the sequence of GOPs. Each subsequence of depth difference maps in the multiple subsequences of depth difference maps may correspond to a respective subsequence of GOPs in the multiple GOPs and may depict the same or similar visual scenes with relatively slow changes in planar displacement of pixel sets represented in the subsequences of depth difference maps.

[0095] Each subsequence of depth difference maps may include, for example, multiple depth difference maps corresponding to multiple GOPs in each subsequence of GOPs. The multiple depth difference maps in the subsequence of depth difference maps may include a particular depth difference map, such as the first depth difference map in the subsequence of depth difference maps. When encoded into the video signal (112), a particular depth difference map in the subsequence of depth difference maps may convey or include a combination of a planar displacement of a set of pixels and a depth of a pixel in the set of pixels relative to a corresponding coded image derived from a corresponding input texture image (e.g., an I-frame, such as the first I-frame in the subsequence of I-frames corresponding to each GOP). Each other depth difference map in the subsequence of depth difference maps may convey or include a combination of another planar displacement of another set of pixels and a depth difference of another pixel in another set of pixels with respect to the depth conveyed in the particular depth difference map for another corresponding coded image derived from another corresponding input texture image (e.g., another I-frame, such as the second or subsequent I-frame in the subsequence of I-frames corresponding to each GOP).

[0096] The receiving device (150) can use a depth in a particular depth difference map in a subsequence of depth difference maps as a reference to derive another depth for the next other corresponding encoded image by adding the depth difference in the next other depth difference map to the reference or depth in the particular depth difference map. The other depth can then be set as a new or current reference for the second depth difference map to derive the second other depth. This process can be repeated for all other difference maps in the same subsequence of depth difference maps. As a result, the downstream receiving device (150) can recover all depths of pixels represented in the received depth difference map. These depths can be used in 2D or 3D image processing and / or rendering operations, including image warping operations, stereoscopic image rendering, multi-view rendering, immersive image display applications, AR or VR image rendering, etc.

[0097] The image MV generator (116) includes software, hardware, a combination of software and hardware, etc., configured to perform interpolation and / or extrapolation operations to generate motion vectors or motion parameters from a depth difference map for a sequence of encoded images in the same manner as the same motion vectors or motion parameters are generated by the upstream device (100) as part of generating and encoding a sequence of encoded images or a sequence of GOPs into the video signal (100). The image MV generator (116) can use these motion vectors or motion parameters to perform MCP or motion vector prediction operations to generate predicted sample values ​​for image blocks in the encoded images. These predicted sample values ​​can be added with residuals decoded from the video signal (112) for these image blocks to generate reconstructed image blocks of a sequence of reconstructed images (e.g., consecutive, sequential, etc.).

[0098] The target display image renderer (118), which may include software, hardware, a combination of software and hardware, etc., is configured to receive a sequence of reconstructed images, generate a sequence of display images (e.g., consecutive, sequential, etc.) from the sequence of reconstructed images, and render the sequence of display images on the target image display (120) for or at multiple time points each covered by the sequence of encoded images in the video signal (112).

[0099] For purposes of example only, it has been described that depth information, such as individual depths of some or all pixels represented in an input texture image, may be obtained or generated by an upstream device (e.g., 100 in FIG. 1A ) by performing image analysis operations on the input texture image or adjacent input texture images. It should be noted that in other embodiments, the depth information may be provided to an upstream device (e.g., 100-1) along with a sequence of input texture images (e.g., 104) and a sequence of input depth maps (e.g., 122) corresponding to the sequence of input texture images (104) to perform depth difference related image processing operations as described herein, including generating an encoded video signal (e.g., 112) with a sequence of encoded images and a sequence of depth difference maps for the sequence of encoded images, as shown in FIG. 1C .

[0100] 5. Exemplary Process Flow 4A illustrates an exemplary process flow according to an exemplary embodiment of the present invention. In some implementations, one or more computing devices or components may perform this process flow. At block 402, an image processing device (e.g., an upstream device, an encoder device, a transcoder, a media streaming server, etc.) receives a sequence of input texture images.

[0101] At block 404, the image processing device determines a sequence of depth maps for the sequence of input texture images, where each depth map in the sequence of depth maps includes depth values ​​for multiple pixels represented in a corresponding input texture image in the sequence of input texture images.

[0102] In block 406, the image processor generates a sequence of depth difference maps from the sequence of depth maps, the sequence of depth difference maps including at least one depth difference map generated from two consecutive depth maps in the sequence of depth maps.

[0103] In block 408, the image processing device outputs a video signal encoded with compressed versions of the sequence of input texture images and the sequence of depth difference maps, which video signal enables a receiving device of the video signal to generate a sequence of display images from the compressed versions of the sequence of input texture images and the sequence of depth difference maps for rendering on an image display.

[0104] In one embodiment, the sequence of input texture images is partitioned into a sequence of group of pictures (GOPs), and each depth difference map in the sequence of depth difference maps is generated for a respective GOP in the sequence of GOPs.

[0105] In one embodiment, the depth difference map includes a depth value difference between a first depth value in a first I-frame of each GOP and a second depth value in a second I-frame of a subsequent GOP that follows each GOP in a sequence of GOPs.

[0106] In one embodiment, the plurality of pixels of the corresponding input texture image includes one or more pixel sets of adjacent and co-moving pixels.

[0107] In one embodiment, the sequence of depth difference maps includes a particular depth difference map including one or more planar displacements for one or more sets of pixels within the plurality of pixels, each planar displacement in the one or more planar displacements being for a respective set of pixels within the one or more sets of pixels.

[0108] In one embodiment, the compressed version of the sequence of input texture images comprises a plurality of inter-predictive coded images having one or more inter-predictive image blocks, samples in each of the one or more inter-predictive image blocks being predicted based on motion parameters generated from planar displacements contained in at least one depth difference map in the sequence of depth difference maps.

[0109] In one embodiment, the video signal does not include the motion parameters used in the inter-predicted image blocks in the compressed version of the sequence of input texture images.

[0110] In one embodiment, the video signal does not include the sequence of depth maps that are used to generate the sequence of depth difference maps.

[0111] In one embodiment, the sequence of depth maps is generated by image analysis operations performed on the sequence of input texture images by an upstream device that receives the sequence of input texture images.

[0112] In one embodiment, the sequence of depth maps is received by an upstream device that receives a sequence of input texture images.

[0113] 4B illustrates an exemplary process flow according to an exemplary embodiment of the invention. In some implementations, one or more computing devices or components may perform this process flow. At block 452, a receiving device obtains a sequence of encoded images from a video signal. The sequence of encoded images represents a compressed version of a sequence of input texture images that have been used by an upstream device to generate a sequence of encoded images that are encoded into the video signal.

[0114] At block 454, the receiving device obtains from the video signal a sequence of depth difference maps that were generated by an upstream device from a sequence of depth maps determined for the sequence of input texture images.

[0115] At block 456, the receiving device causes the sequence of display images derived from the sequence of encoded images and the sequence of depth difference maps to be rendered on an image display.

[0116] In one embodiment, the video signal is coded with residual values ​​of inter-predicted image blocks in a plurality of inter-predictive coded images in a sequence of coded images.

[0117] In one embodiment, motion parameters for predicting samples in an inter-predicted image block are generated from a sequence of depth difference maps.

[0118] In various example embodiments, a device, system, apparatus, or one or more other computing devices performs any or part of the methods described above. In an embodiment, a non-transitory computer-readable storage medium stores software instructions that, when executed by one or more processors, cause execution of the methods described herein.

[0119] Although separate embodiments have been described herein, any combination of the embodiments and / or sub-embodiments discussed herein may be combined to form further embodiments.

[0120] 6. Implementation mechanism – hardware overview According to one embodiment, the communications described herein are implemented by one or more special purpose computing devices. The special purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) permanently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such special purpose computing devices may combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, network devices, or any other devices incorporating hard-wired and / or program logic to implement the techniques.

[0121] For example, Figure 5 is a block diagram illustrating a computer system 500 in which an exemplary embodiment of the invention may be implemented. Computer system 500 includes a bus 502 or other communication mechanism for communicating information, and a hardware processor 504 coupled to bus 502 for processing information. Hardware processor 504 may be, for example, a general-purpose microprocessor.

[0122] Computer system 500 also includes a main memory 506, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 502 for storing information and instructions to be executed by processor 504. Main memory 506 may also be used for storing time variables or other intermediate information during execution of instructions to be executed by processor 504. Such instructions, when stored in a non-transitory storage medium accessible by processor 504, make computer system 500 a special-purpose machine customized to perform the operations specified in the instructions.

[0123] Computer system 500 further includes a read only memory (ROM) 508 or other static storage device coupled to bus 502 for storing static information and instructions for processor 504 .

[0124] A storage device 510, such as a magnetic or optical disk or solid-state RAM, is provided and coupled to bus 502 for storing information and instructions.

[0125] Computer system 500 may be coupled via bus 502 to a display 512, such as a liquid crystal display, for displaying information to a computer user. An input device 514, including alphanumeric and other keys, is coupled to bus 502 for communicating information and command selections to processor 504. Another type of user input device is a cursor control 516, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to processor 504 and for controlling cursor movement on display 512. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), allowing the device to specify a position in a plane.

[0126] Computer system 500 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic that couple with the computer system to render or program computer system 500 into a special purpose machine. According to an embodiment, the techniques are performed by computer system 500 in response to processor 504 executing one or more sequences of one or more instructions contained in main memory 506. Such instructions may be read into main memory 506 from another storage medium, such as storage device 510. Execution of the sequences of instructions contained in main memory 506 causes processor 504 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.

[0127] The term "storage media" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 510. Volatile media include dynamic memory, such as main memory 506. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge.

[0128] Storage media is distinct from, but may be used in conjunction with, transmission media, which is involved in transferring information between storage media. For example, transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise bus 502. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0129] Various types of media may be involved in carrying one or more sequences of one or more instructions to the processor 504 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 500 can receive the data on the telephone line and use an infrared transmitter to convert the data to an infrared signal. An infrared detector can receive the data carried in the infrared signal and appropriate circuitry can place the data on the bus 502. The bus 502 carries the data to the main memory 506, from which the processor 504 reads and executes the instructions. The instructions received by the main memory 506 may optionally be stored on storage device 510 either before or after execution by the processor 504.

[0130] Computer system 500 also includes a communication interface 518 coupled to bus 502. The communication interface 518 provides a two-way data communication coupling to a network link 520 that is connected to a local network 522. For example, the communication interface 518 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 518 may be a local area network (LAN) card that provides a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, the communication interface 518 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0131] The network link 520 typically provides data communication through one or more networks to other data devices. For example, the network link 520 may provide a connection through a local network 522 to a host computer 524 or to data equipment operated by an ISP (Internet Service Provider) 526. The ISP 526 also provides data communication services through the world wide packet data communication network now commonly referred to as the "Internet" 528. The local network 522 and the Internet 528 use both electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 520 and through the communication interface 518 that carry the digital data to and from the computer system 500 are exemplary forms of transmission media.

[0132] Computer system 500 can send messages and receive data, including program code, through the network(s), network link 520 and communication interface 518. In the Internet, for example, a server 530 may transmit a requested code for a suitable program through Internet 528, ISP 526, local network 522 and communication interface 518.

[0133] The received code may be executed by processor 504 as it is received, and / or stored in storage device 510, or other non-volatile storage for later execution.

[0134] 7. Equivalents, extensions, substitutions, etc. In the foregoing specification, exemplary embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indication of what the invention is, and what the applicant intends the invention to be, is set forth in the claims issued in a particular form hereby, including any subsequent amendments. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Thus, any limitations, elements, features, advantages, or attributes not expressly set forth in the claims should not limit the scope of the claims in any manner. The specification and drawings should therefore be considered in an illustrative, rather than restrictive, sense.

[0135] Various aspects of the invention may be apparent from the enumerated example embodiments (EEE) described below.

[0136] (EEE1) receiving a sequence of input texture images; determining a sequence of depth maps for the sequence of input texture images, each depth map in the sequence of depth maps comprising depth values ​​for a number of pixels represented by a corresponding input texture image in the sequence of input texture images; generating a sequence of depth difference maps from the sequence of depth maps, the sequence of depth difference maps including at least one depth difference map generated from two consecutive depth maps in the sequence of depth maps; outputting a video signal encoded with a compressed version of the sequence of input texture images and the sequence of depth difference maps, the video signal causing a receiving device of the video signal to generate, from the compressed version of the sequence of input texture images and the sequence of depth difference maps, a sequence of display images for rendering on an image display; The method includes:

[0137] (EEE2) The method according to EEE1, wherein the sequence of input texture images is divided into a sequence of group of pictures (GOPs), and each depth difference map in the sequence of depth difference maps is generated for a respective GOP in the sequence of GOPs.

[0138] (EEE3) The method according to EEE1 or 2, wherein the depth difference map includes a depth value difference between a first depth value in a first I frame of each GOP and a second depth value in a second I frame of a subsequent GOP that follows each GOP in the sequence of GOPs.

[0139] (EEE4) The method according to any one of EEE1 to 3, wherein the plurality of pixels of the corresponding input texture image includes one or more pixel sets of adjacent and co-moving pixels.

[0140] (EEE5) the sequence of depth difference maps includes a particular depth difference map including one or more planar displacements for the one or more sets of pixels within the plurality of pixels; The method of any of EEE1-4, wherein each planar displacement in the one or more planar displacements is for a respective pixel set in the one or more pixel sets.

[0141] (EEE6) A method as described in any of EEE1 to 5, wherein the compressed version of the sequence of input texture images comprises a plurality of inter-predictive coded images, the plurality of inter-predictive coded images comprising an inter-predictive coded image having one or more inter-predictive image blocks, samples in each of the one or more inter-predictive image blocks being predicted based on motion parameters generated from planar displacements contained in at least one depth difference map in the sequence of depth difference maps.

[0142] (EEE7) The method according to any of EEE1 to 6, wherein the video signal does not include motion parameters used in inter-predicted image blocks in a compressed version of the sequence of input texture images.

[0143] (EEE8) The method according to any of EEE1-7, wherein the video signal does not include the sequence of depth maps used to generate the sequence of depth difference maps.

[0144] (EEE9) The method of any of EEE1-8, wherein the sequence of depth maps is generated by image analysis operations performed on the sequence of input texture images by an upstream device receiving the sequence of input texture images.

[0145] (EEE10) A method according to any of claims 1 to 9, wherein the sequence of depth maps is received by an upstream device that receives the sequence of input texture images.

[0146] (EEE11) 1. A method comprising: retrieving a sequence of coded images from a video signal, said sequence of coded images representing a compressed version of said sequence of input texture images, said sequence of input texture images having been used by an upstream device to generate a sequence of coded images which have been encoded into said video signal; retrieving, from the video signal, a sequence of depth difference maps, the sequence of depth difference maps having been generated by the upstream device from a sequence of depth maps determined for the sequence of input texture images; rendering on an image display a sequence of display images derived from the sequence of encoded images and the sequence of depth difference maps; The method includes:

[0147] (EEE12) The method according to EEE11, wherein the video signal is coded with residual values ​​of inter-predicted image blocks in a plurality of inter-predictive coded images in the sequence of coded images.

[0148] (EEE13) The method according to any one of claims 8 to 12, wherein motion parameters for predicting samples in the inter-predicted image block are generated from the sequence of depth difference maps.

[0149] (EEE14) Apparatus for carrying out the method according to any one of EEE1 to 13.

[0150] (EEE15) A non-transitory computer-readable storage medium storing software instructions that, when executed by one or more processors, cause the method to be performed according to any one of the methods described in EEE1-13.

Claims

1. 1. A method comprising: receiving a sequence of input texture images; determining a sequence of depth maps for the sequence of input texture images, each depth map in the sequence of depth maps comprising depth values ​​for a number of pixels represented by a corresponding input texture image in the sequence of input texture images; generating a sequence of depth difference maps from the sequence of depth maps, the sequence of depth difference maps including at least one depth difference map generated from two consecutive depth maps in the sequence of depth maps; outputting a video signal encoded with a compressed version of the sequence of input texture images and the sequence of depth difference maps, the video signal causing a receiving device of the video signal to generate, from the compressed version of the sequence of input texture images and the sequence of depth difference maps, a sequence of display images for rendering on an image display, the video signal not including motion parameters used in inter-predicted image blocks in the compressed version of the sequence of input texture images; The method includes:

2. The method of claim 1 , wherein the sequence of input texture images is partitioned into a sequence of group of pictures (GOPs), and each depth difference map in the sequence of depth difference maps is generated for a respective GOP in the sequence of GOPs.

3. 3. The method of claim 1, wherein the depth difference map comprises a depth value difference between a first depth value in a first I-frame of each GOP and a second depth value in a second I-frame of a subsequent GOP following the respective GOP in the sequence of GOPs.

4. The method according to any of claims 1 to 3, wherein the plurality of pixels of the corresponding input texture image comprises one or more pixel sets of adjacent and co-moving pixels.

5. the sequence of depth difference maps includes a particular depth difference map including one or more planar displacements for the one or more sets of pixels within the plurality of pixels; The method of any of claims 1 to 4, wherein each planar displacement in the one or more planar displacements is for a respective pixel set in the one or more pixel sets.

6. 6. The method of claim 1, wherein the compressed version of the sequence of input texture images comprises a plurality of inter-predictively coded images having one or more inter-predictive image blocks, samples in each of the one or more inter-predictive image blocks being predicted based on motion parameters generated from planar displacements contained in at least one depth difference map in the sequence of depth difference maps.

7. The method according to any of the preceding claims, wherein the video signal does not include the sequence of depth maps used to generate the sequence of depth difference maps.

8. A method according to any preceding claim, wherein the sequence of depth maps is generated by image analysis operations performed on the sequence of input texture images by an upstream device receiving the sequence of input texture images.

9. The method according to any of the preceding claims, wherein the sequence of depth maps is received by an upstream device which receives the sequence of input texture images.

10. 1. A method comprising: retrieving a sequence of coded images from a video signal, said sequence of coded images representing a compressed version of a sequence of input texture images, said sequence of input texture images having been used by an upstream device to generate a sequence of coded images which have been encoded into said video signal; retrieving, from the video signal, a sequence of depth difference maps, the sequence of depth difference maps having been generated by the upstream device from a sequence of depth maps determined for the sequence of input texture images; rendering on an image display a sequence of display images derived from the sequence of encoded images and the sequence of depth difference maps; The method includes:

11. The method of claim 10 , wherein the video signal is coded with residual values ​​of inter-predicted image blocks in a plurality of inter-predictive coded images in the sequence of coded images.

12. The method according to claim 10 or 11, wherein motion parameters for predicting samples in the inter-predicted image block are generated from the sequence of depth difference maps.

13. Apparatus for carrying out the method according to any one of claims 1 to 12.

14. A non-transitory computer readable storage medium storing software instructions which, when executed by one or more processors, cause the medium to perform the method according to any of claims 1 to 12.

Images