A coding method or apparatus based on camera motion information

Abstract

At least a method and an apparatus are presented for efficiently encoding or decoding video. For example, motion information is determined that is representative of camera motion between a current image and a reference image, where the current image and the reference image are part of a game engine 2D rendered video. For instance, motion information representative of camera motion is determined from depth value of the samples and camera parameters. Advantaesgoulsy, a list of depth candidates is determined for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block. An encoding or a decoding method determines a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate of the list.

Description

[0001] A CODING METHOD OR APPARATUS BASED ON CAMERA MOTION INFORMATION

[0002] CROSS REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of European Patent Application No 23305957.5, filed on June 16, 2023, which is incorporated herein by reference in its entirety.

[0004] TECHNICAL FIELD

[0005] At least one of the present embodiments generally relates to a method or an apparatus for video encoding or decoding, and more particularly, to a method or an apparatus comprising determining a list of depth candidates for a block coded with motion information representative of camera motion.

[0006] BACKGROUND

[0007] To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transform to leverage spatial and temporal redundancy in the video content. Generally, intra or inter prediction is used to exploit the intra or inter frame correlation, then the differences between the original image and the predicted image, often denoted as prediction errors or prediction residuals, are transformed, quantized, and entropy coded. T o reconstruct the video, the compressed data are decoded by inverse processes corresponding to the entropy coding, quantization, transform, and prediction.

[0008] To obtain coding gains, modern codec standards define more and more sophisticated tools, and let the codec encoder decide the best ones to use. In the scope of cloud gaming compression, minimizing the latency is key. Although intensive computation capabilities are required in recent encoders that introduce a latency between the rendering of the game content and its coding.

[0009] Existing methods for coding and decoding show some limitations in the domain of coding 2D rendered video of a game engine. Therefore, there is a need to improve the state of the art.

[0010] SUMMARY

[0011] The drawbacks and disadvantages of the prior art are solved and addressed by the general aspects described herein.

[0012] According to a first aspect, there is provided a method. The method comprises video encoding by obtaining a coding block in a current image; determining a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determining a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and encoding the coding block based on the motion compensated prediction. According to a second aspect, there is provided a method. The method comprises video decoding by obtaining a coding block in a current image; determining a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determining a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and decoding the coding block based on the motion compensated prediction. According to another aspect, there is provided an apparatus. The apparatus comprises one or more processors, wherein the one or more processors are configured to implement the method for video encoding according to any of its variants. According to another aspect, the apparatus for video encoding comprises means for implementing the method for video decoding according to any of its variants.

[0013] According to another aspect, there is provided another apparatus. The apparatus comprises one or more processors, wherein the one or more processors are configured to implement the method for video decoding according to any of its variants. According to another aspect, the apparatus for video decoding comprises means for implementing the method for video decoding according to any of its variants.

[0014] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments; and at least one of (i) an antenna configured to receive a signal, the signal including the video block, (ii) a band limiter configured to limit the received signal to a band of frequencies that includes the video block, or (iii) a display configured to display an output representative of the video block.

[0015] According to another general aspect of at least one embodiment, there is provided a non- transitory computer readable medium containing data content generated according to any of the described encoding embodiments or variants.

[0016] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variants. According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variants.

[0017] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the described encoding / decoding embodiments or variants.

[0018] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in connection with the accompanying drawings.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In the drawings, examples of several embodiments are illustrated.

[0021] Figure 1 illustrates a block diagram of an example apparatus in which various aspects of the embodiments may be implemented.

[0022] Figure 2 illustrates a block diagram of an embodiment of video encoder in which various aspects of the embodiments may be implemented.

[0023] Figure 3 illustrates a block diagram of an embodiment of video decoder in which various aspects of the embodiments may be implemented.

[0024] Figure 4 illustrates an example texture frame of a video game with a corresponding depth map.

[0025] Figure 5 illustrates an example architecture of a cloud gaming system.

[0026] Figure 6 illustrates a camera motion inter tool in a codec in which various aspects of the embodiments may be implemented.

[0027] Figure 7 illustrates 4 exemplary representations of a plane of a depth model according to at least one embodiment.

[0028] Figure 8 illustrates an exemplary encoding method according to at least one embodiment.

[0029] Figure 9 illustrates an exemplary decoding method according to at least one embodiment.

[0030] Figure 10 illustrates various depth candidates for a coding block according to at least one embodiment.

[0031] Figures 1 1 and 12 illustrate two examples of a list of depth candidates according to at least one embodiment.

[0032] Figure 13 illustrates a generic encoding method according to at least one embodiment.

[0033] Figure 14 illustrates a generic decoding method according to at least one embodiment. DETAILED DESCRIPTION

[0034] Various embodiments relate to a video coding system in which, in at least one embodiment, it is proposed to adapt video coding tools to the cloud gaming system. Different embodiments are proposed hereafter, introducing some tools modifications to increase coding efficiency and improve the codec consistency when processing 2D rendered game engine video. Amongst others, an encoding method, a decoding method, an encoding apparatus, a decoding apparatus based on this principle are proposed. Although the present embodiments are presented in the context of the cloud gaming system, they may apply to any system where a 2D video may be associated to with camera parameters, such as a video captured by mobile device along with sensor’s information allowing to determine the position and characteristics of the device’s camera capturing the video. Depth information may be made available either from a sensor or other processing.

[0035] Moreover, the present aspects, although describing principles related to particular drafts of VVC (Versatile Video Coding) or to HEVC (High Efficiency Video Coding) specifications, or to ECM (Enhanced Compression Model) reference software are not limited to VVC or HEVC or ECM, and can be applied, for example, to other standards and recommendations, whether pre-existing or future-developed, and extensions of any such standards and recommendations (including VVC and HEVC and ECM). Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.

[0036] The acronyms used herein are reflecting the current state of video coding developments and thus should be considered as examples of naming that may be renamed at later stages while still representing the same techniques.

[0037] Figure 1 illustrates a block diagram of an example of a system in which various aspects and embodiments can be implemented. System 100 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this application. Examples of such devices, include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 100, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 100 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 100 is communicatively coupled to other systems, or to other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 100 is configured to implement one or more of the aspects described in this application.

[0038] The system 100 includes at least one processor 1 10 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this application. Processor 1 10 may include embedded memory, input output interface, and various other circuitries as known in the art. The system 100 includes at least one memory 120 (e.g. a volatile memory device, and / or a non-volatile memory device). System 100 includes a storage device 140, which may include non-volatile memory and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drive, and / or optical disk drive. The storage device 140 may include an internal storage device, an attached storage device, and / or a network accessible storage device, as non-limiting examples.

[0039] System 100 includes an encoder / decoder module 130 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 130 may include its own processor and memory. The encoder / decoder module 130 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 130 may be implemented as a separate element of system 100 or may be incorporated within processor 110 as a combination of hardware and software as known to those skilled in the art.

[0040] Program code to be loaded onto processor 1 10 or encoder / decoder 130 to perform the various aspects described in this application may be stored in storage device 140 and subsequently loaded onto memory 120 for execution by processor 1 10. In accordance with various embodiments, one or more of processor 1 10, memory 120, storage device 140, and encoder / decoder module 130 may store one or more of various items during the performance of the processes described in this application. Such stored items may include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0041] In several embodiments, memory inside of the processor 110 and / or the encoder / decoder module 130 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 110 or the encoder / decoder module 130) is used for one or more of these functions. The external memory may be the memory 120 and / or the storage device 140, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as for HEVC, or VVC.

[0042] The input to the elements of system 100 may be provided through various input devices as indicated in block 105. Such input devices include, but are not limited to, (i) an RF portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0043] In various embodiments, the input devices of block 105 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) down converting the selected signal, (iii) bandlimiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the down converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion may include a tuner that performs various of these functions, including, for example, down converting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, down converting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements may include inserting elements in between existing elements, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna. Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 100 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 1 10 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 1 10 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 110, and encoder / decoder 130 operating in combination with the memory and storage elements to process the datastream as necessary for presentation on an output device.

[0044] Various elements of system 100 may be provided within an integrated housing, Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 1 15, for example, an internal bus as known in the art, including the I2C bus, wiring, and printed circuit boards.

[0045] The system 100 includes communication interface 150 that enables communication with other devices via communication channel 190. The communication interface 150 may include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 190. The communication interface 150 may include, but is not limited to, a modem or network card and the communication channel 190 may be implemented, for example, within a wired and / or a wireless medium.

[0046] Data is streamed to the system 100, in various embodiments, using a Wi-Fi network such as IEEE 802. 1 1 . The Wi-Fi signal of these embodiments is received over the communications channel 190 and the communications interface 150 which are adapted for Wi-Fi communications. The communications channel 190 of these embodiments is typically connected to an access point or router that provides access to outside networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 100 using a set-top box that delivers the data over the HDMI connection of the input block 105. Still other embodiments provide streamed data to the system 100 using the RF connection of the input block 105.

[0047] The system 100 may provide an output signal to various output devices, including a display 165, speakers 175, and other peripheral devices 185. The other peripheral devices 185 include, in various examples of embodiments, one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide a function based on the output of the system 100. In various embodiments, control signals are communicated between the system 100 and the display 165, speakers 175, or other peripheral devices 185 using signaling such as AV. Link, CEC, or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 100 via dedicated connections through respective interfaces 160, 170, and 180. Alternatively, the output devices may be connected to system 100 using the communications channel 190 via the communications interface 150. The display 165 and speakers 175 may be integrated in a single unit with the other components of system 100 in an electronic device, for example, a television. In various embodiments, the display interface 160 includes a display driver, for example, a timing controller (T Con) chip.

[0048] The display 165 and speaker 175 may alternatively be separate from one or more of the other components, for example, if the RF portion of input 105 is part of a separate set-top box. In various embodiments in which the display 165 and speakers 175 are external components, the output signal may be provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0049] Figure 2 illustrates an example video encoder 200, such as VVC (Versatile Video Coding) encoder. Figure 2 may also illustrate an encoder in which improvements are made to the VVC standard or an encoder employing technologies similar to VVC.

[0050] In the present application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “encoded” or “coded” may be used interchangeably, and the terms “image,” “picture” and “frame” may be used interchangeably. Usually, but not necessarily, the term “reconstructed” is used at the encoder side while “decoded” is used at the decoder side.

[0051] Before being encoded, the video sequence may go through pre-encoding processing (201 ), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing, and attached to the bitstream.

[0052] In the encoder 200, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of, for example, CUs. Each unit is encoded using, for example, either an intra or inter mode. When a unit is encoded in an intra mode, it performs intra prediction (260). In an inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which one of the intra mode or inter mode to use for encoding the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting (210) the predicted block from the original image block.

[0053] The prediction residuals are then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the non-transformed residual signal. The encoder can bypass both transform and quantization, i. e. , the residual is coded directly without the application of the transform or quantization processes.

[0054] The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (240) and inverse transformed (250) to decode prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (265) are applied to the reconstructed picture to perform, for example, deblocking / SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (280).

[0055] Figure 3 illustrates a block diagram of an example video decoder 300. In the decoder 300, a bitstream is decoded by the decoder elements as described below. Video decoder 300 generally performs a decoding pass reciprocal to the encoding pass as described in FIG. 2. The encoder 200 also generally performs video decoding as part of encoding video data.

[0056] In particular, the input of the decoder includes a video bitstream, which can be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. The decoder may therefore divide (335) the picture according to the decoded picture partitioning information. The transform coefficients are de-quantized (340) and inverse transformed (350) to decode the prediction residuals. Combining (355) the decoded prediction residuals and the predicted block, an image block is reconstructed. The predicted block can be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). In-loop filters (365) are applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380).

[0057] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201 ). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0058] A video coding system such as a cloud gaming server or a device with light detection and ranging (LiDAR) capabilities may receive input video frames (e.g., texture frames) together with depth information (e.g., a depth map) and / or motion information, which may be correlated.

[0059] Figure 4 illustrates an example texture frame 402 of a video game with a corresponding depth map 404 that may be extracted (e.g., directly) from a game engine that is rendering the game scene. A depth map may be provided by the game engine in a floating-point representation. A depth map may be represented by a grey-level image, which may indicate the distance between a camera and an actual object. A depth map may represent the basic geometry of the captured video scene. A depth map may correspond to a texture picture of a video content and may include a dense monochrome picture of the same resolution as the luma picture. In examples, the depth map and the luma picture may be of different resolutions.

[0060] Figure 5 shows an example architecture of a cloud gaming system, where a game engine may be running on a cloud server. The gaming system may render a game scene based on the player actions. The rendered game scene may be represented as a 2D video including a set of texture frames. The rendered game engine 2D video may be encoded into a bitstream, for example, using a video encoder. The bitstream may be encapsulated by a transport protocol and may be sent as a transport stream to the player’s device. The player’s device may de-encapsulate and decode the transport stream and present the decoded 2D video representing the game scene to the player. According to various implementations, additional information such as a depth information, motion information, an object ID, an occlusion mask, camera parameters, etc. may be obtained from a game engine (e.g., as outputs of the game engine) and made available to the cloud server (e.g., an encoder of the cloud) as prior information. According to at least one embodiment, a video to encode is generated by 3D game engine as shown in the cloud gaming system of figure 5 where the video only includes texture information and synchronized camera parameters. As additional information (e.g. depth map) obtained from the game engine may represent a large amount of data, it is desirable to skip some of the available additional information. Additional information described herein such as the motion information issued from state of the art motion estimation in the encoder, or camera parameters or a combination thereof may be utilized to perform motion compensation in the rendered game engine 2D video in a video processing device (e.g., the encoder side of a video codec) as for instance described in the EP application 22306847.9 filed on 12-Dec-2022 by the same applicant which is incorporated herein by reference. Advantageously, the motion compensation generates inter prediction based on new motion information that is responsive to a new motion model in order to improve coding gains (e.g., compression gains). This new motion model, described in the EP application 22306847.9, may render the motion of a camera in the 3D game engine. The processing based on the camera parameters is referred to as Camera Motion tool or Camera Motion Inter tool in the present disclosure. The Camera Motion tool allows predicting motion in areas of a current image where motion is only affected by the virtual camera of the game engine (its characteristics and position). The present principles address the deriving of depth information to be used Camera Motion Inter tool in both the encoder and the decoder from a list of depth candidates.

[0061] Figure 6 illustrates a camera motion inter tool in a codec in which various aspects of the embodiments may be implemented. According to a salient feature, camera motion inter tool 650 approximates the depth of the CU to be compressed to a plane. Depending on the orientation of this plane in the 3D space, four different depth models can be used to approximate the plane, characterized by 1 , 2 or 3 depth parameters Pi. These depth parameters do not represent the depth in the 3D scene, but a third coordinate representing the depth associated to the 2D image coordinates. Usually, this third coordinate may be obtained from the projection of a 3D point by the game engine’s virtual camera. According to a salient feature described in the EP application 22306847.9, this third coordinate may be obtained from the projection of a reconstructed 3D point, the 3D point being reconstructed from motion information, such as a motion vector and reference picture, associated with the 2D image sample. Thanks to the depth models, characterized by only few depth parameters i.e. a few motion vectors such as the motion vector of an already reconstructed sample, a depth information is available per pixel (or per 4 x4 pixel blocs in VTM) in a camera motion coded block. This depth information, associated with the camera parameters, is used to compute a motion vector per sample. Then the motion compensation 640 can be performed, as it is performed in the state of art.

[0062] Figure 7 illustrates 4 exemplary representations of a plane of a depth model as disclosed in the EP application 22306847.9. The hatched planes represent some planes in the 3D game scene which are only affected by the game engine’s camera. Below these 3D hatched planes, an exemplary Camera Motion Coding Block (CB) 710, 720, 730 corresponding to the projection of a part of the hatched planes by the camera is represented.

[0063] According to a first variant, a depth model for the coding block includes a plane parallel to a camera’s sensor and is characterized by one depth parameter. In this variant, the plane 710 of the coding block may be approximated by a plane parallel to the camera’s sensor, the coding block is represented by only one depth parameter (Depth Modell ). The depth parameter represents the depth value of the central sample P1 in the coding block, which is also the depth value of any sample in the coding block.

[0064] According to a second variant, a depth model for the coding block includes a plane 720 tilted vertically or horizontally with respect to a camera’s sensor and the depth model is characterized by two depth parameters. In this variant, the plane may either be tilted horizontally (Depth Model 2H implying a horizontal depth interpolation) or vertically (Depth Model 2V implying a vertical depth interpolation). In this case, two depth parameters are required to define the depth plane. For instance, for the Depth Model 2V, a first depth parameter represents a depth value of a central sample P2V-T on a top border line of the coding block and a second depth parameter represents a depth value of a central sample P2V-B on a bottom border line of the coding block. For instance, for the Depth Model 2H, a first depth parameter represents a depth value of a central sample P2H-L on a left border line of the coding block and a second depth parameter represents a depth value of a central sample P2H-R on a right border line of the coding block. Then, the depth of any sample in the coding block is determined using an interpolation between the depth values indicated by the two depth parameters.

[0065] According to a third variant, a depth model for the coding block includes a plane tilted vertically and horizontally with respect to a camera’s sensor and the depth model is characterized by three depth parameters. In the variant, the plane is tilted in both directions (Depth Model 3) and three parameters are required. For instance, for the Depth Model 3, the three parameters respectively represent the depth value of a top-left sample P3-TL of the coding block, a depth value of a topright sample P3-TR of the coding block, a depth value of a bottom-left sample P3-B of the coding block. The skilled in the art will appreciate that the positions of samples used in the depth plane model are non-limiting examples, and that the present principles may contemplate any implementation of depth parameters allowing to define the 4 plane models.

[0066] Back to figure 6, the new Camera Motion Inter tool consists in computing the motion vectors in a new way for contents such as game engine contents. The Camera Motion inter tool 650 is indicated by the dotted line in encoder and decoder scheme of figure 6. Firstly, for each sample of the block (or a sub-sampled set in the block, sub-sampling by 4 in both direction for instance), an estimate of the depth of a sample is computed 620 depending on its position, the Camera Motion Depth Model and its associated parameters, where the depth in the coding block is represented with a parametric plane. Secondly, a motion vector is computed 630 depending on the sample position, the estimate depth and the camera parameters. The block diagram of figure 6 partially represents modules of an encoder or encoding method, for instance implemented in the exemplary encoder of figure 2. The block diagram of figure 6 further partially represents modules of a decoder or decoding method, for instance implemented in the exemplary decoder of figure 3. The Camera Motion inter tool receives some depth model parameters Pi along with camera parameters and provides motion vectors MVs used to compute the motion compensation 640. The camera parameters represent the characteristics and the position of the game engine’s virtual camera. They are provided for the reference image and for the current image to be encoded. In an embodiment, the encoder may determine 610 a depth parameter Pi for a depth model i by approximating the depth at the coding block position with a plane. Depending on the depth model, the coding block depth is approximated with a plane characterized by up to 3 parameters Pi. For instance, for the Depth Modell representing a block with constant depth, a single depth parameter P1 may be obtained by taking one of the depth of the central pixel of the coding block, an average depth around the central pixel of the coding block, or the average depth of the coding block, with or without sub-sampling. Advantageously, the parameter of depth model for the coding block is determined from a list of depth candidates from a causal neighborhood. According to a first variant, the depth parameter of a depth candidate is derived from a depth information associated to a neighboring block previously reconstructed. According to a second variant, the depth parameter of a depth candidate is derived from a motion vector information associated to a neighboring block previously reconstructed. An example of the deriving of an estimate depth value used as depth parameter is described in the EP application 23305419.6 filed on 28-Mar-2023 by the same applicant which is incorporated herein by reference. Then, the encoder reconstructs 620 depth values of the coding block based on the depth model parameters Pi. It determines an estimation of the depth value of any sample of the coding blocks. A motion vector responsive to camera motion compensation per sample is computed depending on its position, its approximated depth, and the camera parameters. Depending on the implementation, to reduce the complexity, a motion vector may be computed for a block of samples. For instance, a motion vector is computed per block of 4x4 samples. These motion vectors are then used to perform the motion compensation 640 as known by the skilled in the art. Since this vector is computed with the depth and the camera parameters, it represents the displacement of the current sample between the reference frame and the current frame due to a camera motion (translations and / or rotations), or a modification of the camera’s characteristics (focal length, ...). Different depth candidates processed by the Camera motion inter tool may be put into competition into a RDO loop to determine the motion model along with an associated depth candidate that result in the lower rate distortion cost. Additionally, the encoder may further provide adequate signaling of the selected depth model and depth candidate to enable a decoder to recover 670 the one or more parameters Pi to be used at the input of the camera motion inter tool 650. According to at least one embodiment, the depth parameters Pi may be derived using motion information or depth information associated to a depth candidate in a list. Additionally, the encoder may further signal 660 camera parameters for the images.

[0067] On the decoder side, the Camera motion inter tool computes Camera Motion MVs as done in the encoder. To that end, the decoder obtains 670, for instance from a list of depth candidates of a causal neighborhood as disclosed in the present application, the one or more parameters Pi of the depth model of a coding block to decode. In a variant, To that end, the decoder further obtains 670 an indication of the particular depth candidate in the list of depth candidates to derive the depth parameter. Additionally, the decoder obtains 680 camera parameters for the current image and for the reference image. The camera parameters for the reference image may be stored locally in the decoder at the reconstruction of the reference image. The depth parameter of the particular depth candidate may be stored locally in the decoder for processing of the next coding blocks to decode.

[0068] In the following, at least one embodiment of the determining of depth information 610, 670 of the depth model for the camera motion inter tool are detailed. The input parameters Pi characterizing the depth model of the Camera Motion inter tool represent a depth information. In the EP application 22306847.9, this depth information could be available at the encoder side, for instance when they are provided by a game engine as a depth map associated to the texture. But in this case, the amount of information to be transmitted to the decoder is not acceptable in the scope of video compression. In the EP application 23305419.6, this depth information may be obtained from motion information of the 2D video. It is desirable to provide such parameters representing the depth to the Camera Motion tool while limiting extra cost due to the transmission of depth information to the decoder and limiting the processing complexity. According to at least one embodiment, a list of depth candidates is detemrined for the coding block, wherein a depth candidate comprises a depth parameter of a depth model for the coding block, and a motion compensated prediction of the coding block with respect to a reference image is detemined from a particular depth candidate in the list.

[0069] As presented with figure 7, the Camera Motion Depth Model 1 only requires one depth parameter P1 to approximate the CU to a plane. Then, this model can be used to compute a motion vector per pixel or bloc of pixels, before performing the motion compensation.

[0070] In the context of video compression codecs, to compress the current CU, the encoder selects the tool providing the lower Rate-Distortion cost, considering the number of bits required to code the information and the distortion of the decompressed CU. This selection is performed by the Rate- Distortion optimization, consisting in evaluating, for different partitioning, the RD-Cost of each tool with its different configurations. In legacy inter tools, the RD-Cost optimization may be based on the evaluation of motion vector candidates. These candidates are some motion vectors provided by CUs that have already been reconstructed and forming the causal neighborhood.

[0071] For the Camera Motion tool using the Depth Model 1 , the P1 parameter candidates are referred to as depth candidates. According to a first aspect of the present principles, a list of depth candidates is built, where a depth value associated to a depth candidates may be derived either from motion vectors or from depth information. Accordingly, the depth candidates list may comprise motion vectors and / or depth information. When the candidate comes from a CU coded by a legacy inter tool, a depth candidate may comprise a motion vector information. Such depth candidate may be referred to as a depth candidate of type motion vector (Type_MV). In this case, this motion vector information is used to estimate a depth value providing a depth parameter of the depth candidate, at the encoder and the decoder side. The deriving of this depth estimate is, for instance, described in the EP application 23305419.6. Advantageously, as also described in the EP application 23305419.6, a depth estimate may be obtained from only one component of the motion vector or from the two motion vector components. According to different variants, a depth parameter of a depth candidate may be derived from the two components of a motion vector, or by only one component (either horizontal or vertical) of a motion vector. To derive the depth of a current pixel knowing its associated motion vector, the position of the pixel is required. Therefore, when building the list of depth candidates, when the depth candidate is issued from a motion vector information, the position of the pixel to which the motion information is associated in the image should be stored. According to a particular feature, a depth candidate of type motion vector in the list of candidates may further comprises an information associated with the spatial position of the pixel associated with the motion information. As two different embodiments are known to estimate a depth value, considering only one component of the motion vector (vertical or horizontal) or both, according to a particular embodiment, a depth candidate may comprise two components of a motion vector or by only one of the components of the motion vector. When only one component is required, a single motion vector candidate may provide two depth candidates, one per component. When the candidate comes from a CU that has previously been compressed by a Camera Motion tool, a depth candidate may comprise a depth information. Such depth candidate may be referred to as a depth candidate of type depth (Type_DEPTH).

[0072] A motion vector of a depth candidate provided by a legacy inter tool is the vector minimizing the RD-Cost. This vector may represent the real motion vector of the video. In this case, converting the vector to depth will provide the real depth, and hence a good candidate for the Camera Motion tool. But in some cases, the motion vector minimizing the RD-Cost may be different from the real motion vector (for instance because this motion vector minimizes the distortion with the reconstructed frame, not the original one, or due to a change of illumination, ...). In this case, computing the depth from the motion vector may provide a depth candidate that does not represent a good approximation of the depth. This candidate may be a bad candidate for Camera Motion. Conversely, if a depth information of a depth candidate is a depth value provided by a CU coded with Camera Motion, it means that this depth candidate minimizes the Camera Motion RD-Cost. Since it is issue from Camera Motion, it should provide a good depth candidate. For this reason, according to an aspect of the current invention, the candidates may be re-ordered depending on if they are motion vector type candidates or depth type candidates.

[0073] To have some depth candidates of type depth available when building the list of candidates, when the Camera Motion tool is selected as the best tool for coding a CU, it must store the pixel depth in the codec (in the same way that the motion vectors and their Refldx are stored in prior art by legacy inter tools). According to an embodiment using the Depth Model 1 , the depth is constant for all the pixels of the CU. But when using other Depth Models than the Depth Model 1 (Depth Models 2V, 2H or Depth Model 3 as illustrated on Figure 7), the depth may vary for each pixel or block of pixels. According to another embodiment, when a CU is coded with Camera Motion, a depth information is stored per pixel or block of pixels, to be available as a depth value for a depth candidate for the next CUs.

[0074] Figure 8 illustrates an exemplary encoding method according to at least one embodiment. The embodiment of Figure 8 implements the Depth Model 1 of the Camera Motion tool at the encoder for a CU, where no depth information is provided. Usually, an encoder has to decide which inter tool, with which parameter, must be used to compress the current CU. To make this decision, a list of motion vector candidates is usually built. For each available inter tool and for a given CU partitioning, the encoder computes a prediction using motion compensation for each motion vector candidate. The tool giving the best RD-Cost is chosen by the encoder. The encoder signals to the decoder which tool must be used with which candidate in the list of candidates.

[0075] In the video codec, the Camera Motion tool is in competition with other inter tools. But to be used, this tool needs one or several parameters representing a depth information, not available in the codec. In a legacy inter prediction, motion vector information (the motion vector itself, the reference image to be considered,...) is stored per pixel or per block of pixels. For instance, in VVC, this information is stored per block of 4 x4 pixels. This motion vector information may be used as candidate to encode the next CUs. According to a particular feature, the list of depth candidates built by the encoder 810 (but also at the decoder 910) for the Camera Motion tool may comprise of two types of depth candidates. The first type TYPE MV corresponds to depth candidate comprising the legacy motion vector candidates, the second type TYPE DEPTH corresponds to depth candidates provided by the Camera Motion tool. The video codec applies a different processing 820 (or 930 for the decoder) if the depth candidate is of type motion vector TYPE MV or of type depth TYPE DEPTH. When the depth candidate is a motion vector type candidate, the codec estimates 830 the depth parameter P1 from the motion vector, as described in the EP application 23305419.6. To perform this computation, the codec needs the camera parameters and the position of the motion vector candidate. According to another particular feature, when building the list of candidates, the spatial position (ie the (x, y) coordinates of sample to which the motion vector is associated to) of the motion vector candidate may be stored as a characteristic of the depth candidate. When the depth candidate is a depth type candidate, it can be used as depth parameter P1 for the Camera Motion tool.

[0076] We have seen that the depth parameter P1 may be derived either by a depth type candidate or by an estimate depth value 830 derived from the motion vector type candidate. With this parameter, the Camera Motion tool 840 is able to compute a motion vector per pixel or block of pixels, the motion vector being used as input to the state of art motion compensation, followed by the RD-Cost estimation.

[0077] According to yet another particular feature, when the Camera Motion tool is selected by the encoder as the best tool to encode the current CU, a new depth information is stored 850 in the codec (per pixel or block of pixels). This depth information P1 is the depth of the pixel or block of pixels computed by the Depth Model. Since this depth information is used to efficiently encode (best RD-Cost) the current CU, it will provide a good depth candidate for the next CUs. By this way, each time the Camera Motion tool is used, it provides new depth candidates for the next CUs. With this mechanism, in a codec where no depth information is provided or signaled, we have introduced depth information available as candidates, at the encoder and the decoder side. Advantageously, at least one embodiment relates to a Camera Motion tool implementing the Depth Model 1 in the codec. In this case, since the depth is constant for the whole CU, the depth information is the same for each pixel or block of pixels. The skilled in the art will appreciate that the same mechanism may be easily applied to any of the other Depth Models represented on Figure 7. The Depth Models 2V, 2H and 3 provide a different depth information per pixel or block of pixels. This depth information may also stored in the codec to provide depth candidates for the Depth Model 1 . In this case, the depth varies depending on the position in the CU.

[0078] Figure 9 illustrates an exemplary decoding method according to at least one embodiment. The embodiment of Figure 9 implements the Depth Model 1 of the Camera Motion tool at the decoder for a CU, where no depth information is provided. The decoding method of figure 9 implement the same method as the encoding method of figure 8 except that the selected depth candidate in the list is retrieved form the encoder (either by using an index in the list or by signaling that the most probable candidate may be used).

[0079] Figure 10 illustrates various depth candidates for a coding block according to at least one embodiment. Building a list of motion vectors candidates is already known in the prior art of video codecs. The algorithm applied to build this list may vary from one tool to another. It may also vary from one codec to another. For the same tool, it can for instance be slightly different between HEVC and VVC. But the principle remains the same: it consists in building a list of the most probable candidates to encode the current CU. The same principles are applied here Figure 10 to build a list of depth candidates. These candidates may be issued from a spatial neighborhood around the current CU (and causally available when encoding the current block) as shown in Figure 10 but the skilled in the art will appreciate that depth candidates may further be issued from collocated blocs, history-based candidates, as non-limiting examples. To reduce the signaling cost, the most probable candidates are usually placed first in the list.

[0080] On Figure 10, the hatched Coding Units CUO and CU2 have been coded with the Camera Motion tool. It means that for these CUs, the depth per pixel or block of pixels is known and previously stored in the codec. This depth information may provide depth candidates for the current CU. The other adjacent CUs CU1 and CU3 have been coded with a legacy inter tool. No depth information is available. They provide a motion vector type candidate. In our case, the list of candidates may comprise the depth type candidates CO and C2 when the CU (hatched CUs CUO and CU2 on figure 10) has been coded with Camera Motion or the motion vector type candidates C1 and C3 when the CU (pointed CUs in figure 10) has been coded by a legacy inter tool. Note that the Coding Unit CUO has been coded with Camera Motion using the Depth Model 1. Since for this model the depth is constant, the same depth value was stored per pixel or block of pixel in CUO. But the Coding Unit CU2 may also be coded with Camera Motion using the Depth Model 2H. When the Depth Model 2H has been applied, it has provided a depth per pixel or block of pixel depending on the horizontal position. In this case, the value Zn stored in CU2 is not the same as the candidate C2 (we can make the analogy with motion vectors provided by tools like Standard Merge where the MV stored and used as candidates are the same for the whole CU, and motion vectors provided by tools like Affine where a different vector is stored per bloc of pixels). Note that even if we have two types of candidates, no extra signaling is required to the decoder: since the decoder builds the same list of candidates as the encoder, the decoder implicitly knows the type of the candidate signaled by the encoder.

[0081] Figure 11 illustrates a first example of a list of depth candidates according to at least one embodiment. For the example illustrated on figure 10, the Figure 11 gives a first example of list of candidates (encoder 820, decoder 930) built to compress the current CU. It is composed of depth candidates of TYPE MV when the CU has been coded with a legacy inter tool, and depth candidates of TYPE DEPTH when the CU has been coded by Camera Motion. The depth candidates TYPE DEPTH are characterized by the depth value stored in the codec (850 and 960) when a Camera Motion Depth Model has been used. The motion vector type candidates TYPE MV are characterized by the motion vector MV itself and its reference frame Refldx, but also by its spatial position. This spatial position is required to estimate the depth (830 and 940) from the motion vector and the camera parameters.

[0082] A depth type candidate is available when the CU has been coded by Camera Motion. The Camera Motion tool was selected by the encoder for this CU because it was minimizing the RD-cost. It means that the depth parameter was a good approximation of the depth: this depth candidate should be a good candidate for the current CU. Conversely, the motion vector minimizing the RD- Cost may be different from the real motion vector of the video. In this case, the computation of the depth from the motion vector will not provide a good approximation of the depth. To minimize the signaling cost, the best candidates must be placed before the others in the list. For these reasons, depth candidates should be placed before the motion vectors candidates in the list. According to a particular embodiment, the list of candidates may be reordered to foster the depth candidates. In the example Figure 1 1 , the initial list CO, C1 , C2, C3 will be rearranged to CO, C2, C1 , C3 to foster the depth type candidate C2 compared to the motion vector type candidate C1 . Besides, as previously explained, two embodiments are possible to compute the depth from a motion vector. The first one using the two components of the motion vector, the second one only one component. The list of candidates Figure 1 1 corresponds to the first embodiment using the two motion vector components. Figure 12 illustrates a second example of a list of depth candidates according to at least one embodiment. In the second embodiment, a motion vector type candidate may be characterized by only one of its components. In this case, a motion vector can be of type TYPE MV X when only the horizontal component is used or type TYPE MV Y when only the vertical component is used to approximate the depth as shown on Figure 12. Note that we have provided here an example of list composed only of spatial candidates. Other candidates well known form the skilled in the art such as collocated candidates, history-based candidates, ... may also be added to the list.

[0083] According to yet another variant, we can limit the number of depth type candidates in the list to add diversity in the candidates. For instance, the number of depth candidates may be limited to two candidates, the other ones being motion vector candidates. These depth candidates may be placed at the beginning of the list. When we have more than two depth candidates available, two of them for instance can be placed at the beginning of the list. If the list may be completed with motion vector candidates, the remaining depth type candidates may be dropped. If the list cannot be completed with motion vector type candidates (not enough candidates are available), the remaining depth candidates may be placed at the end of the list to complete it.

[0084] According to yet another variant, a same candidate value is not set twice in the list. For instance, when a motion vector type candidate is already present in the list, a new potential motion vector candidate having the same value should be dropped. In the same way, if a depth candidate is already present, a potential new depth candidate with the same depth should be dropped. In another variant, if a motion vector candidate provides a depth value already present as depth candidate in the candidates list, it may be dropped.

[0085] In the context of cloud gaming (where the camera parameters are provided by a game engine) the depth values in the Z buffer are usually coded between two depth planes. The near plane can for instance be coded by the float value 0.0 and the far plane by 1.0 (or conversely). By construction, depth values cannot be outside this range. In an embodiment, when a motion vector type candidate provides a depth estimate outside this range, it is considered as invalid and removed from the list of candidates (it means that the motion vector does not represent the real motion in the video). Alternatively, it could be kept as candidate, but the corresponding depth value can be limited in the valid range. For instance, a motion vector providing a depth value higher than 1 .0 could be kept as a valid candidate, but the depth value used as depth parameter should be 1.0. Only one of these candidates could be kept. In the same way, only one motion vector providing a depth value lower than 0.0 could be kept, providing a depth parameter limited to 0.0.

[0086] Figure 13 illustrates a generic encoding method 1300 according to at least one embodiment. In a first step 1310, a coding block in a current image is obtained. In a step 1320, a list of depth candidates is determined for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block. The encoding method determines for instance responsively to a RD-cost a particular depth candidate to be used for the encoding. In a step 1330, a motion compensated prediction of the coding block with respect to a reference image is computed from the particular depth candidate of the list. Here, the motion information used in motion compensation is representative of camera motion between the current image and the reference image. Finally in a step, 1340, the coding block is encoded based on the motion compensated prediction. In a step, not shown on figure 13, an index identifying the particular depth candidate selected for the encoding of the coding block is encoded and signaled to a decoder.

[0087] Figure 14 illustrates a generic decoding method 1400 according to at least one embodiment. In a first step 1410, a coding block to decode in a current image is obtained. In a step 1420, a list of depth candidates is determined for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block. The decoding method determines a particular depth candidate to be used for the decoding. For instance, an index identifying the particular depth candidate selected at the encoding is received and parsed in the decoding. In a step 1430, a motion compensated prediction of the coding block with respect to a reference image is computed from the particular depth candidate of the list. Here, the motion information used in motion compensation is representative of camera motion between the current image and the reference image. Finally in a step, 1440, the coding block is encoded based on the motion compensated prediction.

[0088] Additional Embodiments and Information

[0089] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding. Various methods and other aspects described in this application can be used to modify modules, for example, the inter prediction modules (270, 275, 375), of a video encoder 200 and decoder 300 as shown in figure 2 and figure 3. Moreover, the present aspects are not limited to VVC or HEVC, and can be applied, for example, to other standards and recommendations, and extensions of any such standards and recommendations. Unless indicated otherwise, or technically precluded, the aspects described in this application can be used individually or in combination.

[0090] Various numeric values are used in the present application. The specific values are for example purposes and the aspects described are not limited to these specific values.

[0091] Various implementations involve decoding. “Decoding,” as used in this application, may encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0092] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application may encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. Note that the syntax elements as used herein are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0093] The implementations and aspects described herein may be implemented as various pieces of information, such as for example syntax, that can be transmitted or stored, for example. This information can be packaged or arranged in a variety of manners, including for example manners common in video standards such as putting the information into an SPS, a PPS, a NAL unit, a header (for example, a NAL unit header, or a slice header), or an SEI message. Other manners are also available, including for example manners common for system level or application level standards such as putting the information into one or more of the following:

[0094] SDP (session description protocol), a format for describing multimedia communication sessions for the purposes of session announcement and session invitation, for example as described in RFCs and used in conjunction with RTP (Real-time Transport Protocol) transmission;

[0095] DASH MPD (Media Presentation Description) Descriptors, for example as used in DASH and transmitted over HTTP, a Descriptor is associated to a Representation or collection of Representations to provide additional characteristic to the content Representation;

[0096] RTP header extensions, for example as used during RTP streaming;

[0097] ISO Base Media File Format, for example as used in OMAF and using boxes which are object- oriented building blocks defined by a unique type identifier and length also known as 'atoms' in some specifications;

[0098] HLS (HTTP live Streaming) manifest transmitted over HTTP. A manifest can be associated, for example, to a version or collection of versions of a content to provide characteristics of the version or collection of versions.

[0099] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, for example, computers, cell phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate communication of information between end-users.

[0100] Reference to “one embodiment” or “an embodiment” or “one implementation” or “an implementation”, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment” or “in an embodiment” or “in one implementation” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0101] Additionally, this application may refer to “determining” various pieces of information. Determining the information may include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Further, this application may refer to “accessing” various pieces of information. Accessing the information may include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information. Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information may include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations, for example, storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0102] It is to be appreciated that the use of any of the following “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0103] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments the encoder signals a quantization matrix for de-quantization. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as others, then signaling can be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling can be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0104] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

Claims

CLAIMS1 . A method, comprising: obtaining a coding block in a current image; determining a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determining a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and encoding the coding block based on the motion compensated prediction.

2. The method of claim 1 , wherein a depth candidate comprises a depth information associated to a neighboring block.

3. The method of claim 1 , wherein a depth candidate comprises a motion vector information associated to a neighboring block.

4. The method of claim 1 , wherein a depth candidate comprises a horizontal motion vector information associated to a neighboring block or wherein a depth candidate comprises a vertical motion vector information associated to a neighboring block.

5. The method of any of claims 3 or 4, wherein a depth candidate further comprises a coordinate information of a sample of a neighboring block associated to the vector information.

6. The method of of any of claims 1 to 5, wherein the list of depth candidates for the coding block is ordered and wherein determining a list of depth candidates for the coding block further comprises placing the depth candidates comprising depth information first.

7. The method of any of claims 1 to 6, wherein the list of depth candidates for the coding block is ordered and wherein determining a list of depth candidates for the coding block further comprises pruning the list of candidates.

8. The method of any of claims 1 to 7, further comprising storing a depth value of at least onesample of the coding block.

9. The method of any of claims 1 to 7, further comprising encoding an indication of the particular depth candidate in the list of depth candidates.

10. The method of any of claims 1 to 9, wherein determining the motion compensated prediction of the coding block from a particular depth candidate further comprises determining the particular depth candidate by applying a rate distortion optimization on the encoding.1 1. The method of any of claims 1 to 10, further comprising encoding an indication of camera motion coding mode associated with the coding block.

12. The method of any of claims 1 to 10, wherein the depth model includes a plane parallel to a camera’s sensor.

13. A method, comprising: obtaining a coding block in a current image; determining a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determining a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and decoding the coding block based on the motion compensated prediction.

14. The method of claim 13, wherein a depth candidate comprises a depth information associated to a neighboring block.

15. The method of claim 13, wherein a depth candidate comprises a motion vector information associated to a neighboring block.

16. The method of claim 13, wherein a depth candidate comprises a horizontal motion vector information associated to a neighboring block or wherein a depth candidate comprises a vertical motion vector information associated to a neighboring block.

17. The method of any of claims 15 or 16, wherein a depth candidate further comprises a coordinate information of a sample of a neighboring block associated to the vector information.

18. The method of any of claims 13 to 17, wherein the list of depth candidates for the coding block is ordered and wherein determining a list of depth candidates for the coding block further comprises placing the depth candidates comprising depth information first.

19. The method of any of claims 13 to 18, wherein the list of depth candidates for the coding block is ordered and wherein determining a list of depth candidates for the coding block further comprises pruning the list of depth candidates.

20. The method of any of claims 13 to 19, further comprising storing a depth value of at least one sample of the coding block.

21. The method of any of claims 13 to 20, further comprising decoding an indication of the particular depth candidate in the list of depth candidates.

22. The method of any of claims 13 to 21 , further comprising decoding an indication of camera motion coding mode associated with the coding block.

23. The method of any of claims 13 to 22, wherein the depth model includes a plane parallel to a camera’s sensor.

24. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: obtain a coding block in a current image; determine a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determine a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and encode the coding block based on the motion compensated prediction.

25. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: obtain a coding block in a current image; determine a list of depth candidates for the coding block, wherein a depth candidate allows deriving a depth parameter of a depth model for the coding block, the depth model being representative of a depth value of samples of the coding block; determine a motion compensated prediction of the coding block with respect to a reference image from a particular depth candidate in the list, where motion information used in motion compensation is representative of camera motion between the current image and the reference image; and decode the coding block based on the motion compensated prediction.

26. A computer program product which is stored on a non-transitory computer readable medium and comprises program code instructions for implementing the steps of a method according to at least one of claims 1 to 23 when executed by at least one processor.

27. A computer program comprising program code instructions for implementing the steps of a method according to at least one of claims 1 to 23 when executed by a processor.

28. A non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer for performing the method according to any one of claims 1 to 23.

Images