Method for encoding and decoding a data stream representing omnidirectional video

Abstract

The present invention relates to a method and an encoding device for encoding a data stream representing an omnidirectional video, and relatedly, to a method and a decoding device for decoding a data stream representing an omnidirectional video. According to the present invention, the data stream representing the omnidirectional video comprises encoded data representing at least one base layer of a 2D or 3D video representing a view of a scene captured by the omnidirectional video, and the data stream further comprises encoded data representing at least one enhancement layer of the omnidirectional video, the at least one enhancement layer being encoded by prediction relative to the at least one base layer.

Description

Technical Field

[0001] The present invention is in the field of video compression, and more particularly in the technical field for encoding and decoding immersive or omnidirectional (e.g. 180°, 360° in 2D or 3D) video. Background Art

[0002] Omnidirectional video can be used to represent a scene from a central point and can do so in all directions. When the entire field is used, the term "360° video content" is used. It is also possible to capture a subset of the field, for example covering only 180°. The content can be captured monoscopically (2D) or stereoscopically (3D). This type of content can be generated by combining sequences of images captured by different cameras, or it can be generated synthetically by a computer (for example, in VR video games). Images of this video content enable video to be rendered in any direction via appropriate means. The user can control the direction in which the captured scene is displayed and navigate continuously in every possible direction.

[0003] Such 360° video content may be rendered, for example, by using a virtual reality headset that provides the user with the impression of being immersed in the scene captured by the 360° video content.

[0004] Such 360° video content requires a receiving device suitable for this type of content (e.g., a virtual reality headset) in order to provide the user with the ability to immerse and control the displayed view.

[0005] However, most currently used video content receivers are not compatible with this type of 360° video content and can only render classic 2D or 3D video content. In fact, rendering 360° video content requires geometric transformation of the video image to render the desired viewing direction.

[0006] Therefore, the broadcast of 360° video content is not backwards compatible with the existing fleet of video receivers and is limited to receivers suitable for this type of content.

[0007] However, it can be observed that content captured specifically for 360° video broadcasting may have been captured for 2D or 3D video broadcasting. In this case, the broadcast is the entire 360° content projected on a flat surface.

[0008] Additionally, since video streams in as many formats as possible need to be sent, i.e., 2D, 3D, 360° views of the same captured scene, the bandwidth cost of simultaneously broadcasting the same content captured in different formats (2D or 3D and 360°) to address different video receivers is high.

[0009] Therefore, there is a need to optimize the encoding and broadcasting of omnidirectional video content representing a portion of a scene (180°) or the entirety of a scene (360°), and doing so monoscopically (2D) or stereoscopically (3D).

[0010] There are techniques for layer-by-layer video coding known as scalable video coding, which are used to encode a 2D video stream in several successive layers of refinement, thereby providing reconstruction of the 2D video at different levels. For example, spatial scalability enables encoding a video signal in several layers of increasing spatial resolution. Scalability in terms of PSNR (peak signal-to-noise ratio) enables encoding a video signal for a fixed spatial resolution in several layers of improved quality. Scalability in colorimetric space enables encoding a video signal in several layers presented in increasingly wider colorimetric spaces.

[0011] However, none of the prior art techniques is able to generate a video data stream representing a scene that can be decoded by a classic 2D or 3D video decoder as well as a 360° video decoder.

[0012] US document 2016 / 156917 describes a method for scalable coding of a video, which may be a multi-view video, and wherein each view of the multi-view video is encoded in a layer of a stream and predicted by another view of the multi-view video. Summary of the Invention

[0013] The present invention improves upon the prior art. To this end, the present invention relates to a method for encoding a data stream representing omnidirectional video, comprising:

[0014] - encoding in said stream at least one base layer representing 2D or 3D video representing views of the same scene captured by omnidirectional video (360°, 180°, etc.),

[0015] - encoding in said stream at least one enhancement layer representing omnidirectional video, the at least one enhancement layer being encoded by prediction with respect to at least one base layer.

[0016] The present invention reduces the cost of transmitting video streams when video content must be transmitted in both 2D and 360° views, or in both 3D and 3D-360° views. Thus, a classic 2D or 3D video decoder will only decode the base layer or one of the base layers to reconstruct a 2D or 3D video of the scene, while a compatible 360° decoder will decode one or more base layers or at least one enhancement layer to reconstruct a 360° video. Using predictions from at least one base layer to encode the enhancement layers reduces the cost of encoding the enhancement layers.

[0017] Relatedly, the present invention also relates to a method for decoding a data stream representing omnidirectional video, comprising:

[0018] - decoding from said stream at least one base layer representing a 2D or 3D video representing views of the same scene captured by omnidirectional video,

[0019] - decoding from said stream at least one enhancement layer representing omnidirectional video, the at least one enhancement layer being decoded by prediction with respect to at least one base layer.

[0020] The term "omnidirectional video" is understood herein to refer equally to video of a scene for which the entirety of the field (360°) is captured, as well as video of a scene for which sub-portions of the 360° field are captured, such as 180°, 160°, 255.6°, etc. Thus, omnidirectional video refers to a scene that is captured for at least one continuous portion of a 360° field.

[0021] According to a particular embodiment of the invention, predicting an enhancement layer with respect to at least one base layer comprises, for encoding or reconstructing at least one picture of the enhancement layer:

[0022] - generating a reference image obtained by geometrically projecting an image, called a base image, reconstructed from at least one base layer onto the reference image,

[0023] - storing said reference picture in a memory of reference pictures of the enhancement layer.

[0024] Advantageously, during encoding or decoding of an image of the enhancement layer, prediction in the enhancement layer is performed by adding a reference image onto which the image reconstructed from the base layer is projected. Thus, a new reference image is added to the memory of reference images of the enhancement layer. This new reference image is generated by geometrically projecting all base images reconstructed from the base layer at a given moment.

[0025] According to another specific embodiment of the present invention, the data stream includes a piece of information indicating the type of geometric projection used to represent the omnidirectional video.

[0026] According to another specific embodiment of the present invention, the view represented by the 2D or 3D video is a view extracted from the omnidirectional video.

[0027] According to another specific embodiment of the present invention, the data stream includes a piece of information indicating the type of geometric projection used to extract the view and position parameters of the omnidirectional video.

[0028] According to one variant, the piece of information representing the parameters of the projection and position of the base image is encoded in the data stream in each image of the 360° video. Advantageously, this variant allows for taking into account shifts in the scene of the views used as predictions for the enhancement layers. For example, the images of the base layer video may correspond to images captured while moving through the scene, for example to track moving objects in the scene. For example, the views may be captured by a moving camera, or by multiple cameras positioned at different viewpoints in the scene in succession, for example to track a ball or a player during a football match.

[0029] According to another specific embodiment of the present invention, the data stream comprises at least two base layers, each base layer representing a 2D or 3D video, the 2D or 3D video respectively representing a view of a scene, the at least two base layers being coded independently of each other.

[0030] Therefore, it is possible to have several independent base layers in the stream, allowing to independently reconstruct several 2D or 3D views of the 360° video.

[0031] According to another specific embodiment of the present invention, an image of an enhancement layer is encoded by means of a set of tiles, each tile covering an area of ​​the image of the enhancement layer, each area being different and separate from other areas of the image of the enhancement layer, each tile being encoded by prediction relative to at least one base layer. Decoding the enhancement layer comprises: reconstructing a portion of the image of the enhancement layer, reconstructing the portion of the image comprising decoding tiles of the enhancement layer covering a portion of the image of the enhancement layer to be reconstructed; and decoding at least one base layer comprising decoding the base layer used for predicting tiles covering a portion of the image of the enhancement layer to be reconstructed.

[0032] This particular embodiment of the present invention enables reconstruction of only a portion of the omnidirectional image, rather than the entire image. Typically, only the portion currently being viewed by the user is reconstructed. Consequently, it is not necessary to decode all base layers of the video stream or even transmit them to the receiver. In fact, in situations where the user cannot simultaneously see the entire omnidirectional video image, it is possible to encode the omnidirectional image using a tile mechanism, thereby enabling independent encoding of omnidirectional image regions, making it possible to decode only those regions of the omnidirectional image visible to the user.

[0033] By means of a particular embodiment of the invention, the independent coding of the base layer thus makes it possible to reconstruct the tiles of the omnidirectional image separately and to limit the complexity in decoding by avoiding unnecessary decoding of the base layer.

[0034] Advantageously, for each tile of an enhancement layer to be decoded, a piece of information is decoded from the data stream, said information identifying at least one base layer used for predicting the tile.

[0035] The present invention also relates to an apparatus for encoding a data stream representing omnidirectional video, the encoding apparatus comprising means for encoding in the stream at least one base layer representing 2D or 3D video, the 2D or 3D video representing views of the same scene captured by the omnidirectional video, and means for encoding in the stream at least one enhancement layer representing the omnidirectional video, the means for encoding the enhancement layer comprising means for predicting the enhancement layer relative to the at least one base layer.

[0036] The present invention also relates to an apparatus for decoding a data stream representing omnidirectional video, the decoding apparatus comprising means for decoding, in the stream, at least one base layer representing 2D or 3D video, the 2D or 3D video representing views of the same scene captured by the omnidirectional video, and means for decoding, in the stream, at least one enhancement layer representing the omnidirectional video, the means for decoding the enhancement layer comprising means for predicting the enhancement layer relative to the at least one base layer.

[0037] The encoding device and decoding device are particularly suitable for implementing the encoding method and decoding method described above, respectively. Of course, according to the present invention, the encoding device and decoding device can respectively include different features of the corresponding encoding method and decoding method. Therefore, the features and advantages of the encoding device and decoding device are the same as the features and advantages of the encoding method and decoding method, respectively, and will not be described in more detail.

[0038] According to a specific embodiment of the present invention, the decoding device is included in a terminal.

[0039] The present invention also relates to a signal representing an omnidirectional video, said signal comprising coded data representing at least one base layer of a 2D or 3D video, said 2D or 3D video representing views of the same scene captured by the omnidirectional video, said signal further comprising coded data representing at least one enhancement layer of the omnidirectional video, said at least one enhancement layer being coded by prediction with respect to at least one base layer.

[0040] According to a specific embodiment of the invention, the image of the enhancement layer is encoded using a set of tiles, each tile covering a region of the image of the enhancement layer, each region being distinct and separate from other regions of the image of the enhancement layer, each tile being encoded by a prediction relative to at least one base layer. According to a specific embodiment of the invention, the signal further includes, for each tile, a piece of information identifying at least one base layer used to predict the tile. Thus, only the base layer required to decode the tile to be decoded is decoded, thereby optimizing the use of decoder resources.

[0041] The present invention also relates to a computer program comprising instructions for implementing the encoding method or decoding method according to any one of the above-described embodiments when the program is executed by a processor. Such a program may be written in any programming language. It may be downloaded from a communication network and / or recorded on a computer-readable medium. Such a program may be written in any programming language and may take the form of source code, object code, or intermediate code between source code and object code, such as a partially compiled form, or any other desired form.

[0042] According to another aspect, the present invention relates to a computer-readable recording medium or medium or information carrier or medium, which includes the instructions of a computer program, such as described above. The recording medium mentioned above can be any entity or device capable of storing a program. For example, the medium can include a storage member, such as a read-only memory (ROM) type memory (such as a CD-ROM or microelectronic circuit ROM), a flash memory installed on a removable storage medium, such as a USB stick, or a magnetic mass storage of a hard disk drive (HDD) or a solid-state drive (SSD) type, or a combination of memories working according to one or more data recording technologies. In addition, the recording medium can correspond to a transmittable medium such as an electrical signal or an optical signal, which can be transmitted via an electrical cable or an optical cable, by radio or by other means. Specifically, the proposed computer program can be downloaded from an Internet type network.

[0043] Alternatively, the recording medium may correspond to an integrated circuit in which the program is incorporated, the circuit being adapted to execute the use in the method in question.

[0044] Therefore, the encoding method or decoding method according to the present invention can be implemented in various ways, in particular, in a wired form or in a software form. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Other features and advantages of the invention will become more apparent from the following description of specific embodiments given by way of simple illustrative and non-exhaustive examples and from the accompanying drawings, in which:

[0046] - Figure 1A The following shows the steps of an encoding method according to a specific embodiment of the present invention.

[0047] - Figure 1B shows an example of a signal generated based on a coding method implemented according to a specific embodiment of the present invention,

[0048] - Figure 2A An image showing a view of a scene captured by a 360° video encoded in the base layer,

[0049] - Figure 2BThe image of the 360° video is used as a reference projection. Figure 2A The image shown in

[0050] - Figure 2C An image showing a 360° video encoded in the enhancement layer,

[0051] - Figure 2D and 2E images each showing two views of a scene captured by a 360° video and each encoded in the base layer,

[0052] - Figure 2F The image of the 360° video is used as a reference projection. Figure 2D and 2E The two views shown in the image,

[0053] - Figure 2G An image showing a 360° video encoded in the enhancement layer,

[0054] - Figure 3 The following shows the steps of a decoding method according to a specific embodiment of the present invention.

[0055] - Figure 4A An example of an encoder configured to implement an encoding method according to a specific embodiment of the present invention is shown.

[0056] - Figure 4B FIG. 4 shows a device suitable for implementing an encoding method according to another specific embodiment of the present invention.

[0057] - Figure 5A An example of a decoder configured to implement a decoding method according to a specific embodiment of the present invention is shown.

[0058] - Figure 5B FIG. 4 shows a device suitable for implementing a decoding method according to another specific embodiment of the present invention.

[0059] - Figure 6A and 6B The images of a 360° omnidirectional video encoded by independent tiles and generated from two views of two base layers and used for Figure 6A The image is encoded as a reference image,

[0060] - Figures 7A to 7C The projection of a 360° omnidirectional video in a 2D plane with a cubemap type projection, the 3D spherical representation of the 360° omnidirectional video in an XYZ reference, and the views extracted from the 360° fully immersive content in a 2D plane according to a rectilinear projection are shown, respectively.

[0061] - Figure 7Dshows the relationship between different geometric projections,

[0062] - Figure 8 The process of constructing a reference image is shown.

[0063] Figure 2A 、 2C to 2E and 2G and Figures 7A to 7B The images are extracted from 360° videos provided by LetInVR within the framework of JVET (Joint Video Exploration Team) (JVT-D0179: Test Sequences for Virtual Reality Video Decoding by Letin VR, October 15–21, 2016). DETAILED DESCRIPTION

[0064] General principles

[0065] The general principle of the invention is to scalably encode a data stream so that a 360° video can be reconstructed and rendered when the receiver is adapted to receive and render such a 360° video, and a 2D or 3D video can be reconstructed and rendered when the receiver is adapted only to render 2D or 3D video.

[0066] In order to reduce the transmission cost of a stream comprising 2D or 3D video and 360° video, according to the present invention, the 2D or 3D video is encoded in a base layer, while the 360° video is encoded in an enhancement or improvement layer predicted from the base layer.

[0067] According to a particular embodiment of the invention, the stream may comprise several base layers, each base layer corresponding to a 2D or 3D video corresponding to a view of the scene. Thus, the enhancement layers are encoded by prediction based on all or part of the base layers included in the stream.

[0068] Implementation example

[0069] Figure 1AThe steps of an encoding method according to a specific embodiment of the present invention are shown. According to this specific embodiment of the present invention, a 360° video is scalably encoded by extracting views from the 360° video and by encoding each view in a base layer. The term "view" is understood here to mean a sequence of images obtained from a viewpoint of a scene captured by the 360° video. Such an image sequence can be a sequence of monoscopic images in the case of a 2D 360° video or a sequence of stereoscopic images in the case of a 3D 360° video. In the case of a stereoscopic image sequence, each image comprises a jointly encoded left view and a right view, for example in the form of images generated by placing the left view and the right view side by side or one above the other. The encoder that encodes such a stereoscopic image sequence in a base layer or an enhancement layer then encodes each image comprising the left view and the right view as a classic 2D image sequence.

[0070] In the following, we describe an embodiment where the omnidirectional video is a 360° video in 2D.

[0071] Here, we describe an embodiment where two base layers are used to encode an enhancement layer.In general, the method described here is applicable to the case where multiple views N (N greater than or equal to 1) are used for encoding of an enhancement layer.

[0072] The number of base layers is independent of the number of views used to generate the 360° video. For example, the number of base layers encoded in the scalable data stream is determined during content generation, or the number of base layers can be determined by the encoder to optimize bit rate.

[0073] During steps 10 and 11, a first view and a second view are extracted from the 360° video. Views [1] and [2] are encoded during encoding steps 12 for encoding the base layer BL[1] and 13 for encoding the base layer BL[2], respectively.

[0074] In a specific embodiment described herein, the base layers BL[1] and BL[2] are coded independently of each other, i.e., there are no coding dependencies (prediction, coding context, etc.) between the coding of an image of the base layer BL[1] and the coding of an image of the base layer BL[2]. Each base layer BL[1] or BL[2] can be decoded independently of the other base layer.

[0075] According to another embodiment, the base layers BL[1] and BL[2] may be encoded dependently, for example to improve compression efficiency. However, this embodiment of the invention requires that the decoder should be able to decode both base layers to render classic 2D video.

[0076] Each coded / reconstructed image of the base layers BL[1] and BL[2] is then geometrically projected (steps 14 and 15, respectively) onto the same reference image I ref The result is a partially filled reference image containing samples interpolated from one or more projected views of the base layer. Figure 8 The construction of the reference image is described in more detail.

[0077] Figures 2A to 2C An embodiment is shown in which a single base layer is used. According to this embodiment, the images of the 360° video have a spatial resolution of 3840×1920 pixels and are generated by equirectangular projection, and the 360° image sequence has a frequency of 30 images per second. Figure 2C An image of a 360° video at time instant t encoded in the enhancement layer is shown.

[0078] exist Figure 2A The image at time t of a view extracted from a 360° video is shown in FIG. For example, such a view is extracted from a 360° video with the following coordinates: yaw = 20°, pitch = 5°, horizontal FOV (field of view) = 110° and vertical FOV = 80°, the spatial resolution of the image of the extracted view is 1920×960 pixels, and the temporal frequency is 30 images per second. The yaw and pitch coordinates correspond to the center of the geometric projection of the image of the base layer view ( Figure 2B The yaw and pitch coordinates correspond to Figure 7B The angle θ and the angle at the midpoint P of the pivot format shown The horizontal FOV and vertical FOV parameters correspond to Figure 7B The horizontal and vertical dimensions of the image of the extracted view centered at point P in the pivoted format shown; Figure 7C Renders this image of the extracted view in .

[0079] Figure 2B The reference image I is shown ref , the reference image is used in Figure 2A The image of the predicted 360° video at time t after the equirectangular geometric projection of the base layer image shown.

[0080] Figures 2D to 2G An embodiment is shown in which two base layers are used. According to this embodiment, the images of the 360° video have a spatial resolution of 3840×1920 pixels and are generated by equirectangular projection, and the 360° image sequence has a frequency of 30 images per second. Figure 2G An image of a 360° video at time instant t encoded in the enhancement layer is shown.

[0081] exist Figure 2D1 shows an image at time t of the first view extracted from the 360° video. For example, such a first view is extracted from the 360° video using the following coordinates: yaw = 20°, pitch = 5°, horizontal FOV (field of view) = 110°, and vertical FOV = 80°; the spatial resolution of the image of the extracted first view is 1920×960 pixels, and the temporal frequency is 30 images per second.

[0082] exist Figure 2E ] shows an image of the second view extracted from the 360° video at time t. For example, such a second view is extracted from the 360° video using the following coordinates: yaw = 20°, pitch = 5°, horizontal FOV (field of view) = 110°, and vertical FOV = 80°; the spatial resolution of the image of the extracted second view is 1920×960 pixels, and the temporal frequency is 30 images per second.

[0083] Figure 2F The reference image I is shown ref , the reference images are used to Figure 2D and 2E The image of the 360° video is predicted at time t after the equirectangular geometric projection of the images of the first view and the second view shown.

[0084] To project the reconstructed image of the base layer in the reference image, the following geometric transformation steps are applied.

[0085] The representation of 360° omnidirectional video in a plane is defined by a geometric transformation that characterizes how the 360° omnidirectional content represented in a sphere fits into the representation in a plane. The spherical representation of the data serves as a pivot format; it can represent points captured by an omnidirectional video device. Figure 7B Such an XYZ 3D spherical representation is shown in .

[0086] For example, a 360° video is represented by an equirectangular geometric transformation, which can be seen as the projection of a point on a cylinder surrounding a sphere. Of course, other geometric transformations are also possible, for example, a cube map projection corresponding to the projection of a point on a cube surrounding a sphere. The faces of the cube are ultimately unfolded onto a plane to form a 2D image. For example, in Figure 7A Such a stereographic projection is shown in .

[0087] Figure 7D A more detailed view of the relationship between the different formats mentioned above is shown. The conversion from the equirectangular format A to the cubic diagram format B is done via the pivot format C, which is Figure 7B The samples in the spherical XYZ system shown in are characterized. Similarly, the extraction of view D from format A is done through this pivot format C. The extraction of immersive content view is characterized by geometric transformation, such as by Figure 7C The geometric transformations are realized by straight line projections of points on the sphere of the plane shown in the ABCD plane in FIG. This projection is characterized by position parameters such as yaw, pitch, and horizontal and vertical fields of view (FOV). The mathematical properties of these different geometric transformations are recorded in the document JVET-G1003 (Algorithmic Description of Projection Format Conversion and Video Quality Indicators in 360Lib, 4th Edition, Y.Ye, E.Alshina, J.Boyce, JVET of ITU-T SG16 WP3 and ISO / IEC JTC 1 / SC29 / WG 11, 7th Meeting, Torino, Italy, July 13-21, 2017).

[0088] Figure 8 The different steps to achieve the conversion between the two formats are shown. First, a correspondence table is created at E80 to place the position of each sample in the target image (I ref ) (corresponding to Figure 1A For each location (u,v) in the target image, the following steps apply:

[0089] At E81: Convert the coordinates (u,v) of the target image to the pivot system XYZ.

[0090] In E82: Project the XYZ coordinates of the pivot system in the source image (u',v').

[0091] At E83: Update the correspondence table relating the positions in the target format and the source format.

[0092] Once the correspondence table is created, the image is compared to the source image (corresponding to the reference image) during a step E84. Figure 1A The corresponding positive values ​​(u', v') in the reconstructed images of the base layers BL[1] and BL[2] in the described example correspond to the values ​​in the target image (I ref ) is interpolated for each pixel value (u, v) in the base layer. When applying a Lanczos type interpolation filter to the decoded image of the base layer at the correspondingly placed position, the interpolation (u'v') can be done before assigning the value.

[0093] exist Figure 1A Step 16 of the encoding method shown is by using a reference image I generated from the base ref The 360° video is encoded in the enhancement layer EL using prediction with respect to the base layers BL[1] and BL[2].

[0094] In step 17, the data encoded during steps 12, 13 and 16 are multiplexed to form a binary stream including the encoded data of the base layers BL[1] and BL[2] and the enhancement layer EL. ref The projection data is also encoded in a binary stream and transmitted to the decoder.

[0095] The encoding steps 12 , 13 and 16 may advantageously be implemented by a standard video encoder, for example by a standard scalable SHVC encoder of the HEVC standard.

[0096] Figure 1B Shown according to reference Figure 1A An example of a binary stream generated by the described method. In this example, the binary stream includes:

[0097] - coded data of the base layers BL[1] and BL[2],

[0098] - a piece of information PRJ indicating the type of geometric projection used to represent the omnidirectional content, for example a value indicating an equirectangular projection,

[0099] - a piece of information PRJ_B1, PRJ_B2 representing the projection parameters used to extract the views and their position in the 360° video from the views of the base layers BL[1] and BL[2] respectively.

[0100] The information representing the projection and position parameters of the base layer view may be encoded, for example, in the form of view coordinates (yaw, pitch, HFOV, VFOV) matching the projection type used to extract the view (rectilinear projection).

[0101] The information representing the projection and position parameters of the base layer view can be encoded only once in the binary stream. Therefore, it is valid for the entire image sequence.

[0102] The information representing the projection and position parameters of the base layer view can be encoded several times in the binary stream, for example, at each picture, or at each group of pictures. Therefore, it is only valid for one picture or a group of pictures.

[0103] When encoding at each image information representing the projection and position parameters of a view, this variant obtains the advantage that the view extracted at each moment of the sequence can correspond to a view of an object in motion in the scene and tracked over time.

[0104] When information representing the projection and position parameters of a view is encoded for a set of images, this variant obtains the advantage that a video sequence encoded in the base layer can change its viewpoint over time, thus allowing events to be tracked through different viewpoints over time.

[0105] Figure 3 The steps of a decoding method according to a specific embodiment of the present invention are shown.

[0106] According to this particular embodiment of the invention, the scalable binary stream representing the 360° video is demultiplexed during step 30. The encoded data of the base layers BL[1] and BL[2] in the example described herein are sent to the decoder for decoding (steps 31, 33 respectively).

[0107] Then, similar to the encoding method, the reconstructed image of the base layer is projected (steps 32, 34 respectively) on the reference image I ref The geometric projection is performed according to the projection data (projection type, information about the projection and position of the view) provided in the binary stream.

[0108] As described above, the coded data of the enhancement layer EL is decoded (step 35), and the reference image I generated from the geometric projection of the base layer is used. ref To reconstruct images from 360° videos.

[0109] The scalable binary stream representing the 360° video can therefore handle any type of receiver. This scalable stream also enables each receiver to decode and reconstruct either 2D video or 360° video, depending on its capabilities.

[0110] According to the above decoding method, classic receivers such as PCs, TVs, and tablets will only decode one base layer and render a 2D image sequence, while receivers suitable for 360° video, such as virtual reality helmets and smartphones, will decode the base layer and enhancement layer and render the 360° video.

[0111] Figure 4A A more detailed illustration of the steps of encoding the base layer and the enhancement layer of the method described above according to a specific embodiment of the present invention is provided. Here, the case of encoding the enhancement layer by prediction from encoding view k from the base layer and encoding 360° omnidirectional video is described.

[0112] Each image of view k to be encoded is subdivided into pixel blocks, and each pixel block is then encoded classically by spatial or temporal prediction using previously constructed reference images of the image sequence of view k.

[0113] Classically, the prediction module P determines the k c The prediction of the current block B k cEncoded by spatial prediction with respect to other blocks of the same image, or by spatial prediction with respect to previously coded and reconstructed blocks stored in memory MEM b is encoded using temporal prediction of blocks of the reference picture of view k in k.

[0114] By calculating the current block B k c The prediction residual is obtained by taking the difference between the prediction and the one determined by the prediction module P.

[0115] This prediction residual is then transformed by a transformation module T implementing, for example, a DCT (Discrete Cosine Transform). The transformed coefficients of the residual block are then quantized by a quantization module Q and then coded by an entropy coding module C to form the coded data of the base layer BL[k].

[0116] Via module Q -1 Perform inverse quantization and module T -1 The inverse transform is performed to reconstruct the prediction residual and added to the prediction determined by the prediction module P to reconstruct the current block.

[0117] The reconstructed current block is then stored in order to reconstruct the current image and so that this reconstructed current image can be used as a reference during encoding of subsequent images of view k.

[0118] When reconstructing the current image of view k, the projection module PROJ is as follows Figure 2B As shown and according to the geometric transformation described above, the reconstructed image is the reference image I of the 360° video ref Perform geometric projection in .

[0119] The reference image I is obtained by projecting the reconstructed image of the base layer ref is stored in the enhancement layer MEM e in the memory.

[0120] As in the case of the base layer, the 360° omnidirectional video is encoded image by image and block by block. e Each pixel block is classically encoded using spatial or temporal prediction based on the reference image in [1].

[0121] Classically, the prediction module P determines the current block B of the current image of the 360° omnidirectional video e c The prediction of the current block B e c Encoded by spatial prediction with respect to other blocks of the same image, or by encoding with respect to a block stored in memory MEM e The 360° video is encoded using temporal prediction of blocks of previously encoded and reconstructed reference images.

[0122] According to the present invention, advantageously, the current block B e c It can also be obtained by comparing the reference image I obtained from the base layer ref For example, this coding mode is reported in the coded data EL of the enhancement layer by an inter-frame coding mode that signals the temporal coding of the block, a zero motion vector and a reference index indicating the time at which the block is located in the indicated image I. ref The memory MEM used e This information is encoded by the entropy encoder C. This specific embodiment of the present invention enables the reuse of the existing syntax of the time coding mode of the existing standard. Of course, other types of signaling are also possible.

[0123] For example, the prediction mode determined for the current block B is selected from the possible prediction modes and by selecting the mode that minimizes the bit rate / distortion criterion. e c The prediction mode of the encoding.

[0124] Once the current block B e c If the prediction mode is selected, the current block B is calculated e c The prediction residual is obtained by taking the difference between the prediction and the one determined by the prediction module P.

[0125] This prediction residue is then transformed by a transformation module T implementing, for example, a DCT (Discrete Cosine Transform).The transformed coefficients of the residual block are then quantized by a quantization module Q and then coded by an entropy coding module C to form coded data of the base layer EL.

[0126] Via module Q -1 Perform inverse quantization and module T -1 The inverse transform is performed to reconstruct the prediction residual and added to the prediction determined by the prediction module P to reconstruct the current block.

[0127] The reconstructed current blocks are then stored in order to reconstruct the current image, and each reconstructed current image can be used as a reference during encoding of subsequent images of the 360° omnidirectional video.

[0128] Here, the encoding has been described for the case where a single view k is coded in the base layer. The method can be easily converted to the case where there are several coded views in the equivalent number of base layers. Each image reconstructed at time t in the base layer is projected onto the same reference image I of the 360° video. ref To encode the image of the 360° video at time t.

[0129] Figure 4B The simplified structure of a coding device COD suitable for implementing the coding method according to any of the specific embodiments of the present invention described above is shown.

[0130] Such a coding device comprises a memory MEM4 and a processing unit UT4 which is equipped with a processor PROC4, for example.

[0131] According to one embodiment of the invention, the coding method is implemented by a computer program PG4 stored in the memory MEM4 and managing the processing unit UT4. The computer program PG4 comprises instructions for implementing the steps of the coding method as described above when the program is executed by the processor PROC4.

[0132] During initialization, the code instructions of the computer program PG4 are loaded into a memory (not shown) and then executed by the processor PROC4. The processor PROC4 of the processing unit UT4 implements the reference program according to the instructions of the computer program PG4. Figure 1A Or the steps of the encoding method described in 4A.

[0133] According to another specific embodiment of the present invention, through the functional modules (P, T, Q, Q -1 、T -1 , C, PROJ) to implement the coding method. To this end, the processing unit UT4 cooperates with different functional modules and the memory MEM4 to implement the steps of the coding method. The memory MEM4 includes in particular the memory MEM b MEM e .

[0134] The various functional modules described above may be implemented in hardware or software. In hardware, such functional modules may include a processor, memory, and program code instructions to implement the functions corresponding to the modules when the code instructions are executed by the processor. In hardware, such functional modules may be implemented by any type of adaptive coding circuit, such as, but not limited to, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or logic unit wiring.

[0135] Figure 5A A more detailed illustration of the steps for decoding the base layer and enhancement layer of the method described above according to a specific embodiment of the present invention is provided. Here, the case of decoding the enhancement layer EL by prediction from the base layer BL[k] encoding view k and decoding 360° omnidirectional video is described.

[0136] View k and 360° omnidirectional video are decoded image by image and block by block.

[0137] Typically, the data of the base layer BL[k] is decoded by the entropy decoding module D. Then, for the current block of the current image to be reconstructed, the inverse quantization module Q -1 The entropy decoded coefficients are inverse quantized and transformed by the inverse transform module T -1 The inverse transform is performed to reconstruct the prediction residual. The prediction module P determines the prediction for the current block based on the signaling data decoded by the entropy decoding module D. The prediction is added to the reconstructed prediction residual to reconstruct the current block.

[0138] The reconstructed current block is then stored so as to reconstruct the current image, and this reconstructed current image is stored in the base layer MEM b and can therefore be used as reference during decoding of subsequent pictures of view k.

[0139] When reconstructing the current image of view k, the projection module PROJ is as follows Figure 2B As shown and according to the geometric transformation described above, the reconstructed image is the reference image I of the 360° video ref Perform geometric projection in .

[0140] The reference image I is obtained by projecting the reconstructed image of the base layer ref is stored in the enhancement layer MEM e The reference image is stored in memory.

[0141] The data of the enhancement layer EL is decoded by the entropy decoding module D. Then, for the current block of the current image to be reconstructed, the inverse quantization module Q -1 The entropy decoded coefficients are inverse quantized and transformed by the inverse transform module T -1 The prediction module determines the prediction of the current block based on the signaling data decoded by the entropy decoding module D.

[0142] For example, the decoded syntax data indicates that the image is obtained by comparing it to a reference image I obtained from the base layer. ref The inter-layer prediction of the co-located blocks is used for the current block B e c Therefore, the prediction module determines that the prediction corresponds to the reference image I ref The current block B in e c Colocalized blocks.

[0143] The prediction is added to the reconstructed prediction residual to reconstruct the current block. The reconstructed current block is then stored to reconstruct the current image of the enhancement layer.

[0144] This reconstructed image is stored in the enhancement layer MEM e to a reference image memory for use as a reference during decoding of subsequent images of the 360° video.

[0145] Figure 5B The simplified structure of a decoding device DEC suitable for implementing the decoding method according to any of the specific embodiments of the present invention described above is shown.

[0146] Such a decoding device comprises a memory MEM5 and a processing unit UT5, which is equipped with a processor PROCR5, for example.

[0147] According to one embodiment of the invention, the decoding method is implemented by a computer program PG5 stored in the memory MEM5 and driving the processing unit UT5. When the program is executed by the processor PROC5, the computer program PG5 comprises instructions for implementing the steps of the decoding method as described above.

[0148] During initialization, the code instructions of the computer program PG5 are loaded into a memory (not shown) and then executed by the processor PROC5. The processor PROC5 of the processing unit UT5 implements the instructions of the computer program PG5 in particular. Figure 3 Or the steps of the decoding method described in 5A.

[0149] According to another specific embodiment of the present invention, through the functional modules (P, Q -1 、T -1 , D, PROJ) to implement the decoding method. To this end, the processing unit UT5 cooperates with different functional modules and the memory MEM5 to implement the steps of the decoding method. The memory MEM5 may in particular include a memory MEM b MEM e .

[0150] The various functional modules described above may be implemented in hardware or software. In hardware, such functional modules may include a processor, memory, and program code instructions to implement the functions corresponding to the modules when the code instructions are executed by the processor. In hardware, such functional modules may be implemented by any type of adaptive coding circuit, such as, but not limited to, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or logic unit wiring.

[0151] According to a specific embodiment of the present invention, the blocks of the image of the enhancement layer are encoded by block groups. Such block groups are also called tiles. Each block group, that is, each tile, is encoded independently of other tiles.

[0152] Each tile can then be decoded independently of the other tiles. Figure 6A Such tiles (TE0 to TE11) are shown in , which represent an image of a 360° omnidirectional video at a time when 12 tiles are defined and the image is completely overlapped.

[0153] The term "independent coding of a tile" is understood here to mean the coding of blocks of a tile without using any spatial prediction from blocks of another tile of the image or without using temporal prediction from blocks of tiles of a reference image that are not co-located with the current tile.

[0154] like Figure 6A and 6B As shown, each tile is encoded based on one or more base layers through temporal prediction or inter-layer prediction. Figure 6A and 6B In , tiles TE4 and TE7 are encoded by inter-layer prediction with respect to the image projected in the reference image Iref of view 1, and tiles TE3 and TE6 are encoded by inter-layer projection with respect to the image projected in the reference image Iref of view 2.

[0155] According to this particular embodiment of the invention, it may happen that a receiver suitable for decoding and rendering 360° video decodes only the tiles required for the current area of ​​the 360° image viewed by the user. In fact, during the rendering of the 360° video, the user cannot see the entire image of the video at time t, i.e., he cannot see in all directions simultaneously and can only see the area of ​​the image facing his line of sight at time t.

[0156] For example, such a viewing area consists of Figure 6A Therefore, according to this embodiment, only the base layer for predicting the area watched by the user is decoded in step 31. Figure 6A and 6B In the example described, based on the enhancement layer EL, Figure 3 Only the base layer corresponding to view 1 is decoded during step 31 of Figure 3 During step 35 of FIGURE 3, only tiles TE4, TE5, TE7 and TE8 are decoded. During step 35, only the portion of the image of the enhancement layer corresponding to tiles TE4, TE5, TE7 and TE8 is reconstructed. Here, the reference image is described in the case where the tiles of the enhancement layer EL to be decoded depend on only one base layer (the base layer of view 1). Figure 6A and 6B According to other variants, for example according to the bitrate / distortion optimization chosen during the coding of the blocks of the enhancement layer, the tiles of the enhancement layer EL may be coded by prediction from several base layers, one block of the tile may be coded by prediction relative to a first base layer, while another block of the same tile may be coded by a base layer different from the first. In this case, all base layers used to predict the blocks of the tiles of the enhancement layer must be decoded.

[0157] To this end, for each tile of the enhancement layer, the coded data stream includes a piece of information identifying the base layer used to predict the tile.

[0158] For example, for each tile, a syntax element indicating the number of base layers used and an identifier of each base layer used is encoded in the data stream.During the step 35 of decoding the enhancement layer, this syntax element is decoded for each tile of the enhancement layer to be decoded.

[0159] The specific embodiment described here limits the use of decoder resources and avoids decoding unnecessary data because the user will not watch it. This embodiment can be implemented by any encoding device and any decoding device described above.

[0160] exist Figure 1A Steps 14 and 15 and Figure 3 Steps 32, 34 project the reconstructed image of the base layer onto the same reference image inserted in the memory of the enhancement layer reference image, in which case the above encoding and decoding methods have been described.

[0161] When the number of base layers is limited (eg when it is 1 or 2), such reference pictures have large undefined areas, eg set to zero by default, thereby unnecessarily using memory resources.

[0162] According to another variant, the reconstructed image of the base layer projected onto the enhancement layer can be stored in a reference sub-image. For example, a sub-image can be used for each base layer. Each sub-image is stored in association with decoding information, allowing the encoder and / or decoder to determine the position of the sub-image in the enhancement image. This variant has the advantage of saving storage space by avoiding the need for a reference image in the enhancement layer, for which most samples are zero.

[0163] This variation can be implemented independently of the decoder and / or encoder.

Claims

1. A method for encoding a data stream representing omnidirectional video, characterized in that The method comprises: a step of encoding in said stream at least two base layers each representing a 2D or 3D video, said 2D or 3D video of each base layer representing a different view of the same scene captured by said omnidirectional video, said at least two base layers being encoded independently of each other, a step of encoding in said stream an enhancement layer representing said omnidirectional video, said enhancement layer being encoded by prediction with respect to at least one of said at least two base layers, The image of the enhancement layer is encoded with the aid of a set of tiles, each tile covering an area of ​​the image of the enhancement layer, each area being different and separate from other areas of the image of the enhancement layer, and each tile being encoded by prediction relative to at least one of the at least two base layers.

2. The method of claim 1 , wherein predicting the enhancement layer with respect to the at least one base layer comprises, for encoding or reconstructing at least one image of the enhancement layer: generating a reference image obtained by geometrically projecting an image, called base image, reconstructed from said at least one base layer onto said reference image, The reference picture is stored in a memory of reference pictures of the enhancement layer.

3. The method of claim 2, wherein the data stream includes a piece of information indicating a type of geometric projection used to represent the omnidirectional video.

4. The method according to claim 2 or 3, wherein the data stream comprises a piece of information representing parameters of the projection and position of the base image in the image of the omnidirectional video, the information being used to project the base image onto the reference image.

5. The method according to claim 4, wherein the piece of information representing the parameters of the projection and position of the base image is encoded in the data stream at each image of the omnidirectional video. The method of claim 1 , wherein the view represented by the 2D or 3D video is a view extracted from the omnidirectional video. 7 . The method according to claim 1 , wherein the encoding of the enhancement layer comprises, for each tile of the enhancement layer to be encoded, encoding information identifying at least one base layer used to predict the tile.

8. A method for decoding a data stream representing omnidirectional video, characterized in that The method comprises: a step of decoding from said stream at least one base layer representing a 2D or 3D video representing a view of the same scene captured by said omnidirectional video, a step of decoding from said stream an enhancement layer representing said omnidirectional video, said enhancement layer being decoded by prediction with respect to said at least one base layer, wherein the data stream comprises at least two base layers, the at least two base layers comprising the at least one base layer representing a 2D or 3D video and another base layer, the 2D or 3D video of the another base layer representing different views of the same scene captured by the omnidirectional video, the at least two base layers being encoded independently of each other; wherein the image of the enhancement layer is encoded by means of a set of tiles, each tile covering a region of the image of the enhancement layer, each region being distinct and separate from other regions of the image of the enhancement layer, each tile being encoded by prediction relative to the at least one base layer, and decoding the enhancement layer comprises: reconstructing a portion of the image of the enhancement layer, comprising decoding the tiles of the enhancement layer covering the portion of the image of the enhancement layer to be reconstructed, and Decoding the at least one base layer comprises decoding the base layer for predicting the tiles covering the portion of the image of the enhancement layer to be reconstructed.

9. The method of claim 8, wherein predicting the enhancement layer with respect to the at least one base layer comprises, for encoding or reconstructing at least one image of the enhancement layer: generating a reference image obtained by geometrically projecting an image, called base image, reconstructed from said at least one base layer onto said reference image, The reference picture is stored in a memory of reference pictures of the enhancement layer.

10. The method of claim 9, wherein the data stream includes a piece of information indicating a type of geometric projection used to represent the omnidirectional video.

11. The method according to claim 9 or 10, wherein the data stream comprises a piece of information representing parameters of the projection and position of the base image in the images of the omnidirectional video, the information being used to project the base image onto the reference image.

12. The method according to claim 11, wherein the piece of information representing the parameters of the projection and position of the base image is encoded in the data stream at each image of the omnidirectional video.

13. The method of claim 8, wherein the view represented by the 2D or 3D video is a view extracted from the omnidirectional video.

14. The method according to claim 8, further comprising: For each tile of the enhancement layer to be decoded, the decoding is performed on information identifying at least one base layer used to predict the tile.

15. An apparatus for encoding a data stream representing omnidirectional video, characterized in that The device comprises: means for encoding in said stream at least two base layers, each base layer representing a 2D or 3D video, said 2D or 3D video of each base layer representing a different view of the same scene captured by said omnidirectional video, said at least two base layers being encoded independently of each other, means for encoding in said stream an enhancement layer representing said omnidirectional video, said means for encoding said enhancement layer comprising means for predicting said enhancement layer relative to at least one of said at least two base layers, The image of the enhancement layer is encoded with the aid of a set of tiles, each tile covering an area of ​​the image of the enhancement layer, each area being different and separate from other areas of the image of the enhancement layer, and each tile being encoded by prediction relative to at least one of the at least two base layers.

16. An apparatus for decoding a data stream representing omnidirectional video, characterized in that The device comprises: means for said decoding in said stream at least one base layer representing a 2D or 3D video representing views of the same scene captured by said omnidirectional video, means for decoding, in said stream, an enhancement layer representing said omnidirectional video, said means for decoding said enhancement layer comprising means for predicting said enhancement layer relative to said at least one base layer, wherein the data stream comprises at least two base layers, the at least two base layers comprising the at least one base layer representing a 2D or 3D video and another base layer, the 2D or 3D video of the another base layer representing different views of the same scene captured by the omnidirectional video, the at least two base layers being encoded independently of each other; wherein the image of the enhancement layer is encoded by means of a set of tiles, each tile covering a region of the image of the enhancement layer, each region being distinct and separate from other regions of the image of the enhancement layer, each tile being encoded by prediction relative to the at least one base layer, and decoding the enhancement layer comprises: reconstructing a portion of the image of the enhancement layer, comprising decoding the tiles of the enhancement layer covering the portion of the image of the enhancement layer to be reconstructed, and Decoding the at least one base layer comprises decoding the base layer for predicting the tiles covering the portion of the image of the enhancement layer to be reconstructed.

Images