Signaling encapsulated data representing primary video sequence and associated secondary video sequence

Abstract

The present application relates to a method and apparatus for signaling encapsulated data representing a primary video sequence associated with a secondary video sequence, the primary video sequence and the secondary video sequence being generated by an image projection method applied to a signal captured by a sensor. The method comprises writing (110) visual content of the primary video sequence and visual content of the secondary video sequence as encapsulated data; and writing (120) an alignment state as encapsulated data, the alignment state indicating whether the visual content of the primary video sequence is aligned with the visual content of the secondary video sequence.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based upon and claims the benefit of priority from European patent application No. “22305922.1” filed on June 24, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application generally relates to signaling packaged data representing both visual content and visual content description of both a primary video sequence and an auxiliary video sequence associated with the primary video sequence. In particular, but not exclusively, the present application relates to signaling packaged data representing visual content and visual content description of a color video sequence associated with a depth video sequence. Background Art

[0004] This section is intended to introduce the reader to various aspects of the technology that may be related to various aspects of at least one exemplary embodiment of the present application described and / or claimed below. It is believed that this discussion helps to provide the reader with background information to facilitate a better understanding of the various aspects of the present application. Therefore, it should be understood that these statements are to be read in this light, and not as an admission of prior art.

[0005] A pixel corresponds to the smallest display unit on the screen, which may be composed of one or more light sources (1 for a monochrome screen, or 3 or more for a color screen).

[0006] A video frame (also denoted as an image) comprises at least one component (also referred to as a channel) determined by a specific picture / video format that specifies all information associated with sample values ​​and all information that can be used by a display unit and / or any other device to display and / or decode video picture data associated with the video frame in order to generate pixel values.

[0007] A video frame comprises at least one component which is typically represented in the shape of a 2D array of samples.

[0008] A monochrome video frame includes a single component, whereas a color video frame (also denoted as a texture video frame) may include three components.

[0009] For example, a color video frame may include a luma (or luminance) component and two chrominance components when the image / video format is the well-known (Y, Cb, Cr) format, or may include three color components (one for red, one for green, and one for blue) when the image / video format is the well-known (R, G, B) format.

[0010] Each component of a video frame may include a number of samples relative to the number of pixels of the screen on which the video picture will be displayed. For example, the number of samples included in a component may be the same as the number of pixels of the screen on which the video frame will be displayed, or a multiple (or fraction) thereof.

[0011] The number of samples included in a component may also be a multiple (or fraction) of the number of samples included in another component of the same video frame.

[0012] For example, in case the video format comprises a luma component and two chroma components like in (Y, Cb, Cr) format, the chroma components may contain half the number of samples in width and / or height relative to the luma component, depending on the color format considered.

[0013] A sample is the smallest visual information unit of a component that makes up a video frame. A sample value may be, for example, a luminance or chrominance value or a color value of a red, green or blue component in the (R, G, B) format.

[0014] A pixel value of the screen can be represented by one sample for a monochrome video picture and by multiple co-located samples for a color video frame. A co-located sample associated with a pixel means a sample that corresponds to the position of the pixel in the screen.

[0015] A video frame is usually viewed as a set of pixel values, with each pixel represented by at least one sample.

[0016] The RGBD format associates a color image with a single-channel image that encodes and decodes depth information for each corresponding pixel in the color image at a certain bit depth.

[0017] The same concept can be extended to video sequences, where each image of the color video sequence is associated with a corresponding depth image of the depth video sequence.

[0018] The RGBD format is suitable for encapsulating a primary video sequence and associated auxiliary video sequences into a container for file-based storage or packet-based network transmission.

[0019] The main video sequence may be a color video sequence, that is, the samples of the main video sequence represent pixel colors, but the main video sequence may also be any other type of video sequence, such as an infrared video sequence, a reflectance video sequence, etc.

[0020] The auxiliary video sequence may be a depth video sequence, ie, the samples of the auxiliary video sequence represent depth information associated with the samples of the primary video sequence. However, the auxiliary video sequence may also be any other video sequence, such as an infrared video sequence, a reflectance video sequence, etc.

[0021] Depth information can follow different conventions, but it usually represents the distance from the camera center C to a point P on the depth image plane IP, along Figure 1 The main z-axis shown gives the direction.

[0022] Compared to other alternative formats such as point clouds or meshes, the RGBD format is a very attractive choice for representing volumetric information as color and depth information because it has strong advantages when it comes to real-time (no sensor fusion, 3D reconstruction required at intermediate nodes), low cost (depth sensors are very ubiquitous), and distribution friendliness (depth data can be carried as video, which can utilize video encoders and decoders, especially in hardware, and widely deployed packaging and streaming formats similar to 2D video streams).

[0023] The RGBD format may be used in several applications such as "holographic" calling, which provides a user with the ability to view another person as a volumetric object rather than as a flat image as is conventionally experienced in popular video calling applications.

[0024] Looking Glass Portrait is also an application for which the RGBD format is well suited, as the device features a depth camera sensor coupled to a conventional camera. Another example is AR glasses with 6DoF capabilities that blend remote participants into the receiving user's real scene.

[0025] The RGBD format may also be well suited for remote rendering of complex 3D objects in complex applications such as architecture or mechanical design, where the 3D objects may quickly exceed the capacity of the local computer. One solution is to rely on lightweight workstations connected to a network node that renders the complex objects. The rendering results of the user's current perspective can be streamed in a video stream in RGBD format. This allows volumetric rendering of the object on the user side, while the rendering is effectively done remotely. This is typically relevant when the user is wearing a VR or AR HMD, but a flat screen setup can be applied on the user side as well.

[0026] From a video sequence in RGBD format, it is indeed possible to create stereoscopic views from it, i.e. a left eye view (primary video sequence) and a right eye view (auxiliary video sequence) for the end user. This process is possible in cases where there is minimal occlusion in the scene from both left and right eye viewpoints. When occlusion occurs, there are several techniques in the field (inner coloring, AI-based generation, patch texture transfer) to visually repair those areas where no visual information is available, i.e. those that are occluded. Note that some services can also decide to stream both the left and right views back to the user at once. However, this would mean that the inter-view distance would be incorporated and targeted to that user and his device, which would require more negotiation on the rendering nodes and per-device adaptation, which hinders large-scale distribution of the same content to thousands of users. Providing video sequences in RGBD format allows for more device independence, since the left and right views (in the case of stereoscopic displays, such as AR or VR HMDs) would be generated on the user side based on the device characteristics and the inter-eye distance (also known as inter-pupillary distance) of the user.

[0027] The RGBD format can also be used for free viewpoint applications, such as 3D TV. In free viewpoint applications, the user is able to choose the viewing angle from which to view a 3D scene. However, since the number of views is almost infinite (the user can choose any given viewing angle in a continuous range), this use case usually requires more than one input view of the 3D scene to achieve better results. However, the processing corresponding to synthesizing views from the same input data has similarities to the 3D TV use case where the receiver needs to display two views to the user (one for the left eye and one for the right eye). As a result, the basic processing on the device corresponds to synthesizing at least one view (possibly two) from the RGBD format, which can store a main video sequence representing the color information of the 3D scene and an auxiliary video sequence representing the depth information of the 3D scene.

[0028] Range imaging is the name of a collection of techniques for producing 2D images showing the distance from a specific point, usually associated with some type of sensor device, to a point in the real scene. The resulting range image (also called a depth image) has pixel values ​​(sample values) corresponding to that distance. If the sensor used to produce the depth image is properly calibrated, the pixel values ​​can be given directly in physical units (such as meters).

[0029] There are different sensors and techniques to generate depth images from the sensed environment, such as stereo triangulation, light sheet triangulation, structured light, time of flight, interferometry, or coded aperture (https: / / en.wikipedia.org / w / index.php?title=Range_imaging&oldid=1050828429 (accessed on May 10, 2022)).

[0030] A depth video sequence captured from a real scene is typically decoupled from a captured color video sequence of the same scene. Therefore, starting from the acquisition step, the visual content of the captured depth and color video sequences cannot be aligned. The visual content of the video sequence is aligned with the visual content of the other video sequence when the samples of the video frames of a video sequence are aligned on the samples of the video frames of another video sequence (i.e., each of the samples of the video sequence corresponds to a sample of the other video sequence). Full alignment means that all samples of all video frames of a video sequence are aligned with the samples of the video frames of the other video sequence, and when only some samples of the video frames of a video sequence are aligned with the samples of the video frames of the other video sequence, the alignment is partial.

[0031] One of the reasons why the sampling of the video frames of the captured depth video sequence is not aligned with the sampling of the video frames of the captured color video sequence is that the captured depth video sequence may have different spatial resolution, different frame rate (temporal sampling / resolution), and temporal incoherence (images taken at different points in time) compared to the captured color video sequence.

[0032] Another reason is that the color and depth video sequences may be captured by different sensors that are not physically at the same location in space. It is also possible that the use of different imaging systems (lens systems) and / or different image projection methods (e.g. planar vs omnidirectional) for projecting the captured scene signal onto the image plane of the sensor may produce different types of distortions to the resulting depth and color images.

[0033] Some imaging systems have both a depth sensor and a color sensor that are distinct and spatially distant from each other, such as Figure 2 Therefore, the images captured by these sensors do not exactly represent the same part of the real scene, as shown in Figure 3 As can be seen in the figure, the sensor captures depth information (depth FOV) and the RGB camera captures color information (color FOV). Since the two images are captured from different sensors, the two images exhibit different characteristics. However, the application expects to be able to associate a pixel from the depth image with a pixel from the color image. Therefore, the Kinect SDK also provides transformation functions (Use Azure Kinect Sensor SDK Image Transformation | Microsoft Docs, https: / / docs.microsoft.com / en-us / azure / kinect-dk / use-image-transformation) to convert color or depth images to other coordinate systems.

[0034] The standard ISO / IEC 14496-12 (ISO / IEC 14496-12 Information technology—Coding of audio-visual objects—Part 12: ISO base media file format) defines a file format (ISOBMFF file structure) that can be used to signal both the visual content and the visual content description of both primary and auxiliary video sequences.

[0035] Basically, the standard ISO / IEC 14496-12 formats a so-called ISOBMFF (ISO Base Media File Format) file as a series of objects, called boxes. An example of a box is a media box containing video data. The media box contains a handler box, defining the properties of the data of the media box as defined in clause 8.4.3 of the standard.

[0036] The standard ISO / IEC 14496-12 allows storing multiple consecutive sample sequences (audio sequences, video sequences, subtitles, etc.) into so-called tracks, which are distinguished by the concept of media handlers. A primary video track can use a "video" handler type in a handler box of a media box containing visual content data of the primary video sequence, and an auxiliary video track can use an "auxiliary" handler type in a handler box of another media box containing visual content data of the auxiliary video sequence data.

[0037] Auxiliary video tracks are encoded identically to the primary video track, but use a different processing type, and are not intended to be displayed visually as video tracks are normally displayed.

[0038] When multiple tracks co-exist in the same ISOBMFF file structure, it is useful to express their relationship so that the application can handle them in an appropriate way. For example, two video tracks may replace each other, in which case the application should only process and display one of the two tracks. In another scenario, the ISOBMFF file structure may contain a primary video track (e.g. a color video sequence) and an auxiliary video track (e.g. a depth video sequence). In this case, the complementarity of the auxiliary video track to the primary video track may be implicit, however, this implicit relationship is broken when there are multiple pairs of primary and auxiliary video sequences. For this reason and many others, the standard ISO / IEC 14496-12 provides the ability to link tracks to express a certain relationship, which is due to the track reference mechanism. This is a track-to-track signaling that can be implemented through a special box called TrackReferenceBox as defined in clause 8.3.3 of the standard ISO / IEC 14496-12.

[0039] In addition to the media handler and track reference concepts, standard ISO / IEC 14496-12 also allows the metadata defined in standard ISO / IEC 23002-2 (ISO / IEC 23002-3 Information technology—MPEG video technologies—Part 3: Representation of auxiliary video and supplemental information) to be stored as metadata items in the video track.

[0040] ISO / IEC 23002-3 is a standard that defines metadata to describe characteristics of auxiliary video sequences that supplement a primary video sequence.

[0041] In its first edition published in 2007, the specification ISO / IEC 23002-3 defines two types of metadata for auxiliary video sequences: depth video sequences (depth maps) and disparity video sequences (disparity maps).

[0042] In short, the standard ISO / IEC 23002-3 defines a syntax element "generic_params" that allows to align the auxiliary video sequence exactly with the primary video sequence (eg in case of subsampling).

[0043] The syntax element "generic params" can be used with any type of auxiliary video sequence. Figure 4As shown, the syntax element "generic params" specifies the signaling differences between the primary video sequence and the auxiliary video sequence in terms of progressive vs interlaced spatial resolution and temporal resolution.

[0044] More specifically, the syntax element "generic params" contains a syntax element "aux_is_one_field" representing a Boolean value, where, if true (=1), it indicates that the auxiliary video sequence is only related to one view of the main video sequence, and, if false (=0), it indicates that the auxiliary video sequence is related to two views of the main video sequence.

[0045] The syntax element "generic params" contains a syntax element "aux_is_bottom_field" representing a Boolean value, which is true (=1), indicating that the auxiliary video sequence only corresponds to the bottom view of the primary video sequence, and which is false (=0), indicating that the auxiliary video sequence only corresponds to the top view of the primary video sequence. If the syntax element "aux_is_one_field" is false, the syntax element "aux_is_bottom_field" is not applicable.

[0046] The syntax element "generic params" also contains a syntax element "aux_is_interlaced" representing a Boolean value, which is true, indicating that any spatial resampling operations on the auxiliary video sequence should be view-based, and which is false, indicating that any spatial resampling operations on the auxiliary video sequence should be frame-based. If the syntax element "aux_is_one_field" is true, the syntax element "aux_is_interlaced" is inferred to be true.

[0047] The syntax element "generic params" also contains syntax elements "position_offset_h" and "position_offset_v", which represent the horizontal and vertical position offsets of the auxiliary video sequence data expressed in 1 / 16 sample positions in the primary video spatial sampling grid, respectively. If the syntax element "primary_is_interlaced" is false, the latter grid is the grid of the video frames of the primary video sequence, and if the syntax element "primary_is_interlaced" is true, the latter grid is the grid of the views of the video frames of the primary video sequence.

[0048] Figure 5An example of a position offset between the primary video sequence data and the auxiliary video sequence data is shown. The auxiliary video sequence data is subsampled by a factor of 4 in the horizontal direction and by a factor of 2 in the vertical direction compared to the spatial resolution of the primary video sequence. The offset due to subsampling corresponds here to "position_offset_h" = 20 and "position_offset_v" = 8 (expressed in 1 / 16 sample positions, i.e. 1.25 sample positions and 0.5 sample positions, respectively).

[0049] The syntax elements "position_offset_h" and "position_offset_v" are useful when the spatial resolution of the auxiliary video sequence is lower than that of its primary counterpart, ie when filtering and decimation processes have been performed which result in shifting.

[0050] When the spatial resolutions of the primary and auxiliary video sequences are different, the receiver shall perform an upsampling operation (although the resampling process is voluntarily kept open) according to ISO / IEC 23002-3. Therefore, the syntax elements "position_offset_h" and "position_offset_v" allow mapping samples of the video frames of the auxiliary video sequence to samples of the video frames of the primary video sequence, without specifying that the resampling process of different implementations may result in different resampled mapped samples.

[0051] A bitstream conforming to standard ISO / IEC 23002-3 shall observe the following restrictions:

[0052] If the auxiliary video sequence and the primary video sequence have the same width, the syntax element "position_offset_h" shall be zero;

[0053] If the syntax element "aux_is_one_field" is false, and if the auxiliary video sequence and the primary video sequence have the same height, the syntax element "position_offset_v" shall be set to zero;

[0054] If the syntax element "aux_is_one_field" is true, and if the height of the auxiliary video sequence is half the height of the primary video sequence, the syntax element "position_offset_v" shall be set to zero.

[0055] The ISO / IEC 23002-3 standard also defines Figure 6 The syntax element "depth params" is shown.

[0056] The syntax element "depth_params" contains the syntax element "nkfar" representing the parameter kfar, which is inferred as follows:

[0057]

[0058] And the syntax element "nknear" represents the parameter knear, which is inferred as follows:

[0059]

[0060] The depth value zp is represented by the unsigned N-bit value m. The depth value zp shall be inferred non-normatively as follows:

[0061]

[0062] kfar and knear specify the extent of the depth information behind and in front of the image, respectively, relative to W. W represents the screen width on the receiver side. W and zp are expressed in the same distance units. W is receiver-dependent and can also be viewed as an arbitrary scaling control for the depth range to be displayed.

[0063] Negative (or positive) values ​​of zp should correspond to positions behind the display (or in front of the display), respectively. If zp is equal to zero, the corresponding position should be on the display.

[0064] Generally, it is recommended that the value of kfar is larger than the value of knear to meet the requirement that the depth range behind the image is larger than the depth range in front of the image. For example, nkfar=128 and nknear=128 correspond to kfar=8 and knear=2.

[0065] In summary, standard ISOBMFF does provide for signaling that a track of an ISOBMFF file is an auxiliary video track to a main video track, where the auxiliary track may contain a depth video sequence.

[0066] The standard ISO / IEC 14496-12 in conjunction with ISO / IEC 23002-3 defines a storage file format for storing primary and auxiliary video sequences, wherein the auxiliary video data may be encoded with any compression scheme.

[0067] However, the standard ISO / IEC 14496-12 in combination with ISO / IEC 23002-3 has disadvantages.

[0068] One of these drawbacks is regarding the signaling of the position offset (syntax elements "position_offset_h" and "position_offset_v"). Signaling the position offset only allows a change in the spatial resolution, but the standard ISO / IEC 14496-12 in combination with ISO / IEC 23002-3 does not provide any solution when the primary and auxiliary video sequences are not projectively aligned (i.e. the primary and auxiliary video sequences result from different image projection methods). The standard ISO / IEC 14496-12 in combination with ISO / IEC 23002-3 assumes that the primary and auxiliary video sequences are always projectively aligned, i.e. using the same image projection method.

[0069] Another disadvantage is that the standard ISO / IEC 14496-12 in combination with ISO / IEC 23002-3 does not specify how exactly to generate the correct sampling (e.g. depth sampling) of the auxiliary video sequence when the primary and auxiliary video sequences have different spatial resolutions due to unspecified resampling operations.

[0070] In short, the prior art does not allow for the simultaneous encapsulation of primary and auxiliary video sequences, both of which may have different characteristics beyond spatial and temporal resolution, such as projections and geometrical characteristics, physical displacements caused by the way the physical sensors are manufactured.

[0071] With the foregoing in mind, at least one exemplary embodiment of the present application has been devised. Summary of the invention

[0072] The following section presents a simplified overview of at least one exemplary embodiment to provide a basic understanding of some aspects of the present application. This overview is not an extensive review of the exemplary embodiments. It is not intended to indicate the key or important elements of the exemplary embodiments. The following overview only presents some aspects of at least one exemplary embodiment in a simplified form as a prelude to a more detailed description provided elsewhere in the document.

[0073] According to a first aspect of the present application, there is provided a method for signaling encapsulation data representing a main video sequence associated with an auxiliary video sequence, the main video sequence and the auxiliary video sequence being generated by an image projection method applied to a signal captured by a sensor, the method comprising writing the visual content of the main video sequence and the visual content of the auxiliary video sequence as encapsulation data; and writing an alignment status as encapsulation data, the alignment status indicating whether the visual content of the main video sequence is aligned with the visual content of the auxiliary video sequence.

[0074] According to a second aspect of the present application, there is provided a method for parsing encapsulated data representing a main video sequence associated with an auxiliary video sequence, wherein the main video sequence and the auxiliary video sequence are generated by an image projection method applied to a signal captured by a sensor, the method comprising obtaining visual content of the main video sequence and visual content of the auxiliary video sequence by parsing the encapsulated data; and obtaining an alignment status by parsing the encapsulated data, the alignment status indicating whether the visual content of the main video sequence is aligned with the visual content of the auxiliary video sequence.

[0075] In one variation, the alignment status further indicates one of the following states:

[0076] - there is no alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence;

[0077] - The main video sequence and the auxiliary video sequence have been generated by the same image projection method, but the sampling of their video frames may not be aligned;

[0078] - there is a one-to-one alignment between the samples of the video frames of the primary video sequence and the samples of the video frames of the auxiliary video sequence; or

[0079] - The alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence is unspecified.

[0080] In a variant, the alignment status indicates that the main video sequence and the auxiliary video sequence have been generated from the same image projection method but the sampling of their video frames may not be aligned, the signaling method further includes writing alignment reference data as encapsulation data, and the parsing method further includes obtaining the alignment reference data by parsing the encapsulation data, the alignment reference data indicating that the sampling of the video frames of the auxiliary video sequence has been aligned on the sampling of the video frames of the main video sequence or vice versa.

[0081] In an exemplary embodiment, if the alignment status indicates that the visual content of the main video sequence and the visual content of the auxiliary video sequence are aligned, the signal transmission method further includes writing the overlap status as encapsulation data, and the parsing method further includes obtaining the overlap status by parsing the encapsulation data, wherein the overlap status indicates whether the visual content of the main video sequence completely overlaps or partially overlaps with the visual content of the auxiliary video sequence.

[0082] In one exemplary embodiment, the overlapping state further indicates:

[0083] - a first overlap window flag indicating the presence of an explicit overlap window within a video frame of the auxiliary video sequence aligned with a video frame of the primary video sequence; and

[0084] - A second overlap window flag indicating the presence of an explicit overlap window within the video frames of the primary video sequence aligned with the video frames of the auxiliary video sequence.

[0085] If the first or second overlapped window flag indicates that an explicit overlapped window exists, the overlap state further indicates:

[0086] -Window horizontal position, which specifies the horizontal position of the upper left corner of the overlapping window;

[0087] - Window vertical position, which specifies the horizontal position of the upper left corner of the overlapping window;

[0088] - Window width, which specifies the horizontal size of the upper left corner of the overlapping window;

[0089] - Window height, which specifies the horizontal size of the upper left corner of the overlapping window;

[0090] - a window position horizontal offset, which specifies the horizontal offset of the position of the upper left corner of the overlapping window, for example in units of 1 / 16 of a pixel unit; and

[0091] - Window position vertical offset, which specifies the vertical offset of the position of the upper left corner of the overlapping window, for example in units of 1 / 16 of a pixel unit.

[0092] If the first and second overlapping window flags indicate that explicit overlapping windows exist, the overlapping state indicates a window horizontal position, a window vertical position, a window width, a window height, a window position horizontal offset, and a window position vertical offset of each overlapping window.

[0093] In an exemplary embodiment, when the primary video sequence and the auxiliary video sequence have different sizes or the overlapping windows have different sizes, the signaling method further includes writing the resampling data as encapsulation data, and the parsing method further includes obtaining the resampling data by parsing the encapsulation data, wherein the resampling data indicates how the application can resample the video frames of the auxiliary video sequence or the primary video sequence to the size of the video frames of the primary video sequence or the auxiliary video sequence.

[0094] In a variant, the signaling method further comprises writing the interpolation data as encapsulated data, and the parsing method further comprises obtaining the interpolation data by parsing the encapsulated data, the interpolation data indicating an interpolation method for resampling samples of the video frames of the video sequence.

[0095] In one exemplary embodiment, the interpolation data may represent coefficients of a resampling filter to be applied to samples of video frames of a video sequence.

[0096] In an exemplary embodiment, the parsing method further includes writing the calibration data as encapsulated data, and the parsing method further includes obtaining the calibration data by parsing the encapsulated data, wherein the calibration data specifies the calibration file in a format and content manner and contains a string representing the calibration file or a binary structure representing calibration data parameters.

[0097] According to a third aspect of the present application, there is provided a container formatted to include encapsulated data obtained by the method according to the first aspect of the present application.

[0098] According to a fourth aspect of the present application, an apparatus includes means for executing one of the methods according to the first and / or second aspect of the present application.

[0099] According to a fifth aspect of the present application, a computer program product includes instructions, which, when executed by one or more processors, causes the one or more processors to execute the method according to the first and / or second aspect of the present application.

[0100] According to a sixth aspect of the present application, a non-transitory storage medium (storage medium or storing medium) is provided, which carries instructions of program code for executing any method according to the first and / or second aspect of the present application.

[0101] The specific nature of at least one exemplary embodiment and other objects, advantages, features and uses of the at least one exemplary embodiment will become more apparent from the following description of the examples in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Reference will now be made by way of example to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0103] Figure 1 The geometric representation of the depth information is schematically shown;

[0104] Figure 2 Schematic diagram showing schematic color and depth sensors;

[0105] Figure 3 An example of the field of view of the color and depth sensors is shown;

[0106] Figure 4 An example of the syntax of the syntax element "generic params" is shown;

[0107] Figure 5 An example of positional offset between primary video sequence data and auxiliary video sequence data is shown;

[0108] Figure 6An example of the syntax of the syntax element "depth params" is shown;

[0109] Figure 7 A schematic block diagram showing the steps of a method 100 of signaling encapsulated data representing a primary video sequence associated with a secondary video sequence according to at least one exemplary embodiment of the present invention;

[0110] Figure 8 A schematic block diagram showing the steps of a method 200 for parsing encapsulated data representing a primary video sequence associated with a secondary video sequence according to at least one exemplary embodiment of the present invention;

[0111] Fig. 9 An example of encapsulating a single primary video sequence associated with a single auxiliary video sequence into an ISOBMFF file according to an exemplary embodiment of the present invention is shown;

[0112] Fig.10 Shows Fig. 9 An example of a variation of;

[0113] Fig.11 Shows Fig. 9 An example of a variation of ; and

[0114] Fig.12 A schematic block diagram illustrating an example of a system in which various aspects and exemplary embodiments are implemented is shown.

[0115] Similar or identical elements are denoted by the same reference numerals. DETAILED DESCRIPTION

[0116] At least one exemplary embodiment will be described more fully below with reference to the accompanying drawings, wherein an example of at least one exemplary embodiment is depicted. However, the exemplary embodiments can be embodied in many alternative forms and should not be construed as being limited to the examples described herein. Therefore, it should be understood that there is no intention to limit the exemplary embodiments to the disclosed specific forms. On the contrary, the application is intended to cover all modifications, equivalents, and substitutions that fall within the spirit and scope of the application.

[0117] At least one aspect generally relates to signaling encapsulated data representing a primary video sequence associated with an auxiliary video sequence. Another aspect generally relates to generating / writing that encapsulated data and possibly storing it in a file or transmitting it into a bitstream, and reading / accessing that encapsulated data from a file or decoding that encapsulated data from a bitstream.

[0118] In the following, exemplary embodiments of the present invention are discussed by considering the primary video sequence as a color video sequence and the associated auxiliary video sequence as a depth video sequence (because this is the usual association between the primary video sequence and the auxiliary video sequence). However, the present invention can be extended to any other type of primary video sequence and auxiliary video sequence.

[0119] Encapsulation is the process of wrapping a media representation in a container. Binary structures such as binary files (e.g., ISOBMFF files) are an example of a container. Binary files define a syntax for parsing and understanding file structures. They have a beginning and an end, and usually hold self-contained information. Binary files are usually transmitted and stored as a whole. However, for the purpose of transmission over a network, such as using HTTP-based transmission, binary files can be further divided into smaller file units. Popular standards for streaming over HTTP are Apple HTTP Live Streaming (IETF Internet Draft) and MPEG-DASH (Dynamic Adaptive Streaming over HTTP, ISO / IEC 23009-1).

[0120] When transmitting media data, real-time protocol can also be selected to meet the requirements of target applications. For example, this is the case for video conversion applications. In this case, the main video sequence bit stream and the auxiliary video sequence bit stream are usually encapsulated in a transport packet (container) rather than as a file structure. For example, a real-time transport protocol (RTP) (RFC3550) can be used. However, without limiting the scope of the invention, other transport packet encapsulations can be used. The container can be a bit stream, a network packet or a binary file.

[0121] In the following, exemplary embodiments of the present invention are described by considering encapsulating the primary video sequence and the auxiliary video sequence into a file based on an example of a binary file, which is called an ISOBMFF file as defined in the standard ISO / IEC 14496-12. However, other file examples may be used without limiting the scope of the present invention in any way.

[0122] Furthermore, the exemplary embodiment of transport packet encapsulation can also be derived from the exemplary embodiment of file encapsulation described below by considering the transport standard as an example of transport packet encapsulation instead of the described file encapsulation.

[0123] The at least one exemplary embodiment may be applied to pre-existing or future developed standards and proposed extensions.Unless otherwise indicated or technically excluded, the various aspects described in this application may be used alone or in combination.

[0124] The principle of the present invention is to signal the encapsulation data representing both the visual content and the visual content description of a main video sequence and an associated auxiliary video sequence, so that the visual content of the two video sequences can be compressed, stored and transmitted as efficiently as possible based on their different signal characteristics, while at a later stage enabling the processing of obtaining aligned visual content from the decoded video sequences based on the signaled encapsulation data.

[0125] The encapsulation data represents an alignment status indicating whether visual content of the primary video sequence is aligned with visual content of the auxiliary video sequence.

[0126] Furthermore, the encapsulation data may represent alignment reference data indicating that samples of video frames of the auxiliary video sequence have been aligned with samples of video frames of the primary video sequence or vice versa.

[0127] Furthermore, the encapsulation data may represent an overlap status indicating whether the visual content of the primary video sequence completely overlaps or partially overlaps the visual content of the auxiliary video sequence.

[0128] The overlap status may also provide information for resampling the samples of the video frames of the video sequence.

[0129] The encapsulation data may also represent resampling data indicating how the application can resample the video frames of the auxiliary video sequence or the primary video sequence to the size of the video frames of the primary video sequence or the auxiliary video sequence and which samples are to be resampled.

[0130] Furthermore, the encapsulated data may represent interpolation data indicating an interpolation method for resampling samples of video frames of the video sequence or coefficients of a resampling filter to be applied to samples of video frames of the video sequence.

[0131] It is therefore no longer necessary to store the visual content of the primary video sequence and the visual content of the auxiliary video sequences at the same spatial resolution, the same projection (realignment) and / or the same frame rate (temporal frequency), etc. This allows, thus, to select for each video sequence (e.g. depth and color video sequences) the format that yields the best performance in terms of quality vs bitrate performance ratio, while enabling realignment of the sampling of the visual content of the primary video sequence and the sampling of the visual content of the auxiliary video sequences after decoding by the application receiving / reading the encapsulated data.

[0132] Furthermore, since alignment of the visual content of the primary video sequence and the visual content of the auxiliary video sequence is no longer a prerequisite on the sending side, but can be performed by an intermediate node entity that may have a higher computing capacity, or in extreme cases by the receiving device, the sending device can provide the capture stream faster under real-time conditions (lower computing cost).

[0133] Figure 7 A schematic block diagram showing the steps of a method 100 of signaling encapsulated data representing a primary video sequence V1 associated with a secondary video sequence V2 according to at least one exemplary embodiment of the present invention is shown.

[0134] A main video sequence V1 is captured from a sensor. The sensor uses an imaging system based on a specific image projection method to generate a video sequence from the captured signals.

[0135] An auxiliary video sequence V2 is also captured from the sensor.The sensor uses an imaging system based on a specific image projection method to generate a video sequence from the captured signals.

[0136] In step 110, the visual content of the primary video sequence and the visual content of the auxiliary video sequence are written as encapsulated data.

[0137] The alignment status is written as encapsulation data in step 120. The alignment status indicates whether the visual content of the primary video sequence V1 is aligned with the visual content of the auxiliary video sequence V2.

[0138] Signaling the alignment status is useful because, depending on its value, processing may or may not be applied to the primary and auxiliary video sequences carried by the encapsulated data.

[0139] Figure 8 A schematic block diagram showing the steps of a method 200 of parsing encapsulated data representing a primary video sequence V1 associated with a secondary video sequence V2 according to at least one exemplary embodiment of the present invention is shown.

[0140] In step 210, visual contents of the primary video sequence and visual contents of the auxiliary video sequence are obtained by parsing the encapsulation data.

[0141] In step 220, the alignment status is obtained by parsing the encapsulation data. The alignment status indicates whether the visual content of the primary video sequence V1 is aligned with the visual content of the auxiliary video sequence V2.

[0142] In a variation of steps 120 or 220, the alignment status provides further details about the alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence.

[0143] In the variant, the alignment state

[0144] - equal to 0, then specifies: there is no alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence;

[0145] - equal to 1, it specifies that: the main video sequence and the auxiliary video sequence have been generated by the same image projection method, but the sampling of their video frames may not be aligned;

[0146] - is equal to 2, which specifies that there is a one-to-one alignment between the samples of the video frames of the primary video sequence and the samples of the video frames of the auxiliary video sequence; or

[0147] - equal to 3, specifies: the alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence is unspecified.

[0148] In a variant, when the alignment status is equal to 1, the method 100 further comprises a step 130 , and the method 200 further comprises a step 230 .

[0149] In step 130, the alignment reference data is written as packing data, and the alignment status is obtained by parsing the packing data in step 230. The alignment reference data indicates whether samples of the video frame of the auxiliary video sequence are aligned with samples of the video frame of the primary video sequence or vice versa.

[0150] In one variation, the alignment reference data

[0151] - equal to 0, specifies: the samples of the video frames of the auxiliary video sequence have been aligned with the samples of the video frames of the primary video sequence;

[0152] - equal to 1, specifies that: the samples of the video frames of the primary video sequence have been aligned with the samples of the video frames of the auxiliary video sequence;

[0153] - equal to 2, specifies: The reference used for alignment is unspecified.

[0154] In an exemplary embodiment, if the alignment status indicates that the primary video sequence and the auxiliary video sequence are aligned, method 100 further includes step 140 , and method 200 further includes step 240 .

[0155] In step 140, the overlap status is written as encapsulation data, and in step 240, the overlap status is obtained by parsing the encapsulation data. The overlap status indicates whether the visual content of the primary video sequence completely overlaps or partially overlaps the visual content of the auxiliary video sequence.

[0156] Signaling the overlap status is useful because, depending on its value, a specific resampling process may or may not be applied to the primary video sequence or the auxiliary video sequence carried by the encapsulated data.

[0157] In one exemplary embodiment, the overlap status indicates:

[0158] - a first overlap window flag indicating the presence of an explicit overlap window within a video frame of the auxiliary video sequence aligned with a video frame of the primary video sequence; and

[0159] - a second overlap window flag indicating the presence of an explicit overlap window within video frames of said primary video sequence aligned with video frames of said auxiliary video sequence.

[0160] If the first or second overlapped window flag indicates that an explicit overlapped window exists, the overlap state further indicates:

[0161] -Window horizontal position, which specifies the horizontal position of the upper left corner of the overlapping window;

[0162] - Window vertical position, which specifies the horizontal position of the upper left corner of the overlapping window;

[0163] - Window width, which specifies the horizontal size of the upper left corner of the overlapping window;

[0164] - Window height, which specifies the horizontal size of the upper left corner of the overlapping window;

[0165] - a window position horizontal offset, which specifies the horizontal offset of the position of the upper left corner of the overlapping window, for example in units of 1 / 16 of a pixel unit; and

[0166] - Window position vertical offset, which specifies the vertical offset of the position of the upper left corner of the overlapping window, for example in units of 1 / 16 of a pixel unit.

[0167] If the first and second overlapping window flags indicate that explicit overlapping windows exist, the overlapping state indicates a window horizontal position, a window vertical position, a window width, a window height, a window position horizontal offset, and a window position vertical offset of each overlapping window.

[0168] When the alignment status indicates that the visual content of the primary video sequence is not aligned with the visual content of the auxiliary video sequence, it is useful to signal how the receiving device (player) can transform one video sequence (e.g., the auxiliary video sequence) so that the sampling of the video frames of the video sequence is aligned with the sampling of the video frames of the other video sequence (e.g., the primary video sequence).

[0169] In an exemplary embodiment, method 100 further includes step 150 , and method 200 further includes step 250 .

[0170] When the primary video sequence and the auxiliary video sequence have different sizes or the overlapping windows have different sizes, resampling data is written as packing data in step 150, and the resampling data is obtained by parsing the packing data in step 250. The resampling data indicates how the application can resample the video frames of the auxiliary video sequence or the primary video sequence to the size of the video frames of the primary video sequence or the auxiliary video sequence and which samples are to be resampled.

[0171] This way, the behavior of different players can be consistent across vendors and increase the consistency of the content presented.

[0172] In one exemplary embodiment, the data is resampled:

[0173] - is equal to 0, then it specifies that the samples to be resampled are samples of the video frame of the auxiliary video sequence;

[0174] - is equal to 1, it specifies that the samples to be resampled are samples of the video frames of the main video sequence;

[0175] - is equal to 2, it specifies that: the sample to be resampled is the sample of the video frame of the video sequence with the smallest size between the primary video sequence and the auxiliary video sequence;

[0176] - is equal to 3, it specifies that: the samples to be resampled are samples of the video frames of the video sequence with the largest size between the primary video sequence and the auxiliary video sequence.

[0177] In one variant, step 150 includes step 151 and step 250 includes step 251 .

[0178] In step 151, the interpolation data is written as packed data, and in step 251, the interpolation data is obtained by parsing the packed data. The interpolation data indicates an interpolation method for resampling samples of a video frame of a video sequence.

[0179] In one example embodiment, the interpolated data

[0180] - equal to 0, specifies that the nearest neighbor interpolation method should be used;

[0181] - equal to 1, specifies that bilinear interpolation should be used.

[0182] - equal to 2, specifies that bicubic interpolation should be used.

[0183] - equal to 3, specifies that Lanczos interpolation should be used.

[0184] In an exemplary embodiment, the interpolated data may represent coefficients of a resampling filter to be applied to samples of video frames of a video sequence, but the goal remains the same, namely to instruct the application to resample samples of video frames of a video sequence according to a particular method.

[0185] When the alignment status indicates that the image projection methods used to capture the main video sequence V1 and the auxiliary video sequence V2 are different and / or when the viewpoints used to capture those video sequences are different, for example when the sensors are a few millimeters apart or when the optical systems are different, it may be useful to provide what are generally called intrinsic and extrinsic camera parameters. The intrinsic camera parameters describe the optical system of the camera used to generate the video sequence, while the extrinsic camera parameters describe the position and orientation of the camera in the real scene. When only two cameras are used, it is sufficient to know the position and orientation of one camera relative to the other. For both types of parameters of both cameras being known, it is mathematically possible to transform one video sequence of a scene captured by a first camera into another video sequence of the same scene, as if it had been shot by another camera.

[0186] In such a scenario, and for easier integration with available hardware, it may be desirable to write a calibration file as packaged data. The calibration file contains the intrinsic and extrinsic camera parameters of the sensor used to capture the primary and auxiliary video sequences. In an exemplary embodiment, method 100 further includes step 160, and method 200 further includes step 260.

[0187] In step 160, the calibration data is written as packaged data, and in step 260, the calibration data is obtained by parsing the packaged data. The calibration data specifies the calibration file in terms of format (e.g., JSON, XML, etc.) and content mode (i.e., which schema and associated specification it conforms to), and contains a string representing the calibration file or a binary structure representing calibration data parameters.

[0188] In an exemplary embodiment, the calibration data contains binary data representing the calibration file and parameters specifying the length of the file containing the encapsulated data (eg, the length of an ISOBMFF file) before the location of the calibration file itself in the box syntax.

[0189] When both primary and auxiliary video sequences are encapsulated, it is desirable for distribution and applications to be able to quickly identify the nature of the encapsulated data without having to parse large amounts of data. Among other uses, this can allow content selection by a streaming client when parsing a manifest file, such as a Media Presentation Document (MPD) defined by the MPEG-DASH standard, in which high-level information about the nature of the media data is signaled.

[0190] In an exemplary embodiment, method 100 further includes step 170 , and method 200 further includes step 270 .

[0191] In step 170, the code is written as encapsulation data and in step 270, the code is obtained by parsing the encapsulation data. The code defines a brand for identifying the file as containing media content and descriptive metadata about the primary video sequence associated with the secondary video sequence.

[0192] For example, a four-character code such as 'rgbd' is written as the package data.

[0193] In an exemplary embodiment of method 100 , the encapsulated data is written into a binary file, and the binary file is parsed in method 200 .

[0194] For example, the binary file is an ISOBFMFF file conforming to the standards ISO / IEC 14496-12 in combination with ISO / IEC 23002-3, such as Fig. 9 and 11 shown.

[0195] Fig. 9 An example of encapsulating a single primary video sequence associated with a single auxiliary video sequence into an ISOBMFF file according to an exemplary embodiment of the present invention is shown.

[0196] ISOBMFF files contain boxes as defined in standard ISO / IEC 14496-12 discussed above.

[0197] In steps 110 and 210, the visual content of the main video sequence V1 is encapsulated as the main video track T1 of the ISOBMFF file, and the auxiliary video sequence V2 is encapsulated as the auxiliary video track T2.

[0198] Note that for simplicity, Fig. 9 Only relevant boxes are represented in . The Media Data box stores data representing the visual content of the primary video track T1 corresponding to data representing the visual content of the video sequence V1 (primary video), and the Media Data box stores data representing the visual content of the secondary video track T1 corresponding to data representing the visual content of the video sequence V2 (secondary video). Boxes M1 and M2 represent the visual content description of the video sequences V1 and V2, i.e. metadata describing the properties of the video sequences V1 and V2.

[0199] More specifically, in Fig. 9In the ISOBMFF file of , the primary video track T1 is identified by the identifier track_ID = 1 and the auxiliary video track T2 is identified by the identifier track_ID = 2. These identifiers are signaled in the track header box 'tkhd'. The track reference mechanism defined in the standard is used to represent the association between the primary video sequence V1 and the auxiliary video sequence V2, i.e. the association between the primary video track T1 and the auxiliary video track T2, in the ISOBMFF file.

[0200] For example, when the auxiliary video sequence V2 is a depth video sequence, a 'vdep' box contained in a box called TrackReferenceBox "tref" as defined in clause 8.3.3.2 of standard ISO / IEC 14496-12 is used.

[0201] The 'vdep' box contains the identifier of the reference track (here track_ID=1) that links the depth video track T2 with the main video track T1.

[0202] In one variant, a single primary video sequence may be associated with multiple auxiliary video sequences in an ISOBMFF file.

[0203] The visual content of the primary video sequence is then encapsulated as a primary video track of the ISOBMFF file, and each auxiliary video sequence is encapsulated as an auxiliary video track.

[0204] This may be the case, for example, when the color video sequence is a large field of view and the depth video sequence has a smaller field of view, which effectively means that the captured portion of the scene of each depth video sequence partially overlaps the entire content captured in the color video sequence. A specific use case for this scenario is a panoramic camera capturing a scene and various depth cameras capturing each part of the scene. Another scenario could be a 360° omnidirectional camera capturing a scene and several 2D depth cameras capturing the scene in some directions.

[0205] When the auxiliary video sequence is a depth video sequence, it can be used in combination with Fig. 9 A similar track reference mechanism as described above, except that multiple deep video tracks point to the same main video track.

[0206] In one exemplary embodiment of step 120 of method 100 and step 220 of method 200 , the alignment status is encapsulated in a box RGBDAlignmentInformationBox as specified in Appendix 1 .

[0207] In one exemplary embodiment of step 130 of method 100 and step 230 of method 200 , the alignment reference data is encapsulated in a box RGBDAlignmentInformationBox as specified in Appendix 2.

[0208] In one exemplary embodiment of step 140 of method 100 and step 240 of method 200 , the overlap status is encapsulated in a box DepthAlignmentInformationBox as specified in Appendix 3.

[0209] In one exemplary embodiment of steps 150 and 151 of method 100 and steps 250 and 251 of method 200 , the resampled and interpolated data are enclosed in a box DepthAlignmentInformationBox as specified in Appendix 4.

[0210] In one exemplary embodiment of step 160 of method 100 and step 260 of method 200 , the overlap status is encapsulated in a box DepthAlignmentInformationBox as specified in Appendix 5.

[0211] In an exemplary embodiment of step 170 of method 100 and step 270 of method 200, as Fig.11 As shown, the four-character code 'rgbd' signals the brand as part of the "compatible_brands parameter" of the box FileTypeBox 'ftyp' defined in standard ISO / IEC 14496-12.

[0212] Fig.12 A schematic block diagram illustrating an example of a system 300 in which various aspects and exemplary embodiments are implemented is shown.

[0213] System 300 may be embodied as one or more devices including various components described below. In various exemplary embodiments, system 300 may be configured to implement one or more aspects described herein.

[0214] Examples of devices that can form all or part of system 300 include personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, connected vehicles and their associated processing systems, head-mounted display devices (HMD, transparent glasses), projectors (beamers), "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing outputs from video decoders, pre-processors for providing inputs to video encoders, web servers, video servers (e.g., broadcast servers, video-on-demand servers or web servers), static or video cameras, encoding or decoding chips or any other communication devices. The elements of system 300 can be included in a single integrated circuit (IC), multiple ICs and / or discrete components individually or in combination. For example, in at least one exemplary embodiment, the processing and encoder / decoder elements of system 300 can be distributed on multiple ICs and / or discrete components. In various exemplary embodiments, system 300 can be coupled to other similar systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports.

[0215] The system 300 may include at least one processor 310, which is configured to execute instructions loaded therein, for implementing various aspects described in the present application, for example. The processor 310 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 300 may include at least one memory 320 (e.g., a volatile memory device and / or a non-volatile memory device). The system 300 may include a storage device 340, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a disk drive, and / or an optical drive. As a non-limiting example, the storage device 340 may include an internal storage device, an additional storage device, and / or a network accessible storage device.

[0216] The system 300 may include an encoder / decoder module 330, which is configured to, for example, process data to provide encoded / decoded video picture data, and the encoder / decoder module 330 may include its own processor and memory. The encoder / decoder module 330 may represent a module that may be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or two encoding and decoding modules. In addition, the encoder / decoder module 330 may be implemented as a separate element of the system 300 or may be incorporated into the processor 310 as a combination of hardware and software known to those skilled in the art.

[0217] Program code to be loaded onto the processor 310 or the encoder / decoder 330 to perform various aspects described in this application may be stored in the storage device 340 and subsequently loaded onto the memory 320 for execution by the processor 310. According to various exemplary embodiments, one or more of the processor 310, the memory 320, the storage device 340, and the encoder / decoder module 330 may store one or more of various items during the execution of the processes described in this application. Such stored items may include, but are not limited to, video picture data, information data for encoding / decoding video picture data, bit streams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0218] In several exemplary embodiments, memory within processor 310 and / or encoder / decoder module 330 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0219] However, in other exemplary embodiments, memory external to the processing device (e.g., the processing device may be the processor 310 or the encoder / decoder module 330) may be used for one or more of these functions. The external memory may be a memory 320 and / or a storage device 340, such as a dynamic volatile memory and / or a non-volatile flash memory. In several exemplary embodiments, an external non-volatile flash memory may be used to store an operating system for the television. In at least one exemplary embodiment, a fast external dynamic volatile memory such as RAM may be used as working memory for video encoding and decoding operations, such as for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), AVC, HEVC, EVC, VVC, AV1, etc.

[0220] Inputs to the elements of system 300 may be provided by various input devices, as shown in block 390. When the present invention is implemented in the automotive field, such input devices include, but are not limited to: (i) an RF section that can receive RF signals transmitted over the air, for example, by a broadcaster; (ii) a composite input terminal; (iii) a USB input terminal; (iv) an HDMI input terminal; (v) a bus, such as a CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), FlexRay (ISO17458) or Ethernet (ISO / IEC 802-3) bus.

[0221] In various exemplary embodiments, the input device of block 390 may have associated corresponding input processing elements known in the art. For example, the RF portion may be associated with elements required to (i) select a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-convert the selected signal, (iii) again band-limit to a narrower frequency band to select a signal frequency band that may be referred to as a channel in certain exemplary embodiments, (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF portion of various exemplary embodiments may include one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near baseband frequency) or baseband.

[0222] In a set-top box implementation, the RF section and its associated input processing elements can receive RF signals sent through a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band.

[0223] Various exemplary embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0224] Adding components may include inserting components between existing components, such as inserting an amplifier and an analog-to-digital converter.In various exemplary embodiments, the RF portion may include an antenna.

[0225] In addition, the USB and / or HDMI terminals may include respective interface processors for connecting the system 300 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or in the processor 310 as desired. Similarly, various aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or in the processor 310 as desired. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, the processor 310 and an encoder / decoder 330, which operates in conjunction with memory and storage elements to process the data streams as desired for presentation on an output device.

[0226] The various elements of the system 300 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transferred between them using suitable connection devices 390 (eg, internal buses known in the art, including I2C buses, wiring, and printed circuit boards).

[0227] System 300 may include a communication interface 350 that enables communication with other devices via a communication channel 351. Communication interface 350 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 351. Communication interface 350 may include, but is not limited to, a modem or a network card, and communication channel 351 may be implemented, for example, in a wired and / or wireless medium.

[0228] In various exemplary embodiments, a Wi-Fi network such as IEEE 802.11 may be used to stream data to the system 300. The Wi-Fi signals of these exemplary embodiments may be received via a communication channel 351 suitable for Wi-Fi communications and a communication interface 350. The communication channel 351 of these exemplary embodiments may typically connect to an access point or router that provides access to external networks including the Internet to allow streaming applications and other top-level communications.

[0229] Other exemplary embodiments may provide streaming data to system 300 using a set-top box that passes data through the HDMI connection of input block 390 .

[0230] Other exemplary embodiments may use the RF connection of input block 390 to provide streaming data to system 300 .

[0231] Streaming data may be used as a means of signaling information used by the system 300. The signaled information may include a bitstream and / or information such as the number of pixels in a video frame, any encoding / decoding setting parameters, alignment status, alignment reference data, overlap status, resampling data, interpolation data, and / or calibration data.

[0232] It should be understood that signaling can be implemented in a variety of ways. For example, in various exemplary embodiments, one or more syntax elements, flags, etc. can be used to signal information to a corresponding decoder.

[0233] The system 300 can provide output signals to various output devices, including a display 361, speakers 371, and other peripherals 381. In various examples of exemplary embodiments, the other peripherals 381 can include one or more of a stand-alone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 300.

[0234] In various exemplary embodiments, control signals may be transmitted between the system 300 and the display 361, speakers 371, or other peripheral devices 381 using signaling transmissions such as AV Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocols that allow device-to-device control with or without user intervention.

[0235] Output devices may be communicatively coupled to system 300 through respective interfaces 360 , 370 , and 380 via dedicated connections.

[0236] Alternatively, output devices may be connected to system 300 via communication interface 350 using communication channel 351. Display 361 and speaker 371 may be integrated into a single unit with other components of system 300 in an electronic device such as, for example, a television.

[0237] In various exemplary embodiments, the display interface 360 ​​may include a display driver, such as, for example, a timing controller (TCon) chip.

[0238] Alternatively, the display 361 and the speaker 371 may be separate from one or more other components, for example, if the RF portion of the input 390 is part of a separate set-top box. In various exemplary embodiments where the display 361 and the speaker 371 may be external components, the output signals may be provided via dedicated output connections, including, for example, an HDMI port, a USB port, or a COMP output.

[0239] exist Figure 1-12In the present invention, various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0240] Some examples are described with respect to block diagrams and / or operational flow charts. Each box represents a circuit element, module or code portion including one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions indicated in the box may not occur in the order indicated. For example, depending on the functions involved, two boxes shown in succession may actually be executed substantially simultaneously, or the boxes may sometimes be executed in reverse order.

[0241] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if only discussed in the context of a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a computer program).

[0242] The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device.

[0243] Additionally, the methods may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by the implementation) may be stored on a computer-readable storage medium, such as, for example, storage device 340 ( Fig.12 ). The computer-readable storage medium may take the form of a computer-readable program product included in one or more computer-readable media and including thereon a computer-readable program code executable by a computer. The computer-readable storage medium used herein may be considered to be a non-transient storage medium having an inherent ability to store information therein and to provide an inherent ability to retrieve information therefrom. The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. It should be understood that while more specific examples of computer-readable storage media to which the exemplary embodiment may be applied are provided, the following is merely illustrative and not an exhaustive list that is readily understood by a person of ordinary skill in the art: a portable computer floppy disk; a hard disk; a read-only memory (ROM); an erasable programmable read-only memory (EPROM or flash memory); a portable compact disk read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination of the foregoing.

[0244] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0245] Instructions may be, for example, hardware, firmware, software, or a combination thereof. Instructions may be found, for example, in an operating system, a separate application, or a combination of both. Thus, a processor may be characterized as both a device configured to perform a process and a device including a processor-readable medium (e.g., a storage device) having instructions for performing the process. Furthermore, in addition to or in lieu of instructions, a processor-readable medium may store data values ​​generated by an implementation.

[0246] Device can be realized in suitable hardware, software and firmware for example.The example of such device comprises personal computer, laptop computer, smart phone, tablet computer, digital multimedia set-top box, digital television receiver, personal video recording system, connected household appliances, head-mounted display device (HMD, transparent glasses), projector (beamer), "caves" (system including multiple displays), server, video encoder, video decoder, post-processor for processing the output from video decoder, pre-processor for providing input to video encoder, web server, set-top box and any other equipment or other communication equipment for processing video picture.Should be clear that the device can be mobile, even installed in a mobile vehicle.

[0247] Computer software may be implemented by processor 310 or hardware, or a combination of hardware and software. As a non-limiting example, the exemplary embodiment may also be implemented by one or more integrated circuits. Memory 320 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory as non-limiting examples. As a non-limiting example, processor 310 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0248] Based on the present application, it is obvious to those skilled in the art that the implementation can generate various signals formatted to carry information that can be stored or sent, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the exemplary embodiment. Such a signal can be formatted as, for example, an electromagnetic wave (for example, using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted through various different wired or wireless links. The signal can be stored on a processor readable medium.

[0249] The terms used herein are only used to describe the purpose of specific exemplary embodiments, and are not intended to be limiting. As used herein, the singular forms "a, an" and "the" may be intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that when used in this specification, the term "includes / comprises and / or including / comprising" can specify the presence of the features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts and / or their combinations. In addition, when an element is referred to as "responding" or "connected" to or "associated with" another element, it can directly respond or be connected to or associated with another element, or there may be an intermediate element. On the contrary, when an element is referred to as "directly responding" or "directly connected" to or "directly associated with" other elements, there is no intermediate element.

[0250] It should be understood that the use of any symbol / term " / ", "and / or", and "at least one of...", for example, in the case of "A / B", "A and / or B", and "at least one of A and B", can be intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As another example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). As will be apparent to one of ordinary skill in this and related arts, this can be extended to as many items as are listed.

[0251] Various numerical values ​​may be used in the present application. Specific values ​​may be used for example purposes, and the described aspects are not limited to these specific values.

[0252] It should be understood that although the terms first, second, etc. can be used here to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the teachings of this application, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. There is no suggestion of ordering between the first element and the second element.

[0253] References to "an exemplary embodiment" or "exemplary embodiment" or "an implementation" or "implementations" and other variations thereof are often used to convey that a particular feature, structure, characteristic, etc., described in conjunction with that embodiment / implementation, is included in at least one embodiment / implementation. Thus, appearances of the phrase "in an exemplary embodiment" or "in an exemplary embodiment" or "in an implementation" or "in implementations," and any other appearances throughout this application, are not necessarily all referring to the same exemplary embodiment.

[0254] Similarly, references herein to "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" and other variations thereof are often used to convey that a particular feature, structure, or characteristic (described in conjunction with an exemplary embodiment / example / implementation) may be included in at least one exemplary embodiment / example / implementation. Therefore, the phrases "according to an exemplary embodiment / example / implementation" or "in an exemplary embodiment / example / implementation" appearing in different places in this application do not necessarily all refer to the same exemplary embodiment / example / implementation, nor are they necessarily separate or alternative exemplary embodiments / examples / implementations that are mutually exclusive of other exemplary embodiments / examples / implementations.

[0255] Reference numerals appearing in the claims are for illustration purposes only and have no limiting effect on the scope of the claims.The exemplary embodiments / examples and variants may be used in any combination or sub-combination even if not explicitly described.

[0256] When the accompanying drawings are presented as flow charts, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when the drawings are presented as block diagrams, it should be understood that it also provides a flow chart of the corresponding method / process.

[0257] Although some diagrams include arrows on communication paths to illustrate a primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0258] Various implementations involve decoding. "Decoding" as used in the present application may encompass, for example, all or part of the processes performed on a received video sequence (which may include a received bitstream encoding one or more video sequences) to produce a final output suitable for display or for further processing in a reconstructed video domain. In various exemplary embodiments, such a process includes one or more processes typically performed by a decoder. In various exemplary embodiments, such a process also includes or alternatively includes a process performed by a decoder of the various implementations described in the present application.

[0259] As a further example, in one exemplary embodiment, "decoding" may refer only to dequantization, in one exemplary embodiment, "decoding" may refer to entropy decoding, in another exemplary embodiment, "decoding" may refer only to differential decoding, and in another exemplary embodiment, "decoding" may refer to a combination of dequantization, entropy decoding, and differential decoding. Whether the phrase "decoding process" may be intended to refer specifically to a subset of operations or generally to a broader decoding process will become clear based on the context of the specific description and is deemed to be well understood by those skilled in the art.

[0260] Various implementations involve encoding. In a manner similar to the above discussion about "decoding", "encoding" used in the present application may encompass, for example, all or part of the processing performed on an input video sequence to produce an output bitstream. In various exemplary embodiments, such a process includes one or more processes typically performed by an encoder. In various exemplary embodiments, such a process also includes or alternatively includes a process performed by an encoder of the various implementations described in the present application.

[0261] As a further example, in one exemplary embodiment, "encoding" may refer only to quantization, in one exemplary embodiment, "encoding" may refer only to entropy coding, in another exemplary embodiment, "encoding" may refer only to differential coding, and in another exemplary embodiment, "encoding" may refer to a combination of quantization, differential coding, and entropy coding. Whether the phrase "encoding process" may be intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is deemed to be well understood by those skilled in the art.

[0262] Additionally, the present application may refer to "obtaining" various information. Obtaining information may include, for example, estimating information, calculating information, predicting information, or retrieving information from a memory, processing information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0263] Additionally, the present application may refer to "receiving" various information. Receiving information may include, for example, one or more of accessing information or receiving information from a communication network.

[0264] In addition, as used herein, the word "signaling" refers in particular to indicating something to a corresponding decoder. For example, in some exemplary embodiments, the encoder signals specific information such as coding parameters or encoded video picture data. In this way, in exemplary embodiments, the same parameters can be used on the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be made without sending (implicit signaling) to simply allow the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various exemplary embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various exemplary embodiments, one or more grammatical elements, signs, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signaling" is related to the foregoing, the word "signaling" can also be used as a noun in this article.

[0265] Many implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or deleted to produce other implementations. In addition, it will be understood by those of ordinary skill in the art that other structures and processes may replace those disclosed, and that the resulting implementations will perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results as the disclosed implementations. Therefore, the present application contemplates these and other implementations.

[0266] Appendix 1

[0267] Appendix 1 provides an example of the binary syntax for representing the box RGBDAlignmentInformationBox.

[0268] The container RGBDAlignmentInformationBox is an instance of a class that inherits from the general class FullBox defined in clause 4.2.2 Object definition of standard ISO / IEC 14496-12. Thus, the container C1 is a box defined in standard ISO / IEC 14496-12.

[0269] In this example, the alignment status called "image_alignment_status_idc" is an unsigned integer, and "reserved" refers to a reserved future value.

[0270] The binary syntax is as follows:

[0271]

[0272] Appendix 2

[0273] Appendix 2 provides another example of a binary syntax for representing a variant of the box RGBDAlignmentInformationBox of Appendix 1.

[0274] In this example, the alignment reference data called "image_alignment_reference_idc" is an unsigned integer, and "reserved" refers to a reserved future value.

[0275] The binary syntax is as follows:

[0276]

[0277] Appendix 3

[0278] Appendix 3 provides an example of the binary syntax for representing the box DepthAlignmentInformationBox. The name of the box refers to the depth video track, since a common use case is to associate the depth video track with the main video track.

[0279] The box DepthAlignmentInformationBox is an instance of a class that inherits from the generic class FullBox defined in clause 4.2.2 Object definition of standard ISO / IEC 14496-12.

[0280] In this example, the first (second) overlap window flag called "depth_overlap_window_flag" ("texture_overlap_window_flag") is an unsigned integer, the window horizontal position called "depth_window_position_h" ("texture_window_position_h") is an unsigned integer, the window vertical position called "depth_window_position_v" ("texture_window_position_v") is an unsigned integer, the window width called "depth_window_width" ("texture_window_width") is an unsigned integer, the window height called "depth_window_height" ("texture_window_height"), the window position horizontal offset called "depth_window_position_offset_h" ("texture_window_position_offset_h") is an unsigned integer, the window position vertical offset called "depth_window_position_offset_v" ("texture_window_position_offset_v") is an unsigned integer, and "reserved" refers to a reserved future value.

[0281] The binary syntax is as follows:

[0282]

[0283]

[0284] When no overlap window is signaled for the auxiliary video track or the primary video track, the alignment is expressed for the entire frame.

[0285] There are probably several places in ISOBMFF where such a box could be placed, but the ideal way to keep the technical scheme codec-agnostic and have the main video (texture video) accessible to legacy devices would be to define a constrained scheme of type 'depa' for the auxiliary video track and place the new box inside the constrained scheme video box ('resv'), like Fig.10 This way, it also makes the auxiliary video track appear "special", and legacy devices that do not recognize this restricted scheme will ignore the depth video track and only process the main video track.

[0286] Appendix 4

[0287] Appendix 4 provides an example of a binary syntax for representing a variant of the box DepthAlignmentInformationBox of Appendix 3.

[0288] In this example, the image resampling data called "image_to_be_resampled" is an unsigned integer and the interpolation data called "interpolation_method" is an unsigned integer.

[0289]

[0290] Appendix 5

[0291] Appendix 5 provides an example of a binary syntax for representing a variant of the box DepthAlignmentInformationBox of Appendix 3 or 4.

[0292] In this example, the calibration file format referred to as "calibration_file_format_uri" is a utf8string, and the string representing the calibration file is a utf8string that allows access to the calibration file.

[0293]

Claims

1. A method (100) for signaling encapsulated data representing a primary video sequence associated with an auxiliary video sequence, the primary video sequence and the auxiliary video sequence being generated by an image projection method applied to a signal captured by a sensor, the method comprising: - writing (110) the visual content of the primary video sequence and the visual content of the auxiliary video sequence as encapsulated data; and - writing (120) an alignment status as encapsulation data, said alignment status indicating whether the visual content of said primary video sequence is aligned with the visual content of said auxiliary video sequence.

2. A method (200) for parsing encapsulated data representing a primary video sequence associated with an auxiliary video sequence, the primary video sequence and the auxiliary video sequence being generated by an image projection method applied to a signal captured by a sensor, the method comprising: - obtaining (210) the visual content of the primary video sequence and the visual content of the auxiliary video sequence by parsing the encapsulation data; and - obtaining (220) an alignment status by parsing the encapsulation data, the alignment status indicating whether the visual content of the primary video sequence is aligned with the visual content of the auxiliary video sequence.

3. The method of claim 1 or 2, wherein the alignment status further indicates one of the following states: - there is no alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence; - the main video sequence and the auxiliary video sequence have been generated by the same image projection method, but the sampling of their video frames may not be aligned; - there is a one-to-one alignment between samples of the video frames of the primary video sequence and samples of the video frames of the auxiliary video sequence; or - The alignment between the visual content of the primary video sequence and the visual content of the auxiliary video sequence is unspecified.

4. The method of claim 3, wherein when the alignment status indicates that the main video sequence and the auxiliary video sequence have been generated from the same image projection method but the sampling of their video frames may not be aligned, the signaling method (100) further includes writing (130) alignment reference data as encapsulation data, and the parsing method (200) further includes obtaining (230) alignment reference data by parsing the encapsulation data, the alignment reference data indicating that the sampling of the video frames of the auxiliary video sequence has been aligned on the sampling of the video frames of the main video sequence or vice versa.

5. A method as described in any one of claims 1 to 4, wherein if the alignment status indicates that the visual content of the main video sequence is aligned with the visual content of the auxiliary video sequence, the signal transmission method (100) further includes writing (140) an overlap status as encapsulation data, and the parsing method (200) further includes obtaining (240) an overlap status by parsing the encapsulation data, wherein the overlap status indicates whether the visual content of the main video sequence is completely overlapped or partially overlapped with the visual content of the auxiliary video sequence.

6. The method of claim 5, wherein the overlapping state further indicates: - a first overlap window flag indicating the presence of an explicit overlap window within a video frame of the auxiliary video sequence aligned with a video frame of the primary video sequence; and - a second overlap window flag indicating the presence of an explicit overlap window within video frames of said primary video sequence aligned with video frames of said auxiliary video sequence.

7. The method of claim 6, wherein if the first or second overlapping window flag indicates that an explicit overlapping window exists, the overlapping state further indicates: -Window horizontal position, which specifies the horizontal position of the upper left corner of the overlapping window; -Window vertical position, which specifies the horizontal position of the upper left corner of the overlapping window; - Window width, which specifies the horizontal size of the upper left corner of the overlapping window; - Window height, which specifies the horizontal size of the upper left corner of the overlapping window; - a horizontal window position offset, which specifies the horizontal offset of the position of the upper left corner of the overlapping window, for example, in units of 1 / 16 of a pixel unit; and - A vertical window position offset, which specifies the vertical offset of the upper left corner of the overlapping window, for example, in units of 1 / 16 of a pixel unit.

8. A method as claimed in claim 6 or 7, wherein if the first and second overlapping window flags indicate the presence of an explicit overlapping window, the overlapping status indication includes a window horizontal position, a window vertical position, a window width, a window height, a window position horizontal offset and a window position vertical offset for each overlapping window.

9. The method according to any one of claims 1 to 8, wherein when the main video sequence and the auxiliary video sequence have different sizes or the overlapping windows have different sizes, the signaling method (100) further comprises writing (150) resampling data as encapsulation data, and the parsing method (200) further comprises obtaining (250) resampling data by parsing the encapsulation data, the resampling data indicating how an application can resample the video frames of the auxiliary video sequence or the video frames of the main video sequence to the size of the video frames of the main video sequence or the size of the video frames of the auxiliary video sequence.

10. The method of claim 9, wherein the method (100) further comprises writing (151) the interpolation data as encapsulation data, and the method (200) further comprises obtaining the interpolation data by parsing the encapsulation data, the interpolation data indicating an interpolation method for resampling samples of a video frame of a video sequence.

11. A method as claimed in claim 10, wherein the interpolated data can represent coefficients of a resampling filter to be applied to samples of a video frame of a video sequence.

12. A method as claimed in any one of claims 1 to 11, wherein the signal transmission method (100) further includes writing (160) calibration data as encapsulated data, and the parsing method (200) further includes obtaining (260) calibration data by parsing the encapsulated data, wherein the calibration data specifies a calibration file in terms of format and content and contains a string representing the calibration file or a binary structure representing parameters of the calibration data.

13. A container formatted to include encapsulated data obtained by the method of any one of claims 1, 3 to 12.

14. An apparatus comprising means for carrying out one of the methods as claimed in any one of claims 1 to 12.

15. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 12.

16. A non-transitory storage medium carrying instructions of a program code for executing the method of any one of claims 1 to 12.

Images