Processing volume data

Abstract

At least one embodiment relates to a method and apparatus for encoding a volumetric video representing a scene, the encoding being based on patches of color and depth representing 2D projections of sub-portions of the scene, wherein a first patch is packed into a second patch within a given time interval less than or equal to a time period along which the second patch is defined, when the first patch can be packed into the second patch within the time interval. A decoding method and apparatus are also provided.

Description

Technical Field

[0001] At least one embodiment of the present invention generally relates to encoding / decoding volumetric video. Background Art

[0002] This section is intended to introduce the reader to various aspects of technology that may be related to at least one of the embodiments of the present invention described and / or claimed below. It is believed that this discussion will help provide the reader with background information to facilitate a better understanding of the various aspects of at least one embodiment.

[0003] With the recent emergence of consumer head-mounted displays (Oculus, HTC Vive, etc.) associated with improved screen resolution, field of view, and frame rate, virtual reality is indeed developing rapidly. This phenomenon has opened the door to new applications in many fields, including gaming, education / training, and multimedia. In the latter field, new types of video content have emerged, including so-called 360° videos. This video allows the user to view his surroundings by purely rotating around a fixed viewpoint.

[0004] Even if pure rotation is sufficient for a first omnidirectional video experience, it may quickly become frustrating for viewers who expect more freedom. More importantly, it may also cause vertigo, because head rotation includes small translations of the head that cannot be reproduced by this experience.

[0005] An alternative to these 360° contents is called volumetric video or 6 degrees of freedom (6DoF) video. When watching such a video, in addition to rotating, the user can also translate his head within the content being viewed and experience parallax. This type of video significantly increases the sense of immersion and the perception of scene depth, and prevents vertigo by providing consistent visual feedback during head translation. The relevant content is essentially created by means of dedicated sensors that allow the color and depth of the scene of interest to be recorded simultaneously. Using a color camera rig combined with photogrammetry is a common way to perform such recordings. While 360° video simply boils down to a temporal succession of specific images (e.g., width-length / equirectangular images) generated by demapping of a spherical texture, 6DoF video "frames" are more complex because they should embed information from several viewpoints. They can be viewed as a time series of colored point clouds generated by the above-mentioned 3D capture.

[0006] Two different types of volumetric video can be considered depending on the viewing conditions. The more permissive one allows completely free navigation within the video content, while the second (which we consider here) restricts the user viewing space to a limited volume. The latter case is a natural compromise between free navigation and passive viewing conditions for audience members sitting in an armchair. Furthermore, it is conceivable that in conditions shorter than the free viewpoint case, less effort is required to acquire the content (due to the limitation of the visualization box) and is currently being considered in the standards bodies under the name 3DoF+ in the standard MPEG-I Part 7 / Metadata for Immersive Video (MIV) (ISO / IEC JTC 1 / SC29 / WG11 Working Draft 1 for Metadata N18464 and W18470 for Immersive Media (Video)).

[0007] 3DoF+ content (volumetric video) is typically provided as a collection of multi-view+depth (MVD) frames over a period of time, i.e., as a collection of MVD frames. At each moment in the period, the MVD frame represents the color and depth of the volumetric video as multiple view pairs. Each pair of views includes one view for color and one view for depth. The volumetric video can be from real-world capture or can be generated from existing computer graphics content through specialized (potentially photorealistic) rendering.

[0008] Recently, video-based frameworks have been proposed to encode / decode volumetric videos.

[0009] Roughly speaking, volumetric video is transmitted as pairs of color and depth patches encapsulated in color and depth atlases (altas), which are then encoded using a conventional video codec (usually HEVC). Each pair of color and depth patches represents a 2D (typically spherical) projection of a sub-portion of the scene over that time period, i.e., it captures the color and depth of a sub-portion of a pair of views over that time period, and the set of all color and depth patch pairs is designed during the encoding phase to represent the 2D projection of the entire scene while being as non-redundant as possible. In other words, volumetric video is defined as a set of views (color + depth) captured and represented by a set of cameras (real cameras for natural content or virtual cameras for computer-generated content). These views are typically redundant. A patch is a sub-portion of a view that is sufficient to fully represent the scene, with no or almost no redundancy.

[0010] Using a video-based decoder to encode color and depth atlases takes advantage of using statically located color and depth patches within a frame to encode at each time segment. This makes temporal prediction for video encoding much easier and more efficient.

[0011] Therefore, each pair of color and depth region patches should have static properties over a given time period (usually a multiple of the MVD frame duration). The patches are said to be defined (valid) over that time period. As for the content of the color or depth patch in a pair of color and depth patches, it can change over time within that time period.

[0012] During the decoding phase, the color and depth atlases are first decoded using a video-based decoder (typically HEVC), and the color and depth patches are not projected during the view synthesis process to restore the viewport associated with the desired viewing position. In addition to the color and depth atlases, specific metadata describing the patch arrangement is signaled along with the color and depth atlases for proper decoding. In particular, the metadata may list the 2D / spherical projection (also known as the camera) used for color and depth atlas creation, as well as patch properties (position, size, ...). Summary of the Invention

[0013] A simplified summary of at least one of the embodiments of the present invention is presented below to provide a basic understanding of some aspects of the present invention. This summary is not an extensive overview of the embodiments. It is not intended to identify key or important elements of the embodiments. The following summary merely presents some aspects of at least one of the embodiments of the present invention in a simplified form as a prelude to the more detailed description provided elsewhere in this document.

[0014] According to a general aspect of at least one embodiment, a method of encoding a volumetric video representing a scene is provided, the encoding being based on patches of color and depth representing 2D projections of sub-portions of the scene, wherein the first patches are packed into the second patches within a given time interval, the given time interval being less than or equal to a time period defining the second patch when the first patch can be packed into the second patch within the time interval.

[0015] In one embodiment, the method further comprises sending information defining the time interval.

[0016] According to another general aspect of at least one embodiment, a method of decoding a volumetric video representing a scene is provided, the decoding being based on patches representing color and depth of 2D projections of sub-portions of the scene, wherein, when the first patch is packed into a second patch within a given time interval that is less than or equal to a time period along which the second patch is defined, the method comprises unpacking the first patch from the second patch over the time interval.

[0017] In one embodiment, upon receiving information defining the time interval, the first patch is packed into the second patch within the time interval.

[0018] According to a general aspect of at least one embodiment, there is provided an apparatus comprising means for encoding a volumetric video representing a scene, the encoding being based on patches of color and depth representing 2D projections of sub-portions of the scene, wherein the first patches are packed into the second patches within a given time interval, the given time interval being less than or equal to a time period defining the second patch when the first patch can be packed into the second patch within the time interval.

[0019] In one embodiment, the apparatus further comprises means for sending information defining the time interval.

[0020] According to a general aspect of at least one embodiment, there is provided an apparatus comprising means for decoding a volumetric video representing a scene, the decoding being based on patches representing color and depth of 2D projections of sub-portions of the scene, wherein, when the first patch is packed into a second patch within a given time interval that is less than or equal to a time period along which the second patch is defined, the method comprises unpacking the first patch from the second patch within the time interval.

[0021] In one embodiment, upon receiving information defining the time interval, the first patch is packed into the second patch within the time interval.

[0022] According to a general aspect of at least one embodiment, there is provided a bitstream comprising a volumetric video representing a scene, the volumetric video being encoded using an encoding of patches based on color and depth representing 2D projections of sub-portions of the scene, wherein the bitstream further comprises information defining a time interval indicating at which a first patch is to be packed into a second patch.

[0023] In one embodiment, the signaled information indicates first and second indices of an ordered list of patches.

[0024] In one embodiment, the time period is divided into a plurality of time intervals, and the information defining the time intervals is a bit indicating one of the time intervals.

[0025] One or more of at least one embodiment also provides a computer program product and a non-transitory computer-readable medium.

[0026] The specific nature of at least one embodiment of the present invention and other objects, advantages, features and uses of at least one embodiment of the present invention will become apparent from the following example description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings, examples of several embodiments are shown. The drawings show:

[0028] - Figure 1 A schematic block diagram showing an example of atlas-based volumetric video encoding / decoding according to the prior art;

[0029] - Figure 2 An example of a patch according to the prior art is shown;

[0030] - Figure 3 A flow chart of a method for encoding volumetric video according to at least one present embodiment is shown;

[0031] - Figure 3a A flow chart of a method for decoding volumetric video according to at least one present embodiment is shown;

[0032] - Figure 4-4d Shown Figure 3 Examples of methods;

[0033] - Figure 5 shows examples of syntax elements according to at least one embodiment; and

[0034] - Figure 6 A schematic block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown. DETAILED DESCRIPTION

[0035] At least one embodiment of the present invention is described more fully below with reference to the accompanying drawings, which illustrate examples of at least one embodiment of the present invention. However, the embodiments may be embodied in many alternative forms and should not be construed as limited to the examples set forth herein. Therefore, it should be understood that the embodiments are not intended to be limited to the specific forms disclosed. On the contrary, this disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present application.

[0036] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0037] Similar or identical elements in the figures are denoted by the same reference numerals.

[0038] Some figures show syntax tables that are widely used in standard specifications to define the structure of bitstreams that conform to those standard specifications. In these syntax tables, the term '...' indicates unchanged parts of the syntax with respect to the original definition given in V-PCC and is removed in the figures for ease of reading. Bold items in the figures indicate that the value of the item is obtained by parsing the bitstream. The right column of the syntax table indicates the number of bits used to encode the data of the syntax element. For example, u(4) indicates that 4 bits are used to encode the data, u(8) indicates 8 bits, and ae(v) indicates a syntax element for context-adaptive arithmetic entropy decoding.

[0039] The aspects described and contemplated below can be implemented in many different forms. Figure 1-6 Some embodiments are provided below, but other embodiments are contemplated and are not intended to be construed as limiting the scope of this disclosure. Figure 1-6 The discussion does not limit the breadth of implementation.

[0040] At least one aspect generally relates to volumetric video encoding and decoding, and at least one other aspect generally relates to transmitting the generated or encoded bitstream.

[0041] Furthermore, aspects of the present invention are not limited to MPEG standards such as MPEG-1 Part 7, and are applicable, for example, to other standards and recommendations (whether pre-existing or developed in the future), as well as extensions to any such standards and recommendations. Unless otherwise indicated or technically excluded, the aspects described in this application may be used alone or in combination.

[0042] The basis for atlas-based volumetric video encoding / decoding is naturally a way to create the color and depth patches of the color and depth atlases in such a way that they are substantially non-redundant and complementary.

[0043] Figure 1 A schematic block diagram showing an example of atlas-based volumetric video encoding / decoding according to the prior art (MIV) is shown.

[0044] Essentially, an atlas-based volumetric video encoder includes a view optimizer, an atlas builder, a video encoder, and a metadata composer. Similarly, an atlas-based volumetric video decoder includes a video decoder, a metadata parser, an atlas patch occupancy map generator, and a renderer.

[0045] The view optimizer splits the view pairs associated with MVD frames within a given time period T into so-called "base" view pairs and "additional" view pairs.

[0046] Possibly, a synthetic view is created as a view of a "base" or "additional" view pair.The synthetic view is not in the initial input set of views to be coded.

[0047] The "base" view pairs are intended to be transmitted in their entirety in so-called "large" patches (one pair of color and depth patches per "base" view pair), whereas the "additional" view pairs are intended to be "pruned", i.e., only some sub-portion of the color and depth of these "additional" view pairs may be encoded and conveyed in the associated color and depth patch pairs, as described below.

[0048] The atlas constructor builds an atlas of color and depth patches representing volumetric video.

[0049] First, for each MVD frame, the associated "additional" view pairs are iteratively pruned one by one: every part of the scene that is already contained in the "base" view pair or in any previously pruned "additional" view pair is discarded. Then, a binary mask is built for each pruned "additional" view pair for each MDV frame of time period T.

[0050] Next, if one of the binary masks from has a valid value at a given spatial location, the binary masks associated with the pruned “additional” view pairs within time period T are merged into a unique aggregate binary mask that then contains the valid value at that spatial location (indicating that some color and depth of the “additional” view pair must be conveyed by color and depth patches).

[0051] Next, the aggregate binary mask associated with each "additional" view pair is clustered. A cluster is a rectangular region that groups connected pixels of the aggregate binary mask with valid values. Each of these clusters represents the region of an "additional" view pair that should be conveyed in a pair of color and depth pictures during time period T.

[0052] Then, for each "additional" view pair, at least one pair of color and depth patches is obtained by packing the clusters together in the color and depth atlas (large frame) using a dedicated algorithm (e.g., "MaxRect"). If necessary, some clusters can be separated and / or rotated to fit into the packing file. The empty areas of some patches can also be used to pack the clusters of another "additional" view pair (the so-called "spatial" patches in the patch) to save space. As mentioned before, the "base" view pair also produces "large" patches, which are also packed in the color and depth atlas after the same process. Note that the packing process only consists in finding patch properties (position, size, ...), not in efficiently generating color and depth atlases.

[0053] Finally, once the patch lists have been shaped, for each MVD frame over time period T, corresponding color and depth atlases are generated. More precisely, the color and depth portions of the "base" and "additional" view pairs corresponding to each pair of color and depth patches are copied (with possible rotation if necessary) at the corresponding spatial locations of the color and depth atlases. At the end of the day, a pair of color and depth atlases is produced for each MVD frame over time period T, and a unique metadata list (including "base" and "additional" view pair attributes and patch list attributes) is produced for the entire time period T.

[0054] An important problem for the atlas constructor occurs when significant motion occurs in the scene during time period T (either due to moving objects or due to camera motion). In this case, aggregation can produce very large patches caused by the "sliding" of a portion of the scene within the patch during time period T. This is particularly true in Figure 2 , where a circle moves from the bottom to the top of the patch over a time period T, the patch instances (i.e., patches) are actually individually almost empty (they contain very little material), but the sliding content causes the portion P of the aggregate binary mask to be quasi-complete, and therefore "large" clusters ("large" patches) are packed in the color and depth atlases.

[0055] Furthermore, in this case, attempting to exploit the sparsity of these patch instances, exploiting the "space" in the patch is not possible, since a given portion of a patch instance may be covered during the remainder of time period T even if it is empty at some time.

[0056] As a straightforward solution, the time period T may be adaptively shortened to reduce the effect of motion, but in that case the patch instances that are not affected by this problem will usefully be extended (since shortening the time period means adding extended intra-coded MVD frames).

[0057] therefore, Figure 1 Atlas-based volumetric video encoding / decoding requires larger atlases to pack all scene content in the presence of large motions that result in potentially undesirable high pixel rates. The pixel rate is the number of pixels to be transmitted per time unit (usually one second) of video transmission. Typically, the pixel rate is defined by W*H*FPS, where W and F are the number of rows and columns of pictures of the video to be transmitted, and FPS is the number of frames per time unit. In this paper, the pixel rate depends on the size of the color and depth atlases.

[0058] Generally speaking, in this embodiment, when the first patch can be packed into the second patch within a time interval less than or equal to the time period T, the first patch is packed into the second patch within the time interval.

[0059] The basic idea is to utilize the empty areas in the patch to reduce the pixel rate.

[0060] Figure 3 A flow chart of a method for encoding volumetric video according to at least one present embodiment is shown.

[0061] In step 310, a first patch and a second patch may be obtained from a color or depth atlas. The second patch is defined over a time period T, and the first patch may be defined over a time period less than or equal to the time period T.

[0062] Alternatively, the first patch may be a cluster that is intended to be packed in color and depth atlases.

[0063] In step 320 , it is checked whether the first patch can be packed into the second patch within a time interval less than or equal to the time period T.

[0064] In step 330, when the first patch can be packed into the second patch within the time interval, the first patch is packed into the second patch within the time interval.

[0065] Figure 4 Shown Figure 3 Consider the time period T M 1 and a second patch P within a time period T. Clearly, since the contents of all instances of patch M do not overlap with the contents of instances of patch P, patch M can be packed into patch P. Here, when patch M is packed on top of patch M, the contents of patch instances P1 and M1, and correspondingly, the contents of P2 and M2, do not overlap. In other words, the contents of all instances M1 and M2 of the first patch M can be recorded (embedded, packed) in the empty spaces of the patch instances P1 and P2 of the second patch ( Figure 4a ).

[0066] This illustrative example can be extended to any first and second patches. Furthermore, multiple first patches can also be packed into the same second patch. Furthermore, first patches within a period of time can be packed into second patches within the same period of time (T M =T).

[0067] Figure 4b Shown Figure 3 Another non-limiting and illustrative example of the method of . Consider a first patch M in time period T and a second patch P in time period T. Clearly, patch M cannot be packed within patch P because if patch M is packed above patch P, then the content of patch instances M3 and M4 overlaps with the content of patch instances P3 and P4, and if patch M is packed below patch P, then the content of patch instances M1 and M2 overlaps with the content of patch instances P1 and P2. In this case, according to Figure 3Method, from patch M( Figure 4b ) obtain two first patches N and O( Figure 4c ). The first patch N is defined over a time interval T1 that is less than the time interval T, and the first patch O is defined over a time interval T2 that is less than the time interval T. Clearly, patch N can be packed above patch P because the contents of all instances of patch N do not overlap with the contents of instances of patch P. Here, the contents of patch instances P1 and N1, and correspondingly P2 and N2, do not overlap. Similarly, patch O can be packed below patch P because the contents of all instances of patch O do not overlap with the contents of instances of patch P. Here, the contents of patch instances P3 and O1, and correspondingly P4 and O2, do not overlap.

[0068] In one embodiment, the time intervals (T1, T2, ...) may be defined so as to minimize the number of time intervals required to pack a patch (M) in a second patch (P).

[0069] In a variant, the time interval is defined by regularly dividing the time period T. Figure 4c In the example of , the time period T is divided into two equal time intervals T1 and T2 , but any other values ​​are allowed.

[0070] Obviously, patch N( Figure 4c ) can be packed on top of patch P because the contents of patch instances N1 and P1, and correspondingly N2 and P2, do not overlap. Patch O can also be packed on the bottom of patch P because the contents of patch instances O1 and P3, and correspondingly O2 and P4, do not overlap ( Figure 4d ).

[0071] Figure 3a A flow chart of a method for decoding volumetric video according to at least one present embodiment is shown.

[0072] In step 350 , a second patch within the time period T may be obtained from a color or depth atlas.

[0073] In step 360 , it is checked whether the first patch is packed into the second patch within a time interval less than or equal to the time period T.

[0074] In step 370, when the first patch is packed into the second patch within the time interval, the first patch is unpacked from the second patch within the time interval.

[0075] exist Figure 3 and 3a In a variant of the method, it is possible to signal information INF defining the time interval from the bitstream, which information INF is transmitted by the encoder and received by the decoder.

[0076] Then, Figure 3The method comprises a step 340 of transmitting information INF and, in one embodiment of a step 360, packing the first patch into the second patch at said time interval when information defining said time interval may be received from the bitstream.

[0077] In one embodiment, said signaled information INF indicates a first and a second index of an ordered list of patches.

[0078] For example, in case of a 32-length time period, a 4-bit field may be considered to store each of these indices.

[0079] In an embodiment, the time period is divided into a plurality of time intervals, and the signaled information INF is a bit indicating one of the time intervals.

[0080] For example, the time period T may be divided into N regular time intervals (e.g., 4 time intervals of 8 frames for a time period T of length 32), and the signaled information may be an N-bit field. If the patch is "present" during the i-th time interval of the time period T, the i-th bit is set to 1.

[0081] Figure 5 An example of the syntax element "altas_params" for parameters of color and depth atlases including the syntax element "patch-lifetime" representing the signaled information INF is shown. The index "a" may refer to a patch index, and the index "i" may refer to a time interval index.

[0082] Figure 6 A schematic block diagram illustrating an example of a system in which various aspects and embodiments are implemented is shown.

[0083] System 6000 can be implemented as one or more devices including the various components described below, and is configured to perform one or more aspects described herein. Examples of devices that can form all or part of system 6000 include personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, connected vehicles and their associated processing systems, head-mounted display devices (HMD, perspective glasses), projectors (projectors), "cave" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing the output from video decoders, pre-processors for providing input to video encoders, web servers, set-top boxes, and any other equipment or other communication devices for processing point clouds, videos or images. The elements of system 6000 can be implemented individually or in combination in a single integrated circuit, multiple ICs and / or discrete components. For example, in at least one embodiment, the processing of system 6000 and encoder / decoder elements can be distributed over multiple ICs and / or discrete components. In various embodiments, the system 6000 can be communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, the system 6000 can be configured to implement one or more aspects described in this document.

[0084] The system 6000 may include at least one processor 6010 configured to execute instructions loaded therein to implement, for example, the various aspects described herein. The processor 6010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 6000 may include at least one memory 6020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 6000 may include a storage device 6040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 6040 may include an internal storage device, an attached storage device, and / or a network accessible storage device.

[0085] The system 6000 may include an encoder / decoder module 6030, which is configured to, for example, process data to provide encoded data or decoded data, and the encoder / decoder module 6030 may include its own processor and memory. The encoder / decoder module 6030 may represent a module (one or more) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 6030 may be implemented as a separate element of the system 6000, or may be incorporated into the processor 6010 as a combination of hardware and software known to those skilled in the art.

[0086] Program code to be loaded onto the processor 6010 or the encoder / decoder 6030 to perform various aspects described in this document may be stored in the storage device 6040 and subsequently loaded onto the memory 6020 for execution by the processor 6010. According to various embodiments, one or more of the processor 6010, the memory 6020, the storage device 6040, and the encoder / decoder module 6030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, point cloud frames, encoded / decoded geometry / texture video / images or portions of encoded / decoded geometry / texture video / images, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0087] In several embodiments, memory internal to the processor 6010 and / or encoder / decoder module 6030 may be used to store instructions and provide working memory for processes that may be performed during encoding or decoding.

[0088] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 6010 or the encoder / decoder module 6030) may be used for one or more of these functions. The external memory may be memory 6020 and / or storage device 6040, such as dynamic volatile memory and / or non-volatile flash memory. In several embodiments, the external non-volatile flash memory may be used to store the operating system of the television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, may be used as working memory for video encoding and decoding operations, such as for MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), HEVC (High Efficiency Video Coding), or VVC (Versatile Video Coding).

[0089] As shown in block 6130, input to the elements of system 6000 may be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section that may receive an RF signal transmitted over the air, for example, by a broadcaster, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0090] In various embodiments, the input device of block 6130 may have associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements necessary for the following processing: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band, which may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments may include one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband.

[0091] In a set-top box embodiment, the RF section and its associated input processing elements can receive RF signals sent over a wired (e.g., cable) medium. The RF section can then perform frequency selection by filtering, down-converting, and filtering again to a desired frequency band.

[0092] Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions.

[0093] Adding components may include inserting components between existing components, such as inserting an amplifier and an analog-to-digital converter.In various embodiments, the RF portion may include an antenna.

[0094] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 6000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within the processor 6010, as desired. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, within a separate interface IC or within the processor 6010, as desired. The demodulated, error-corrected, and demultiplexed streams may be provided to various processing elements, including, for example, the processor 6010 and the encoder / decoder 6030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0095] The various components of system 6000 can be arranged in an integrated housing. In the integrated housing, the various components can be interconnected using a suitable connection arrangement 6140 (e.g., an internal bus known in the art, including an I2C bus, wiring, and a printed circuit board) and transmit data therebetween.

[0096] System 6000 may include a communication interface 6050 that enables communication with other devices via a communication channel 6060. Communication interface 6050 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 6060. Communication interface 6050 may include, but is not limited to, a modem or a network card, and communication channel 6060 may be implemented, for example, within a wired and / or wireless medium.

[0097] In various embodiments, a Wi-Fi network, such as IEEE 802.11, may be used to stream data to the system 6000. The Wi-Fi signals of these embodiments may be received via a communication channel 6060 suitable for Wi-Fi communications and a communication interface 6050. The communication channel 6060 of these embodiments may typically connect to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communications.

[0098] Other embodiments may provide streaming data to the system 6000 using a set-top box that delivers data through an HDMI connection of the input box 6130 .

[0099] Other embodiments may use the RF connection of input box 6130 to provide streaming data to system 6000.

[0100] It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. can be used to signal information to a corresponding decoder.

[0101] The system 6000 can provide output signals to various output devices including a display 6100, speakers 6110, and other peripheral devices 6120. In various examples of embodiments, the other peripheral devices 6120 can include one or more of a stand-alone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 3000.

[0102] In various embodiments, control signals may be transmitted between the system 6000 and the display 6100, speakers 6110, or other peripheral devices 6120 using signaling such as AV.Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocols that enable device-to-device control with or without user intervention.

[0103] Output devices may be communicatively coupled to the system 6000 via dedicated connections through respective interfaces 6070 , 6080 , and 6090 .

[0104] Alternatively, output devices may be connected to system 6000 via communication interface 6050 using communication channel 6060. Display 6100 and speaker 6110 may be integrated with other components of system 6000 into a single unit in an electronic device such as a television.

[0105] In various embodiments, the display interface 6070 may include a display driver, such as a timing controller (T Con) chip.

[0106] For example, if the RF portion of input 6130 is part of a separate set-top box, the display 6100 and speaker 6110 may alternatively be separate from one or more of the other components. In various embodiments where the display 6100 and speaker 6110 may be external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0107] exist Figure 1-6 In the present invention, various methods are described herein, and each method includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0108] Some examples are described with respect to block diagrams and operational flow charts. Each block represents a circuit element, module, or code portion that includes one or more executable instructions for implementing a specified logical function(s). It should also be noted that in other implementations, the functions(s) marked in the blocks may not occur in the order indicated. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0109] The implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a computer program, a data stream, a bit stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., an apparatus or a computer program).

[0110] The method may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. A processor also includes a communication device.

[0111] In addition, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the embodiment) can be stored on a computer-readable storage medium. A computer-readable storage medium can take the form of a computer-readable program product that is contained in one or more computer-readable media (one or more) and has a computer-readable program code contained thereon that can be executed by a computer. As used herein, a computer-readable storage medium can be considered to be a non-transitory storage medium that has an inherent ability to store information therein and an inherent ability to retrieve information therefrom. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or apparatus, or any suitable combination of the foregoing. It should be understood that although more specific examples of computer-readable storage media to which the present embodiment can be applied are provided, the following is merely illustrative and not an exhaustive list as would be readily understood by one of ordinary skill in the art: a portable computer disk; a hard disk; a read-only memory (ROM); an erasable programmable read-only memory (EPROM or flash memory); a portable compact disk read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination of the foregoing.

[0112] The instructions may form an application program tangibly embodied on a processor-readable medium.

[0113] Instructions can be, for example, hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a separate application, or a combination of both. Thus, a processor can be characterized as, for example, a device configured to perform a process and a device that includes a processor-readable medium (such as a storage device) having instructions for performing the process. In addition, the processor-readable medium can store data values ​​generated by the implementation in addition to or instead of the instructions.

[0114] For example, the device can be implemented with appropriate hardware, software and firmware. The example of such a device includes a personal computer, a laptop computer, a smart phone, a tablet computer, a digital multimedia set-top box, a digital television receiver, a personal video recording system, a connected household appliance, a head-mounted display device (HMD, perspective glasses), a projector (projector), "caves" (a system including multiple displays), a server, a video encoder, a video decoder, a post-processor for processing the output from the video decoder, a pre-processor for providing input to the video encoder, a web server, a set-top box, and any other device or other communication device for processing point clouds, videos or images. It should be clear that the device can be mobile and can even be installed in a mobile vehicle.

[0115] The computer software may be implemented by the processor 6010 or by hardware, or by a combination of hardware and software. As a non-limiting example, embodiments may also be implemented by one or more integrated circuits. The memory 6020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 6010 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of the following: a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0116] As will be apparent to one of ordinary skill in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0117] The terms used herein are only for the purpose of describing specific embodiments, rather than being intended to limit. As used herein, the singular "one", "an" and "the" also may be intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" and / or "include / comprise" may specify the existence of, for example, features, integers, steps, operations, elements and / or components when used in this specification, but do not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or their combinations. In addition, when an element is referred to as "response" or "connected" to another element, it may directly respond to or be connected to another element, or there may be an intermediate element. On the contrary, when an element is referred to as "directly responding" or "directly connected" to other elements, there may be no intermediate element.

[0118] It should be understood that the use of any of the symbols / terms " / ," "and / or," and "at least one," such as in the case of "A / B," "A and / or B," and "at least one of A and B," may be intended to include selection of only the first listed option (A), or only the second listed option (B), or selection of both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0119] Various numerical values ​​may be used in this application. Specific values ​​may be, for example, purposes, and the described aspects are not limited to these specific values.

[0120] It should be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the teachings of this application, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. No order is implied between the first element and the second element.

[0121] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations are often used to convey that a particular feature, structure, characteristic, etc. (described in connection with the embodiment / implementation) is included in at least one embodiment / implementation. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in implementations" and any other variations in various places throughout this application are not necessarily all referring to the same embodiment.

[0122] Similarly, references herein to "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" and other variations thereof are often used to convey that a particular feature, structure, or characteristic (described in conjunction with an embodiment / example / implementation) may be included in at least one embodiment / example / implementation. Thus, the appearances of the phrases "according to an embodiment / example / implementation" or "in an embodiment / example / implementation" in various places in the specification do not necessarily all refer to the same embodiment / example / implementation, nor are they necessarily separate or alternative embodiments / examples / implementations that are mutually exclusive of other embodiments / examples / implementations.

[0123] Reference numerals appearing in the claims are by way of illustration only and shall have no limiting effect on the scope of the claims.Embodiments / examples and variations of the invention may be employed in any combination or subcombination although not explicitly described.

[0124] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0125] Although some diagrams include arrows on communication paths to illustrate a primary direction of communication, it is understood that communication can occur in the opposite direction of the depicted arrows.

[0126] Various implementations involve decoding. As used in this application, "decoding" can include, for example, all or part of the processes performed on a received point cloud frame (which may include a received bitstream encoding one or more point cloud frames) to produce a final output suitable for display or for further processing in the reconstructed point cloud domain. In various embodiments, such processes include one or more of the processes typically performed by an image-based decoder.

[0127] As a further example, in one embodiment, "decoding" may refer only to entropy decoding, in another embodiment, "decoding" may refer only to differential decoding, and in another embodiment, "decoding" may refer to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0128] Various implementations involve encoding. In a manner similar to the discussion above regarding "decoding," "encoding," as used in this application, may include, for example, all or part of a process performed on an input point cloud frame to produce an encoded bitstream. In various embodiments, such a process includes one or more of the processes typically performed by an image-based decoder.

[0129] As a further example, in one embodiment, "encoding" may refer only to entropy encoding, in another embodiment, "encoding" may refer only to differential encoding, and in another embodiment, "encoding" may refer to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0130] Note that the syntax elements used here are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0131] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is often considered, often given a computational complexity constraint. Rate-distortion optimization can generally be formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or coding parameter values, with a complete evaluation of their coding costs and associated distortion in the reconstructed signal after transcoding and decoding. Faster approaches can also be used to save coding complexity, particularly by computing approximate distortion based on a prediction or prediction residual signal rather than the reconstructed signal. A hybrid of these two approaches can also be used, for example by using approximate distortion for only some possible coding options and the full distortion for others. Other approaches evaluate only a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization does not necessarily require a complete evaluation of both coding cost and associated distortion.

[0132] Additionally, this application may refer to “determining” various information. Determining information may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from a memory.

[0133] Furthermore, this application may refer to “accessing” various information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0134] Additionally, this application may refer to "receiving" various types of information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of: accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0135] In addition, as used herein, the term "signal / signaling" refers in particular to indicating something to a corresponding decoder. Thus, in embodiments, the same parameters can be used on both the encoder and decoder sides. Thus, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending them (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to the corresponding decoder. Although the foregoing relates to the verb form of the term "signal / signaling", the term "signal / signaling" can also be used as a noun in this document.

[0136] A number of implementations have been described. However, it should be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or removed to produce other implementations. Additionally, it will be understood by those skilled in the art that other structures and processes may replace those disclosed, and that the resulting implementations may perform at least substantially the same functions in at least substantially the same manner to achieve at least substantially the same results (one or more) as the disclosed implementations. Therefore, the present application contemplates these and other implementations.

Claims

1. A video encoding method, the method comprising: encoding a volumetric video representing a scene, wherein the encoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of 2D projections of sub-portions of the scene over a period of time; When content of each of the instances of a first patch in the plurality of patches does not overlap content of an instance of the second patch in a time period associated with the second patch, packing the first patch into the second patch in a time interval, wherein the time interval is less than or equal to the time period; and Information defining said time interval is transmitted.

2. The method according to claim 1, wherein: The information indicates a first index and a second index of an ordered list of patches.

3. The method according to claim 1, wherein: The time period is divided into a plurality of time intervals, and the information defining the time intervals is a bit indicating one of the time intervals.

4. A video decoding method, the method comprising: Receive information at defined time intervals; decoding a volumetric video representing a scene, wherein the decoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of a 2D projection of a sub-portion of the scene over a time period, wherein a first patch of the plurality of patches is packed into a second patch of the plurality of patches over the time interval, wherein the time interval is less than or equal to a time period associated with the second patch; and The first patch is unpacked from the second patch within the time interval.

5. The method according to claim 4, wherein: The information indicates a first index and a second index of an ordered list of patches.

6. The method according to claim 4, wherein: The time period is divided into a plurality of time intervals, and the information defining the time intervals is a bit indicating one of the time intervals.

7. A video encoding device, comprising: The processor is configured as: encoding a volumetric video representing a scene, wherein the encoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of 2D projections of sub-portions of the scene over a period of time; When content of each of the instances of a first patch in the plurality of patches does not overlap content of an instance of the second patch in a time period associated with the second patch, packing the first patch into the second patch in a time interval, wherein the time interval is less than or equal to the time period; and Information defining said time interval is transmitted.

8. The device according to claim 7, wherein: The information indicates a first index and a second index of an ordered list of patches.

9. The device according to claim 7, wherein: The time period is divided into a plurality of time intervals, and the information defining the time intervals is a bit indicating one of the time intervals.

10. A video decoding device, comprising: The processor is configured as: Receive information at defined time intervals; decoding a volumetric video representing a scene, wherein the decoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of a 2D projection of a sub-portion of the scene over a time period, wherein a first patch of the plurality of patches is packed into a second patch of the plurality of patches over the time interval, wherein the time interval is less than or equal to a time period associated with the second patch; and The first patch is unpacked from the second patch within the time interval.

11. The device according to claim 10, wherein: The information indicates a first index and a second index of an ordered list of patches.

12. The device according to claim 10, wherein: The time period is divided into a plurality of time intervals, and the information defining the time intervals is a bit indicating one of the time intervals.

13. A computer program product comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform a video encoding method, the method comprising: encoding a volumetric video representing a scene, wherein the encoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of 2D projections of sub-portions of the scene over a period of time; When content of each of the instances of a first patch in the plurality of patches does not overlap content of an instance of the second patch in a time period associated with the second patch, packing the first patch into the second patch in a time interval, wherein the time interval is less than or equal to the time period; and Information defining said time interval is transmitted.

14. A non-transitory computer-readable storage medium comprising instructions for causing one or more processors to perform a video encoding method, the method comprising: encoding a volumetric video representing a scene, wherein the encoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of 2D projections of sub-portions of the scene over a period of time; When content of each of the instances of a first patch in the plurality of patches does not overlap content of an instance of the second patch in a time period associated with the second patch, packing the first patch into the second patch in a time interval, wherein the time interval is less than or equal to the time period; and Information defining said time interval is transmitted.

15. A computer program product comprising instructions, which, when executed by one or more processors, cause the one or more processors to perform a video decoding method, the method comprising: Receive information at defined time intervals; decoding a volumetric video representing a scene, wherein the decoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of a 2D projection of a sub-portion of the scene over a time period, wherein a first patch of the plurality of patches is packed into a second patch of the plurality of patches over the time interval, wherein the time interval is less than or equal to a time period associated with the second patch; and The first patch is unpacked from the second patch within the time interval.

16. A non-transitory computer-readable storage medium comprising instructions for causing one or more processors to perform a video decoding method, the method comprising: Receive information at defined time intervals; decoding a volumetric video representing a scene, wherein the decoding of the volumetric video representing the scene is based on a plurality of patches representing at least one of color or depth of a 2D projection of a sub-portion of the scene over a time period, wherein a first patch of the plurality of patches is packed into a second patch of the plurality of patches over the time interval, wherein the time interval is less than or equal to a time period associated with the second patch; and The first patch is unpacked from the second patch within the time interval.

Images