Method, apparatus and article of manufacture for generating packed video frames for volumetric and immersive video bitstreams

Abstract

Methods, apparatus, systems, and articles of manufacture to generate packed video frames are disclosed. A video encoding system disclosed herein includes a configuration determiner to create a packed video frame layout that includes an area into which a video component is to be placed. The system also includes a frame generator to form packed video frames including the video components placed into the different regions. The encoding system also includes a frame information generator that generates packed video frame information that identifies characteristics of the packed video frame, including (i) an identity of a region included in the packed video frame layout, (ii) a video component type included in the region, or (iii) information that identifies a location and size of the region. A video encoder of an encoding system encodes frames and includes packed video frame information indicating that packed video frames are included in an encoded bitstream.

Description

[0001] Related applications

[0002] This patent claims the benefit of U.S. Patent Application Serial No. 63 / 040369, filed June 17, 2020. The entirety of U.S. Patent Application Serial No. 63 / 040369 is incorporated herein by reference. Priority of U.S. Patent Application Serial No. 63 / 040369 is hereby claimed. Technical Field

[0003] This disclosure relates generally to immersive video, and more specifically to methods and apparatus for generating data packets for immersive video bitstreams. Background Technology

[0004] Volumetric video is captured using several video processing tools and cameras, and the result is formatted in three degrees of freedom. Therefore, users are provided with the feeling and perspective of being immersed in another time / place through the virtual environment depicted by the volumetric video. Furthermore, users can move around within the virtual environment, further enhancing the feeling that they have been transported to another space or time. Consequently, commercial applications using volumetric video, including virtual / augmented / mixed reality applications, are rapidly improving, and the widespread adoption and promising future of volumetric video technology are causing a surge in interest in this field. Attached Figure Description

[0005] Figure 1A This is an illustration of an example first packaging configuration that can be implemented according to the packaging atlas information of the present invention.

[0006] Figure 1B This is an illustration of the second packaging configuration, which is an example of packaging atlas information.

[0007] Figure 1C This is an illustration of the third packaging configuration, which is an example of packaging atlas information.

[0008] Figure 2 This is a block diagram of an example video encoder with an example frame packer according to the present invention.

[0009] Figure 3 This is an example embodiment of the present invention. Figure 2 A block diagram of an example frame packer.

[0010] Figure 4A It is a table that represents an example modification to the V3C unit header type to include a unit header type representing packaged video data.

[0011] Figure 4BIt is pseudocode that identifies the V3C unit information to be obtained and / or generated when the V3C unit header indicates that the V3C unit contains packaged video data.

[0012] Figure 4C It is pseudocode that identifies a way to modify the V3C parameter set to determine when a V3C unit includes packaged video and corresponding packaging information.

[0013] Figure 4D It is a table that represents the definition of a V3C profile, indicating the allowed values ​​for syntax elements of the MIV main profile and MIV extension profiles.

[0014] Figure 5 It is to achieve Figure 2 and Figure 3 The pseudocode for the frame packer's packing information generator.

[0015] Figure 6 It is to achieve Figure 2 and Figure 3 The example frame packer includes pseudocode for an example packing information generator.

[0016] Figure 7 This is a block diagram of an example decoder with an example frame unpacker according to the present invention.

[0017] Figure 8 This is an example embodiment of the present invention, including Figure 7 Example frame unpacker Figure 7 A block diagram of a portion of the decoder.

[0018] Figure 9 It can be used to implement Figure 6 and Figure 7 The pseudocode for at least a portion of the example frame unpacker.

[0019] Figure 10A and Figure 10B The implementation is illustrated in the diagram. Figure 2 and Figure 3 The flowchart of the example encoder's example program.

[0020] Figure 11 The implementation is illustrated in the diagram. Figure 2 and Figure 3 The flowchart shows an example program for a frame packer, encoder, and multiplexer.

[0021] Figure 12 The implementation is illustrated in the diagram. Figure 2 and Figure 3 The flowchart shows an example program for verifying the number of encoders and decoders.

[0022] Figure 13 It can be used to implement Figure 8 Here is the pseudocode for an example decoder validator.

[0023] Figure 14 The implementation is illustrated in the diagram. Figure 7 and Figure 8 A flowchart of an example program for a decoder.

[0024] Figure 15 The implementation is illustrated in the diagram. Figure 7 and Figure 8 The flowchart shows an example program for a frame unpacker.

[0025] Figure 16 It is constructed to execute Figure 10A , Figure 10B , Figure 11 and Figure 12 The instructions are shown in the block diagram of the example processing platform for implementing the example encoder.

[0026] Figure 17 It is constructed to execute Figure 14 and Figure 15 The instructions are shown in the block diagram of the example processing platform for implementing the example decoder.

[0027] The accompanying drawings are not to scale. Generally, the same reference numerals will be used throughout the drawings and the accompanying written description to refer to the same or similar parts. As used herein, references to connections (e.g., attachment, coupling, joining, engagement) may include intermediate members between the elements mentioned by the connection and / or relative movement between these elements, unless otherwise indicated. Therefore, a reference to a connection does not necessarily imply that two elements are directly connected and / or have a fixed relationship with each other.

[0028] Unless otherwise specifically stated, this document uses descriptive terms such as “first,” “second,” “third,” etc., without indicating or otherwise specifying any priority, physical order, arrangement in a list, and / or any sorting, but merely as labels and / or arbitrary names to distinguish elements for ease of understanding of the disclosed examples. In some examples, the descriptive term “first” may be used to refer to an element in a detailed description, while the same element may be referred to in the claims using different descriptive terms, such as “second” or “third.” In such cases, it should be understood that such descriptive terms are only used to explicitly identify those elements that may, for example, share the same name in other cases. Detailed Implementation

[0029] Visual volumetric video refers to a collection of visual volumetric video coded (V3C) frames. V3C frames are encoded by converting the three-dimensional information of a V3C frame into a collection of two-dimensional images. These two-dimensional images can then be encoded using any video and image coding standard (e.g., ISO / IEC 14492-10, ISO / IEC 23008-2, etc.). The V3C data associated with the two-dimensional encoded images can be used to reconstruct the V3C to reproduce the three-dimensional video.

[0030] A view captured by a camera can be represented as a base view or as an additional view. The base view includes all video data corresponding to the view captured by the camera. In contrast, an additional view contains video data corresponding to one or more patches (parts) of the captured view, whose reprojection relative to the position of the base view from another view is different. Therefore, after obtaining one or more base views, instead of fully retransmitting all the information needed to reconstruct each view, only information relating to the parts (patches) of the base view that have changed between time instances is transmitted.

[0031] The transmitted view portions are called patches and are contained / represented in an atlas. Within an atlas or atlas tile, a patch represents a component at a specific location within the view. An atlas may contain one or more atlas tiles. Components are various types of video data and may include data related to geometry, color, reflectivity, surface information, texture, occupancy, materials, objects, etc. An access unit comprises one or more atlases corresponding to multiple views captured on multiple cameras at the same time instance; therefore, an access unit comprises a collection of atlases all corresponding to the same time instance.

[0032] Often, to provide users with a high-quality immersive experience, V3C video is captured using numerous cameras, resulting in a large amount of video data. Additionally, in some cases, attributes (and other types of information) associated with V3C video and / or MPEG immersive video data need to be explicitly identified in separate video sub-bitstreams. For example, an MPEG immersive (MIV) video access unit refers to a sub-bitstream assembly unit that shares the same decoding order count. Sub-bitstream assembly units may include those containing occupancy information, those containing attributes, those containing geometric information, and so on. Thus, large qubitstreams, each requiring a separate decoder, can be included in the encoded visual volumetric video transmission.

[0033] Due to the number of bitstreams and the large amount of video data, multiple decoder instances are typically required to reconstruct the video and synchronize their operation. While high-end devices such as personal computers generally have enough decoder instances to synchronously process the various bitstreams of visual volumetric video data, mobile electronic devices typically lack the ability to synchronize multiple video decoder instances. Therefore, mobile electronic devices often cannot provide users with a high-quality, immersive video experience.

[0034] The systems, methods, apparatus, and articles of art disclosed herein utilize frame packing techniques prior to encoding V3C video data. Using the frame packing techniques disclosed herein results in fewer bitstreams, thus requiring fewer decoder instances. Frame packing results in one or more packed video frames, the information of which will be used at the decoder to reconstruct a 3D image from the original video data. As used herein, a “packed video frame” refers to a video layout of a packed frame comprising multiple regions, i.e., individual regions. Each region contains one type of video component data (also referred to herein as a “video data type” or “type of video data”) associated with an atlas or atlas tile (e.g., geometry, occupancy, attributes, etc.). Each region may represent a geometric, occupancy, or attribute video sub-bitstream of an atlas or atlas tile. Each region represents data that would otherwise be carried by a video sub-bitstream containing only one type of component data (e.g., geometry, occupancy, attributes, etc.). Additionally, in some examples, the packed video frame does not need to be fully occupied by a region, thus allowing unoccupied space to be padded (if necessary, for example, to make the encoded output data more friendly to the video codec). As used herein, the terms rect, rectangle, reg, and region are used interchangeably. As described in this article, while regions can take a rectangular geometry, they can in turn take any shape, and are by no means limited to a rectangular shape. As used in this article, some variables include "rect", in such examples, "rect" is used to refer to a region (also abbreviated as "reg").

[0035] The frame packing technique disclosed in this paper uses packed frame video information, which is provided along with the encoded video sub-bitstream of volumetric video data. This packed frame video information is used at the decoder to identify regions included in the packed frame, information about the location and size of the regions, information about the types of video components included in the regions, mapping information that can be used to map the regions to a sequence of video frames to be rendered, and so on. Generating this information, along with the encoded volumetric video data, allows for the generation of packed frames at the encoder, and then unpacking at the decoder. Additionally, in some examples, independent decodeable regions can be identified, and supplemental enhancement information (SEI) messages can be generated to indicate the presence of such regions and provide information about where these regions are located. In such examples, multiple independent decodeable regions can be decoded in parallel by multiple decoders.

[0036] The frame packing technique disclosed in this paper reduces the number of decoder instances required to decode encoded video because fewer bitstreams are needed to carry the encoded video signal. This is because the information included in the atlas (which is typically carried in multiple sub-bitstreams, each representing a component type) can be placed into a single packed video frame that can be decoded by a single decoder. Even when the frame packing technique results in the creation of more than one packed video frame, the number of packed video frames created and the corresponding number of decoders required to decode them are still less than the number of decoders would be needed if the encoded video were not packed video frames, as further described below.

[0037] Figure 1A This is an illustration of an example of a first packaged video frame configuration 100A that can be configured according to the packaged atlas information of the present invention. Figure 1A In the example, metadata about texture data, geometric data, and occupancy data for different atlases is included in areas that are inserted into a set of packed video frames (e.g., packed 0A, packed 1A, packed 2A, packed 3A, packed 4A, and packed 5A). Figure 1A In the example, the initial region "r0" 102A of package 0A is filled with texture data from the first atlas "a0", the second region "r1" 104A of package 0A is filled with geometric data from atlas a0, and part 106A of package 0A is unfilled. Since the base view does not include occupancy data, it is assumed that atlas a0 represents the base view.

[0038] exist Figure 1AIn the example, the packed video frame (“Pack 1A”) includes data from atlases a0 and a1, contained within three regions (e.g., region r0 108A, region r1 110A, and region r2 112A). Region r0 of Pack 1A contains texture data from atlas “a1”. Region r1 110A of Pack 1A contains geometric data from atlas a0. In some examples, region r2 112A of Pack 1A contains occupancy data from atlas a1. Additionally, region r2 occupies portion 114A, which is otherwise an empty region.

[0039] exist Figure 1A In the example, region “r0” 116A of packing 2A is filled with texture data from atlas a2, region “r1” 118A is filled with geometric data from atlas a2, and part 120A of packing 2A is unfilled.

[0040] exist Figure 1A In the example, region r0 122A of packaged 3A is filled with texture data of atlas "a3", and region r1 124A is filled with geometric data of atlas a3. In addition, region r2 126A is filled with occupancy data of atlas a3, and part of 128A (where region r2 is arranged) is otherwise unfilled.

[0041] exist Figure 1A In the example, region r0 130A of the packaged 4A is filled with texture data of the atlas "a4", region r1 132A is filled with geometric data of the atlas a4, and part 134A of the packaged 4A is not filled.

[0042] In addition, Figure 1A In the example, region r1 136A of packaged 5A is filled with texture data of atlas "a5", and region r1 138A is filled with geometric data of atlas a5. Additionally, region r2 140A is filled with occupancy data of atlas a5, and portion 142A containing region r2 140A is otherwise unfilled.

[0043] thereby, Figure 1A The frame packing configuration 100A illustration shows a sequence of six packed video frames (packing 0A, packing 1A, packing 2A, packing 3A, packing 4A, and packing 5A) from six different atlases. As shown, data associated with the six atlases is packed, with three atlases associated with two types of data (texture and geometry), and three atlases associated with three types of data (texture, geometry, and occupancy). If conventional transmission techniques are used... Figure 1AThe data represented in the six packed video frames would require fifteen sub-bitstreams (e.g., (3*2) + (3*3) = 15) to carry the data, and 15 different decoders to decode the data. By using... Figure 1A Configuration 100A requires only six bitstreams to carry the data (one for each packed video frame), therefore, only six decoders are needed to decode these six bitstreams. As described above, the first packed video frame configuration 100A illustrates the packing of different video components of the same atlas within the same packed video frame in a sequence of packed video frames comprising six packed video frames (packed 0, packed 1, packed 2, packed 3, packed 4, packed 5).

[0044] like Figure 1A As shown, each packed video frame is formed by a set of blocks that have specific positions within the video pack, as illustrated in Pack 2. In some examples, information from each block of each pack is ultimately mapped into the volumetric video sequence. Examples of this mapping will be described below. Blocks are associated with each region, although for clarity they are only illustrated in Pack 2.

[0045] Figure 1B This is an example of a second packed video frame configuration 100B, where information / data from multiple atlas tiles can be packed. The second packed video frame configuration includes three packed video frames (e.g., packed video frame packed 0B, packed video frame packed 1B, and packed video frame packed 2B). In the second packed video frame configuration 100B, the first packed video frame packed 0B contains six regions (region r0102B, region r1104B, region r2106B, region r3108B, region r4110B, and region r5112B). Figure 1B In the example second configuration, all six regions of the package 0B contain texture data, and the texture data contained in each region is associated with different atlas tiles (e.g., atlas tile a0, atlas tile a1, atlas tile a2, atlas tile a3, atlas tile a4, and atlas tile a5, respectively).

[0046] The second packed video frame 1B also contains six regions (region r0 114B, region r1 116B, region r2 118B, region r3 120B, region r4 122B, and region r5 124B). All six regions contain geometric data, and the geometric data contained in each region of packed 1B is associated with different atlas tiles (e.g., atlas tile a0, atlas tile a1, atlas tile a2, atlas tile a3, atlas tile a4, and atlas tile a5). The third packed video frame, packed 2B, contains three regions: region r0 126B, region r1 128B, and region r3 130B. Each region includes occupancy data for different atlas tiles (atlas tile a0, atlas tile a3, and atlas tile a5). Thus, the second packaged video frame configuration 100B includes atlas data of six atlas tiles, wherein geometry and texture data are included in three of the atlas tiles, and geometry, texture, and occupancy data are included in three of the atlas tiles.

[0047] Figure 1C The illustration shows an example third packed video frame configuration 100C, which includes packed 0C and packed 1C. In the third packed video frame configuration 100C, packed 0C comprises six regions (reg r0 102C, reg r1 104C, reg r2 106C, reg r3 108C, reg r4 110C, reg r5 112C). reg r0 102C, reg r1 104C, and reg r2 106C each occupy one-quarter of the area of ​​packed 0C. Each of reg r0 102C, reg r1 104C, and reg r2 106C contains texture data corresponding to atlas tile a0, atlas tile a2, and atlas tile a4, respectively. Packed 0C also includes three regions (reg r3 108C, r4 110C, r5 114C), each occupying one-sixteenth of the area of ​​packed 0C. Each of these three regions (regr3 108C, r4 110C, r5 114C) contains geometric data from atlas tile a0, atlas tile a2, and atlas tile a4, respectively. Therefore, pack 0 includes the texture and geometric data of atlas tile a0, atlas tile a2, and atlas tile a4, each representing a base view. One-sixteenth of pack 0C does not contain any data.

[0048] Figure 1CPacket 1C comprises three regions (reg r0 116C, reg r1 118C, and reg r2 120C), each occupying one-quarter of the area of ​​Packet 1C. Each of these three regions (reg r0 116C, reg r1 118C, and reg r2 120C) contains texture data corresponding to atlas tiles a1, a3, and a5, respectively. Packet 1C also includes three regions (reg r3 122C, reg r4 124C, and reg r5 126C), each occupying one-sixteenth of the area of ​​Packet 1C. Each of these three regions (reg r3 122C, reg r4 124C, and reg r5 126C) contains geometric data from atlas tiles a1, a3, and a5, respectively. Furthermore, packing 1C comprises three regions (reg r6 130C, reg 7 128C, and rec r8 132C). reg r6 and reg r8 each occupy 1 / 32 of the area of ​​packing 1C, while reg r7 occupies less than 1 / 32 of the area of ​​packing 1C. Each of these three regions (reg r6 130C, rec7 128C, and reg r8 132C) contains occupancy data from atlas tile a1, atlas tile a3, and atlas tile a5, respectively. Therefore, Figure 1C The packed 1C includes texture and geometry data of atlases (atlas tile a0, atlas tile a2, and atlas tile a4), and also includes occupancy data of atlases (atlas a1, atlas a3, and atlas a5). The packed 1C also includes a portion 134C without data. Note that individual components packed together within the same packed video frame can be encoded at different frame rates (e.g., a packed video frame of texture atlases can be sent at 60 fps, while a packed video frame of geometry atlases can be sent at 30 fps to achieve further compression or support desired functionality, depending on the application). Although, in Figure 1A , Figure 1B and Figure 1C In the example configuration, it is assumed that it is based on atlases or atlas tiles of the same size, but the atlases or atlas tiles can be changed to be of different sizes.

[0049] The encoded video bitstream contains an encoded representation of the video data. More specifically, the bitstream comprises a series of Network Abstraction Layer (NAL) units. Each NAL unit includes a NAL unit header and a Raw Byte Sequence Payload (RBSP). Information also included in the bitstream can be used to identify the end of the RBSP. In some examples, a unit may include more than one packed video frame (e.g., a sequence of packed video frames).

[0050] For illustrative purposes only. Figure 1A , Figure 1B and Figure 1C The region is rectangular. In practice, the region can take any shape, as long as the regions do not overlap.

[0051] Now go to Figure 2 , Figure 2 This is a block diagram of a simplified encoding system 200 according to the teachings of this disclosure. The simplified encoding system 200 includes an example captured video data and metadata storage device 202, an example video and metadata preprocessor 204, an example parameter / atlas data provider 206, an example geometric video data provider 208, an example attribute video data provider 210, an example occupancy video data provider 212, an example bitstream formatter 216, an example frame packer 218, an example encoder 220, and an example multiplexer 222.

[0052] In some examples, captured video data and metadata are stored in example captured video data and metadata storage device 202. Example video metadata preprocessor 204 examines the video data and metadata and separates the metadata (e.g., parameter and atlas data) from the different types of video included in the video data (e.g., geometry, attributes, occupancy, etc.). Additionally, video and metadata preprocessor 204 separates different types of video data (separating geometry, attribute, and occupancy data from each other) and provides each to geometry video data provider 208, attribute video data provider 210, and occupancy video data provider 212, respectively. In some examples, parameter and atlas data are provided to parameter and atlas data provider 206. Geometry video data provider 208, attribute video data provider 210, and occupancy video data provider 212 provide geometry video data, attribute video data, and occupancy video data to example frame packer 218, respectively, and parameter / atlas data provider 206 provides parameter / atlas data to example bitstream formatter 216. The video data type is also referred to herein as video component type.

[0053] In some examples, geometric video data, attribute video data, and occupancy video data are provided to the example packed video frame packer 218. Additionally, parameter / atlas data is provided to the example bitstream formatter 216. In some examples, the bitstream formatter 216 and the packed video frame packer 218 share a configuration for determining the packed video frames (e.g., Figure 1A Packed video frame configuration, Figure 1B Packed video frame configuration and / or Figure 1C The packaged video frame packer 218 provides the necessary information for its configuration, including geometry, attributes, and occupancy data. In some examples, the packaged video frame packer 218 provides the packaged video frames to the example encoder 220 for encoding using any suitable encoding method. In addition to generating the packaged video frames, the packaged video frame packer 218 also generates packaged video frame information for the decoder to determine where the regions of video included in the packaged video frames will be placed, in order to generate a volumetric video sequence for rendering.

[0054] Example bitstream formatter 216 determines the bitstream format of parameter / atlas data 206 corresponding to the video data contained in the packaged video frames generated by frame packer 218. In some examples, parameter / atlas data 206 also includes SEI messages. In some examples, SEI message information is provided to bitstream formatter 216 by packaged video frame packer 218. In some examples, bitstream formatter 216 provides bitstream format information, and encoder 220 provides encoded packaged video frames and packaged video frame information to example multiplexer 222, which operates to multiplex the bitstream format information with the encoded packaged video frames. In some examples, the SEI message identifies one or more regions of one or more packaged video frames that can be decoded independently of other regions. In some examples, such regions can be decoded in parallel by various decoders.

[0055] Figure 3 yes Figure 2 A block diagram of example frame packer 218. In some examples, Figure 3The frame packer 218 includes an example grade / level / level storage device 302, an example packed video frame flag setter 304, an example packed video frame packing configuration determiner 306, an example atlas / atlas tile recognizer 307, an example geometric video collector 308, an example attribute video reg collector generator 310, an example occupancy video collector generator 312, an example packed video frame generator 314, an example packed video frame information generator 316, an example metadata storage device 318, an example packed video frame storage device 322, an example metadata supplier 324, an example packed video frame supplier 326, an example decoder number verifier 328, an example evaluator 330, an example packed video frame flag storage device 332, an example packed video frame flag setter 333, an example packed video frame flag adder 334, an example packed video frame checker 336, an example reg deduction device 338, and an example comparator 340.

[0056] In some examples, encoder 220 operates according to a video coding standard with syntax defining profile, tier, and level fields. In some examples, the "profile" of the video coding standard defines what encoding tools can be used. For example, for encoding system 200, a profile can specify a set of encoding tools that encoding system 200 can use to generate encoded bitstreams. Similarly, an encoder profile can specify the type of video decoder that can decode video data generated by a video encoder (including the set of encoding tools that a video decoder must have to be able to decode the bitstream associated with that profile). In some examples, a level is a defined set of constraints on the possible values ​​that syntax elements and variables of the video coding standard can take. A tier is a level constraint of a specified category imposed on the values ​​of syntax elements or variables in the bitstream. Level constraints are nested within tiers, and a decoder conforming to a certain tier and level will be able to decode all bitstreams conforming to the same tier, lower tiers, or any tier below that level. Therefore, the level of a hierarchy is a set of specified constraints imposed on the values ​​of syntax elements in the bitstream or on the values ​​of variables used for bitstream decoding. In some examples, the hierarchy / level information specifies a constraint on the number of decoders required to decode the bitstream at the decoder. In some examples, this constraint identifies the maximum number of encoders that can be used to decode the bitstream, since a decoder with fewer decoders will not be able to decode the bitstream. In some examples, the encoding system can use the hierarchy / level information to identify the number of decoders required and ensure that the encoding occurring at the encoding system does not violate the constraint on the number of decoders. In some examples, the encoding system can determine the number of decoders that will be needed to decode the bitstream based on the characteristics of the information to be encoded, in order to form a correctly formatted bitstream with the corresponding video components.

[0057] In some examples, as further described below, to limit the number of decoder instances required to decode the encoded bitstream, the profile / level / tier syntax is modified to include a set of bits defined to indicate the maximum number of such decoder instances. In some examples, the profile / level / tier syntax may include, for example, a set of bits (e.g., 4 bits) represented by the variable “ptl_max_decodes_idc”, and used to indicate the constraint on the number of sub-bitstreams of video decoder instances that the encoded video sequence conforms to. The bitstream contains a value for “ptl_max_decodes_idc”, specifically defined as follows. For example, the 4 bits of the variable “ptl_profile_codec_group_idc” can indicate one of the values ​​0, 1, 2, 3, 4, 5, 6, 7, 8, 9, and each of these values ​​can correspond to a different maximum number of decoder instances (e.g., 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, respectively). Values ​​10-14 can be reserved for future use, and value 15 can be unconstrained. Because some devices (decoders) can support a large number of simultaneous decoders, while others may not, the same content can be encoded according to multiple levels so that the output bitstream can be decoded by decoding devices with different decoding capabilities. Thus, the addition of levels / layers / grades corresponding to the maximum number of decoders enables the use of multiple levels to support multiple encodings of the same bitstream, so that the bitstream can be decoded by decoding devices with different decoding capabilities. In some examples, example levels / layers / grades supported by encoding system 200 are stored in example level / layer / grade storage device 302. As described above, in some examples, encoding system 200 may be able to encode multiple video bitstreams, each corresponding to a different level among multiple levels, and each level having a different maximum number of decoders (represented by the variable "max_decode_idc") to support decoding devices with different levels of decoding capabilities.

[0058] In some examples, the information stored in the grade / level / tier signal storage device is set by the manufacturer of the encoding system 200. In some examples, the information stored in the grade / level / tier signal storage device can be populated and / or reset / revised by the operator of the encoding system 200. In some examples, it can be obtained from... Figure 2Another encoder block included in the process receives information stored in a grade / level / tier signal storage device. In some examples, the configuration of the packed video frames used by frame packer 218 to form a region is at least partially governed by the information stored in the grade / level / tier signal storage device, because each packed video frame included in the configuration of the packed video frames requires a decoder instance on the decoding side. Thus, by specifying the maximum number of decoder instances, grade / level / tier information can be taken into account when generating the packed video frames.

[0059] In some examples, when a Natural Abstraction Layer (NAL) unit of video is to be packed, the packing video frame flag setter 304 causes the packing frame video flag to be set. In some examples, setting the packing frame video flag includes setting one or more bits in the metadata storage device 318. In some examples, this is based on information in the tier / level / grade storage device 302 or based on information from... Figure 2 Information provided by any other block allows the Packed Video Frame Configuration Determiner 306 to determine the Packed Video Frame Configuration, which identifies the regions of the Packed Video Frame, the size of the regions, the location of the regions, the type of data to be included in each region, and so on. Figure 1A , Figure 1B and / or Figure 1C The diagram illustrates an example packaged video frame configuration. In some examples, any of the various packaged video frame configurations can be determined based on aspects / characteristics of the video stream to be packaged into packaged video frames. For example, the determined packaged video frame configuration may depend on the number of available views and the typical number of decoder instances for the product used in that use case.

[0060] As described above, in some examples, the packaged video frame configuration includes information identifying the number of regions to be included in the packaged video frame, the dimensions of each region (e.g., width, height), the type of data to be included in the regions, and so on. In some examples, the packaged video frame configuration uses an atlas tile recognizer 307 to determine the number of atlases or atlas tiles associated with the incoming geometry, attribute, and occupancy video data to be processed. In some examples, the incoming geometry data, attribute data, and occupancy data each include information identifying the atlas or atlas tile corresponding to that information. In some examples, the atlas tile recognizer 307 examines the incoming geometry, attribute, and occupancy data to determine one or more atlases or atlas tiles corresponding to each type of data. In some examples, all geometry, attribute, and occupancy data associated with a first atlas tile are received at the packaged video frame packer 218 at the same time (or within the same time period). In some such examples, the atlas recognizer 307 may tag or otherwise associate incoming geometric attributes and occupancy data with atlas tile numbers / types (e.g., the atlas tile recognizer 307 identifies information about atlas tiles as associated with a base view or an additional view, etc.).

[0061] In some examples, the example packaged video frame generator 314 uses packaged video frame information from the packaged video frame information generator 316 to generate packaged video frames with individual regions, including geometric data provided by the example geometry data collector 308, attribute data provided by the example attribute data collector 310, and occupancy data (if any) provided by the example occupancy data collector 312. In some examples, each region has a size (determined by the packaged video frame configuration determiner 306) and / or the amount of collected geometry data, attribute data, and occupancy data. Note that some atlases may not have occupancy data, and therefore, occupancy regions will not always exist for every existing geometric and attribute region.

[0062] In some examples, the packaged video frame generator 314 uses a determined packaged video frame configuration and / or the packaged video frame information generator 316 to ensure sufficient packaged video frame information for the decoder to retrieve the region of the corresponding packaged video frame. In some examples, the packaged video frame generator 314 stores the packaged video frames in an example packaged video frame storage device 322. Furthermore, the packaged video frame generator 314 includes the packaged video frame information as metadata carried with the corresponding packaged video frame or in the metadata associated with the corresponding packaged video frame. In some examples, the packaged video frame information metadata may include dimensional information about the size of the region, the placement of the region in the sequence of volumetric video frames, atlas information identifying the atlas or atlas tiles corresponding to each region, and any other information required by the decoder to unpack the packaged video frames of the video data, thereby enabling the unpacked data to be used to reconstruct a 3D image / view. In some examples, the example packaged video frame information generator 316 stores the packaged video frame information in a metadata storage device 318, and the metadata provider 324 provides the packaged video frame information to the packaged video frame storage device for inclusion with the corresponding packaged video frame. In some examples, metadata information and the corresponding video are not stored in the packaged video frame storage device, but are directly provided to the example encoder 220 for encoding. In some examples, one or more packaged video frames and one or more corresponding packaged video frames are provided to the example packaged video frame provider, which then provides them to the example encoder 220. In some examples, the metadata provider 324 can provide... Figure 2 The example bitstream formatter provides information corresponding to atlases or atlas tiles of packed video frames (or multiple packed video frames). In some examples, Figure 2 The bitstream formatter formats the parameter data and / or SEI messages associated with the corresponding packetized video frames. In some examples, the example packetized video frame provider 326 provides packetized video frames (obtained from the packetized video frame storage device 322) to... Figure 2 Example encoder 220 of example encoding system 200, at which packaged video frames are encoded.

[0063] Furthermore, example multiplexer 222 multiplexes the bitstream of metadata / parameter / SEI message data with the corresponding encoded data for delivery to the decoder for decoding and rendering. Note that in some examples, the packetized video frame generator 314 does not generate the packetized video frames themselves, but rather associates the packetized video information 314 with the incoming, collected video data to decode the video data using the packetized video information. In some examples, multiple packetized video frames are generated simultaneously (e.g., associated with corresponding packetized video frame information), so that multiple packetized video frames are generated at the same time (or within the same time period of each other). Thus, the packetized video frame includes packetized video frame information (to the receiving decoder) indicating the existence of the packetized video frame and further indicating how the volumetric video data contained in the packetized video frame will be placed to be included in the reconstructed volumetric video view.

[0064] As briefly described above, in some examples, the encoding system 200 includes features for specifying regions of packed video frames(s) that can be independently decoded. In some examples, the encoding system aligns these regions with regions (e.g., tiles or sub-pictures) that are independently encoded by the video codec. In some such examples, the bitstream formatter creates SEI messages to identify the correspondence between packed video frames and independently decodeable regions. In some such examples, the SEI message identifies the top-left and bottom-right tile in a rectangular region of tile rows and columns, such as that used in the HEVC encoding standard's time-motion-constrained tile set. In some examples, the SEI message identifies a sub-picture ID, such as that used in the V3C encoding standard.

[0065] The above enables use cases where a Media Aware Network Element (MANE) or MIV decoder can discard some video bitstream while still enabling the decoding of the remaining bitstream. The encoder can choose different quantization parameter (QP) values ​​for different regions within the same packed video frame.

[0066] Figure 4A The diagram illustrates a set of V3C unit header types, and Figure 4B The illustration shows example information associated with packaged video frame data units. In some examples, video coding standards include information identifying the type of data included in a packaged video frame unit. For example, the V3C standard includes the following unit types: V3C parameter set data, atlas data, occupancy video data, attribute video data, and packaged video frame data. In some examples, the data unit associated with or included in the packaged video frame data is... Figure 4AThe header “V3C_PVD” shown is used for identification. In some examples, when the V3C_PVD unit header is used to identify encoded data, the metadata associated with the packetized video frame data unit header includes a set of bits (e.g., 4 bits), represented by the variable “vuh_v3c_parameter_set_id”, which identifies the location of the parameter set associated with the packetized video frame data unit. (See also...) Figure 4B Furthermore, the encoded data identified by the V3C_PVD unit header is associated with metadata, which includes a set of bits (e.g., 4 bits) represented by the variable "vuh_pack_index". "vuh_pack_index" identifies the number of the packed video frame assigned to the unit of data associated with the packed video frame. Thus, in some examples, Figure 2 The bitstream formatter is configured to include bits for "vuh_v3c_parameter_set_id" and "vuh_pack_index" in the metadata associated with the packed video frame data unit. In some examples, each packed video frame sub-bitstream is associated with a separate value for "vuh_pack_index" and with the codec specified by the variable "pi_pack_codec_id".

[0067] Figure 4C The diagram illustrates the V3C parameter set data and a set of bits that are set or not, depending on the presence of various types of data in the bitstream to be decoded. In some examples, to accommodate the processing of packed video frame data units, the V3C parameter set is modified to include "vps_pack_video_enabled_flag 402C". When this flag is set as determined by pseudocode line 404C, the bitstream to be decoded (e.g., the corresponding video packed video frame data units) includes data from... Figure 2 The frame packer 218 generates packed video frames. As described above, the packed video frame flag setter 304 sets flags by generating (or toggling) bits included in the metadata to be provided to the example bitstream formatter. Figure 8 As indicated by the V3C parameter set, in some examples, when vps_pack_video_enabled_flag indicates that the corresponding bitstream contains packed video frame data, the decoder decoding the bitstream will obtain packed video frame information (pseudocode line 406C), which includes information needed to unpack the packed video frames. In some examples, the information included in the packed video frame information is at least partially generated by the packing information generator 316, as described below.

[0068] refer to Figure 4DThe diagram illustrates a method for modifying the MIV extension profile to account for the presence of packaged video frames. In some examples, the "vuh unit type" of the MIV extension profile is modified to include a fifth bit, which, when set, indicates that packaged video frames are included in the corresponding video bitstream. In some examples, the syntax value of the profile toolset idc (of the MIV extension profile) is modified to equal 65 when packaged video frames are included in the processed video stream. Additionally, a VME embedded occupancy variable is added, which can take a value of 0 or 1, and indicates whether the corresponding bitstream includes embedded occupancy data, where occupancy information is embedded within the geometric component. A VPS occupancy video presence flag is added, which can take a value of 0 or 1, and indicates whether the corresponding bitstream includes occupancy video data. A VPS packaging enable flag is also added, which can take a value of 0 or 1, and indicates whether the corresponding bitstream includes packaged video frames.

[0069] Refer again Figure 3 In some examples, the packaged video frame information generated by the example packaged video information generator 306 includes a set of bits (e.g., 4 bits) that provides an identifier for a set of V3C parameter data corresponding to the packaged video frame. In some examples, the packaged video frame information generator 316 generates a unique packaged video frame index number for each generated packaged video frame and provides the packaged video frame index number to the metadata provider 324 for distribution to the example bitstream formatter 216. In some examples, this information is instead converted into metadata to be included in the corresponding encoded packaged video frame.

[0070] Still referencing Figure 3 The example decoder verifier 328 stores flag information in the example flag storage device 332, which includes bits corresponding to a set of flags. These flags are identified in a parameter set associated with the packaged video frame unit. Each flag included in the flag set corresponds to a different type of video data (e.g., geometry, auxiliary, attribute, occupancy, etc.), and each flag indicates whether the corresponding type of video data exists in the corresponding packaged video frame unit. In some examples, attribute data may include subcategories of attribute data, and the variable "ai_attribute_count" is used to represent the number of subcategories included in the attribute video data.

[0071] In some examples, the decoder number verifier 328 also includes an example flag adder 334 that sums the number of flags associated with each atlas / atlas tile of the packaged video frame unit to determine the total number of types of video data included in the packaged video frame unit. The sum of the flags is the number of decoders that the decoder receiving the encoded video can decode, and is equal to the variable “NumDecodes”. As the total number of flags is summed, the example packaged video frame checker 336 checks whether the packaged video frame video flags indicate that the video unit includes the packaged video frame unit. If the packaged video frame video flags indicate that the video unit includes the packaged video frame, then for each packaged video frame, the example region subtractor 338 subtracts 1 from the number of regions included in each packaged video frame from the “NumDecodes” value. The number of regions included in each packaged video frame is represented by the variable “pi_num_rect[k]-1”, where the variable “k” represents the number of packaged video frames included in the total number of packaged video frames. Next, the example comparator 340 compares the value of NumDecodes with a value corresponding to the maximum number of decoders that the receiving decoder can decode. The maximum number of decodes is represented by the variable "MaxDecodes". In some examples, the frame packer 218 is pre-programmed with information that identifies the corresponding MaxDecodes value for the various types of decoders to which the video stream may be transmitted. In some examples, the maximum number of decodes is included in the grade / layer / level syntax information.

[0072] If the example comparator 340 determines that the value of NumDecodes is less than or equal to the value of MaxDecodes, then the frame packer 218 can continue to generate packed video frames. If the comparator 340 determines that the value of NumDecodes is not less than or equal to the value of MaxDecodes, then the example encoder 220 will not encode the corresponding video data. Thus, the example decoder number verifier 328 determines whether the number of decoders required to decode the encoded bitstream exceeds the constraint MaxDecodes. In some examples, the decoder number verifier 328 is included in the decoding system to ensure that the decoding system has a sufficient number of decoders. In some examples, the required number of decoders is... Figure 2 The parameter data sent by the encoding system is provided, and Figure 7 and Figure 8 The decoding system simply compares the required number with the available number, and if the available number is insufficient, it does not decode the incoming bitstream.

[0073] Figure 5This is pseudocode that can be used by the packed video frame information generator 316 to generate packed video frame information for a set of packed video frames. In some examples, the packed video frame information generated by the packed video frame information generator 316 (for use by the decoder) specifies the number of V3C_PVD units that will be used to identify the region information, and is represented by the variable "pi_num_packs". The packed video frame information also specifies the codec used to compress the corresponding packed video frame for each packed video frame (in... Figure 2 The codec identifier is represented as an identifier for the encoding system 200, and is indicated by the variable "pi_pack_codec_id[k]", where "k" is the index of the number of video frames packed. The value of "pi_pack_codec_id" ranges from 0 to 255, inclusive. In some examples, the codec identifier is transmitted to the decoder in the Supplemental Enhancement (SEI) message associated with the packed video frames (e.g., via a component codec mapping SEI message) or via any other means.

[0074] In some examples, the packed video frame information also indicates the nominal 2D bit depth to which the k-th packed video frame should be converted, and is represented by the variable “pi_pack_nominal_2d_bitdepth_minus1[k]+1”. In some examples, the value of “pi_pack_nominal_2d_bitdepth_minus1[k]” ranges from 0 to 31, inclusive. The nominal 2D bit depth represents the number of bits used to represent component information. Thus, for example, the geometric data of a packed video frame can be represented by the number of bits “A”, and the attribute data corresponding to the same packed video frame can be represented by the number of bits “B”. In some such examples, the packed video frame information generator 316 can determine that all components of the packed video frame will be represented by the same number of bits (e.g., the larger of the values ​​A and B), and any unfilled bit positions can be filled.

[0075] The packed video frame information can also indicate the width and height of each packed video frame separately. The width and height can be represented by the variables "pi_pack_width[k]" and "pi_pack_height[k]", respectively. The packed video frame information also specifies the number of regions included in the k-th packed video frame, represented by the variable pi_num_rect[k]. The packed video frame information also specifies the vertical (x) and horizontal (y) position of the top-left corner of each region included in each packed video frame, in units of "AtlasPatchPackingBlockSize". The variable "AtlasPatchPackingBlockSize" is a unit that represents the size of the blocks included in the patch of the atlas currently being processed. All patch, atlas, and atlas tile sizes are integer multiples of AtlasPatchPackingBlockSize, so sizes can be represented more efficiently in these units rather than in a single pixel. In some examples, patches corresponding to atlases are configured to be stored in blocks within the region being processed. The vertical and horizontal positions can be represented by the variables “pi_rect_pos_x[k][i]” and “pi_rect_pos_y[k][i]”.

[0076] The packaged video frame information also specifies the width and height of the region for each packaged video frame, in units of "AtlasPatchPackingBlockSize". The width and height of the region are represented by the variables "pi_rect_width[k][i]" and "pi_rect_height[k][i]", respectively. Conditions can be applied to restrict the position and size of the region so that the region fits within the size of the packaged video frame, for example: the value of "pi_rect_pos_x[k][i]" + "pi_rect_width[k][i]" should be less than or equal to pi_pack_width[k], and the value of "pi_rect_pos_y[k][i]" + "pi_rect_height[k][i]" should be less than or equal to pi_pack_height[k].

[0077] Additionally, to ensure that no regions overlap, the frame-packed video frame 218 may include an evaluator 330 to ensure that for all values ​​of “m” and “n”, at most one value of “i” (representing the index number of the packed video frame) satisfies the following statement to be evaluated: [(pi_rect_pos_x[k][i]<=m<pi_rect_pos_x[k][i]+pi_rect_width[k][i])]&&[(pi_rect_pos_y[k][i]<=n<pi_rect_pos_y[k][i]+pi_rect_height[k][i])].

[0078] The packaged video frame information also identifies the rotation (if any) of the i-th region in the k-th packaged video frame. This rotation can be represented by the variable "pi_rect_orientation[k][i]". In some examples, the packaged video frame generator 314 can rotate these regions in a way that allows the packaged video frame to have a smaller size than it would be possible without rotation when including one or more regions in the packaged video frame.

[0079] The packaged video frame information may also include the atlas or atlas tile identifier (ID) of the i-th region of the k-th packaged video frame, and is represented by the variable "pi_rect_atlas_id[k][i]". The atlas ID identifies which of the multiple atlases is the source of the data stored in the corresponding rect.

[0080] In some examples, the packaged video frame information also identifies which of several maps (each represented by a different "map index" number) is associated with the i-th region of the k-th packaged video frame. The map index number used to identify the map is represented by the variable "pi_rect_map_index[k][i]".

[0081] In some examples, the packaged video frame information includes a flag that identifies whether the i-th region of the k-th packaged video frame contains auxiliary video data. In some examples, the variable "pi_rect_auxiliary_video_flag[k][i]" represents the auxiliary video flag for the i-th region of the k-th packaged video frame.

[0082] The packaged video frame information also specifies the V3C type of the data contained in the i-th region of the k-th packaged video frame atlas. The V3C type of the data can be represented by the variable "pi_rect_type_id[k][i]". In some examples, the value of "pi_rect_type_id[k][i]" spans from 2 to 4. In some examples, the V3C type of the data contained in the region can be occupancy type, geometry type, attribute type, etc.

[0083] In the packaged video frame information, the variable "pi_rect_attr_index[k][i]" specifies the attribute index of the i-th region of the k-th packaged video frame. This attribute index indicates which type of attribute data is included in the i-th region. In some examples, various types of data are categorized as attribute data, and each type of data is numbered in the index; thus, the attribute index number reflects the type of attribute data included in the i-th region.

[0084] “pi_rect_attr_partition_index[k][i]”, when present, specifies the attribute partition index of the i-th region in the k-th packed video frame. When not present, the value of “pi_rect_attr_partition_index[k][i]” is inferred to be 0. The attribute partition index identifies which of a set of subcategories of the attribute type are included in the attribute.

[0085] Therefore, the variables identified above represent examples of packaged video frame information generated by the example packaged information generator 316 before, during, and / or after the generation of packaged video frames. Additionally, as described above, the packaged video frame information is provided to the bitstream formatter and multiplexer for multiplexing with the corresponding packaged video frames according to the operating standards of the control coding system 200.

[0086] Still referencing Figure 5 In some examples, the packed video frame information generator 316 generates packed video frame information for each of k packed video frames by generating the packed video frame codec index number, the packed video frame nominal 2D bit depth minus 1, the packed video frame width, the packed video frame height, and the number of regions to be included in the packed video frame. Next, for each region included in each region, the packed video frame information generator 316 generates the region position within the packed video frame as the x, y position within the packed video frame. The packed video frame information generator 316 also generates width, height, orientation, atlas index number and / or atlas tile index number, data type index number, atlas index number, auxiliary data flag (set when auxiliary data is included in the region), and attribute index number. Next, Figure 3The evaluator 330 can be used to determine whether the value of the attribute dimension partition minus 1 multiplied by the atlas index number (or atlas tile index number) and then multiplied by the attribute index number is greater than zero. If so, the packaged video frame information generator 316 generates region attribute partition index numbers for the attribute data. The packaged video frame information generator 316 continues for the next region included in the packaged video frame until data has been generated for all regions of the packaged video frame. Then, the packaged video frame information generator 316 selects the next set of incoming geometry, attribute, and / or occupancy data and generates the aforementioned data for the next packaged video frame. Region information is generated for each region included in the next packaged video frame in the above manner, and the packaged video frame information generator 316 continues in this manner until the incoming video stream or video stream sequence is encoded.

[0087] Figure 6 The illustration shows the information that the packetized video frame information generator 316 will generate when the packetized video frame independent region SEI message is used to identify the correspondence between regions in the packetized video frame and independent decodable regions in the video sub-bitstream. Furthermore, Figure 6 This is pseudocode that can be used to generate SEI messages to indicate the correspondence between regions of a packed video frame and independent decodable regions in the video sub-bitstream. In some such examples, the example packed video frame information generator 316 can generate signaling that will be included in the SEI message for an independent region of the packed video frame. Thus, the example packed video frame information generator 316 generates a set of variables, such as Figure 6 The pseudocode includes the following variables. In some examples, this group of variables includes the variable "pir_num_pack", which represents the number of packed video frames for which it identifies information about independently decodable regions. The variable "pir_description_type_idc[k] equals 0" indicates the tile indices of the top-left and bottom-right tiles of the independently decodable region (e.g., a set of temporally motion-constrained tiles) for the region of the k-th packed video frame. Based on some encoding criteria, the atlas can be subdivided into tiles and can be manipulated in tile form. The variable "pir_description_type_idc equals 1" indicates the sub-picture ID for the region of the k-th packed video frame. The variable "pir_description_type_idc[k]" is in the range of 0 to 1. Other values ​​(e.g., values ​​in the range 2..3) can be reserved for future use.

[0088] In some examples, the variable "pir_num_rect[k]" specifies the number of regions in the k-th packed video frame that are labeled with information about independently decodable regions. A set of variables, "pir_top_left_tile_idx[k][i]" and "pir_bottom_right_tile_idx[k][i]", when present, identifies the tile indices of the top-left and bottom-right tiles of the independently decodable regions (e.g., a set of temporally motion-constrained tiles), corresponding to the i-th region of the video sub-bitstream of the k-th packed video frame in tile raster scan order. The variable "pir_subpic_id[k][i]", when present, identifies the subpicture ID corresponding to the i-th region of the video sub-bitstream of the k-th packed video frame.

[0089] Still referencing Figure 6 In some examples, information identifying packet-independent regions (PIRs) of a video frame can be used by the packet-independent video frame information generator. Figure 6 The pseudocode generation is as follows. In some such examples, for each of several packed video frames "k" for which information corresponding to independently decodable regions is assigned, the packed video frame information generator generates packed video frame information. In some such examples, a PIR description type is obtained, which (as described above) indicates, when equal to 0, the tile indices of the top-left and bottom-right corner tiles of the PIR independently decodable regions (e.g., a set of time-motion-constrained tiles) for the region of the k-th PIR packed video frame. Next, the packed video frame information generator determines, based on, for example, the packed video frame configuration, the number of regions for which the independently decodable region information of the k-th PIR packed video frame should be assigned. Then, for each packed video frame with a PIR description type equal to zero, the packed video frame information generator generates / determines the tile indices of the top-left and bottom-right corner tiles of the PIR independently decodable regions corresponding to each i-region of the video sub-bitstream of the k-th packed video frame. When the value of the variable pi_description_type_idc is equal to 1, the packaged video frame information generator generates a sub-image ID for each region of each packaged video frame.

[0090] Figure 7 This is a block diagram of a portion of an example packetized video frame video decoding system 700. In some examples, the packetized video frame video decoding system 700 includes an example demultiplexer 702, an example packetized video frame video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718.

[0091] In some examples, example demultiplexer 702 from Figure 2 The multiplexer of the video encoding system 200 receives the encoded video stream. The demultiplexer 702 separates metadata / SEI messages from the encoded video stream and provides the separated information to an example metadata / SEI message extractor 716. In some examples, the metadata / SEI message extractor 716 examines the extracted information to identify SEI messages included in the extracted information and identifies individual data fields and associated data corresponding to the video stream parser 704. In some examples, the video stream parser 704 uses the vuh unit header (“vuh_pack_index”) corresponding to the unit of the packed video frame sub-bitstream to identify the unit containing the packed video frame. In some examples, the video stream parser 704 identifies the codec (using the variable “pi_pack_codec_id”). In some examples, the values ​​of the variables vuh_pack_index and pi_pack_codec_id are provided to the video stream parser 704 by the example metadata / SEI message decoder 718. In some examples, the vuh_pack_index and pi_pack_codec_id information may be included in the packed video frame sub-bitstream.

[0092] In some examples, the video stream parser 704 uses information provided by the metadata / SEI message decoder 718 to identify units in the sub-bitstream corresponding to packed video frames, and identifies the information corresponding to those units. In some examples, the information provided by the metadata / SEI message decoder 718 for each packed video frame includes the color component index, bit depth, packed video frame width, and packed video frame height. Furthermore, the metadata / SEI message decoder 718 provides the number of spectral streams associated with the packed video frame, and, if necessary, obtains the spectral count associated with the currently being operated on packed frame.

[0093] In some examples, the example video stream parser 704 uses collected / determined information to assign values ​​to a set of variables associated with the decoded packetized video frames, including: 1) "decPackFrame[frameIdx][compIdx][y][x]", 2) "decPackBitdepth[frameIdx]", 3) "decPackWidth[frameIdx]", 4) "decPackHeight[frameIdx]", and 5) "mapIdx". The value of the variable "frameIdx" is the index of the decoded packetized video frame, and the value of the variable "compIdx" is the color component index. The variable "y" is the row index in the decoded packetized video frame and is in the range of 0 to the value represented by the variable "decPackHeight[frameIdx]-1", inclusive; and the variable "x" is the column index in the decoded frame and is in the range of 0 to "decPackWidth[frameIdx]-1", inclusive. The value of the map index (“mapIdx”) falls within the following range: this range starts at zero and extends to the value evaluated by the statement: “vps_multiple_map_streams_present_flag?vps_map_count_minus1:0”, including both ends of the range. In some examples, the above statement evaluates to the value of the variable “vps_map_count_minus1” when the value of the variable “vps_multiple_map_streams_present_flag” is true (or not equal to zero), otherwise it evaluates to zero. The variable “vps_map_count_minus1” is a value that represents the number of maps included in or otherwise associated with the currently packed video frame unit.

[0094] In some examples, packaged video frame information is included with the packaged video frames and parsed by video stream parser 704, where the packaged video frame information is extracted from the encoded video. The encoded video is provided to example frame decoder 708 for decoding, and example packaged video frame information is provided to frame unpacker 710. In some examples, frame decoder 708 decodes one or more packaged video frame units according to any video decoding standard and provides the resulting decoded one or more packaged video frame units to example frame unpacker 710. Frame unpacker 710 uses the packaged video frame information associated with the decoded packaged video frames, variables recognized by video stream parser 704, etc., to unpack the packaged video frame units to unpack one or more decoded packaged video frame units. The information extracted from the unpacked frames is then provided to reconstructor 712 to reconstruct the original 3D image for rendering at renderer 714. In some examples, the frame unpacker 710 operates to use the packed frame information to identify regions, region characteristics, blocks of identified regions, and / or to map blocks to a map of the volumetric video sequence in real-time or near real-time for display by the renderer. In some such examples, the frame unpacker 710 is deployed for... Figure 8 The operations described by the decoding system may not necessarily be performed in the same order or in exactly the same way.

[0095] Figure 8 This is a block diagram of the example metadata / SEI message decoder 718, the example frame unpacker 710, and the example reconstructor 712, all of which are also... Figure 7 As shown in the figure. As described above, the metadata / SEI message decoder 718 decodes information (including packed video frame information associated with the packed video frame(s) to be unpacked) for identifying packed video frame units, atlas data headers corresponding to packed video frame units, decoded patch units of packed video frame units, decoded block-to-patch maps, etc.

[0096] In some examples, the example frame unpacker 710 includes an example reg (region) counter 801, an example variable value setter 802, an example packaged video frame unit getter 804, an example reg type determiner 806, an example reg selector 807, an example reg extractor 808, an example reg height and reg width incrementer 810, an example reg positioning determiner 812, and an example mapper 814.

[0097] In some examples, the example packed video frame unit getter 804 receives and / or retrieves (from...) Figure 7The frame decoder 708) processes the packed video frame units. In some examples, "k" packed video frames are included in a packed video frame unit, and each of the k packed video frames is operated by the block of the frame unpacker 710 in the manner described below. In some examples, the packed video frame unit acquirer 804 does not receive (or acquire) the frames, but instead receives information about the position of the packed video frames within the packed video provided by the decoder.

[0098] An individual packaged video frame within a packaged video frame is referred to as the k-th packaged video frame, and the value of k is incremented each time a packaged video frame is processed by the frame unpacker 710. In some examples, the example region selector 807 selects one of a set of "i" regions associated with the k-th packaged video frame. An individual region within these regions is referred to as the i-th rect. Next, the atlas index number (or atlas tile index number) corresponding to the i-th region of the k-th packaged video frame is identified, and the atlas identifier corresponding to the i-th region of the k-th packaged video frame is identified. As described above, in some examples, the packaged video frame information is received from the metadata / SEI message decoder 718, and / or from the inspection of packaged video frame units and / or from the information provided by the frame decoder 708, and is used to identify the region from which to select the i-th region.

[0099] In some examples, the position of the block within the selected i-th region is determined by the example region locator 812. The determined position is represented by variables "x, y", where the value of x is the vertical position of the block within the region, and y is the horizontal position of the block within the region. In some examples, the block corresponds to a square of the same size contained within the region, so that all the blocks together form the region. Next, the frame index number and component index number of the currently being operated on packing video frame unit (e.g., a frame in which the i-th region of the k-th packing video frame is inserted) are obtained.

[0100] In some examples, the region type determiner 806 determines the type of video data contained in the i-th region of the k-th packed video frame (e.g., occupancy data, geometric data, attribute data, etc.). In some examples, when the region type determiner 806 determines the data type, it also generates information indicating that the type of data residing at the x, y positions of the i-th region of the k-th packed video frame is the data type determined by the region type determiner. In some examples, the example frame unpacker 710 provides the decoded video data of the i-th region of the k-th packed video frame, along with the corresponding collected information, as output to the example reconstructor 712. In some examples, the output provided to the reconstructor 712 includes the atlas index number of the i-th region, the spectral index number of the i-th region, the frame index number of the current packed video frame unit, the component index number, and the x, y position information. The example reconstructor 712 uses this information to insert the corresponding raw video data into the image being reconstructed. In some examples, when the data type is determined to be attribute data, frame unpacker 710 generates the information identified above, along with an attribute index number that identifies which of the various attributes resides at the indicated location. In some examples, after generating the information output, frame unpacker 710 performs the same operation on the next block at the next position (e.g., (x+1, y=1)) of the i-th region of the k-th packed video frame. In some examples, after all blocks of the i-th region have been processed, the next region of the k-th packed video frame (e.g., the (i+1)-th region) is processed in the manner described above. Similarly, after all regions of the packed video frame have been processed, the next packed video frame (e.g., the (k+1)-th packed video frame unit) is obtained, and the operation of frame unpacker 710 is repeated. In some examples, when all packed video frames of a packed video frame unit have been processed, the next set of packed video frames associated with another packed video frame unit is processed as described above. In some examples, when all the packaged video frames of a packaged video frame unit have been processed, the packaged video frame unit acquirer 804 responds by acquiring or receiving the next packaged video frame unit and associated information from the video decoder.

[0101] In some examples, for blocks containing regions with occupancy type data, frame unpacker 710 generates the variable "decOccFrame[atlasIdx][mapIdx][frameIdx][compIdx][y][x]" and its corresponding value. For blocks containing regions with geometry type data, frame unpacker 710 generates the variable "decGeoFrame[atlasIdx][mapIdx][frameIdx][compIdx][y][x]", and its corresponding variable. Similarly, for blocks containing regions with attribute data, frame unpacker 710 generates the variable: ("decAttrFrame[atlasIdx][attrIdx][mapIdx][partIdx][frameIdx][compIdx][y][x]") and its corresponding data. As described above, this information is used by reconstructor 712 to reconstruct the view represented by packed video frame units. In some examples, the frame unpacker may be... Figure 9 The example pseudocode is used to implement this.

[0102] Although Figure 2 and Figure 3 The diagram illustrates an example of implementing encoding system 200, but... Figure 2 and Figure 3 One or more of the components, processes, and / or devices shown may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other way. Additionally, the example includes: video capture data and metadata storage device 202; video and metadata preprocessor 204; parameter / atlas data supplier 206; geometric video data supplier 208; attribute video data supplier 210; occupancy video data supplier 212; bitstream formatter 216; frame packer 218; encoder 220; multiplexer 222; grade / hierarchy / level storage device 302; pack video flag setter 304; pack configuration selector 306; atlas recognizer 307; and geometric area. Domain generator 308, example attribute region generator 310, example occupied region generator 312, example packaging generator 314, example packaging information generator 316, example metadata storage device 318, example packaging storage device 322, example metadata provider 324, example packaging provider 326, example decoder verifier 328, example evaluator 330, example flag storage device 332, example flag setter 333, example flag adder 334, example packaged video inspector 336, example region subtraction device 338, and example comparator 340, and / or more generally, Figure 2 and Figure 3The example encoding system 200 can be implemented using hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, it includes: example capture video data and metadata storage device 202, example video and metadata preprocessor 204, example parameter / atlas data supplier 206, example geometric video data supplier 208, example attribute video data supplier 210, example occupancy video data supplier 212, example bitstream formatter 216, example frame packer 218, example encoder 220, and example multiplexer 222; example grade / level / hierarchy storage device 302, example pack video flag setter 304, example pack configuration selector 306, example atlas recognizer 307, example geometric region generator 308, example attribute region generator 310, example occupancy region generator 312, and example pack generator. 314, Example Packaging Information Generator; 316, Example Metadata Storage Device; 318, Example Packaging Storage Device; 322, Example Metadata Provider; 324, Example Packaging Provider; 326, Example Decoder Verifier; 328, Example Evaluator; 330, Example Flag Storage Device; 332, Example Flag Setter; 333, Example Flag Adder; 334, Example Packaging Video Inspector; 336, Example Region Subtractor; 338, and Example Comparator; and / or more generally, any one of the Example Encoding System 200, may be implemented by one or more analog or digital circuits, logic circuits, (one or more) programmable processors, (one or more) programmable controllers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application-specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs), and / or (one or more) field-programmable logic devices (FPLDs).When any device or system claim in this patent covers a purely software and / or purely firmware implementation, examples include: a video capture data and metadata storage device 202, a video and metadata preprocessor 204, a parameter / atlas data supplier 206, a geometric video data supplier 208, an attribute video data supplier 210, an occupancy video data supplier 212, a bitstream formatter 216, a frame packer 218, an encoder 220, a multiplexer 222, a grade / level / hierarchy storage device 302, a pack video flag setter 304, a pack configuration selector 306, an atlas recognizer 307, a geometric region generator 308, an attribute region generator 310, and an occupancy region generator 312. Example package generator 314, example package information generator 316, example metadata storage device 318, example package storage device 322, example metadata provider 324, example package provider 326, example decoder verifier 328, example evaluator 330, example flag storage device 332, example flag setter 333, example flag adder 334, example packaged video inspector 336, example region subtraction device 338, and example comparator 340 are hereby explicitly defined as including non-transitory computer-readable storage devices or disks containing the software and / or firmware, such as memory, digital versatile disks (DVDs), compact disks (CDs), Blu-ray discs, etc. Furthermore, ... Figure 2 and Figure 3 Example encoding system 200 may include, in addition to Figure 2 and Figure 3 Those other than or replacing those shown Figure 2 and Figure 3 One or more of the elements, processes, and / or devices shown, and / or may include any or all of more than one of the elements, processes, and devices shown in the illustrations. As used herein, the phrase “communicate with”—including its variations—covers direct communication and / or indirect communication via one or more intermediate components, without requiring direct physical (e.g., wired) communication and / or continuous communication, but also including selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-off events.

[0103] exist Figure 10A , Figure 10B , Figure 11 and / or Figure 12 The diagram shows a representative used for implementation. Figure 2 and Figure 3The coding system 200 includes example hardware logic, machine-readable instructions, hardware implementation state machines, and / or any combination thereof, as shown in the flowcharts. Machine-readable instructions may be one or more executable programs or portions thereof for execution by a computer processor and / or processor circuitry, such as those described below. Figure 16 The processor 1612 shown in the example processor platform 1600 discussed herein. Programs may be embodied in software stored on non-transitory computer-readable storage media such as CD-ROM, floppy disk, hard disk, DVD, Blu-ray disc, or memory associated with the processor 1612, but the entire program and / or portions thereof may instead be executed by a device other than the processor 1612 and / or embodied in firmware or dedicated hardware. Furthermore, although this is a reference... Figure 10A , Figure 10B , Figure 11 and Figure 12 The flowchart shown is used to describe the example program, but many other methods for implementing the example coding system 200 may be used alternatively. For example, the execution order of the blocks may be changed, and / or some of the blocks described may be altered, eliminated, or combined. Additionally or alternatively, any or all blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. Processor circuitry may be distributed across different network locations and / or local to one or more devices (e.g., a multi-core processor in a single machine, multiple processors distributed across a server rack, etc.).

[0104] The machine-readable instructions described herein can be stored in one or more formats, including compressed formats, encrypted formats, segmented formats, compiled formats, executable formats, packaged formats, etc. The machine-readable instructions described herein can be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, machine-readable instructions can be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations within a network or set of networks (e.g., in the cloud, in edge devices, etc.). Machine-readable instructions can require installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., to make them directly readable, interpretable, and / or executable by computing devices and / or other machines. For example, machine-readable instructions can be stored as multiple parts that are individually compressed, encrypted, and stored on separate computing devices, wherein these parts, when decrypted, decompressed, and combined, form a set of executable instructions that implement one or more functions, which together form a program, such as that described herein.

[0105] In another example, machine-readable instructions may be stored in a state in which they can be read by processor circuitry, but require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., to execute these instructions on a specific computing device or other device. In another example, machine-readable instructions may need to be configured (e.g., storage settings, input data, recording network addresses, etc.) before they can be executed in whole or in part. Therefore, machine-readable media as used herein may include machine-readable instructions and / or (one or more) programs, regardless of the specific format or state of these machine-readable instructions and / or (one or more) programs at the time of storage or otherwise at rest or in transit.

[0106] The machine-readable instructions described in this article can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0107] As described above, executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transitory computer and / or machine-readable medium can be used to implement... Figure 10A , Figure 10B , Figure 11 and Figure 12 The example process, wherein the medium is, for example, a hard disk drive, flash memory, read-only memory, compact disk, digital multifunction disk, cache, random access memory, and / or any other storage device or disk in which information may be stored for any duration (e.g., long-term storage, permanent storage, transient storage, temporary buffering, and / or caching for information). For the purposes of this document, the term nontransitory computer-readable medium is explicitly defined as including any type of computer-readable storage device and / or disk, and excludes propagating signals and transmission media.

[0108] "Comprising" and "including" (and all its forms and tenses) are used herein as introductory terms. Thus, whenever a claim uses any form of "comprising" or "including" (e.g., including, comprising, having, etc.) as a preamble or in any kind of claim statement, it is understood that additional elements, terms, etc., may exist without falling outside the scope of the corresponding claim or statement. For the purposes of this document, when the phrase "at least" is used as a transitional term in, for example, the preamble of a claim, it is introductory, just as the terms "comprising" and "including" are introductory. The term "and / or," when used, for example, in the form of, say, A, B, and / or C, refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase “at least one of A or B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing the execution or operation of processes, instructions, actions, activities, and / or steps, the phrase “at least one of A and B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, for the purposes of this document in the context of describing the execution or operation of processes, instructions, actions, activities and / or steps, the phrase “at least one of A or B” is intended to refer to an implementation that includes any one of the following: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0109] As used herein, singular references (e.g., “a,” “first,” “second,” etc.) do not exclude pluralism. For the purposes of this document, the term “a” refers to one or more of the same entity. The terms “a,” “one or more,” and “at least one” may be used interchangeably herein. Furthermore, although listed separately, multiple means, elements, or method actions may be implemented by, for example, a single unit or processor. Moreover, while individual features may be included in different examples or claims, they may be combined, and inclusion in different examples or claims does not imply that the combination of features is infeasible and / or not advantageous.

[0110] Although Figure 7 and Figure 8 The diagram illustrates an example of implementing decoder 700, but... Figure 7 and Figure 8 One or more of the components, processes, and / or devices shown may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other way. Additionally, the example demultiplexer 702, example packetized video stream parser 704, example frame decoder 708, example frame unpacker 710, example reconstructor 712, example renderer 714, example metadata / SEI message extractor 716, and example metadata / SEI message decoder 718, example region counter 801, example variable value setter 802, example packetized frame unit retriever 804, example region type determiner 806, example region selector 807, example region extractor 808, example region height and region width incrementer 810, and example region positioning determiner 812, example mapper 814, and / or more generally, Figure 7 and Figure 8 The example decoder 700 can be implemented through hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, it may include an example demultiplexer 702, an example packetized video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, an example metadata / SEI message decoder 718, an example region counter 801, an example variable value setter 802, an example packetized frame unit retriever 804, an example region type determiner 806, an example region selector 807, an example region extractor 808, an example region height and region width incrementer 810, and a... Any of the example region localizer 812, example mapper 814, and / or more generally example decoder 700, may be implemented by one or more analog or digital circuits, logic circuits, one or more programmable processors, one or more programmable controllers, one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more programmable logic devices (PLDs), and / or one or more field-programmable logic devices (FPLDs).

[0111] When any device or system claim in this patent covers a purely software and / or purely firmware implementation, at least one of the following: example demultiplexer 702, example packetized video stream parser 704, example frame decoder 708, example frame unpacker 710, example reconstructor 712, example renderer 714, example metadata / SEI message extractor 716, example metadata / SEI message decoder 718, example region counter 801, example variable value setter 802, example packetized frame unit acquirer 804, example region type determiner 806, example region selector 807, example region extractor 808, example region height and region width incrementer 810, example region positioning determiner 812, and example mapper 814, is hereby explicitly defined as including a non-transitory computer-readable storage device or disk containing such software and / or firmware, such as a memory, digital versatile disc (DVD), compact disc (CD), Blu-ray disc, etc. Furthermore, Figure 7 and Figure 8 Example decoder 700 may include, in addition to Figure 7 and Figure 8 Those other than or replacing those shown Figure 7 and Figure 8 One or more of the elements, processes, and / or devices shown, and / or may include any or all of more than one of the illustrated elements, processes, and devices. As used herein, the phrase “communicating with”—including its variations—covers direct communication and / or indirect communication via one or more intermediate components, without requiring direct physical (e.g., wired) communication and / or continuous communication, but also including selective communication at periodic intervals, scheduled intervals, non-periodic intervals, and / or one-off events.

[0112] exist Figure 14 and Figure 15 The diagram shows a representative used for implementation. Figure 7 and Figure 8 The decoder 700 includes example hardware logic, machine-readable instructions, a hardware implementation state machine, and / or any combination thereof flowcharts. Machine-readable instructions may be one or more executable programs or portions thereof for execution by a computer processor and / or processor circuitry, such as those described below. Figure 17 The processor 1712 shown in the example processor platform 1700 discussed herein. Programs may be embodied in software stored on non-transitory computer-readable storage media such as CD-ROM, floppy disk, hard disk, DVD, Blu-ray disc, or memory associated with the processor 1712, but the entire program and / or portions thereof may instead be executed by a device other than the processor 1712 and / or embodied in firmware or dedicated hardware. Furthermore, although this is a reference... Figure 14and Figure 15 The flowchart shown is used to describe the example program, but many other methods for implementing the example decoder 700 may be used alternatively. For example, the execution order of the blocks may be changed, and / or some of the blocks described may be altered, eliminated, or combined. Additionally or alternatively, any or all blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) configured to perform the corresponding operations without executing software or firmware. Processor circuitry may be distributed across different network locations and / or local to one or more devices (e.g., a multi-core processor in a single machine, multiple processors distributed across a server rack, etc.).

[0113] The machine-readable instructions described herein can be stored in one or more formats, including compressed formats, encrypted formats, segmented formats, compiled formats, executable formats, packaged formats, etc. The machine-readable instructions described herein can be stored as data or data structures (e.g., portions of instructions, code, representations of code, etc.) that can be used to create, manufacture, and / or produce machine-executable instructions. For example, machine-readable instructions can be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located in the same or different locations within a network or set of networks (e.g., in the cloud, in edge devices, etc.). Machine-readable instructions can require installation, modification, adaptation, updating, combination, supplementation, configuration, decryption, decompression, unpacking, distribution, reassignment, compilation, etc., to make them directly readable, interpretable, and / or executable by computing devices and / or other machines. For example, machine-readable instructions can be stored as multiple parts that are individually compressed, encrypted, and stored on separate computing devices, wherein these parts, when decrypted, decompressed, and combined, form a set of executable instructions that implement one or more functions, which together form a program, such as that described herein.

[0114] In another example, machine-readable instructions may be stored in a state in which they can be read by processor circuitry, but require the addition of libraries (e.g., dynamic link libraries (DLLs)), software development kits (SDKs), application programming interfaces (APIs), etc., to execute these instructions on a specific computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., storage settings, input data, recording network addresses, etc.) before they can be executed in whole or in part. Therefore, the machine-readable medium used herein may include machine-readable instructions and / or (one or more) programs, regardless of the specific format or state of these machine-readable instructions and / or (one or more) programs at the time of storage or otherwise at rest or in transit.

[0115] The machine-readable instructions described in this article can be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, machine-readable instructions can be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, HyperText Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0116] As described above, executable instructions (e.g., computer and / or machine-readable instructions) stored on a non-transitory computer and / or machine-readable medium can be used to implement... Figure 14 and Figure 15 The example process, wherein the medium is, for example, a hard disk drive, flash memory, read-only memory, compact disk, digital multifunction disk, cache, random access memory, and / or any other storage device or disk in which information may be stored for any duration (e.g., long-term storage, permanent storage, transient storage, temporary buffering, and / or caching for information). For the purposes of this document, the term nontransitory computer-readable medium is explicitly defined as including any type of computer-readable storage device and / or disk, and excludes propagating signals and transmission media.

[0117] Figure 10A and Figure 10B The diagram illustrates what can be executed to achieve this. Figure 2The program of the example encoding system 200 is shown in Figure 10. The program includes block 1002, where captured video data and parameter / atlas data are stored in the example captured video data and metadata storage device 202. The example video and metadata preprocessor 204 separates the parameter / atlas data from the different types of video included in the video data (e.g., geometry, attributes, occupancy, etc.). (Block 1004) Next, the different types of video data are provided to the packed video frame packer 218 via video data providers 208, 210, and 212, and the parameter and atlas data are provided to the bitstream formatter 216 via the parameter and atlas data provider 206. (Block 1006)

[0118] Example video frame packer 218 determines the video frame packing configuration in part based on the type and size of the video data provided by video data providers 208, 210, 212, or based on any of a variety of other factors (block 1008). In some examples, as referenced... Figure 3 As described, the example packed video frame packer 218 determines the number of decodes required to decode the bitstream generated by the encoding system 200 (block 1010). The packed video frame packer 218 tests to determine whether the number of decodes will meet a threshold (e.g., be less than or equal to the maximum number of decodes) (block 1012). In some examples, when the maximum number of decodes is exceeded, the packed video frame packer 218 does not encode the video, and the procedure terminates.

[0119] When the maximum number of decodes is met (e.g., the number of decodes is less than or equal to the maximum number of decodes), the packet video frame packer 218 generates a packet video frame (block 1014).

[0120] In some examples, when generating packaged video frames (or before or after generation), example packaged video frame packer 218 ( Figure 2 Generate packed video frame information (block 1016). In some examples, the packed video frame includes the packed video frame itself, in addition to the data types included in the regions of the packed video frame. In some examples, the packed video frame information includes any of various information about the packed video frame, including: the regions included, the type of video components in each region, and the atlas or atlas tiles corresponding to each region. Bitstream formatter 216 ( Figure 2The parameter / atlas data (and any SEI messages applicable to the packaged video frames) is formatted (block 1018). In some examples, the parameter data may include parameter data and / or camera-related data corresponding to the camera views used to generate the video packaged video frames, such as identifying the number of cameras collecting data from them, the views associated with each camera, etc. In some examples, as described above, the packaged video frame information may include information about the packaged video frame as a whole, individual packaged video frames, individual regions included in each packaged video frame, the size of the regions, etc. The packaged video frames are encoded at encoder 220 (block 1020), and example multiplexer 222 operates to multiplex the bitstream information with the encoded packaged video frames (block 1022). The multiplexed bitstream is then transmitted to decoder (block 1024), and the procedure ends.

[0121] Figure 11 It is to achieve Figure 2 and Figure 3 Example packaged video frame packer 218 program 1100. Program 1100 begins at block 1102, where grade / level / hierarchy information is stored in grade / level / hierarchy storage. Next, example packaged video flag setter 304 sets video flags indicating that packaged video frame data is to be processed (block 1104). In some examples, setting packaged video flags includes setting one or more bits in metadata storage 318. In some examples, this is based on information in grade / level / hierarchy storage 302 or based on information provided by... Figure 2 Information provided by any other block allows the packaged video frame configuration selector 306 to determine the video frame configuration / layout (block 1106). In some examples, it can select... Figure 1A , Figure 1B and / or Figure 1C Any of the packaged video frame configurations shown. In some examples, it may be determined differently. Figure 1A , Figure 1B Any of the various configurations (or additional configurations besides those shown in Figure 1) and the configuration shown in Figure 1.

[0122] Next, the geometry region generator 308 generates geometry rects / regs, the attribute region generator 310 generates regions, and the example occupancy region generator 312 generates occupancy rects. (Block 1108). In some examples, the packed video frame generator 314 uses the selected packed video frame configuration and the generated regions to generate (or insert regions into one or more packed video frames according to the selected packed video frame configuration). (Block 1110). In some examples, the generated packed video frames identify the location of the regions in the volumetric video being encoded and associate that location (and other information) with the position of the volumetric video in the bitstream. Furthermore, as described above, the packed video frame information generator 316 generates packed video frame information about each packed video frame created by the packed video frame generator 314. (Block 1112). The packed video frame information can be stored in the metadata storage device 318. In some examples, the metadata provider 324 provides metadata (packed video frame information, parameter / atlas data, etc.) corresponding to the packed video frame (or multiple packed video frames) to... Figure 2 Example bitstream formatter 216. (Block 1114). In some examples, packetized video frames are appended or otherwise associated with the corresponding packetized video frames to be encoded. In some examples, example packetized video frame provider 326 provides packetized video frames (obtained from packetized storage device 322) to... Figure 2 Example encoding system 200, where packetized video frames are encoded (block 1116). Furthermore, packetized video frame information can be associated with the encoded packetized frames. Additionally, an example multiplexer multiplexes the metadata bitstream with the corresponding encoded data for delivery to the decoder for decoding and rendering (block 1118), and the multiplexed stream is transmitted to the decoder (block 1120). Thereafter, program 1100 is repeated or terminated.

[0123] In some examples, the packetized video frame generator 314 inserts regions simultaneously into different packetized video frames, thereby generating multiple packetized video frames at the same time (or within the same time period of each other). In some examples, the packetized video frame information generator 316 examines the packetized video frames stored in the packetized video frame storage device 322 to determine packetized video frame information, and in some examples, the packetized video frame generator 314 provides one or more types of packetized video frame information to the packetized video frame information generator 316. In some examples, the packetized video frame information generator 316 determines the order in which the packetized video frame information is arranged according to the video coding standard.

[0124] Figure 12 It is to achieve Figure 3 Example decoder verifier 328 program 1200. Figure 10A and Figure 10B Example program 1000 illustrates how the decoder verifier 328 operates within the context of the operation of the encoding system 200. For clarity, the operation of the decoder verifier 328 is not reflected in... Figure 11 In program 1100, and the specific operation of decoder verifier 328 is not reflected in Figure 10. Instead, the operation of decoder verifier 328 is referenced... Figure 12 The program is described using 1200. (See reference.) Figure 10A and Figure 10B To illustrate the broader context of how example program 1200 operates to execute blocks 1010 and 1012.

[0125] This example program may include block 1202, at which point, Figure 3Example flag setter 333 sets flags in example flag storage device 332. Each flag corresponds to a different type of video data, and each flag indicates whether the corresponding type of video data exists in the corresponding packaged video frame video unit. Different types of video data include auxiliary video data, occupancy video data, geometric video data, and attribute video data. In some examples, attribute data may include subcategories of attribute data, and the variable "ai_attribute_count" is used to represent the number of subcategories included in the attribute video data. In addition, example flag adder 324 sums the number of flags associated with each atlas of packaged video frame units to determine the total number of types of video data included in the packaged video frame unit (block 1204). The sum of the flags is the number of decoders that the decoder receiving the encoded video can decode, and is equal to the variable "NumDecodes". When the number of flags for all atlases has been summed, example packaged video checker 336 checks whether the packaged video frame flags indicate that the packaged video frame unit includes a packaged video frame (block 1206). If the packaged video frame flags do not indicate that the packaged video frame unit includes a packaged video frame, then program 1200 ends. If the packetized video frame video flag indicates that the packetized video frame video unit includes packetized video frames, then for each packetized video frame, the example region subtractor 338 subtracts the number of regions included in each packetized video frame from the value of NumDecodes until all regions of all packetized video frames have been subtracted (block 1208). The number of regions included in each packetized video frame minus 1 can be represented by the variable "pi_num_rect[k]-1", where the variable "k" represents the number of packetized video frames included in the total number of packetized video frames. Next, the example comparator 340 compares the value of NumDecodes with a value corresponding to the maximum number of decoders that the receiving decoder can decode (block 1210). The maximum number of decoders is represented by the variable "MaxDecodes" (also known as the threshold). In some examples, the frame packer 218 is pre-programmed with information that identifies the corresponding MaxDecodes value for the various types of decoders to which the video stream may be transmitted.

[0126] If the maximum number of decodes is exceeded (also known as the threshold) (e.g., the threshold is not met), as determined at block 1212, the corresponding video is not encoded, and program 1200 ends.

[0127] If the threshold is met (as determined at block 1212), the encoder continues generating the packed video frames (block 1214), and program 1200 ends. In some examples, Figure 13 The pseudocode can be used to implement program 1200 or replace it. Figure 12 Program 1200. Figure 12The purpose of procedure 1200 is to determine the number of video decoders required for bitstream decoding. For all atlases, if a specific component type exists, it is added. Then, if packetization is used, the number of regions minus 1 can be subtracted from the total number of decoders, since the entire packetized frame only requires one decoder, regardless of how many regions it contains.

[0128] Figure 14 It is to achieve Figure 6 Example of a packaged video frame decoder 600 is program 1400. In some examples, program 1400 includes block 1402, at which... Figure 7 Example demultiplexer 702 receives and demultiplexes data by... Figure 2 The encoding system 200 generates an encoded video bitstream (block 1402). Furthermore, in block 1402, demultiplexer 702 separates the metadata / SEI message from the encoded video sub-bitstream and provides the metadata / SEI message to the example metadata / SEI message extractor 716. Additionally, demultiplexer 702 provides the encoded video sub-bitstream and the packetized video frame information corresponding to the metadata / SEI message to the example video stream parser 704.

[0129] In some examples, the example video stream parser 704 uses information provided by the example metadata / SEI message decoder 718 to identify units of the sub-bitstream corresponding to the packed video frames and to identify information corresponding to the packed video frames. Furthermore, the video stream parser 704 parses the packed video frame information from the encoded packed video sub-bitstream received from the autoencoder (block 1404). In some examples, the information provided by the metadata / SEI message decoder 718 for each packed video frame includes a color component index, bit depth, packing width, and packing height. Additionally, the metadata / SEI message decoder 718 provides the number of spectral streams associated with the packed video frames and, if necessary, obtains a spectral count associated with one or more packed video frames currently being operated on.

[0130] In some examples, example decoding process 1200 includes using a video stream parser 704 to parse the packed video frame information from the bitstream (block 1404). In some examples, the video stream parser associates the packed video frame information with a set of variables describing the packed video frame, including: 1) "decPackFrame[frameIdx][compIdx][y][x]", 2) "decPackBitdepth[frameIdx]", 3) "decPackWidth[frameIdx]", 4) "decPackHeight[frameIdx]", and 5) "mapIdx". The value of the variable "frameIdx" is the index of the decoded packed video frame, and the value of the variable "compIdx" is the color component index. The variable "y" is the row index in the packed video frame and falls within the range of 0 to the value represented by the variable "decPackHeight[frameIdx]-1", inclusive. The variable "x" is the column index in the packed video frame and falls within the range of 0 to "decPackWidth[frameIdx]-1", inclusive. The map index ("mapIdx") falls within the range that starts at zero and extends to the value evaluated by the statement: "vps_multiple_map_streams_present_flag?vps_map_count_minus1:0", inclusive. In some examples, the above statement evaluates to the value of variable "vps_map_count_minus1" when the value of variable "vps_multiple_map_streams_present_flag" is true (or not equal to zero), otherwise it evaluates to zero. The variable “vps_map_count_minus1” has the following value: it represents the value of the map included in or otherwise associated with the currently being processed packaged video frame.

[0131] The decoded packetized video frames are provided to the example frame decoder 708. The decoded packetized video information is also provided to the frame parser. The frame decoder 708 performs any decoding process on the packetized video frame stream according to any video decoding standard and provides the resulting decoded(one or more) packetized video frame units to the example frame unpacker 710 (block 1406). The packetized video frame unpacker 710 uses the packetized video frame information associated with the decoded packetized video frames, variables recognized by the video stream parser 704, etc., to unpack the packetized video frame units, thereby unpacking the decoded packetized video frames (block 1408).

[0132] The example frame mapper uses the decoded packed video stream and unpacked frame information to map regions of the decoded packed video stream to a volumetric video sequence (block 1410). The mapped video is then fed to the example reconstructor 712 (see [link]). Figure 7 The reconstructor uses the stream of information to perform any additional processing required to reconstruct the 3D image (block 1412) for display at the example renderer (block 1414).

[0133] Figure 15 It can be used to implement Figure 8 The example frame unpacker program 1500 is shown. In some examples, program 1500 begins at block 1502, where the metadata / SEI message decoder 718 decodes the metadata (including packed video frame information) and SEI messages generated by the frame packer of the example encoder. The metadata / SEI messages may include packed video frame information associated with one or more packed video frame units to be unpacked, atlas data headers corresponding to the packed video frame units, decoded patch units of the packed video frame units, decoded block-to-patch maps, etc. The information from the metadata / SEI message decoder 718 may be provided to the frame unpacker as needed or upon request from the frame unpacker.

[0134] In some examples, the example packed video frame unit retriever 704 receives and / or retrieves (from frame decoder 708) the packed video frame units to be processed (block 1504). In some examples, for each region of each packed video frame, information describing each region is collected from the metadata / SEI message decoder, as described above for each region. Figure 7 As described. (Block 1508). For each block of each rect, determine the position and obtain the corresponding information, as described above for... Figure 7 As described above. (Block 1510). The block location, corresponding block information, and raw data are provided to the reconstructor so that, as described above... Figure 7 The image is placed in the image. (Block 1512). Repeat blocks 1508, 1510, and 1512 until all blocks in all regions of all packed video frames have been processed. (Block 1514). After this, program 1500 ends. In some examples, it can be used... Figure 9 The pseudocode is used to implement program 1500.

[0135] Figure 16 It is constructed to execute Figure 10A , Figure 10B , Figure 11 , Figure 12 Instructions to achieve Figure 2 and Figure 3A block diagram of an example processor platform 1600 for an encoder. Processor platform 1600 can be, for example, a server, personal computer, workstation, self-learning machine (e.g., neural network), mobile device (e.g., cellular phone, smartphone, such as iPad). TM Tablet devices, personal digital assistants (PDAs), internet-connected appliances, DVD players, CD players, digital video recorders, Blu-ray players, game consoles, personal video recorders, set-top boxes, headphones or other wearable devices, or any other type of computing device.

[0136] The illustrated processor platform 1600 includes a processor 1612. The illustrated processor 1612 is hardware. For example, the processor 1612 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements an example video and metadata preprocessor 204, an example parameter / atlas data provider 206, an example geometry video data provider 208, an example attribute video data provider 210, an example occupancy video data provider 212, an example bitstream formatter 216, an example frame packer 218, an example encoder 220, an example multiplexer 222, an example packed video frame video flag setter 304, an example packed video frame configuration selector 306, an example atlas recognizer 307, and an example geometry region generator. 308, Example Attribute Region Generator 310, Example Occupied Region Generator 312, Example Packed Video Frame Generator 314, Example Packed Video Frame Information Generator 316, Example Metadata Provider 324, Example Packed Video Frame Provider 326, Example Decoder Verifier 328, Example Evaluator 330, Example Flag Setter 333, Example Flag Adder 334, Example Packed Video Frame Video Inspector 336, Example Region Subtraction Unit 338, and Example Comparator 340, and / or more generally, Example Encoding System 200.

[0137] The illustrated processor 1612 includes local memory 1613 (e.g., cache). The illustrated processor 1612 communicates via bus 1618 with main memory, which includes volatile memory 1614 and non-volatile memory 1616. The volatile memory 1614 may be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, It can be implemented using flash memory and / or any other type of random access memory device. The non-volatile memory 1616 can be implemented using flash memory and / or any other desired type of memory device. Access to the main memory 1614, 1616 is controlled by the memory controller.

[0138] The processor platform 1600 illustrated also includes interface circuitry 1620. Interface circuitry 1620 can be implemented using any type of interface standard, such as an Ethernet interface, Universal Serial Bus (USB), etc. Interfaces include near field communication (NFC) interfaces and / or PCI fast interfaces.

[0139] In the illustrated example, one or more input devices 1622 are connected to interface circuitry 1620. The input devices 1622 allow users to input data and / or commands into processor 1612. The input devices may be implemented as, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, touchpads, trackballs, isopoints, and / or voice recognition systems.

[0140] One or more output devices 1624 are also connected to the interface circuitry 1620 illustrated in the figure. The output devices 1624 may be implemented, for example, by display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube displays (CRTs), in-place switching (IPS) displays, touchscreens, etc.), haptic output devices, virtual reality headsets, etc. The interface circuitry 1620 illustrated in the figure thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0141] The interface circuit 1620 illustrated also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any type of computing device) via network 1626. Communication may be via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-to-line wireless systems, cellular telephone systems, and so on.

[0142] The illustrated processor platform 1600 also includes one or more mass storage devices 1628 for storing software and / or data. Examples of such mass storage devices 1628 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0143] Figure 10A , Figure 10B , Figure 11 and Figure 12 The machine-executable instructions 1632 may be stored in mass storage device 1628, volatile memory 1614, non-volatile memory 1616, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0144] Figure 17 It is constructed to execute Figure 14 and Figure 15 Instructions to achieve Figure 7 and Figure 8 A block diagram of an example processor platform 1700 for the decoder. Processor platform 1700 can be, for example, a server, personal computer, workstation, self-learning machine (e.g., neural network), mobile device (e.g., cellular phone, smartphone, such as iPad). TM Tablet devices, personal digital assistants (PDAs), internet-connected appliances, DVD players, CD players, digital video recorders, Blu-ray players, game consoles, personal video recorders, set-top boxes, headphones or other wearable devices, virtual reality systems, or any other type of computing device.

[0145] The illustrated processor platform 1700 includes a processor 1712. The illustrated processor 1712 is hardware. For example, the processor 1712 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements an example demultiplexer 702, an example packetized video frame video stream parser 704, an example frame decoder 708, an example frame unpacker 710, an example reconstructor 712, an example renderer 714, an example metadata / SEI message extractor 716, and an example metadata / SEI message decoder 718, an example region counter 801, an example variable value setter 802, an example packetized video frame unit retriever 804, an example region type determiner 806, an example region selector 807, an example region extractor 808, an example region height and region width incrementer 810, an example region positioning determiner 812, and an example mapper 814.

[0146] The illustrated processor 1712 includes local memory 1713 (e.g., cache). The illustrated processor 1712 communicates via bus 1718 with main memory, which includes volatile memory 1714 and non-volatile memory 1716. The volatile memory 1714 may be Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), etc. Dynamic Random Access Memory (DRAM) Dynamic Random Access Memory, It can be implemented using flash memory and / or any other type of random access memory device. The non-volatile memory 1716 can be implemented using flash memory and / or any other desired type of memory device. Access to the main memory 1714, 1716 is controlled by the memory controller.

[0147] The processor platform 1700 illustrated also includes interface circuitry 1720. Interface circuitry 1720 can be implemented using any type of interface standard, such as an Ethernet interface, Universal Serial Bus (USB), etc. Interfaces include near field communication (NFC) interfaces and / or PCI fast interfaces.

[0148] In the illustrated example, one or more input devices 1722 are connected to interface circuitry 1720. The input devices 1722 allow users to input data and / or commands into processor 1712. The input devices may be implemented as, for example, audio sensors, microphones, cameras (still or video), keyboards, buttons, mice, touchscreens, touchpads, trackballs, isopoints, and / or voice recognition systems.

[0149] One or more output devices 1724 are also connected to the interface circuitry 1720 of the illustrated example. The output devices 1724 may be implemented, for example, by display devices (e.g., light-emitting diodes (LEDs), organic light-emitting diodes (OLEDs), liquid crystal displays (LCDs), cathode ray tube displays (CRTs), in-place switching (IPS) displays, touchscreens, etc.), haptic output devices, virtual reality headsets, etc. The interface circuitry 1720 of the illustrated example thus typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0150] The interface circuit 1720 illustrated also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces, to facilitate data exchange with external machines (e.g., any kind of computing device) via network 1726. Communication may be via, for example, Ethernet connections, digital subscriber line (DSL) connections, telephone line connections, coaxial cable systems, satellite systems, line-to-line wireless systems, cellular telephone systems, and so on.

[0151] The illustrated processor platform 1700 also includes one or more mass storage devices 1728 for storing software and / or data. Examples of such mass storage devices 1728 include floppy disk drives, hard disk drives, compact disk drives, Blu-ray disc drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0152] Figure 14 and Figure 15The machine-executable instructions 1732 may be stored in mass storage device 1728, volatile memory 1714, non-volatile memory 1716, and / or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

[0153] As will be clear from the foregoing, example methods, apparatuses, and articles of art for generating data-packed video frames for immersive video bitstreams have been disclosed. The disclosed methods, apparatuses, and articles of art improve the efficiency of computing devices by expanding the pool of devices capable of decoding video frames by reducing the number of decoders required to decode the video frames. Furthermore, the methods, apparatuses, and articles of art disclosed herein also allow for the inclusion of different components of the same or different atlases within the same "packed video frame," and these components can be encoded at different frame rates (e.g., a packed video frame of a texture atlas can be sent at 60 fps, while a packed video frame of a geometry atlas can be sent at 30 fps) to achieve further compression or, depending on the desired features supported by the application. Moreover, the present invention allows for the operation of volumetric video data and / or immersive data using standard encoders and decoders without requiring modifications to such encoders or decoders. The disclosed methods, apparatuses, and articles of art are therefore directed toward one or more improvements to the functionality of a computer.

[0154] This paper discloses example methods, apparatuses, systems, and artifacts for generating packaged video frames for volumetric video bitstreams and immersive video bitstreams.

[0155] Further examples and combinations thereof include the following:

[0156] Example 1 includes a volumetric video coding system comprising: a configuration determiner for creating a packaged video frame layout, the packaged video frame layout including regions into which video components will be placed; and a packaged video frame generator for forming packaged video frames, the packaged video frames including video components placed in different regions of the packaged video frame layout. It also includes a packaged video frame information generator for generating packaged video frame information that identifies characteristics of the packaged video frames, the characteristics including at least one of: (i) the identity of the regions included in the packaged video frame layout, (ii) the types of video components included in each region, or (iii) information identifying the location and size of the regions. The volumetric video coding system further includes a video encoder for encoding the packaged video frames in an encoded bitstream. The encoded bitstream includes the packaged video frame information, and the packaged video frame information indicates that the encoded bitstream includes the packaged video frames.

[0157] Example 2 includes a volumetric video coding system as described in Example 1, wherein the video component type includes at least one of the following: auxiliary data, attribute data, geometric data, or occupancy data.

[0158] Example 3 includes a volumetric video coding system as described in Example 1, wherein the regions of the packaged video frames do not overlap, and the packaged video frame information further includes information identifying the types of video components associated with the respective regions, and at least one of an atlas identifier or an atlas tile identifier.

[0159] Example 4 includes a volumetric video coding system as described in Example 1, wherein the volumetric video data in the same packed video frame has the same sampling format and bit depth.

[0160] Example 5 includes a volumetric video coding system as described in Example 1, and further includes a decoder number verifier to verify that there is no violation of the constraint on the number of decoders to be used for decoding the bitstream.

[0161] Example 6 includes a volumetric video coding system as described in Example 1, further comprising: an independently decodeable region identifier for identifying (i) regions of the packaged video frames that can be independently decoded and (ii) corresponding region identifiers; a message generator for generating a message including the region identifiers and indicating that the regions corresponding to the region identifiers can be independently decoded; and a multiplexer for multiplexing the message with the encoded bitstream for transmission to a decoder.

[0162] Example 7 includes one or more non-transitory computer-readable media having instructions that, when executed by at least one processor, cause the at least one processor to create at least a packaged video frame layout. The packaged video frame layout includes regions into which video components will be placed. Furthermore, the processor forms packaged video frames, each including video components placed in a different region of the packaged video frame layout, and generates packaged video frame information that identifies characteristics of the packaged video frames, including at least one of: (i) the identity of the regions included in the packaged video frame layout, (ii) the types of video components included in each region, or (iii) information identifying the location and size of the regions. The processor also encodes the packaged video frames in an encoded bitstream including the packaged video frame information. The packaged video frame information indicates that the encoded bitstream includes the packaged video frames.

[0163] Example 8 includes one or more non-transitory computer-readable media as described in Example 6, wherein the video component type includes at least one of the following: auxiliary data, attribute data, geometric data, or occupancy data.

[0164] Example 9 includes one or more non-transitory computer-readable media as described in Example 6, wherein the regions of the packaged video frames do not overlap, and the packaged video frame information further includes information identifying the types of video components associated with the respective regions, and at least one of atlas identifiers or atlas tile identifiers.

[0165] Example 10 includes one or more non-transitory computer-readable media as described in Example 6, wherein volumetric video data placed in the same packaged video frame has the same sampling format and bit depth.

[0166] Example 11 includes one or more non-transitory computer-readable media as described in Example 6, and also includes a decoder number verifier to verify that there is no violation of the constraint on the number of decoders to be used for decoding the bitstream.

[0167] Example 12 includes one or more non-transitory computer-readable media as described in Example 6, wherein, when executed, the instructions further cause the processor to identify (i) regions of the packaged video frames that can be independently decoded and (ii) corresponding region identifiers, and generate a message including the region identifiers and indicating that the regions corresponding to the region identifiers can be independently decoded. The instructions also cause the processor to multiplex the message with the encoded bitstream for transmission to the decoder.

[0168] Example 13 includes one or more non-transient media as described in Example 6, wherein, when the instructions are executed, the processor identifies regions of the packaged video frames that can be independently decoded, generates a message identifying regions of the packaged video frames to be independently decoded, and multiplexes the message with the encoded bitstream for transmission to the decoder.

[0169] Example 15 includes a volumetric video decoding system having a parser for extracting packed video frame information from a sub-bitstream containing packed video frames. The packed video frame information includes (i) a packed video frame layout, (ii) a region identifier that uniquely identifies a region of the packed video frame layout, and (iii) region location and size information. It also includes: a video decoder for decoding the packed video frames of the sub-bitstream; and a mapper for mapping regions of the packed video frames to at least a portion of a volumetric video sequence to be rendered, based on the packed video frame information.

[0170] Example 16 includes a volumetric video decoding system as described in Example 15, wherein each region in the region includes a video component type, the video component type including at least one of the following: auxiliary data, attribute data, geometric data, or occupancy data.

[0171] Example 17 includes a volumetric video decoding system as described in Example 15, wherein the regions of the packaged video frames do not overlap, and the packaged video frame information further includes information identifying the types of video components associated with the respective regions, and at least one of an atlas identifier or an atlas tile identifier.

[0172] Example 18 includes a volumetric video decoding system as described in Example 15, wherein the parser is further configured to extract messages from the sub-bitstream, the messages including supplemental enhancement information with region identifiers, the regions corresponding to the region identifiers being encoded independently.

[0173] Example 19 includes a volumetric video decoding system as described in Example 18, wherein the size and location of the regions to be encoded independently are determined based on tile indexes included in the packaged video frame information.

[0174] Example 20 includes a volumetric video decoding system as described in Example 18, wherein the size and location of the regions to be encoded independently are determined based on sub-picture identifiers included in the packaged video frame information.

[0175] Example 21 includes a volumetric video decoding system having a video bitstream parser to parse a parameter set associated with a video bitstream. The parameter set indicates the number of video decoders to be used for decoding the bitstream. A decoder verifier determines, based on a grade, level, and tier associated with the volumetric video decoding system, whether a plurality of video decoders in the volumetric video decoding system are greater than or equal to the number of decoders indicated by the parameter set. Furthermore, a decoding controller stops decoding of the sub-bitstream when the plurality of video decoders is less than the number of decoders indicated by the parameter set, and continues decoding of the sub-bitstream when the plurality of video decoders is greater than or equal to the number of decoders indicated by the parameter set.

[0176] Example 22 includes a method for decoding volumetric video, the method comprising: extracting packed video frame information from a sub-bitstream containing packed video frames, the packed video frame information including (i) a packed video frame layout, (ii) a region identifier that uniquely identifies a region of the packed video frame layout, and (iii) region location and size information; decoding the packed video frames of the sub-bitstream; and mapping regions of the packed video frames to at least a portion of a volumetric video sequence to be rendered based on the packed video frame information.

[0177] Example 23 includes a volumetric video decoding method as described in Example 22, wherein each region in the region includes a respective video component type, and the video component type includes at least one of the following: auxiliary data, attribute data, geometric data, or occupancy data.

[0178] Example 24 includes a volumetric video decoding method as described in Example 22, wherein the regions of the packaged video frames do not overlap, and the packaged video frame information further includes information identifying the types of video components associated with the respective regions, and at least one of an atlas identifier or an atlas tile identifier.

[0179] Example 25 includes a volumetric video decoding method as described in Example 22, wherein the parser is further configured to extract messages from the sub-bitstream, the messages including supplemental enhancement information with region identifiers, the regions corresponding to the region identifiers being independently encoded.

[0180] Example 26 includes a volumetric video decoding method as described in Example 22, wherein the size and location of the regions to be encoded independently are determined based on tile indexes included in the packaged video frame information.

[0181] Example 27 includes a volumetric video decoding method as described in Example 22, wherein the size and position of the region to be encoded independently are determined based on sub-picture identifiers included in the packaged video frame information.

[0182] Example 28 is any one of Examples 2-6, which is included in Example 1.

[0183] Example 29 is any one of Examples 8-13, which is Example 7.

[0184] Example 30 is any one of Examples 16-17, 19, and 20, which are included in Example 15.

[0185] Example 31 includes Example 21 and also includes any one of Examples 22-27.

[0186] Example 32 includes one or more non-transitory media, the media including instructions that, when executed, cause a processor to perform the method described in any of Examples 22-27.

[0187] While certain example methods, apparatuses, and articles of manufacture are disclosed herein, the scope of this patent is not limited thereto. Rather, this patent covers all methods, apparatuses, and articles of manufacture that fairly fall within the scope of the claims of this patent.

[0188] The appended claims are hereby incorporated by reference into this “Detailed Description” section, wherein each claim is an independent embodiment of this disclosure.

Claims

1. At least one non-transitory computer-readable medium comprising instructions to cause at least one processor circuit to perform at least the following: generate a single packed video frame comprising a first region comprising a texture attribute component associated with a first atlas and a second region comprising at least one of a geometry component or an occupancy component associated with the first atlas; generate frame packing information identifying characteristics of the packed video frame; and encode the packed video frame and the frame packing information in a bitstream.

2. The at least one non-transitory computer-readable medium of claim 1, wherein the instructions are to cause one or more of the at least one processor circuit to generate the single packed video frame comprising the first region comprising the texture attribute component associated with the first atlas, the second region comprising the geometry component associated with the first atlas, and a third region comprising the occupancy component associated with the first atlas.

3. The at least one non-transitory computer-readable medium of claim 1, wherein the regions of the packed video frame are non-overlapping.

4. The at least one non-transitory computer-readable medium of claim 1, wherein the texture attribute component and the at least one of the geometry component or the occupancy component have a same bit depth.

5. The at least one non-transitory computer-readable medium of claim 4, wherein the frame packing information is to specify the bit depth.

6. The at least one non-transitory computer-readable medium of claim 1, wherein the frame packing information comprises a first type value to identify the texture attribute component and a second type value to identify the geometry component.

7. The at least one non-transitory computer-readable medium of claim 1, wherein the bitstream is an immersive video bitstream.

8. A video encoder comprising: an interface circuit; machine-readable instructions; and at least one processor circuit programmed based on the machine-readable instructions to: generate a single packed video frame comprising a first region comprising a texture attribute component associated with a first atlas and a second region comprising at least one of a geometry component or an occupancy component associated with the first atlas; generate frame packing information identifying characteristics of the packed video frame; and encode the packed video frame and the frame packing information in a bitstream.

9. The video encoder of claim 8, wherein one or more of the at least one processor circuit is to generate the single packed video frame comprising the first region comprising the texture attribute component associated with the first atlas, the second region comprising the geometry component associated with the first atlas, and a third region comprising the occupancy component associated with the first atlas. ​ 10. The video encoder of claim 8, wherein the regions of the packed video frame are non-overlapping.

11. The video encoder of claim 8, wherein the texture attribute component and the at least one of the geometry component or the occupancy component have a same bit depth.

12. The video encoder of claim 11, wherein the frame packing information is to specify the bit depth.

13. The video encoder of claim 8, wherein the frame packing information includes a first type value to identify the texture attribute component and a second type value to identify the geometry component.

14. The video encoder of claim 8, wherein the bitstream is an immersive video bitstream.

15. At least one non-transitory computer-readable medium comprising instructions to cause at least one processor circuit to perform at least the following: extract frame packing information from a bitstream comprising a packed video frame, the frame packing information identifying characteristics of the packed video frame; decode regions of the packed video frame based on the frame packing information, the regions including a first region comprising a texture attribute component associated with a first atlas and a second region comprising at least one of a geometry component or an occupancy component associated with the first atlas; and map the regions of the packed video frame to at least a portion of a volumetric video sequence to be rendered based on the frame packing information.

16. The at least one non-transitory computer-readable medium of claim 15, wherein the regions include the first region, the second region, and a third region, the first region comprising the texture attribute component associated with the first atlas, the second region comprising the geometry component associated with the first atlas, and the third region comprising the occupancy component associated with the first atlas.

17. The at least one non-transitory computer-readable medium of claim 15, wherein the regions of the packed video frame are non-overlapping.

18. The at least one non-transitory computer-readable medium of claim 15, wherein the texture attribute component and the at least one of the geometry component or the occupancy component have a same bit depth.

19. The at least one non-transitory computer-readable medium of claim 18, wherein the frame packing information is to specify the bit depth.

20. The at least one non-transitory computer-readable medium of claim 15, wherein the frame packing information includes a first type value to identify the texture attribute component and a second type value to identify the geometry component.

21. A method comprising: generating a single packed video frame comprising a first region comprising a texture attribute component associated with a first atlas and a second region comprising at least one of a geometry component or an occupancy component associated with the first atlas; generating frame packing information identifying attributes of the packed video frame; and encoding the packed video frame and the frame packing information in a bitstream. ​ 22. The method of claim 21, further comprising generating the single packed video frame comprising the first region, the second region, and a third region, the first region comprising the texture attribute component associated with the first atlas, the second region comprising the geometry component associated with the first atlas, and the third region comprising the occupancy component associated with the first atlas.

23. The method of claim 21, wherein the regions of the packed video frame are non- overlapping.

24. The method of claim 21, wherein the texture attribute component and the at least one of the geometry component or occupancy component have a same bit depth.

25. The method of claim 24, wherein the frame packing information is to specify the bit depth.

26. The method of claim 21, wherein the frame packing information comprises a first type value to identify the texture attribute component and a second type value to identify the geometry component.

27. The method of claim 21, wherein the bitstream is an immersive video bitstream.

28. One or more computer-readable media storing instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any of claims 21-27.

29. An apparatus comprising means for performing the method of any of claims 21-27.

30. A computer program product comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any of claims 21-27.

Images