Methods, apparatus, and computer programs for instance-based mesh coding

Abstract

A method and apparatus comprising computer code configured to cause one or more processors to obtain an input mesh corresponding to volumetric data of at least one three-dimensional (3D) visual content, partition the input mesh into a plurality of sub-meshes, each of the sub-meshes including a respective one of the instances of an object, determine a similarity between at least two of the instances of the object, group the sub-meshes based on the determined similarity, and code the volumetric data based on the grouping of the sub-meshes.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 392,409, filed on 26 July 2022, and U.S. Patent Application No. 18 / 312,338, filed on 4 May 2023, the disclosures thereof being incorporated herein by reference in their entirety.

[0002] This disclosure is directed towards a set of advanced video coding techniques, including both reversible and reversible mesh coding techniques based on instances of meshes. [Background technology]

[0003] Advances in 3D capture, modeling, and rendering are driving the universal presence of 3D content across multiple platforms and devices. Today, it may be possible to capture a baby's first steps on one continent and allow grandparents on another continent to see (and sometimes even interact with) that child, sharing a truly immersive experience together. Nevertheless, to achieve such reality, the models have become more sophisticated than ever before, and a considerable amount of data is tied to the creation and consumption of those models.

[0004] VMesh is an ongoing MPEG standard for compressing static and dynamic meshes. VMesh separates the input mesh into a simplified base mesh and a residual mesh. The base mesh can be encoded in high quality, while the remaining mesh can be encoded using refined surface fitting and displacement encoding to take advantage of local properties.

[0005] However, complex meshes often contain information about multiple instances in order to associate texture maps. This information is available during encoding. On the other hand, a mesh can be segmented into several parts based on its characteristics. For example, the face region of a human mesh will have more polygons.

[0006] Thus, a constant quantization step size applied to all instances, objects, and parts within a mesh results in large quantization errors, mesh regions may not be equally important, the number of faces may vary significantly in different parts of the mesh, and the base mesh may be simpler than the original mesh and displacement, thus requiring less precision in bit depth. Therefore, for any of these reasons, a technical solution to such problems arising in video coding techniques is desired. [Overview of the project] [Means for solving the problem]

[0007] The invention includes a method and apparatus comprising a memory configured to store computer program code, and one or more processors configured to access the computer program code and to operate as instructed by the computer program code. The computer program comprises acquisition code configured to cause at least one processor to acquire an input mesh corresponding to volumetric data of at least one three-dimensional (3D) visual content, and partitioning code configured to cause at least one processor to partition the input mesh corresponding to the volumetric data into a plurality of submeshes, each of which includes one instance of an object, and is configured to cause at least one processor to execute partitioning code, determination code configured to cause at least one processor to determine the similarity between at least two of the instances of the object, grouping code configured to cause at least one processor to group the submeshes based on the determined similarity, and coding code configured to cause at least one processor to code volumetric data based on the grouping of the plurality of submeshes.

[0008] According to an exemplary embodiment, determining similarity may involve comparing at least two bounding boxes of a submesh.

[0009] According to an exemplary embodiment, comparing bounding boxes may include comparing the ratio of the dimensions of the bounding boxes.

[0010] According to an exemplary embodiment, coding volumetric data may include signaling each of at least two submeshes with a different codec.

[0011] According to an exemplary embodiment, coding volumetric data may include signaling a prediction of a first one of at least two of the sub-meshes based on a prediction of a second one of at least two of the sub-meshes.

[0012] According to an exemplary embodiment, coding volumetric data may include signaling only a single prediction index for an entire first one of at least two of the sub-meshes.

[0013] According to an exemplary embodiment, coding volumetric data may include signaling only a plurality of prediction indexes for a first one of at least two of the sub-meshes.

[0014] According to an exemplary embodiment, a first one of the plurality of prediction indexes may be based on a first prediction index, a second one of the plurality of prediction indexes may be based on a second prediction index, the first prediction index may be based on a first other prediction index that is a prediction index of a second one of at least two sub-meshes, and a second one of the plurality of prediction indexes is based on a second other prediction index that is a prediction index of a third one of at least two sub-meshes.

[0015] According to an exemplary embodiment, signaling a prediction of a first one of at least two of the sub-meshes may include signaling a first prediction index and a second prediction index, the first prediction index may be a single index collectively set for each of a plurality of vertices of a first one of at least two of the sub-meshes, and the second prediction index may be set for at least one other vertex of a first one of at least two of the sub-meshes.

[0016] According to an exemplary embodiment, at least one of the first and second prediction indices may be based on a prediction index set at at least one vertex of the second of at least two submeshes.

[0017] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0018] [Figure 1] This is a schematic diagram of an embodiment. [Figure 2] This is a simplified block diagram of an embodiment. [Figure 3] This is a simplified diagram of an embodiment. [Figure 4] This is a simplified diagram of an embodiment. [Figure 5] This is a simplified diagram of an embodiment. [Figure 6] This is a simplified diagram of an embodiment. [Figure 7] This is a simplified diagram of an embodiment. [Figure 8] This is a simplified diagram of an embodiment. [Figure 9] This is a simplified diagram of an embodiment. [Figure 10] This is a simplified diagram of an embodiment. [Figure 11] This is a simplified diagram of an embodiment. [Figure 12] This is a simplified diagram of an embodiment. [Figure 13] This is a simplified flowchart of an embodiment. [Figure 14] This is a simplified flowchart of an embodiment. [Figure 15] This is a simplified flowchart of an embodiment. [Figure 16] This is a simplified diagram of an embodiment. [Figure 17]This is a simplified diagram of an embodiment. [Figure 18] This is a simplified diagram of an embodiment. [Figure 19] This is a simplified diagram of an embodiment. [Figure 20] This is a simplified flowchart of an embodiment. [Figure 21] This is a simplified diagram of an embodiment. [Figure 22] This is a simplified flowchart of an embodiment. [Figure 23] This is a simplified flowchart of an embodiment. [Figure 24] This is a simplified diagram based on the embodiment. [Modes for carrying out the invention]

[0019] The proposed features described below may be used individually or in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium.

[0020] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102, 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may code video data at its local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the coded video data from the other terminal via the network 105, decode the coded data, and display the restored video data. Unidirectional data transmission may be common in media delivery applications, etc.

[0021] Figure 1 illustrates a second pair of terminals 101 and 104 provided to support the bidirectional transmission of coded video, which may occur, for example, during a video conference. For bidirectional data transmission, each terminal 101 and 104 may code video data captured at its local location for transmission to the other terminal via the network 105. Each terminal 101 and 104 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the restored video data on a local display device.

[0022] In Figure 1, terminals 101, 102, 103, and 104 may be exemplified as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure find applications involving laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that transmit coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network 105 may not be important to the operation of this disclosure unless described below herein.

[0023] Figure 2 illustrates an example of the deployment of a video encoder and video decoder in a streaming environment as an example of the application of the disclosed subject matter. The subject matter of this disclosure can be equally applied to other video-enabled applications, such as video conferencing, digital television, and the storage of compressed video on digital media including CDs, DVDs, and memory sticks.

[0024] The streaming system may include a capture subsystem 203 which can include a video source 201, such as a digital camera, which creates, for example, an uncompressed video sample stream 213. The sample stream 213 may be highlighted as having a higher data volume compared to an encoded video bitstream and can be processed by an encoder 202 coupled to the video source 201, which may be a camera as described above. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the subject disclosed, as will be described in more detail below. The encoded video bitstream 204 may be highlighted as having a lower data volume compared to the sample stream and can be stored in a streaming server 205 for future use. One or more streaming clients 212 and 207 can access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include a video decoder 211 which decodes an incoming copy 208 of the encoded video bitstream and creates an outgoing video sample stream 210 that can be rendered on a display 209 or other rendering device (not shown). In some streaming systems, video bitstreams 204, 206, and 208 can be encoded according to specific video coding / compression standards. Examples of these standards are mentioned above and will be further described herein.

[0025] Figure 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present invention.

[0026] Receiver 302 may receive one or more codec video sequences to be decoded by decoder 300, which in the same or different embodiments may be one coded video sequence at a time, with the decoding of each coded video sequence being independent of other coded video sequences. Coded video sequences may be received from channel 301, which may be a hardware / software link to a storage device that stores coded video data. Receiver 302 may receive coded video data together with other data that may be transferred to their respective usage entities (not shown), such as coded audio data and / or auxiliary data streams. Receiver 302 may isolate coded video sequences from other data. To counteract network jitter, a buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter, "Parser"). Buffer 303 may not be necessary or may be small if receiver 302 is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network. When used in best-effort packet networks such as the internet, a buffer of 303 may be required, which can be relatively large and, advantageously, can be adapted to an adaptive size.

[0027] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from an entropy-coded video sequence. The categories of these symbols may include information used to manage the operation of the decoder 300, and potentially information for controlling rendering devices such as a display 312, which are not integral parts of the decoder but can be coupled to it. The rendering device control information may be in the form of supplemental extension information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and arithmetic coding with or without context dependency. From the coded video sequence, the parser 304 may extract from the coded video sequence a set of at least one subgroup parameter of subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), and predictive units (PU). The entropy decoder / parser may also extract information from coded video sequence information such as transform coefficients, quantizer parameter values, and motion vectors.

[0028] The parser 304 may perform entropy decoding / parse operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 may determine whether a particular symbol 313 should be provided to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0029] The reconstruction of symbol 313 may involve multiple different units, depending on the type of coded video picture or part thereof (such as interpictures and intrapictures, interblocks and intrablocks), as well as other factors. Which units are involved and how can be controlled by parser 304 using subgroup control information parsed from the coded video sequence. The flow of such subgroup control information between parser 304 and the following multiple units is not illustrated for clarity.

[0030] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units can interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the disclosed subject, the following conceptual subdivision into functional units is appropriate.

[0031] The first unit is the scaler / inverse unit 305. The scaler / inverse unit 305 receives control information from the parser 304 as symbol 313, including the quantized transformation coefficients and the transformation to be used, block size, quantization factor, quantization scaling matrix, etc. It can output a block containing sample values ​​that can be input to the aggregator 310.

[0032] In some cases, the output samples of the scaler / inverse transform 305 may relate to intracoded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information can be provided by the intrapicture predictive unit 307. In some cases, the intrapicture predictive unit 307 generates a block of the same size and shape as the block being reconstructed, using already reconstructed surrounding information fetched from the current (partially reconstructed) picture 309. The aggregator 310, in some cases, adds the predictive information generated by the intrapredictive unit 307 to the output sample information provided by the scaler / inverse transform unit 305, sample by sample.

[0033] In other cases, the output samples of the scaler / inverse unit 305 may relate to an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit 306 can access the reference picture memory 308 to fetch samples to be used for prediction. After the fetched samples have been motion-compensated according to the symbols 313 relating to the block, these samples can be added to the output of the scaler / inverse unit by the aggregator 310 to generate output sample information (in this case, called residual samples or residual signals). The address in the reference picture memory format from which the motion-compensated unit fetches prediction samples can be controlled by a motion vector, which can be made available to the motion-compensated unit in the form of a symbol 313 that can have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when the exact motion vector of the subsample is used, a motion vector prediction mechanism, etc.

[0034] The output samples of the aggregator 310 can be subjected to various loop filtering techniques in the loop filter unit 311. The video compression technique is controlled by parameters contained in the coded video bitstream and may include in-loop filtering techniques available in the loop filter unit 311 as symbols 313 from the parser 304, but may also respond to metadata obtained during decoding of previous (in decoding order) portions of the coded picture or coded video sequence, or to previously reconstructed and loop-filtered sample values.

[0035] The output of the loop filter unit 311 can be output to the rendering device 312, or it can be a sample stream that can be stored in the reference picture memory 557 for use in future interpicture prediction.

[0036] Once fully reconstructed, a particular coded picture can be used as a reference picture for future predictions. When a coded picture is fully reconstructed and identified as a reference picture (for example, by parser 304), the current reference picture 309 can become part of the reference picture buffer 308, allowing for the reallocation of new current picture memory before initiating the reconstruction of subsequent coded pictures.

[0037] The video decoder 300 may perform decoding operations according to a predetermined video compression technique that may be documented in a standard such as ITU-T Rec.H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that it conforms to the syntax of the video compression technique or standard as specified in the video compression technique documentation or standard, and specifically as specified in the profile documented therein. Furthermore, compliance may require that the complexity of the coded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limitations set by the level may, in some cases, be further limited by the virtual reference decoder (HRD) specification and metadata for HRD buffer management signaled within the coded video sequence.

[0038] In one embodiment, receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by video decoder 300 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time layer, a spatial layer, or a signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, or forward error correction codes.

[0039] Figure 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0040] The encoder 400 may receive video samples from a video source 401 (not part of the encoder) that can capture video images to be coded by the encoder 400.

[0041] The video source 401 may provide a source video sequence to be coded by the encoder (303) in the form of a digital video sample stream, which can be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a series of separate pictures that give motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0042] According to the embodiment, the encoder 400 can encode pictures of a source video sequence in real time or under any other time constraints as required by the application and compress them into a coded video sequence 410. One function of the controller 402 is to ensure an appropriate coding speed. The controller controls and is functionally coupled to other functional units, as described below. For simplicity, the coupling is not depicted. Parameters set by the controller may include rate control-related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art will be able to easily identify the other functions of the controller 402, as they may relate to the video encoder 400 optimized for a particular system design.

[0043] Some video encoders operate in what is readily recognizable to those skilled in the art as a “coding loop.” In an overly simplified explanation, the coding loop may consist of an encoding portion of encoder 400 (hereafter “source coder”) (which is responsible for creating symbols based on the input picture to be coded and a reference picture), and a (local) decoder 406 built into encoder 400 that creates sample data which will also create a (remote) decoder by reconstructing the symbols (since any compression between the symbols and the coded video bitstream is reversible in the video compression techniques considered in the disclosed subject). The reconstructed sample stream is input to reference picture memory 405. Because decoding the symbol stream yields bit-exact results regardless of the decoder’s location (local or remote), the reference picture buffer contents are also bit-exact between the local encoder and the remote encoder. In other words, the prediction portion of the encoder “sees” the exact same sample values ​​as reference picture samples that the decoder “sees” when using predictions during decoding. This basic principle of the synchronization of reference pictures (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0044] The operation of the “local” decoder 406 can be the same as that of the “remote” decoder 300, which has already been described in detail above in relation to Figure 3. However, also briefly referring to Figure 4, since symbols are available and the encoding / decoding of symbols into the coded video sequence by the entropy coder 408 and parser 304 may be reversible, the entropy decoding portion of decoder 300, including channel 301, receiver 302, buffer 303, and parser 304, may not be fully performed by the local decoder 406.

[0045] At this point, it can be said that any decoder technology present within the decoder, excluding parsing / entropy decoding, must also necessarily exist in the corresponding encoder in substantially the same functional form. The description of encoder technology can be omitted as it is the inverse of the comprehensively described decoder technology. More detailed explanations are necessary only in specific areas, and are provided below.

[0046] As part of its operation, the source coder 403 may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from a video sequence designated as “reference frames”. In this manner, the coding engine 407 codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame which may be selected as a predictive reference to the input frame.

[0047] The local video decoder 406 can decode the coded video data of a frame that may be designated as a reference frame based on symbols created by the source coder 403. The operation of the coding engine 407 may preferably be a lossy process. When the coded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 406 can replicate the decoding process that may be performed on the reference frame by the video decoder and store the reconstructed reference frame in a reference picture memory 405, which may be a cache, for example. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that has common content as the reconstructed reference frame that will be acquired by a video decoder at the far end (without transmission errors).

[0048] The predictor 404 may perform a predictive search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for specific metadata, such as reference picture motion vectors and block shapes, which can function as sample data (as candidate reference pixel blocks) or as appropriate predictive references for the new picture. The predictor 404 may operate on a sample block basis for each pixel block to find an appropriate predictive reference. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 405, as determined by the search results obtained by the predictor 404.

[0049] The controller 402 can manage the coding operations of the source coder 403, which may be a video coder, for example, including setting parameters and subgroup parameters used to encode video data.

[0050] The output of all the aforementioned functional units can undergo entropy coding in the entropy coder 408. The entropy coder converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0051] Transmitter 409 may buffer the coded video sequence created by entropy coder 408 and prepare it for transmission via communication channel 411, which may be a hardware / software link to a storage device that will store the coded video data. Transmitter 409 may merge the coded video data from source coder 403 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (source not shown).

[0052] The controller 402 may manage the operation of the encoder 400. During coding, the controller 402 may assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following frame types:

[0053] An intra-picture (I-picture) may be one that can be coded and decoded without using other frames in the sequence as a source for prediction. Some video codecs enable various types of intra-pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are familiar with these variations of I-pictures, as well as their respective applications and characteristics.

[0054] A predictive picture (P-picture) may be one that can be coded and decoded using intra-prediction or inter-prediction, which uses up to one motion vector and reference index to predict the sample values ​​for each block.

[0055] A bidirectional predictive picture (B-picture) may be one that can be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, a multiple predictive picture can use more than two reference pictures and associated metadata to reconstruct a single block.

[0056] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded unpredictably or predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded unpredictably via spatial prediction or via temporal prediction by referencing one previously coded reference picture. Blocks of picture B may be coded unpredictably, via spatial prediction, or via temporal prediction by referencing one or two previously coded reference pictures.

[0057] For example, the encoder 400, which may be a video coder, may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec.H.265. In its operation, the encoder 400 may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0058] In one embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source coder 403 may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0059] FIG. 5 shows a simplified block style workflow diagram 500 of exemplary viewport dependent processing in an omnidirectional media application format (OMAF) that enables 360-degree virtual reality (VR360) streaming described in the omnidirectional media application format (OMAF).

[0060] In acquisition block 1001, when the image data can represent a VR360 scene, video data A such as a plurality of images and audio data at the same time instance is acquired. In processing block 1003, an image B at the same time instance i is processed by being stitched, mapped to a picture projected with respect to one or more virtual reality (VR) angles or other angles / viewpoints, and packed for each region, among one or more of such processes. In addition, metadata indicating either such processed information or other information is created to assist in distribution and rendering processing.

[0061] Regarding data D, in image encoding block 1005, the projected picture is encoded into data E i and configured into a media file. In viewport-independent streaming, in video encoding block 1004, the video picture is encoded as data E v as, for example, a single-layer bitstream, and regarding data B a audio data can also be encoded into data E a in audio encoding block 1002.

[0062] Data E a 、E v 、and E i 、the encoded bitstream F iand / or the entire F may be stored on a (Content Delivery Network (CDN) / cloud) server, typically in the delivery block 1007, or otherwise fully transmitted to the OMAF player 1020, and fully decoded by a decoder, thereby rendering to the user in the display block 1016, from the head / eye tracking block 1008, various metadata, file playback, and orientation / viewport metadata, e.g., with respect to the angles the user may be viewing through the VR image device with respect to the viewport specifications of the device. A distinct feature of VR360 is that only the viewport may be displayed at any given time, and such feature can be used to improve the performance of the omnidirectional video system by selective delivery depending on the user's viewport (or any other criterion such as recommended viewport timed metadata). For example, viewport-dependent delivery may be enabled by tile-based video coding in an exemplary embodiment.

[0063] Similar to the encoding block described above, the OMAF player 1020 in an exemplary embodiment uses data F' and / or F' i Furthermore, with respect to decapsulation of one or more files / segments of the metadata, one or more facets of such encoding are similarly reversed, and audio data E' in audio decoding block 1010. i , video data E' in video decoding block 1013 v , and image data E' in image decoding block 1014 i Decode the data B' in audio rendering block 1011. a Audio rendering and image rendering of data D' in block 1015 proceed, and display data A' in VR360 format in display block 1016 according to various metadata such as orientation / viewport metadata. iand audio data A' in speaker / headphone block 1012 s The output may include various metadata which may affect one of the data decoding and rendering processes depending on the various tracks, languages, qualities, and views that may be selected by or for the user of the OMAF player 1020, and it should be understood that the order of processing described herein is presented for exemplary embodiments and may be carried out in other orders according to other exemplary embodiments.

[0064] Figure 6 shows a simplified block-style content flow process diagram 600 of (coded) point cloud data (hereinafter referred to as "V-PCC") with view position and angle-dependent processing for capture / generation / (de)coding / rendering / display of 6-degree-of-freedom media. It should be understood that the described features may be used separately or in any order, and elements such as encoding and decoding may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits), particularly as illustrated, and one or more processors may execute a program stored on a non-temporary computer-readable medium according to the exemplary embodiment.

[0065] Diagram 600 shows an exemplary embodiment for streaming coded point cloud data using V-PCC.

[0066] In the volumetric data acquisition block 1101, a real-world visual scene or a computer-generated visual scene (or a combination thereof) may be captured by a set of camera devices or synthesized by a computer as volumetric data, which may have any format. The volumetric data can be converted to a (quantized) point cloud data format via image processing in the point cloud conversion block 1102. For example, data derived from volumetric data can be converted to one point in a point cloud, area by area, by pulling one or more of the values ​​described below from the volumetric data and any related data into a desired point cloud format. In the exemplary embodiment, the volumetric data may be a 3D dataset of 2D images, such as slices from which a 2D projection of the 3D dataset can be projected. According to exemplary embodiments, a point cloud data format may include a representation of data points in one or more different spaces and may be used to represent volumetric data and may provide improvements with respect to sampling and data compression with respect to temporal redundancy, for example, point cloud data in x, y, z format may represent color values ​​(e.g., RGB), brightness, intensity, etc., at each of multiple points in the cloud data and may be used with progressive decoding, polygon meshes, direct rendering, and octree 3D representations of 2D quadtree data.

[0067] In the projection block 1103 onto the image, the acquired point cloud data can be projected onto a 2D image and encoded as an image / video picture using video-based point cloud coding (V-PCC). The projected point cloud data may consist of attributes, geometry, occupancy maps, and other metadata used to reconstruct the point cloud data using, for example, the Painter's algorithm, raycasting algorithm, (3D) binary space partitioning algorithm, etc.

[0068] Meanwhile, in the scene generator block 1109, the scene generator may generate several metadata used to render and display 6-degree-of-freedom (DoF) media, for example, depending on the director's intent or the user's preference. Such 6DoF media may include 360VR, such as a 3D view of the scene from rotational changes on the 3D axes X, Y, and Z, in addition to additional dimensions that enable forward, backward, up, and left and right movement within, or at least corresponding to, the point cloud coded data for the virtual experience. Scene description metadata defines one or more scenes consisting of coded point cloud data and other media data, including VR360, light fields, audio, etc., and may be provided to one or more cloud servers and / or file / segment encapsulation / deencapsulation processes, as shown in Figure 6 and the related description.

[0069] Following the video encoding block 1104 and image encoding block 1105, similar to the video and image encoding described above (as understood, audio encoding may also be provided as described above), the file / segment encapsulation block 1106 processes the coded point cloud data so that it consists of a media file for file playback, or a sequence of initialization segments, and media segments for streaming according to a specific media container file format, such as one or more video container formats, and such descriptions may be used with respect to DASH described later, such as representing exemplary embodiments. The file container may also include scene description metadata in the file or segment, such as from the scene generator block 1109.

[0070] According to an exemplary embodiment, a file is encapsulated according to scene description metadata so that such a file can be transmitted in response to a request in response to user or author input, and each contains at least one view position and at least one or more angular views at (one or more) view positions, each at one or more times in 6DoF media. Furthermore, according to an exemplary embodiment, a segment of such a file may contain one or more parts of such a file, e.g., a portion of 6DoF media showing a viewpoint and an angle at one or more times, but these are merely exemplary embodiments and may be modified depending on various conditions such as the network, user, author capabilities and input.

[0071] According to an exemplary embodiment, point cloud data is partitioned into multiple 2D / 3D regions that are independently coded in one or more of the video encoding block 1104 and the image encoding block 1105. Each independently coded partition of the point cloud data can then be encapsulated in a file / segment encapsulation block 1106 as a file and / or segment track. According to an exemplary embodiment, each point cloud track and / or metadata track may contain some useful metadata for view position / angle-dependent processing.

[0072] According to an exemplary embodiment, metadata useful for view position / angle-dependent processing, such as that contained in encapsulated files and / or segments with respect to file / segment encapsulation blocks, includes one or more of the following: layout information of 2D / 3D partitions with an index; (dynamic) mapping information that associates a 3D volume partition with one or more 2D partitions (e.g., any of tiles / tile groups / slice / subpictures); the 3D position of each 3D partition on a 6DoF coordinate system; a representative view position / angle list; a selected view position / angle list corresponding to the 3D volume partition; an index of the 2D / 3D partition corresponding to the selected view position / angle; quality (rank) information for each 2D / 3D partition; and rendering information for each 2D / 3D partition according to each view position / angle, for example. By retrieving such metadata when requested, such as by a V-PCC player user or by a content creator on behalf of a V-PCC player user, more efficient processing of a desired portion of the 6DoF media may be enabled with respect to such metadata, thereby enabling the V-PCC player to deliver high-quality images that focus on the portion of the 6DoF media rather than the unused portion of the media.

[0073] From the file / segment encapsulation block 1106, a file or one or more segments of a file can be delivered directly to either the V-PCC player 1125 or a cloud server such as the cloud server block 1107 using a delivery mechanism (e.g., Dynamic Adaptive Streaming over HTTP (DASH)), and the cloud server can extract one or more tracks and / or one or more specific 2D / 3D partitions from the file and merge multiple coded point cloud data into a single data.

[0074] According to data such as the position / view angle tracking block 1108, if the current view position and angle are defined in a 6DoF coordinate system in the client system, the view position / angle metadata may be delivered from the file / segment encapsulation block 1106 or processed in other ways from a file or segment already present on the cloud server in the cloud server block 1107. As a result, the cloud server may extract the appropriate partitions from the store file and merge them (if necessary) according to the metadata from the client system, for example, having a V-PCC player 1125, and the extracted data can be delivered to the client as a file or segment.

[0075] With respect to such data, in the file / segment decapsulation block 1115, the file decapsulator processes the file or received segment, extracts the coded bitstream, parses the metadata, and in the video decoding and image decoding blocks, the coded point cloud data is then decoded and reconstructed into point cloud data in the point cloud reconstruction block 1112, the reconstructed point cloud data can be displayed in the display block 1114 and / or initially configured in the scene configuration block 1113 according to one or more different scene descriptions for scene description data by the scene generator block 1109.

[0076] In consideration of the foregoing, such exemplary V-PCC flows represent advantages over the V-PCC standard, including one or more of the following: the ability to partition described for multiple 2D / 3D areas, the ability to compress domain assembly of coded 2D / 3D partitions into a single conforming coded video bitstream, and the ability to extract coded 2D / 3D bitstreams of coded pictures into a conforming coded bitstream. Such V-PCC system support is further enhanced by including containerization for VVC bitstreams to support a mechanism containing metadata that carries one or more of the metadata described above.

[0077] From that perspective, according to the exemplary embodiments further described below, the term “mesh” refers to a composition of one or more polygons representing the surface of a volumetric object. Each polygon is defined by its vertices in 3D space and information about how those vertices are connected, which is called connectivity information. Optionally, vertex attributes such as color and normals can be associated with the mesh vertices. Attributes can also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such a mapping is called UV coordinates or texture coordinates and can be described by a set of parametric coordinates associated with the mesh vertices. The 2D attribute map (referred to as a texture map in some examples) is used to store high-resolution attribute information such as texture, normals, and displacements. Such information can be used for various purposes, such as texture mapping and shading.

[0078] Nevertheless, dynamic mesh sequences can require large amounts of data because they can consist of a considerable amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with always-on connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information. DCC (Digital Content Creation) tools typically generate such dynamic meshes. In response, it is difficult for volumetric acquisition techniques to generate always-on connectivity dynamic meshes, especially under real-time constraints. This type of content is not supported by existing standards. According to exemplary embodiments herein, embodiments of a new mesh compression standard for directly handling dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps are described, and this standard targets lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, AR, and VR. Features such as random access and scalable / progressive coding are also considered. A "mesh," or "input mesh," is data consisting of multiple vertices, at least two of them.

[0079] Figure 7 represents an exemplary framework 700 of dynamic mesh compression for a 2D atlas sampling-based method. Each frame of the input mesh 1201 can be preprocessed by a series of operations such as tracking, remeshing, parameterizing, and voxelizing. Note that these operations can be performed by the encoder only, meaning they may not be part of the decoding process, and such possibilities can be signaled in the metadata by flags such as showing 0 to the encoder only and 1 to others. A mesh 1202 with a 2D UV atlas can then be obtained, in which each vertex of the mesh has one or more associated UV coordinates of the 2D atlas. The mesh can then be transformed into multiple maps, including geometry maps and attribute maps, by sampling with the 2D atlas. These 2D maps can then be coded by video / image codecs such as HEVC, VVC, AV1, and AVS3. On the decoder side 1203, the mesh can be reconstructed from the decoded 2D maps. Any post-processing and filtering can also be applied to the reconstructed mesh 1204. Note that other metadata may be signaled to the decoder side for the purpose of 3D mesh reconstruction. Note that chart boundary information, including the uv and xyz coordinates of boundary vertices, can be predicted, quantized, and entropy coded within the bitstream. The quantization step size can be configured on the encoder side due to the trade-off between quality and bitrate.

[0080] In some embodiments, a 3D mesh can be partitioned into several segments (or patches / charts). Each segment consists of a set of connected vertices associated with their geometry, attributes, and connectivity information. As shown in the example volumetric data 800 in Figure 8, a UV parameterization process 1302 that maps 3D mesh segments to a 2D chart, such as the 2D UV atlas 1202 block described above, maps one or more mesh segments 1301 to a 2D chart 1303 in a 2D UV atlas 1304. Each vertex (v) in the mesh segment n The vertices (v) in the 2D chart are assigned 2D UV coordinates within the 2D UV atlas. n It should be noted that these vertices form connecting components as their 3D counterparts. The geometry, attributes, and connectivity information of each vertex can also be inherited from their 3D counterparts. For example, information that vertex v4 is directly connected to vertices v0, v5, v1, and v3 may be shown, and information for each other vertex may be shown in the same way. Furthermore, such a 2D texture mesh may, according to an exemplary embodiment, further show information such as color information for each patch of triangles, e.g., v2, v5, v3, etc.

[0081] For example, see Example 900 in Figure 9, which shows that in addition to the features of Example 800 in Figure 8, the 3D mesh segment 1301 can also be mapped to multiple separate 2D charts 1401 and 1402. In this case, the 3D vertices can correspond to multiple vertices in the 2D UV atlas. As shown in Figure 9, the same 3D mesh segment is mapped to multiple 2D charts in the 2D UV atlas instead of a single chart as in Figure 8. For example, 3D vertices v1 and v4 correspond to two 2D correspondence relationships v1, v4, respectively. 1’ and v4,v 4’Therefore, a typical 2D UV atlas of a 3D mesh may consist of multiple charts, as shown in Figure 14, each chart may contain multiple (usually three or more) vertices associated with their 3D geometry, attributes, and connectivity information.

[0082] Figure 10 shows Example 1000, which illustrates a derived triangulation in a chart having boundary vertices B0, B1, B2, B3, B4, B5, B6, B7. Given such information, any triangulation can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). For example, for each vertex, find the two closest vertices. Or, for all vertices, generate triangles sequentially until a minimum number of triangles are achieved after a set number of trials. As shown in Example 1000, there are various regular shapes of repeating triangles and various irregular shapes, which are generally closest to boundary vertices and have their own dimensions, which may or may not be shared with other triangles. Connectivity information can also be reconstructed by explicit signaling. If the polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream according to an exemplary embodiment.

[0083] Looking at the patches described above, Example 1000 could represent one such patch, such as the patch formed from vertices v3, v2, and v5 shown in either Figure 14 or Figure 15.

[0084] The boundary vertices B0, B1, B2, B3, B4, B5, B6, B7 are defined in 2D UV space. As shown in Figure 15, the filled vertices are boundary vertices because they lie on the boundary edges of connected components (patches / charts). Boundary edges can be determined by checking whether the edge appears in only one triangle. The following boundary vertex information is important and should be signaled in the bitstream according to the exemplary embodiment, and the information is geometric information, e.g., 3D XYZ coordinates and 2D UV coordinates, even if currently in 2D UV parametric format.

[0085] As shown in Figure 9, when a 3D boundary vertex corresponds to multiple vertices in a 2D UV atlas, the mapping from 3D XUZ to 2D UV can be one-to-many. Therefore, a UV-to-XYZ (or UV2XYZ) index can be signaled to indicate the mapping function. UV2XYZ may also be a 1D array of indices that map each 2D UV vertex to a 3D XYZ vertex.

[0086] According to exemplary embodiments, in order to efficiently represent the mesh signal, a subset of mesh vertices may be coded first, along with connectivity information between them. In the original mesh, since these vertices are subsampled from the original mesh, connectivity between them may not exist. There are various ways to signal connectivity information between vertices, and therefore such a subset is called the base mesh or base vertices.

[0087] However, other vertices can be predicted by applying interpolation between two or more already decoded mesh vertices. Since predictor vertices have their geometric positions along the edges of two connected existing vertices, the geometric information of a predictor can be calculated based on nearby decoded vertices. In some cases, a displacement vector or prediction error from the vertex to be coded to the vertex predictor should be further coded. For example, referring to Example 1100 in Figure 11, an example of such edge-based vertex prediction is shown, more specifically, an example of vertex geometry prediction using extrapolation by extending a triangle into a parallelogram as shown on the left, and intra-prediction by interpolation by a weighted average of two existing vertices as shown on the right. After decoding the base vertices (i.e., the solid triangle 1801 on the left side of Figure 11), interpolation between these base vertices can be performed along connected edges. For example, the midpoint of each edge can be generated as a predictor. Thus, the geometric positions of these interpolated points are the (weighted) average of two nearby decoded vertices (dashed point 1802 on the left side of Figure 11). Having multiple intermediate points between two already decoded vertices can be done in a similar manner. Thus, the actual vertices to be coded can be reconstructed by adding displacement vectors to the predictors (Figure 7, center). After decoding these additional vertices, the connections between the newly decoded vertices and the existing base vertices are still maintained. Furthermore, connections between the newly decoded vertices can be further established. By connecting these newly decoded vertices 1803 together with the base vertices, along with the base vertices, more intermediate vertex predictors can be generated along new edges (Figure 7, right). Thus, more actual vertices to be decoded exist, along with their associated displacement vectors.

[0088] According to an exemplary embodiment, the mesh vertices of mesh frame 1902 can also be predicted from the decoded vertices of a previously coded mesh frame 1901. This prediction mechanism is called interpretation. An example of interpretation of mesh geometry is shown in Example 1200 in Figure 12, which demonstrates vertex geometry prediction using interpretation (vertices from the previous mesh frame become predictors of the vertices in the current frame). In some cases, a displacement vector or prediction error from the vertex to be coded to the vertex predictor should be further coded.

[0089] According to exemplary embodiments, several methods are employed for dynamic mesh compression, which are part of the edge-based vertex prediction framework described above, where a base mesh is first coded, and then more additional vertices are predicted based on connectivity information from the edges of the base mesh. It should be noted that the methods may be applied individually or in any form of combination.

[0090] For example, consider flowchart 1300 of an example of vertex grouping for prediction modes in Figure 13. In S201, vertices in a mesh may be acquired and in S202, they may be divided into different groups for prediction purposes, see, for example, Figure 10. In one example, the division is performed using patch / chart partitioning in S204 as described above. In another example, the division is performed under each patch / chart S205. The decision in S203 to proceed to S204 or S205 may be signaled by a flag or the like. In the case of S205, some vertices in the same patch / chart may form a prediction group and share the same prediction mode, while some other vertices in the same patch / chart may use other prediction modes. Such grouping in S206 can be assigned at different levels by determining the number of vertices to be included in each group. For example, every 64, 32, or 16 vertices in the scan order within the patch / chart may be assigned the same prediction mode according to the exemplary embodiment, while other vertices may be assigned differently. For each group, the prediction mode can be either intra-prediction mode or inter-prediction mode. This can be signaled or assigned. According to the illustrative flowchart 1300, if it is determined in S207 that a mesh frame or mesh slice is of intra-type, for example by checking whether the flag of the mesh frame or mesh slice indicates an intra-type, then all vertex groups within that mesh frame or mesh slice shall use intra-prediction mode; otherwise, in S208, either intra-prediction mode or inter-prediction mode may be selected for each group of vertices within it.

[0091] Furthermore, in the case of a group of mesh vertices using intra-prediction mode, the vertices can only be predicted by using previously coded vertices within the same subpartition of the current mesh. Sometimes, the subpartition can be the current mesh itself, according to an exemplary embodiment, and in the case of a group of mesh vertices using inter-prediction mode, the vertices can only be predicted by using previously coded vertices from other mesh frames, according to an exemplary embodiment. Each of the above pieces of information can be determined and signaled by flags, etc. The prediction features may occur in S210, and the results of the prediction and signaling may occur in S211.

[0092] According to an exemplary embodiment, for each vertex in a group of vertices in exemplary flowchart 1300 and flowchart 1400 described later, after prediction, the residual is a 3D displacement vector indicating the shift from the current vertex to its predictor. The residuals of a group of vertices need to be further compressed. In one example, the transformation in S211, along with its signaling, can be applied to the residuals of a group of vertices before entropy coding. To handle the coding of groups of displacement vectors, the following methods can be employed. For example, one method appropriately signals when a group of displacement vectors, some displacement vectors, or its components have only zero values. In another embodiment, a flag is signaled for each displacement vector indicating whether it has non-zero components, and if not, the coding of all components of this displacement vector can be skipped. Furthermore, in yet another embodiment, a flag is signaled for each group of displacement vectors indicating whether the group has a non-zero vector, and if not, the coding of all displacement vectors in this group can be skipped. Furthermore, in other embodiments, a flag may be signaled for each component of the group of displacement vectors to indicate whether this component of the group has any non-zero vectors, and if not, the coding of this component for all displacement vectors in this group may be skipped. Furthermore, in other embodiments, if a group of displacement vectors or a component of a group of displacement vectors requires a transformation, and if not, the transformation may be skipped, and there may be signaling when quantization / entropy coding can be applied directly to the group or a component of the group. Furthermore, in other embodiments, a flag may be signaled for each group of displacement vectors to indicate whether this group needs to undergo a transformation, and if not, the transformation coding of all displacement vectors in this group may be skipped.Furthermore, in other embodiments, a flag is signaled for each component of the group of displacement vectors to indicate whether this component of the group needs to undergo a transformation; if not, the transformation coding for this component of all displacement vectors in this group can be skipped. The embodiments described above in this paragraph relating to the processing of vertex prediction residuals can also be combined and implemented in parallel on different patches.

[0093] Figure 14 shows an exemplary flowchart 1400, where in S221, a mesh frame can be coded and acquired as a whole data unit, meaning that all vertices or attributes of the mesh frame may have correlations with each other. Alternatively, depending on the decision in S222, the mesh frame can be divided in S223 into smaller, independent subpartitions, similar in concept to slices or tiles of a 2D video or image. The coded mesh frame or coded mesh subpartition can be assigned a prediction type in S224. Possible prediction types include intra-coding types and inter-coding types. For intra-coding types, only predictions from reconstructed portions of the same frame or slice are allowed in S225. Inter-prediction types, on the other hand, allow predictions from previously coded mesh frames in addition to intra-mesh frame predictions in S225. Inter-prediction types may also be further classified into more subtypes, such as P-type and B-type. In P-type, only one predictor can be used for prediction purposes, while in B-type, two predictors from two previously coded mesh frames may be used to generate a predictor. A weighted average of two predictors can be one example. When a mesh frame is coded as a whole, the frame can be considered an intra-coded or intercoded mesh frame. In the case of an intermesh frame, the P or B type may be further identified via signaling. Alternatively, if the mesh frame is further divided and coded within the frame, a prediction type is assigned to each subpartition in S224. Each of the above pieces of information may be determined and signaled by flags, etc., and the prediction features may occur in S226, and the results of the prediction and signaling may occur in S227, as in S210 and S211 in Figure 13.

[0094] Therefore, dynamic mesh sequences can consist of a considerable amount of information that changes over time and thus may require a large amount of data. Efficient compression techniques are needed to store and transmit such content, and the features described above for Figures 20 and 21 represent such improved efficiency by enabling at least improved mesh vertex 3D position prediction by using either previously decoded vertices within the same mesh frame (intra-prediction) or previously decoded vertices from previously coded mesh frames (inter-prediction).

[0095] Furthermore, exemplary embodiments may generate displacement vectors for a third layer 2303 of a mesh based on one or more reconstructed vertices of a preceding layer(s), such as a second layer 2302 and a first layer 2301. Assuming the index of the second layer 2302 is T, the predictor T+1 for a vertex in the third layer 2303 is generated based on at least the reconstructed vertices of the current layer or the second layer 2302. An example of such a layer-based prediction structure is shown in Example 1600 of Figure 16, which demonstrates reconstruction-based vertex prediction, i.e., progressive vertex prediction using edge-based interpolation, where the predictor is generated based on previously decoded vertices rather than predictor vertices. The first layer 2301 may be a mesh bounded by a first polygon 2340, whose vertices include vertices decoded along its boundary and vertices interpolated along one of the lines between those decoded vertices. As progressive coding progresses from the first layer 2301 to the second layer 2302, an additional polygon 2341 may be formed by a displacement vector from one of the interpolated vertices of the first layer to an additional vertex in the second layer 2302, so that the total number of vertices in the second layer 2302 may be greater than the total number of vertices in the first layer 2301. Similarly, when progressing to the third layer 2303, the additional vertices in the second layer 2302, along with the decoded vertices from the first layer 2301, may function in coding in the same way as the decoded vertices supplied when progressing from the first layer 2301 to the second layer 2303, i.e., multiple additional polygons may be formed. As a note, referring to Example 1900 in Figure 19, which illustrates such progressive coding, unlike Figure 16, Example 1900 shows that as you progress from the first layer 2601 to the second layer 2603 and then to the third layer 2603, each of the additionally formed polygons may be entirely within the polygon formed by the boundary of the first layer 2601.

[0096] For such an example 1600, referring to the illustrative flowchart 1500 in Figure 15, the interpolated vertices of the current layer are predicted values, and therefore such values ​​need to be reconstructed before they can be used to generate predictors for the vertices of the next layer. This is done by coding the base mesh in S231, performing vertex prediction in S232, and then adding the decoded displacement vectors of the current layer to the predictors of vertices such as layer 2302 in S233. The reconstructed vertices of this layer 2303 can then be used in S235 to generate and signal predictor vertices for the next layer 2303, along with all the decoded vertices of the previous layer(s), such as checking for additional vertex values ​​for such a layer in S234. This process can also be summarized as follows: Let P[t](Vi) represent the predictor of vertex Vi on layer t, R[t](Vi) represent the reconstructed vertex Vi on layer t, D[t](Vi) represent the displacement vector of vertex Vi on layer t, and f(*) represent the predictor generator, which can in particular be the average of two existing vertices. Next, for each layer t, according to an exemplary embodiment, the following exists: P[t](Vi)=f(R[s|s <t](Vj)、R[m|m<t](Vk)) Here, Vj and Vk are the reconstructed vertices of the previous layer. R[t](Vi)=P[t](Vi)+D[t](Vi) Equation (1)

[0097] Next, for every vertex in a single mesh frame, they are divided into layers 0 (base mesh), 1, 2, ..., etc. Then, the reconstruction of vertices in one layer depends on the reconstruction of vertices in the previous layer. In the above, P, R, and D each represent a 3D vector in the context of the 3D mesh representation. D is the decoded displacement vector, and quantization may or may not be applied to this vector.

[0098] According to an exemplary embodiment, vertex prediction using reconstructed vertices may be applied to only certain layers, for example, layers 0 and 1. For other layers, vertex prediction can still use neighbor predictor vertices without adding displacement vectors to them for reconstruction. Thus, these other layers can be processed simultaneously without waiting for the previous layer to be reconstructed. According to an exemplary embodiment, for each layer, it may be signaled whether to select reconstruction-based vertex prediction or predictor-based vertex prediction, or whether to use reconstruction-based vertex prediction for the layer (and subsequent layers).

[0099] For displacement vectors generated by reconstructed vertices, quantization can be applied to them without performing further transformations such as wavelet transforms. For displacement vectors generated by other predictor vertices, transformations may be necessary, and quantization can be applied to the transformation coefficients of those displacement vectors.

[0100] Therefore, dynamic mesh sequences can require large amounts of data, as they can consist of a considerable amount of information that changes over time. Consequently, efficient compression techniques are needed to store and transmit such content. In the interpolation-based vertex prediction method framework described above, compressing the displacement vectors is one of the key steps, which constitute the majority of the coded bitstream and is the focus of this disclosure. For example, the feature in Figure 15 mitigates such problems by providing such compression.

[0101] Furthermore, as with the other examples described above, even in these embodiments, dynamic mesh sequences can still require a large amount of data because they can consist of a considerable amount of information that changes over time, and therefore efficient compression techniques are needed to store and transmit such content. In the framework of the 2D atlas sampling-based method described above, a significant advantage can be achieved by inferring connectivity information from the sampled vertices and boundary vertices on the decoder side. This is a key part of the decoding process and is the focus of the further examples described below.

[0102] According to an exemplary embodiment, connectivity information for the base mesh can be inferred (derived) from the decoded boundary vertices and sampled vertices of each chart on both the encoder and decoder sides.

[0103] As described above, any triangulation can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). For charts where none of the internal vertices are sampled, such as the internal vertices shown in Example 1000 in Figure 10 and Example 1800 in Figure 18, which are described further below, a similar method for creating connectivity still applies, although according to the exemplary embodiment, it may be signaled to use a different triangulation method for boundary vertices and sampled vertices.

[0104] For example, according to an exemplary embodiment, for each of four neighboring points at an arbitrary sampled location, it may be determined whether the number of occupied points is three or more (examples of occupied or unoccupied points are highlighted in Figure 18, which shows an occupation map example 1800 where each circle represents an integer number of pixels), and the triangular connectivity between the four points can be inferred by certain rules. For example, as shown in Example 1700 in Figure 17, if it is determined that three of the four points shown in Examples (2), (3), (4), and (4) are occupied, those points can be directly interconnected to form a triangle as in the example, while if it is determined that all four points are occupied, those points are used to form two triangles as shown in Example (1) in Figure 17. Note that different rules can be applied to different numbers of neighboring points. This process can be carried out over many points, as further shown in Figure 18. In this embodiment, the reconstructed mesh is a triangular mesh as shown in Figure 17, which cannot be determined to not be signaled according to such rules, at least like the regular triangular mesh inside Figure 18, but can instead be coded and decoded by inference rather than individual signaling, and as the surrounding irregular triangles that are individually signaled.

[0105] Furthermore, in an attempt to further reduce complexity and data processing, such regular internal triangular quadrilateral meshes shown in Figure 18 may be inferred as such quadrilateral meshes in Example (1) of Figure 17, thereby reducing even the degree of complexity caused by inferring the regular internal triangles, and instead inferring a reduced number of regular internal quadrilateral meshes.

[0106] According to an exemplary embodiment, as shown in example (1) of Figure 17, if it is determined that all four neighboring points are occupied, the quadrilateral mesh can be reconstructed.

[0107] From the above description, it can be inferred that, as shown in Figure 18, the reconstructed mesh of Example 1800 may be of a hybrid type, that is, some areas of the mesh frame generate triangular meshes and other areas generate quadrilateral meshes, some of which may be regular compared to other triangular meshes within them, and some may be irregular, such as being one of the boundaries but not necessarily all of the boundary meshes.

[0108] According to exemplary embodiments, such connectivity types can be signaled with high-level syntax such as sequence headers or slice headers.

[0109] As described above, connectivity information can also be reconstructed by explicit signaling, such as with irregularly shaped triangular meshes. That is, if it is determined that a polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream. Furthermore, according to exemplary embodiments, the overhead of such explicit signaling can be reduced depending on the polygon boundary. For example, as shown in Example 1800 of Figure 18, the connectivity information of a triangle is signaled to be reconstructed by both implicit rules, such as the regular Example 2400 of Figure 17, which can be inferred, and explicit signaling of at least one of the irregularly shaped polygons shown at the mesh boundary in Figure 18.

[0110] According to one embodiment, it is determined that only connectivity information between boundary vertices and sampled locations is signaled, and connectivity information between the sampled locations themselves is inferred.

[0111] Furthermore, in any embodiment, connectivity information may be signaled by prediction such that only the difference between the predicted connectivity (as a prediction) from one mesh to another can be signaled in the bitstream.

[0112] As a note, the inferred triangle orientation (e.g., inferred clockwise or counterclockwise for each triangle) can be signaled for all charts in high-level syntax such as sequence headers or slice headers, or fixed (assumed) by encoders and decoders in exemplary embodiments. The inferred triangle orientation can also be signaled differently for each chart.

[0113] As a further note, any reconstructed mesh may have different connectivity than the original mesh. For example, the original mesh may be a triangular mesh, while the reconstructed mesh may be a polygon mesh (e.g., a quad mesh).

[0114] According to exemplary embodiments, connectivity information for any base vertices does not need to be signaled; instead, edges between base vertices may be derived using the same algorithm on both the encoder and decoder sides. For example, see how the bottommost vertices in Example 1800 are all occupied, so that coding can utilize such information by determining that such vertices are occupied as bases, thereby inferring that connectivity information for any base vertices does not need to be signaled later; instead, edges between base vertices can be derived using the same algorithm on both the encoder and decoder sides. Also according to exemplary embodiments, interpolation of predicted vertices for additional mesh vertices may be based on derived edges of the base mesh.

[0115] According to an exemplary embodiment, a flag may be used to signal whether base vertex connectivity information should be signaled or derived, and such a flag may be signaled at different levels of the bitstream, such as the sequence level or the frame level.

[0116] According to an exemplary embodiment, the edges between base vertices are first derived using the same algorithm on both the encoder and decoder sides. Then, the difference between the derived edges and the actual edges is signaled by comparing them with the original connectivity of the base mesh vertices. Thus, after decoding the difference, the original connectivity of the base vertices can be restored.

[0117] For example, if a derived edge is found to be incorrect when compared to the original edge, such information may be signaled in a bitstream (by indicating the pair of vertices that form the edge), and if the original edge is not derived, it may be signaled in a bitstream (by indicating the pair of vertices that form the edge). Furthermore, connectivity on boundary edges and vertex interpolation including boundary edges may be performed separately from internal vertices and edges.

[0118] Accordingly, the technical problems pointed out above can be advantageously improved by one or more of the exemplary embodiments described herein. For example, since dynamic mesh sequences can consist of a considerable amount of information that changes over time, they may require large amounts of data, and therefore the exemplary embodiments described herein represent at least efficient compression techniques for storing and transmitting such content.

[0119] The embodiments described above may be further applied to instance-based mesh coding, where an instance may be a mesh of an object or part of an object. For example, the illustrated example 2100 in Figure 21 shows mesh example 2801, and various instances 2802 (representing the mesh of a cup), 2803 (representing the mesh of a spoon), and 2804 (representing the mesh of a plate) exist, each of which can be separated and coded. Also, each of instances 2801, 2802, 2803, and 2804 is shown by each of the bounding boxes described further below, but note that instance 2801 may be thought to be shown bounded by a “mesh-based bounding box”, and each of instances 2802, 2803, and 2804 may be thought to be shown as bounding by each of the “instance-based bounding boxes”.

[0120] According to exemplary embodiments, the proposed method may be used separately or combined in any order. The proposed method may be used with any polygon mesh, but only triangular meshes may be used to demonstrate various embodiments. As stated above, the input mesh may contain one or more instances, and a submesh is a part of the input mesh having one or more instances, and multiple instances can be grouped to form a submesh.

[0121] From that perspective, Figure 20 shows an example 2000 in which it is proposed to separately quantize different objects or parts at a given input bit depth (which may be called "QP"). For example, in 2701, one or more input meshes may be acquired, each of which can be separated into multiple submeshes. The submeshes can be objects, instances of objects, or segmented regions, which are independently quantized in S2702 according to an exemplary embodiment.

[0122] According to an exemplary embodiment, a mesh M having m points in (x,y,z) coordinates can be quantized by a QP bit depth in S2702. The quantization step size for all three dimensions (x,y,z) may be determined based on the maximum length of the bounding box in all dimensions, d bbox >0. Also, the same quantization step size may be applied in S2704 to all objects in the mesh identified in S2703 as follows:

number

number

number

number

number

[0123] However, in complex scenes, the largest object, the background, may often be relatively simple and can tolerate a higher quantization step size. The main object, on the other hand, is smaller and suffers from a large quantization error, which can be explained by various embodiments described further below.

[0124] Therefore, as shown in Example 2200 in Figure 22, the bounding box d of the input mesh bbox The maximum length is always as follows for each instance

number

number

number

[0125] In a given bit-depth QP, the quantization step size for each instance of instance 2802 (representing the cup mesh), 2803 (representing the spoon mesh), and 2804 (representing the plate mesh) is always:

number

[0126] Therefore, the quantization error for each instance becomes smaller, and the overall quantization error becomes smaller.

[0127] In various embodiments, the bit depth may be adaptively assigned to each instance / region called a “submesh” in S2902, and may be determined based on the face density of that particular instance. Each submesh may be obtained from volumetric data of a mesh, which itself may individually signal each instance within the mesh, and each submesh is derived from that mesh for each instance in S2902. For example, each of instances 2802, 2803, and 2804 may be assigned its own bit depth in S2904 according to its own particular face density or number of vertices, forming one or more of the aforementioned polygons within it. In general, the more faces each instance has, which can be determined in S2903 by counting the number of such polygons within it, the less quantization should be applied to that instance in S2702. For example, given a mesh M, the total number of faces is n, and the faces corresponding to the k-th submesh are n satisfying the following: k And,

number

number

number

number

[0128] According to various embodiments, the mesh is represented as a base mesh B and its corresponding displacement D, and is quantized in S2702 with different bit depths. For example, for the k-th object, the bit depth base mesh

number

number

number

[0129] In various embodiments, an adaptive bit depth parameter based on minimizing distortion may be used. For example, given an input bit depth QP, the mean squared error (MSE) of the quantization method is ε_QP, which can be given by equation (4). The MSE for each submesh is derived as ε_QP^k = ω_k * ε_QP, ∀k ∈ [1,...,K], where ω_k > 0 is a weight coefficient. In one example, ω_k = 1 ∀k. A linear search is performed for each submesh to find the best bit depth for the base mesh that satisfies the following:

number

[0130] In addition, the best bit depth for displacement can also be obtained via the following:

number

[0131] According to an exemplary embodiment, there may be signaling of the quantization of each object, such as signaling the bit depth via the bitstream with the signal S2907. The ascending set of base quantization bit depths is:

number

number

[0132] [Table 1]

[0133] Here, -u(n) is an unsigned integer using n bits, i(n) is an integer using n bits, and mips_quant() is a sequence of signaling data. -mips_min_bbox[k] is the minimum bounding box in the i-th dimension, -mips_num_instances_minus1 is the number of instances minus 1 in the mesh. -mips_base_bitdepth_minus1 is the bit depth of the first instance in this order, -mips_base_quant[k] is the difference between the quantizations of the (k+1)th and kth submesh. This number is always non-negative when the quantization sets are sorted in ascending order. -mips_dist_quant[k] is the k-th quantization data relative to the bit depth of the base mesh.

[0134] According to various embodiments, to reduce signaling overhead, multiple instances may be grouped into K groups with the same bit depth. The instances are bounded by a simple clustering method such as K-means clustering.

number

[0135] However, since such 3D scenes can often consist of multiple instances with similar mesh structures that reuse assets, further improvements can be achieved not only by utilizing local properties but also by considering the similarities between instances as in exemplary embodiments.

[0136] For example, Figure 23 shows an exemplary flowchart 2300 in which an instance-based matching prediction (IMP) method is used to find redundant meshes and encode the corresponding displacements, which may also involve favorably normalizing instances to maximize their similarity, and which may be used in any of the embodiments described above.

[0137] For example, in S2801, an input mesh may be acquired and partitioned into multiple submeshes as described above. A submesh may be an instance of an individual object or a part of an object, according to an exemplary embodiment.

[0138] In S2802, grouping from instances to similarity groups is possible using simple scaling features and similarity measures. For example, according to the embodiment, instances may be aligned and normalized so that only transient assets can be reused. Scale and orientation information may also be signaled via channels in IMP mode, and given an input mesh M having m instances, the i-th instance is,

number

[0139] According to an exemplary embodiment, in S2803, depending on the flag instruction, IMP is used to specify a size greater than 1, i.e., m k Asset groups that are >1

number

[0140] [Table 2]

[0141] Depending on flag S2804, coding in S2805 may be signaled to code submeshes independently into subbitstreams. Each submesh can be coded by a mesh codec having different coding parameters. Each submesh can also be coded by a different mesh codec, in which case it should be noted that a codec index indicating which mesh codec is used must be signaled in the subbitstream header or elsewhere. According to an exemplary embodiment, the subbitstreams of submeshes can be encoded and decoded in parallel without data dependency issues.

[0142] If the flag in S2804 instead indicates dependent coding, an additional flag may be considered in S2806 to also indicate a mode of dependent coding of submeshes. For example, according to one embodiment, a submesh can be coded by predictions from other submeshes that have already been coded. The prediction index can be coded to indicate which submesh is used as the prediction. The prediction index can be signaled at different levels.

[0143] For example, in S2807, only one prediction index may be coded for the entire submesh, and as a result, all vertices in the current submesh are predicted from the same submesh as indicated by the index in the exemplary embodiment.

[0144] In S2808, a prediction index can be signaled for each vertex of the current submesh so that each vertex can be predicted from a different submesh. Note that the prediction index can also be coded by prediction coding, and the prediction index of a vertex can be predicted from neighboring coded vertices. Next, according to an exemplary embodiment, the prediction index residual can be coded by arithmetic coding.

[0145] In S2809, the prediction index may be signaled at an intermediate level between the vertex level and the submesh level, for example, at the vertex group level, and groups of vertices share the same prediction index. Note that prediction indices for different groups can also be coded by prediction coding according to the exemplary embodiment.

[0146] Next, given a predicted index for each vertex of the current submesh, each vertex can be predicted from the corresponding vertex in the submesh, as described above in any of Figures 13, 14, and 15. According to an exemplary embodiment, rigid body motion may be estimated from the predicted submesh to the current submesh, and the parameters of the rigid body motion (e.g., rotation and translation parameters) can be coded. Then, after applying the rigid body motion to the predicted submesh, the residuals of the attributes of the current vertex can be obtained by subtracting the attributes of the corresponding vertex in the transformed predicted submesh. The vertex attributes may include, but are not limited to, geometry, color, normal, UV coordinates, connectivity, etc. Next, the residual information can be coded by arithmetic coding according to an exemplary embodiment.

[0147] Furthermore, such coding may be applied to the coding of material and texture information for each submesh. This information may include, but is not limited to, ambient color, diffuse color, specular color, focus of specular highlights, dissolve coefficient, lighting model, and texture image ID.

[0148] According to an exemplary embodiment, one submesh may allow only one set of material and texture information, in which case this information can simply be coded in the header of the subbitstream.

[0149] Alternatively, according to an exemplary embodiment, a submesh may have two or more sets of material and texture information, in which case these sets may be coded in the header of the subbitstream. Note that these parameters in different sets may be coded independently or dependently. When dependent coding is applied, prediction may be applied, and instead, the predicted residuals of the material parameters may be coded. Then, for each vertex in the submesh, a material ID may be coded to indicate which set of material information is used for that vertex. Note that, according to an exemplary embodiment, to reduce redundancy, the material ID (identifier) ​​may be coded by prediction from coded neighboring vertices.

[0150] Therefore, according to embodiments of this specification, a complex mesh can often contain information about multiple instances for associating texture maps, and since this information may be available at encoding time, each instance can be considered a 3D asset, as the asset can be frequently used in 3D design according to exemplary embodiments of this specification to reduce the cost of designing models, particularly for the synthesis of complex scenes. For example, a 3D model can be reused with or without modified textures due to differences in scale, orientation, etc. This addresses the problems associated with matching and retrieving 3D objects, as the mesh can be normalized to position, size, and orientation based on PCA (Principal Component Analysis) or bilateral planes of symmetry, thereby enabling more efficient retrieval.

[0151] The aforementioned techniques can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media, or by one or more hardware processors specifically configured. For example, Figure 24 shows a computer system 2400 suitable for implementing a particular embodiment of the disclosed subject matter.

[0152] Computer software can be coded using any suitable machine code or computer language that is subject to mechanisms such as assembly, compilation, and linking, in order to create code that contains instructions that can be executed directly or through interpretation, microcode execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0153] Instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0154] The components shown in Figure 24 with respect to the computer system 2400 are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components shown in the exemplary embodiments of the computer system 2400.

[0155] The computer system 2400 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images acquired from still image cameras, etc.), and video (2D video, 3D video including stereoscopic video, etc.).

[0156] The input human interface device may include one or more of the following (only one of each is illustrated): a keyboard 2401, a mouse 2402, a trackpad 2403, a touchscreen 2410, a joystick 2405, a microphone 2406, a scanner 2408, and a camera 2407.

[0157] The computer system 2400 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., touchscreen 2410, or tactile feedback via joystick 2405, although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speaker 2409, headphones (not shown)), visual output devices (screens 2410 including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touchscreen input capabilities, each with or without tactile feedback capabilities, some of which may be capable of outputting two-dimensional visual output or output beyond three dimensions via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0158] The computer system 2400 may also include human-accessible storage devices and the media associated with them, such as CD / DVD ROM / RW 2420 with CD / DVD 2411 or similar media, thumb drive 2422, removable hard drive or solid-state drive 2423, legacy magnetic media such as tape and floppy disks (not shown), and optical media including dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0159] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0160] The computer system 2400 may also include an interface 2499 to one or more communication networks 2498. The networks 2498 can be, for example, wireless, wired, or optical. The networks 2498 can further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks 2498 include local area networks such as Ethernet and Wi-Fi; cellular networks including GSM, 3G, 4G, 5G, and LTE; wired or wireless wide-area digital networks for TV including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and industrial networks including CANBus. Certain networks 2498 generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (2450 and 2451) (e.g., a USB port on the computer system 2400), while other networks are generally integrated into the core of the computer system 2400 by attachment to a system bus, as described later (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 2498, the computer system 2400 can communicate with other entities. Such communication may be unidirectional (e.g., broadcast television), unidirectional (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., to other computer systems using local area or wide area digital networks). Specific protocols and protocol stacks may be used for each of these networks and network interfaces, as described above.

[0161] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core 2440 of the computer system 2400.

[0162] The core 2440 may include one or more central processing units (CPUs) 2441, graphics processing units (GPUs) 2442, graphics adapters 2417, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 2443, and hardware accelerators 2444 for specific tasks. These devices may be connected via a system bus 2448, along with read-only memory (ROM) 2445, random access memory 2446, and internal mass storage 2447 such as internal, user-inaccessible hard drives or SSDs. In some computer systems, the system bus 2448 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus 2448 or via a peripheral bus 2449. Architectures for the peripheral bus include PCI, USB, etc.

[0163] The CPU 2441, GPU 2442, FPGA 2443, and accelerator 2444 can work together to execute specific instructions that constitute the aforementioned computer code. This computer code can be stored in ROM 2445 or RAM 2446. Transient data can also be stored in RAM 2446, while persistent data can be stored, for example, in internal mass storage 2447. By using cache memory that can be closely associated with one or more CPUs 2441, GPUs 2442, mass storage 2447, ROM 2445, RAM 2446, etc., high-speed storage and retrieval to any of the memory devices can be enabled.

[0164] A computer-readable medium may contain computer code for performing various computer implementation operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0165] For example, but not limited to, a computer system 2400 having an architecture, specifically a core 2440, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core 2440 that is non-transient in nature, such as the core internal mass storage 2447 or ROM 2445. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 2440. The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core 2440, and specifically the processors within it (including a CPU, GPU, FPGA, etc.), to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM 2446 and modifying such data structures according to processes defined by the software. In addition, or as an alternative, a computer system may also provide functionality as a result of hardwired or otherwise embodied logic in a circuit (e.g., accelerator 2444) that can perform a particular process or a particular part of a particular process as described herein, instead of, or in conjunction with, software. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry that houses software for execution (such as an integrated circuit (IC)), circuitry that embodies logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software.

[0166] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within its spirit and scope. [Explanation of Symbols]

[0167] 100 Communication system, 101 Terminal, 102 Second terminal, 103 First terminal, 104 Terminal, 105 Communication network, 201 Video source, 202 Encoder, 203 Capture subsystem, 204 Video bitstream, 205 Streaming server, 206 Video bitstream, 207 Streaming client, 208 Video bitstream, Incoming copy, 209 Display, 210 Outgoing video sample stream, 211 Video decoder, 212 Streaming client, 213 Uncompressed video sample stream, 300 Video decoder, 301 Channel, 302 Receiver, 303 Buffer memory, 304 Entropy decoder / parser, 305 Scaler / Inverse unit, 306 Motion compensation prediction unit, 307 Intra-picture prediction unit, 308 Reference picture memory, Reference picture buffer, 309 Reference picture, 310 Aggregator, 311 Loop filter unit, 312 Display, rendering device, 313 Symbol, 400 Video encoder, 401 Video source, 402 Controller, 403 Source coder, 404 Predictor, 405 Reference picture memory, 406 Local video decoder, 407 Coding engine, 408 Entropy coder, 409 Transmitter, 410 Coded video sequence, 411 Communication channel, 500 Simplified block-style workflow diagram, 600 Simplified block-style content flow process diagram, 700 Framework, 1001 Acquisition block, 1002 Audio encoding block, 1003 Processing block, 1004 Video encoding block, 1005 Image encoding block, 1007 Distribution block, 1008 Eye tracking block, 1010 Audio decoding block, 1011 Audio rendering block, 1012 Headphone block, 1013 Video decoding block, 1014 Image decoding block, 1015 Image rendering block, 1016 Display block, 1020 OMAF player, 1101 Volumetric data acquisition block, 1102 Transform block, 1103 Projection block, 1104Video encoding block, 1105 Image encoding block, 1106 Segment encapsulation block, 1107 Cloud server block, 1108 View angle tracking block, 1109 Scene generator block, 1112 Point cloud reconstruction block, 1113 Scene composition block, 1114 Display block, 1115 Segment deencapsulation block, 1125 V-PCC player, 1201 Input mesh, 1202 Mesh with 2D UV atlas, 2D UV atlas, 1203 Decoder, 1204 Reconstructed mesh, 1300 Exemplary flowchart, 1301 3D mesh segment, 1302 UV parameterization process, 1303 2D chart, 1304 2D UV atlas, 1400 Exemplary flowchart, 1401 2D chart, 1402 2D chart, 1500 Exemplary flowchart, 1700 Example, 1801 Triangle, 1802 Dashed line points, 1803 vertices, 1901 mesh frames, 1902 mesh frames, 2300 flowcharts, 2301 first layer, 2302 second layer, 2303 third layer, 2340 first polygon, 2341 additional polygons, 2400 computer system, 2401 keyboard, 2402 mouse, 2403 trackpad, 2405 joystick, 2406 microphone, 2407 camera, 2408 scanner, 2409 speaker, 2410 touchscreen, 2417 graphics adapter, 2420 CD / DVD ROM / RW, 2421 CD / DVD, 2422 thumb drive, 2423 solid state drive, 2440 core, 2441 central processing unit (CPU), 2442 graphics processing unit (GPU), 2443 Field-Programmable Gate Area (FPGA), 2444 Hardware Accelerator, 2445 Read-Only Memory (ROM), 2446 Random Access Memory, 2447 Core Internal Mass Storage, 2448 System Bus, 2449 Peripheral Bus, 2450 Peripheral Bus, 2451 Peripheral Bus, 2498 Communication Network, 2499 Interface, 2601 First Layer, 2602 Second Layer, 2603 Third Layer, 2801 Instance, 2802Instances 2803, 2804

Claims

1. A method for video decoding performed by at least one processor, wherein the method is The steps include obtaining an input mesh corresponding to volumetric data of at least one three-dimensional (3D) visual content, A step of partitioning the input mesh corresponding to the volumetric data into a plurality of submeshes, wherein each of the submeshes includes one instance of the object; A step of determining similarity between at least two instances of the object, comprising the step of comparing the bounding boxes of at least two instances of the object by comparing the ratio of the dimensions of the bounding boxes, The steps include: grouping the plurality of submeshes based on the determined similarity; The steps include: decoding the volumetric data based on the grouping of the plurality of submeshes; Methods that include...

2. The step of decoding the volumetric data includes signaling each of the at least two instances of the object to a different codec. The method for video decoding according to claim 1.

3. The step of decoding the volumetric data includes signaling a prediction of a first instance of the at least two instances of the object based on a prediction of a second instance of the at least two instances of the object. The method for video decoding according to claim 1.

4. The step of decoding the volumetric data includes signaling a single predictive index for the entirety of the first of the at least two instances of the object, The method for video decoding according to claim 3.

5. The step of decoding the volumetric data includes signaling a predicted index to each vertex of the first of the at least two instances of the object, The method for video decoding according to claim 3.

6. The step of decoding the volumetric data includes signaling a predictive index to a vertex group level which is an intermediate level between the vertex level and the submesh level of the first of the at least two instances of the object, The aforementioned vertex groups at the vertex group level share the same prediction index. The method for video decoding according to claim 3.

7. The step of decoding the volumetric data includes predicting each vertex from the vertices in the corresponding instance based on the predicted index of each vertex of the first of the at least two instances of the object. The method for video decoding according to claim 3.

8. An apparatus configured to perform the method described in any one of claims 1 to 7.

9. A computer program for causing a computer to perform the method described in any one of claims 1 to 7.

Images