Displacement coding for mesh compression

Abstract

a method and apparatus, the method comprising computer code configured to: cause one or more processors to obtain volumetric data for at least one frame of three-dimensional (3D) visual content; derive a mesh from the frame of the volumetric data, the mesh including a plurality of base mesh vertices; determine a displacement of at least one vertex that is not a base mesh vertex based on a series of projections from at least one of the plurality of base mesh vertices that are neighbors of the plurality of base mesh vertices to the at least one vertex; predict at least one vertex based on at least the determined displacements; and encode the volumetric data based on the predicted at least one vertex.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 406,174, filed September 13, 2022, and U.S. Patent Application No. 18 / 314,986, filed May 10, 2023, the disclosures thereof being incorporated herein by reference in their entirety.

[0002] This disclosure covers a set of advanced video coding techniques, including both lossless and lossy mesh coding techniques based on mesh instances. [Background technology]

[0003] Advances in 3D capture, modeling, and rendering are driving the widespread presence of 3D content across multiple platforms and devices. Today, it's possible to film a baby's first steps on one continent and have their grandparents on another continent watch (or interact with) it, enjoying a fully immersive experience with their child. Nevertheless, to achieve such realism, models have become more sophisticated than ever before, and a considerable amount of data is tied to the creation and consumption of these models.

[0004] VMesh is an ongoing MPEG standard for compressing static and dynamic meshes. VMesh separates the input mesh into a simplified base mesh and a residual mesh. The base mesh can be encoded in high quality, while the remaining mesh can be encoded using subdivision surface fitting and displacement encoding, taking advantage of local characteristics.

[0005] However, complex meshes often contain information about multiple instances in order to associate and relate texture maps. This information is available during encoding. On the other hand, a mesh can be segmented into several parts based on its characteristics. For example, there are more polygons in the face region of a human mesh.

Summary of the Invention

Problems to be Solved by the Invention

[0006] Therefore, a fixed quantization step size applied to all instances, objects, and parts within a mesh results in a large quantization error, the mesh regions may not be equally important, the number of faces can vary significantly in different parts of the mesh, the base mesh may be simpler than the original mesh and the displacement, and thus may not require much bit-depth accuracy. Therefore, for any of these reasons, a technical solution to such problems arising in video coding technology is desired.

Means for Solving the Problems

[0007] The invention includes a method and apparatus comprising a memory configured to store computer program code, and one or more processors configured to access the computer program code and to operate as instructed by the computer program code. The computer program is configured to cause the processors to implement acquisition code configured to cause at least one processor to acquire volumetric data of at least one three-dimensional (3D) visual content, and derivation code configured to cause at least one processor to derive a mesh from a frame of volumetric data, wherein the mesh comprises derivation code including a plurality of base mesh vertices, determination code configured to cause at least one processor to determine the displacement of at least one vertex that is not a base mesh vertex based on a set of projections from a plurality of base mesh vertices or at least one vertex that is a neighbor of the plurality of base mesh vertices, prediction code configured to cause at least one processor to predict at least one vertex based on at least the determined displacement, and encoding code configured to cause at least one processor to encode volumetric data based on the determination of the predicted at least one vertex.

[0008] According to an exemplary embodiment, the step of encoding volumetric data includes the step of encoding a scalar distance from a projected vertex to at least one vertex.

[0009] According to an exemplary embodiment, a series of projections includes the step of determining that the projected vertex lies on a line from a first base mesh vertex of a plurality of base mesh vertices to a second base mesh vertex of a plurality of base mesh vertices, where the scalar distance is perpendicular to the line.

[0010] According to an exemplary embodiment, a series of projections further includes the step of deriving neighboring vertices as being on a line, and the step of determining that the projected vertices are on a line includes the step of applying a second scalar distance to the neighboring vertices.

[0011] According to an exemplary embodiment, a series of projections includes the steps of determining that a first neighboring vertex lies on a first line from a first base mesh vertex among a plurality of base mesh vertices to a second base mesh vertex among a plurality of base mesh vertices, and determining that the projected vertex is the distance from the first neighboring vertex.

[0012] According to an exemplary embodiment, a series of projections further includes the steps of determining that a second neighboring vertex lies on a second line from a first base mesh vertex among a plurality of base mesh vertices to a third base mesh vertex among a plurality of base mesh vertices, and determining that the projected vertex is the distance from both the first neighboring vertex and the second neighboring vertex.

[0013] According to an exemplary embodiment, the step of encoding volumetric data includes the step of encoding the distance from a projected vertex to at least one vertex.

[0014] According to an exemplary embodiment, a plurality of base mesh vertices define a face of the mesh, and a series of projections further includes the steps of subdividing the face into a plurality of subdivisions and determining the subdivision point that is closest to the projected vertex among the subdivisions.

[0015] According to an exemplary embodiment, the step of encoding volumetric data includes the step of coding the distance from the nearest subdivision point to at least one vertex.

[0016] According to an exemplary embodiment, the mesh faces include quad faces.

[0017] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0018] [Figure 1] This is a schematic diagram of an embodiment. [Figure 2] This is a simplified block diagram of an embodiment. [Figure 3] This is a simplified diagram of an embodiment. [Figure 4] This is a simplified diagram of an embodiment. [Figure 5] This is a simplified diagram of an embodiment. [Figure 6] This is a simplified diagram of an embodiment. [Figure 7] This is a simplified diagram of an embodiment. [Figure 8] This is a simplified diagram of an embodiment. [Figure 9] This is a simplified diagram of an embodiment. [Figure 10] This is a simplified flowchart of an embodiment. [Figure 11] This is a simplified flowchart of an embodiment. [Figure 12] This is a simplified flowchart of an embodiment. [Figure 13] This is a simplified diagram of an embodiment. [Figure 14] This is a simplified diagram of an embodiment. [Figure 15] This is a simplified flowchart of an embodiment. [Figure 16] This is a simplified diagram of an embodiment. [Figure 17] This is a simplified flowchart of an embodiment. [Figure 18] This is a simplified flowchart of an embodiment. [Figure 19] This is a simplified diagram of an embodiment. [Figure 20] This is a simplified flowchart of an embodiment. [Figure 21] This is a simplified diagram of an embodiment. [Figure 22] This is a simplified diagram of an embodiment. [Figure 23] This is a simplified diagram of an embodiment. [Modes for carrying out the invention]

[0019] The proposed features described below may be used separately or combined in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium.

[0020] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may code video data at its local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the coded video data from the other terminal via the network 105, decode the coded data, and display the restored video data. Unidirectional data transmission may be common in applications such as media serving.

[0021] Figure 1 shows a second pair of terminals 101 and 104 provided to support the bidirectional transmission of coded video, for example, during a video conference. For bidirectional data transmission, each terminal 101 and 104 may code video data captured at its local location for transmission to the other terminal via the network 105. Each terminal 101 and 104 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the restored video data on a local display device.

[0022] In Figure 1, terminals 101, 102, 103, and 104 may be represented as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure may include applications involving laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that transmit coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 105 may not be important to the operation of this disclosure unless described below herein.

[0023] Figure 2 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and video decoder in a streaming environment. The disclosed subject matter can be equally applied to other video-enabled applications, including, for example, video conferencing, digital TV, and the storage of compressed video on digital media such as CDs, DVDs, and memory sticks.

[0024] The streaming system may include a capture subsystem 203 which can include a video source 201, such as a digital camera, which creates an uncompressed video sample stream 213. The sample stream 213 may be highlighted as having a higher data volume compared to an encoded video bitstream and can be processed by an encoder 202 coupled to the video source 201, which may be a camera as described above. The encoder 202 may include hardware, software, or a combination thereof to enable or implement embodiments of the disclosed subject, as will be described in more detail below. The encoded video bitstream 204 may be highlighted as having a lower data volume compared to the sample stream and can be stored in the streaming server 205 for future use. One or more streaming clients 212 and 207 can access the streaming server 205 and retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include a video decoder 211 which decodes an incoming copy 208 of the encoded video bitstream and creates an outgoing video sample stream 210 that can be rendered on a display 209 or other rendering device (not shown). In some streaming systems, video bitstreams 204, 206, and 208 can be encoded according to specific video coding / compression standards. Examples of these standards are mentioned above and will be further described herein.

[0025] Figure 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present invention.

[0026] Receiver 302 may receive one or more codec video sequences to be decoded by decoder 300, one coded video sequence at a time in the same or different embodiments, and the decoding of each coded video sequence is independent of other coded video sequences. Coded video sequences may be received from channel 301, which may be a hardware / software link to a storage device that stores encoded video data. Receiver 302 may receive encoded video data together with other data that may be transferred to their respective usage entities (not shown), e.g., coded audio data and / or auxiliary data streams. Receiver 302 may isolate coded video sequences from other data. To counteract network jitter, a buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter, "parser"). Buffer 303 may not be necessary or can be made smaller if receiver 302 is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network. For use on best-effort packet networks such as the internet, buffer 303 may be required and can be relatively large, or advantageously, its size can be adapted.

[0027] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from an entropy-coded video sequence. These symbol categories may include information used to manage the operation of the decoder 300, and potentially information for controlling rendering devices such as a display 312, which are not integral parts of the decoder but can be coupled to it. The rendering device control information may be in the form of supplemental extension information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser 304 may parse / entropy-decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards, and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, and context-dependent or non-context-dependent arithmetic coding. From the coded video sequence, the parser 304 may extract from the coded video sequence a set of at least one subgroup parameters of subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups can include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), and predictive units (PU). The entropy decoder / parser can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the coded video sequence information.

[0028] The parser 304 may perform entropy decoding / analysis operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 may determine whether a particular symbol 313 should be provided to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0029] The reconstruction of symbol 313 may involve multiple different units, depending on the type of coded video picture or its parts (interpicture and intrapicture, interblock and intrablock, etc.), as well as other factors. Which units are involved and how can be controlled by parser 304 using subgroup control information parsed from the coded video sequence. The flow of such subgroup control information between parser 304 and the following multiple units is not illustrated for clarity.

[0030] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units can interact closely with each other and be integrated with each other at least partially. However, for the purpose of illustrating the disclosed subject, the following conceptual subdivision into functional units is appropriate.

[0031] The first unit is the scaler / inverse unit 305. The scaler / inverse unit 305 receives control information from the parser 304 as symbol 313, including the quantized transformation coefficients, as well as the transformation to be used, block size, quantization factor, and quantization scaling matrix. The scaler / inverse unit 305 can output a block containing sample values ​​that can be input to the aggregator 310.

[0032] In some cases, the output samples of the scaler / inverse transform 305 may also relate to intracoded blocks, i.e., blocks that do not use predictive information from a previously reconstructed picture, but can use predictive information from a previously reconstructed portion of the current picture. Such predictive information can be provided by the intrapicture predictive unit 307. In some cases, the intrapicture predictive unit 307 generates a block of the same size and shape as the block being reconstructed, using the surrounding already reconstructed information fetched from the current (partially reconstructed) picture 309. The aggregator 310, in some cases, adds the predictive information generated by the intrapredictive unit 307 to the output sample information provided by the scaler / inverse transform unit 305, sample by sample.

[0033] In other cases, the output samples of the scaler / inverse unit 305 may relate to an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit 306 may access the reference picture memory 308 and fetch samples to be used for prediction. After the fetched samples have been motion-compensated according to the symbols 313 related to the block, these samples may be added to the output of the scaler / inverse unit by the aggregator 310 to generate output sample information (in this case, called residual samples or residual signals). The address in the reference picture memory format from which the motion-compensated unit fetches the prediction samples may be controlled by a motion vector, which can be made available to the motion-compensated unit in the form of a symbol 313 that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when the exact motion vector of the subsample is used, a motion vector prediction mechanism, etc.

[0034] The output samples of the aggregator 310 can be subjected to various loop filtering techniques in the loop filter unit 311. The video compression technique may include in-loop filtering techniques controlled by parameters contained in the coded video bitstream and made available to the loop filter unit 311 as symbols 313 from the parser 304, but it may also respond to metadata obtained during decoding of earlier parts (in decoding order) of the coded picture or coded video sequence, and may also respond to previously reconstructed and loop-filtered sample values.

[0035] The output of the loop filter unit 311 can be output to the rendering device 312, or it can be a sample stream that can be stored in the reference picture memory 557 for use in future interpicture prediction.

[0036] A particular coded picture, once fully reconstructed, can be used as a reference picture for future predictions. Once a coded picture is fully reconstructed and identified as a reference picture (for example, by parser 304), the current reference picture 309 can become part of the reference picture buffer 308, allowing for the reallocation of new current picture memory before initiating the reconstruction of subsequent coded pictures.

[0037] The video decoder 300 may perform decoding operations according to a predetermined video compression technique documented in standards such as ITU-T Rec.H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that it is faithful to the syntax of the video compression technique or standard, as specified in the video compression technique documentation or standard, specifically in the profile documentation therein. The complexity of the coded video sequence may also be required for compliance to be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited by the virtual reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0038] In one embodiment, receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by video decoder 300 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time layer, a spatial layer, or a signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0039] Figure 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0040] The encoder 400 may receive video samples from a video source 401 (not part of the encoder) that can capture video images to be coded by the encoder 400.

[0041] The video source 401 may provide a source video sequence coded by an encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and a suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0042] According to one embodiment, the encoder 400 can encode and compress pictures of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. One function of the controller 402 is to ensure an appropriate coding speed. The controller controls and is functionally coupled to other functional units, as described below. The coupling is not illustrated for clarity. Parameters set by the controller may include rate control-related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, and maximum motion vector search range. Those skilled in the art will readily identify the other functions of the controller 402, as they may relate to the video encoder 400 optimized for a particular system design.

[0043] Some video encoders operate in what is readily recognizable to those skilled in the art as a “coding loop.” In an overly simplified explanation, the coding loop may consist of an encoding portion of encoder 400 (hereafter “source coder”) (which is responsible for creating symbols based on the input picture and reference picture to be coded) and a (local) decoder 406 built into encoder 400 that reconstructs the symbols and creates sample data, which (since any compression between the symbols and the coded video bitstream is reversible in the video compression techniques considered in the disclosed subject) will also create a (remote) decoder. The reconstructed sample stream is input to reference picture memory 405. Since decoding the symbol stream results in a bit-exact outcome regardless of the decoder’s location (local or remote), the contents of the reference picture buffer are also bit-exact between the local encoder and the remote encoder. In other words, the prediction portion of the encoder “sees” the exact same sample values ​​as reference picture samples that the decoder “sees” when using predictions during decoding. This basic principle of the synchronization of a reference picture (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0044] The operation of the “local” decoder 406 may be the same as that of the “remote” decoder 300, which has already been described in detail above in relation to Figure 3. However, also briefly referring to Figure 4, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder 408 and parser 304 may be reversible, the entropy decoding portion of decoder 300, including channel 301, receiver 302, buffer 303, and parser 304, may not be fully performed by the local decoder 406.

[0045] At this point, it can be said that any decoder techniques within the decoder, excluding analysis / entropy decoding, must also necessarily exist in the corresponding encoder in substantially the same functional form. The description of encoder techniques can be omitted as it is the inverse of the comprehensively described decoder techniques. More detailed explanations are necessary only in specific areas, and are provided below.

[0046] As part of its operation, the source coder 403 may perform motion-compensated predictive coding, which predictively codes the input frame by referencing one or more previously coded frames from the video sequence, designated as “reference frames”. In this way, the coding engine 407 codes the difference between the pixel blocks of the input frame and the pixel blocks of the reference frame, which may be selected as a predictive reference to the input frame.

[0047] The local video decoder 406 can decode coded video data of a frame that may be designated as a reference frame, based on symbols created by the source coder 403. The operation of the coding engine 407 may, advantageously, be a lossy process. If coded video data can be decoded by a video decoder (not shown in Figure 4), the reconstructed video sequence may typically be a copy of the source video sequence with some error. The local video decoder 406 can reproduce the decoding process that may be performed by the video decoder on the reference frame and store the reconstructed reference frame in a reference picture memory 405, which may be a cache, for example. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame having common content as the reconstructed reference frame that will be acquired by a far-end video decoder (without transmission errors).

[0048] The predictor 404 may perform a predictive search for the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., which can function as appropriate predictive references for the new picture. The predictor 404 may operate on sample blocks pixel by pixel to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 405, as determined by the search results obtained by the predictor 404.

[0049] The controller 402 can manage the coding operations of the source coder 403, which may be a video coder, including, for example, setting parameters and subgroup parameters used to encode video data.

[0050] The outputs of all the aforementioned functional units may be entropy coded by the entropy coder 408. The entropy coder converts the symbols generated by the various functional units into coded video sequences by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable-length coding, and arithmetic coding.

[0051] Transmitter 409 may buffer the coded video sequence created by entropy coder 408 for transmission over communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 may merge the coded video data from source coder 403 with other data being transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0052] The controller 402 may manage the operation of the encoder 400. During coding, the controller 402 may assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following frame types:

[0053] An intra-picture (I-picture) can be a picture that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs enable various types of intra-pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0054] A predictive picture (P-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses up to one motion vector and reference index to predict the sample values ​​for each block.

[0055] A bidirectional predictive picture (B-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​for each block. Similarly, a multiple predictive picture can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0056] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks can be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded unpredictably or predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded unpredictably via spatial prediction or via temporal prediction referencing one previously coded reference picture. Pixel blocks of picture B may be coded unpredictably via spatial prediction or via temporal prediction referencing one or two previously coded reference pictures.

[0057] The encoder 400 may be, for example, a video coder, and may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Rec.H.265. In its operation, the encoder 400 may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0058] In one embodiment, the transmitter 409 may transmit additional data along with the encoded video. The source coder 403 may include such data as part of the encoded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and redundant slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, and the like.

[0059] FIG. 5 shows a simplified block style workflow diagram 500 of an exemplary viewport-dependent process in an omnidirectional media application format (OMAF) that enables 360-degree virtual reality (VR360) streaming described in the omnidirectional media application format (OMAF).

[0060] In acquisition block 501, when the image data can represent a scene within VR360, video data A such as data of a plurality of images and audio at the same time instance is acquired. In processing block 503, image B at the same time instance a , v , a is processed by one or more of being stitched, mapped to a picture projected with respect to one or more virtual reality (VR) angles or other angles / viewpoints, and packed for each region. In addition, metadata indicating either such processed information or other information is created to assist in distribution and rendering processing.

[0061] Regarding data D, in image encoding block 505, the projected picture is encoded into data E i and configured into a media file. In viewport-independent streaming, in video encoding block 504, the video picture is encoded as data E v as, for example, a single-layer bitstream, and regarding data B a the audio data can also be encoded into data E a in audio encoding block 502.

[0062] Data E a 、E v 、and E i 、the encoded bitstream F iand / or the entire F may be stored on a (Content Delivery Network (CDN) / cloud) server, typically in a delivery block 507, and fully transmitted to the OMAF player 520, and fully decoded by a decoder, thereby rendering to the user in the display block 516 a decoding area of ​​the decoded picture corresponding to at least the current viewport from the head / eye tracking block 508, along with various metadata, file playback, and orientation / viewport metadata, such as the angles the user can see through the VR image device with respect to the viewport specifications of that device. A distinct feature of VR360 is that it can display only the viewport at any given time, and such a feature can be used to improve the performance of the omnidirectional video system through selective delivery (or any other criterion such as recommended viewport timed metadata) depending on the user's viewport. For example, viewport-dependent delivery may be enabled by tile-based video coding according to an exemplary embodiment.

[0063] Similar to the encoding block described above, the OMAF player 520 in an exemplary embodiment uses data F' and / or F' i Furthermore, with respect to decapsulation of one or more files / segments of the metadata, one or more facets of such encoding are similarly reversed, and audio data E' in audio decoding block 510. i , video data E' in video decoding block 513 v , and image data E' in image decoding block 514 i Decode the data B' in audio rendering block 511. a The audio rendering and image rendering block 515 proceeds with the image rendering of data D' in VR360 format according to various metadata such as orientation / viewport metadata, and the display block 516 displays the display data A'. i and audio data A' in speaker / headphone block 512s The output may be configured to include various metadata, which may affect one of the data decoding and rendering processes depending on the various tracks, languages, quality, and views that may be selected by or for the user of the OMAF Player 520. It should be understood that the order of processing described herein is presented for exemplary embodiments and may be carried out in other orders according to other exemplary embodiments.

[0064] Figure 6 shows a simplified block-style content flow process diagram 600 of (coded) point cloud data (hereinafter referred to as "V-PCC") with view position and angle-dependent processing for capturing / generating / (de)coding / rendering / displaying 6-degree-of-freedom media. It should be understood that the described features may be used separately or in any order, and elements such as encoding and decoding may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits), particularly as illustrated, and one or more processors may execute a program stored on a non-temporary computer-readable medium according to the exemplary embodiment.

[0065] Figure 600 shows an exemplary embodiment for streaming coded point cloud data using V-PCC.

[0066] In volumetric data acquisition block 601, a real-world visual scene or a computer-generated visual scene (or a combination thereof) can be captured by a set of camera devices or synthesized by a computer as volumetric data, which may have any format. This volumetric data can then be converted to a (quantized) point cloud data format via image processing in the point cloud conversion block 602. For example, according to an exemplary embodiment, the data from the volumetric data may be area data converted to one of the points of a point cloud by pulling one or more of the values ​​described below from the volumetric data and any related data into a desired point cloud format. According to an exemplary embodiment, the volumetric data may be a 3D dataset of 2D images, such as slices onto which a 2D projection of the 3D dataset can be projected. According to exemplary embodiments, a point cloud data format may include a representation of data points in one or more different spaces and may be used to represent volumetric data and may provide improvements with respect to sampling and data compression with respect to temporal redundancy, for example, point cloud data in x,y,z format may represent color values ​​(e.g., RGB), brightness, intensity, etc., at each of multiple points in the cloud data and may be used in conjunction with progressive decoding, polygon meshes, direct rendering, and octree 3D representations of 2D quadtree data.

[0067] In the projection block 603 onto the image, the acquired point cloud data may be projected onto a 2D image and encoded as an image / video picture using video-based point cloud coding (V-PCC). The projected point cloud data may consist of attributes, geometry, occupancy maps, and other metadata used to reconstruct the point cloud data using, for example, a painter's algorithm, a raycasting algorithm, a (3D) binary space partitioning algorithm, etc.

[0068] Meanwhile, in the scene generator block 609, the scene generator may generate several metadata used to render and display 6-degree-of-freedom (DoF) media, for example, depending on the director's intent or the user's preference. Such 6DoF media may include 360VR, such as a 3D view of the scene from rotational changes on the 3D axes X, Y, Z, in addition to additional dimensions that enable forward, backward, up, and left and right movement within, or at least corresponding to, the point cloud coded data for the virtual experience. Scene description metadata defines one or more scenes consisting of coded point cloud data and other media data, including VR360, light fields, audio, etc., and may be provided to one or more cloud servers and / or file / segment encapsulation / deencapsulation processes, as shown in Figure 6 and related descriptions.

[0069] Following the video encoding block 604 and image encoding block 605, similar to the video and image encoding described above (as understood, audio encoding may also be provided as described above), the file / segment encapsulation block 606 processes the encoded point cloud data so that it consists of a media file for file playback, or a sequence of initialization segments, and media segments for streaming according to a specific media container file format, such as one or more video container formats, and such descriptions may be used with respect to the DASH described later, such as representing exemplary embodiments. The file container may also contain scene description metadata, such as from the scene generator block 1109, in the file or segment.

[0070] According to an exemplary embodiment, a file is encapsulated according to scene description metadata so that such a file can be transmitted in response to a request in response to user or author input, and each contains at least one view position and at least one or more angular views at the view position, each at one or more times in a 6DoF medium. Furthermore, according to an exemplary embodiment, a segment of such a file may contain one or more parts of such a file, e.g., a portion of a 6DoF medium showing a viewpoint and an angle at one or more times, but these are merely exemplary embodiments and may be modified depending on various conditions such as the network, user, author capabilities and input.

[0071] According to an exemplary embodiment, point cloud data is partitioned into multiple 2D / 3D regions that are independently coded in one or more of the video encoding block 604 and the image encoding block 605. Each partition of the independently coded point cloud data can then be encapsulated in the file / segment encapsulation block 606 as a track within a file and / or segment. According to an exemplary embodiment, each point cloud track and / or metadata track may contain some useful metadata for view position / angle-dependent processing.

[0072] According to an exemplary embodiment, metadata useful for view position / angle-dependent processing, such as that contained in encapsulated files and / or segments with respect to file / segment encapsulation blocks, includes one or more of the following: layout information of 2D / 3D partitions with an index; (dynamic) mapping information that associates a 3D volume partition with one or more 2D partitions (e.g., any of tiles / tile groups / slice / subpictures); the 3D position of each 3D partition on a 6DoF coordinate system; a representative view position / angle list; a selected view position / angle list corresponding to the 3D volume partition; an index of the 2D / 3D partition corresponding to the selected view position / angle; quality (rank) information for each 2D / 3D partition; and rendering information for each 2D / 3D partition according to each view position / angle, for example. By retrieving such metadata when requested, such as by a V-PCC player user or by a content creator on behalf of a V-PCC player user, more efficient processing of a desired portion of 6DoF media with respect to such metadata may be possible, thereby enabling the V-PCC player to deliver high-quality images that focus on the 6DoF media portion rather than the unused portion of the media.

[0073] From the file / segment encapsulation block 606, a file or one or more segments of a file may be delivered directly to either the V-PCC player 625 or a cloud server such as the cloud server block 607 using a delivery mechanism (e.g., Dynamic Adaptive Streaming over HTTP (DASH)), the cloud server may extract one or more tracks and / or one or more specific 2D / 3D partitions from the file and merge multiple coded point cloud data into a single data.

[0074] According to data such as the position / field of view tracking block 608, if the current field of view position and angle are defined in a 6DoF coordinate system in the client system, then in the cloud server block 607, the view position / angle metadata may be delivered from the file / segment encapsulation block 606 or processed in other ways from a file or segment already present on the cloud server. As a result, the cloud server may extract the appropriate partitions from the store file based on metadata from a client system, for example, having a V-PCC player 625, merge them (if necessary), and deliver the extracted data to the client as a file or segment.

[0075] With respect to such data, in the file / segment decapsulation block 615, the file decapsulator processes the file or received segment, extracts the coded bitstream, parses the metadata, and in the video decoding and image decoding blocks 610 and 611, the coded point cloud data is then decoded and reconstructed into point cloud data in the point cloud reconstruction block 612, the reconstructed point cloud data can be displayed in the display block 614, and / or can be initially configured in the scene configuration block 613 with respect to scene description data according to one or more different scene descriptions in the scene generator block 609.

[0076] In consideration of the foregoing, such exemplary V-PCC flows represent advantages over the V-PCC standard, including one or more of the following: the ability to partition described for multiple 2D / 3D areas, the ability to compress domain assembly of coded 2D / 3D partitions into a single conforming coded video bitstream, and the ability to extract coded 2D / 3D bitstreams of coded pictures into a conforming coded bitstream. Such V-PCC system support is further enhanced by including containerization for VVC bitstreams to support a mechanism containing metadata that carries one or more of the metadata described above.

[0077] From that perspective, according to the exemplary embodiments further described below, the term “mesh” or “input mesh” as initially obtained “mesh” refers to a configuration of one or more polygons describing the surface of a volumetric object. Each vertex of the mesh can be signaled as part of the volumetric data. Each polygon is defined by its vertices in 3D space and information on how those vertices are connected, called connectivity information. The vertices of the “input mesh” can be considered as “base mesh vertices” as a base for considering various displacement vectors to other vertices of “projected” and “predicted” vertices for, for example, compression. Optionally, vertex attributes such as color and normals may be associated with the mesh vertices. Attributes may also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such a mapping is called UV coordinates or texture coordinates and may be described by a set of parametric coordinates associated with the mesh vertices. The 2D attribute map is used to store high-resolution attribute information such as texture, normals, and displacements. Such information may be used for various purposes, such as texture mapping and shading, according to the exemplary embodiments.

[0078] Nevertheless, dynamic mesh sequences can require large amounts of data because they can consist of a considerable amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with always-on connectivity and time-varying geometry and vertex attributes. However, these standards do not take into account time-varying attribute maps and connectivity information. DCC (Digital Content Creation) tools typically generate such dynamic meshes. In response, volumetric acquisition techniques struggle to generate always-on connectivity dynamic meshes, especially under real-time constraints. This type of content is not supported by existing standards. According to exemplary embodiments herein, a new mesh compression standard is described for directly processing dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps, the standard covering lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, AR, and VR. Features such as random access and scalable / progressive coding are also considered.

[0079] Figure 7 represents an exemplary framework 700 for one dynamic mesh compression method, such as a 2D atlas sampling-based method. Each frame of the input mesh 701 can be preprocessed by a series of operations such as tracking, remeshing, parameterizing, and voxelizing. Note that these operations can be performed by the encoder only, meaning they may not be part of the decoding process, and such possibility may be signaled in the metadata by a flag, such as indicating 0 only to the encoder and 1 to others. A mesh 702 with a 2D UV atlas can then be obtained, where each vertex of the mesh has one or more associated UV coordinates on the 2D atlas. The mesh can then be converted into multiple maps, including geometry maps and attribute maps, by sampling on the 2D atlas. These 2D maps can then be coded by video / image codecs such as HEVC, VVC, AV1, and AVS3. On the decoder side 703, the mesh can be reconstructed from the decoded 2D maps. Any post-processing and filtering can also be applied to the reconstructed mesh 704. Note that metadata may be signaled to the decoder side for the purpose of 3D mesh reconstruction. Note that chart boundary information, including the UV and XYZ coordinates of boundary vertices, can be predicted, quantized, and entropy coded within the bitstream. The quantization step size can be configured on the encoder side due to the trade-off between quality and bitrate.

[0080] In some implementations, a 3D mesh can be partitioned into several segments (or patches / charts), and one or more 3D mesh segments can be considered a “3D mesh” according to the exemplary embodiment. Each segment consists of a set of connected vertices associated with their geometry, attributes, and connectivity information. As shown in the example volumetric data 800 in Figure 8, a UV parameterization process 802 that maps from 3D mesh segments to 2D charts, such as the 2D UV atlas 702 block described above, maps one or more mesh segments 801 to a 2D chart 803 in the 2D UV atlas 804. Each vertex (v) in the mesh segment n ) will be assigned 2D UV coordinates within the 2D UV atlas. Vertices (v) in the 2D chart n It should be noted that these vertices form connecting components as their 3D counterparts. The geometry, attributes, and connectivity information of each vertex can also be inherited from their 3D counterparts. For example, it may indicate that vertex v4 is directly connected to vertices v0, v5, v1, and v3, and similarly, the information of each other vertex may be indicated. Furthermore, such a 2D texture mesh may, according to an exemplary embodiment, further indicate information such as color information for each patch of triangles, for example, v2, v5, v3, etc., as a single "patch".

[0081] For example, see Example 900 in Figure 9, which shows that in addition to the features of Example 800 in Figure 8, a 3D mesh segment 801 can also be mapped to multiple separate 2D charts 901 and 902. In this case, vertices in 3D can also correspond to multiple vertices in the 2D UV atlas. As shown in Figure 9, the same 3D mesh segment is mapped to multiple 2D charts in the 2D UV atlas, instead of a single chart as in Figure 8. For example, 3D vertices v1 and v4 have two 2D correspondences v1, v1 and v4, v4, respectively. Thus, a typical 2D UV atlas of a 3D mesh may consist of multiple charts, as shown in Figure 14, each chart may contain multiple (usually three or more) vertices associated with their 3D geometry, attributes, and connectivity information.

[0082] Figure 9 shows Example 903, which illustrates the derived triangulation in a chart having boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7. Given such information, any triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). For example, for each vertex, find the two closest vertices. Or, for all vertices, generate triangles sequentially until a minimum number of triangles are achieved after a set number of trials. As shown in Example 903, there are various regular-shaped repeating triangles and various irregular triangles that are generally closest to boundary vertices and have their own dimensions, which may or may not be shared with other triangles. Connectivity information can also be reconstructed by explicit signaling. If the polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream according to an exemplary embodiment.

[0083] The boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7 are defined in 2D UV space. Boundary edges can be determined by checking whether the edge appears in only one triangle. The following boundary vertex information is important and should be signaled in a bitstream according to the exemplary embodiment, and the information is geometric information, e.g., 3D XYZ coordinates and 2D UV coordinates, even if currently in 2D UV parametric format.

[0084] When boundary vertices in 3D correspond to multiple vertices in a 2D UV atlas, the mapping from 3D XUZ to 2D UV can be one-to-many, as shown in Figure 9. Therefore, a UV-to-XYZ (or UV2XYZ) index can be signaled to indicate the mapping function. UV2XYZ may be a 1D array of indices that map each 2D UV vertex to a 3D XYZ vertex.

[0085] According to exemplary embodiments, in order to efficiently represent the mesh signal, a subset of mesh vertices may be coded first, along with connectivity information between them. In the original mesh, since these vertices are subsampled from the original mesh, connectivity between them may not exist. There are various ways to signal connectivity information between vertices, and therefore such a subset is called the base mesh or base vertices.

[0086] According to exemplary embodiments, several methods are employed for dynamic mesh compression, which are part of the edge-based vertex prediction framework described above, where a base mesh is first coded, and then more additional vertices are predicted based on connectivity information from the edges of the base mesh. It should be noted that the methods may be applied individually or in any combination.

[0087] For example, consider vertex grouping for the example prediction mode flowchart 1001 in Figure 10. In S201, vertices in the mesh may be obtained, and in S202, they may be divided into different groups for prediction purposes, see, for example, Figure 9. In one example, the division is performed in S204 using patch / chart partitioning. In another example, the division is performed under each patch / chart in S205. The decision to proceed to S204 or S205 in S203 may be signaled by a flag or the like. In the case of S205, some vertices in the same patch / chart may form a prediction group and share the same prediction mode, while some other vertices in the same patch / chart may use a different prediction mode. Here, “prediction mode” can be thought of as a specific mode that the decoder uses to make predictions of video content containing patches, and prediction modes can be categorized into intra-prediction modes and inter-prediction modes, and within each category there may be different specific modes that the decoder selects. According to exemplary embodiments, each group, “prediction group,” may share the same specific mode (e.g., an angular mode at a specific angle) or the same category prediction mode (e.g., all intra-prediction modes, but capable of predicting at different angles) according to exemplary embodiments. Such grouping in S206 can be assigned at different levels by determining the number of vertices each group contains. For example, every 64, 32, or 16 vertices in the scan order within the patch / chart may be assigned the same prediction mode according to exemplary embodiments, while other vertices are assigned differently. For each group, the prediction mode can be an intra-prediction mode or an inter-prediction mode, which can be signaled or assigned.According to the example flowchart 1000, if it is determined in S207 that a mesh frame or mesh slice is of an intra type, for example by checking whether the flag of the mesh frame or mesh slice indicates an intra type, then all vertex groups within that mesh frame or mesh slice shall use the intra prediction mode. Otherwise, in S208, either the intra prediction mode or the inter-prediction mode may be selected for each group of vertices within it.

[0088] Furthermore, in the case of a group of mesh vertices using intra-prediction mode, the vertices can only be predicted by using previously coded vertices within the same subpartition of the current mesh. Sometimes, the subpartition can be the current mesh itself, according to an exemplary embodiment, and in the case of a group of mesh vertices using inter-prediction mode, the vertices can only be predicted by using previously coded vertices from another mesh frame, according to an exemplary embodiment. Each of the above pieces of information can be determined and signaled by flags, etc. The prediction features may occur in S210, and the results of the prediction and signaling may occur in S211.

[0089] According to an exemplary embodiment, for each vertex in a group of vertices in exemplary flowchart 1000 and flowchart 1100 described later, after prediction, the residual is a 3D displacement vector indicating the shift from the current vertex to its predictor. The residuals of a group of vertices need to be further compressed. In one example, the transformation in S211, along with its signaling, can be applied to the residuals of a group of vertices before entropy coding. To handle the coding of groups of displacement vectors, the following methods may be employed. For example, one method appropriately signals when a group of displacement vectors, some displacement vectors, or its components have only zero values. In another embodiment, a flag is signaled for each displacement vector to indicate whether it has non-zero components, and if not, the coding of all components of this displacement vector can be skipped. Furthermore, in yet another embodiment, a flag is signaled for each group of displacement vectors to indicate whether it has non-zero vectors, and if not, the coding of all displacement vectors in this group can be skipped. Furthermore, in another embodiment, a flag is signaled for each component of the group of displacement vectors to indicate whether this component of the group has any non-zero vectors, and if not, the coding of this component for all displacement vectors s in this group can be skipped. Furthermore, in another embodiment, there may be signaling for cases where a group of displacement vectors or a component of a group of displacement vectors requires a transformation, and if not, the transformation can be skipped, and quantization / entropy coding can be applied directly to the group or group component. Furthermore, in another embodiment, a flag may be signaled for each group of displacement vectors to indicate whether this group needs to undergo a transformation, and if not, the transformation coding for all displacement vectors in this group can be skipped. Furthermore, in another embodiment, a flag may be signaled for each component of the group of displacement vectors to indicate whether this component of the group needs to undergo a transformation, and if not, the transformation coding for this component for all displacement vectors in this group can be skipped.The embodiments described above in this paragraph relating to the processing of vertex prediction residuals can also be combined and implemented in parallel on different patches.

[0090] Figure 11 shows an illustrative flowchart 1150, which means that in S221, a mesh frame can be coded and obtained as a whole data unit, and all vertices or attributes of the mesh frame may be correlated with each other. Alternatively, depending on the decision in S222, the mesh frame can be divided in S223 into smaller, independent subpartitions, similar in concept to slices or tiles of a 2D video or image. A prediction type can be assigned to the coded mesh frame or coded mesh subpartition in S224. Possible prediction types include intra-coding types and inter-coding types. For intra-coding types, only predictions from reconstructed portions of the same frame or slice are allowed in S225. Inter-prediction types, on the other hand, allow predictions from previously coded mesh frames in addition to intra-mesh frame predictions in S225. Inter-prediction types may also be further classified into more subtypes, such as P-type and B-type. In P-type, only one predictor can be used for prediction purposes, while in B-type, two predictors from two previously coded mesh frames may be used to generate a predictor. A weighted average of two predictors can be one example. When a mesh frame is coded as a whole, the frame can be considered an intra-coded or intercoded mesh frame. In the case of an intermesh frame, the P or B type may be further identified via signaling. Alternatively, if the mesh frame is further divided and coded within the frame, a prediction type is assigned to each subpartition in S224. Each of the above pieces of information may be determined and signaled by flags, etc., and the prediction features may occur in S226, and the results of the prediction and signaling may occur in S227, as in S210 and S211 in Figure 10.

[0091] Therefore, since dynamic mesh sequences can consist of a considerable amount of information that changes over time, they may require a large amount of data. Efficient compression techniques are needed to store and transmit such content, and the features described above for Figures 20 and 21 represent such improved efficiency by enabling at least improved mesh vertex 3D position prediction by using either previously decoded vertices within the same mesh frame (intra-prediction) or previously decoded vertices from a previously coded mesh frame (inter-prediction).

[0092] Furthermore, exemplary embodiments may generate displacement vectors for a third layer 1303 of a mesh based on one or more reconstructed vertices of a preceding layer, such as a second layer 1302 and a first layer 1301. Assuming the index of the second layer 1302 is T, the predictor T+1 of the vertices of the third layer 1303 is generated based on at least the reconstructed vertices of the current layer or the second layer 1302. An example of such a layer-based prediction structure is shown in Example 1300 of Figure 13, which demonstrates reconstruction-based vertex prediction, i.e., progressive vertex prediction using edge-based interpolation, where the predictor is generated based on previously decoded vertices rather than predictor vertices. The first layer 1301 may be a mesh bounded by a first polygon 1340 having, as its vertices, vertices decoded along its boundaries and vertices interpolated along one of the lines between those decoded vertices. As progressive coding progresses from the first layer 1301 to the second layer 1302, an additional polygon 1341 may be formed by a displacement vector from one of the interpolated vertices of the first layer to an additional vertex in the second layer 1302, so that the total number of vertices in the second layer 1302 may be greater than the total number of vertices in the first layer 1301. Similarly, when progressing to the third layer 1303, the additional vertices in the second layer 1302, along with the decoded vertices from the first layer 1301, may function in coding in the same way as the decoded vertices that functioned when progressing from the first layer 1301 to the second layer 1303, i.e., multiple additional polygons may be formed. As a note, unlike Figure 13, see Example 1400 in Figure 14, which illustrates such progressive coding, in which each of the additionally formed polygons can be entirely within the polygon formed by the extent of the first layer 1401 as you progress from the first layer 1401 to the second layer 1403 and then to the third layer 1403.

[0093] For such examples 1300 and / or 1400, according to an exemplary embodiment, reference is made to the exemplary flowchart 1200 of FIG. 12, where the interpolated vertices on the current layer are prediction values, so such values need to be reconstructed before being used to generate predictors for vertices on the next layer. This is done by coding the base mesh at S231, performing vertex prediction at S232, and then adding the decoded displacement vectors of the current layer to predictors for vertices such as layer 1302 at S233. Next, the reconstructed vertices of this layer 2303 can be used, together with all the decoded vertices of the previous layer, such as checking for additional vertex values of such a layer at S234, to generate and signal predictor vertices for the next layer 1303 at S235. This process can also be summarized as follows. Let P[t](Vi) represent the predictor for vertex Vi on layer t, R[t](Vi) represent the reconstructed vertex Vi on layer t, D[t](Vi) represent the displacement vector of vertex Vi on layer t, and f(*) represent the predictor generator, which can in particular be the average of two existing vertices. In that case, according to an exemplary embodiment, the following exists for each layer t. P[t](Vi)=f(R[s|s<t](Vj), R[m|m<t](Vk)), where Vj and Vk are the reconstructed vertices of the previous layer R[t](Vi)=P[t](Vi)+D[t](Vi) Equation (1)

[0094] Next, for all vertices of a mesh frame, they are divided into layers 0 (base mesh), layer 1, layer 2,.... Then, the reconstruction of vertices in one layer depends on the reconstruction of vertices in the previous layer. Above, each of P, R, and D represents a 3D vector in the context of a 3D mesh representation. D is the decoded displacement vector, and quantization may or may not be applied to this vector.

[0095] According to an exemplary embodiment, vertex prediction using reconstructed vertices may be applied to only certain layers, for example, layers 0 and 1. For other layers, vertex prediction can still use neighbor predictor vertices without adding displacement vectors to them for reconstruction. Thus, these other layers can be processed simultaneously without waiting for the previous layer to be reconstructed. According to an exemplary embodiment, each layer can be signaled to choose reconstruction-based vertex prediction or predictor-based vertex prediction, or to signal a layer (and subsequent layers) that does not use reconstruction-based vertex prediction.

[0096] For displacement vectors generated by reconstructed vertices, quantization can be applied to them without further transformations such as wavelet transforms. For displacement vectors generated by other predictor vertices, transformations may be necessary, and quantization can be applied to the transformation coefficients of those displacement vectors.

[0097] Therefore, dynamic mesh sequences can require large amounts of data, as they can consist of a considerable amount of information that changes over time. Consequently, efficient compression techniques are needed to store and transmit such content. In the interpolation-based vertex prediction method framework described above, compressing the displacement vectors is one of the key steps, which constitute the majority of the coded bitstream and is the focus of this disclosure. The feature shown in Figure 15, for example, mitigates such problems by providing such compression.

[0098] Furthermore, as with the other examples described above, even in these embodiments, dynamic mesh sequences can still require a large amount of data because they can consist of a considerable amount of information that changes over time, and therefore efficient compression techniques are needed to store and transmit such content. In the framework of the 2D atlas sampling-based method described above, a significant advantage can be achieved by inferring connectivity information from the sampled vertices and boundary vertices on the decoder side. This is a key part of the decoding process and is the focus of the further examples described below.

[0099] According to an exemplary embodiment, connectivity information for the base mesh can be inferred (derived) from the decoded boundary vertices and sampled vertices of each chart on both the encoder and decoder sides.

[0100] As described above, any triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). According to exemplary embodiments, connectivity types can be signaled with high-level syntax such as sequence headers and slice headers.

[0101] As described above, connectivity information can also be reconstructed by explicitly signaling, such as with irregularly shaped triangular meshes. That is, if it is determined that a polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream. Furthermore, according to exemplary embodiments, the overhead of such explicit signaling can be reduced depending on the polygon boundaries.

[0102] According to the embodiment, it is determined that only connectivity information between boundary vertices and sampled locations is signaled, and connectivity information between the sampled locations themselves is inferred.

[0103] Furthermore, in any embodiment, connectivity information may be signaled by prediction such that only the difference between the predicted connectivity (as a prediction) from one mesh to another can be signaled in the bitstream.

[0104] As a note, the inferred triangle orientation (e.g., whether each triangle is inferred to be clockwise or counterclockwise) can be signaled for all charts in high-level syntax such as sequence headers or slice headers, or it can be fixed (assumed) by encoders and decoders in exemplary embodiments. The inferred triangle orientation can also be signaled differently for each chart.

[0105] As a further note, any reconstructed mesh may have different connectivity than the original mesh. For example, the original mesh may be a triangular mesh, while the reconstructed mesh may be a polygon mesh (e.g., a quad mesh).

[0106] According to the exemplary embodiment, connectivity information for arbitrary base vertices does not need to be signaled; instead, edges between base vertices may be derived using the same algorithm on both the encoder and decoder sides. Also according to the exemplary embodiment, interpolation of predicted vertices for additional mesh vertices may be based on the derived edges of the base mesh.

[0107] According to an exemplary embodiment, a flag may be used to signal whether base vertex connectivity information should be signaled or derived, and such a flag may be signaled at different levels of the bitstream, such as the sequence level or the frame level.

[0108] According to an exemplary embodiment, the edges between base vertices are first derived using the same algorithm on both the encoder and decoder sides. Then, the difference between the derived edges and the actual edges is signaled by comparing them with the original connectivity of the base mesh vertices. Thus, after decoding the difference, the original connectivity of the base vertices can be restored.

[0109] For example, if a derived edge is determined to be incorrect when compared to the original edge, such information may be signaled in a bitstream (by indicating the pair of vertices that form the edge), and if the original edge is not derived, it may also be signaled in a bitstream (by indicating the pair of vertices that form the edge). Furthermore, connectivity on boundary edges and vertex interpolation including boundary edges may be performed separately from internal vertices and edges.

[0110] Accordingly, the technical problems pointed out above can be advantageously improved by one or more of the exemplary embodiments described herein. For example, since dynamic mesh sequences can consist of a considerable amount of information that changes over time, they may require large amounts of data, and therefore the exemplary embodiments described herein represent at least efficient compression techniques for storing and transmitting such content.

[0111] The embodiments described above may be further applied to instance-based mesh coding, where instances may be the mesh of an object or part of an object. For example, the illustrated example 1600 in Figure 16 shows mesh example 1601, each of which can be separated and coded, with various instances 1602 (representing the mesh of a cup), 1603 (representing the mesh of a spoon), and 1604 (representing the mesh of a plate). Each of instances 1601, 1602, 1603, and 1604 is shown with respect to the respective bounding boxes, which will be further described below. Note that instance 1601 can be thought of as being bounded by a "mesh-based bounding box," and each of instances 1602, 1603, and 1604 can be thought of as being shown as bounding by the respective "instance-based bounding boxes."

[0112] According to exemplary embodiments, the proposed method may be used separately or combined in any order. The proposed method may be used with any polygon mesh, but only triangular meshes may be used to demonstrate various embodiments. As stated above, the input mesh may contain one or more instances, and a submesh is a part of the input mesh having one or more instances, and multiple instances can be grouped to form a submesh.

[0113] From that perspective, Figure 15 shows Example 1500 in which it is proposed to separately quantize different objects or parts at a given input bit depth (which may be called "QP"). For example, in S1501, one or more input meshes can be taken and each separated into multiple submeshes. The submeshes can be objects, instances of objects, or segmented regions, which are independently quantized in S1502 according to the exemplary embodiment.

[0114] According to an exemplary embodiment, a mesh M having m points in (x, y, z) coordinates can be quantized by a QP bit depth in S1502. The quantization step size for all three dimensions (x, y, z) can be determined based on the maximum length of the bounding box in all dimensions, d bbox >0. Also, the same quantization step size may be applied in S1504 to all objects in the mesh identified in S1503.

number

number

number

number

number

[0115] However, in complex scenes, the largest object, the background, may often be relatively simple and can tolerate a higher quantization step size. The main object, on the other hand, is smaller and suffers from a large quantization error, which can be explained by various embodiments described further below.

[0116] Therefore, as shown in Example 1500 of Figure 15, the bounding box d of the input mesh bbox The maximum length is always the bounding box of each instance, as follows:

number

number

number

[0117] In a given bit-depth QP, the quantization step size for each instance of instance 1602 (representing the cup mesh), 1603 (representing the spoon mesh), and 1604 (representing the plate mesh) is always:

number

[0118] Therefore, the quantization error for each instance becomes smaller, and the overall quantization error becomes smaller.

[0119] In various embodiments, as seen in the flowchart 1700 of Figure 17, the bit depth may be adaptively assigned to each instance / region called a “submesh” in S1702, and may be determined based on the face density of that particular instance. Each submesh may be obtained from volumetric data of a mesh, which itself may individually signal each instance within the mesh, and each submesh is derived from that mesh for each instance in S1702. For example, each of instances 1602, 1603, and 1604 may be assigned its own bit depth in S1704, depending on its particular face density or number of vertices, forming one or more of the aforementioned polygons within it. In general, the more faces each instance has, which can be determined in S1703 by counting the number of such polygons within it, the less quantization should be applied to that instance in S1702. For example, given a mesh M, the total number of faces is n, and the faces corresponding to the k-th submesh are n satisfying the following: k And,

number

number

number

number

[0120] According to various embodiments, the mesh is represented as a base mesh B and its corresponding displacement D, and is quantized at different bit depths in S1702. For example, for the k-th object, the bitdepth base mesh

number

number

number

[0121] According to various embodiments, an adaptive bit depth parameter based on minimizing distortion may be used. For example, given an input bitdepthQP, the mean squared error (MSE) of the quantization method is

number

number

number

[0122] In addition, the best bitdepth for displacement can also be obtained via the following:

number

[0123] According to an exemplary embodiment, there may be signaling of the quantization of each object, such as signaling the bitdepth via the bitstream with the signal in S1707. The ascending set of base quantization bit depths is:

number

number

[0124] [Table 1]

[0125] Here, u(n) is an unsigned integer using n bits, i(n) is an integer using n bits, and mips_quant() is a sequence of signaling data. mips_min_bbox[k] is the minimum bounding box in i dimensions, mips_num_instances_minus1 is the number of instances minus 1 in the mesh. mips_base_bitdepth_minus1 is the bit depth of the first instance in this order, mips_base_quant[k] is the difference between the quantizations of the (k+1)th and kth submesh. Since the quantization sets are sorted in ascending order, this number is always non-negative. mips_dist_quant[k] is the k-th quantization data of the bithdepth of the base mesh.

[0126] In various embodiments, to reduce signaling overhead, multiple instances may be grouped into K groups with the same bit depth. The instances are bounded by a simple clustering method such as K-means clustering.

number

[0127] However, since such 3D scenes can often consist of multiple instances with similar mesh structures that reuse assets, further improvements can be achieved not only by utilizing local properties but also by considering the similarities between instances as in exemplary embodiments.

[0128] For example, continuing from above, Figure 18 shows an exemplary flowchart 1800 that uses an instance-based matching prediction (IMP) method to find redundant meshes and encode the corresponding displacements, which may favorably normalize instances to maximize their similarity and may be used in any of the embodiments described above.

[0129] For example, in S1801, an input mesh is acquired and can be partitioned into multiple submeshes as described above. According to exemplary embodiments, a submesh may be an instance of an individual object or a part of an object.

[0130] In S1802, instances can be grouped into similarity groups using simple scaling features and similarity measures. For example, according to one embodiment, instances can be aligned and normalized so that only transient assets can be reused. Scale and orientation information can be signaled via an IMP mode channel, and given an input mesh M having m instances, the i-th instance is,

number

[0131] According to an exemplary embodiment, in S1803, in accordance with the flag instruction, IMP is used to specify a size greater than 1, i.e., m k Asset groups that are >1

number

[0132] [Table 2]

[0133] In the case of lossless compression,

number

number

[0134] Depending on flag S1804, coding in S1805 may be signaled to code submeshes independently into subbitstreams. Each submesh can be coded by a mesh codec having different coding parameters. Each submesh can also be coded by a different mesh codec, in which case it should be noted that a codec index indicating which mesh codec is used must be signaled in the subbitstream header or elsewhere. According to an exemplary embodiment, the subbitstreams of submeshes can be encoded and decoded in parallel without data dependency issues.

[0135] If the flag in S1804 instead indicates dependent coding, an additional flag may be considered in S1806 to also indicate a mode of dependent coding of submeshes. For example, according to one embodiment, a submesh can be coded by predictions from other submeshes that have already been coded. The prediction index can be coded to indicate which submesh to use as the prediction. The prediction index can be signaled at different levels.

[0136] For example, in S1807, only one prediction index may be coded for the entire submesh, and as a result, all vertices in the current submesh are predicted from the same submesh as indicated by the index in the exemplary embodiment.

[0137] In S1808, a prediction index can be signaled for each vertex of the current submesh so that each vertex can be predicted from a different submesh. Note that the prediction index can also be coded by prediction coding, and the prediction index of a vertex can be predicted from neighboring coded vertices. Next, according to an exemplary embodiment, the prediction index residual can be coded by arithmetic coding.

[0138] In S1809, the prediction index may be signaled at an intermediate level between the vertex level and the submesh level, for example, at the vertex group level, where groups of vertices share the same prediction index. It is also possible to code the prediction indices of different groups using prediction coding according to the exemplary embodiment. Signaling may be performed in S1810.

[0139] Next, given a predicted index for each vertex of the current submesh, each vertex can be predicted from the corresponding vertex in the submesh, as described herein using one of the exemplary embodiments. According to the exemplary embodiments, the rigid motion may be estimated from the predicted submesh to the current submesh, and the parameters of the rigid motion (e.g., rotation and translation parameters) can be coded. Then, after applying the rigid motion to the predicted submesh, the residuals of the attributes of the current vertex can be obtained by subtracting the attributes of the corresponding vertex in the transformed predicted submesh. The vertex attributes may include, but are not limited to, geometry, color, normal, UV coordinates, connectivity, etc. Next, the residual information can be coded by arithmetic coding according to the exemplary embodiments.

[0140] Furthermore, such coding may be applied to the coding of material and texture information for each submesh. This information may include, but is not limited to, ambient color, diffuse color, specular color, focus of specular highlights, dissolve coefficient, lighting model, and texture image ID.

[0141] According to an exemplary embodiment, one submesh may allow only one set of material and texture information, in which case this information can be simply coded in the header of the subbitstream.

[0142] Alternatively, according to an exemplary embodiment, a submesh may have two or more sets of material and texture information, in which case these sets can be coded in the header of the subbitstream. Note that these parameters in different sets can be coded independently or dependently. When dependent coding is applied, predictions can be applied, and instead, predicted residuals of the material parameters can be coded. Then, for each vertex in the submesh, a material ID can be coded to indicate which set of material information is used for that vertex. Note that, according to an exemplary embodiment, to reduce redundancy, the material ID (identifier) ​​can be coded by prediction from coded neighboring vertices.

[0143] Therefore, according to embodiments of this specification, a complex mesh may contain information about multiple instances for relating and associating texture maps, and since this information may be available at encoding time, each instance may be considered a 3D asset, as the asset may be frequently used in 3D design according to exemplary embodiments of this specification to reduce the cost of designing models, particularly for the synthesis of complex scenes. For example, a 3D model may be reused with or without modified textures due to differences in scale, orientation, etc. This addresses the problems associated with matching and retrieving 3D objects, as the mesh may be normalized to position, size, and orientation based on PCA (Principal Component Analysis) or bilateral planes of symmetry, thereby allowing for more efficient retrieval.

[0144] Figure 19 shows an example of block partitioning using a quadtree-binary tree (QTBT) 1901 and its corresponding tree representation 1902. Solid lines represent quadtree partitions, and dotted lines represent binary tree partitions. At each partition (i.e., non-leaf) node in the binary tree, one flag is signaled to indicate which partition type is used (i.e., horizontal or vertical), where 0 indicates a horizontal partition and 1 indicates a vertical partition. In the case of a quadtree partition, there is no need to indicate the partition type because the quadtree partition always divides the block both horizontally and vertically, producing four subblocks of the same size.

[0145] A coding tree unit (CTU) is divided into coding units (CUs) by using a quadtree structure represented as a coding tree to suit various local characteristics. The decision of whether to use interpicture (temporal) or intrapicture (spatial) prediction to code a picture area is made at the CU level. Each CU can be further divided into one, two, or four prediction units (PUs) according to the PU partition type. Within a single PU, the same prediction process is applied, and relevant information is transmitted to the decoder on a PU basis. After obtaining residual blocks by applying the prediction process based on the PU partition type, the CU can be partitioned into transformation units (TUs) according to another quadtree structure, such as the coding tree of the CU.

[0146] According to exemplary embodiments, both reversible and irreversible mesh coding techniques exist. The base mesh may be extracted as a subset of the original mesh, and the remaining vertices are encoded based on distance-based predicted displacement coding.

[0147] According to exemplary embodiments, the embodiments described herein may be used separately, in any order, or in any combination, or in any polygon mesh, and the geometry may be encoded by a base mesh and predicted displacement coding. For example, looking at flowchart 2000, in S2002, given a base mesh which is a subset of the original mesh obtained in S2001, the original vertices may be encoded by their predicted points (projected vertices) and the displacement between the predicted points (projected vertices) and the original points (remaining vertices), as seen in Example 2100. The base mesh is constrained such that the remaining vertices (not included in the vertices) are always on the normal side of the intermediate vertices.

[0148] For example, if the displacement coding of a 2D mesh is determined in S2004, then looking at example 2101 of two distance-based displacement codings for a 2D mesh, point

number

number

number

number

number

number

[0149] According to an exemplary 2D embodiment, and in 3D according to another exemplary embodiment, in S2006, an additional point y lies on the line between neighboring points y1 and y2. n This is derived from the neighborhood. For example, point y n The scalar distances from the center of points y1 and y3 are derived as 0, 1 / 2, 1 / 3, and 2 / 3, respectively. Then, in S2007, the best candidates for rate and distortion are selected and signaled.

[0150] Such an embodiment uses points y1, y3 to form an intermediate point y n Obtain the point from there.

number

number

[0151] Looking at Example 2102 in Figure 21, which shows two distance-based displacement codings for nearly reversible 3D mesh displacement coding that can be considered reversible in this specification, the 3D mesh is described based on the selection of 3D coding in S2004 according to an exemplary embodiment. For example, if it is determined that irreversible coding has not been selected in S2008, then in S2009, vertex z4 is predicted from adjacent vertices in the base mesh: points z1, z2, z3. As in the 2D case of Example 2101, distance h h If is known, then point z4 is point

number

number

number

number

number

[0152] Looking at Example 2103, which demonstrates subdivision and distance-based mesh coding, such exemplary embodiments similarly introduce displacement coding of irreversible 3D meshes, such as selected in S2008, based on distance and face subdivision in S2011. That is, similar to Example 2102, Example 2103 uses base mesh face point coding.

number

number

number

number

number

number

[0153] As described above for Example 2101, Example 2103 is also compared to Example 2102, where point z4 and point

number

number

number

number

number

[0154] According to an exemplary embodiment, one embodiment involves face-level processing of an irreversible (selected in S2008), 3D (selected in S2004), or quad (selected in S2010) mesh based on distance and face subdivision. For example, instead of sequentially predicting and encoding, in S2012, there is pixelation of the mesh into corresponding rectangular blocks, such as one or more of the blocks shown in Example 1901. This technique enables block portioning and block merging frameworks in video coding.

[0155] For example, when pixelating a surface in S2012, given a quad surface, first the original vertices subdivided are n 2 They are subdivided so that they occupy 4 points. The example ternary tree shown in Example 2201 of Level 1 for group representation of a quad mesh and corresponding displacement in Example 2200 of Figure 22 is 4 2 It can be used to divide a quad surface to have a number of faces. Based on the assumption of a smooth, changing surface of the mesh, groups of displacements are likely to have a high correlation. Therefore, using intra-like prediction and local transformation, displacement

number

[0156] According to an exemplary embodiment, displacement

number

[0157] For example, looking at group displacement representation 2252, we can use already encoded base mesh vertices (bottom left LL, bottom right LR, top left TL, top right TF) to predict the displacement according to their positions. That is, given a group displacement representation of size B1 × B2, we can use four weight matrices to derive the prediction at partition grid positions i,j as follows:

number

[0158] Furthermore, Example 2202 demonstrates a neighborhood-based intra-displacement prediction in which the current displacement group can be predicted using base mesh vertices and already encoded neighboring displacement vertices. This prediction may be an angle prediction in the intra-prediction. Additional base mesh vertices can be used to perform corrections for the angle prediction and post-smoothing according to equation (11) in the exemplary embodiment.

[0159] Thus, the aspects described herein address technical deficiencies by avoiding creating additional vertices with false connectivity that reduce the throughput of GPU-based mesh rendering and thereby prevent the use of advanced coding methods in video compression for mesh coding.

[0160] The techniques described above can be implemented using computer-readable instructions and as computer software physically stored on one or more computer-readable media or by one or more hardware processors specifically configured. For example, FIG. 23 shows a computer system 2300 suitable for implementing certain embodiments of the disclosed subject matter.

[0161] The computer software can be coded using any suitable machine code or computer language that can receive mechanisms such as assembly, compilation, linking, etc. to create code that includes instructions that can be executed directly by a computer central processing unit (CPU), a graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.

[0162] The instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, etc.

[0163] The components shown in FIG. 23 for the computer system 2300 are exemplary in nature and are not intended to suggest any limitations as to the use or functionality of the computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependencies or requirements with respect to any one or combination of the components shown in the exemplary embodiment of the computer system 230°.

[0164] The computer system 2300 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, applause), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., voices, music, ambient sounds), images (e.g., scanned images, photographic images, still images, etc., acquired from a camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0165] The input human interface device may include one or more of the following (only one of each is shown): a keyboard 2301, a mouse 2302, a trackpad 2303, a touchscreen 2310, a joystick 2305, a microphone 2306, a scanner 2308, and a camera 2307.

[0166] The computer system 2300 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 2310 or joystick 2305, although tactile feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers 2309, headphones (not shown)), visual output devices (e.g., screens 2310 including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability, some of which may be capable of outputting two-dimensional visual output or output beyond three dimensions via means such as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0167] The computer system 2300 may also include human-accessible storage devices and associated media, such as optical media including a CD / DVD ROM / RW 2320 with media like a CD / DVD 2311, a thumb drive 2322, a removable hard drive or solid-state drive 2323, legacy magnetic media such as tape or floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as a security dongle (not shown).

[0168] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0169] The computer system 2300 may also include an interface 2399 to one or more communication networks 2398. The networks 2398 can be, for example, wireless, wired, or optical. The networks 2398 can further be local, wide-area, urban, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks 2398 include local area networks such as Ethernet and Wi-Fi; cellular networks such as GSM, 3G, 4G, 5G, and LTE; wide-area digital networks for wired or wireless TV, including cable TV, satellite TV, and terrestrial broadcast TV; and vehicle and industrial networks including CANBus. Certain networks 2398 generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (2350 and 2351) (e.g., a USB port on the computer system 2300), while other networks are generally integrated into the core of the computer system 2300 by attachment to the system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks 2398, the computer system 2300 can communicate with other entities. Such communication may be unidirectional (e.g., broadcast TV), unidirectional (e.g., CANbus to a specific CANbus device), or bidirectional (e.g., communication to other computer systems using a local area or wide area digital network). Specific protocols and protocol stacks can be used on each of those networks and network interfaces, as described above.

[0170] The aforementioned human interface device, human-accessible memory device, and network interface can be mounted on the core 2340 of the computer system 2300.

[0171] The core 2340 may include one or more central processing units (CPUs) 2341, graphics processing units (GPUs) 2342, graphics adapters 2317, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 2343, hardware accelerators 2344 for specific tasks, and the like. These devices may be connected via a system bus 2348, along with read-only memory (ROM) 2345, random access memory 2346, internal mass storage such as internal non-user-accessible hard drives, SSDs, etc. 2347. In some computer systems, the system bus 2348 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus 2348 or via a peripheral bus 2349. Architectures for peripheral buses include PCI, USB, and the like.

[0172] The CPU 2341, GPU 2342, FPGA 2343, and accelerator 2344 can execute certain instructions that, when combined, can constitute the aforementioned computer code. This computer code can be stored in ROM 2345 or RAM 2346. Temporary data can also be stored in RAM 2346, while persistent data can be stored, for example, in internal mass storage 2347. High-speed storage and retrieval to any of the memory devices can be enabled by the use of cache memory, which can be closely associated with one or more CPUs 2341, GPUs 2342, mass storage 2347, ROM 2345, RAM 2346, etc.

[0173] A computer-readable medium may contain computer code for performing various computer implementation operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0174] For example, but not limited to, a computer system 2300 having an architecture, specifically a core 2340, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media related to specific storage of the core 2340 of a non-transient nature, such as the core internal mass storage 2347 or ROM 2345. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 2340. The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core 2340, specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to execute specific processes or specific parts of specific processes as described herein, including defining data structures stored in RAM 2346 and modifying such data structures according to processes defined by the software. In addition, or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in a circuit (e.g., accelerator 2344), which may operate in place of or in conjunction with software to perform a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuits that store software for execution (e.g., integrated circuits (ICs)), circuits that embody logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software.

[0175] While this disclosure describes several exemplary embodiments, there are many variations, substitutions, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure. [Explanation of symbols]

[0176] 100 Communication system, 101 Terminal, 102 Terminal, 103 Terminal / Second terminal, 104 Terminal / First terminal, 105 Communication network, 201 Video source, 202 Encoder, 203 Capture subsystem, 204 Encoded video bitstream, 205 Streaming server, 206 Copy of encoded video bitstream / Video bitstream, 207 Streaming client, 208 Copy of encoded video bitstream / Incoming copy of encoded video bitstream / Video bitstream, 209 Display, 210 Outgoing video sample stream, 211 Video decoder, 212 Streaming client, 213 Uncompressed video sample stream, 300 Video decoder, 301 Channel, 302 Receiver, 303 Buffer memory / Encoder, 304 Entropy decoder / Parser, 305 Scaler / Inverse unit, 306 Motion compensation prediction unit, 307 Intra-picture prediction unit, 308 Reference picture memory / reference picture buffer, 309 Current reference picture, 310 Aggregator, 311 Loop filter unit, 312 Display / rendering device, 313 Symbol, 400 Video encoder, 401 Video source, 402 Controller, 403 Source coder, 404 Predictor, 405 Reference picture memory, 406 Local video decoder, 407 Coding engine, 408 Entropy coder, 409 Transmitter, 410 Coded video sequence, 411 Communication channel, 500 Simplified block-style workflow diagram, 501 Acquisition block, 502 Audio encoding block, 503 Processing block, 504 Video encoding block, 505 Image encoding block, 507 Distribution block, 508 Head / eye tracking block, 510 Audio decoding block, 511 Audio rendering block, 512 Speaker / headphone block, 513 Video decoding block, 514 Image decoding block, 515 Image rendering block, 516 Display block, 520OMAF Player, 600 Simplified Block Style Content Flow Process Diagram, 601 Volumetric Data Acquisition Block, 602 Conversion to Point Cloud Block, 603 Projection to Image Block, 604 Video Encoding Block, 605 Image Encoding Block, 606 File / Segment Encapsulation Block, 607 Cloud Server Block, 608 Position / Field of View Tracking Block, 609 Scene Generator Block, 610 Video Decoding Block, 611 Image Decoding Block, 612 Point Cloud Reconstruction Block, 613 Scene Configuration Block, 614 Display Block, 615 File / Segment Decapsulation Block, 625 V-PCC Player, 700 Framework, 701 Input Mesh, 702 Mesh with 2D UV Atlas, 703 Decoder, 704 Reconstructed Mesh, 800 Example of Volumetric Data, 801 3D Mesh Segment, 802 UV Parameterization Process, 803 2D Chart, 804 2D UV Atlas, 900 Examples, 901 2D Chart, 902 2D Chart, 903 Example, 1000 Exemplary Flowcharts, 1001 Prediction Mode Example Flowchart, S201 Step, S202 Step, S203 Step, S204 Step, S205 Step, S206 Step, S207 Step, S208 Step, S209 Step, S210 Step, S211 Step, 1150 Exemplary Flowcharts, S221 Step, S222 Step, S223 Step, S224 Step, S225 Step, S226 Step, S227 Step, 1200 Exemplary Flowcharts, S231 Step, S232 Step, S233 Step, S234 Step, S235 Step, 1300 Examples, 1301 First Layer, 1302 Second Layer, 1303 Third Layer, 1340 First polygon, 1341 Additional polygons, 1400 Examples, 1401 First layer, 1402 Second layer, 1403 Third layer, 1500 Examples, S1501 Steps, S1502 Steps, S1503 Steps, S1504 Steps, 1600 Examples, 1601 Mesh examples / instances, 1602 Instances, 1603Instance, 1604 Instance, 1700 Flowchart, S1701 Step, S1702 Step, S1703 Step, S1704 Step, S1705 Step, S1706 Step, S1707 Step, S1708 Step, 1800 Flowchart, S1801 Step, S1802 Step, S1803 Step, S1804 Step, S1805 Step, S1806 Step, S1807 Step, S1808 Step, S1809 Step, S1810 Step, 1900 Example, 1901 Quadriarch-Binary Tree (QTBT), 1902 Tree Representation, 2000 Flowchart, S2001 Step, S2002 Step, S2003 Step, S2004 Step, S2005 Step, S2006 Step, S2007 Step, S2008 Step, S2009 Step, S2010 Step, S2011 Step, S2012 Step, S2013 Step, S2014 Step, 2100 Example, 2101 Example, 2102 Example, 2103 Example, 2200 Example, 2201 Example, 2202 Example, 2251 Displacement Prediction, 2252 Group Displacement Representation, 2253 Already Encoded Surface, 2254 Already Encoded Surface, 2300 Computer System, 2301 Keyboard, 2302 Mouse, 2303 Trackpad, 2305 Joystick, 2306 Microphone, 2307 Camera, 2308 Scanner, 2309 Speaker, 2310 Touchscreen, 2311 CD / DVD, 2317 Graphics Adapter, 2320 CD / DVD ROM / RW, 2322 Thumb Drive, 2323 Removable hard drive / solid state drive, 2340 cores, 2341 central processing unit (CPU), 2342 graphics processing unit (GPU), 2343 field-programmable gate area (FPGA), 2344 hardware accelerator, 2345 read-only memory (ROM), 2346 random access memory (RAM), 2347 internal mass storage, 2348 system bus, 2349 peripheral bus, 2350 general-purpose data port, 2351 peripheral bus, 2398 communication network, 2399 interface

Claims

1. A method for video encoding, wherein the method is performed by at least one processor. The steps include obtaining volumetric data of at least one three-dimensional (3D) visual content, A step of deriving a mesh from the frame of volumetric data, wherein the mesh includes a plurality of base mesh vertices and at least one vertex that is not one of the plurality of base mesh vertices. A step of determining the displacement of the at least one vertex that is not a base mesh vertex, based on a series of projections from at least one base mesh vertex located in the vicinity of the at least one vertex among the plurality of base mesh vertices to the at least one vertex, The steps include predicting at least one vertex based on the determined displacement, A step of encoding the volumetric data based on the predicted at least one vertex. Methods for video encoding, including those mentioned above.

2. The step of encoding the volumetric data includes the step of encoding the scalar distance from the projected vertex to at least one of the vertices. The method for video encoding according to claim 1.

3. The series of projections includes the step of determining that the projected vertex lies on a line from a first base mesh vertex among the plurality of base mesh vertices to a second base mesh vertex among the plurality of base mesh vertices, The scalar distance is perpendicular to the line. The method for video encoding according to claim 2.

4. The series of projections further includes the step of deriving neighboring vertices as being on the line, The step of determining that the projected vertex lies on the line includes the step of applying a second scalar distance to the neighboring vertex. The method for video encoding according to claim 3.

5. The aforementioned series of projections The steps include determining that a first neighboring vertex lies on a first line from a first base mesh vertex among the plurality of base mesh vertices to a second base mesh vertex among the plurality of base mesh vertices, The steps include determining that the projected vertex is the distance from the first neighboring vertex, and A method for video encoding according to claim 1, including the method described in claim 1.

6. The aforementioned series of projections The steps include determining that the second neighboring vertex lies on a second line from the first base mesh vertex among the plurality of base mesh vertices to the third base mesh vertex among the plurality of base mesh vertices, The steps include determining that the projected vertex is the distance from both the first neighboring vertex and the second neighboring vertex, A method for video encoding according to claim 5, further comprising:

7. The step of encoding the volumetric data includes the step of encoding the distance from the projected vertex to at least one vertex, The method for video encoding according to claim 6.

8. The aforementioned plurality of base mesh vertices define the faces of the mesh, The series of projections further includes the steps of subdividing the surface into a plurality of subdivisions and determining the subdivision point closest to the projected vertex among the subdivisions, The method for video encoding according to claim 6.

9. The step of encoding the volumetric data includes the step of encoding the distance from the nearest subdivision point to at least one vertex. The method for video encoding according to claim 8.

10. The surface of the mesh includes a quad surface. The method for video encoding according to claim 8.

11. An apparatus configured to perform the method described in any one of claims 1 to 10.

12. A computer program for causing a computer to perform the method described in any one of claims 1 to 10.

13. A method for storing a bitstream, the method being performed by at least one processor, A step of generating a bitstream, The steps include obtaining volumetric data of at least one three-dimensional (3D) visual content, A step of deriving a mesh from the frame of volumetric data, wherein the mesh includes a plurality of base mesh vertices and at least one vertex that is not one of the plurality of base mesh vertices. A step of determining the displacement of the at least one vertex that is not a base mesh vertex, based on a series of projections from at least one base mesh vertex located in the vicinity of the at least one vertex among the plurality of base mesh vertices to the at least one vertex, The steps include predicting at least one vertex based on the determined displacement, The steps include encoding the volumetric data based on the predicted at least one vertex, The steps include generating a bitstream and The steps include storing the bitstream in a computer-readable storage medium, Methods that include...

Images