Mesh vertex grouping for position compression

Abstract

The method includes: dividing a plurality of vertices corresponding to a mesh of a current frame that is positionally tracked into a plurality of vertex groups based on a predetermined constant integer; the plurality of vertex groups includes a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimation error indicating a difference between a reference vertex of the reference frame and a current vertex in the first vertex group; encoding a prediction mode of a vertex in the first vertex group based on at least the estimated error between the reference vertex and the current vertex; generating prediction information of a vertex in the first vertex group based on the encoded prediction mode of a vertex in the first vertex group.

Description

[Technical Field]

[0001] This application claims priority to U.S. provisional application Ser. No. 63 / 388525, filed July 12, 2022, entitled "Grouping of Mesh Vertices in Position Compression," and to U.S. patent application Ser. No. 18 / 144703, filed May 8, 2023, entitled "Grouping of Mesh Vertices in Position Compression," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure includes embodiments relating to mesh processing. [Background technology]

[0003] The "Background" discussion provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein, to the extent described in this Background, along with aspects described therein that would not normally be considered prior art at the time of filing, is not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) filming, modeling, and rendering have facilitated the ubiquity of 3D content across multiple platforms and devices. It is now possible to film a baby's first steps on one continent while the baby's grandparents can view (and potentially interact with) the baby on another continent, enjoying a fully immersive experience. To achieve such realism, models are becoming ever more sophisticated, and large amounts of data are linked to the creation and consumption of those models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Problem to be solved by the invention]

[0005] Aspects of the present disclosure provide methods and apparatus for mesh processing. In some examples, an apparatus for mesh processing includes a processing circuit. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a mesh processing method executed by a video encoder is provided. The method includes: dividing a plurality of vertices corresponding to a positionally tracked mesh of a current frame into a plurality of vertex groups based on a predetermined constant integer; the plurality of vertex groups includes a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimation error indicating a difference between a reference vertex in the reference frame and a current vertex in the first vertex group; encoding a prediction mode of a vertex in the first vertex group based on at least the estimated error between the reference vertex and the current vertex; and generating prediction information of the vertices in the first vertex group based on the encoded prediction mode of the vertices in the first vertex group.

[0007] In some embodiments, the method includes determining an average neighbor estimation error of a plurality of neighbor estimation errors, the plurality of neighbor estimation errors being associated with a plurality of neighboring vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicating a difference between a respective one of a plurality of neighboring vertices of the current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighboring vertices of the current vertex in the first vertex group, determining a first cost value based on at least the estimation error associated with the current vertex, determining a second cost value based on at least the average neighbor estimation error associated with the current vertex, and encoding the prediction mode based on at least the first cost value and the second cost value.

[0008] In some embodiments, a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as the current vertex in the first vertex group in the current frame.

[0009] In one example, to determine an average neighbor estimation error associated with the plurality of neighboring vertices, a first neighbor estimation error associated with a first neighboring vertex of a plurality of neighboring vertices in the first vertex group is determined, the first neighbor estimation error indicating a difference between a first neighboring vertex of the current vertex and a reference vertex of a reference frame corresponding to the first neighboring vertex of the current vertex. A second neighbor estimation error associated with a second neighboring vertex of a plurality of neighboring vertices in the first vertex group is determined, the second neighbor estimation error indicating a difference between a second neighboring vertex of the current vertex and a reference vertex of a reference frame corresponding to the second neighboring vertex of the current vertex. An average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error is determined, the average neighbor estimation error being associated with the current vertex in the first vertex group. An estimated difference between the estimation error of the current vertex and an average neighbor estimation error associated with the current vertex is further determined.

[0010] In some embodiments, the first cost value is determined as a sum of estimation errors of vertices in the first group of vertices, the estimation errors of the vertices including the estimation error of the current vertex.

[0011] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each of the vertices in the first group of vertices and an average neighboring estimated error associated with corresponding vertices in the first group of vertices is determined, and the second cost value is determined as a sum of estimated differences associated with vertices in the first group of vertices, where the estimated difference includes an estimated difference associated with the current vertex.

[0012] In one example, the prediction mode is determined as the first mode based on the first cost value being less than or equal to the second cost value. In another example, the prediction mode is determined as the second mode based on the first cost value being greater than the second cost value.

[0013] In some embodiments, the method generates a first prediction residual for each vertex in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimated error for each vertex in the first vertex group, and generates a second prediction residual for each vertex in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated error associated with each vertex in the first vertex group.

[0014] In some embodiments, the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

[0015] In some embodiments, the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag including one of an exclusive-or (XOR) flag and a binary flag.

[0016] According to another aspect of the present disclosure, there is provided an apparatus including a processing circuit, the processing circuit may be configured to perform any of the methods for mesh processing described.

[0017] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for mesh processing. [Brief explanation of the drawings]

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of an exemplary block diagram of a video processing system (100). [Figure 2] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4] FIG. 1 is a schematic diagram of vertices of a mesh according to some embodiments of the present disclosure. [Figure 5] FIG. 1 is a schematic diagram of an exemplary apex position compression device according to some embodiments of the present disclosure. [Figure 6] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 7] 1 is a flowchart outlining another process according to some embodiments of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0019] 1 shows a block diagram of a video processing system 100 in some examples. The video processing system 100 is an example of an application of the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter is equally applicable to other image and video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media, including CDs, DVDs, memory sticks, etc.

[0020] The video processing system 100 includes a capture subsystem 113, which may include a video source 101. The video source 101 may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera may create a stream of uncompressed video pictures 102. In one example, the stream of video pictures 102 includes samples captured by the digital camera. The stream of video pictures 102, shown in bold to emphasize its high data volume compared to the encoded video data 104 (or coded video bitstream), may be processed by an electronic device 120, which includes a video encoder 103, coupled to the video source 101. The video encoder 103 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 104 (or encoded video bitstream), shown with a thin line to emphasize its lower data volume compared to the stream of video pictures 102, can be stored on a streaming server 105 for future use. One or more streaming client subsystems, such as the client subsystems 106 and 108 of FIG. 1, can access the streaming server 105 to retrieve copies 107 and 109 of the encoded video data 104. The client subsystem 106 can include, for example, a video decoder 110 in an electronic device 130. The video decoder 110 decodes the incoming copy 107 of the encoded video data and creates an outgoing stream of video pictures 111 that can be rendered on a display 112 (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data 104, 107, and 109 (e.g., video bitstreams) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, an evolving video coding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0021] It should be noted that the electronic devices 120 and 130 may include other components (not shown). For example, the electronic device 120 may include a video decoder (not shown), and the electronic device 130 may include a video encoder (not shown).

[0022] 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include receiving circuitry, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) in the example of FIG. 1.

[0023] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, with the decoding of each coded video sequence being independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (231) may separate the coded video sequences from other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be external to the video decoder (210) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (210), for example, to prevent network jitter, and another buffer memory (215) internal to the video decoder (210), for example, to handle playback timing. When the receiver (231) receives data from a store-and-forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be necessary and may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, may be relatively large, may advantageously be adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (210).

[0024] The video decoder (210) may include a parser (220) that reconstructs symbols (221) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (210) and potentially include information to control a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but can be coupled to the electronic device (230) as shown in FIG. 2. Control information for the rendering device may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence can be aligned with a video coding technique or standard and can follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a set of subgroup parameters for at least one subgroup of pixels in a video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0025] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0026] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0027] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0028] The first unit is a scalar / inverse transform unit (251), which receives control information from the parser (220) including the transform to be used, block size, quantization factor, quantization scaling matrix, etc., as well as quantized transform coefficients as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to the aggregator (255).

[0029] In some cases, the output samples of the scaler / inverse transform unit (251) may pertain to intra-coded blocks. Intra-coded blocks do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information retrieved from the current picture buffer (258). The current picture buffer (258), for example, buffers the partially reconstructed and / or fully reconstructed current picture. The aggregator (255) may, in some cases, perform intra-coded block reconstructing on a sample-by-sample basis. Picture The prediction information generated by the prediction unit (252) is added to the output sample information provided by the scaler / inverse transform unit (251).

[0030] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols (221) related to the block, these samples may be added to the output of the scalar / inverse transform unit (251) by the aggregator (255) to generate output sample information (in this case, referred to as residual samples or residual signals). The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) retrieves prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (221) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (257) when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0031] The output samples of the aggregator (255) may be applied to various loop filtering techniques in a loop filter unit (256). Video compression techniques can include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression can also depend on meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed, loop-filtered sample values.

[0032] The output of the loop filter unit (256) may be a sample stream that can be output to a rendering device (212) and stored in a reference picture memory (257) for use in future inter-picture prediction.

[0033] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before beginning reconstruction of the following coded picture.

[0034] The video decoder (210) can perform decoding operations according to a given video compression technology or standard, such as ITU-T Recommendation H.265. A coded video sequence may comply with the syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile may select specific tools from all tools available in the video compression technology or standard as the only tools available in that profile. Compliance also requires that the complexity of the coded video sequence be within the range limited by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the coded video sequence.

[0035] In one embodiment, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0036] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0037] The video encoder (303) can receive video samples from a video source (301) (not part of the electronic device (320) in the example of FIG. 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0038] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed in sequence, give the appearance of motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following discussion focuses on samples.

[0039] According to one embodiment, the video encoder (303) can encode and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Enforcing the appropriate coding rate is one of the functions of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) can include rate control-related parameters (picture skip, quantization, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured with other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0040] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop can include a source coder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to that created by the (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding the symbol stream produces bit-exact results regardless of the location of the (local or remote) decoder, the contents of the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained due to, for example, channel errors) is also used in several related technologies.

[0041] The operation of the "local" decoder (333) may be similar to the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in connection with Figure 2. However, and with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0042] In one embodiment, decoder technology other than parsing / entropy decoding present in the decoder is present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder technology can be omitted, as it is the reverse of the decoder technology described generically. In certain areas, more detailed descriptions are provided below.

[0043] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The local video decoder (333) can decode coded video data for pictures that may be designated as reference pictures based on symbols created by the source coder (330). The operation of the coding engine (332) can advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence may typically be a replica of the source video sequence, with some errors. The local video decoder (333) can replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in the reference picture memory (334). In this way, the video encoder (303) can locally store copies of reconstructed reference pictures that have content in common (without transmission errors) with the reconstructed reference pictures obtained by the far-end video decoder.

[0044] The predictor (335) can perform the prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata that can serve as suitable prediction criteria for the new picture, such as the reference picture's motion vectors, block shapes, etc. The predictor (335) can operate on a sample block / pixel block basis to find a suitable prediction criteria. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction criteria drawn from multiple reference pictures stored in the reference picture memory (334).

[0045] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0046] The output of all of the aforementioned functional units may be entropy coded by an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0047] The transmitter (340) can buffer the coded video sequence created by the entropy coder (345) in preparation for transmission over a communication channel (360), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) can also merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0048] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned one of the following picture types:

[0049] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I-pictures and their respective uses and characteristics.

[0050] A predictive picture (P picture) may be coded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0051] Bidirectionally predicted pictures (B-pictures) may be coded and decoded by intra- or inter-prediction using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than one reference picture and associated metadata for the reconstruction of a single block.

[0052] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to one pre-coded reference picture. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to one or two pre-coded reference pictures.

[0053] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. During operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard used.

[0054] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0055] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as "intra-prediction") exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0056] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, each of which is earlier in decoding order than a current picture in a video (but can be earlier and later in display order, respectively). A block in the current picture can be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted from a combination of the first reference block and the second reference block.

[0057] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0058] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, a prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0059] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0060] The present disclosure includes embodiments that relate to methods and systems for grouping mesh vertices for positional compaction.

[0061] A mesh may include a number of polygons that describe the surface of a volumetric object. Each polygon in a mesh can be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information about how the vertices are connected (which can be referred to as connectivity information). In some embodiments, vertex attributes, such as color, normals, etc., can be associated with mesh vertices. Attributes (or vertex attributes) can also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a two-dimensional (2D) attribute map. Such mapping can typically be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps can be used to store high-resolution attribute information, such as texture, normals, and displacements. Such information can be used for various purposes, such as texture mapping and shading.

[0062] Dynamic mesh sequences can require large amounts of data because dynamic meshes can contain a large amount of information that changes over time. Therefore, efficient compression techniques are necessary to store and transmit such content. Mesh compression standards such as IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information. Digital content creation (DCC) tools typically generate such dynamic meshes. However, generating constant connectivity dynamic meshes, especially under real-time constraints, is challenging for volumetric acquisition techniques. This type of content (e.g., constant connectivity dynamic meshes) may not be supported by existing standards. MPEG plans to develop a new mesh compression standard that directly handles dynamic meshes with time-varying connectivity information and, optionally, time-varying attribute maps. This new mesh compression standard targets lossy and lossless compression for various applications, such as real-time communication, storage, free-viewpoint video, augmented reality (AR), and virtual reality (VR). Functionality such as random access and scalable / progressive coding is also considered.

[0063] Mesh geometry information can include vertex connectivity information, 3D coordinates, 2D texture coordinates, etc. Compression of vertex 3D coordinates, also called vertex positions, is important because they can often consume a significant portion of the overall geometry-related data.

[0064] A dynamic mesh sequence M at time instance t can be represented as M(t). If there is a mapping (or mapping operation) f from the vertex positions of M(t) to the vertex positions of another time instance M(t), M(t) can be called a positional tracking frame, where t and t are different time instances. Therefore, M(t) can be called a reference frame, and the corresponding vertices in the reference frame can be called reference vertices for the vertices in M(t).

[0065] This disclosure proposes methods and / or systems for grouping mesh vertices for position compression (e.g., vertex position compression). Note that these methods and / or systems may be applied individually or in any combination. Furthermore, the disclosed methods and systems are not limited to vertex position compression. The disclosed methods and systems may also be applied, for example, to two-dimensional (2D) texture coordinate compression or more general temporal prediction-based schemes.

[0066] For a vertex V in the positional tracking frame M(t), the neighbors of a vertex may be vertices connected to V via edges, and these vertices are called neighbors (or adjacent vertices) of V. For example, as shown in Figure 4, vertex A may have four neighbors: C, D, E, and B. Vertex E may have five neighbors: A, B, F, H, and D.

[0067] 5 is a schematic diagram of an example vertex position compression device (500) according to some embodiments of the present disclosure. As shown in FIG. 5, the vertex position compression device (500) may include a vertex grouping module (502) configured to group vertices of a mesh, a position prediction module (504) configured to calculate predicted positions of the vertices, a prediction mode coding module (506) configured to code (or determine) a prediction mode for the position compression, and a prediction residual coding module (508) configured to code a position prediction residual.

[0068] The term module in this disclosure may refer to a software module, a hardware module, or a combination thereof. A software module (e.g., a computer program) may be developed using a computer programming language. A hardware module may be implemented using processing circuitry and / or memory. Each module may be implemented using one or more processors (or a processor and memory). Similarly, a processor (or a processor and memory) can be used to implement one or more modules. Furthermore, each module can be part of an entire module that includes the functionality of that module.

[0069] In this disclosure, temporal prediction can be applied to predict a vertex of a current frame based on a reference vertex of a reference frame. For a vertex V in a positionally tracked frame (or current frame) M(t), the position of the vertex V can be estimated by the position of a reference vertex f(V) in a reference frame (e.g., M(t), where t is a different time instance), where f is a mapping operation between M(t) and the reference frame. In some embodiments, the vertex V and the reference vertex f(V) in the reference frame are aligned. Thus, the reference vertex can have the same relative position in the reference frame as the vertex in the current frame M(t). When the vertex V is predicted from the reference vertex f(V), an estimation error E can be determined as the difference between the position of V and the position of f(V) in Equation (1).

[0070] E=Vf(V) Equation (1)

[0071] Since each vertex of the frame M(t) can have a 3D coordinate, the 3D coordinate components of the estimated error E can be provided based on equation (1). For example, assuming that the subscripts x, y, and z represent 3D coordinates in the xyz space, the 3D coordinate components of the estimated error E can be provided by equations (2) to (4).

[0072] E x =V x -(f(V)) x Formula (2)

[0073] E y =V y -(f(V)) y Formula (3)

[0074] E z =V z -(f(V)) z Formula (4)

[0075] The estimated error E of a vertex V can be predicted or determined from the neighboring vertices (or adjacent vertices) of vertex V. For the neighboring vertices (or adjacent vertices) of V, if the neighboring vertices are coded and available for prediction, the estimated error of the adjacent vertices can be applied to predict E.

[0076] V is a set of N neighboring vertices (or adjacent vertices) V1, V2...V that are coded and can be used for prediction. N Suppose we have a neighboring vertex V i For a neighboring vertex V i The estimated error (or adjacent estimated error) of E i =V i -f(V i ) (i=1, 2...N) can be determined as f(V i ) is the neighboring vertex V in the reference frame. i can be the reference vertex of E i can be called the neighbor estimation error associated with vertex V, and each E i can be a predictive candidate for E.

[0077] If N≧2, multiple estimation errors E i is available. In one embodiment, the estimation error E i The average estimation error (or average neighbor estimation error) E0 of the vectors can be defined as follows in equation (5):

[0078] E0=(E1+E2+…+E N ) / N formula (5)

[0079] For a group G of a frame M(t), the encoder can determine a coding cost C0 and a coding cost C1. C0 may represent the sum of the estimated errors E of all vertices in the group G. For example, if the estimated errors E of each of the vertices in the group G are determined based on equation (1), the coding cost C0 may be determined as the sum of the estimated errors E of the vertices in the group G. C1 may represent the sum of the estimated residuals (E-E0) of all vertices in the group G. To determine the coding cost C1, the estimated errors E of the neighboring vertices of each vertex in the group G are calculated. i ,E i =V i -f(V i ) can be determined as follows. Furthermore, the estimation error E i The average neighbor estimation error E0 of can be determined based on, for example, equation (5). An estimation residual (or estimation difference) (E-E0) can be determined for each of the vertices in group G. A coding cost C1 can be determined as the sum of the estimation residuals associated with the vertices in group G.

[0080] In the present disclosure, a prediction mode can be determined based on the coding cost C0 and the coding cost C1.

[0081] If the coding cost C0 is less than or equal to C1, prediction mode 0 is applied to group G. Prediction mode 0 indicates that the prediction residual of each vertex in group G is the estimated error E of each vertex. Therefore, at the decoder side, each vertex in group G can be reconstructed as V = f(V) + E, where f(V) is the reference vertex in reference frame M(t0) corresponding to each vertex in group G in frame M(t).

[0082] If the coding cost C0 is greater than C1, prediction mode 1 is applied to group G. Prediction mode 1 indicates that the prediction residual of each vertex in group G is (E-E0), where E is the estimation error of each vertex, and E0 is the average estimation error of the estimation errors of each vertex's neighboring vertices. Therefore, at the decoder side, each vertex in group G can be reconstructed as V=f(V)+(E-E0), where f(V) is the reference vertex in reference frame M(t0) corresponding to each vertex in group G in frame M(t).

[0083] In this disclosure, the prediction mode of group G may be coded. The prediction mode may be a binary number such as 0 or 1. In one embodiment, the prediction mode may be coded using entropy coding. In one embodiment, the prediction mode may be coded using arithmetic coding. In one embodiment, the prediction mode may be coded using context-based arithmetic coding, such as context-adaptive binary arithmetic coding (CABAC). In one embodiment, the prediction mode may be coded using spatial context-based arithmetic coding, where the context may be conditioned depending on (or based on) previously coded groups of the same frame.

[0084] In one embodiment, if the prediction mode in a reference frame (e.g., M(t0)) has already been coded and is available, the prediction mode of the group in frame M(t) can be coded using a temporal context, such as temporal context-based arithmetic coding. Thus, based on the temporal context-based arithmetic coding, the prediction mode of the group in frame M(t) can be determined based on the prediction mode of the corresponding (or associated) group in the reference frame. This is because each vertex in the group in frame M(t) has a corresponding reference vertex in the associated group in the reference frame, and a one-to-one association between the group in positional tracking frame M(t) and the associated group in reference frame M(t0) can also be established. The associated group in the reference frame (e.g., M(t0), where t0 is a different time instance from t) can be called a reference group.

[0085] In one embodiment, a logical operation (or flag) such as XOR (exclusive OR) may be coded to indicate the relationship between the prediction mode of group G and the prediction mode of the reference group of group G. Thus, if group G and the reference group have the same prediction mode, the logical operation XOR may be coded as 0. If group G and the reference group have different prediction modes, the logical operation XOR may be coded as 1.

[0086] In one embodiment, a binary flag (e.g., 0 or 1) may be coded to indicate whether group G and the reference group of group G have the same prediction mode. Thus, if group G and the reference group have the same prediction mode, a binary 1 may be coded. If group G and the reference group have different prediction modes, a binary 0 may be coded.

[0087] In one embodiment, the prediction mode of group G in the positional tracking frame M(t) can be coded using temporal context-based arithmetic coding, and the context of the temporal context-based arithmetic coding can be based on the prediction mode of the reference group of group G.

[0088] In some embodiments, the prediction residual (e.g., E or (E-E)) may be coded using a coding algorithm such as fixed-length coding, exponential-Golomb coding, arithmetic coding, etc. In some embodiments, the prediction residual may undergo a compression transform such as a fast Fourier transform (FFT), a discrete cosine transform (DCT), a discrete sine transform (DST), or a discrete wavelet transform (DWT). The output from the compression transform may be coded using a coding algorithm such as fixed-length coding, exponential-Golomb coding, arithmetic coding, etc.

[0089] 6 is a flowchart outlining a process (600) according to one embodiment of the present disclosure. The process (600) may be used in an encoder, such as a video encoder. In various embodiments, the process (600) is performed by a processing circuit, such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), or the like. In some embodiments, the process (600) is implemented by software instructions, such that the processing circuit performs the process (600) when executing the software instructions. The process starts at (S601) and proceeds to (S610).

[0090] In step S610, a plurality of vertices corresponding to the mesh of the current frame that are positionally tracked are divided into a plurality of vertex groups based on a predetermined constant integer, and the plurality of vertex groups includes a first vertex group.

[0091] In (S620), a first position of a current vertex in the first vertex group is estimated based on a second position of a reference vertex in a reference frame different from the current frame.

[0092] At (S630), an estimated error is determined that indicates the difference between the reference vertex in the reference frame and the current vertex in the first vertex group.

[0093] At (S640), prediction modes of vertices in the first vertex group are encoded based on at least the estimated error between the reference vertex and the current vertex.

[0094] At (S650), prediction information for the vertices in the first vertex group is generated based on the encoded prediction modes of the vertices in the first vertex group.

[0095] In some embodiments, the method includes determining an average neighbor estimation error of a plurality of neighbor estimation errors, the plurality of neighbor estimation errors being associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicating a difference between a respective one of a plurality of neighbor vertices of the current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of the current vertex in the first vertex group, determining a first cost value based on the estimation error associated with at least the current vertex, determining a second cost value based on at least the average neighbor estimation error associated with the current vertex, and encoding the prediction mode based on at least the first cost value and the second cost value.

[0096] In some embodiments, a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as the current vertex in the first vertex group in the current frame.

[0097] In one example, to determine an average neighbor estimation error associated with the plurality of neighboring vertices, a first neighbor estimation error associated with a first neighboring vertex of a plurality of neighboring vertices in the first vertex group is determined. The first neighbor estimation error indicates a difference between a first neighboring vertex of the current vertex and a reference vertex of a reference frame corresponding to the first neighboring vertex of the current vertex. A second neighbor estimation error associated with a second neighboring vertex of a plurality of neighboring vertices in the first vertex group is determined. The second neighbor estimation error indicates a difference between a second neighboring vertex of the current vertex and a reference vertex of a reference frame corresponding to the second neighboring vertex of the current vertex. An average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error is determined, the average neighbor estimation error being associated with the current vertex in the first vertex group. An estimated difference between the estimation error of the current vertex and an average neighbor estimation error associated with the current vertex is further determined.

[0098] In some embodiments, the first cost value is determined as a sum of estimated errors of the vertices in the first group of vertices, wherein the estimated errors of the vertices include the estimated error of the current vertex.

[0099] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each of the vertices in the first group of vertices and an average neighboring estimated error associated with corresponding vertices in the first group of vertices is determined, and the second cost value is determined as a sum of estimated differences associated with vertices in the first group of vertices, including the estimated difference associated with the current vertex.

[0100] In one example, the prediction mode is determined as the first mode based on the first cost value being less than or equal to the second cost value. In another example, the prediction mode is determined as the second mode based on the first cost value being greater than the second cost value.

[0101] In some embodiments, the method generates a first prediction residual for each vertex in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimated error for each vertex in the first vertex group, and generates a second prediction residual for each vertex in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated error associated with each vertex in the first vertex group.

[0102] In some embodiments, the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

[0103] In some embodiments, the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

[0104] The process then proceeds to (S699) and ends.

[0105] The process 600 may be adapted as appropriate. Steps of the process 600 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0106] 7 is a flowchart outlining a process (700) according to one embodiment of the present disclosure. The process (700) may be used in a decoder, such as a video decoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit that performs the functions of the video decoder (110), a processing circuit that performs the functions of the video decoder (210), or the like. In some embodiments, the process (700) is implemented with software instructions, such that the processing circuit performs the process (700) when executing the software instructions. The process starts at (S701) and proceeds to (S710).

[0107] At (S710), coding information of a first vertex group of a mesh in a current frame is received, where the coding information indicates a prediction mode of the vertices in the first vertex group.

[0108] At (S720), estimation errors for vertices in the first vertex group are determined based on the coding information indicating a first prediction mode of the prediction modes, each of the estimation errors indicating a difference between a reference vertex of a reference frame corresponding to a vertex in the first vertex group and a corresponding vertex in the first vertex group.

[0109] At (S720), average neighbor estimation errors of a plurality of neighboring vertices associated with each of the vertices in the first vertex group are determined based on coding information indicating a second prediction mode of the prediction modes, each of the plurality of neighbor estimation errors indicating a difference between one of a plurality of neighboring vertices of each vertex in the first vertex group and a reference vertex of a reference frame corresponding to the one of the plurality of neighboring vertices of each vertex in the first vertex group.

[0110] At (S730), the prediction residuals of each vertex in the first vertex group are determined as (i) an estimation error for each vertex in the first vertex group based on the coding information indicating the first prediction mode, and (ii) an average neighboring estimation error associated with each vertex in the first vertex group based on the coding information indicating the second prediction mode.

[0111] In (S740), the vertices in the first vertex group are reconstructed based on the determined prediction residuals.

[0112] The process then proceeds to (S799) and ends.

[0113] The process 700 may be adapted as appropriate. Steps of the process 700 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0114] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 8 illustrates a computer system (800) suitable for implementing certain embodiments of the disclosed subject matter.

[0115] Computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or by interpretation, microcode execution, etc.

[0116] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0117] 8 for computer system (800) are exemplary in nature and are not intended to suggest any limitation on the use or scope of functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system (800).

[0118] The computer system (800) may include certain human interface input devices that can respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0119] The input human interface devices may include one or more of a keyboard (801), a mouse (802), a trackpad (803), a touchscreen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808) (only one of each is shown).

[0120] The computer system (800) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (810), data gloves (not shown), or joystick (805), but may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (809), headphones (not shown), etc.), visual output devices (e.g., screens (810), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input and haptic feedback capabilities, some of which may output two-dimensional visual output or three-dimensional or higher-dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0121] The computer system (800) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (820) or similar media (821) with CDs / DVDs, thumb drives (822), and removable hard drives or solid state drives (823), legacy magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0122] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0123] The computer system 800 may also include an interface 854 to one or more communications networks 855. The networks may be, for example, wireless, wired, or optical networks. The networks may further include local, wide-area, metropolitan, vehicular, industrial, real-time, and delay-tolerant networks. Examples of networks include local area networks such as Ethernet and wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicular and industrial networks including CAN buses and the like. Particular networks typically require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 849 (e.g., a USB port on the computer system 800). Other networks are typically integrated into the core of the computer system 800 by connecting to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 800 can communicate with other entities. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from a CAN bus to a particular CAN bus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. As noted above, specific protocols and protocol stacks may be used for each of these networks and network interfaces.

[0124] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to the core (840) of the computer system (800).

[0125] A core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), field programmable gate array The system may include dedicated programmable processing units in the form of FPGAs (843), task-specific hardware accelerators (844), and graphics adapters (850). These devices may be connected via a system bus (848), along with read-only memory (ROM) (845), random access memory (846), and internal mass storage devices (847) such as non-user-accessible internal hard drives or SSDs. In some computer systems, the system bus (848) is accessible in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core system bus (848) or via a peripheral bus (849). In one example, a screen (810) may be connected to the graphics adapter (850). Peripheral bus architectures include PCI, USB, etc.

[0126] The CPU (841), GPU (842), FPGA (843), and accelerator (844) may combine to execute specific instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (845) or RAM (846). Transient data may be stored in RAM (846), while permanent data may be stored, for example, in internal mass storage (847). Cache memory, which may be closely associated with one or more of the CPU (841), GPU (842), mass storage (847), ROM (845), RAM (846), etc., may be used to enable fast storage and retrieval from any memory device.

[0127] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0128] By way of example and not limitation, a computer system having the architecture (800), and in particular the core (840), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device, as described above, in addition to specific storage of the core (840) that is non-transitory in nature, such as the core's internal mass storage device (847) or ROM (845). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (840). The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause the core (840), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processes or portions of certain processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (844)) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may, where appropriate, include logic, and vice versa. References to computer-readable media may, where appropriate, include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0129] While this disclosure has described certain exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It should thus be understood that those skilled in the art can devise numerous systems and methods that embody the principles of the disclosure, but which are within its spirit and scope.

Claims

1. 1. A mesh processing method performed by a video encoder, comprising: Dividing a plurality of vertices corresponding to a mesh of a current frame that have been positionally tracked into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimated error indicative of a difference between a reference vertex of the reference frame and a current vertex in the first group of vertices; encoding prediction modes for vertices in the first vertex group based on at least an estimated error between the reference vertex and the current vertex; generating prediction information for vertices in the first vertex group based on encoded prediction modes of vertices in the first vertex group.

2. determining an average neighbor estimation error of a plurality of neighbor estimation errors associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicating a difference between a respective one of a plurality of neighbor vertices of a current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of the current vertex in the first vertex group; determining a first cost value based on at least an estimation error associated with the current vertex, and determining a second cost value based on at least an average neighbor estimation error associated with the current vertex; The method of claim 1 , wherein encoding the prediction mode further comprises encoding the prediction mode based on at least the first cost value and the second cost value.

3. The method of claim 1 , wherein a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as a current vertex in the first vertex group in the current frame.

4. Determining an average neighbor estimation error associated with the plurality of neighboring vertices comprises: determining a first adjacent estimation error associated with a first adjacent vertex of a plurality of adjacent vertices in the first vertex group, the first adjacent estimation error indicating a difference between a first adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the first adjacent vertex of the current vertex; determining a second adjacent estimation error associated with a second adjacent vertex of a plurality of adjacent vertices in the first vertex group, the second adjacent estimation error indicating a difference between a second adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the second adjacent vertex of the current vertex; determining an average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error associated with a current vertex in the first vertex group; The method of claim 2 , further comprising determining an estimated difference between the current vertex's estimated error and an average neighboring estimated error associated with the current vertex.

5. The step of determining the first cost value comprises:

5. The method of claim 4, further comprising determining the first cost value as a sum of estimation errors of vertices in the first group of vertices, the estimation errors of the vertices including the estimation error of the current vertex.

6. The step of determining the second cost value comprises: determining an estimated difference between an estimated error for each of the vertices in the first group of vertices and an average neighboring estimated error associated with each vertex in the first group of vertices; 6. The method of claim 5, further comprising determining the second cost value as a sum of estimated differences associated with vertices in the first group of vertices, the estimated differences including an estimated difference associated with the current vertex.

7. The step of encoding the prediction mode includes: determining the prediction mode as a first mode based on the first cost value being equal to or less than the second cost value; The method of claim 6 , further comprising: determining the prediction mode as a second mode based on the first cost value being greater than the second cost value.

8. The step of generating prediction information includes: generating a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimation error for each vertex in the first vertex group; 8. The method of claim 7, further comprising: generating a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated difference associated with each vertex in the first vertex group.

9. The method of claim 7 , wherein the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

10. 8. The method of claim 7, wherein the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

11. 1. An apparatus for mesh processing including processing circuitry, The processing circuitry Dividing a plurality of vertices corresponding to a mesh of a current frame that have been positionally tracked into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimated error indicative of a difference between a reference vertex of the reference frame and a current vertex in the first group of vertices; encoding prediction modes for vertices in the first vertex group based on at least an estimated error between the reference vertex and the current vertex; generating prediction information for vertices in the first vertex group based on the encoded prediction modes of vertices in the first vertex group.

Images