Mesh vertex grouping for position compression

Abstract

The method includes: dividing a plurality of vertices corresponding to a mesh of a current frame that is positionally tracked into a plurality of vertex groups based on a predetermined constant integer; the plurality of vertex groups includes a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimation error indicating a difference between a reference vertex of the reference frame and a current vertex in the first vertex group; encoding a prediction mode of a vertex in the first vertex group based on at least the estimated error between the reference vertex and the current vertex; generating prediction information of a vertex in the first vertex group based on the encoded prediction mode of a vertex in the first vertex group.

Description

[Technical field]

[0001] This application claims priority to U.S. provisional application serial number 63 / 388525, filed July 12, 2022, entitled "Grouping of Mesh Vertices in Position Compression," and to U.S. patent application serial number 18 / 144703, filed May 8, 2023, entitled "GROUPING OF MESH VERTICES IN POSITION COMPRESSION," the entire contents of which are incorporated herein by reference.

[0002] The present disclosure includes embodiments that relate to mesh processing. [Background technology]

[0003] The "Background" discussion provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein, to the extent described in this Background, along with aspects of the description that would not ordinarily be considered prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) filming, modeling, and rendering have facilitated the ubiquity of 3D content across multiple platforms and devices. It is now possible to film a baby's first steps on one continent while the baby's grandparents can view (and potentially interact with) the baby on another continent, enjoying a fully immersive experience. To achieve such realism, models are becoming ever more sophisticated, and vast amounts of data are linked to the creation and consumption of those models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Problem to be solved by the invention]

[0005] Aspects of the present disclosure provide methods and apparatus for mesh processing. In some examples, an apparatus for mesh processing includes a processing circuit. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a mesh processing method is provided for execution by a video encoder. The method includes: dividing a plurality of vertices corresponding to a positionally tracked mesh of a current frame into a plurality of vertex groups based on a predetermined constant integer; the plurality of vertex groups includes a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex of a reference frame different from the current frame; determining an estimation error indicating a difference between a reference vertex of the reference frame and a current vertex in the first vertex group; encoding a prediction mode of a vertex in the first vertex group based on at least the estimated error between the reference vertex and the current vertex; generating prediction information of a vertex in the first vertex group based on the encoded prediction mode of a vertex in the first vertex group.

[0007] In some embodiments, the method includes determining an average neighbor estimation error of a plurality of neighbor estimation errors, the plurality of neighbor estimation errors being associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicative of a difference between a respective one of a plurality of neighbor vertices of a current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of a current vertex in the first vertex group, determining a first cost value based on at least the estimation error associated with the current vertex, determining a second cost value based at least on an average neighbor estimation error associated with the current vertex, and encoding the prediction mode based on at least the first cost value and the second cost value.

[0008] In some embodiments, a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as a current vertex in the first vertex group in the current frame.

[0009] In one example, to determine an average neighbor estimation error associated with the plurality of adjacent vertices, a first neighbor estimation error associated with a first adjacent vertex of a plurality of adjacent vertices in the first vertex group is determined, the first neighbor estimation error indicating a difference between a first adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the first adjacent vertex of the current vertex. A second neighbor estimation error associated with a second adjacent vertex of a plurality of adjacent vertices in the first vertex group is determined, the second neighbor estimation error indicating a difference between a second adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the second adjacent vertex of the current vertex. An average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error is determined, the average neighbor estimation error being associated with a current vertex in the first vertex group. An estimated difference between an estimation error of the current vertex and an average neighbor estimation error associated with the current vertex is further determined.

[0010] In some embodiments, the first cost value is determined as a sum of estimation errors of vertices in the first group of vertices, the estimation errors of the vertices including the estimation error of the current vertex.

[0011] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each of the vertices in the first group of vertices and an average neighboring estimated error associated with corresponding vertices in the first group of vertices is determined, the second cost value is determined as a sum of estimated differences associated with vertices in the first group of vertices, the estimated difference including an estimated difference associated with the current vertex.

[0012] In one example, the prediction mode is determined as a first mode based on the first cost value being less than or equal to the second cost value. In another example, the prediction mode is determined as a second mode based on the first cost value being greater than the second cost value.

[0013] In some embodiments, the method generates a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimated error for each of the vertices in the first vertex group, and generates a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated error associated with each of the vertices in the first vertex group.

[0014] In some embodiments, the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

[0015] In some embodiments, the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

[0016] According to another aspect of the present disclosure, there is provided an apparatus, the apparatus including a processing circuit, the processing circuit may be configured to perform any of the methods for mesh processing described.

[0017] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for mesh processing. [Brief description of the drawings]

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4] FIG. 2 is a schematic diagram of vertices of a mesh according to some embodiments of the present disclosure. [Diagram 5] FIG. 13 is a schematic diagram of an exemplary vertex position compression according to some embodiments of the present disclosure. [Figure 6] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 7] 1 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] 1 illustrates a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application of the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter is equally applicable to other image and video enabled applications including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0020] The video processing system (100) includes a capture subsystem (113) that may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera may create a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples captured by the digital camera. The stream of video pictures (102), shown in bold to emphasize the high amount of data compared to the encoded video data (104) (or coded video bitstream), may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown with thin lines to emphasize its lower amount of data compared to the stream of video pictures (102), can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include, for example, a video decoder (110) in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an outgoing stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, an evolving video coding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0021] Additionally, electronic devices 120 and 130 may include other components (not shown). For example, electronic device 120 may include a video decoder (not shown), and electronic device 130 may include a video encoder (not shown).

[0022] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include a receiving circuit, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) in the example of FIG. 1.

[0023] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, with the decoding of each coded video sequence being independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (231) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be external to the video decoder (210) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (210) to, for example, prevent network jitter, and another buffer memory (215) internal to the video decoder (210) to, for example, handle playback timing. When the receiver (231) receives data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be necessary and may be made smaller. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, and may be relatively large, advantageously of adaptive size, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (210).

[0024] The video decoder (210) may include a parser (220) that reconstructs symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and potentially include information to control a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230) as shown in FIG. 2. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be aligned with a video coding technique or standard and may follow a variety of principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a set of subgroup parameters for at least one subgroup of pixels in a video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0025] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to create symbols (221).

[0026] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0027] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into several functional units as described below. In an actual implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0028] The first unit is a scalar / inverse transform unit (251), which receives control information from the parser (220) including the transform to be used, block size, quantization factor, quantization scaling matrix, etc., as well as the quantized transform coefficients as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0029] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks, which do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information retrieved from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0030] In other cases, the output samples of the scaler / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (253) may access the reference picture memory (257) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (221) related to the block, these samples may be added to the output of the scaler / inverse transform unit (251) by the aggregator (255) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (257) from which the motion compensated prediction unit (253) retrieves prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (253), for example in the form of symbols (221) that may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (257) when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0031] The output samples of the aggregator (255) may be applied by various loop filtering techniques in a loop filter unit (256). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also depend on meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed and loop filtered sample values.

[0032] The output of the loop filter unit (256) may be a sample stream that can be output to a rendering device (212) and also stored in a reference picture memory (257) for use in future inter-picture prediction.

[0033] Once a particular coded picture has been fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before beginning reconstruction of the following coded picture.

[0034] The video decoder (210) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Recommendation H.265. A coded video sequence may comply with a syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence complies with both the syntax of the video compression technique or standard and with a profile documented in the video compression technique or standard. In particular, a profile may select certain tools as the only tools available in the profile, from all tools available in the video compression technique or standard. It is also necessary for compliance that the complexity of the coded video sequence be within the range limited by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the coded video sequence.

[0035] In one embodiment, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0036] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0037] The video encoder (303) can receive video samples from a video source (301) (not part of the electronic device (320) in the example of FIG. 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0038] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g. 8-bit, 10-bit, 12-bit...), any color space (e.g. BT.601 Y CrCB, RGB...) and any suitable sampling structure (e.g. Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that, when viewed in sequence, give the impression of motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. The following discussion focuses on samples.

[0039] According to one embodiment, the video encoder (303) can code and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraint required. Enforcing an appropriate coding rate is one of the functions of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) can include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured with other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0040] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (330) (e.g., responsible for creating symbols, such as a symbol stream based on the input picture to be coded and reference pictures), and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a similar manner as the (remote) decoder would create them. The reconstructed sample stream (sample data) is input to a reference picture memory (334). The content of the reference picture memory (334) is also bit-accurate between the local and remote encoders, since the decoding of the symbol stream gives bit-accurate results regardless of the location of the (local or remote) decoder. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder "sees" when using prediction during decoding, as reference picture samples. This basic principle of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0041] The operation of the "local" decoder (333) may be similar to the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in connection with Figure 2. However, with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0042] In one embodiment, decoder techniques other than parsing / entropy decoding present in the decoder are present in the same or substantially the same functional form in the corresponding encoder. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted since they are the inverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0043] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as prediction references for the input picture. The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (333) may replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in the reference picture memory (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content (no transmission errors) with the reconstructed reference pictures obtained by the far-end video decoder.

[0044] The predictor (335) can perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata that can serve as suitable prediction criteria for the new picture, such as the motion vectors of the reference picture, block shapes, etc. The predictor (335) can operate on a sample block / pixel block basis to find a suitable prediction criteria. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction criteria drawn from multiple reference pictures stored in the reference picture memory (334).

[0045] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0046] The output of all of the aforementioned functional units may be entropy coded in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0047] The transmitter (340) can buffer the coded video sequence created by the entropy coder (345) in preparation for transmission over a communication channel (360), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) can also merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0048] A controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned any of the following picture types:

[0049] An Intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I-pictures and their respective uses and characteristics.

[0050] A predictive picture (P-picture) may be one that can be coded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0051] Bidirectionally predicted pictures (B-pictures) may be coded and decoded by intra- or inter-prediction using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than one reference picture and associated metadata for the reconstruction of a single block.

[0052] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to one pre-coded reference picture. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to one or two pre-coded reference pictures.

[0053] The video encoder (303) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Recommendation H.265. During operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard used.

[0054] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0055] A video may be shot as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as "intra prediction") exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called a current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0056] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, each of which is earlier in decoding order than a current picture in a video (but may be in the past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted from a combination of the first reference block and the second reference block.

[0057] Furthermore, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0058] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking luma prediction blocks as an example of prediction blocks, prediction blocks include matrices of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0059] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0060] The present disclosure includes embodiments that relate to methods and systems for grouping mesh vertices for positional compaction.

[0061] A mesh may include a number of polygons that describe the surface of a volumetric object. Each polygon of a mesh may be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information on how the vertices are connected (which may be referred to as connectivity information). In some embodiments, vertex attributes such as color, normal, etc. may be associated with the mesh vertices. Attributes (or vertex attributes) may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a two-dimensional (2D) attribute map. Such mapping may typically be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normal, displacement, etc. Such information may be used for various purposes such as texture mapping and shading.

[0062] Dynamic mesh sequences may require a large amount of data since dynamic meshes may contain a large amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards such as IC, MESHGRID and FAMC were previously developed by MPEG to address dynamic meshes with always-on connectivity, time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information. Typically, DCC (Digital Content Creation) tools generate such dynamic meshes. However, generating always-on dynamic meshes, especially under real-time constraints, is challenging for volume acquisition techniques. This type of content (e.g., always-on dynamic meshes) may not be supported by existing standards. MPEG plans to develop a new mesh compression standard that directly handles dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. This new mesh compression standard targets lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, augmented reality (AR), virtual reality (VR), etc. Functionality such as random access and scalable / progressive coding are also considered.

[0063] Mesh geometry information can include vertex connectivity information, 3D coordinates, 2D texture coordinates, etc. Compression of vertex 3D coordinates, also called vertex positions, is important because they can often consume a significant portion of the overall geometry-related data.

[0064] A dynamic mesh sequence M at time instance t can be represented as M(t). If there is a mapping (or mapping operation) f from vertex positions in M(t) to vertex positions at another time instance M(t0), where t and t0 are different time instances, M(t) can be called a positional tracking frame, and thus M(t0) can be called a reference frame, and the corresponding vertices in the reference frame can be called reference vertices for the vertices in M(t).

[0065] In this disclosure, methods and / or systems are proposed for grouping mesh vertices in position compression (e.g., vertex position compression). It should be noted that these methods and / or systems may be applied individually or in any combination. Furthermore, the disclosed methods and systems are not limited to vertex position compression. The disclosed methods and systems may also be applied, for example, to two-dimensional (2D) texture coordinate compression or more general temporal prediction-based schemes.

[0066] For a vertex V in the positional tracking frame M(t), the neighbors of a vertex may be vertices that are connected to V via edges, and these vertices are called the neighbors (or adjacent vertices) of V. For example, as shown in Figure 4, vertex A may have four neighbors: C, D, E, and B. Vertex E may have five neighbors: A, B, F, H, and D.

[0067] Figure 5 is a schematic diagram of an example vertex position compression device (500) according to some embodiments of the present disclosure. As shown in Figure 5, the vertex position compression device (500) can include a vertex grouping module (502) configured to group vertices of a mesh, a position prediction module (504) configured to calculate predicted positions of the vertices, a prediction mode coding module (506) configured to code (or determine) a prediction mode for the position compression, and a prediction residual coding module (508) configured to code a position prediction residual.

[0068] The term module in this disclosure may refer to a software module, a hardware module, or a combination thereof. A software module (e.g., a computer program) may be developed using a computer programming language. A hardware module may be implemented using processing circuitry and / or memory. Each module may be implemented using one or more processors (or a processor and memory). Similarly, a processor (or a processor and memory) may be used to implement one or more modules. Furthermore, each module may be part of an entire module that includes the functionality of that module.

[0069] In this disclosure, temporal prediction can be applied to predict a vertex of a current frame based on a reference vertex of a reference frame. For a vertex V of a positionally tracked frame (or current frame) M(t), the position of the vertex V can be estimated by the position of a reference vertex f(V) of a reference frame (e.g., M(t0), where t0 is a different time instance), where f is a mapping operation between M(t) and the reference frame. In some embodiments, the vertex V and the reference vertex f(V) of the reference frame are aligned. Thus, the reference vertex can have the same relative position in the reference frame as the vertex in the current frame M(t). When the vertex V is predicted from the reference vertex f(V), an estimation error E can be determined as the difference between the position of V and the position of f(V) in equation (1).

[0070] E=Vf(V) Equation (1)

[0071] Since each vertex of the frame M(t) can have a 3D coordinate, the 3D coordinate components of the estimated error E can be provided based on equation (1). For example, assuming that the subscripts x, y, and z represent 3D coordinates in the xyz space, the 3D coordinate components of the estimated error E can be provided by equations (2) to (4).

[0072] E x =V x -(f(V)) x Formula (2)

[0073] E y =V y -(f(V)) y Formula (3)

[0074] E z =V z -(f(V)) z Formula (4)

[0075] The estimated error E of a vertex V can be predicted or determined from the neighboring vertices (or adjacent vertices) of vertex V. For the neighboring vertices (or adjacent vertices) of V, if the neighboring vertices are coded and available for prediction, the estimated error of the neighboring vertices can be applied to predict E.

[0076] Let V be a set of N neighboring vertices (or adjacent vertices) V1, V2...V N Suppose we have a neighboring vertex V i For the neighboring vertex V i Let E be the estimated error (or the adjacent estimated error) i =V i -f(V i ) (i=1, 2…N) can be determined as f(V i ) is the neighboring vertex V in the reference frame. i Also, E i can be called the neighbor estimation error associated with vertex V, and each E i can be a predictive candidate for E.

[0077] If N ≥ 2, multiple estimation errors E i is available. In one embodiment, the estimation error E i The average estimation error (or average neighboring estimation error) E0 of can be defined as follows in equation (5):

[0078] E0=(E1+E2+…+E N ) / N formula (5)

[0079] For a group G of a frame M(t), the encoder may determine a coding cost C0 and a coding cost C1. C0 may denote the sum of the estimated errors E of all vertices in the group G. For example, if the estimated errors E of each of the vertices in the group G are determined based on equation (1), the coding cost C0 may be determined as the sum of the estimated errors E of the vertices in the group G. C1 may denote the sum of the estimated residuals (E-E0) of all vertices in the group G. To determine the coding cost C1, the estimated errors E of the neighboring vertices of each vertex in the group G may be determined as i E i =V i -f(V i ) can be determined as follows. Furthermore, the estimation error E i The average neighbor estimation error E0 of may be determined based on, for example, equation (5). An estimated residual (or estimated difference) (E-E0) may be determined for each of the vertices in group G. A coding cost C1 may be determined as the sum of the estimated residuals associated with the vertices in group G.

[0080] In this disclosure, a prediction mode may be determined based on the coding cost C0 and the coding cost C1.

[0081] If the coding cost C0 is less than or equal to C1, prediction mode 0 is applied to group G. Prediction mode 0 indicates that the prediction residual of each of the vertices in group G is the estimated error E of each vertex. Therefore, at the decoder side, each vertex in group G can be reconstructed as V=f(V)+E, where f(V) is the reference vertex of reference frame M(t0) corresponding to each vertex in group G of frame M(t).

[0082] If the coding cost C0 is greater than C1, prediction mode 1 is applied to group G. Prediction mode 1 indicates that the prediction residual of each vertex in group G is (E-E0), where E is the estimation error of each vertex, and E0 is the average estimation error of the estimation errors of the neighboring vertices of each vertex. Therefore, at the decoder side, each vertex in group G can be reconstructed as V=f(V)+(E-E0), where f(V) is the reference vertex of reference frame M(t0) corresponding to each vertex in group G in frame M(t).

[0083] In this disclosure, a prediction mode of group G may be coded. The prediction mode may be a binary number such as 0 or 1. In one embodiment, the prediction mode may be coded using entropy coding. In one embodiment, the prediction mode may be coded using arithmetic coding. In one embodiment, the prediction mode may be coded using context-based arithmetic coding, such as context-adaptive binary arithmetic coding (CABAC). In one embodiment, the prediction mode may be coded using spatial context-based arithmetic coding, where the context may be conditioned on (or based on) previously coded groups of the same frame.

[0084] In one embodiment, if the prediction mode in a reference frame (e.g., M(t0)) is already coded and available, the prediction mode of the group in frame M(t) can be coded using a temporal context, such as temporal context-based arithmetic coding. Thus, based on the temporal context-based arithmetic coding, the prediction mode of the group in frame M(t) can be determined based on the prediction mode of the corresponding (or associated) group in the reference frame. The reason is that each vertex in the group in frame M(t) has a corresponding reference vertex in the associated group in the reference frame, and a one-to-one association between the group in the positional tracking frame M(t) and the associated group in the reference frame M(t0) can also be established. The associated group in the reference frame (e.g., M(t0), where t0 is a different time instance from t) can be called a reference group.

[0085] In one embodiment, a logical operation (or flag) such as XOR (exclusive OR) may be coded to indicate a relationship between the prediction mode of group G and the prediction mode of the reference group of group G. Thus, if group G and the reference group have the same prediction mode, the logical operation XOR may be coded as 0. If group G and the reference group have different prediction modes, the logical operation XOR may be coded as 1.

[0086] In one embodiment, a binary flag (e.g., 0 or 1) may be coded to indicate whether group G and the reference group of group G have the same prediction mode. Thus, if group G and the reference group have the same prediction mode, a binary 1 may be coded. If group G and the reference group have different prediction modes, a binary 0 may be coded.

[0087] In one embodiment, the prediction mode of group G in the positional tracking frame M(t) may be coded using temporal context-based arithmetic coding, and the context of the temporal context-based arithmetic coding may be based on the prediction mode of a reference group of group G.

[0088] In some embodiments, the prediction residual (e.g., E or (E-E0)) may be coded using a coding algorithm such as fixed length coding, exponential-Golomb coding, arithmetic coding, etc. In some embodiments, the prediction residual may undergo a compression transform such as a fast Fourier transform (FFT), a discrete cosine transform (DCT), a discrete sine transform (DST), a discrete wavelet transform (DWT), etc. The output from the compression transform may be coded using a coding algorithm such as fixed length coding, exponential-Golomb coding, arithmetic coding, etc.

[0089] 6 is a flow chart outlining a process (600) according to one embodiment of the present disclosure. The process (600) may be used in an encoder, such as a video encoder. In various embodiments, the process (600) is performed by a processing circuit, such as a processing circuit performing the functions of a video encoder (103), a processing circuit performing the functions of a video encoder (303), or the like. In some embodiments, the process (600) is implemented with software instructions, such that the processing circuit performs the process (600) when executing the software instructions. The process starts at (S601) and proceeds to (S610).

[0090] In step S610, a plurality of vertices corresponding to a mesh of the current frame that are positionally tracked are divided into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group.

[0091] In (S620), a first position of a current vertex in the first vertex group is estimated based on a second position of a reference vertex in a reference frame different from the current frame.

[0092] At (S630), an estimated error is determined that indicates a difference between the reference vertex in the reference frame and the current vertex in the first vertex group.

[0093] At (S640), prediction modes of vertices in the first vertex group are encoded based on at least the estimated error between the reference vertex and the current vertex.

[0094] At (S650), prediction information for vertices in the first vertex group is generated based on the encoded prediction modes of vertices in the first vertex group.

[0095] In some embodiments, the method includes determining an average neighbor estimation error of a plurality of neighbor estimation errors, the plurality of neighbor estimation errors being associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicative of a difference between a respective one of a plurality of neighbor vertices of a current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of a current vertex in the first vertex group, determining a first cost value based on the estimation error associated with at least the current vertex, determining a second cost value based on at least the average neighbor estimation error associated with the current vertex, and encoding the prediction mode based on at least the first cost value and the second cost value.

[0096] In some embodiments, a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as a current vertex in the first vertex group in the current frame.

[0097] In one example, to determine an average neighbor estimation error associated with the plurality of neighbor vertices, a first neighbor estimation error associated with a first neighbor vertex of a plurality of neighbor vertices in the first vertex group is determined. The first neighbor estimation error indicates a difference between a first neighbor vertex of the current vertex and a reference vertex of a reference frame corresponding to the first neighbor vertex of the current vertex. A second neighbor estimation error associated with a second neighbor vertex of a plurality of neighbor vertices in the first vertex group is determined. The second neighbor estimation error indicates a difference between a second neighbor vertex of the current vertex and a reference vertex of a reference frame corresponding to the second neighbor vertex of the current vertex. An average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error is determined, the average neighbor estimation error being associated with the current vertex in the first vertex group. An estimated difference between the estimation error of the current vertex and an average neighbor estimation error associated with the current vertex is further determined.

[0098] In some embodiments, the first cost value is determined as a sum of estimated errors of the vertices in the first group of vertices, where the estimated errors of the vertices include the estimated error of the current vertex.

[0099] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each of the vertices in the first group of vertices and an average neighboring estimated error associated with corresponding vertices in the first group of vertices is determined, and the second cost value is determined as a sum of estimated differences associated with vertices in the first group of vertices, the estimated difference including an estimated difference associated with the current vertex.

[0100] In one example, the prediction mode is determined as a first mode based on the first cost value being less than or equal to the second cost value. In one example, the prediction mode is determined as a second mode based on the first cost value being greater than the second cost value.

[0101] In some embodiments, the method further comprises generating a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimated error for each of the vertices in the first vertex group, and generating a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated error associated with each of the vertices in the first vertex group.

[0102] In some embodiments, the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

[0103] In some embodiments, the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

[0104] The process then proceeds to (S699) and ends.

[0105] The process 600 may be adapted as appropriate. Steps of the process 600 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0106] 7 is a flow chart outlining a process (700) according to one embodiment of the present disclosure. The process (700) may be used in a decoder, such as a video decoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), or the like. In some embodiments, the process (700) is implemented with software instructions, such that the processing circuit performs the process (700) when executing the software instructions. The process starts at (S701) and proceeds to (S710).

[0107] At (S710), coding information of a first vertex group of a mesh in a current frame is received, the coding information indicating a prediction mode of the vertices in the first vertex group.

[0108] At (S720), based on the coding information indicating a first prediction mode of the prediction modes, determine estimation errors for vertices in the first vertex group, each of the estimation errors indicating a difference between a reference vertex of a reference frame corresponding to a vertex in the first vertex group and a corresponding vertex in the first vertex group.

[0109] At (S720), an average neighbor estimation error of a plurality of neighbor vertices associated with each of the vertices in the first vertex group is determined based on coding information indicating a second prediction mode of the prediction modes, each of the plurality of neighbor estimation errors indicating a difference between one of the plurality of neighbor vertices of each vertex in the first vertex group and a reference vertex of a reference frame corresponding to the one of the plurality of neighbor vertices of each vertex in the first vertex group.

[0110] At (S730), the prediction residuals of each vertex in the first vertex group are determined as (i) an estimation error for each vertex in the first vertex group based on the coding information indicating the first prediction mode, and (ii) an average adjacent estimation error associated with each vertex in the first vertex group based on the coding information indicating the second prediction mode.

[0111] At (S740), the vertices in the first vertex group are reconstructed based on the determined prediction residuals.

[0112] The process then proceeds to (S799) and ends.

[0113] The process (700) may be adapted as appropriate. Steps of the process (700) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0114] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 8 illustrates a computer system (800) suitable for implementing certain embodiments of the disclosed subject matter.

[0115] The computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly configured to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or by interpretation, microcode execution, etc.

[0116] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0117] 8 for computer system (800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system (800).

[0118] The computer system (800) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0119] The input human interface devices may include one or more of a keyboard (801), a mouse (802), a trackpad (803), a touch screen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808) (only one of each shown).

[0120] The computer system (800) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (810), data gloves (not shown), or joystick (805), but may also be haptic feedback devices that do not function as input devices), audio output devices (speakers (809), headphones (not shown), etc.), visual output devices (such as screens (810), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, haptic feedback capability, some of which may output two-dimensional visual output or three or more dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0121] The computer system (800) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (820) or similar media (821) with CDs / DVDs, thumb drives (822), and removable hard drives or solid state drives (823), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0122] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0123] The computer system (800) may also include an interface (854) to one or more communication networks (855). The networks may be, for example, wireless, wired, or optical networks. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay-tolerant networks, and the like. Examples of networks include local area networks, such as Ethernet and wireless LAN, cellular networks, including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks, including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks, including CAN bus, and the like. Certain networks typically require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (849) (e.g., a USB port of the computer system (800)). Other networks are typically integrated into the core of the computer system (800) by connecting to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (800) can communicate with other entities. Such communications may be, for example, unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from a CAN bus to a particular CAN bus device), or bidirectional, to other computer systems using local or wide area digital networks. As noted above, specific protocols and protocol stacks may be used for each of these networks and network interfaces.

[0124] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to a core (840) of the computer system (800).

[0125] The cores (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (843), hardware accelerators for specific tasks (844), and graphics adapters (850), etc. These devices may be connected via a system bus (848), along with read-only memory (ROM) (845), random access memory (846), and internal mass storage devices (847), such as internal hard drives, SSDs, etc., that are not accessible to the user. In some computer systems, the system bus (848) is accessible in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (848) or through a peripheral bus (849). In one example, a screen (810) may be connected to the graphics adapter (850). Peripheral bus architectures include PCI, USB, etc.

[0126] The CPU (841), GPU (842), FPGA (843) and accelerator (844) may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (845) or RAM (846). Transient data may also be stored in RAM (846), while permanent data may be stored, for example, in internal mass storage device (847). A cache memory, which may be closely associated with one or more of the CPU (841), GPU (842), mass storage device (847), ROM (845), RAM (846), etc., may be used to enable fast storage and retrieval from any memory device.

[0127] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0128] By way of example and not of limitation, a computer system having the architecture (800), and in particular the core (840), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as introduced above, as well as specific storage of the core (840) that is non-transitory in nature, such as the core internal mass storage device (847) or ROM (845). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (840). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core (840), and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (844)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may include logic, where appropriate, and vice versa. References to computer-readable media may include circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0129] While this disclosure has described some exemplary embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of this disclosure. It should thus be understood that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are within its spirit and scope.

Claims

1. 1. A mesh processing method performed by a video encoder, comprising: Dividing a plurality of vertices corresponding to a mesh of a current frame that are positionally tracked into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimated error indicative of a difference between a reference vertex of the reference frame and a current vertex in the first group of vertices; encoding prediction modes for vertices in the first vertex group based on at least an estimated error between the reference vertex and the current vertex; generating prediction information for vertices in the first vertex group based on encoded prediction modes of vertices in the first vertex group.

2. determining an average neighbor estimation error of a plurality of neighbor estimation errors associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicating a difference between a respective one of a plurality of neighbor vertices of a current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of a current vertex in the first vertex group; determining a first cost value based on at least an estimated error associated with the current vertex, and determining a second cost value based on at least an average neighbor estimated error associated with the current vertex; The method of claim 1 , wherein encoding the prediction mode further comprises encoding the prediction mode based on at least the first cost value and the second cost value.

3. The method of claim 1 , wherein a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as a current vertex in the first vertex group in the current frame.

4. The step of determining an average adjacent estimation error associated with the plurality of adjacent vertices comprises: determining a first adjacent error estimation associated with a first adjacent vertex of a plurality of adjacent vertices in the first vertex group, the first adjacent error estimation indicating a difference between a first adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the first adjacent vertex of the current vertex; determining a second adjacent estimation error associated with a second adjacent vertex of a plurality of adjacent vertices in the first vertex group, the second adjacent estimation error indicating a difference between a second adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the second adjacent vertex of the current vertex; determining an average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error associated with a current vertex in the first vertex group; The method of claim 2 , further comprising: determining an estimated difference between the current vertex's estimated error and an average neighbor estimated error associated with the current vertex.

5. The step of determining a first cost value comprises:

5. The method of claim 4, further comprising determining the first cost value as a sum of estimation errors of vertices in the first group of vertices, the estimation errors of the vertices including an estimation error of the current vertex.

6. The step of determining the second cost value comprises: determining an estimated difference between an estimated error for each of the vertices in the first group of vertices and an average neighboring estimated error associated with each vertex in the first group of vertices; 6. The method of claim 5, further comprising: determining the second cost value as a sum of estimated difference associated with vertices in the first group of vertices, the estimated difference including an estimated difference associated with the current vertex.

7. The step of encoding the prediction mode includes: determining the prediction mode as a first mode based on the first cost value being equal to or less than the second cost value; The method of claim 6 , further comprising: determining the prediction mode as a second mode based on the first cost value being greater than the second cost value.

8. The step of generating prediction information includes: generating a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimation error for each vertex in the first vertex group; 8. The method of claim 7, further comprising: generating a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated difference associated with each vertex in the first vertex group.

9. The method of claim 7 , wherein the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

10. 8. The method of claim 7, wherein the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

11. 1. An apparatus for mesh processing comprising a processing circuit, The processing circuitry includes: Dividing a plurality of vertices corresponding to a mesh of a current frame that are positionally tracked into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; estimating a first position of a current vertex in the first vertex group based on a second position of a reference vertex in a reference frame different from the current frame; determining an estimated error indicative of a difference between a reference vertex of the reference frame and a current vertex in the first group of vertices; encoding prediction modes for vertices in the first vertex group based on at least an estimated error between the reference vertex and the current vertex; generating prediction information for vertices in the first vertex group based on encoded prediction modes of vertices in the first vertex group.

12. The processing circuitry includes: determining an average neighbor estimation error of a plurality of neighbor estimation errors associated with a plurality of neighbor vertices of a current vertex in the first vertex group, each of the plurality of neighbor estimation errors indicating a difference between a respective one of a plurality of neighbor vertices of a current vertex in the first vertex group and a reference vertex of a reference frame corresponding to each of the plurality of neighbor vertices of a current vertex in the first vertex group; determining a first cost value based on at least an estimated error associated with the current vertex and determining a second cost value based on at least an average neighbor estimated error associated with the current vertex; 12. The apparatus of claim 11, further configured to: encode the prediction mode based on at least the first cost value and the second cost value.

13. The apparatus of claim 11 , wherein a reference vertex corresponding to a current vertex in the first vertex group is positioned in the reference frame at the same relative position as a current vertex in the first vertex group in the current frame.

14. the processing circuitry determines a first adjacent estimation error associated with a first adjacent vertex of a plurality of adjacent vertices in the first vertex group, the first adjacent estimation error indicating a difference between a first adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the first adjacent vertex of the current vertex; determining a second adjacent estimation error associated with a second adjacent vertex of a plurality of adjacent vertices in the first vertex group, the second adjacent estimation error indicating a difference between a second adjacent vertex of the current vertex and a reference vertex of a reference frame corresponding to the second adjacent vertex of the current vertex; determining an average neighbor estimation error of the first neighbor estimation error and the second neighbor estimation error associated with a current vertex in the first vertex group; determining an estimated difference between an estimated error of the current vertex and an average neighboring estimated error associated with the current vertex.

15. The processing circuitry includes:

15. The apparatus of claim 14, further configured to perform the step of determining the first cost value as a sum of estimation errors of vertices in the first group of vertices, the estimation errors of the vertices including an estimation error of the current vertex.

16. The processing circuitry includes: determining an estimated difference between an estimated error for each of the vertices in the first group of vertices and an average neighboring estimated error associated with each vertex in the first group of vertices; 16. The apparatus of claim 15, further configured to perform the steps of: determining the second cost value as a sum of estimated difference associated with vertices in the first group of vertices, the estimated difference including an estimated difference associated with the current vertex.

17. The processing circuitry includes: determining the prediction mode as a first mode based on the first cost value being equal to or less than the second cost value; 17. The apparatus of claim 16, further configured to: determine the prediction mode as a second mode based on the first cost value being greater than the second cost value.

18. The processing circuitry includes: generating a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating an estimation error for each vertex in the first vertex group; 20. The apparatus of claim 17, further configured to perform: generating a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicating an estimated difference associated with each vertex in the first vertex group.

19. 20. The apparatus of claim 17, wherein the prediction mode is coded based on one of entropy coding, arithmetic coding, context-based arithmetic coding, spatial context-based arithmetic coding, and temporal context-based arithmetic coding.

20. 20. The apparatus of claim 17, wherein the prediction information includes a flag indicating whether the first vertex group and a corresponding reference group in the reference frame have the same prediction mode, the corresponding reference group including reference vertices of vertices in the first vertex group, and the flag includes one of an exclusive-or (XOR) flag and a binary flag.

Images