METHOD AND APPARATUS FOR MESH PROCESSING - Patent application

Abstract

The method includes determining a plurality of candidate predictions for a current vertex, the current vertex being included in a plurality of vertices of a mesh, a plurality of the candidate predictions being determined based on a plurality of triangles in the mesh, each of the plurality of predictions corresponding to a respective candidate prediction, determining a prediction for the current vertex from the plurality of candidate predictions, the predictions being identified by a predetermined prediction index based on a number of the plurality of candidate predictions for the current vertex being equal to or greater than a threshold, encoding prediction information for the current vertex based on the determined predictions, the prediction information including a prediction index identifying a prediction for the current vertex.

Description

[Technical field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 359,669, filed on July 8, 2022, entitled "Vertex Position Coding in Mesh Compression," which claims priority to U.S. Provisional Application No. 18 / 144,686, filed on May 8, 2023, and entitled "Vertex Position Coding in Mesh Compression," all of which are incorporated by reference herein in their entirety.

[0002] The present disclosure includes embodiments that relate to mesh processing. [Background technology]

[0003] The "Background" discussion provided herein is intended to provide a general context for the present disclosure. The work of the inventors named herein, to the extent described in this Background, along with aspects of the description that would not ordinarily be considered prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) filming, modeling, and rendering have facilitated the ubiquity of 3D content across multiple platforms and devices. It is now possible to film a baby's first steps on one continent while the baby's grandparents can view (and in some cases interact with) that baby on another continent, enjoying a fully immersive experience. To achieve such realism, models are becoming ever more sophisticated, and vast amounts of data are linked to the creation and consumption of those models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Problem to be solved by the invention]

[0005] Aspects of the present disclosure provide a method and apparatus for mesh processing. In some examples, an apparatus for mesh processing includes a processing circuit. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, there is provided a method for mesh processing performed by a video encoder, comprising: determining a plurality of candidate predictions for a current vertex; the current vertex being included in a plurality of vertices of a mesh; determining the plurality of candidate predictions based on a plurality of triangles, each of the plurality of triangles corresponding to a respective candidate prediction; determining a prediction for the current vertex from the plurality of candidate predictions; identifying the prediction by a predetermined prediction index based on a number of the plurality of candidate predictions for the current vertex being equal to or greater than a threshold; encoding prediction information for the current vertex based on the determined prediction; and the prediction information including a prediction index identifying a prediction for the current vertex.

[0007] In some embodiments, the threshold is greater than or equal to 2 and less than or equal to 5.

[0008] In one example, to determine a plurality of candidate predictions for the current vertex, a first triangle of the plurality of triangles is determined for the current vertex, the first triangle having a shared edge with a second triangle of the plurality of triangles, the current vertex being included in the second triangle and on the opposite side of the shared edge, and a first candidate prediction of the plurality of candidate predictions is determined based on the first triangle determined by a parallelogram prediction, where the first candidate prediction and the first triangle form a first parallelogram.

[0009] In one example, to determine multiple candidate predictions for the current vertex, an average prediction of multiple of the candidate predictions is determined.

[0010] In one example, to determine a prediction for the current vertex, a prediction list for the current vertex is determined, where candidate predictors in the prediction list include an average prediction in the prediction list and a number of the candidate predictions following the average prediction, where an order of the candidate predictions in the prediction list is based on an order of the triangles corresponding to the candidate predictions, where each of the candidate predictors in the prediction list is associated with a respective prediction index.

[0011] In some embodiments, the plurality of triangles are ordered based on an edge breaker algorithm that labels them in a spiral triangle spanning tree order.

[0012] In one example, the prediction of the current vertex is determined as a first candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a first prediction index in response to the number of the plurality of the candidate predictions of the current vertex being equal to the threshold. In one example, the prediction of the current vertex is determined as a second candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a second prediction index in response to the number of the plurality of the candidate predictions of the current vertex being greater than the threshold. In one example, the prediction of the current vertex is determined as a third candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a third prediction index in response to the number of the plurality of the candidate predictions of the current vertex being equal to or greater than the threshold.

[0013] In one example, to determine a prediction for the current vertex, a prediction difference between each of the candidate predictors in the prediction list and the current vertex is determined, and further, the prediction corresponding to a minimum prediction difference among the prediction differences from the candidate predictors in the prediction list is determined.

[0014] In one example, to determine a prediction for the current vertex, the prediction with the smallest prediction difference within a subset of candidate predictors in the prediction list is determined.

[0015] In some embodiments, a respective prediction index is determined for each of the candidate predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

[0016] In some embodiments, the prediction information for the current vertex includes residual information, the residual information indicating a difference between the current vertex and the determined prediction. In some embodiments, the residual information may be encoded, for example, based on one of fixed length coding, exponential-Golomb coding, and arithmetic coding.

[0017] According to another aspect of the present disclosure, there is provided an apparatus, the apparatus including a processing circuit, the processing circuit may be configured to perform any of the methods for mesh processing described.

[0018] Aspects of the present disclosure further provide a non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for mesh processing. [Brief description of the drawings]

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0020] [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4A]FIG. 2 is a schematic diagram of a first exemplary parallelogram prediction according to some embodiments of the present disclosure. [Figure 4B] FIG. 13 is a schematic diagram of a second exemplary parallelogram prediction according to some embodiments of the present disclosure. [Figure 5A] We present five example patch configurations for the Edge Breaker algorithm. [Figure 5B] 1 illustrates an exemplary mesh traversal with the Edge Breaker algorithm. [Figure 6] FIG. 1 is a schematic diagram illustrating an example of predicting vertices based on parallelogram prediction; [Figure 7] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 8] 1 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] 1 illustrates a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application of the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter is equally applicable to other image and video enabled applications including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] The video processing system (100) includes a capture subsystem (113) that may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera creates a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples captured by the digital camera. The stream of video pictures (102), shown in bold to emphasize its high amount of data compared to the encoded video data (104) (or coded video bitstream), may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown with thin lines to emphasize its lower amount of data compared to the stream of video pictures (102), can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) may include a video decoder (110), for example, in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and creates an outgoing stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, an evolving video coding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0023] Additionally, electronic devices 120 and 130 may include other components (not shown). For example, electronic device 120 may include a video decoder (not shown), and electronic device 130 may include a video encoder (not shown).

[0024] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include a receiving circuit, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) in the example of FIG. 1.

[0025] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). The receiver (231) may separate the coded video sequences from the other data. To prevent network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be external to the video decoder (210) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (210) to, for example, prevent network jitter, and another buffer memory (215) internal to the video decoder (210) to, for example, handle playback timing. When the receiver (231) receives data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be necessary and may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, and may be relatively large, advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (210).

[0026] The video decoder (210) may include a parser (220) that reconstructs symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and potentially include information to control a rendering device such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230) as shown in FIG. 2. The control information for the rendering device may be in the form of a supplementary enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be aligned with a video coding technique or standard and may follow a variety of principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a set of subgroup parameters for at least one subgroup of pixels in a video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0027] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to produce symbols (221).

[0028] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0029] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into several functional units as described below. In an actual implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0030] The first unit is a scalar / inverse transform unit (251), which receives control information from the parser (220) including the transform to be used, block size, quantization factor, quantization scaling matrix, etc., and the quantized transform coefficients as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0031] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks, which are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information retrieved from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) adds, in some cases, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0032] In other cases, the output samples of the scaler / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (253) may access the reference picture memory (257) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (221) related to the block, these samples may be added to the output of the scaler / inverse transform unit (251) by the aggregator (255) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (257) from which the motion compensated prediction unit (253) retrieves prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (253), for example in the form of symbols (221) that may have X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​retrieved from the reference picture memory (257) when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0033] The output samples of the aggregator (255) may be applied by various loop filtering techniques in a loop filter unit (256). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also depend on meta-information obtained during decoding of previous parts (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed and loop filtered sample values.

[0034] The output of the loop filter unit (256) may be a sample stream that can be output to a rendering device (212) and also stored in a reference picture memory (257) for use in future inter-picture prediction.

[0035] Once a particular coded picture has been fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0036] The video decoder (210) may perform decoding operations according to a given video compression technology or standard, such as ITU-T Recommendation H.265. A coded video sequence may comply with a syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence complies with both the syntax of the video compression technology or standard and with a profile documented in the video compression technology or standard. In particular, a profile may select certain tools as the only tools available in the profile, from all tools available in the video compression technology or standard. It is also necessary for compliance that the complexity of the coded video sequence be within the range limited by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the coded video sequence.

[0037] In one embodiment, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0039] The video encoder (303) can receive video samples from a video source (301) (not part of the electronic device (320) in the example of FIG. 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0040] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g. 8-bit, 10-bit, 12-bit...), any color space (e.g. BT.601 Y CrCB, RGB...) and any suitable sampling structure (e.g. Y CrCb 4:2:0, Y CrCb 4:4:4). In a media delivery system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that, when viewed in sequence, give the impression of motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. The following description focuses on the samples.

[0041] According to one embodiment, the video encoder (303) can code and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraint required. Enforcing an appropriate coding rate is one of the functions of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) can include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured with other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0042] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (330) (e.g., responsible for creating symbols, such as a symbol stream based on the input picture to be coded and reference pictures), and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a similar manner as the (remote) decoder would create them. The reconstructed sample stream (sample data) is input to a reference picture memory (334). The content of the reference picture memory (334) is also bit-accurate between the local and remote encoders, since decoding of the symbol stream results in bit-accurate results regardless of the location of the (local or remote) decoder. In other words, the predictive part of the encoder "sees" as reference picture samples exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the drift that occurs when synchrony cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0043] The operation of the "local" decoder (333) may be similar to the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in connection with Figure 2. However, referring also briefly to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0044] In one embodiment, decoder techniques other than parsing / entropy decoding present in the decoder are present in the same or substantially the same functional form in the corresponding encoder. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted since they are the inverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0045] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as prediction references for the input picture.

[0046] The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (333) may replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in the reference picture memory (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content (no transmission errors) with the reconstructed reference pictures obtained by the far-end video decoder.

[0047] The predictor (335) can perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata that can serve as suitable prediction criteria for the new picture, such as the motion vectors of the reference picture, block shapes, etc. The predictor (335) can operate on a sample block / pixel block basis to find a suitable prediction criteria. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction criteria drawn from multiple reference pictures stored in the reference picture memory (334).

[0048] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The output of all of the aforementioned functional units may be entropy coded in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0050] The transmitter (340) can buffer the coded video sequence created by the entropy coder (345) in preparation for transmission over a communication channel (360), which can be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) can also merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] A controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned any of the following picture types:

[0052] An Intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of the variations of I-pictures and their respective uses and characteristics.

[0053] A predictive picture (P-picture) may be one that can be coded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] Bidirectionally predicted pictures (B-pictures) may be coded and decoded by intra- or inter-prediction using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source picture may generally be spatially subdivided into a number of sample blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture may be predictively coded via spatial or temporal prediction with reference to one pre-coded reference picture. Blocks of a B-picture may be predictively coded via spatial or temporal prediction with reference to one or two pre-coded reference pictures.

[0056] The video encoder (303) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. During operation, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard used.

[0057] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0058] A video may be shot as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as "intra prediction") exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0059] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, each of which is earlier in decoding order than a current picture in a video (but may be in the past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted from a combination of the first reference block and the second reference block.

[0060] Furthermore, merge mode techniques can be used for inter-picture prediction to improve coding efficiency.

[0061] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks, such as polygon or triangular blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking luma prediction blocks as an example of prediction blocks, prediction blocks include matrices of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0062] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0063] The present disclosure includes embodiments that relate to methods and systems for vertex position coding in mesh compression.

[0064] A mesh may include a number of polygons that describe the surface of a volumetric object. Each polygon of a mesh may be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information on how the vertices are connected (which may be referred to as connectivity information). In some embodiments, vertex attributes such as color, normal, etc. may be associated with the mesh vertices. Attributes (or vertex attributes) may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a two-dimensional (2D) attribute map. Such mapping may typically be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normal, displacement, etc. Such information may be used for various purposes such as texture mapping and shading.

[0065] Dynamic mesh sequences may require a large amount of data since dynamic meshes may contain a large amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards such as IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with always-on connectivity, time-varying geometry, and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information. Typically, DCC (Digital Content Creation) tools generate such dynamic meshes. However, generating always-on dynamic meshes, especially under real-time constraints, is challenging for volume acquisition techniques. This type of content (e.g., always-on dynamic meshes) may not be supported by existing standards. MPEG plans to develop a new mesh compression standard that directly handles dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. This new mesh compression standard targets lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, augmented reality (AR), and virtual reality (VR). Functionality such as random access and scalable / progressive coding are also considered.

[0066] Mesh geometry information can include vertex connectivity information, 3D coordinates, and 2D texture coordinates, etc. Compression of vertex 3D coordinates, also called vertex positions, is very important as they can often consume a significant portion of the overall geometry related data.

[0067] For simplicity and clarity, the present disclosure uses a triangular mesh as an example to describe the embodiments. However, other polygon meshes (e.g., quadrilateral meshes or pentagonal meshes) can also be applied to the embodiments of the present disclosure. In one example, a triangular mesh is a type of polygon mesh that includes a set of triangles. The set of triangles can be connected by a common edge or corner. The common edge or corner can be represented as a mesh vertex.

[0068] Vertex position compression can be implemented by parallelogram prediction. In parallelogram prediction, a compression algorithm can introduce a new vertex that is included in a predicted triangle. The predicted triangle can be formed based on edges shared with adjacent triangles. In one example, the adjacent triangles can be pre-coded. The new vertex can indicate a predicted position of the current vertex. The new vertex can form a parallelogram with three vertices included in the adjacent triangles. For example, as shown in FIG. 4A, a mesh (400A) can include multiple triangles, such as triangle (412). Each of the triangles can include three vertices. For example, triangle (412) can include vertices (406), (408), and (410). To code a current vertex (402), a predicted vertex (or prediction) (404) of the current vertex (402) can be generated based on a predicted triangle (414) according to parallelogram prediction. The predicted vertex (404) is included in the predicted triangle (414). The predicted triangle 414 may share an edge 416 with triangle 412. The vertices 406, 408, 410, and the predicted vertex 404 may form a parallelogram. Furthermore, the current vertex 402 is contained within a triangle 418 in the mesh 400A, and triangle 418 and triangle 412 share an edge 416.

[0069] Multiple parallelogram predictions can be applied to predict the current vertex. Multiple parallelogram predictions can, for example, use the average position obtained by two or more parallelogram predictions whenever they are available. FIG. 4B shows an example of two parallelogram predictions. As shown in FIG. 4B, the current vertex (420) is included in a first triangle (434) of the mesh (400B). The first triangle (434) can include vertices (420), (438), and (442). The first triangle (434) can share an edge (444) with a neighboring triangle (428) of the mesh (400B). The current vertex (420) is included in a second triangle (436) that includes vertices (420), (438), and (440). The second triangle (436) can share an edge (446) with a neighboring triangle (426) of the mesh (400B). Based on the shared edges (444), a first predicted vertex (or first prediction) (424) may be determined. The first predicted vertex (424) may form a first parallelogram with an adjacent triangle (428). Based on the shared edges (446), a second predicted vertex (or second prediction) (422) may be determined. The second predicted vertex (422) may form a second parallelogram with an adjacent triangle (426). Thus, the current vertex (420) may further be predicted by the average of the first predicted vertex (424) and the second predicted vertex (422).

[0070] In the present disclosure, a method and / or system of vertex position coding for mesh compression is provided. In one embodiment, a predicted position value (or prediction or predicted vertex) of a current vertex can be selected from multiple candidate position values ​​(or candidate predictions or candidate predicted vertices) based on a parallelogram prediction. The selected predicted vertex can be coded (or identified) by an index. A prediction residual between the selected predicted vertex and the current vertex can be further coded.

[0071] In this disclosure, many methods and / or systems can be proposed for vertex position prediction in mesh compression. It should be noted that these methods and / or systems can be applied individually or in any combination. It should be noted that these methods can be applied to static meshes as well as dynamic meshes. A static mesh may only include one frame, and the mesh content of a static mesh may not change over time. Furthermore, the disclosed methods and / or systems are not limited to vertex position prediction. The disclosed methods and / or systems can also be applied to, for example, 2D texture coordinate prediction, or more general multi-prediction based schemes.

[0072] The triangles in a triangular mesh (or meshes) may be ordered. The order of the triangles in the mesh may be traversed according to an edge breaker algorithm or other division algorithm. Additionally, in one example, the vertices in the mesh may also be ordered based on the order of the triangles. In one example, the order or traversal order of the triangles and vertices may be determined based on the same scheme or different schemes.

[0073] 5A and 5B show an example ordering of triangles and vertices in a mesh based on the Edge Breaker algorithm. FIG. 5A shows five example patch configurations for the Edge Breaker algorithm. As shown in FIG. 5A, v is the central vertex of the patch, and T is the current triangle. The active gate (or current triangle) in each patch can be represented as T. In patch C, a complete triangle can be provided that fans out (or rotates) around v. In patch L, one or more missing triangles can be located to the left of the active gate T. In patch R, one or more missing triangles can be located to the right of the active gate T. In patch E, v is adjacent to only T. In patch S, one or more missing triangles can be located in a position other than the left or right of the active gate T. FIG. 5B shows an example traversal of a mesh (500) in which the triangles of the mesh (500) can be ordered based on the traversal of the Edge Breaker algorithm. As shown in FIG. 5B, the triangles in the mesh (500) can be traversed along a spiral triangle spanning tree. For example, the traversal may start with a triangle (502) of type C (or C-patch). Then, the traversal may proceed along the branch adjacent to the right edge of the triangle (e.g., (502)). The traversal may stop when an E-type triangle (e.g., (504)) is reached. According to the edge breaker algorithm, the triangles of the mesh (500) may be traversed (or ordered) in the sequence CRSRLECRRRLE, which may be shown in FIG. 5B. The vertices of each of the triangles of the mesh (500) may also be ordered based on the triangle order.

[0074] In a triangular mesh, each triangle (also called a face) has three vertices. For two triangles that share an edge, parallelogram prediction can be applied to use one of the two opposite vertices of the two triangles as a predictor of the other. As shown in FIG. 6, two triangles ABC and DBC can share an edge BC. The coordinates of vertices A, B, and C that have already been coded can be applied to predict the coordinate of D. For example, a predicted vertex (or prediction) D' can be determined based on parallelogram prediction. The coordinate of the predicted vertex D' can be determined in equation (1) as follows:

[0075]

number

[0076] In this way, the four vertices (D', B, A, C) can form a parallelogram, as shown in Figure 6. Since each vertex has a 3D coordinate, equation (1) can be used to calculate each coordinate component of the predicted vertex D'. For example, assuming that the subscripts x, y, and z represent 3D coordinates in xyz space, the coordinate components of the predicted vertex D' in xyz space can be provided by equations (2)-(4).

[0077]

number

number

number

[0078] If the position values ​​of vertices A, B, and C can be coded and used for prediction, triangle ABC can be represented as a prediction candidate (or predicted triangle) for vertex D. Depending on the number of existing coded vertices and shared edges, a vertex may have zero, one, two, or more prediction candidates. If there are multiple prediction candidates for the current vertex, the prediction candidates can be ordered based on the order of the triangles in the mesh. For example, the triangles can be ordered based on an edge breaker algorithm, etc.

[0079] In the present disclosure, multiple predictions (or parallelogram predictions) of the current vertex may be determined. An average (or average prediction) of the multiple parallelogram predictions may further be determined. A prediction (or predicted vertex) of the current vertex may be selected from the prediction list. The predictors (or candidate predictors) of the prediction list may include, for example, the available parallelogram predictions and the average of the multiple parallelogram predictions. The associated prediction error (also called prediction residual) may be coded. The prediction error (or prediction residual) may indicate the difference between the prediction and the current vertex. In one embodiment, the prediction may correspond to the minimum prediction error between the current vertex and each of the predictors (or candidate predictors) in the prediction list. In one embodiment, the prediction may be predefined in the prediction list based on the number of predictors (or candidate predictors) in the prediction list being equal to or greater than a threshold. Thus, a particular (or predetermined) predictor in the prediction list may be selected as a prediction for the current vertex according to the number of predictors (or candidate predictors) in the prediction list being equal to or greater than a threshold.

[0080] In the present disclosure, all triangles in a triangular mesh may be ordered. For example, the triangles may be ordered based on an edge breaker algorithm. Also, all or a portion of the vertices in the triangular mesh may be ordered. In one example, the vertices may be ordered based on the order of the triangles.

[0081] Given a vertex V in a triangular mesh, the vertex V includes N prediction candidates (or prediction triangles). The N prediction candidates are N prediction values ​​V1', V2'...V N For example, as shown in FIG. 4B, a current vertex (420) may have two predicted triangles (430) and (432). Each of the predicted triangles (430) and (432) may generate a respective predicted value (or predicted vertex), e.g., (422) and (424), respectively. The predicted values ​​for the current vertex may be ordered based on the order of the corresponding predicted triangles in the mesh. If duplicates are determined in the predicted values, such duplicates may be removed from the list of predicted values.

[0082] In one example, if N=0, there is no prediction candidate available for vertex V. If V is the first vertex to be coded in the mesh, the predicted value V' of the first vertex can be set to 0 and the prediction residual R can be set to V. Thus, V'=0, R=V. If V is not the first vertex to be coded, the predicted value V' of V can be set to the coded value of a vertex that was previously coded in the vertex order. Thus, the previously coded vertex can be a predictor for vertex V. The prediction residual R can be determined as the difference between vertex V and the predicted value (or predicted vertex) V', i.e., R=V-V'.

[0083] In one example, if N=1, then there is only one prediction candidate available for vertex V. Therefore, a prediction value V' can be determined based on the only prediction candidate, and a prediction residual R can be determined as the difference between V and V', i.e., R=V-V'.

[0084] In one example, if N≧2, there are multiple prediction candidates available for vertex V. Therefore, multiple predicted values ​​(or predicted vertices) can be determined based on the multiple prediction candidates. In one embodiment, the average of the predicted values ​​can be considered as an additional predictor V0′. V0′ can be determined by equation (5).

[0085]

number

[0086] The average predicted value V0' is the predicted value V1', V2'...V N If either the average prediction V0' or the duplicate predictions are equal to V0' or V0', then either the average prediction V0' or the duplicate predictions can be removed.

[0087] In one embodiment, on the encoder side, V i ' can be compared to V, where 0 ≤ i ≤ N. A prediction index j can be selected. The selected prediction index j is the sum of V and the predicted value V i ' and V corresponding to the minimum prediction residual i '. In some embodiments, the minimization error (or minimum prediction residual) is 0 Norm, L 1 Norm, L 2 norm, or some other norm. For example, L 0 The norm can be determined in equation (6) as follows:

[0088]

number

[0089] Here, V x , V y , V z are the coordinates of V in the xyz space, and V 0x ', V 0y ', V 0z ' is the coordinate of V0' in xyz space.

[0090] At the decoder side, the selected index (e.g., prediction index) can be decoded, and the selected predictor (or predicted value) can be restored from a list of predictors (or candidate predictors). For example, the decoder can determine one or more predicted values ​​for the current vertex based on a prediction mode such as parallelogram prediction. Also, an average of multiple predicted values ​​can be considered as a predictor. The average predicted value and the one or more predicted values ​​can be ordered to form a prediction list. In one example, the prediction list generated at the decoder side can be the same as the prediction list generated at the encoder side. Based on the decoded prediction index, the selected predictor can be restored based on the prediction index from the prediction list.

[0091] In one embodiment, at the encoder side, if N is equal to some specific value (e.g., 2≦N≦5), a fixed prediction index (e.g., any of indexes 0, 1...N) may be selected in the prediction list. The fixed prediction index indicates a given predictor (or predicted vertex) of the current vertex. Therefore, the prediction index may not be coded. At the decoder side, the decoder may first build the prediction list. Then, the decoder may recover (or determine) the predictor based on the fixed prediction index. In one example, if N>2, prediction index 0 may be selected, which may indicate the average of multiple predictions. In one example, if N>2, the prediction index may not be coded.

[0092] In one embodiment, at the encoder side, a fixed prediction index may be selected or determined if N is equal to or greater than a particular value (or threshold). In one example, the particular value (or threshold) may be 2. In one example, the threshold (or particular value) may be between 2 and 10. Thus, all prediction indexes in the prediction list may not be coded. At the decoder side, the decoder may recover (or determine) a prediction index based on the number of N. For example, if N=2, prediction index 1 may be selected. If N>2, prediction index 0 may be selected. In another example, if n≧2, prediction index 0 may be selected.

[0093] In one embodiment, an upper limit M can be set. The upper limit M can indicate the number of prediction candidates that can be considered (or applied) in the prediction list. In some cases, if N>M, then for each vertex in the mesh, only the first M prediction candidates are considered. Thus, the first M prediction candidates V1', V2'...V M ' is applied. In one embodiment, the average of the first M predicted values ​​may be determined by equation (7).

[0094]

number

[0095] M can be an integer, such as 4. Thus, up to M (e.g., 4) prediction candidates can be considered. Note that the average prediction V0' can be placed at different positions in the candidate list. In one example, the average prediction V0' can be the first predictor in the prediction candidate list. In one example, the average prediction V0' can be the last predictor in the prediction candidate list. In one example, the average prediction V0' is the first ... M ' can be placed in

[0096] If a vertex V has N prediction candidates V1', V2'...V N ' and the mean V0', the selected prediction index may be coded.

[0097] In some embodiments, the prediction index needs to be coded only if there are multiple prediction candidates. For example, if there are more than one prediction candidate currently available for a vertex, an index indicating the selected prediction candidate can be coded at the encoder side. At the decoder side, the decoder can determine the two or more prediction candidates in the same order as the encoder. The decoder can decode the coded prediction index and reconstruct the selected prediction candidate based on the prediction index from the two or more prediction candidates.

[0098] In one embodiment, if no prediction candidate is determined or only one prediction candidate is determined, the prediction index does not need to be coded. Thus, the predictor value may be the current vertex itself and may be the only prediction candidate. In one embodiment, the prediction index may always be coded regardless of the number of available predictor candidates. If no predictor is available, the signaled index value may not affect the decoding process at the decoder side.

[0099] In some embodiments, the prediction index may not need to be coded if a fixed prediction index is always selected or predetermined for some particular value of N. Thus, the prediction index of a predictor (or candidate predictor) in the candidate list may not need to be coded, and the prediction index selected from the candidate list may not need to be signaled to the decoder.

[0100] In some embodiments, the prediction index may not be fixed and may be selected from prediction candidates. If multiple prediction candidates are determined for the current vertex, the prediction index may be coded, for example, using fixed-length coding. For example, if three prediction candidates are determined for the current vertex, four possible prediction indices may be required: 0, 1, 2, and 3 (0 indicates the average value of candidates 1, 2, and 3). Therefore, two-digit binary numbers may be applied to represent each of the four prediction indices. It should be noted that different vertices of a triangular mesh may use different fixed lengths. For example, if another vertex has seven prediction candidates, the other vertex may use a three-digit binary number to represent the prediction index. The output of the fixed-length coding may be further compressed by entropy coding, such as arithmetic coding.

[0101] Alternatively, the prediction index can be coded using variable length coding. For example, if four prediction candidates are determined for the current vertex, five possible prediction indexes may be required: 0, 1, 2, 3, and 4. Variable length codes of 0, 100, 101, 110, and 111 can be assigned to represent the five prediction indexes: 0, 1, 2, 3, and 4, respectively. Alternatively, variable length codes of 1, 01, 001, 0001, and 00001 can be applied to represent the five prediction indexes: 0, 1, 2, 3, and 4, respectively. Note that different vertices of the triangular mesh may use different variable lengths. The output of the variable length coding can be further compressed by entropy coding, such as arithmetic coding.

[0102] Alternatively, the prediction index can be differentially coded, where the prediction index of the current vertex can be predicted from a previously coded prediction index, and the prediction error associated with the prediction index can also be coded. For example, the difference between a first prediction index of the predictor and a second prediction index of the predictor can be coded first. The second prediction index can be further coded based on the difference between the first prediction index and the second prediction index.

[0103] If an upper limit M is set to limit the maximum number of prediction candidates, then if there are more than M prediction candidates available for vertex V, only the first M prediction candidates may be used to predict vertex V. Thus, (M+1) possible prediction indices may be coded, where M indicates the first M prediction candidates and 1 indicates the average of the first M prediction candidates. If duplicates are determined in the predicted values, the duplicates may be removed. Thus, the probability of a prediction index may also be reduced. If a fixed prediction index is always selected, the prediction index may not be coded. If the prediction index is not fixed but is selected from among multiple prediction candidates, the prediction index may be coded. The prediction index may be coded by fixed length coding, variable length coding, differential coding, etc.

[0104] In one embodiment, a prediction residual R of a position value (or predicted value) can be coded (or determined). The prediction residual R can indicate the difference between the predicted value of the current vertex and the current vertex. The prediction residual R can be coded by fixed-length coding, exponential-Golomb coding, arithmetic coding, etc. Thus, the decoder can reconstruct the current vertex based on the predicted value and the prediction residual, and the predicted value can be determined based on a decoded prediction index from a prediction list.

[0105] 7 is a flow chart outlining a process (700) according to one embodiment of the present disclosure. The process (700) may be used in an encoder, such as a video encoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit performing the functions of a video encoder (103), a processing circuit performing the functions of a video encoder (303), or the like. In some embodiments, the process (700) is implemented with software instructions, such that the processing circuit performs the process (700) when executing the software instructions. The process starts at (S701) and proceeds to (S710).

[0106] In step S710, a plurality of candidate predictions of a current vertex are determined. The current vertex is included in the plurality of vertices of the mesh. The plurality of candidate predictions are determined based on a plurality of triangles. Each of the plurality of triangles corresponds to a respective one of the candidate predictions.

[0107] In step S720, a prediction for the current vertex is determined from the plurality of candidate predictions, the prediction being identified by a predetermined prediction index based on the number of the plurality of candidate predictions for the current vertex being equal to or greater than a threshold.

[0108] In step S730, prediction information for the current vertex is encoded based on the determined prediction, where the prediction information includes a prediction index that identifies the prediction for the current vertex.

[0109] In some embodiments, the threshold is greater than or equal to 2 and less than or equal to 5.

[0110] In one example, to determine multiple candidate predictions for a current vertex, a first triangle of the multiple triangles is determined for the current vertex, the first triangle having a shared edge with a second triangle of the multiple triangles, the current vertex is included in the second triangle and is on the opposite side of the shared edge, and the first candidate prediction of the multiple candidate predictions is determined based on the determined first triangle by a parallelogram prediction, where the first candidate prediction and the first triangle form a first parallelogram.

[0111] In one example, to determine multiple candidate predictions for the current vertex, an average prediction of the multiple candidate predictions is determined.

[0112] In one example, to determine a prediction for the current vertex, a prediction list for the current vertex is determined, where the candidate predictors in the prediction list include an average prediction and a number of candidate predictions following the average prediction in the prediction list, where the order of the number of candidate predictions in the prediction list is based on the order of the number of triangles corresponding to the number of candidate predictions, where each of the candidate predictors in the prediction list is associated with a respective prediction index.

[0113] In some embodiments, the triangles are ordered based on an edge breaker algorithm that labels them in a spiral triangle spanning tree order.

[0114] In one example, in response to the number of the plurality of candidate predictions of the current vertex being equal to a threshold value, a prediction of the current vertex is determined as a first candidate predictor of the candidate predictors in the prediction list according to a predetermined prediction index based on the prediction index being predefined as a first prediction index. In one example, in response to the number of the plurality of candidate predictions of the current vertex being greater than a threshold value, a prediction of the current vertex is determined as a second candidate predictor of the candidate predictors in the prediction list according to a predetermined prediction index based on the prediction index being predefined as a second prediction index. In one example, in response to the number of the plurality of candidate predictions of the current vertex being equal to or greater than a threshold value, a prediction of the current vertex is determined as a third candidate predictor of the candidate predictors in the prediction list according to a predetermined prediction index.

[0115] In one example, to determine a prediction for the current vertex, a prediction difference between each of the candidate predictors in the prediction list and the current vertex is determined, and a prediction corresponding to a minimum prediction difference among the prediction differences from the candidate predictors in the prediction list is further determined.

[0116] In one example, to determine the prediction for the current vertex, the prediction with the smallest prediction difference within the subset of candidate predictors in the prediction list is determined.

[0117] In some embodiments, a respective prediction index is determined for each of the candidate predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

[0118] The process then proceeds to (S799) and ends.

[0119] The process (700) may be adapted as appropriate. Steps of the process (700) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0120] 8 is a flow chart outlining a process (800) according to one embodiment of the present disclosure. The process (800) may be used in a decoder, such as a video decoder. In various embodiments, the process (800) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), or the like. In some embodiments, the process (800) is implemented with software instructions, such that the processing circuit performs the process (800) when executing the software instructions. The process starts at (S801) and proceeds to (S810).

[0121] In step S810, coding information of a mesh including a plurality of triangles and a plurality of vertices is received.

[0122] In step S820, a plurality of candidate predictions of a current vertex are determined, the current vertex being included in the plurality of vertices of the mesh, and the plurality of candidate predictions are determined based on a plurality of triangles, each of the plurality of triangles corresponding to a respective one of the candidate predictions.

[0123] In step S830, a prediction of the current vertex is determined from the plurality of candidate predictions, the prediction being indicated by a predetermined prediction index determined based on the number of the plurality of candidate predictions of the current vertex being equal to or greater than a threshold.

[0124] In step S840, the current vertex is reconstructed based on the determined prediction.

[0125] The process then proceeds to (S899) and ends.

[0126] The process 800 may be adapted as appropriate. Steps of the process 800 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0127] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media, such as a non-transitory computer-readable storage medium. For example, Figure 9 illustrates a computer system (900) suitable for implementing certain embodiments of the disclosed subject matter.

[0128] The computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly configured to create code containing instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or other processing circuitry, or by interpretation, microcode execution, or the like.

[0129] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0130] 9 for computer system (900) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system (900).

[0131] The computer system (900) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0132] The input human interface devices may include one or more of a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data glove (not shown), a joystick (905), a microphone (906), a scanner (907), and a camera (908) (only one of each shown).

[0133] The computer system (900) may further include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (910), data gloves (not shown), or joystick (905), but may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (909), headphones (not shown)), visual output devices (such as screens (910) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, haptic feedback capability, some of which may output two-dimensional visual output or three or more dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0134] The computer system (900) may further include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (920) or similar media (921) with CDs / DVDs, thumb drives (922), and removable hard drives or solid state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0135] Those skilled in the art should further understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0136] The computer system (900) may further include an interface (954) to one or more communication networks (955). The network may be, for example, a wireless, wired, or optical network. The network may further be a local, wide area, metropolitan, vehicular, industrial, real-time, delay tolerant network, and the like. Examples of networks include local area networks such as Ethernet and wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks including CAN bus and the like. Certain networks typically require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (949) (e.g., a USB port of the computer system (900)). Other networks are typically integrated into the core of the computer system (900) by connecting to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system (900) can communicate with other entities. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from a CAN bus to a particular CAN bus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. As noted above, specific protocols and protocol stacks may be used for each of these networks and network interfaces.

[0137] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to a core (940) of the computer system (900).

[0138] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (943), hardware accelerators for specific tasks (944), and graphics adapters (950), etc. These devices may be connected via a system bus (948), along with read only memory (ROM) (945), random access memory (946), and internal mass storage devices (947) such as internal hard drives, SSDs, etc. that are not accessible to the user. In some computer systems, the system bus (948) is accessible in the form of one or more physical plugs, allowing expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (948) or through a peripheral bus (949). In one example, a screen (910) may be connected to the graphics adapter (950). Peripheral bus architectures include PCI, USB, etc.

[0139] The CPU (941), GPU (942), FPGA (943) and accelerator (944) may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (945) or RAM (946). Transient data may also be stored in RAM (946), while permanent data may be stored, for example, in internal mass storage (947). A cache memory, which may be closely associated with one or more of the CPU (941), GPU (942), mass storage (947), ROM (945), RAM (946), etc., may be used to enable fast storage and retrieval from any memory device.

[0140] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0141] By way of example and not of limitation, a computer system having the architecture (900), and in particular the core (940), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with a user-accessible mass storage device as introduced above, as well as specific storage of the core (940) that is non-transitory in nature, such as the core internal mass storage device (947) or ROM (945). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core (940), and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (946) and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (944)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may include logic, where appropriate, and vice versa. References to computer-readable media may include circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software.

[0142] While this disclosure has described some exemplary embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of this disclosure. It should thus be understood that those skilled in the art can invent numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are within its spirit and scope.

Claims

1. 1. A method for mesh processing performed by a video encoder, comprising: determining a plurality of candidate predictions for a current vertex, the current vertex being included in a plurality of vertices of a mesh, the plurality of candidate predictions being determined based on a plurality of triangles in the mesh, each of the plurality of triangles corresponding to a respective one of the candidate predictions; determining a prediction for the current vertex from a plurality of the candidate predictions, the prediction being identified by a predetermined prediction index based on a number of the plurality of candidate predictions for the current vertex being equal to or greater than a threshold; encoding prediction information for the current vertex based on the determined prediction, the prediction information including a prediction index that identifies a prediction for the current vertex.

2. The method of claim 1 , wherein the threshold is greater than or equal to 2 and less than or equal to 5.

3. The step of determining a plurality of candidate predictions for the current vertex includes: determining a first triangle of the plurality of triangles for the current vertex, the first triangle having a shared edge with a second triangle of the plurality of triangles, the current vertex being contained within the second triangle and on an opposite side of the shared edge; 2. The method of claim 1 , further comprising: determining the first candidate prediction based on the first triangle determined by a parallelogram prediction, where a first candidate prediction of the plurality of candidate predictions and the first triangle form a first parallelogram.

4. The step of determining a plurality of candidate predictions for the current vertex includes: The method of claim 3 , further comprising determining an average prediction of a plurality of the candidate predictions.

5. The step of determining a prediction of the current vertex comprises:

5. The method of claim 4, further comprising: determining a prediction list for the current vertex, wherein candidate predictors in the prediction list include an average prediction in the prediction list and a plurality of the candidate predictions following the average prediction, an order of the plurality of candidate predictions in the prediction list based on an order of the plurality of triangles corresponding to the plurality of candidate predictions, and each of the candidate predictors in the prediction list is associated with a respective prediction index.

6. The method of claim 5 , wherein the plurality of triangles are ordered based on an edge breaker algorithm that labels them in a spiral triangle spanning tree order.

7. The step of determining a prediction of the current vertex comprises: determining a prediction of the current vertex as a first candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index, based on the prediction index being predetermined as a first prediction index, in response to a number of the plurality of candidate predictions of the current vertex being equal to the threshold; determining a prediction of the current vertex as a second candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a second prediction index in response to the number of the plurality of candidate predictions of the current vertex being greater than the threshold; and determining a prediction of the current vertex as a third candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a third prediction index in response to the number of the plurality of candidate predictions of the current vertex being equal to or greater than the threshold.

8. The step of determining a prediction of the current vertex comprises: determining a prediction difference between each of the candidate predictors in the prediction list and the current vertex; and determining the prediction that corresponds to a minimum one of the prediction differences from candidate predictors in the prediction list.

9. The step of determining a prediction of the current vertex comprises: The method of claim 5 , further comprising determining a prediction having a minimum prediction difference within a subset of candidate predictors of the prediction list.

10. The step of determining each prediction index comprises:

6. The method of claim 5, further comprising determining a respective prediction index for each of the candidate predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

11. the prediction information of the current vertex includes residual information indicating a difference between the current vertex and the determined prediction, The method comprises: The method of claim 1 , further comprising the step of encoding the residual information.

12. 1. An apparatus for mesh processing comprising a processing circuit, The processing circuitry includes: determining a plurality of candidate predictions for a current vertex, the current vertex being included in a plurality of vertices of a mesh, the plurality of candidate predictions being determined based on a plurality of triangles in the mesh, each of the plurality of triangles corresponding to a respective one of the candidate predictions; determining a prediction for the current vertex from a plurality of the candidate predictions, the prediction being identified by a predetermined prediction index based on a number of the plurality of candidate predictions for the current vertex being equal to or greater than a threshold; encoding prediction information for the current vertex based on the determined prediction, the prediction information including a prediction index that identifies a prediction for the current vertex.

13. The apparatus of claim 12 , wherein the threshold is greater than or equal to 2 and less than or equal to 5.

14. The processing circuitry includes: determining a first triangle of the plurality of triangles for the current vertex, the first triangle having a shared edge with a second triangle of the plurality of triangles, the current vertex being contained within the second triangle and on an opposite side of the shared edge; 13. The apparatus of claim 12, further configured to: determine the first candidate prediction based on the first triangle determined by a parallelogram prediction, where a first candidate prediction of the plurality of candidate predictions and the first triangle form a first parallelogram.

15. The processing circuitry includes: The apparatus of claim 14 , further configured to perform the step of determining an average prediction of a plurality of the candidate predictions.

16. The processing circuitry includes:

16. The apparatus of claim 15, further configured to perform a step of determining a prediction list for the current vertex, wherein candidate predictors in the prediction list include an average prediction in the prediction list and a plurality of the candidate predictions following the average prediction, an order of the plurality of candidate predictions in the prediction list based on an order of the plurality of triangles corresponding to the plurality of candidate predictions, and each of the candidate predictors in the prediction list is associated with a respective prediction index.

17. The apparatus of claim 16 , wherein the plurality of triangles are ordered based on an edge breaker algorithm labeled with a spiral triangle spanning tree order.

18. The processing circuitry includes: determining a prediction of the current vertex as a first candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index, based on the prediction index being predetermined as a first prediction index, in response to a number of the plurality of candidate predictions of the current vertex being equal to the threshold; determining a prediction of the current vertex as a second candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a second prediction index in response to the number of the plurality of candidate predictions of the current vertex being greater than the threshold; 17. The apparatus of claim 16, further configured to: in response to a number of the plurality of candidate predictions of the current vertex being equal to or greater than the threshold, determine a prediction of the current vertex as a third candidate predictor of candidate predictors in the prediction list according to the predetermined prediction index based on the prediction index being predetermined as a third prediction index.

19. The processing circuitry includes: determining a prediction difference between each of the candidate predictors in the prediction list and the current vertex; 17. The apparatus of claim 16, further configured to perform the step of: determining the prediction that corresponds to a minimum one of the prediction differences from candidate predictors in the prediction list.

20. The processing circuitry includes: The apparatus of claim 16 , further configured to perform the step of determining a prediction having a minimum prediction difference within a subset of candidate predictors of the prediction list.

Images