Texture Coordinate Encoding in Mesh Compression

Abstract

A plurality of candidate predictions of a current vertex are determined. The current vertex is one of a plurality of vertices of the mesh. Each of the plurality of candidate predictions is determined based on each of a plurality of triangles in the mesh and includes a two-dimensional (2D) coordinate in a 2D region. An average prediction of the plurality of candidate predictions is calculated. A 2D prediction of the current vertex is selected from a prediction list including the plurality of candidate predictions and the average prediction. A three-dimensional (3D) prediction of the current vertex is determined based on one or more neighboring vertices of the current vertex in the mesh. A prediction residual including a prediction error of the current vertex is encoded. The prediction residual is obtained based on the selected 2D prediction of the current vertex and the determined 3D prediction.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims benefit of priority to U.S. Provisional Application No. 63 / 388,522, entitled "Texture Coordinate Coding in Mesh Compression," filed on July 12, 2022, which claims benefit of priority to U.S. Provisional Application No. 18 / 215,230, entitled "TEXTURE COORDINATE CODING IN MESH COMPRESSION," filed on June 28, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] The present disclosure includes embodiments that relate to mesh processing. [Background technology]

[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent described in this Background section, is not admitted expressly or impliedly as prior art to the present disclosure, as are aspects of the specification that may not be admitted as prior art at the time of filing.

[0004] Advances in three-dimensional (3D) capture, modeling, and rendering have facilitated the ubiquitous presence of 3D content across various platforms and devices. Today, a baby's first steps can be captured on one continent while that baby's grandparents on another continent watch (and potentially interact with) and enjoy a fully immersive experience with the baby. To achieve such realism, models are becoming increasingly sophisticated, and a significant amount of data is associated with the creation and consumption of those models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide a method and apparatus for mesh processing. In some examples, the apparatus for mesh processing includes a processing circuit.

[0006] According to one aspect of the present disclosure, a method of mesh processing performed in a video encoder is provided. In the method, a plurality of candidate predictions of a current vertex are determined. The current vertex is one of a plurality of vertices of the mesh. Each of the plurality of candidate predictions is determined based on each of a plurality of triangles in the mesh and includes a two-dimensional (2D) coordinate in a 2D domain. An average prediction of the plurality of candidate predictions is calculated. A 2D prediction of the current vertex is selected from a prediction list including the plurality of candidate predictions and the average prediction. The selected 2D prediction corresponds to a threshold prediction difference between the current vertex and each predictor in the prediction list. A three-dimensional (3D) prediction of the current vertex is determined based on one or more neighboring vertices of the current vertex in the mesh. The 3D prediction includes a 3D coordinate in the 3D domain. A prediction residual including a prediction error of the current vertex is encoded. The prediction residual is obtained based on the selected 2D prediction of the current vertex and the determined 3D prediction.

[0007] In one example, to determine multiple candidate predictions for the current vertex, a first triangle of the multiple triangles is determined for the current vertex, the first triangle shares an edge with a second triangle of the multiple triangles, and the current vertex is included in the second triangle and is on the opposite side of the shared edge. The first prediction of the multiple candidate predictions is determined based on the determined first triangle via a parallelogram prediction, where the first prediction and the first triangle form a first parallelogram.

[0008] In some embodiments, to select a 2D prediction for the current vertex, a prediction list for the current vertex is determined. The predictors in the prediction list include an average prediction and a plurality of candidate predictions following the average prediction in the prediction list. The order of the plurality of candidate predictions in the prediction list is based on the order of the plurality of triangles corresponding to the plurality of candidate predictions. A prediction index for each predictor in the prediction list is further determined.

[0009] In some embodiments, the triangles are ordered based on an edge breaker algorithm in which the triangles are labeled in a spiral triangle spanning tree order.

[0010] In some embodiments, to select a 2D prediction for the current vertex, a prediction difference between each of the predictors in the prediction list and the current vertex is determined. The 2D prediction is selected from the predictors in the prediction list such that the selected 2D prediction corresponds to the smallest prediction difference of the prediction differences.

[0011] In some embodiments, a 2D prediction is selected within the subset of the prediction list such that the 2D prediction corresponds to the smallest prediction difference within the subset of the prediction list.

[0012] In one example, the 3D prediction of the current vertex is determined based on a delta prediction, where the 3D prediction is determined as a neighboring vertex of one of the one or more neighboring vertices. In one example, the 3D prediction of the current vertex is determined based on a parallelogram prediction, where the 3D prediction and three neighboring vertices of the one or more neighboring vertices form a parallelogram.

[0013] In some embodiments, to encode the prediction residual, a predicted value of the current vertex is determined based on one of the 2D prediction and the 3D prediction, the one of the 2D prediction and the 3D prediction having a smaller prediction difference than another one of the 2D prediction and the 3D prediction, and a prediction residual indicating a difference between the predicted value and the current vertex is further encoded.

[0014] In some embodiments, to encode the prediction residual, a prediction value of the current vertex is determined based on an average of the 2D prediction and the 3D prediction, and a prediction residual indicating the difference between the prediction value and the current vertex is further encoded.

[0015] In one example, index information indicating a prediction index of a selected 2D prediction in a prediction list is encoded. In one example, a flag is further encoded, the flag indicating that a prediction value of the current vertex is determined based on one of a 2D prediction, a 3D prediction, or an average of a 2D prediction and a 3D prediction.

[0016] According to another aspect of the present disclosure, an apparatus is provided, the apparatus including a processing circuit, the processing circuit may be configured to perform any of the described methods for mesh processing.

[0017] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform any of the methods for mesh processing.

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4A] FIG. 2 is a schematic diagram of a first exemplary parallelogram prediction according to some embodiments of the present disclosure. [Figure 4B] FIG. 13 is a schematic diagram of a second exemplary parallelogram prediction according to some embodiments of the present disclosure. [Figure 5A] 5 shows an example of a patch configuration for the edge breaker algorithm. [Figure 5B] 1 illustrates an exemplary mesh traversal with the Edge Breaker algorithm. [Figure 6] FIG. 13 is a schematic diagram of an example of predicting vertices based on parallelogram prediction; [Figure 7] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 8] 1 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] 1 shows a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application of the disclosed subject matter and a video encoder and video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other image and video enabled applications including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] The video processing system (100) includes a capture subsystem (113) that may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera creates a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples taken by the digital camera. The stream of video pictures (102) is shown in bold to emphasize its large amount of data when compared to the encoded video data (104) (or encoded video bitstream) and may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown with thin lines to emphasize its small amount of data when compared to the stream of video pictures (102), can be stored on the streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to obtain copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example in an electronic device (130). The video decoder (110) decodes an input copy (107) of the encoded video data and creates an output stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown).In some streaming systems, the encoded video data (104), (107), (109) (e.g., a video bitstream) may be encoded according to a particular video encoding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video encoding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0022] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0023] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include a receiving circuit, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) of the example of FIG. 1.

[0024] The receiver (231) can receive one or more encoded video sequences to be decoded by the video decoder (210). In one embodiment, one encoded video sequence is received at a time, and the decoding of each encoded video sequence is independent of the decoding of the other encoded video sequences. The encoded video sequences can be received from a channel (201), which can be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) can receive the encoded video data with other data, e.g., encoded audio data and / or auxiliary data streams, which can be forwarded to a respective using entity (not shown). The receiver (231) can separate the encoded video sequences from the other data. To combat network jitter, a buffer memory (215) can be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other applications, it may be external to the video decoder (210) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (210), e.g., to combat network jitter, and another buffer memory (215) internal to the video decoder (210), e.g., to handle playout timing. When the receiver (231) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be needed and may be relatively large, advantageously adaptively sized, and may be implemented at least partially within an operating system or similar element (not shown) external to the video decoder (210).

[0025] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the encoded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and potential information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230), as shown in FIG. 2. The control information for the rendering device(s) may be in the form of a Supplemental Enhancement Information (SEI) message, or a parameter set fragment (not shown) of a Video Usability Information (VUI). The parser (220) may parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence may conform to a video encoding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser (220) can extract from the coded video sequence a set of subgroup parameters for at least one of a subgroup of pixels in a video decoder based on at least one parameter corresponding to the group. The subgroups can include a collection of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (220) can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0026] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to produce symbols (221).

[0027] The reconstruction of the symbols (221) may require different units depending on the type of encoded video picture or part thereof (inter-picture and intra-picture, inter-block and intra-block, etc.) and other factors. Which units are required and how can be controlled by subgroup control information parsed from the encoded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0028] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0029] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients and control information from the parser (220) as symbol(s) (221), including the transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0030] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) may add the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251) on a sample-by-sample basis.

[0031] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensation prediction unit (253) may access the reference picture memory (257) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (221) associated with the block, these samples may be added by the aggregator (255) to the output of the scalar / inverse transform unit (251) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion compensation prediction unit (253) fetches the prediction samples may be controlled by a motion vector, available to the motion compensation prediction unit (253), for example in the form of a symbol (221) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0032] The output samples of the aggregator (255) may be subject to various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and are available to the loop filter unit (256) from the parser (220) as symbols (221). Video compression may also accommodate meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and may also accommodate previously reconstructed and loop filtered sample values.

[0033] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) or stored in a reference picture memory (257) for use in future inter-picture prediction.

[0034] Once a particular coded picture has been fully reconstructed, it may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture has been fully reconstructed and the coded picture has been identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and the new current picture buffer may be relocated before beginning reconstruction of a subsequent coded picture.

[0035] The video decoder (210) may perform decoding operations according to a predefined video compression technique or standard, such as ITU-T Recommendation H.265. The encoded video sequence may comply with the syntax specified by the video compression technique or standard being used, in the sense that the encoded video sequence adheres to both the syntax of the video compression technique or standard and to the profile documented in the video compression technique or standard. Specifically, the profile may select certain tools from among all tools available in the video compression technique or standard as tools that are only available to them under the profile. Also, what is required for compliance may be that the complexity of the encoded video sequence falls within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the encoded video sequence.

[0036] In one embodiment, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0037] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the example video encoder (103) of FIG.

[0038] The video encoder (303) can receive video samples from a video source (301) (not part of the electronic device (320) in the example of FIG. 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0039] The video source (301) may provide a source video sequence to be encoded by the video encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following description focuses on samples.

[0040] According to one embodiment, the video encoder (303) can encode and compress pictures of a source video sequence into an encoded video sequence (343) in real-time, or under any other time constraint as needed. Enforcing an appropriate encoding rate is a function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured with other appropriate functions associated with the video encoder (303) optimized for a particular system design.

[0041] In some embodiments, the video encoder (303) is configured to operate in an encoding loop. As an oversimplified explanation, in one example, the encoding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to that which a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since the decoding of the symbol stream produces bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is also used in some related techniques.

[0042] The operation of the "local" decoder (333) may be the same as the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in conjunction with Figure 2. However, with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0043] In one embodiment, the decoder techniques, except for parsing / entropy decoding, present in the decoder, are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the decoder operation. The description of the encoder techniques can be omitted since they are the opposite of the decoder techniques described generically. Only in certain areas will more detailed descriptions be provided below.

[0044] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0045] The local video decoder (333) may decode the encoded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (330). The operation of the encoding engine (332) may advantageously be a lossy process. When the encoded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained (without transmission errors) by the far-end video decoder.

[0046] The predictor (335) may perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or some metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (335) may operate on one sample block per pixel block to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0047] The controller (350) can manage the encoding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0048] The output of all the aforementioned functional units may be entropy coded in an entropy coder (345), which converts the symbols produced by the various functional units into an encoded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0049] The transmitter (340) may buffer the encoded video sequence(s) created by the entropy coder (345) to prepare it for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the encoded video data from the video encoder (303) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0050] The controller (350) can manage the operation of the video encoder (303). During encoding, the controller (350) can assign a particular encoded picture type to each encoded picture, which can affect the encoding technique that can be applied to each picture. For example, pictures are often assigned as one of the following picture types:

[0051] An intra picture (I-picture) may be a picture that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0052] A predictive picture (P picture) may be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0053] A bidirectionally predicted picture (B-picture) may be one that can be encoded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multiple predicted picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0054] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the individual picture of the block. For example, blocks of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction with reference to one previously coded reference picture or via temporal prediction. Blocks of a B picture may be predictively coded via spatial prediction with reference to one or two previously coded reference pictures or via temporal prediction.

[0055] The video encoder (303) may perform encoding operations according to a predefined video encoding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0056] In one embodiment, the transmitter (340) can transmit additional data along with the encoded video. The source coder (330) can include such data as part of the encoded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0057] A video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0058] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but may be past and future, respectively, in display order). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0059] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.

[0060] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, 16×16 pixels, etc. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed on a prediction block basis. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0061] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0062] The present disclosure includes embodiments directed to methods and systems for texture coordinate encoding in mesh compression.

[0063] A mesh may include a number of polygons that represent the surface of a volumetric object. Each polygon of a mesh may be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information, also called connectivity information, of how the vertices are connected. In some embodiments, vertex attributes, such as color, normal, etc., may be associated with the mesh vertices. Attributes (or vertex attributes) may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a two-dimensional (2D) attribute map. Such mapping may be described by a set of parametric coordinates, usually called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps may be used to store high-resolution attribute information, such as texture, normal, displacement, etc. Such information may be used for various purposes, such as texture mapping and shading.

[0064] Dynamic mesh sequences may require a large amount of data since dynamic meshes may contain a significant amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards such as IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards may not consider time-varying attribute maps and connectivity information. DCC (Digital Content Creation) tools typically generate such dynamic meshes. However, it may be difficult to generate constant connectivity dynamic meshes with volume acquisition techniques, especially under real-time constraints. This type of content (e.g., constant connectivity dynamic meshes) may not be supported by existing standards. MPEG plans to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. The new mesh compression standard targets lossy and lossless compression for various applications, such as real-time communication, storage, free viewpoint video, augmented reality (AR), and virtual reality (VR). Features such as random access and scalable / progressive coding are also possible.

[0065] Mesh geometry information can include vertex connectivity information, 3D coordinates, and 2D texture coordinates, etc. Compression of vertex 3D coordinates, which may also be referred to as vertex positions, can be very important in many cases because compressing vertex 3D coordinates can consume a significant portion of the overall geometry-related data.

[0066] In this disclosure, for simplicity and clarity, the embodiments are described using a triangular mesh as an example. However, other polygon meshes (e.g., quadrilateral or pentagonal meshes) can also be applied to the embodiments of the present disclosure. In one example, a triangular mesh is a type of polygon mesh that includes a set of triangles. The set of triangles can be connected by a common edge or corner of the set of triangles. The common edge or corner can be denoted as a mesh vertex.

[0067] Vertex position compression may be implemented by parallelogram prediction. In parallelogram prediction, the compression algorithm may introduce a new vertex that is included in a predicted triangle. The predicted triangle may be formed based on edges shared with adjacent triangles. In one example, the adjacent triangle may be pre-encoded. The new vertex may indicate a predicted position of the current vertex. The new vertex may form a parallelogram with three vertices that are included in the adjacent triangles. For example, as shown in FIG. 4A, the mesh (400A) may include multiple triangles, such as triangle (412). Each of the triangles may include three vertices. For example, triangle (412) may include vertices (406), (408), and (410). To encode the current vertex (402), a predicted vertex (or prediction) (404) of the current vertex (402) may be generated based on the predicted triangle (414) according to the parallelogram prediction. The predicted vertex (404) may be included in the predicted triangle (414). The predicted triangle (414) may share an edge (416) with triangle (412). The vertices (406), (408), (410), and the predicted vertex (404) may form a parallelogram. Further, note that the current vertex (402) is contained within triangle (418) of the mesh (400A), and triangle (418) and triangle (412) share an edge (416).

[0068] A multi-parallelogram prediction can be applied to predict the current vertex. The multi-parallelogram prediction can, for example, use the average position given by two or more parallelogram predictions whenever they are available. Figure 4B provides an example of two parallelogram predictions. As shown in Figure 4B, the current vertex (420) can be included in the first triangle (434) of the mesh (400B). The first triangle (434) may include vertices (420), (438), and (442). The first triangle (434) may share an edge (444) with an adjacent triangle (428) of the mesh (400B). The current vertex (420) may be included in a second triangle (436) that includes vertices (420), (438), and (440). The second triangle (436) may share an edge (446) with an adjacent triangle (426) of the mesh (400B). Based on the shared edge (444), the first predicted vertex A point (or first prediction) (424) may be determined. The first predicted vertex (424) may form a first parallelogram with an adjacent triangle (428). A second predicted vertex (or second prediction) (422) may be determined based on the shared edge (446). The second predicted vertex (422) may form a second parallelogram with an adjacent triangle (426). Thus, the current vertex (420) may be further predicted by an average of the first predicted vertex (424) and the second predicted vertex (422).

[0069] In the present disclosure, a method and / or system for texture coordinate encoding in mesh compression is provided. In one embodiment, a predicted position value (or prediction or predicted vertex) of a current vertex can be selected from multiple candidate position values ​​(or candidate predictions or candidate predicted vertices) based on a parallelogram prediction. The selected predicted vertex can be coded (or identified) by an index. A prediction residual between the selected predicted vertex and the current vertex can be further coded.

[0070] In the present disclosure, many methods and / or systems can be proposed for vertex position prediction in mesh compression. It is noted that the methods and / or systems can be applied individually or in any form of combination. It is also noted that the method can be applied to static meshes as well as dynamic meshes. Static meshes may only include one frame or the mesh content of static meshes may not change over time. Furthermore, the disclosed method and / or system is not limited to vertex position prediction. The disclosed method and / or the disclosed method can also be applied to, for example, 3D coordinate encoding or more general multi-prediction based schemes.

[0071] The triangles in a triangular mesh (or meshes) may be ordered. The order of the triangles in the mesh may be traversed according to an edge breaker algorithm or other partition algorithm. Additionally, in one example, the vertices in the mesh may be ordered based on the order of the triangles. In one example, the order or traversal order of the triangles and vertices may be determined based on the same scheme or different schemes.

[0072] 5A and 5B show an example ordering of triangles and vertices in a mesh based on the edge breaker algorithm. FIG. 5A shows five examples of patch configurations for the edge breaker algorithm. As shown in FIG. 5A, V is the patch center vertex, and T is the current triangle. The active gate (or current triangle) in each patch can be represented as T. In patch C, a complete triangle can be provided that fans out (or rotates) around V. In patch L, one or more missing triangles can be located to the left of the active gate T. In patch R, one or more missing triangles can be located to the right of the active gate T. In patch E, V is adjacent only to T. In patch S, one or more missing triangles can be located in a position other than the left or right of the active gate T. FIG. 5B shows an example traversal of a mesh (500), whose triangles can be ordered based on the traversal of the edge breaker algorithm. As shown in FIG. 5B, the triangles in the mesh (500) can be traversed along a spiral triangle spanning tree. For example, the traversal may begin at a triangle (502) of type C (or patch C). The traversal may then proceed along the branch adjacent to the right edge of the triangle (e.g., (502)). The traversal may stop when a triangle of type E (e.g., (504)) is reached. According to the edge breaker algorithm, the triangles of the mesh (500) may be traversed (or ordered) in the sequence CRSRLECRRRLE, which may be shown in FIG. 5B. The vertices within each triangle of the mesh (500) may also be ordered based on the triangle's order.

[0073] In a triangular mesh, each triangle, also called a face, has three vertices. For two triangles that share an edge, parallelogram prediction can be applied to use one of the two opposite vertices of the two triangles as a predictor of another triangle. As shown in FIG. 6, two triangles ABC and DBC can share an edge BC. The coordinates of vertices A, B, and C already coded can be applied to predict the coordinates of D. For example, a predicted vertex (or prediction) D' can be determined based on parallelogram prediction. The coordinates of the predicted vertex D' can be determined in equation (1) as follows: D'=B+CA formula (1) In this way, the four vertices (D', B, A, C) can form a parallelogram as shown in Figure 6. Since each vertex has a 2D texture coordinate, each coordinate component of the predicted vertex D' can be calculated using equation (1). For example, assuming that the subscripts x, y, and z indicate the 2D texture coordinates in uv space, the coordinate components of the predicted vertex D' in uv space can be provided by equations (2)-(3). D u '=B u +C u -A u Formula (2) D v '=B v +C v -A v Formula (3)

[0074] Triangle ABC may be denoted as a prediction candidate (or predicted triangle) for vertex D if the position values ​​of vertices A, B, and C are encoded and may be used for prediction. Depending on the number of existing encoded vertices and shared edges, a vertex may have zero, one, two, or more prediction candidates. If there is more than one prediction candidate for the current vertex, the prediction candidates may be ordered based on the order of the triangles in the mesh. For example, the triangles may be ordered based on an edge breaker algorithm, etc.

[0075] In the present disclosure, one or more predictions (or parallelogram predictions) of the current vertex may be determined. An average of the one or more parallelogram predictions may further be determined. Further, each parallelogram prediction may be compared to a true value of the vertex position of the current vertex. Further, the average of the one or more parallelogram predictions may be compared to the true position value. A prediction of the current vertex may be determined from the one or more parallelogram predictions and the average of the one or more parallelogram predictions. In one example, a prediction of the current vertex may be determined based on a minimum prediction error of the comparison. An index of the prediction with the minimum prediction error may be stored. The index of the prediction may be referred to as a prediction index. The prediction index may indicate one of the parallelogram predictions or the average of the parallelogram predictions. Both the prediction index and the associated prediction error, which may also be referred to as a prediction residual, may further be encoded. The predictor error or prediction residual may indicate a difference between the current vertex and the predicted vertex.

[0076] In the present disclosure, all triangles in a triangle mesh may be ordered. For example, the triangles may be ordered based on an edge breaker algorithm. Additionally, all or a subset of the vertices in the triangle mesh may be ordered. In one example, the vertices may be ordered based on the order of the triangles.

[0077] Consider a vertex V in a triangular mesh, and V with N prediction candidates (or prediction triangles). The N prediction candidates are defined as N predictions V1', V2', ..., V N For example, as shown in FIG. 4B, a current vertex (420) may have two predicted triangles (430) and (432). Each of the predicted triangles (430) and (432) may generate a separate predicted value (or predicted vertex), such as (422) and (424), respectively. The predicted values ​​for the current vertex may be ordered based on the order of the corresponding predicted triangles in the mesh. If duplicates are determined in the predicted values, such duplicates may be removed from the list of predicted values.

[0078] In one example, when N=0, there are no prediction candidates available for vertex V. If V is the first vertex to be coded in the mesh, the predicted value V' of the first vertex can be set as 0, and the prediction residual R can be set as V. Thus, V'=0, R=V. If V is not the first vertex to be coded, the predicted value V' of V can be set as the coded value of a previously coded vertex in vertex order. Thus, the previously coded vertex can be the predictor of vertex V. The prediction residual R can be determined as the difference between vertex V and the predicted value (or predicted vertex) V', where R=V-V'.

[0079] In one example, when N=1, only one prediction candidate is available for vertex V. Thus, a prediction value V' can be determined based on only one prediction candidate, and a prediction residual R can be determined as the difference between V and V', where R=V-V'.

[0080] In one example, when N>=2, multiple prediction candidates are available for a vertex V. Therefore, multiple predicted values ​​(or predicted vertices) can be determined based on the multiple predicted candidates. In one embodiment, the average of the predicted values ​​can be considered as an additional predictor V0'. V0' can be determined by equation (4). V0'=(V1'+V2'+...+V N ') / N formula (4)

[0081] The average predicted value V0' is the predicted value V1', V2', ..., V N ', then either the average forecast V0' or the duplicated forecast can be deleted.

[0082] On the encoder side, V i ' can be compared to V, where 0<=i<=N. A prediction index j can be selected. The selected prediction index j is the sum of V and each prediction value V i V corresponding to the minimum prediction error between iIn some embodiments, the minimization error (or minimum prediction residual) is 0 Norm, L 1 Norm, L 2 It can be measured by the L norm, or some other norm. For example, 0 The norm can be determined in equation (5) as follows:

number

[0083] At the decoder side, the index of the selection (e.g., prediction index) can be decoded, and the selected predictor (or prediction value) can be restored from the list of predictors. For example, the decoder can determine one or more prediction values ​​of the current vertex based on a prediction mode such as parallelogram prediction. Furthermore, the average of the one or more prediction values ​​can be considered as a predictor. The average prediction value and the one or more prediction values ​​can be ordered to form a prediction list. In one example, the prediction list generated at the decoder side can be the same as the prediction list generated at the encoder side. According to the decoded prediction index, the selected predictor can be restored based on the prediction index from the prediction list.

[0084] In one embodiment, at the encoder side, if the number of prediction candidates N is equal to a specified (or default) value, a fixed prediction index may be selected (or default). In one example, N may be 2 or more and 5 or less. Thus, the prediction index associated with the selected predictor in the prediction list may not need to be coded. At the decoder side, the decoder may restore (or identify) the prediction index (or fixed prediction index). For example, if N>2, the decoder may select (or identify) prediction index 0. Thus, if N>2, the prediction index does not need to be coded.

[0085] In one embodiment, at the encoder side, a fixed prediction index may be selected (or defaulted) if N is equal to or greater than a constant, such as 2. Thus, not all prediction indexes need to be coded. At the decoder side, the decoder may recover (or identify) a prediction index (or a fixed prediction index). In one example, if N=2, a first prediction index (e.g., prediction index 1) may be selected. In one example, if N>2, a second prediction index (e.g., prediction index 0) may be selected. In one example, if N>=2, a third prediction index (e.g., prediction index 0 or prediction index 3) may be selected.

[0086] In one embodiment, an upper limit M can be set. The upper limit M can indicate the number of prediction candidates that can be considered (or applied) in the prediction list. If N>M, then for each vertex in the mesh, only the first M prediction candidates are considered. This results in the first M prediction candidates V1', V2', ..., V M ' is applied. In one embodiment, the average of the first M predicted values ​​may be determined by equation (6). V0'=(V1'+V2'+...+V M ') / M formula (6) M may be an integer, such as 4. Thus, up to M (e.g., 4) prediction candidates are possible. Note that the average prediction V0' may be placed in various positions in the candidate list. In one example, the average prediction V0' may be the first predictor in the prediction candidate list. In one example, the average prediction V0' may be the last predictor in the prediction candidate list. In one example, the average prediction V0' may be the first predictor in the prediction candidate list. In one example, the average prediction V0' may be the first predictor in the prediction candidate list. In one example, the average prediction V0' may be the first predictor in the prediction candidate list. M ' can be placed in

[0087] In one embodiment, 3D coordinate-based prediction can be applied to predict texture coordinates if the 3D coordinates are encoded and available for prediction. For example, the 3D coordinate-based prediction for vertex D in FIG. 6 is D x ', D y ', and D z '. Various prediction methods, such as inter-frame prediction or intra-frame prediction, can be applied to operate the 3D coordinate-based prediction. In one example, the intra-frame prediction can be a parallelogram prediction. The 3D coordinate-based prediction can be compared with a parallelogram-based prediction (e.g., a 2D coordinate-based prediction), and the one with the smaller coding cost can be selected. A binary flag can be further coded. The binary flag can indicate whether the 3D coordinate-based prediction or the parallelogram-based prediction (e.g., a 2D coordinate-based prediction) is selected.

[0088] In one embodiment, the average of the 3D coordinate-based prediction and the parallelogram-based prediction may be applied as a predictor for vertex V.

[0089] If a vertex V includes N prediction candidates V1', V2', ..., VN' and an average V0', then a selected prediction index can be coded. The selected prediction index indicates the selected predictor in the prediction list. The prediction list includes N prediction candidates V1', V2', ..., V N ', and 'average V0'.

[0090] In some embodiments, the prediction index needs to be encoded only if there are two or more prediction candidates. For example, if two or more prediction candidates are available for the current vertex, an index indicating the selected prediction candidate can be encoded at the encoder side. At the decoder side, the decoder can determine the two or more prediction candidates in the same order as the encoder. The decoder can decode the encoded prediction index and reconstruct the selected prediction candidate based on the prediction index from the two or more prediction candidates.

[0091] In one embodiment, if no prediction candidate is determined or only one prediction candidate is determined, the prediction index may not need to be coded. Thus, the predicted value may be the current vertex itself or may be the only prediction candidate. In one embodiment, the prediction index may always be coded regardless of the number of available predictor candidates. If no predictor is available, the signaled index value may not affect the decoding process on the decoder side.

[0092] In some embodiments, when a fixed prediction index is selected (or predefined) in response to N being a specified (or predefined) value, such as 2≦N≦5, the prediction index may not be encoded.

[0093] In some embodiments, the prediction index is not fixed but is selected from the prediction candidates. If more than one prediction candidate is determined for the current vertex, the prediction index can be encoded using a fixed-length coding. For example, if three prediction candidates are determined for the current vertex, four possible prediction indices may be required: 0, 1, 2, and 3 (0 indicates the average value of candidates 1, 2, and 3). Thus, a two-digit binary number can be applied to represent each of the four prediction indices. It should be noted that different vertices of a triangular mesh can use different fixed lengths. For example, if another vertex has seven prediction candidates, the other vertex can use a three-digit binary number for the prediction index representation. The output from the fixed-length coding can be further compressed by entropy coding, such as arithmetic coding.

[0094] Alternatively, the prediction index can be coded using variable length coding. For example, if four prediction candidates are determined for the current vertex, five possible prediction indexes, 0, 1, 2, 3, and 4, may be required. Variable length codes of 0, 100, 101, 110, and 111 can be assigned to represent the five prediction indexes 0, 1, 2, 3, and 4, respectively. Alternatively, variable length codes of 1, 01, 001, 0001, and 00001 can be applied to represent the five prediction indexes 0, 1, 2, 3, and 4, respectively. Note that different vertices of a triangular mesh can use different variable lengths. The output from the variable length coding can be further compressed by entropy coding, such as arithmetic coding.

[0095] Alternatively, the prediction index can be differentially coded, where the prediction index of the current vertex can be predicted from a previously coded prediction index, and the prediction error associated with the prediction index can also be coded. For example, the difference between a first prediction index of the predictor and a second prediction index of the predictor can be coded first. The second prediction index can be further coded based on the difference between the first prediction index and the second prediction index.

[0096] If an upper limit value M is set to limit the maximum number of prediction candidates and more than M prediction candidates are available for a vertex V, then only the first M prediction candidates can be used to predict the vertex V. Thus, (M+1) possible prediction indices can be coded, where M indicates the first M prediction candidates and 1 indicates the average of the first M prediction candidates. If duplicates are determined in the predicted values, the duplicates can be removed. This can also reduce the likelihood of the prediction index. If a fixed prediction index is selected (or defaulted), the prediction index may not be coded. If the prediction index is not fixed and is selected from among multiple prediction candidates, the prediction index can be coded. The prediction index can be coded by fixed-length coding, variable-length coding, differential coding, etc.

[0097] If 3D coordinate-based prediction is also a prediction candidate, a 1-bit binary flag can be used to signal the prediction mode, which can indicate whether the prediction mode is 3D coordinate-based prediction or parallelogram-based prediction.

[0098] If the average of 3D coordinate-based prediction and parallelogram-based prediction is also a prediction candidate, a 3-symbol flag can be used to signal the prediction mode. The 3-symbol flag can indicate whether the prediction mode is 3D coordinate-based prediction, parallelogram-based prediction, or the average of 3D coordinate-based prediction and parallelogram-based prediction.

[0099] In one embodiment, a prediction residual R of a position value (or predicted value) can be coded (or determined). The prediction residual R can indicate a difference between a predicted value of a current vertex and the current vertex. The prediction residual R can be coded by fixed-length coding, exponential-Golomb coding, arithmetic coding, etc. Thus, the decoder can reconstruct the current vertex based on the predicted value and the prediction residual, and the predicted value can be determined based on a prediction index decoded from a prediction list.

[0100] 7 shows a flow chart outlining a process (700) according to one embodiment of the disclosure. The process (700) may be used in an encoder, such as a video encoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit performing the functions of a video encoder (103), a processing circuit performing the functions of a video encoder (303), etc. In some embodiments, the process (700) is implemented in software instructions, such that the processing circuit performs the process (700) when the processing circuit executes the software instructions. The process begins at (S701) and proceeds to (S710).

[0101] In step (S710), a plurality of candidate predictions of a current vertex is determined. The current vertex is one of a plurality of vertices of a mesh. Each of the plurality of candidate predictions is determined based on each of a plurality of triangles in the mesh and includes a two-dimensional (2D) coordinate in a 2D region.

[0102] In step S720, an average prediction of the multiple candidate predictions is calculated.

[0103] In step S730, a 2D prediction for the current vertex is selected from a prediction list including multiple candidate predictions and an average prediction, the selected 2D prediction corresponding to a threshold prediction difference between the current vertex and each predictor in the prediction list.

[0104] In step S740, a three-dimensional (3D) prediction of the current vertex is determined based on one or more neighboring vertices of the current vertex in the mesh, the 3D prediction including 3D coordinates within the 3D domain.

[0105] In step (S750), a prediction residual including a prediction error of the current vertex is encoded. The prediction residual is obtained based on the selected 2D prediction and the determined 3D prediction of the current vertex.

[0106] In one example, to determine multiple candidate predictions for the current vertex, a first triangle of the multiple triangles is determined for the current vertex, the first triangle shares an edge with a second triangle of the multiple triangles, and the current vertex is included in the second triangle and is on the opposite side of the shared edge. The first prediction of the multiple candidate predictions is determined based on the determined first triangle via a parallelogram prediction, where the first prediction and the first triangle form a first parallelogram.

[0107] In some embodiments, to select a 2D prediction for the current vertex, a prediction list for the current vertex is determined. The predictors in the prediction list include an average prediction and a plurality of candidate predictions following the average prediction in the prediction list. The order of the plurality of candidate predictions in the prediction list is based on the order of the plurality of triangles corresponding to the plurality of candidate predictions. A prediction index for each predictor in the prediction list is further determined.

[0108] In some embodiments, the triangles are ordered based on an edge breaker algorithm in which the triangles are labeled in a spiral triangle spanning tree order.

[0109] In some embodiments, to select a 2D prediction for the current vertex, a prediction difference between each of the predictors in the prediction list and the current vertex is determined. The 2D prediction is selected from the predictors in the prediction list such that the selected 2D prediction corresponds to the smallest prediction difference of the prediction differences.

[0110] In some embodiments, a 2D prediction is selected within the subset of the prediction list such that the 2D prediction corresponds to the smallest prediction difference within the subset of the prediction list.

[0111] In one example, the 3D prediction of the current vertex is determined based on a delta prediction, where the 3D prediction is determined as a neighboring vertex of one of the one or more neighboring vertices. In one example, the 3D prediction of the current vertex is determined based on a parallelogram prediction, where the 3D prediction and three neighboring vertices of the one or more neighboring vertices form a parallelogram.

[0112] In some embodiments, to encode the prediction residual, a predicted value of the current vertex is determined based on one of the 2D prediction and the 3D prediction, the one of the 2D prediction and the 3D prediction having a smaller prediction difference than another one of the 2D prediction and the 3D prediction, and a prediction residual indicating a difference between the predicted value and the current vertex is further encoded.

[0113] In some embodiments, to encode the prediction residual, a prediction value of the current vertex is determined based on an average of the 2D prediction and the 3D prediction, and a prediction residual indicating the difference between the prediction value and the current vertex is further encoded.

[0114] In one example, index information indicating a prediction index of a selected 2D prediction in a prediction list is encoded. In one example, a flag is further encoded, the flag indicating that a prediction value of the current vertex is determined based on one of a 2D prediction, a 3D prediction, or an average of a 2D prediction and a 3D prediction.

[0115] The process then proceeds to (S799) and ends.

[0116] The process (700) may be adapted as appropriate. Steps of the process (700) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0117] 8 shows a flow chart outlining a process (800) according to one embodiment of the disclosure. The process (800) may be used in a decoder, such as a video decoder. In various embodiments, the process (800) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), or the like. In some embodiments, the process (800) is implemented in software instructions, such that the processing circuit performs the process (800) when the processing circuit executes the software instructions. The process begins at (S801) and proceeds to (S810).

[0118] In step (S810), coding information of a mesh including a plurality of triangles and a plurality of vertices is received. The coding information includes index information of a current vertex of the plurality of vertices. The index information indicates a two-dimensional (2D) prediction of the current vertex. The 2D prediction of the current vertex includes a 2D coordinate within the 2D domain.

[0119] In step (S820), a plurality of candidate predictions for the current vertex are determined based on the plurality of triangles, each of the plurality of candidate predictions corresponding to a respective one of the plurality of triangles.

[0120] In step S830, a 2D prediction of the current vertex is selected from a plurality of candidate predictions based on the index information, the selected 2D prediction corresponding to a threshold prediction difference between the current vertex and each of the plurality of candidate predictions.

[0121] In step (S840), a three-dimensional (3D) prediction of the current vertex is determined based on one or more neighboring vertices of the current vertex in the mesh, the 3D prediction including 3D coordinates within the 3D domain.

[0122] In step (S850), the current vertex is reconstructed based on the selected 2D prediction and the determined 3D prediction of the current vertex.

[0123] Thereafter, the process proceeds to (S899) and ends.

[0124] The process (800) may be adapted as appropriate. Steps in the process (800) may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0125] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media, such as non-transitory computer readable storage media. For example, Figure 9 illustrates a computer system (900) suitable for implementing certain embodiments of the disclosed subject matter.

[0126] Computer software can be encoded using any suitable machine code or computer language that can be subject to mechanisms such as assembly, compilation, linking, etc. to create code including instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., processing circuitry.

[0127] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, IoT devices, etc.

[0128] 9 for the computer system (900) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (900).

[0129] The computer system (900) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0130] The input human interface devices may include one or more (only one of each) of a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data glove (not shown), a joystick (905), a microphone (906), a scanner (907), and a camera (908).

[0131] The computer system (900) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (910), data gloves (not shown), or joystick (905), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (909), headphones (not shown), etc.), visual output devices (e.g., screens (910) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability and each with or without haptic feedback capability - some of which are capable of outputting two-dimensional visual output, or output in more than three dimensions via means such as stereo output, virtual reality glasses (not shown), holographic displays, smoke tanks (not shown), and printers (not shown).

[0132] The computer system (900) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (920) along with media such as CDs / DVDs (921), thumb drives (922), removable hard drives or solid state drives (923), legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0133] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0134] The computer system (900) may also include an interface (954) to one or more communication networks (955). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial television, vehicular and industrial including CANBus, etc. Certain networks require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port of the computer system (900)) or peripheral bus (949), while others are typically integrated into the core of the computer system (900) by being attached to a system bus as described below (e.g., an Ethernet interface is integrated into a PC computer system, or a cellular network interface is integrated into a smartphone computer system). Using any of these networks, the computer system (900) can communicate with other entities. Such communications may be one-way, receive-only (e.g., TV broadcasts), transmit-only one-way (e.g., CANbus to a particular CANbus device), or bidirectional, for example to other computer systems using local or wide-area digital networks. As explained above, each of these networks and network interfaces may use specific protocols and protocol stacks.

[0135] The aforementioned human interface devices, human access storage devices, and network interfaces may be attached to the core (940) of the computer system (900).

[0136] The cores (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units (943) in the form of field programmable gate areas (FPGAs), hardware accelerators for specific tasks (944), graphics adapters (950), and the like. These devices may be connected through a system bus (948) along with read only memory (ROM) (945), random access memory (946), internal mass storage (947) such as a non-user accessible internal hard drive, SSD, and the like. In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus (948) or through a peripheral bus (949). In one example, a screen (910) may be connected to the graphics adapter (950). Architectures for peripheral buses include PCI, USB, and the like.

[0137] The CPU (941), GPU (942), FPGA (943), and accelerator (944) may execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in a ROM (945) or a RAM (946). Temporary data may also be stored in the RAM (946), while permanent data may be stored, for example, in an internal mass storage (947). Rapid storage and retrieval from any of the memory devices may be made possible through the use of a cache memory that may be closely associated with one or more of the CPU (941), GPU (942), mass storage (947), ROM (945), RAM (946), etc.

[0138] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0139] By way of example and not limitation, a computer system having the architecture (900), and in particular the cores (940), can provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as previously described, as well as media associated with specific storage of the cores (940) of a non-transitory nature, such as the core internal mass storage (947) or ROM (945). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the cores (940). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the cores (940), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes, or certain portions of certain processes, described herein, including defining data structures stored in RAM (946) and modifying such data structures in accordance with the software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic embodied in hardwired or otherwise circuitry (e.g., accelerator (944)), which may operate in place of or together with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0140] The use of "at least one of" or "one of" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; at least one of A through C is intended to include A only, B only, C only, or any combination thereof. Reference to one of A or B and one of A and B is intended to include A or B, or (A and B). The use of "one of" does not exclude any combination of the listed elements, where applicable, such as when the elements are not mutually exclusive.

[0141] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0142] 100 video processing system, 101 video source, 102 stream, 103 video encoder, 104 video data, 105 streaming server, 106 client subsystem, 107 video data, input copy of video data, 108 client subsystem, 109 video data, 110 video decoder, 111 video picture, 112 display, 113 capture subsystem, 120 electronic device, 130 electronic device, 201 channel, 210 video decoder, 212 rendering device, 215 buffer memory, 220 parser, 221 symbol, 230 electronic device, 231 receiver, 251 scaler / inverse transform unit, 252 intra picture prediction unit, intra prediction unit, 253 motion compensation prediction unit, 255 aggregator, 256 loop filter unit, 257 reference picture memory, 258 picture buffer, 301 video source, 303 video encoder, 320 Electronic device, 330, source coder, 332, coding engine, 333, local video decoder, local decoder, 334, reference picture memory, 335, predictor, 340, transmitter, 343, video sequence, 345, entropy coder, 350, controller, 360, communication channel, 400A, mesh, 400B, mesh, 402, vertices, 404, predicted vertices, 406, vertices, 408, vertices, 410, vertices, 412, triangles, 414, predicted triangles, 416, edges, 418, triangles, 420, vertices, 422, predicted vertices, 424, predicted vertices, 426, triangles, 428, triangles, 430, predicted triangles, 432, predicted triangles, 434, triangles, 436, triangles, 438, vertices, 440, vertices, 442, vertices, 444, shared edges, 446, shared edges, 500 Mesh, 502 Triangle, 504 Type E triangle, 700 Process, 800 Process, 900 Computer system, architecture, 901 Keyboard, 902 Mouse, 903 Trackpad, 905 Joystick, 906 Microphone, 907 Scanner, 908 Camera, 909 Speaker, 910 Touch screen, 921 Media, 922 Thumb drive, 923 Solid state drive, 940 Core, 941central processing unit, CPU, 942 graphics processing unit, GPU, 943 dedicated programmable processing unit, FPGA, 944 hardware accelerator, 945 read only memory, ROM, 946 random access memory, 947 core internal mass storage, 948 system bus, 949 peripheral bus, 950 graphics adapter, 954 interface, 955 communication network

Claims

1. 1. A method of mesh processing performed in a video encoder, comprising: determining a plurality of candidate predictions for a current vertex, the current vertex being one of a plurality of vertices of a mesh, each of the plurality of candidate predictions being determined based on each of a plurality of triangles in the mesh and including a two-dimensional (2D) coordinate within a 2D domain; calculating an average prediction of the plurality of candidate predictions; selecting a 2D prediction for the current vertex from a prediction list including the plurality of candidate predictions and the average prediction, the selected 2D prediction corresponding to a threshold prediction difference between the current vertex and each of the predictors in the prediction list; determining a three-dimensional (3D) prediction of the current vertex based on one or more neighboring vertices of the current vertex in the mesh, the 3D prediction including 3D coordinates within a 3D domain; encoding a prediction residual including a prediction error of the current vertex, the prediction residual being obtained based on the selected 2D prediction and the determined 3D prediction of the current vertex; A method comprising:

2. The step of determining the multiple candidate predictions for the current vertex comprises: determining a first triangle of the plurality of triangles for the current vertex, the first triangle sharing an edge with a second triangle of the plurality of triangles, the current vertex being contained in the second triangle and on an opposite side of the shared edge; determining a first prediction of the plurality of candidate predictions based on the determined first triangle via a parallelogram prediction, where the first prediction and the first triangle form a first parallelogram; The method of claim 1 further comprising:

3. The step of selecting a 2D prediction of the current vertex comprises: determining a prediction list for the current vertex, where predictors in a prediction list include the average prediction and the plurality of candidate predictions following the average prediction in the prediction list, and an order of the plurality of candidate predictions in the prediction list is based on an order of the plurality of triangles corresponding to the plurality of candidate predictions; determining a prediction index for each of the predictors in the prediction list; The method of claim 2 , further comprising:

4. The method of claim 3 , wherein the plurality of triangles is ordered based on an edge breaker algorithm in which the plurality of triangles are labeled in a spiral triangle spanning tree order.

5. The step of selecting the 2D prediction of the current vertex comprises: determining a prediction difference between each of the predictors in the prediction list and the current vertex; selecting the 2D prediction from the predictors in the prediction list such that the selected 2D prediction corresponds to a minimum prediction difference of the prediction differences; The method of claim 3 further comprising:

6. The step of selecting the 2D prediction of the current vertex comprises: selecting the 2D prediction in the subset of prediction lists such that the 2D prediction corresponds to a minimum prediction difference in the subset of prediction lists. The method of claim 5 further comprising:

7. The step of determining the 3D prediction of the current vertex comprises: determining a 3D prediction of the current vertex based on a delta prediction, the 3D prediction being determined as a neighboring vertex of one or more neighboring vertices; and determining the 3D prediction of the current vertex based on a parallelogram prediction, where the 3D prediction and three neighboring vertices of the one or more neighboring vertices form a parallelogram; The method of claim 5 , further comprising any one of the following:

8. The step of encoding the prediction residual comprises: determining a predicted value of the current vertex based on one of the 2D prediction and the 3D prediction, the one of the 2D prediction and the 3D prediction having a smaller prediction difference than another one of the 2D prediction and the 3D prediction; encoding the prediction residual indicative of a difference between the prediction and the current vertex; The method of claim 1 further comprising:

9. The step of encoding the prediction residual comprises: determining a predicted value for the current vertex based on an average of the 2D prediction and the 3D prediction; encoding the prediction residual indicative of a difference between the prediction and the current vertex; The method of claim 1 further comprising:

10. encoding index information indicating the prediction index of the selected 2D prediction in the prediction list; encoding a flag indicating that the predicted value of the current vertex was determined based on one of the 2D prediction, the 3D prediction, or an average of the 2D prediction and the 3D prediction; The method of claim 3 further comprising:

11. 1. An apparatus for mesh processing comprising: determining a plurality of candidate predictions for a current vertex, the current vertex being one of a plurality of vertices of a mesh, each of the plurality of candidate predictions being determined based on each of a plurality of triangles in the mesh and including a two-dimensional (2D) coordinate within a 2D domain; calculating an average prediction of said plurality of candidate predictions; selecting a 2D prediction for the current vertex from a prediction list including the plurality of candidate predictions and the average prediction, the selected 2D prediction corresponding to a threshold prediction difference between the current vertex and each of the predictors in the prediction list; determining a three-dimensional (3D) prediction of the current vertex based on one or more neighboring vertices of the current vertex in the mesh, the 3D prediction including 3D coordinates within a 3D domain; encoding a prediction residual including a prediction error of the current vertex, the prediction residual being obtained based on the selected 2D prediction and the determined 3D prediction of the current vertex; A processing circuit configured to 13. An apparatus comprising:

12. The processing circuitry comprises: determining a first triangle of the plurality of triangles for the current vertex, the first triangle sharing an edge with a second triangle of the plurality of triangles, the current vertex being contained in the second triangle and on an opposite side of the shared edge; determining a first prediction of the plurality of candidate predictions based on the determined first triangle via a parallelogram prediction, where the first prediction and the first triangle form a first parallelogram. The apparatus of claim 11 configured to:

13. The processing circuitry comprises: determining the prediction list for the current vertex, the predictors of the prediction list including the average prediction and the candidate predictions following the average prediction in the prediction list, an order of the candidate predictions in the prediction list based on an order of the triangles corresponding to the candidate predictions; determining a prediction index for each of the predictors in the prediction list; The apparatus of claim 12 configured to:

14. The apparatus of claim 13 , wherein the plurality of triangles is ordered based on an edge breaker algorithm in which the plurality of triangles are labeled in a spiral triangle spanning tree order.

15. the processing circuitry determines a prediction difference between each of the predictors in the prediction list and the current vertex; selecting the 2D prediction from the predictors in the prediction list such that the selected 2D prediction corresponds to a minimum prediction difference of the prediction differences; The apparatus of claim 13 configured to:

16. The processing circuitry comprises: selecting the 2D prediction in the subset of prediction lists such that the 2D prediction corresponds to a minimum prediction difference in the subset of prediction lists; The apparatus of claim 15 configured to:

17. The processing circuitry comprises: determining a 3D prediction of the current vertex based on a delta prediction, the 3D prediction being determined as a neighboring vertex of the one or more neighboring vertices; determining the 3D prediction of the current vertex based on a parallelogram prediction, where the 3D prediction and three neighboring vertices of the one or more neighboring vertices form a parallelogram; The apparatus of claim 15 , configured to:

18. The processing circuitry comprises: determining a predicted value of the current vertex based on one of the 2D prediction and the 3D prediction, the one of the 2D prediction and the 3D prediction having a smaller prediction difference than another one of the 2D prediction and the 3D prediction; encoding the prediction residual indicative of a difference between the predicted value and the current vertex; The apparatus of claim 11 configured to:

19. The processing circuitry comprises: determining a predicted value for the current vertex based on an average of the 2D prediction and the 3D prediction; encoding the prediction residual indicative of a difference between the predicted value and the current vertex; The apparatus of claim 11 configured to:

20. The processing circuitry comprises: encoding index information indicating the prediction index of the selected 2D prediction in the prediction list; encoding a flag indicating that the predicted value of the current vertex is determined based on one of the 2D prediction, the 3D prediction, or an average of the 2D prediction and the 3D prediction; The apparatus of claim 13 configured to:

Images