Method, apparatus and computer program for vertex position prediction in mesh compression

Abstract

In a method of mesh processing, the mesh includes a current vertex within a plurality of vertices. A plurality of candidate predictions of the current vertex are determined based on a plurality of triangles in the mesh, each of the plurality of triangles corresponding to a respective candidate prediction. A prediction of the current vertex is determined from the plurality of candidate predictions. The prediction corresponds to a threshold prediction difference between the current vertex and each of the plurality of candidate predictions. Prediction information for the current vertex is further generated based on the determined predictions.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims benefit of priority to U.S. Provisional Application No. 63 / 344,394, entitled "Vertex Position Prediction in Mesh Compression," filed May 20, 2022, which claims benefit of priority to U.S. Provisional Application No. 18 / 126,152, entitled "VERTEX POSITION PREDICTION IN MESH COMPRESSION," filed March 24, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] The present disclosure includes embodiments relating to mesh processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context for the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or implicitly as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) capture, modeling, and rendering are facilitating the ubiquitous presence of 3D content across various platforms and devices. Today, it is possible to capture a baby's first steps on one continent while the baby's grandparents on another continent watch (and potentially interact with) the child, enjoying an immersive experience. To achieve this immersion, models are becoming ever more sophisticated, and a significant amount of data is tied to the generation and consumption of these models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide methods and apparatus for mesh processing. In some examples, an apparatus for mesh processing includes a processing circuit.

[0006] According to one aspect of the present disclosure, there is provided a method for mesh processing performed in a video encoder, wherein the mesh includes a plurality of vertices. A plurality of candidate predictions for a current vertex at the plurality of vertices are determined based on a plurality of triangles in the mesh, each of the plurality of triangles corresponding to a respective candidate prediction. A prediction for the current vertex is determined from the plurality of candidate predictions. The prediction corresponds to a threshold prediction difference between the current vertex and each of the plurality of candidate predictions. Prediction information for the current vertex is further generated based on the determined predictions.

[0007] In some embodiments, to determine multiple candidate predictions for the current vertex, a first triangle of the multiple triangles is determined for the current vertex. The first triangle shares a side with a second triangle of the multiple triangles, and the current vertex is included in the second triangle and is on the opposite side of the shared side. The first prediction of the multiple candidate predictions is determined based on the first triangle determined by parallelogram prediction, where the first prediction and the first triangle form a first parallelogram.

[0008] In some embodiments, to determine multiple candidate predictions for the current vertex, an average prediction of the multiple candidate predictions is determined.

[0009] In some embodiments, to determine a prediction for the current vertex, a prediction list is determined for the current vertex. The prediction list includes predictors. The predictors include an average prediction and multiple candidate predictions that follow the average prediction in the prediction list. The ordering of the multiple candidate predictions in the prediction list is based on the ordering of multiple triangles corresponding to the multiple candidate predictions. For each of the predictors in the prediction list, a respective prediction index is determined.

[0010] In one example, the triangles are arranged based on an edge breaker algorithm, where the triangles are labeled in an arrangement of a spiral triangle spanning tree.

[0011] In one example, to determine a prediction for the current vertex, a prediction difference between each of the predictors in the prediction list and the current vertex is determined, and a prediction is further determined from the predictor in the prediction list that corresponds to the smallest prediction difference of the prediction differences.

[0012] In one example, to determine the prediction for the current vertex, the prediction is determined within the subset of predictors in the prediction list that has the smallest prediction difference.

[0013] In some embodiments, a respective prediction index is determined for each of the predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

[0014] In some embodiments, each prediction index is determined based on differential coding. In one example, a difference between a first prediction index of a predictor and a second prediction index of a predictor may be determined. The second prediction index is further coded based on the difference between the first prediction index and the second prediction index.

[0015] In some embodiments, the prediction information for the current vertex includes index information indicating the determined prediction in the prediction list, and residual information indicating the difference between the current vertex and the determined prediction.

[0016] According to another aspect of the present disclosure, there is provided an apparatus including a processing circuit, the processing circuit being configured to perform any of the described mesh processing methods.

[0017] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described mesh processing methods for video decoding.

[0018] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is a schematic diagram of an exemplary block diagram of a communication system (100). [Figure 2] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder. [Figure 4A] FIG. 2 is a schematic diagram of a first exemplary parallelogram prediction, according to some embodiments of the present disclosure. [Figure 4B] FIG. 10 is a schematic diagram of a second exemplary parallelogram prediction, according to some embodiments of the present disclosure. [Figure 5A] Five example patch configurations for the Edge Breaker algorithm are shown. [Figure 5B] 1 illustrates an exemplary mesh traversal with the Edge Breaker algorithm. [Figure 6] 1 shows a schematic diagram of an example of vertex prediction based on parallelogram prediction. [Figure 7] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 8] 10 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 9] FIG. 1 is a schematic diagram of a computer system, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] 1 shows a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application for the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter can be equally applicable to other image and video-enabled applications, including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] The video processing system (100) includes a capture subsystem (113) that can include a video source (101). The video source (101) can include one or more images captured by a camera and / or generated by a computer. For example, a digital camera generates an uncompressed video picture stream (102). In one example, the video picture stream (102) includes samples taken by the digital camera. The video picture stream (102) is shown in bold to emphasize its large amount of data compared to the encoded video data (104) (or coded video bitstream), and can be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in detail below. The encoded video data (104) (or encoded video bitstream), shown with a thin line to emphasize its small amount of data compared to the stream of video pictures (102), can be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example, within the electronic device (130). The video decoder (110) decodes the input copy of the encoded video data (107) and generates an output stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown).In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in conjunction with VVC.

[0022] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may also include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0023] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include receiving circuitry, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) in the example of FIG. 1.

[0024] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences are received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may also receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver (231) may separate the coded video sequences from other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) can be external to the video decoder (210) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (210), for example, to combat network jitter, plus another buffer memory (215) internal to the video decoder (210), for example, to handle playout timing. When the receiver (231) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (215) may not be necessary or can be small.For use over a best-effort packet network such as the Internet, a buffer memory (215) may be required, which may be relatively large, advantageously adaptively sized, and may be implemented at least in part within an operating system or similar element (not shown) external to the video decoder (210).

[0025] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (210) and, potentially, information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230), as shown in FIG. 2. The control information for the rendering device(s) may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not depicted). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding with or without context dependency, Huffman coding, arithmetic coding, etc. The parser (220) may extract a set of subgroup parameters for at least one subgroup of pixels within the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), an image, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0026] The parser (220) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to generate symbols (221).

[0027] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by subgroup control information parsed by the parser (220) from the coded video sequence. The flow of such subgroup control information between the parser (220) and the following units is not depicted for clarity.

[0028] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0029] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients as well as control information from the parser (220) as symbol(s) (221), including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (251) can output blocks containing sample values that can be input to an aggregator (255).

[0030] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258), for example, buffers a partially reconstructed and / or fully reconstructed current picture. The aggregator (255) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0031] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit (253) may access a reference picture memory (257) to retrieve samples to use for prediction. After motion-compensating the fetched samples according to the symbols (221) related to the block, these samples (in this case referred to as residual samples or residual signals) may be added to the output of the scalar / inverse transform unit (251) by an aggregator (255) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (253), for example, in the form of symbols (221) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0032] The output samples of the aggregator (255) can be subjected to various loop filtering techniques in the loop filter unit (256). Video compression techniques can include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression can also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and to previously reconstructed loop-filtered sample values.

[0033] The output of the loop filter unit (256) can be a sample stream that can be output to a rendering device (212) and stored in a reference picture memory (257) for use in future inter-picture prediction.

[0034] Once fully reconstructed, a particular coded picture can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and that coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0035] The video decoder (210) may perform decoding operations according to a given video compression technology or standard, such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile may select certain tools from among all tools available in the video compression technology or standard as the only tools available under that profile. Compliance may also require that the complexity of the coded video sequence be within a range defined by the level of the video compression technology or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further constrained by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0036] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0037] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) may be used in place of the video encoder (103) in the example of FIG. 1.

[0038] The video encoder (303) may receive video samples from a video source (301) (which is not part of the electronic device (320) in the example of FIG. 3) that may capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0039] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. The following discussion focuses on samples.

[0040] According to one embodiment, the video encoder (303) may code and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Achieving an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units, as described below. For clarity, coupling is not shown. Parameters set by the controller (350) may include rate control-related parameters (e.g., picture skip, quantization, lambda values for rate-distortion optimization techniques), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functionality associated with the video encoder (303) optimized for a particular system design.

[0041] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop can include a source coder (330) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to that used by a (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding of the symbol stream produces bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronism (and the resulting drift when synchronism cannot be maintained, for example due to channel errors) is also used in some related techniques.

[0042] The operation of the "local" decoder (333) may be the same as the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in conjunction with Figure 2. However, with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) can be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0043] In one embodiment, decoder technology, excluding parsing / entropy decoding, present in the decoder is present in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. The description of the encoder technology can be omitted, since it is the reverse of the decoder technology described generically. Only in certain areas will more detailed descriptions be provided below.

[0044] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0045] The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (334). In this manner, the video encoder (303) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures obtained by a far-end video decoder (free of transmission errors).

[0046] The predictor (335) may perform the predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or some metadata, such as reference picture motion vectors, block shapes, etc., which may serve as suitable predictive references for the new picture. The predictor (335) may operate on sample blocks, pixel block by pixel block, to find suitable predictive references. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (334).

[0047] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode the video data.

[0048] The output of all of the aforementioned functional units may be entropy coded by an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0049] The transmitter (340) may buffer the coded video sequence(s) created by the entropy coder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0050] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign several coding picture types to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0051] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0052] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values of each block.

[0053] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0054] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0055] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In doing so, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0056] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may comprise temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0057] Video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0058] In some embodiments, bi-prediction techniques may be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but their display orders may be past and future, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0059] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0060] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine the prediction type of the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0061] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) can be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0062] The present disclosure includes embodiments that relate to methods and systems for vertex position prediction in mesh compression.

[0063] A mesh can include several polygons that describe the surface of a volumetric object. Each polygon of a mesh can be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information about how the vertices are connected, sometimes referred to as connectivity information. In some embodiments, vertex attributes such as color, normals, etc. can be associated with mesh vertices. Attributes (or vertex attributes) can also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh with a two-dimensional (2D) attribute map. Such mapping can typically be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps can be used to store high-resolution attribute information such as texture, normals, and displacement. Such information can be used for various purposes, such as texture mapping and shading.

[0064] Dynamic meshes can contain a significant amount of information that changes over time, so they can require large amounts of data. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards, such as IC, MESHGRID, and FAMC, were previously developed by MPEG to address dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards may not consider time-varying attribute maps and connectivity information. Digital content generation (DCC) tools typically generate such dynamic meshes. However, volumetric acquisition techniques can have difficulty generating constant connectivity dynamic meshes, especially under real-time constraints. This type of content (e.g., constant connectivity dynamic meshes) may not be supported by existing standards. MPEG is planning to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and, optionally, time-varying attribute maps. The new mesh compression standard targets lossy and lossless compression for various applications, such as real-time communication, storage, free-viewpoint video, augmented reality (AR), and virtual reality (VR). Features such as random access and scalable / progressive coding can also be considered.

[0065] Mesh geometry information can include vertex connectivity information, 3D coordinates, 2D texture coordinates, etc. Compressing vertex 3D coordinates, sometimes also called vertex positions, can be very important because in many cases, compressing vertex 3D coordinates can consume a significant portion of the overall geometry-related data.

[0066] In this disclosure, for simplicity and clarity, embodiments are described using a triangular mesh as an example. However, other polygon meshes (e.g., quadrilateral or pentagonal meshes) can also be applied to the embodiments in this disclosure. In one example, a triangular mesh is a type of polygon mesh that includes a set of triangles. The set of triangles can be connected by a common edge or corner of the set of triangles. The common edge or corner can be represented as a mesh vertex.

[0067] Vertex position compression can be implemented by parallelogram prediction. In parallelogram prediction, a compression algorithm can introduce a new vertex that is included in a predicted triangle. The predicted triangle can be formed based on edges shared with adjacent triangles. In one example, the adjacent triangle may have been previously coded. The new vertex can indicate a predicted position of the current vertex. The new vertex can form a parallelogram with three vertices included in the adjacent triangles. For example, as shown in FIG. 4A, a mesh (400A) can include multiple triangles, such as triangle (412). Each triangle can include three vertices. For example, triangle (412) can include vertices (406), (408), and (410). To code a current vertex (402), parallelogram prediction can generate a predicted vertex (or prediction) (404) for the current vertex (402) based on a predicted triangle (414). The predicted vertex (404) can be included in the predicted triangle (414). The predicted triangle 414 may share an edge 416 with the triangle 412. The vertices 406, 408, and 410 and the predicted vertex 404 may form a parallelogram. Furthermore, note that the current vertex 402 is contained within a triangle 418 in the mesh 400A, and that triangle 418 shares an edge 416 with triangle 412.

[0068] Multi-parallelogram prediction can be applied to predict the current vertex. Multi-parallelogram prediction can, for example, use the average position given by two or more parallelogram predictions whenever available. Figure 4B provides an example of two parallelogram predictions. As shown in Figure 4B, the current vertex (420) can be included in the first triangle (434) of the mesh (400B). The first triangle (434) may include vertices (420), (438), and (442). The first triangle (434) may share an edge (444) with an adjacent triangle (428) of the mesh (400B). The current vertex (420) may also be included in a second triangle (436) that includes vertices (420), (438), and (440). The second triangle (436) may share an edge (446) with an adjacent triangle (426) of the mesh (400B). Based on the shared edge (444), the first predicted vertex (or A first predicted vertex (424) may be determined. The first predicted vertex (424) may form a first parallelogram together with an adjacent triangle (428). A second predicted vertex (422) may be determined based on the shared edge (446). The second predicted vertex (422) may form a second parallelogram together with an adjacent triangle (426). Thus, the current vertex (420) may be further predicted by the average of the first predicted vertex (424) and the second predicted vertex (422).

[0069] This disclosure provides a method and / or system for vertex position prediction for mesh compression. In one embodiment, a predicted position value (or prediction or predicted vertex) of a current vertex may be selected from multiple candidate position values (or candidate predictions or candidate predicted vertices) based on parallelogram prediction. The selected predicted vertex may be coded (or identified) by an index. A prediction residual between the selected predicted vertex and the current vertex may be further coded.

[0070] In this disclosure, several methods and / or systems may be proposed for vertex position prediction in mesh compression. It should be noted that the methods and / or systems may be applied individually or in any combination. It should also be noted that the methods can be applied to static meshes as well as dynamic meshes. Static meshes may only contain one frame, or the mesh content of static meshes may not change over time. Furthermore, the disclosed methods and / or systems are not limited to vertex position prediction. The disclosed methods and / or systems can also be applied to, for example, 2D texture coordinate prediction or more general multi-prediction based schemes.

[0071] The triangles in a triangular mesh (or mesh) may be ordered. The ordering of the triangles in the mesh may be traversed by an edge breaker algorithm or other division algorithm. Furthermore, in one example, the vertices in the mesh may be ordered based on the ordering of the triangles. In one example, the ordering or traversal ordering of the triangles and vertices may be determined based on the same scheme or different schemes.

[0072] 5A and 5B show exemplary arrangements of triangles and vertices in a mesh based on the edge breaker algorithm. FIG. 5A shows five exemplary patch configurations for the edge breaker algorithm. As shown in FIG. 5A, V is the patch center vertex and T is the current triangle. The active gate (or current triangle) in each patch can be denoted by T. Patch C can provide a complete triangle that orbits (or rotates) around V. Patch L can have one or more missing triangles located to the left of the active gate T. Patch R can have one or more missing triangles located to the right of the active gate T. Patch E can have V adjacent only to T. Patch S can have one or more missing triangles located in a position other than the left or right of the active gate T. FIG. 5B shows an exemplary traversal of a mesh (500), in which triangles in the mesh (500) can be arranged based on the traversal of the edge breaker algorithm. As shown in FIG. 5B, triangles in the mesh (500) can be traversed along a spiral triangle spanning tree. For example, traversal may begin with triangle (502) of type C (or patch C). Traversal then proceeds along the branch adjacent to the right edge of the triangle (e.g., (502)). Traversal may stop when a triangle of type E (e.g., (504)) is reached. According to the edge breaker algorithm, the triangles of mesh (500) may be traversed (or ordered) in the order CRSRLECRRRLE, which can be seen in Figure 5B. The vertices within each triangle of mesh (500) may also be ordered based on the ordering of the triangle.

[0073] In a triangular mesh, each triangle, sometimes called a face, has three vertices. For two triangles that share a side, parallelogram prediction can be applied to use one of the two opposite vertices in the two triangles as a predictor for the other triangle. As shown in FIG. 6, two triangles ABC and DBC can share side BC. The coordinates of vertices A, B, and C have already been coded and can be applied to predict the coordinate of D. For example, a predicted vertex (or prediction) D' can be determined based on parallelogram prediction. The coordinate of the predicted vertex D' can be determined by the following equation (1): D'=B+CA formula (1) Thus, four vertices (D', B, A, C) can form a parallelogram, as shown in Figure 6. Since each vertex has a 3D coordinate, Equation (1) can be used to calculate each coordinate component of the predicted vertex D'. For example, assuming that the subscripts x, y, and z indicate 3D coordinates in xyz space, the coordinate components of the predicted vertex D' in xyz space can be given by Equations (2) through (4). D x '=B x +C x -A x Formula (2) D y '=B y +C y -A y Formula (3) D z '=B z +C z -A z Formula (4)

[0074] If the position values of vertices A, B, and C are coded and available for prediction, triangle ABC can be a predicted candidate (or predicted triangle) for vertex D. A vertex can have zero, one, two, or more predicted candidates, depending on the existing coded vertices and the number of shared edges. If there are two or more predicted candidates for the current vertex, the predicted candidates can be ordered based on the triangle ordering of the mesh. For example, the triangles may be ordered based on an edge breaker algorithm, etc.

[0075] In the present disclosure, one or more predictions (or parallelogram predictions) of the current vertex may be determined. Furthermore, an average of the one or more parallelogram predictions may be determined. Furthermore, each parallelogram prediction may be compared to a true value of the vertex position of the current vertex. Additionally, an average of the one or more parallelogram predictions may be compared to the true position value. A prediction of the current vertex may be determined from one or more parallelogram predictions and an average of the one or more parallelogram predictions. In one example, a prediction of the current vertex may be determined based on the minimum prediction error between the comparisons. An index of the prediction that results in the smallest prediction error may be stored. The index of the prediction may be referred to as a prediction index. The prediction index may indicate one of the parallelogram predictions or the average of the parallelogram predictions. Both the prediction index and the associated prediction error, which may also be referred to as a prediction residual, may be further coded. The predictor error or prediction residual may indicate the difference between the current vertex and the predicted vertex.

[0076] In the present disclosure, all triangles in a triangular mesh can be coded. For example, the triangles can be ordered based on an edge breaker algorithm. In addition, all or a subset of the vertices in the triangular mesh can also be ordered. In one example, the vertices can be ordered based on the ordering of the triangles.

[0077] Suppose there is a vertex V in a triangular mesh, and the vertex V contains N prediction candidates (or prediction triangles). The N prediction candidates are N prediction values V1', V2', ..., V NFor example, as shown in FIG. 4B, a current vertex (420) may have two predicted triangles (430) and (432). Each of the predicted triangles (430) and (432) may generate a respective predicted value (or predicted vertex), such as (422) and (424). The predicted values for the current vertex may be ordered based on the ordering of the corresponding predicted triangles in the mesh. If overlaps are determined among the predicted values, such overlaps may be removed from the list of predicted values.

[0078] In one example, if N=0, then there are no prediction candidates available for vertex V. If V is the first vertex to be coded in the mesh, then the predicted value V' of the first vertex can be set to 0, and the prediction residual R can be set to V. Thus, V'=0, R=V. If V is not the first vertex to be coded, then the predicted value V' of V can be set to the coding value of a previously coded vertex in the vertex array. Thus, the previously coded vertex becomes the predictor for vertex V. The prediction residual R can be determined as the difference between vertex V and the predicted value (or predicted vertex) V', R=V-V'.

[0079] In one example, if N=1, only one prediction candidate is available for vertex V. Therefore, the predicted value V′ can be determined based on only one prediction candidate, and the prediction residual R can be determined as the difference between V and V′, R=V−V′.

[0080] In one example, if N>=2, two or more prediction candidates can be used for vertex V. Therefore, multiple predicted values (or predicted vertices) can be determined based on two or more prediction candidates. In one embodiment, the average of the predicted values can be considered as an additional predictor V0′. V0′ can be determined by the following equation (5): V0'=(V1'+V2'+...+V N ') / N formula (5)

[0081] The average predicted value V0' is calculated by dividing the predicted values V1', V2', ..., V NEither the average predicted value V0' or the duplicate predicted value can be removed if it is equal to one of the values V0', ...

[0082] V on the encoder side i ' can be compared with V, where 0<=i<=N. A prediction index j can be selected. The selected prediction index j is the sum of V and the predicted value V i V corresponds to the minimum prediction residual between each of i '. In some embodiments, the minimized error (or minimum prediction residual) is 0 Norm, L 1 Norm, L 2 It can be measured by the L norm, or some other norm. For example, 0 The norm can be determined by the following equation (6):

number

[0083] At the decoder side, the selected index (e.g., a prediction index) can be decoded, and the selected predictor (or predicted value) can be restored from the list of predictors. For example, the decoder can determine one or more predicted values for the current vertex based on a prediction mode such as parallelogram prediction. In addition, the average of one or more predicted values can be considered as a predictor. The average predicted value and the one or more predicted values may be arranged to form a prediction list. In one example, the prediction list generated at the decoder side can be the same as the prediction list generated at the encoder side. According to the decoded prediction index, the selected predictor can be restored from the prediction list based on the prediction index.

[0084] In one embodiment, an upper limit M can be set. The upper limit M can indicate the number of prediction candidates that can be considered (or applied) in the prediction list. In some cases, if N>M, then for each vertex in the mesh, only the first M prediction candidates are considered. Thus, the prediction values V1', V2', ..., V associated with the first M prediction candidates are M ' is applied. In one embodiment, the mean of the first M predicted values may be determined by equation (7). V0'=(V1'+V2'+...+V M ') / M formula (7) M can be an integer, such as 4. Thus, up to M (e.g., up to 4) prediction candidates can be considered. Note that the average prediction V0′ is placed in a different position in the candidate list. In one example, the average prediction V0′ can be the first predictor in the prediction candidate list. In one example, the average prediction V0′ can be the last predictor in the prediction candidate list. In one example, the average prediction V0′ is the sum of the predicted values V1′, V2′, ..., V M can be placed inside '.

[0085] In some embodiments, the prediction index needs to be coded only when two or more prediction candidates exist. For example, when two or more prediction candidates are available for the current vertex, an index indicating the selected prediction candidate may be coded at the encoder side. At the decoder side, the decoder can determine two or more prediction candidates having the same arrangement as the encoder. The decoder can decode the coded prediction index based on the prediction index and reconstruct the selected prediction candidate from the two or more prediction candidates.

[0086] In one embodiment, if no prediction candidate is determined or only one prediction candidate is determined, the prediction index may not need to be coded. Thus, the predicted value can be the current vertex itself or only one prediction candidate. In one embodiment, the prediction index can always be coded regardless of the number of available predictor candidates. When no predictor is available, the signaled index value may not affect the decoding process at the decoder side.

[0087] When two or more prediction candidates are determined for the current vertex, the prediction index can be coded using fixed-length coding. For example, if three prediction candidates are determined for the current vertex, four possible prediction indices, 0, 1, 2, and 3, may be required (0 indicates the average value of candidates 1, 2, and 3). Therefore, a two-digit binary number can be applied to represent each of the four prediction indices. Note that different vertices of a triangular mesh may use different fixed lengths. For example, if another vertex has seven prediction candidates, the other vertex may use a three-digit binary number for its prediction index representation. The output from fixed-length coding can be further compressed by entropy coding, such as arithmetic coding.

[0088] Alternatively, the prediction indexes can be coded using variable length coding. For example, if four prediction candidates are determined for the current vertex, five possible prediction indices 0, 1, 2, 3, and 4 may be required. Variable length codes 0, 100, 101, 110, and 111 can be assigned to represent the five prediction indices 0, 1, 2, 3, and 4, respectively. Alternatively, variable length codes 1, 01, 001, 0001, and 00001 can be applied to represent the five prediction indices 0, 1, 2, 3, and 4, respectively. Note that different vertices of a triangular mesh may use different variable lengths. The output from variable length coding can be further compressed by entropy coding, such as arithmetic coding.

[0089] Alternatively, the prediction index may be differentially coded, where the prediction index of the current vertex can be predicted from a previously coded prediction index, and the prediction error associated with the prediction index can also be coded. For example, the difference between a first prediction index of a predictor and a second prediction index of the predictor may be coded first. The second prediction index may then be further coded based on the difference between the first prediction index and the second prediction index.

[0090] When an upper bound M is set to limit the maximum number of prediction candidates, and if more than M prediction candidates are available for vertex V, only the first M prediction candidates may be used to predict vertex V. Thus, (M+1) possible prediction indices can be coded, where M indicates the first M prediction candidates and 1 indicates the average of the first M prediction candidates. If overlaps are determined in the predicted values, the overlaps can be removed. Thus, the likelihood of a prediction index can be reduced. The prediction index can be coded by fixed-length coding, variable-length coding, differential coding, etc.

[0091] In one embodiment, a prediction residual R of a position value (or predicted value) may be coded (or determined). The prediction residual R may indicate a difference between the predicted value of the current vertex and the current vertex. The prediction residual R may be coded by fixed-length coding, exponential-Golomb coding, arithmetic coding, etc. Thus, the decoder can reconstruct the current vertex based on the predicted value and the prediction residual, and the predicted value may be determined based on a decoded prediction index from a prediction list.

[0092] 7 shows a flowchart outlining a process (700) according to one embodiment of the present disclosure. The process (700) can be used in an encoder, such as a video encoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit that performs the functions of the video encoder (103), a processing circuit that performs the functions of the video encoder (303), or the like. In some embodiments, the process (700) is implemented by software instructions, such that the processing circuit performs the process (700) when the processing circuit executes the software instructions. The process begins at (S701) and proceeds to (S710).

[0093] In step (S710), a mesh includes a plurality of vertices, and a plurality of candidate predictions for a current vertex among the plurality of vertices are determined based on a plurality of triangles in the mesh, each of the plurality of triangles corresponding to a respective candidate prediction.

[0094] In step S720, a prediction for the current vertex is determined from a plurality of candidate predictions. In one example, the prediction corresponds to a threshold prediction difference between the current vertex and each of the plurality of candidate predictions.

[0095] In step (S730), prediction information for the current vertex is further generated based on the determined prediction.

[0096] In some embodiments, to determine multiple candidate predictions for the current vertex, a first triangle of the multiple triangles is determined for the current vertex. The first triangle shares a side with a second triangle of the multiple triangles, and the current vertex is included in the second triangle and is on the opposite side of the shared side. The first prediction of the multiple candidate predictions is determined based on the first triangle determined by parallelogram prediction, where the first prediction and the first triangle form a first parallelogram.

[0097] In some embodiments, to determine multiple candidate predictions for the current vertex, an average prediction of the multiple candidate predictions is determined.

[0098] In some embodiments, to determine a prediction for the current vertex, a prediction list is determined for the current vertex. The prediction list includes predictors. The predictors include an average prediction and multiple candidate predictions that follow the average prediction in the prediction list. The ordering of the multiple candidate predictions in the prediction list is based on the ordering of multiple triangles corresponding to the multiple candidate predictions. For each of the predictors in the prediction list, a respective prediction index is determined.

[0099] In one example, the triangles are arranged based on an edge breaker algorithm, where the triangles are labeled in an arrangement of a spiral triangle spanning tree.

[0100] In one example, to determine a prediction for the current vertex, a prediction difference between each of the predictors in the prediction list and the current vertex is determined, and a prediction is further determined from the predictor in the prediction list that corresponds to the smallest prediction difference of the prediction differences.

[0101] In one example, to determine the prediction for the current vertex, the prediction is determined within the subset of predictors in the prediction list that has the smallest prediction difference.

[0102] In some embodiments, a respective prediction index is determined for each of the predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

[0103] In some embodiments, each prediction index is determined based on differential coding. In one example, a difference between a first prediction index of a predictor and a second prediction index of a predictor may be determined. The second prediction index is further coded based on the difference between the first prediction index and the second prediction index.

[0104] In some embodiments, the prediction information for the current vertex includes index information indicating the determined prediction in the prediction list, and residual information indicating the difference between the current vertex and the determined prediction.

[0105] The process then proceeds to (S799) and ends.

[0106] Process 700 may be adapted as appropriate. Steps of process 700 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0107] 8 shows a flowchart outlining a process (800) according to one embodiment of the present disclosure. The process (800) may be used in a decoder, such as a video decoder. In various embodiments, the process (800) is performed by a processing circuit, such as a processing circuit that performs the functions of the video decoder (110), a processing circuit that performs the functions of the video decoder (210), or the like. In some embodiments, the process (800) is implemented in software instructions, such that the processing circuit performs the process (800) when the processing circuit executes the software instructions. The process begins at (S801) and proceeds to (S810).

[0108] In step (S810), coding information of a mesh including a plurality of triangles and a plurality of vertices is received, the coding information including index information of a current vertex among the plurality of vertices, the index information indicating a prediction of the current vertex.

[0109] At (S820), multiple candidate predictions of the current vertex are determined based on the multiple triangles, with each of the multiple triangles corresponding to a respective candidate prediction.

[0110] In step S830, a prediction of the current vertex is determined from the plurality of candidate predictions based on the index information, the determined prediction corresponding to a threshold prediction difference between the current vertex and each of the plurality of candidate predictions.

[0111] In step S840, the current vertex is reconstructed based on the determined prediction.

[0112] Thereafter, the process proceeds to (S899) and ends.

[0113] Process 800 may be adapted as appropriate. Steps in process 800 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0114] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media, such as non-transitory computer-readable storage media. For example, Figure 9 illustrates a computer system (900) suitable for implementing certain embodiments of the disclosed subject matter.

[0115] Computer software may be coded using any suitable machine or computer language, which may be processed by assembly, compilation, linking, or similar mechanisms to generate code comprising instructions that may be executed directly, interpreted, executed by microcode, or otherwise executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), or other processing circuitry.

[0116] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0117] 9 for computer system (900) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (900).

[0118] The computer system (900) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0119] The input human interface devices may include one or more (only one of each shown) of a keyboard (901), a mouse (902), a trackpad (903), a touch screen (910), a data glove (not shown), a joystick (905), a microphone (906), a scanner (907), and a camera (908).

[0120] The computer system (900) may also include certain human interface output devices that may stimulate one or more of the human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (910), data gloves (not shown), or joystick (905), although haptic feedback devices that are not input devices may also exist), audio output devices (e.g., speakers (909), headphones (not shown)), visual output devices (e.g., screens (910), including CRT screens, LCD screens, plasma screens, OLED screens, etc., with or without touchscreen input capability and with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output via means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, smoke tanks (not shown)), and printers (not shown).

[0121] The computer system (900) may also include human-accessible storage devices and media associated with the storage devices, such as optical media including CD / DVD ROM / RW (920) with media (921) such as CD / DVD, thumb drive (922), removable hard drive or solid state drive (923), legacy magnetic media such as tape or floppy disk (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0122] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0123] The computer system 900 may also include an interface 954 to one or more communications networks 955. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; mobile networks, including WLAN, GSM, 3G, 4G, 5G, and LTE; wired or wireless wide-area digital television networks, including cable, satellite, and terrestrial television; and vehicular and industrial networks, including CAN Bus. Certain networks require an external network interface adapter that attaches to a particular general-purpose data port (e.g., a USB port on the computer system 900) or peripheral bus 949. Others are typically integrated into the core of the computer system 900 by attaching to a system bus, as described below (e.g., an Ethernet interface integrated into a PC computer system, or a mobile network interface integrated into a smartphone computer system). Using any of these networks, the computer system (900) may communicate with other entities. Such communications may be one-way, receive-only (e.g., television broadcasts), one-way transmit-only (e.g., CANbus device to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0124] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (940) of the computer system (900).

[0125] The core (940) may include one or more central processing units (CPUs) (941), graphics processing units (GPUs) (942), dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) (943), hardware accelerators (944) for specific tasks, graphics adapters (950), etc. These devices may be connected through a system bus (948), along with read-only memory (ROM) (945), random access memory (946), and internal mass storage (947) such as an internal non-user accessible hard drive or SSD. In some computer systems, the system bus (948) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (948) or through a peripheral bus (949). In one example, a screen (910) may be connected to the graphics adapter (950). Architectures for peripheral buses include PCI, USB, etc.

[0126] The CPU (941), GPU (942), FPGA (943), and accelerator (944) may execute specific instructions that, in combination, may constitute the aforementioned computer code. The computer code may be stored in ROM (945) or RAM (946). Temporary data may also be stored in RAM (946), while permanent data may be stored, for example, in internal mass storage (947). High-speed storage and retrieval from any of the memory devices may be enabled using cache memory, which may be closely associated with one or more of the CPU (941), GPU (942), mass storage (947), ROM (945), RAM (946), etc.

[0127] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0128] By way of example and not limitation, the computer system (900) having the architecture, and in particular the core (940), may provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (940) that is non-transitory in nature, such as the core's internal mass storage (947) or ROM (945). Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (940). The computer-readable media may include one or more memory devices or chips according to particular needs. The software may cause the core (940), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform particular processes, or particular portions of particular processes, described herein, including defining data structures stored in RAM (946) and modifying such data structures according to the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (944)) that may operate in place of or together with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may encompass logic, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0129] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0130] 100 Video processing system, communication system, 101 Video source, 102 Stream of video pictures, 103 Video encoder, 104 Video data, 105 Streaming server, 106 Client subsystem, 107 Input copy of video data, 108 Client subsystem, 109 Copy of video data, 110 Video decoder, 111 Output stream, 112 Display, 113 Capture subsystem, 120 Electronic device, 130 Electronic device, 201 Channel, 210 Video decoder, 212 Rendering device, 215 Buffer memory, 220 Entropy decoder / parser, 221 Symbol, 230 Electronic device, 231 Receiver, 251 Scaler / inverse transform unit, 252 Intra-picture prediction unit, 253 Motion compensation prediction unit, 255 Aggregator, 256 Loop filter unit, 257 Reference picture memory, 258 Current picture buffer, 301 Video source, 303 Video encoder, 320, electronic device, 330, source coder, 332, coding engine, 333, local video decoder, 334, reference picture memory, 335, predictor, 340, transmitter, 343, video sequence, 345, entropy coder, 350, controller, 360, communication channel, 400A, mesh, 400B, mesh, 402, current vertex, 404, predicted vertex, 406, vertex, 408, vertex, 410, vertex, 412, triangle, 414, predicted triangle, 416, edge, 418, triangle, 420, current vertex, 422, second predicted vertex, 424, first predicted vertex, 426, adjacent triangle, 428, adjacent triangle, 430, predicted triangle, 432, predicted triangle, 434, first triangle, 436, second triangle, 438, vertex, 440, vertex, 442, vertex, 444 Edges, 446 Edges, 500 Mesh, 502 Triangles, 504 Triangles, 700 Processes, 800 Processes, 900 Computer systems, 901 Keyboards, 902 Mouse, 903 Trackpad, 905 Joystick, 906 Microphones, 907 Scanners, 908 Cameras, 909 Speakers, 910 Touchscreens, 920 CD / DVD ROM / RW, 921 Media, 922 Thumb drives, 923Solid-state drive, 940 core, 941 central processing unit (CPU), 942 graphics processing unit (GPU), 943 field programmable gate array (FPGA), 944 hardware accelerator, 945 read-only memory (ROM), 946 random access memory (RAM), 947 internal mass storage, 948 system bus, 949 peripheral bus, 950 graphics adapter, 954 interface, 955 communication network

Claims

1. 1. A method of mesh processing performed in a video encoder, the method comprising: determining a plurality of candidate predictions for a current vertex, the current vertex being within a plurality of vertices of a mesh, the plurality of candidate predictions being determined based on a plurality of triangles within the mesh, each of the plurality of triangles corresponding to a respective candidate prediction; determining a prediction for the current vertex from the plurality of candidate predictions that corresponds to a minimum prediction residual between the current vertex and each of the plurality of candidate predictions; generating prediction information for the current vertex based on the determined prediction.

2. determining the plurality of candidate predictions for the current vertex; determining a first triangle of the plurality of triangles of the current vertex, the first triangle sharing an edge with a second triangle of the plurality of triangles, the current vertex being contained within the second triangle and on the opposite side of the shared edge; 2. The method of claim 1 , further comprising: determining a first prediction of the plurality of candidate predictions, based on the determined first triangle by a parallelogram prediction, wherein the first prediction and the first triangle form a first parallelogram.

3. determining the plurality of candidate predictions for the current vertex; The method of claim 2 , further comprising determining an average prediction of the plurality of candidate predictions.

4. determining the prediction of the current vertex by: determining a prediction list for the current vertex, where predictors in the prediction list include the average prediction and the plurality of candidate predictions that follow the average prediction in the prediction list, and an ordering of the plurality of candidate predictions in the prediction list based on an ordering of the plurality of triangles corresponding to the plurality of candidate predictions; and determining a respective prediction index for each of the predictors in the prediction list.

5. The method of claim 4 , wherein the plurality of triangles is arranged based on an edge breaker algorithm, and the plurality of triangles is labeled in an arrangement of a spiral triangle spanning tree.

6. determining the prediction of the current vertex by: determining a prediction difference between each of the predictors in the prediction list and the current vertex; and determining the prediction from the predictor in the prediction list that corresponds to a minimum prediction difference of the prediction differences.

7. determining the prediction of the current vertex by: The method of claim 6 , further comprising determining the prediction within the subset of predictors in the prediction list having the smallest prediction difference.

8. 20. The method of claim 19, wherein the step of determining each of the prediction indexes comprises:

5. The method of claim 4, further comprising determining the respective prediction index for each of the predictors in the prediction list based on one of fixed length coding, variable length coding, and differential coding.

9. wherein said determining each prediction index is based on said differential coding; determining each prediction index based on the differential coding, determining a difference between a first predicted index of the predictor and a second predicted index of the predictor; 9. The method of claim 8, further comprising: coding the second predicted index based on the difference between the first predicted index and the second predicted index.

10. The prediction information for the current vertex is: index information indicating the determined prediction within the list of predictions; and The method of claim 4 , further comprising residual information indicating a difference between the current vertex and the determined prediction.

11. Apparatus for processing meshes, which performs the method according to any one of claims 1 to 10.

12. A computer program comprising instructions which, when executed by at least one processor, cause the at least one processor to perform the method of any one of claims 1 to 10.

Images