Location compression based on duplicate vertices

Abstract

The method includes determining whether each of the plurality of duplicate vertices is skippable in the mesh. One or more skippable duplicate vertices of the plurality of duplicate vertices in the mesh are excluded to generate an updated mesh. The vertices of the updated mesh are divided into a plurality of vertex groups, the plurality of vertex groups including a first vertex group. A prediction mode of the first vertex group is determined based on at least an estimation error of the vertices in the first vertex group of the updated mesh. Each estimation error indicates a difference between a respective vertex in the first vertex group and a corresponding reference vertex in a reference frame. Prediction information of the vertices in the first vertex group is generated based on the determined prediction mode.

Description

[Technical field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 18 / 207,610 entitled "DUPLICATE VERTICES BASED POSITION COMPRESSION," filed June 8, 2023, which claims the benefit of priority to U.S. Provisional Application No. 63 / 394,485 entitled "Duplicate Vertices based Position Compression," filed August 2, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] The present disclosure includes embodiments that relate to mesh processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) capture, modeling, and rendering are facilitating the ubiquity of 3D content across platforms and devices. Today, it is possible to capture a baby's first steps on one continent while the baby's grandparents on another continent watch (and potentially interact with) and enjoy an immersive experience with the child. To achieve this immersion, models are becoming ever more sophisticated, and a significant amount of data is tied to the generation and consumption of these models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Problem to be solved by the invention]

[0005] Aspects of the present disclosure provide a method and apparatus for mesh processing. In some examples, an apparatus for mesh processing includes a processing circuit. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a method of mesh processing performed in a video encoder is provided. In the method, it is determined whether each of a plurality of overlapping vertices is skippable in a mesh of a current frame. Each of the plurality of overlapping vertices is a vertex of the mesh that is an overlap of another corresponding vertex in the mesh. Each of the plurality of overlapping vertices and its corresponding vertex have the same reference vertex in a reference frame. One or more skippable overlapping vertices of the plurality of overlapping vertices in the mesh are excluded to generate an updated mesh. The vertices of the updated mesh are divided into a plurality of vertex groups based on a predetermined constant integer, and the plurality of vertex groups includes a first vertex group. A prediction mode of the first vertex group is determined based on at least an estimation error of the vertices in the first vertex group of the updated mesh. Each estimation error indicates a difference between a respective vertex in the first vertex group and a corresponding reference vertex in the reference frame. Prediction information of the vertices in the first vertex group is generated based on the determined prediction mode of the first vertex group.

[0007] In one example, a first of the multiple overlapping vertices is determined to be skippable based on a first of the multiple overlapping vertices and a vertex in the mesh that corresponds to the first overlapping vertex having the same coordinates. In one example, a second of the multiple overlapping vertices is determined to be non-skippable based on a second of the multiple overlapping vertices and a vertex in the mesh that corresponds to the second overlapping vertex having different coordinates.

[0008] In some embodiments, a number of non-skippable duplicate vertices of the plurality of duplicate vertices is determined. Further, a vertex index of each non-skippable duplicate vertex of the plurality of duplicate vertices is determined.

[0009] In some embodiments, the coding information is generated based on one of: (i) the number of the plurality of overlapping vertices being greater than or equal to a first threshold, (ii) the number of vertices in the mesh being greater than or equal to a second threshold, and (iii) a ratio of the number of the plurality of overlapping vertices to the number of vertices in the mesh being greater than or equal to a third threshold, The coding information indicates (i) a number of non-skippable overlapping vertices among the plurality of overlapping vertices, and (ii) a vertex index for each non-skippable overlapping vertex among the plurality of overlapping vertices.

[0010] In one embodiment, the number of non-skippable overlapping vertices is coded based on one of variable length coding, fixed length coding, a fixed length code of (b+1) bits, and a fixed length code having a code length less than (b+1) bits, where b is a positive integer. In one embodiment, the vertex index of each of the non-skippable overlapping vertices of the multiple overlapping vertices is coded based on one of differential coding, variable length coding, Exp-Golomb coding, a fixed length code of (b+1) bits, and a fixed length code having a code length less than (b+1) bits.

[0011] In some embodiments, to determine a prediction mode for the first vertex group, an estimation error for the vertices in the first vertex group of the updated mesh is determined. An average neighborhood estimation error for a plurality of neighboring vertices associated with each vertex in the first vertex group is determined. Each of the plurality of neighborhood estimation errors indicates a difference between one of the plurality of neighboring vertices of the respective vertex in the first vertex group and a reference vertex in the reference frame corresponding to one of the plurality of neighboring vertices of the respective vertex in the first vertex group. A first cost value is determined based on the estimation error for the vertices in the first vertex group. A second cost value is determined based on the average neighborhood estimation error associated with the vertices in the first vertex group. A prediction mode for the first vertex group is determined based on a comparison between the first cost value and the second cost value.

[0012] In one example, to determine an average neighborhood estimation error for a plurality of neighboring vertices, a first neighborhood estimation error associated with a first one of the vertices in the first vertex group is determined. The first neighborhood estimation error indicates a difference between a first neighboring vertex of the first vertex and a reference vertex in a reference frame corresponding to the first neighboring vertex of the first vertex. A second neighborhood estimation error associated with the first one of the vertices in the first vertex group is determined. The second neighborhood estimation error indicates a difference between a second neighboring vertex of the first vertex and a reference vertex in a reference frame corresponding to the second neighboring vertex of the first vertex. An average neighborhood estimation error of the first neighborhood estimation error and the second neighborhood estimation error is determined, and the average neighborhood estimation error is associated with the first one of the vertices in the first vertex group.

[0013] In some embodiments, the first cost value is determined as a sum of the estimation errors of the vertices in the first group of vertices.

[0014] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each vertex in the first vertex group and an average neighborhood estimated error associated with a corresponding vertex in the first vertex group is determined. The second cost value is determined as a sum of the estimated differences associated with the vertices in the first vertex group.

[0015] In one example, the prediction mode is determined as the first mode based on the first cost value being less than or equal to the second cost value. In one example, the prediction mode is determined as the second mode based on the first cost value being greater than the second cost value.

[0016] In some embodiments, a first prediction residual is generated for each of the vertices in the first vertex group based on the prediction mode being a first mode, the first prediction residual indicating an estimated error of a corresponding vertex in the first vertex group, and a second prediction residual is generated for each of the vertices in the first vertex group based on the prediction mode being a second mode, the second prediction residual indicating an estimated error associated with the corresponding vertex in the first vertex group.

[0017] According to another aspect of the present disclosure, an apparatus is provided that includes a processing circuit that may be configured to perform any of the described methods for mesh processing.

[0018] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for mesh processing.

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4] 1 is a schematic diagram of vertices of a mesh according to some embodiments of the present disclosure. [Diagram 5] 1A-1C are schematic diagrams of exemplary vertex position compression, in accordance with some embodiments of the present disclosure. [Figure 6] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 7] 10 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 8] 1 is a schematic diagram of a computer system, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] 1 shows a block diagram of a video processing system (100) in some examples. The video processing system (100) is one example of an application for the disclosed subject matter, a video encoder and video decoder in a streaming environment. The disclosed subject matter may be equally applicable to other image and video enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] The video processing system (100) includes a capture subsystem (113) that may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera may create a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples taken by the digital camera. The stream of video pictures (102) may be processed by an electronic device (120) that includes a video encoder (103), shown in bold to emphasize the large amount of data compared to the encoded video data (104) (or coded video bitstream), coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in detail below. The encoded video data (104) (or encoded video bitstream), shown with thin lines to emphasize its small amount of data compared to the stream of video pictures (102), may be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, may access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) may include a video decoder (110), for example, within an electronic device (130). The video decoder (110) decodes an input copy (107) of the encoded video data and generates an output stream (111) of video pictures that may be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown).In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in conjunction with VVC.

[0023] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may also include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0024] 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include receiving circuitry, such as a network interface circuit. The video decoder (210) may be used in place of the video decoder (110) of the example of FIG. 1.

[0025] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver (231) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) can be external to the video decoder (210) (not shown). In still other applications, there can be a buffer memory (not shown) external to the video decoder (210), e.g., to combat network jitter, plus another buffer memory (215) internal to the video decoder (210), e.g., to handle playout timing. When the receiver (231) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (215) may not be necessary or can be small.For use over a best-effort packet network such as the Internet, a buffer memory (215) may be required, which may be relatively large and advantageously be adaptively sized, and may be implemented at least in part within an operating system or similar element (not shown) external to the video decoder (210).

[0026] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen), which is not an integral part of the electronic device (230) but may be coupled to the electronic device (230), as shown in FIG. 2. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) may extract a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group from the coded video sequence. The subgroup may include a Group of Pictures (GOP), an image, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0027] The parser (220) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (215) to produce symbols (221).

[0028] The reconstruction of the symbols (221) may involve several different units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are involved and how may be controlled by subgroup control information parsed by the parser (220) from the coded video sequence. The flow of such subgroup control information between the parser (220) and the following units is not depicted for clarity.

[0029] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate:

[0030] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients from the parser (220) as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0031] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0032] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (253) may access the reference picture memory (257) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (221) related to the block, these samples may be added by the aggregator (255) to the output of the scalar / inverse transform unit (251) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion compensated prediction unit (253) fetches the prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (253) in the form of symbols (221), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of fetched sample values ​​from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0033] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and to previously reconstructed loop filtered sample values.

[0034] The output of the loop filter unit (256) may be a sample stream that may be output to a rendering device (212) and may also be stored in a reference picture memory (257) for use in future inter-picture prediction.

[0035] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and a new current picture buffer may be reallocated before beginning reconstruction of a subsequent coded picture.

[0036] The video decoder (210) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Recommendation H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and to the profile documented in the video compression technique or standard. In particular, the profile may select certain tools from among all tools available in the video compression technique or standard as tools that are only available to them under the profile. Also, compliance may require that the complexity of the coded video sequence be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0037] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) may be used in place of the example video encoder (103) of FIG.

[0039] The video encoder (303) may receive video samples from a video source (301) (which is not part of the electronic device (320) in the example of FIG. 3) that may capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0040] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc. in use. The following description focuses on samples.

[0041] According to one embodiment, the video encoder (303) may code and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraints required. Providing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functions associated with the video encoder (303) optimized for a particular system design.

[0042] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (330) (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to that which the (remote) decoder also creates. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since decoding of the symbol stream results in bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0043] The operation of the "local" decoder (333) may be the same as the operation of a "remote" decoder, such as the video decoder (210), already described in detail above in conjunction with Figure 2. However, with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0044] In one embodiment, the decoder techniques, except for parsing / entropy decoding, present in the decoder are present in the corresponding encoder in the same or substantially the same functional form. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques may be omitted, since they are the inverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0045] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference picture(s) that may be selected as predictive reference(s) for the input picture.

[0046] The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a replica of the source video sequence, usually with some errors. The local video decoder (333) may reproduce the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (334). In this manner, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end video decoder (without transmission errors).

[0047] The predictor (335) may perform a prediction search for the coding engine (332). That is, the predictor (335) may search for sample data (as candidate reference pixel blocks) from the reference picture memory (334) for a new picture to be coded, or some metadata such as reference picture motion vectors, block shapes, etc., which may serve as suitable prediction references for the new picture. The predictor (335) may operate on sample blocks, pixel block by pixel block, to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0048] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The output of all the aforementioned functional units may be entropy coded by an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0050] The transmitter (340) may buffer the coded video sequence(s) created by the entropy coder (345) in preparation for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign several coding picture types to each of the coded pictures, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0052] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0053] A predictive picture (P picture) may be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction with reference to one previously coded reference picture or via temporal prediction. A block of a B picture may be predictively coded via spatial prediction with reference to one or two previously coded reference pictures or via temporal prediction.

[0056] The video encoder (303) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0057] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0058] A video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation in a given picture, while inter-picture prediction uses (temporal or other) correlation between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0059] In some embodiments, bi-prediction techniques may be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but the display order may be past and future, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first reference block and the second reference block.

[0060] Furthermore, merge mode techniques may be used in inter-picture prediction to improve coding efficiency.

[0061] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, 16×16 pixels, etc. In general, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal predictability and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0062] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0063] This disclosure includes embodiments directed to a method and system for mesh vertex position compression with duplicate vertices. In some embodiments, each of the duplicate vertices in a mesh is a duplicate of another corresponding vertex in the mesh in a current frame. The duplicate vertex and each of its corresponding vertices may have the same reference vertex in a reference frame.

[0064] A mesh may include a number of polygons that describe the surface of a volumetric object. Each polygon of the mesh may be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information about how the vertices are connected, sometimes referred to as connectivity information. In some embodiments, vertex attributes such as color, normal, etc. may be associated with the mesh vertices. Attributes (or vertex attributes) may also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a two-dimensional (2D) attribute map. Such mapping may be described by a set of parametric coordinates, usually called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps may be used to store high-resolution attribute information such as texture, normal, displacement, etc. Such information may be used for various purposes such as texture mapping and shading.

[0065] Dynamic meshes may require a large amount of data since they may contain a significant amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards such as IC, MESHGRID, and FAMC were previously developed by MPEG to deal with dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards may not consider time-varying attribute maps and connectivity information. DCC (Digital Content Generation) tools usually generate such dynamic meshes. However, volume acquisition techniques may have difficulty generating constant connectivity dynamic meshes, especially under real-time constraints. This kind of content (e.g., constant connectivity dynamic meshes) may not be supported by existing standards. MPEG is planning to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. The new mesh compression standard targets lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, augmented reality (AR), and virtual reality (VR). Features such as random access and scalable / progressive coding may also be considered.

[0066] Mesh geometry information can include vertex connectivity information, 3D coordinates, 2D texture coordinates, etc. Compression of vertex 3D coordinates, sometimes also referred to as vertex positions, can be important because in many cases, compressing vertex 3D coordinates can consume a significant portion of the overall geometry-related data.

[0067] A dynamic mesh sequence M at a time instance t can be represented as M(t). M(t) can be referred to as a positionally tracked frame if there is a mapping (or mapping operation) f from vertex positions in M(t) to vertex positions in another time instance M(t0), where t and t0 are different time instances. Thus, M(t0) can be referred to as a reference frame, and a corresponding vertex in the reference frame can be referred to as a reference vertex of a vertex in M(t).

[0068] In this disclosure, a method and / or system for mesh vertex position compression with overlapping vertices is proposed. It is noted that the method and / or system may be applied individually or in any form of combination. Furthermore, the disclosed method and system is not limited to vertex position compression. The disclosed method and system may also be applied to, for example, two-dimensional (2D) texture coordinate compression or more general temporal prediction-based schemes.

[0069] For a vertex V in a positionally tracked frame M(t), the neighbors of the vertex may also be the vertices connected to V through edges, and these vertices are called the neighbor vertices (or neighboring vertices) of V. For example, as shown in FIG. 4, vertex A may have four neighbor vertices, which are C, D, E, and B. Vertex E may have five neighbor vertices, which are A, B, F, H, and D.

[0070] For a positionally tracked frame M(t) and its reference frame M(t0), let f be a mapping between vertex positions in M(t) and M(t0). Given a vertex V in M(t), let f(V) be the reference vertex for V and the coded vertex in M(t).

number

number

number

number

number

[0071] In one embodiment, for a duplicate vertex V, V and

number

number

[0072] FIG. 5 is a schematic diagram of an exemplary vertex position compression device (or apparatus) (500) according to some embodiments of the present disclosure. As illustrated in FIG. 5, the vertex position compression device (500) may include a duplication signaling module (501) configured to determine whether a duplicate vertex is skippable. The duplication signaling module (501) may also be configured to signal (send) a vertex index of a skippable duplicate vertex or a vertex index of a non-skippable duplicate vertex. The apparatus (500) may include a vertex grouping module (502) configured to group vertices of a mesh, a position prediction module (504) configured to calculate a predicted position of the vertex, a prediction mode coding module (506) configured to code (or determine) a prediction mode for the position compression, and a prediction residual coding module (508) configured to code a position prediction residual.

[0073] The term module in this disclosure can refer to a software module, a hardware module, or a combination thereof. A software module (e.g., a computer program) may be developed using a computer programming language. A hardware module may be implemented using a processing circuit and / or a memory. Each module can be implemented using one or more processors (or a processor and a memory). Similarly, a processor (or a processor and a memory) may be used to implement one or more modules. Furthermore, each module may be a part of an entire module that includes the functionality of the module.

[0074] For a positionally tracked frame M(t), duplicate vertices may be identified using a reference frame M(t0). For each duplicate vertex V, it may be determined as either a skippable duplicate or a non-skippable duplicate. In one embodiment, a set S may be represented as the collection of skippable duplicates, and a set NS may be represented as the collection of non-skippable duplicates.

[0075] In one embodiment, the encoder can signal the vertex indices of the non-skippable duplicates. In one example, the set NS is NS={d1, d2, ..., d s}, where 0≦d1 <d2<…<d s ≦T−1, and d i is the vertex index of the i-th non-skippable duplicate, s is the number (or count) of non-skippable duplicate vertices, and T is the number (or count) of vertices in M(t). In one example, s > 0 and T > 3. If s = 0, the set NS can be the empty set and all duplicate vertices are skippable. The encoder uses the number s and each index d for duplicate signaling. i can be coded.

[0076] The number s may be coded based on various coding methods. In one embodiment, the number s may be coded using variable length coding. In one embodiment, the number s may be coded using fixed length coding. In one embodiment, the number s may be coded using a (b+1)-bit fixed length code, where b can be the largest integer less than or equal to log2(T). In one embodiment, the number s may be coded using a fixed length code, where the code length is less than (b+1)-bits, where b is the largest integer less than or equal to log2(T). Since s may be coded using more than one fixed length code, the first bit (or the last bit) in each fixed length code may be signaled to indicate whether the respective fixed length code is a representation of a new symbol or a continuation of the last (or previous) symbol.

[0077] Vertex index d i may be coded based on various coding methods. In one embodiment, the vertex index d i can be coded using differential coding, in which the encoder calculates multiple parameters D i(i ≧ 1) can be coded. i D1=d1(i=1) and D i =d i -d i-1 -1 (i ≥ 2). Therefore, the vertex index of a non-skippable vertex is the coded D i It can be expressed based on (i≧1).

[0078] In one embodiment, D i may be coded using variable length coding. i may be coded using an Exp-Golomb code. i may be coded using fixed length coding. i may be coded using a (B+1)-bit fixed length code, where B is the largest integer less than or equal to log2(T-2). i may be coded using a fixed-length code, where the code length is less than (B+1) bits, and B is the largest integer less than or equal to log2(T-2). i may be coded using more than one fixed length code, so the first bit (or the last bit) in each fixed length code may be signaled to indicate whether the respective fixed length code is a representation of a new symbol or a continuation of the last (or previous) symbol.

[0079] In some embodiments, the vertex index of a skippable duplicate (or duplicate vertex) may be signaled.

[0080] In some embodiments, the encoder can adaptively turn on or off the duplicate signaling. The adaptive decision to turn on or off the duplicate signaling can be based on vertex statistics. In one example, the adaptive decision to turn on or off the duplicate signaling can be based on the number (or count) of duplicate vertices. If the number of duplicate vertices is equal to or greater than a threshold, duplicate signaling can be turned on. Otherwise, duplicate signaling can be turned off. In one example, the adaptive decision to turn on or off the duplicate signaling can be based on the number (or count) of vertices T. If the number of vertices T is equal to or greater than a threshold, duplicate signaling can be turned off. Otherwise, duplicate signaling can be turned on. In one example, the adaptive decision to turn on or off the duplicate signaling can be based on a ratio between the number of duplicate vertices and the number of vertices. If the ratio is equal to or greater than a threshold, duplicate signaling can be turned on. Otherwise, duplicate signaling can be turned off.

[0081] In the present disclosure, the vertices of a frame M(t) that are positionally tracked may be divided into groups, and each group may contain K vertices, where K is a constant. For example, the vertices of M(t) may be divided into multiple groups, and each group may contain 10 vertices (K=10). In one example, when K=1, each group contains one vertex. In yet another example, when K is equal to the number of vertices in frame M(t), all vertices are included in the same group.

[0082] In one embodiment of the present disclosure, if the vertices of M(t) are divided into groups, the skippable duplicates (or skippable duplicate vertices) can be skipped (or removed). For example, the vertices of M(t) are divided into groups, each group containing 10 vertices. If the first 12 vertices of M(t) have two skippable duplicates, a first group of M(t) can be formed based on the vertices with indexes from 0 to 11, and the two skippable duplicates of the first 12 vertices are removed.

[0083] In other embodiments, when M(t) is divided into groups, all vertices, including skippable duplicates, may be considered (or included).

[0084] In the present disclosure, temporal prediction may be applied to predict a vertex in a current frame based on a reference vertex in a reference frame. For a vertex V in a positionally tracked frame (or current frame) M(t), the position of the vertex V can be estimated by the position of a reference vertex f(V) in a reference frame (e.g., M(t0), where t0 is a different time instance), where f is a mapping operation between M(t) and the reference frame. In some embodiments, the vertex V and the reference vertex f(V) in the reference frame are collocated. Thus, the reference vertex can have the same relative position in the reference frame as the vertex in the current frame M(t). When the vertex V is predicted by the reference vertex f(V), the estimation error E can be determined as the difference between the position of V and the position of f(V) in Equation (7). E=Vf(V) Equation (7)

[0085] Since each vertex in the frame M(t) may have a 3D coordinate, the 3D coordinate components of the estimated error E may be provided based on equation (1). For example, assuming that the subscripts x, y, and z represent 3D coordinates in the xyz space, the 3D coordinate components of the estimated error E may be given by equations (8)-(10). E x =V x -(f(V)) x Formula (8) E y =V y -(f(V)) y Formula (9) E z =V z -(f(V)) z Formula (10)

[0086] The estimated error E of a vertex V can be predicted or determined from the neighboring vertices (or neighboring vertices) of vertex V. For the neighboring vertices (or neighboring vertices) of V, the estimated errors of the neighboring vertices can be applied to predict E if the neighboring vertices have been coded and can be used for prediction.

[0087] V is a set of N neighboring vertices (or neighboring vertices) V1, V2, ..., V N Assume that we have a neighboring vertex V i If , the neighboring vertex V i The estimation error (or neighborhood estimation error) of is E for i=1,2,…,N. i =V i -f(V i ) can be obtained as f(V i ) is the neighboring vertex V in the reference frame. i The reference vertex of E i can also be referred to as the neighborhood estimation error associated with the vertex V, and each E i can be a candidate prediction for E.

[0088] If N>=2, then there are two or more estimated errors E i is available. In one embodiment, the estimation error E i The average estimation error (or average neighborhood estimation error) E0 of can be defined as follows in Equation (11): E0=(E1+E2+…+E N ) / N formula (11)

[0089] For a group G in a frame M(t), the encoder may determine a coding cost C0 and a coding cost C1. C0 may denote the sum of the estimated errors E of all vertices in the group G. Once the estimated errors E of each vertex in the group G are determined, such as based on equation (1), the coding cost C0 may be determined as the sum of the estimated errors E of the vertices in the group G. C1 may denote the sum of the estimated residuals (E-E0) of all vertices in the group G. To determine the coding cost C1, the estimated errors E of the neighboring vertices of each vertex in the group G may be determined as i E i =V i -f(V i ) and so on. Furthermore, the estimation error E i The average neighborhood estimation error E0 of may be determined based on equation (5). An estimated residual (or estimated difference) (E-E0) may be determined for each vertex in group G. A coding cost C1 may be determined as the sum of the estimated residuals associated with the vertices of group G.

[0090] In this disclosure, a prediction mode may be determined based on the coding cost C0 and the coding cost C1.

[0091] If the coding cost of C0 is less than or equal to C1, prediction mode 0 is applied to group G. Prediction mode 0 indicates setting the prediction residual for each vertex in group G as the estimated error E of each vertex. Thus, at the decoder side, each vertex in group G can be reconstructed as V=f(V)+E, where f(V) is a reference vertex in reference frame M(t0) corresponding to each vertex in group G in frame M(t).

[0092] If the coding cost of C0 is greater than C1, prediction mode 1 is applied to group G. Prediction mode 1 indicates that the prediction residual of each vertex in group G is set as (E-E0), where E is the estimation error of each vertex, and E0 is the average estimation error of the estimation errors of the neighboring vertices of each vertex. Therefore, at the decoder side, each vertex in group G can be reconstructed as V=f(V)+(E-E0), where f(V) is a reference vertex in reference frame M(t0) corresponding to each vertex in group G in frame M(t).

[0093] In this disclosure, a prediction mode of group G may be coded. The prediction mode may be a binary number, such as 0 or 1. In one embodiment, the prediction mode may be coded using entropy coding. In one embodiment, the prediction mode may be coded using arithmetic coding. In one embodiment, the prediction mode may be coded using context-based arithmetic coding, such as context-adaptive binary arithmetic coding (CABAC). In one embodiment, the prediction mode may be coded using spatial context-based arithmetic coding, where the context may be conditional (or based) on previously coded groups of the same frame.

[0094] In one embodiment, if the prediction mode in the reference frame (e.g., M(t0)) has already been coded and is available, the prediction mode of the group in frame M(t) may be coded using a temporal context, such as temporal context-based arithmetic coding. Thus, the prediction mode of the group in frame M(t) may be determined based on the prediction mode of the corresponding (or associated) group in the reference frame based on temporal context-based arithmetic coding. The reason is that each vertex in the group in frame M(t) has a corresponding reference vertex in the associated group in the reference frame, and a one-to-one association between the positionally tracked group in frame M(t) and the associated group in reference frame M(t0) may also be established. The associated group in the reference frame (e.g., M(t0), where t0 is a different time instance from t) may be referred to as a reference group.

[0095] In one embodiment, a logical operation (or flag) such as XOR (exclusive OR) may be coded to indicate a relationship between the prediction mode of group G and the prediction mode of the reference group of group G. Thus, if group G and the reference group have the same prediction mode, the logical operation XOR may be coded as 0. If group G and the reference group have different prediction modes, the logical operation XOR may be coded as 1.

[0096] In one embodiment, a binary flag (e.g., 0 or 1) may be coded to indicate whether group G and the reference group of group G have the same prediction mode. Thus, if group G and the reference group have the same prediction mode, a binary 1 may be coded. If group G and the reference group have different prediction modes, a binary 0 may be coded.

[0097] In one embodiment, the prediction mode of group G in positionally tracked frame M(t) may be coded using temporal context-based arithmetic coding, and the context of the temporal context-based arithmetic coding may be based on the prediction mode of a reference group of group G.

[0098] In some embodiments, the prediction residual (e.g., E or (E-E0)) may be coded using a coding algorithm such as fixed length coding, exponential Golomb coding, arithmetic coding, etc. In some embodiments, the prediction residual may undergo a compaction transform such as a fast Fourier transform (FFT), a discrete cosine transform (DCT), a discrete sine transform (DST), a discrete wavelet transform (DWT), etc. The output from the compaction transform may be coded using a coding algorithm such as fixed length coding, exponential Golomb coding, arithmetic coding, etc.

[0099] 6 shows a flow chart outlining a process (600) according to one embodiment of the disclosure. The process (600) may be used in an encoder, such as a video encoder. In various embodiments, the process (600) is performed by a processing circuit, such as a processing circuit performing the functions of a video encoder (103), a processing circuit performing the functions of a video encoder (303), etc. In some embodiments, the process (600) is implemented with software instructions, such that the processing circuit performs the process (600) when it executes the software instructions. The process starts at (S601) and proceeds to (S610).

[0100] At (S610), it is determined whether each of the multiple duplicate vertices can be skipped in the mesh of the current frame. Each of the multiple duplicate vertices is a vertex of the mesh that is a duplicate of another corresponding vertex in the mesh. Each of the multiple duplicate vertices and its corresponding vertex have the same reference vertex in the reference frame.

[0101] At (S620), one or more skippable duplicate vertices of the plurality of duplicate vertices in the mesh are eliminated to generate an updated mesh.

[0102] At (S630), the vertices of the updated mesh are divided into a plurality of vertex groups based on a predetermined constant integer, and the plurality of vertex groups includes a first vertex group.

[0103] At (S640), a prediction mode for the first vertex group is determined based at least on estimation errors of vertices in the first vertex group of the updated mesh, each estimation error indicating a difference between a respective vertex in the first vertex group and a corresponding reference vertex in the reference frame.

[0104] At (S650), prediction information for the vertices in the first vertex group is generated based on the determined prediction mode of the first vertex group.

[0105] In one example, a first of the multiple overlapping vertices is determined to be skippable based on a first of the multiple overlapping vertices and a vertex in the mesh that corresponds to the first overlapping vertex having the same coordinates. In one example, a second of the multiple overlapping vertices is determined to be non-skippable based on a second of the multiple overlapping vertices and a vertex in the mesh that corresponds to the second overlapping vertex having different coordinates.

[0106] In some embodiments, a number of non-skippable duplicate vertices of the plurality of duplicate vertices is determined. Further, a vertex index of each non-skippable duplicate vertex of the plurality of duplicate vertices is determined.

[0107] In some embodiments, the coding information is generated based on one of: (i) the number of the plurality of overlapping vertices being greater than or equal to a first threshold, (ii) the number of vertices in the mesh being greater than or equal to a second threshold, and (iii) a ratio of the number of the plurality of overlapping vertices to the number of vertices in the mesh being greater than or equal to a third threshold, The coding information indicates (i) a number of non-skippable overlapping vertices among the plurality of overlapping vertices, and (ii) a vertex index for each non-skippable overlapping vertex among the plurality of overlapping vertices.

[0108] In one embodiment, the number of non-skippable overlapping vertices is coded based on one of variable length coding, fixed length coding, a fixed length code of (b+1) bits, and a fixed length code having a code length less than (b+1) bits, where b is a positive integer. In one embodiment, the vertex index of each of the non-skippable overlapping vertices of the multiple overlapping vertices is coded based on one of differential coding, variable length coding, Exp-Golomb coding, a fixed length code of (b+1) bits, and a fixed length code having a code length less than (b+1) bits.

[0109] In some embodiments, to determine a prediction mode for the first vertex group, an estimation error for the vertices in the first vertex group of the updated mesh is determined. An average neighborhood estimation error for a plurality of neighboring vertices associated with each vertex in the first vertex group is determined. Each of the plurality of neighborhood estimation errors indicates a difference between one of the plurality of neighboring vertices of the respective vertex in the first vertex group and a reference vertex in the reference frame corresponding to one of the plurality of neighboring vertices of the respective vertex in the first vertex group. A first cost value is determined based on the estimation error for the vertices in the first vertex group. A second cost value is determined based on the average neighborhood estimation error associated with the vertices in the first vertex group. A prediction mode for the first vertex group is determined based on a comparison between the first cost value and the second cost value.

[0110] In one example, to determine an average neighborhood estimation error for a plurality of neighboring vertices, a first neighborhood estimation error associated with a first one of the vertices in the first vertex group is determined. The first neighborhood estimation error indicates a difference between a first neighboring vertex of the first vertex and a reference vertex in a reference frame corresponding to the first neighboring vertex of the first vertex. A second neighborhood estimation error associated with the first one of the vertices in the first vertex group is determined. The second neighborhood estimation error indicates a difference between a second neighboring vertex of the first vertex and a reference vertex in a reference frame corresponding to the second neighboring vertex of the first vertex. An average neighborhood estimation error of the first neighborhood estimation error and the second neighborhood estimation error is determined, and the average neighborhood estimation error is associated with the first one of the vertices in the first vertex group.

[0111] In some embodiments, the first cost value is determined as a sum of the estimation errors of the vertices in the first group of vertices.

[0112] In some embodiments, to determine the second cost value, an estimated difference between an estimated error of each vertex in the first vertex group and an average neighborhood estimated error associated with a corresponding vertex in the first vertex group is determined. The second cost value is determined as a sum of the estimated differences associated with the vertices in the first vertex group.

[0113] In one example, the prediction mode is determined as the first mode based on the first cost value being less than or equal to the second cost value. In one example, the prediction mode is determined as the second mode based on the first cost value being greater than the second cost value.

[0114] In some embodiments, a first prediction residual is generated for each of the vertices in the first vertex group based on the prediction mode being a first mode, the first prediction residual indicating an estimated error of a corresponding vertex in the first vertex group, and a second prediction residual is generated for each of the vertices in the first vertex group based on the prediction mode being a second mode, the second prediction residual indicating an estimated error associated with the corresponding vertex in the first vertex group.

[0115] Thereafter, the process proceeds to (S699) and ends.

[0116] Process (600) may be adapted as appropriate. Step(s) of process (600) may be modified and / or omitted. Further step(s) may be added. Any suitable order of performance may be used.

[0117] 7 shows a flow chart outlining a process (700) according to one embodiment of the disclosure. The process (700) may be used in a decoder, such as a video decoder. In various embodiments, the process (700) is performed by a processing circuit, such as a processing circuit performing the functions of the video decoder (110), a processing circuit performing the functions of the video decoder (210), or the like. In some embodiments, the process (700) is implemented with software instructions, such that the processing circuit performs the process (700) when the processing circuit executes the software instructions. The process begins at (S701) and proceeds to (S710).

[0118] At (S710), coding information for a mesh in a current frame is received. The coding information indicates (i) a number of non-skippable duplicate vertices among a plurality of duplicate vertices in the mesh, and (ii) a vertex index of each of the non-skippable duplicate vertices. Each of the plurality of duplicate vertices is a vertex of the mesh that is a duplicate of another corresponding vertex in the mesh. Each of the plurality of duplicate vertices and its corresponding vertex have the same reference vertex in the reference frame.

[0119] At (S720), one or more skippable duplicate vertices of the plurality of duplicate vertices in the mesh are eliminated to generate an updated mesh.

[0120] At (S720), the vertices of the updated mesh are divided into a plurality of vertex groups based on a predetermined constant integer, and the plurality of vertex groups includes a first vertex group.

[0121] At (S730), a prediction mode for a first vertex group of the plurality of vertex groups is determined based on the coding information.

[0122] At (S740), the vertices in the first vertex group are reconstructed based on the determined prediction mode.

[0123] Thereafter, the process proceeds to (S799) and ends.

[0124] Process (700) may be adapted as appropriate. Steps of process (700) may be modified and / or omitted. Further step(s) may be added. Any suitable order of performance may be used.

[0125] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 8 illustrates a computer system (800) suitable for implementing certain embodiments of the disclosed subject matter.

[0126] Computer software may be coded using any suitable machine code or computer language that may undergo mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that may be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0127] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0128] 8 for computer system (800) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having a dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (800).

[0129] The computer system (800) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0130] The input human interface devices may include one or more (only one of each shown) of a keyboard (801), a mouse (802), a trackpad (803), a touch screen (810), a data glove (not shown), a joystick (805), a microphone (806), a scanner (807), and a camera (808).

[0131] The computer system (800) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (810), data gloves (not shown), or joystick (805), although there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (809), headphones (not shown)), visual output devices (e.g., screens (810), including CRT screens, LCD screens, plasma screens, OLED screens, etc., with or without touch screen input capability and with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output via such means as stereoscopic output, virtual reality glasses (not shown), holographic displays, smoke tanks (not shown), and printers (not shown).

[0132] The computer system (800) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (820), including media such as CDs / DVDs (821), thumb drives (822), removable hard drives or solid state drives (823), traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0133] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0134] The computer system (800) may also include an interface (854) to one or more communication networks (855). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CANBus, and the like. Certain networks generally require an external network interface adapter (e.g., a USB port on the computer system (800)) attached to a particular general-purpose data port or peripheral bus (849), while other networks are generally integrated into the core of the computer system (800) by attachment to a system bus as described below (e.g., the integration of an Ethernet interface into a PC computer system, or the integration of a cellular network interface into a smartphone computer system). Using any of these networks, computer system (800) can communicate with other entities. Such communications may be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., CANBus device to a particular CANBus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0135] The aforementioned human interface devices, human accessible storage, and network interfaces may be attached to the core (840) of the computer system (800).

[0136] The core (840) may include one or more central processing units (CPUs) (841), graphics processing units (GPUs) (842), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (843), hardware accelerators for specific tasks (844), graphics adapters (850), etc. These devices may be connected via a system bus (848), along with read only memory (ROM) (845), random access memory (846), internal mass storage (847) such as internal hard drives or SSDs that are not user accessible, etc. In some computer systems, the system bus (848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may also be attached directly to the core's system bus (848) or via a peripheral bus (849). In one example, a screen (810) may be connected to a graphics adapter (850). Architectures for peripheral buses include PCI, USB, etc.

[0137] The CPU (841), GPU (842), FPGA (843) and accelerator (844) may combine to execute certain instructions that may constitute the computer code described above. The computer code may be stored in ROM (845) or RAM (846). Temporary data may also be stored in RAM (846), while permanent data may be stored, for example, in internal mass storage (847). The use of cache memory, which may be closely associated with one or more CPUs (841), GPUs (842), mass storage (847), ROM (845), RAM (846), etc., may enable fast storage and retrieval in any of the memory devices.

[0138] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the available kind well known to those skilled in the computer software arts.

[0139] By way of example and not limitation, a computer system (800) having an architecture, and specifically a core (840), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (840) of a non-transitory nature, such as a core internal mass storage (847) or ROM (845). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (840). Depending on the particular needs, the computer-readable media may include one or more memory devices or chips. The software may cause the core (840), and specifically the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (846) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of hardwired or otherwise embodied logic in circuitry (e.g., accelerator (844)) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0140] The use of "at least one of" or "one of" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C, i.e., at least one of A, B, and C, at least one of A, B, and / or C, at least one of A through C, is intended to include A only, B only, C only, or any combination thereof. Reference to one of A or B and one of A and B is intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the listed elements where applicable, such as when the elements are not mutually exclusive.

[0141] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0142] 100 Video Processing System 101 Video Sources 102 Stream 103 Video Encoder 104 Encoded video data 105 Streaming Server 106 Client Subsystem 107 Copying Encoded Video Data 108 Client Subsystem 109 Copying Encoded Video Data 110 Video Decoder 111 Output Stream 112 Display 113 Capture Subsystem 120 Electronic Devices 130 Electronic Devices 201 Channel 210 Video Decoder 212 Rendering Devices 215 Buffer Memory 220 Parser 221 Symbols 230 Electronic Devices 231 Receiver 251 Scaler / Descaler Unit 252 Intra Picture Prediction Unit 253 Motion Compensation Prediction Unit 255 Aggregator 256 Loop Filter Unit 257 Reference Picture Memory 258 picture buffer 301 Video Source 303 Video Encoder 320 Electronic Devices 330 Source Coder 332 Coding Engine 333 Local Video Decoder 334 Reference Picture Memory 335 Predictor 340 Transmitter 343 coded video sequence 345 Entropy Coder 350 Controller 360 Communication Channels 500 Vertex Position Compressor 501 Duplicate Signaling Module 502 Vertex Grouping Module 504 Location Prediction Module 506 Prediction Mode Coding Module 508 Prediction Residual Coding Module 600 processes 700 processes 800 Computer Systems 801 Keyboard 802 Mouse 803 Trackpad 805 Joystick 806 Microphone 807 Scanner 808 Camera 809 Speaker 810 screens 821 Medium 822 Thumb Drive 823 Solid State Drive 840 cores 841 Central Processing Unit (CPU) 842 Graphics Processing Unit (GPU) 843 Field Programmable Gate Area (FPGA) 844 Accelerator 845 Read-Only Memory (ROM) 846 Random Access Memory 847 Mass Storage 848 System Bus 849 Peripheral Bus 850 Graphics Adapter 854 Interface 855 Communication Network

Claims

1. 1. A method of mesh processing performed in a video encoder, the method comprising: determining whether each of a plurality of duplicate vertices can be skipped in a mesh of a current frame, each of the plurality of duplicate vertices being a vertex of the mesh that is a duplicate of another corresponding vertex in the mesh, and each of the plurality of duplicate vertices and its corresponding vertex having the same reference vertex in a reference frame; removing one or more skippable duplicate vertices from the plurality of duplicate vertices in the mesh to generate an updated mesh; Dividing the vertices of the updated mesh into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; determining a prediction mode for the first vertex group based at least on estimation errors of vertices in the first vertex group of the updated mesh, each of the estimation errors indicating a difference between a respective vertex in the first vertex group and a corresponding reference vertex in the reference frame; generating prediction information for the vertices in the first vertex group based on the determined prediction mode of the first vertex group; A method comprising:

2. The step of determining whether each of the plurality of overlapping vertices can be skipped comprises: Based on a first overlapping vertex of the plurality of overlapping vertices and a vertex in the mesh that corresponds to the first overlapping vertex and has the same coordinates, determining that the first overlapped vertex of the plurality of overlapped vertices can be skipped; Based on a second overlapping vertex of the plurality of overlapping vertices and a vertex in the mesh that corresponds to the second overlapping vertex and has different coordinates, determining that the second overlapped vertex of the plurality of overlapped vertices is non-skippable; The method of claim 1, further comprising:

3. determining a number of non-skippable overlapping vertices of the plurality of overlapping vertices; determining a vertex index for each of the non-skippable overlapping vertices of the plurality of overlapping vertices; 3. The method of claim 2, further comprising:

4. based on one of: (i) the number of the plurality of overlapping vertices being greater than or equal to a first threshold; (ii) the number of the vertices in the mesh being greater than or equal to a second threshold; and (iii) a ratio of the number of the plurality of overlapping vertices to the number of the vertices in the mesh being greater than or equal to a third threshold. generating coding information indicating (i) the number of non-skippable overlapping vertices among the plurality of overlapping vertices, and (ii) the vertex index for each of the non-skippable overlapping vertices among the plurality of overlapping vertices.

4. The method of claim 3, further comprising:

5. the number of non-skippable overlapping vertices is coded based on one of a variable length coding, a fixed length coding, a (b+1)-bit fixed length code, and a fixed length code having a code length less than (b+1) bits, where b is a positive integer; the vertex index of each of the non-skippable overlapped vertices of the plurality of overlapped vertices is coded based on one of differential coding, the variable length coding, Exp-Golomb coding, the (b+1) bit fixed length code, and a fixed length code having a code length smaller than the (b+1) bit. The method of claim 4.

6. The step of determining the prediction mode for the first vertex group comprises: determining the estimated errors for the vertices in the first group of vertices in the updated mesh; determining an average neighborhood estimation error for a number of neighbor vertices associated with each of the vertices in the first vertex group, each of the number of neighborhood estimation errors indicative of a difference between one of the number of neighbor vertices of the respective vertex in the first vertex group and a reference vertex in the reference frame that corresponds to the one of the number of neighbor vertices of the respective vertex in the first vertex group; determining a first cost value based on the estimation errors of the vertices in the first group of vertices and determining a second cost value based on the average neighborhood estimation errors associated with the vertices in the first group of vertices; determining the prediction mode for the first vertex group based on a comparison of the first cost value and the second cost value; The method of claim 1, further comprising:

7. The step of determining the average neighborhood estimation error for the plurality of neighboring vertices comprises: determining a first neighborhood estimation error associated with a first one of the vertices in the first vertex group, the first neighborhood estimation error indicating a difference between a first neighboring one of the first vertices and a reference vertex in the reference frame that corresponds to the first neighboring one of the first vertices; determining a second neighborhood estimation error associated with the first one of the vertices in the first vertex group, the second neighborhood estimation error indicating a difference between a second neighbor of the first vertices and a reference vertex in the reference frame that corresponds to the second neighbor of the first vertices; determining the average neighborhood estimation error of the first neighborhood estimation error and the second neighborhood estimation error associated with the first one of the vertices in the first group of vertices; 7. The method of claim 6, further comprising:

8. The step of determining the first cost value comprises: determining the first cost value as a sum of the estimated errors of the vertices in the first group of vertices; 8. The method of claim 7, further comprising:

9. The step of determining the second cost value further comprises: determining an estimated difference between the estimated error for each of the vertices in the first group of vertices and the average neighborhood estimated error associated with the corresponding vertex in the first group of vertices; determining the second cost value as a sum of the estimated differences associated with the vertices in the first group of vertices; 9. The method of claim 8, further comprising:

10. The step of determining the prediction mode comprises: determining the prediction mode as a first mode based on the first cost value being less than or equal to the second cost value; determining the prediction mode as a second mode based on the first cost value being greater than the second cost value; 10. The method of claim 9, further comprising:

11. The step of generating the prediction information further comprises: generating a first prediction residual for each of the vertices in the first vertex group based on the prediction mode being the first mode, the first prediction residual indicating the estimation error of the respective vertex in the first vertex group; generating a second prediction residual for each of the vertices in the first vertex group based on the prediction mode being the second mode, the second prediction residual indicative of the estimated difference associated with the respective vertex in the first vertex group; 11. The method of claim 10, further comprising:

12. 1. An apparatus for mesh processing, comprising: determining whether each of a plurality of duplicate vertices can be skipped in a mesh of a current frame, each of the plurality of duplicate vertices being a vertex of the mesh that is a duplicate of another corresponding vertex in the mesh, each of the plurality of duplicate vertices and its corresponding vertex having the same reference vertex in a reference frame; removing one or more skippable duplicate vertices from the plurality of duplicate vertices in the mesh to generate an updated mesh; Dividing the vertices of the updated mesh into a plurality of vertex groups based on a predetermined constant integer, the plurality of vertex groups including a first vertex group; determining a prediction mode for the first vertex group based at least on estimation errors of vertices in the first vertex group of the updated mesh, each of the estimation errors indicating a difference between a respective vertex in the first vertex group and a corresponding reference vertex in the reference frame; generating prediction information for the vertices in the first vertex group based on the determined prediction mode of the first vertex group; A processing circuit configured as follows:

13. An apparatus comprising:

13. The processing circuitry includes: Based on a first overlapping vertex of the plurality of overlapping vertices and a vertex in the mesh that corresponds to the first overlapping vertex and has the same coordinates, determining that the first overlapping vertex of the plurality of overlapping vertices can be skipped; Based on a second overlapping vertex of the plurality of overlapping vertices and a vertex in the mesh that corresponds to the second overlapping vertex and has different coordinates, determining that the second overlapped vertex of the plurality of overlapped vertices is non-skippable; The apparatus of claim 12 configured to:

14. The processing circuitry includes: determining a number of non-skippable overlapping vertices of the plurality of overlapping vertices; determining a vertex index for each of the non-skippable duplicated vertices of the plurality of duplicated vertices; The apparatus of claim 13 configured to:

15. The processing circuitry includes: based on one of: (i) the number of the plurality of overlapping vertices being greater than or equal to a first threshold; (ii) the number of the vertices in the mesh being greater than or equal to a second threshold; and (iii) a ratio of the number of the plurality of overlapping vertices to the number of the vertices in the mesh being greater than or equal to a third threshold. generating coding information indicating (i) the number of non-skippable overlapping vertices among the plurality of overlapping vertices, and (ii) the vertex index for each of the non-skippable overlapping vertices among the plurality of overlapping vertices; The apparatus of claim 14 configured to:

16. the number of non-skippable overlapping vertices is coded based on one of a variable length coding, a fixed length coding, a (b+1)-bit fixed length code, and a fixed length code having a code length less than (b+1) bits, where b is a positive integer; the vertex index of each of the non-skippable overlapped vertices of the plurality of overlapped vertices is coded based on one of differential coding, the variable length coding, Exp-Golomb coding, the (b+1) bit fixed length code, and a fixed length code having a code length smaller than the (b+1) bit.

16. The apparatus of claim 15.

17. The processing circuitry includes: determining the estimated errors for the vertices in the first group of vertices in the updated mesh; determining an average neighborhood estimation error for a plurality of neighbor vertices associated with each of the vertices in the first vertex group, each of the plurality of neighborhood estimation errors indicative of a difference between one of the plurality of neighbor vertices of the respective vertex in the first vertex group and a reference vertex in the reference frame corresponding to the one of the plurality of neighbor vertices of the respective vertex in the first vertex group; determining a first cost value based on the estimation error of the vertices in the first group of vertices, and determining a second cost value based on the average neighborhood estimation error associated with the vertices in the first group of vertices; determining the prediction mode for the first vertex group based on a comparison of the first cost value and the second cost value. The apparatus of claim 12 configured to:

18. The processing circuitry includes: determining a first neighbor estimation error associated with a first one of the vertices in the first vertex group, the first neighbor estimation error indicative of a difference between a first neighbor of the first vertices and a reference vertex in the reference frame that corresponds to the first neighbor of the first vertices; determining a second neighborhood estimation error associated with the first one of the vertices in the first vertex group, the second neighborhood estimation error indicative of a difference between a second neighbor of the first vertices and a reference vertex in the reference frame that corresponds to the second neighbor of the first vertices; determining the average neighborhood estimation error of the first neighborhood estimation error and the second neighborhood estimation error associated with the first one of the vertices in the first group of vertices; The apparatus of claim 17 configured to:

19. The processing circuitry includes: determining the first cost value as a sum of the estimated errors of the vertices in the first group of vertices; 20. The apparatus of claim 18, configured to:

20. The processing circuitry includes: determining an estimated difference between the estimated error for each of the vertices in the first group of vertices and the average neighborhood estimated error associated with the corresponding vertex in the first group of vertices; determining the second cost value as a sum of the estimated differences associated with the vertices in the first group of vertices; 20. The apparatus of claim 19, configured to:

Images