MESH PROCESSING METHOD, APPARATUS AND PROGRAM

Abstract

In the method, a base mesh is generated from a downsampled input mesh in a current frame, the base mesh including a plurality of vertices. A prediction mode to be applied to the base mesh is determined. The prediction mode is an inter prediction mode or an intra prediction mode. Based on the prediction mode being determined as an intra prediction mode, duplicated vertices of the plurality of vertices in the base mesh are merged to generate a subset of the plurality of vertices. Based on the determined intra prediction mode, at least the subset of the plurality of vertices is encoded to generate prediction information for at least the subset of the plurality of vertices.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Provisional Application No. 63 / 408,268, entitled "Duplicate Vertices Merging in Mesh Compression," filed on September 20, 2022, which claims the benefit of priority to U.S. Provisional Application No. 18 / 208,138, entitled "DUPLICATE VERTICES MERGING IN MESH COMPRESSION," filed on June 9, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] The present disclosure includes embodiments that relate to mesh processing. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Advances in three-dimensional (3D) capture, modeling, and rendering have facilitated the ubiquitous presence of 3D content across a variety of platforms and devices. Today, it is possible to capture a baby's first steps on one continent while that baby's grandparents can see (and potentially interact with) the child on another continent and enjoy a fully immersive experience with that child. To achieve such realism, models have become increasingly sophisticated, and a significant amount of data is tied to the creation and consumption of those models. 3D meshes are widely used to represent such immersive content. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide a method and apparatus for mesh processing. In some examples, the apparatus for mesh processing includes a processing circuit.

[0006] According to one aspect of the present disclosure, a method of mesh processing in a video decoder is provided. In the method, a base mesh is generated from a downsampled input mesh in a current frame, the base mesh including a plurality of vertices. A prediction mode to be applied to the base mesh is determined. The prediction mode is an inter prediction mode or an intra prediction mode. Based on the prediction mode being determined as an intra prediction mode, duplicated vertices of the plurality of vertices in the base mesh are merged to generate a subset of the plurality of vertices. Based on the determined intra prediction mode, at least the subset of the plurality of vertices is encoded, and prediction information for at least the subset of the plurality of vertices is generated.

[0007] In one example, duplicate vertices in the base mesh are merged based on the prediction mode being determined as an intra prediction mode. In one example, duplicate vertices in the base mesh are not merged based on the prediction mode being determined as an inter prediction mode.

[0008] In one example, each of the multiple duplicated vertices is a vertex of the base mesh that shares the same reference vertex in the reference frame with another vertex in the base mesh.

[0009] In one embodiment, merging duplicate vertices is included in determining the prediction mode.

[0010] In one embodiment, merging duplicate vertices is included in encoding at least a subset of the plurality of vertices.

[0011] In some embodiments, the vertices of the base mesh are encoded based on the prediction mode being determined as an inter prediction mode. To encode the vertices of the base mesh, a reference vertex in a reference frame is determined for each of the vertices. Inter prediction information is generated for each of the vertices based on the corresponding reference vertex of each vertex in the reference frame.

[0012] In some embodiments, the reference vertex of each of the multiple vertices is a collocated vertex in the reference frame of the respective vertex, and the collocated vertex is located at the same relative position in the reference frame as the respective vertex of the base mesh in the current frame.

[0013] According to another aspect of the present disclosure, a method of mesh processing performed in an encoder is provided. In the method, a base mesh is generated from a downsampled input mesh in a current frame, the base mesh including a plurality of vertices. Duplicate vertices of the plurality of vertices of the base mesh are merged to generate a merged base mesh including a subset of the plurality of vertices. Index information identifying the merged duplicated vertices is generated. A prediction mode to be applied to the base mesh is determined based on the merged base mesh and the generated index information. The prediction mode is an inter prediction mode or an intra prediction mode. Based on the determined prediction mode being an inter prediction mode, inter prediction information of the base mesh is generated according to the generated index information. Based on the determined intra prediction mode being an intra prediction mode, intra prediction information of the base mesh is generated.

[0014] In one example, to determine a prediction mode, the merged overlap vertices of the multiple vertices are de-merged based on index information of the merged overlap vertices. A reference vertex in a reference frame is determined for each of the multiple vertices of the base mesh in a current frame, the reference frame being different from the current frame. A first prediction difference for each of the multiple vertices of the base mesh is determined. The first prediction difference for each of the multiple vertices indicates a difference between a reference vertex in the reference frame and each of the multiple vertices of the base mesh. A second prediction difference for each of the multiple vertices of the base mesh is determined. The second prediction difference for each of the multiple vertices indicates a difference between one or more neighboring vertices in the current frame of the each of the multiple vertices and the each of the multiple vertices.

[0015] In one example, to determine a prediction mode, a first cost value is determined as a sum of first prediction differences associated with a plurality of vertices, a second cost value is determined as a sum of second prediction differences associated with a plurality of vertices, and the prediction mode is determined as (i) an inter prediction mode based on the first cost value being less than or equal to the second cost value, and (ii) an intra prediction mode based on the second cost value being less than the first cost value.

[0016] In some embodiments, a collocated vertex in the reference frame is determined as a reference vertex for each of the multiple vertices, and the collocated vertices are located at the same relative positions in the reference frame as each vertex of the base mesh in the current frame.

[0017] In one embodiment, based on the prediction mode being determined as an inter prediction mode, inter prediction information is generated for each of a subset of the multiple vertices based on a corresponding reference vertex of each index in a reference frame.

[0018] In one embodiment, based on the prediction mode being determined as an intra prediction mode, intra prediction information is generated for each of a subset of the multiple vertices based on one or more neighboring vertices of each vertex in the current frame.

[0019] According to another aspect of the present disclosure, an apparatus is provided, the apparatus including a processing circuit, the processing circuit being configured to perform any of the described methods for mesh processing.

[0020] Aspects of the present disclosure also provide a non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to perform any of the described methods for mesh processing.

[0021] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a schematic diagram of an example block diagram of a communication system (100). [Diagram 2] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Diagram 3] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 4] FIG. 2 is a schematic diagram of an example encoding process according to some embodiments of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of a pre-processing stage according to some embodiments of the present disclosure. [Figure 6] 1 is a first exemplary encoding process with vertex merging according to some embodiments of the present disclosure. [Figure 7] 11 is a second exemplary encoding process with vertex merging according to some embodiments of the present disclosure. [Figure 8] 1 is a flowchart outlining a process according to some embodiments of the present disclosure. [Figure 9] 1 is a flowchart outlining another process in accordance with some embodiments of the present disclosure. [Figure 10] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] 1 shows a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application for the disclosed subject matter, video encoders, and video decoders in a streaming environment. The disclosed subject matter may be equally applicable to other image and video enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0024] The video processing system (100) includes a capture subsystem (113) that may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera may create a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples taken by the digital camera. The stream of video pictures (102), shown as a bold line to emphasize the amount of data compared to the encoded video data (104) (or coded video bitstream), may be processed by an electronic device (120) that includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown as a thin line to emphasize the small amount of data compared to the stream of video pictures (102), can be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to obtain copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example, in an electronic device (130). The video decoder (110) decodes the incoming copy of the encoded video data (107) and creates an outgoing stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown).In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0025] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may also include a video encoder (not shown).

[0026] 2 shows an example block diagram of a video decoder (210). The video decoder (210) can be included in an electronic device (230). The electronic device (230) can include a receiver (231). The receiver (231) can include receiving circuitry, such as a network interface circuit. The video decoder (210) can be used in place of the video decoder (110) in the example of FIG. 1.

[0027] The receiver (231) may receive one or more coded video sequences to be decoded by the video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) may receive the encoded video data with other data, e.g., coded audio data and / or auxiliary data streams, which may be transferred to each using an entity (not shown). The receiver (231) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). In other cases, the buffer memory (215) may be external to the video decoder (210) (not shown). In still other cases, there may be a buffer memory (not shown) external to the video decoder (210), e.g., to combat network jitter, and another buffer memory (215) internal to the video decoder (210), e.g., to handle playback timing. When the receiver (231) receives data from a storage / forwarding device with sufficient bandwidth and controllability, or from an asynchronous network, the buffer memory (215) may be unnecessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary and may be relatively large, may be advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (210).

[0028] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (210) and potential information for controlling a rendering device such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230) as shown in FIG. 2. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (220) can extract a set of subgroup parameters for at least one of a subgroup of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroups can include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs), etc. The parser (220) can also extract coded video sequence information such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0029] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to produce symbols (221).

[0030] The reconstruction of the symbols (221) may involve a number of different units, depending on the type of coded video picture or part thereof (e.g., inter-picture and intra-picture, inter-block and intra-block), and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0031] In addition to the functional blocks already mentioned, the video decoder (210) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0032] The first unit is a scalar / inverse transform unit (251), which receives quantized transform coefficients from the parser (220) as well as control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (221). The scalar / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0033] In some cases, the output samples of the scaler / inverse transform unit (251) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) may add the prediction information generated by the intra-prediction unit (252) to the output sample information from the scaler / inverse transform unit (251) on a sample-by-sample basis.

[0034] In other cases, the output samples of the scalar / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensation prediction unit (253) may access the reference picture memory (257) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (221) related to the block, these samples may be added by the aggregator (255) to the output of the scalar / inverse transform unit (251) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion compensation prediction unit (253) fetches the prediction samples may be controlled by motion vectors available to the motion compensation prediction unit (253) in the form of symbols (221), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0035] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques that are controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of the coded picture or coded video sequence, and may also be responsive to previously reconstructed, loop filtered sample values.

[0036] The output of the loop filter unit (256) may be a sample stream that can be output to a rendering device (212) and also stored in a reference picture memory (257) for use in future inter-picture prediction.

[0037] Once fully reconstructed, a particular coded picture can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (220)) as a reference picture, the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0038] The video decoder (210) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. Also, a requirement for compliance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by metadata for HRD buffer management and Hypothetical Reference Decoder (HRD) specifications signaled in the coded video sequence.

[0039] In one embodiment, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0040] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the example video encoder (103) of FIG.

[0041] The video encoder (303) can receive video samples from a video source (301) (which in the example of FIG. 3 is not part of the electronic device (320)) that can capture video images to be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0042] The video source (301) may provide a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that give motion when viewed in succession. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc., in use. The following discussion focuses on samples.

[0043] According to one embodiment, the video encoder (303) can code and compress pictures of a source video sequence into a coded video sequence (343) in real-time or under any other time constraint as needed. Enforcing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units as described below. For clarity, coupling is not shown. Parameters set by the controller (350) can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) can be configured with other appropriate functions for the video encoder (303) optimized for a particular system design.

[0044] In some embodiments, the video encoder (303) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) built into the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a manner similar to that which a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Since decoding of the symbol stream results in bit-exact results independent of the decoder location (local or remote), the content in the reference picture memory (334) is also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" the exact same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0045] The operation of the "local" decoder (333) may be the same as that of a "remote" decoder, such as the video decoder (210) already described in detail in connection with Figure 2. However, with brief reference also to Figure 2, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215), and the parser (220), may not be fully implemented in the local decoder (333).

[0046] In one embodiment, the decoder techniques, except for analysis / entropy decoding, present in the decoder are present in the corresponding encoder in the same or substantially the same functional form. Thus, the disclosed subject matter focuses on the decoder operation. A description of the encoder techniques can be omitted since the encoder techniques are the inverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0047] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0048] The local video decoder (333) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (330). The operation of the coding engine (332) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 3), the reconstructed video sequence may usually be a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that may be performed on the reference pictures by the video decoder and store the reconstructed reference pictures in the reference picture memory (334). In this way, the video encoder (303) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end video decoder (without transmission errors).

[0049] The predictor (335) may perform a prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (335) may operate on a sample block-by-sample block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0050] The controller (350) can manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0051] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0052] The transmitter (340) may buffer the coded video sequence produced by the entropy coder (345) for transmission over a communication channel (360), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0053] The controller (350) can manage the operation of the video encoder (303). During coding, the controller (350) can assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0054] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0055] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0056] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0057] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 blocks) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture of the block. For example, blocks of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction with reference to one previously coded reference picture or via temporal prediction. Blocks of a B picture may be predictively coded via spatial prediction with reference to one or two previously coded reference pictures or via temporal prediction.

[0058] The video encoder (303) may perform coding operations in accordance with a given video encoding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (303) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video encoding technique or standard being used.

[0059] In one embodiment, the transmitter (340) can transmit additional data along with the encoded video. The source coder (330) can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0060] A video may be captured in time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0061] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are before the decoding order of the current picture in the video (but may be in the past and future in display order, respectively). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0062] Furthermore, to improve coding efficiency, merge mode techniques can be used for inter-picture prediction.

[0063] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks, such as polygonal or triangular blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0064] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0065] The present disclosure includes embodiments that relate to methods and systems for mesh compression using duplicate vertex merging.

[0066] A mesh may include a number of polygons that represent the surface of a volumetric object. Each polygon of a mesh may be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information about how the vertices are connected, which may be referred to as connectivity information. In some embodiments, vertex attributes such as color, normal, etc. may be associated with the mesh vertices. Attributes (or vertex attributes) may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a two-dimensional (2D) attribute map. Such mapping may be described by a set of parametric coordinates, usually called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute maps may be used to store high-resolution attribute information such as texture, normal, displacement, etc. Such information may be used for various purposes, such as texture mapping and shading.

[0067] Dynamic mesh sequences may require a large amount of data since dynamic meshes may contain a significant amount of information that changes over time. Therefore, efficient compression techniques are required to store and transmit such content. Mesh compression standards such as IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with always-on, time-varying geometry, and vertex attributes. However, these standards may not consider time-varying attribute maps and connectivity information. DCC (Digital Content Creation) tools typically generate such dynamic meshes. However, generating always-on dynamic meshes, especially under real-time constraints, may be challenging for volumetric acquisition techniques. This kind of content (e.g., always-on dynamic meshes) may not be supported by existing standards. MPEG plans to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. The new mesh compression standard targets lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, augmented reality (AR), and virtual reality (VR). Features such as random access and scalable / progressive coding are also possible.

[0068] Figure 4 shows an encoding process (400) based on a relevant video codec such as MPEG V-Mesh™ v 1.0. As shown in Figure 4, the encoding process (400) can include a pre-processing stage (400A) configured to generate a base mesh and a displacement vector according to an input mesh, a mode decision stage (400B) configured to determine whether inter-frame prediction or intra-frame prediction is applied in the encoding process (400), and an encoding stage (400C) configured to encode the base mesh, the displacement vector, and texture information of the base mesh.

[0069] With further reference to FIG. 4, the pre-processing stage (400A) may include a mesh thinning process (402), an atlas parameterization process (404), and a subdivision surface fitting process (406). The mesh thinning process (402) is configured to downsample vertices of an input mesh M(i) to generate a thinned mesh dm(i) including a plurality of thinned (or downsampled) vertices. The atlas parameterization process (404) is configured to map the thinned mesh dm(i) to a planar region, such as a UV atlas (or UV map), to generate a re-parameterized mesh pm(i). In one example, the atlas parameterization may be performed based on a video processing tool, such as a UVAtlas tool. The subdivision surface fitting process (406) is configured to take the re-parameterized mesh pm(i) and the input mesh M(i) as inputs and generate a base mesh m(i) together with a set of displacements d(i). In an exemplary subdivision surface fitting process, pm(i) is subdivided using iterative interpolation. The iterative interpolation involves inserting, at each iteration, a new point in the middle of each edge of the reparameterized mesh pm(i). The displacement field d(i) is computed by determining, for each vertex of the subdivided mesh, the closest point on the surface of the original mesh M(i).

[0070] FIG. 5 illustrates an exemplary pre-processing stage (500) for generating a base mesh and displacement vectors. As shown in FIG. 5, an input 2D curve (represented by a 2D polyline) of an input mesh can be referred to as an "original" curve (502). The original curve (502) can be downsampled to generate a base curve / polyline called a "thinned" curve (504). A subdivision scheme can then be applied to the thinned polyline (504) to generate a "subdivision" curve (506). In one example, the subdivision scheme can be an iterative interpolation scheme. The iterative interpolation scheme can include inserting a new point in the middle of each edge of the polyline (or thinned curve) at each iteration (504). For example, a point (510) can be inserted at an edge (508) of the thinned curve (504). Additionally, a point (522) can be added between point (512) and point (510), and a point (516) can be added between point (510) and point (514). The subdivided polyline (506) is then deformed to generate a displacement curve (518). The displacement curve (518) is a better approximation of the original curve (502) and can serve as an edge of the base mesh. More precisely, a displacement vector (e.g., (520)) is calculated for each vertex (e.g., (510)) of the subdivided curve (506) such that the shape of the displacement curve (518) is as close as possible to the shape of the original curve (502). The main advantage of the subdivided curve (506) is that it has a subdivision structure that allows for more efficient compression while providing a faithful approximation of the original curve (502).

[0071] The mode decision step (400B) can compare the cost of the intra-coding mode with the cost of the inter-coding mode, and determine the coding mode for each frame of the base mesh based on which of the cost of the intra-coding mode and the cost of the inter-coding mode is smaller.

[0072] The encoding stage (400C) may include base mesh coding (408), displacement coding (410), and texture coding (412). The base mesh coding (408) is configured to encode geometric information of a base mesh m(i) associated with a current frame. In intra-encoding, the base mesh m(i) may be first quantized (e.g., using uniform quantization) and then encoded using a static mesh encoder. In inter-encoding, a reconstructed quantized reference base mesh m′(j) associated with a reference frame with index j is used to predict a base mesh m(i) associated with a current frame with index i. The displacement coding (410) is configured to encode a set of displacement vectors associated with the subdivided mesh vertices, referred to as the displacement field d(i). The texture coding (412) is configured to encode attribute information of the base mesh. The attribute information may include texture, normal, color, etc. The attribute information may be encoded based on an appropriate codec, such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC).

[0073] FIG. 6 illustrates an encoding process (600) based on a duplicated vertex merging technique for improving base mesh coding efficiency. As shown in FIG. 6, the encoding process (600) may include a vertex merging step (or base mesh vertex merging step) (602) in the pre-processing stage (600A). In one embodiment, a duplicated vertex may be a vertex of a base mesh that shares the same reference vertex in a reference frame with another vertex in the base mesh. In one embodiment, a vertex of the base mesh and a reference vertex of a vertex in the base mesh are collocated such that the reference vertex is located at the same relative position in the reference frame as the vertex of the base mesh in the current frame. In the vertex merging step (602), a duplicated vertex may be merged with a vertex corresponding to the duplicated vertex.

[0074] However, the coding efficiency of the encoding process with vertex merging may not be as efficient as the encoding process without vertex merging. For example, in the encoding process with vertex merging (600), an intra-coding mode may always be selected or may be more likely to be selected for each frame during the mode decision stage. One reason is that the overlapping vertex merging can reduce the number of vertices, and two frames (e.g., a current frame and a reference frame) may not have a one-to-one vertex correspondence (or correlation). The inter-coding mode requires a one-to-one vertex correspondence between the current frame and the reference frame. Therefore, because there is no one-to-one vertex correspondence, the frame selects an intra-coding mode instead of an inter-coding mode. Thus, vertex merging may result in coding loss because an inter-prediction mode may not be selected.

[0075] In this disclosure, a method and system for mesh compression using overlapping vertex merging is provided to address coding loss due to vertex merging. The proposed methods may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium.

[0076] In this disclosure, duplicated vertices may be merged after mode decision. Thus, duplicated vertices may be merged based on the prediction mode (e.g., inter-coding mode or intra-coding mode) determined at the mode decision stage (e.g., (400B)). In some embodiments, duplicated vertex merging may be applied to frames under intra-coding modes.

[0077] In one embodiment, the mode decision and overlapping vertex merging can be located in the second stage (or mode decision stage) of the encoding process. For example, as shown in FIG. 7, the encoding process (700) can include a first stage (or pre-processing stage) (700A), a second stage (700B), and a third stage (or encoding stage) (700C). The second stage (700B) can include a mode decision step (704) and a vertex merging step (or intra-frame vertex merging) (702). The third stage (or encoding stage) (700C) can include base mesh coding (706), displacement coding (708), and texture coding (710). The mode decision step (704) is configured to determine whether an inter-coding mode or an intra-coding mode is applied to a frame associated with the base mesh. According to the encoding process (700), after the coding mode is determined in the mode decision step (704), if the coding mode of the frame is determined as an intra-coding mode, then overlap vertex merging may be applied to the base mesh, and if the coding mode of the frame is determined as an inter-coding mode, then overlap vertex merging may not be applied to the frame.

[0078] In one embodiment, the overlap vertex merging may be placed in a third stage, such as the encoding stage (700C). In one example, the overlap vertex merging may be placed before the base mesh coding (706). Thus, if the coding mode of the frame of the base mesh is determined as an intra-coding mode, the overlap vertex merging may be applied to the base mesh to generate a merged base mesh. The merged base mesh may include a subset of the vertices of the base mesh. The vertices in the subset of the vertices may be unique or non-repeated. The merged base mesh may be further encoded in the base mesh coding (706) according to the intra-coding mode. In this way, the base mesh coding (706) is performed on the merged base mesh, and the overlapped vertices are merged (or skipped), thereby reducing the input information for the base mesh coding (706). If the coding mode of the frame is determined as an inter-coding mode, the overlap vertex merging may not be applied to the frame.

[0079] In one embodiment, the overlap vertex merging can be located (or implemented) in the base mesh coding (e.g., (706)) of the encoding stage (e.g., (700C)). Thus, the base mesh coding (e.g., (706)) is applied to the base mesh. Thus, the base mesh coding can include two sub-steps. In the first sub-step, overlap vertices of the base mesh can be identified. For example, a vertex of the base mesh can be determined as an overlap vertex if the vertex of the base mesh and another vertex of the base mesh have the same reference vertex in a reference frame. In the second step, the base mesh can be encoded based on a coding mode determined in the mode determination step (e.g., (704)). For example, during the base mesh coding, if the coding mode of the frame of the base mesh is determined as an intra-coding mode, the overlap vertex merging can be applied to the base mesh to generate a merged base mesh including a subset of the vertices of the base mesh. The subset of the vertices can be further coded based on a static mesh codec. If the coding mode of the frame of the base mesh is determined as an inter-coding mode, overlapped vertex merging may not be applied and motion field coding may be applied to the base mesh. Compared with base mesh coding in which overlapped vertex merging is placed before base mesh coding, base mesh coding in which overlapped vertex merging is placed inside the encoding stage receives more input information because overlapped vertices are not merged before base mesh coding.

[0080] In one embodiment, overlapped vertex merging can be performed before mode decision. In one example, overlapped vertex merging can be placed in the first stage before mode decision, which can be shown in FIG. 6. As shown in FIG. 6, when overlapped vertices are merged in the base mesh vertex merging step (602) of the pre-processing stage (600A) to generate a merged base mesh, information of the merged vertices can be signaled and sent to the mode decision stage (600B). In some examples, the information of the merged vertices can include indexes. Thus, in the mode decision stage (600B), the indexes of the merged vertices can be identified. Based on the indexes of the merged vertices, the merged vertices can be unmerged. In this way, the number of vertices in the current frame is not reduced, and a one-to-one vertex correspondence (or correlation) can be established across frames (e.g., between the current frame and the reference frame). Furthermore, the cost (or predicted cost) of the intra-coding mode and the cost of the inter-coding mode can be compared to select the intra-coding mode or the inter-coding mode. When a prediction mode (or coding mode) is determined in the mode decision step (600B), whether the base mesh or the merged base mesh is encoded in the base mesh coding (602) is determined based on the determined prediction mode. For example, when an intra prediction mode (or an intra coding mode) is determined in the mode decision step (600B), the base mesh coding (602) may encode a merged base mesh in which overlapping vertices are merged. When an inter prediction mode (or an inter coding mode) is determined in the mode decision step (600B), the base mesh coding (602) may encode a base mesh.

[0081] FIG. 8 shows a flow chart outlining a first decoding process (800), and FIG. 9 shows a flow chart outlining a second decoding process (900), according to an embodiment of the present disclosure. Process (800) and process (900) may be used in a decoder, such as a video decoder. In various embodiments, process (800) and process (900) may be performed by a processing circuit, such as a processing circuit performing the functions of a video decoder (110), a processing circuit performing the functions of a video decoder (210), or the like. In some embodiments, process (800) and process (900) are implemented with software instructions, such that the processing circuit performs process (800) and process (900), respectively, when the processing circuit executes the software instructions.

[0082] In one example, the process (800) starts at (S801) and proceeds to (S810).

[0083] At (S810), a base mesh is generated from the downsampled input mesh in the current frame, where the base mesh includes a plurality of vertices.

[0084] At (S820), a prediction mode to be applied to the base mesh is determined. The prediction mode is an inter prediction mode or an intra prediction mode.

[0085] In (S830), based on the prediction mode being determined as an intra prediction mode, duplicated vertices among the multiple vertices in the base mesh are merged to generate a subset of the multiple vertices.

[0086] At (S840), at least a subset of the plurality of vertices is encoded based on the determined intra-prediction mode, and prediction information for at least the subset of the plurality of vertices is generated.

[0087] In one example, duplicate vertices in the base mesh are merged based on the prediction mode being determined as an intra prediction mode. In one example, duplicate vertices in the base mesh are not merged based on the prediction mode being determined as an inter prediction mode.

[0088] In one example, each of the multiple duplicated vertices is a vertex of the base mesh that shares the same reference vertex in the reference frame with another vertex in the base mesh.

[0089] In one embodiment, merging duplicate vertices is included in determining the prediction mode.

[0090] In one embodiment, merging duplicate vertices is included in encoding at least a subset of the plurality of vertices.

[0091] In some embodiments, the vertices of the base mesh are encoded based on the prediction mode being determined as an inter prediction mode. To encode the vertices of the base mesh, a reference vertex in a reference frame is determined for each of the vertices. Inter prediction information is generated for each of the vertices based on the corresponding reference vertex of each vertex in the reference frame.

[0092] In some embodiments, the reference vertex of each of the multiple vertices is a collocated vertex in the reference frame of the respective vertex, and the collocated vertex is located at the same relative position in the reference frame as the respective vertex of the base mesh in the current frame.

[0093] The process then proceeds to (S899) and ends.

[0094] The process (800) may be adapted as appropriate. Steps of the process (800) may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0095] In one example, the process (900) starts at (S901) and proceeds to (S910).

[0096] At (S910), a base mesh is generated from the downsampled input mesh in the current frame, where the base mesh includes a plurality of vertices.

[0097] At (S920), duplicate vertices of the plurality of vertices of the base mesh are merged to generate a merged base mesh including a subset of the plurality of vertices.

[0098] At (S930), index information is generated that identifies the merged duplicate vertices.

[0099] At (S940), a prediction mode to be applied to the base mesh is determined based on the merged base mesh and the generated index information, where the prediction mode is an inter prediction mode or an intra prediction mode.

[0100] At (S950), based on the determined prediction mode being an inter prediction mode, inter prediction information of the base mesh is generated according to the generated index information.

[0101] At (S960), intra prediction information of the base mesh is generated based on the determined prediction mode being an intra prediction mode.

[0102] In one example, to determine a prediction mode, the merged overlap vertices of the multiple vertices are de-merged based on index information of the merged overlap vertices. A reference vertex in a reference frame is determined for each of the multiple vertices of the base mesh in a current frame, the reference frame being different from the current frame. A first prediction difference for each of the multiple vertices of the base mesh is determined. The first prediction difference for each of the multiple vertices indicates a difference between a reference vertex in the reference frame and each of the multiple vertices of the base mesh. A second prediction difference for each of the multiple vertices of the base mesh is determined. The second prediction difference for each of the multiple vertices indicates a difference between one or more neighboring vertices in the current frame of the each of the multiple vertices and the each of the multiple vertices.

[0103] In one example, to determine a prediction mode, a first cost value is determined as a sum of first prediction differences associated with a plurality of vertices, a second cost value is determined as a sum of second prediction differences associated with a plurality of vertices, and the prediction mode is determined as (i) an inter prediction mode based on the first cost value being less than or equal to the second cost value, and (ii) an intra prediction mode based on the second cost value being less than the first cost value.

[0104] In some embodiments, a collocated vertex in the reference frame is determined as a reference vertex for each of the multiple vertices, and the collocated vertices are located at the same relative positions in the reference frame as each vertex of the base mesh in the current frame.

[0105] In one embodiment, based on the prediction mode being determined as an inter prediction mode, inter prediction information is generated for each of a subset of the multiple vertices based on a corresponding reference vertex of each index in a reference frame.

[0106] In one embodiment, based on the prediction mode being determined as an intra prediction mode, intra prediction information is generated for each of a subset of the multiple vertices based on one or more neighboring vertices of each vertex in the current frame.

[0107] Thereafter, the process proceeds to (S999) and ends.

[0108] The process (900) may be adapted as appropriate. Steps of the process (900) may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0109] The techniques described above can be implemented as computer software physically stored on one or more computer readable media using computer readable instructions. For example, Figure 10 illustrates a computer system (1000) suitable for implementing certain embodiments of the disclosed subject matter.

[0110] Computer software can be coded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to produce code including instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0111] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0112] 10 of the computer system (1000) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having a dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (1000).

[0113] The computer system (1000) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), video (2D video, 3D video including stereoscopic video, etc.).

[0114] The input human interface devices may include one or more (only one of each is depicted) of a keyboard (1001), a mouse (1002), a trackpad (1003), a touch screen (1010), a data glove (not shown), a joystick (1005), a microphone (1006), a scanner (1007), and a camera (1008).

[0115] The computer system (1000) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1010), data gloves (not shown), or joystick (1005), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (1009), headphones (not shown)), visual output devices (screens (1010), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional hyper-output via means such as stereo output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0116] The computer system (1000) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1020) with media (1021) such as CDs / DVDs, thumb drives (1022), removable hard drives or solid state drives (1023), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices such as security dongles (not shown).

[0117] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other ephemeral signals.

[0118] The computer system (1000) may also include an interface (1054) to one or more communication networks (1055). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (1049) (e.g., a USB port of the computer system (1000)). Others are generally integrated into the core of the computer system (1000) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1000) may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast TV), transmit only unidirectional (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0119] The aforementioned human interface devices, human access storage devices, and network interfaces may be attached to the core (1040) of the computer system (1000).

[0120] The cores (1040) may include one or more central processing units (CPUs) (1041), graphics processing units (GPUs) (1042), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1043), hardware accelerators for specific tasks (1044), graphics adapters (1050), and the like. These devices may be connected via a system bus (1048), along with read only memory (ROM) (1045), random access memory (1046), internal mass storage such as an internal non-user accessible hard drive, SSD, and the like (1047). In some computer systems, the system bus (1048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripherals may be attached directly to the core's system bus (1048) or via a peripheral bus (1049). In one example, a screen (1010) may be connected to the graphics adapter (1050). Peripheral bus architectures include PCI, USB, etc.

[0121] The CPU (1041), GPU (1042), FPGA (1043), and accelerator (1044) can execute certain instructions that, in combination, can constitute the computer code described above. That computer code can be stored in ROM (1045) or RAM (1046). Transient data can also be stored in RAM (1046), while permanent data can be stored, for example, in internal mass storage (1047). Rapid storage and retrieval in any of the memory devices can be made possible by the use of cache memories, which can be closely associated with one or more of the CPU (1041), GPU (1042), mass storage (1047), ROM (1045), RAM (1046), etc.

[0122] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0123] By way of example and not limitation, a computer system (1000) having an architecture, particularly a core (1040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with a particular storage of the core (1040) of a non-transient nature, such as a core internal mass storage (1047) or a ROM (1045). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1040). The computer-readable media can include one or more memory devices or chips, depending on the particular need. The software can cause the core (1040), and in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM (1046) and modifying such data structures according to the processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic embodied in hardwired or otherwise circuitry (e.g., accelerator (1044)), which may operate in place of or in conjunction with software to perform particular processes or particular portions of particular processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0124] The use of "at least one of" or "one of" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C, i.e., at least one of A, B, and C, at least one of A, B, and / or C, at least one of A through C, is intended to include A only, B only, C only, or any combination thereof. Reference to one of A or B and one of A and B is intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the listed elements where applicable, such as when the elements are not mutually exclusive.

[0125] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods not explicitly shown or described herein, but which embody the principles of the disclosure and are therefore within the spirit and scope of the disclosure. [Explanation of symbols]

[0126] 100 video processing system, 101 video source, 102 stream of video pictures, 103 video encoder, 104 video data, 105 streaming server, 106 client subsystem, 107 incoming copy, 108 client subsystem, 109 copy, 110 video decoder, 111 video picture, 112 display, 113 capture subsystem, 120 electronic device, 130 electronic device, 201 channel, 210 video decoder, 212 rendering device, 215 buffer memory, 220 parser, 221 symbols, 230 electronic device, 231 receiver, 251 inverse transform unit, 252 intra prediction unit, 253 motion compensation prediction unit, 255 aggregator, 256 loop filter unit, 257 reference picture memory, 258 picture buffer, 301 video source, 303 video encoder, 320 electronic device, 330 source coder, 332 coding engine, 333 decoder, 334 reference picture memory, 335 predictor, 340 transmitter, 343 video sequence, 345 entropy coder, 350 controller, 360 communication channel, 400 encoding process, 400A pre-processing stage, 400B mode decision stage, 400C encoding stage, 402 mesh thinning process, 404 atlas parameterization process, 406 subdivision surface fitting process, 408 base mesh coding, 410 displacement coding, 412 texture coding, 500 pre-processing stage, 502 curve, 504 polyline, 506 polyline, 508 edge, 518 displacement curve, 600 encoding process, 600A pre-processing stage, 600B mode decision stage, 602 vertex merging step, 700 encoding process, 700A pre-processing stage, 700C Encoding stage, 702, intra-frame vertex merging, 704, mode decision step, 706, base mesh coding, 708, displacement coding, 710, texture coding, 800, decoding process, 900, decoding process, 1000, computer system, 1001, keyboard, 1002, mouse, 1003track pad, 1005 joystick, 1006 microphone, 1007 scanner, 1008 camera, 1009 speaker, 1010 touch screen, 1021 media, 1022 thumb drive, 1023 solid state drive, 1040 core, 1041 CPU, 1042 GPU, 1043 field programmable gate area, 1044 hardware accelerator, 1045 read only memory ROM, 1046 random access memory, 1047 mass storage, 1048 system bus, 1049 peripheral bus, 1050 graphics adapter, 1054 interface, 1055 communication network

Claims

1. A method of mesh processing performed in an encoder, comprising the steps of: generating a base mesh from the downsampled input mesh in a current frame, the base mesh including a plurality of vertices; determining a prediction mode to be applied to the base mesh, the prediction mode being an inter prediction mode or an intra prediction mode; based on the prediction mode being determined as the intra prediction mode, merging duplicate vertices among the plurality of vertices in the base mesh to generate a subset of the plurality of vertices; encoding at least the subset of the plurality of vertices based on the determined intra-prediction mode to generate prediction information for at least the subset of the plurality of vertices; A method comprising:

2. the overlapping vertices in the base mesh are merged based on the prediction mode being determined as the intra prediction mode; the duplicated vertices in the base mesh are not merged based on the prediction mode being determined as being the inter prediction mode. The method of claim 1.

3. The method of claim 1 , wherein each of the duplicated vertices is a vertex of the base mesh that shares a same reference vertex in a reference frame with another vertex in the base mesh.

4. The method of claim 1 , wherein the step of merging the duplicated vertices is included in the step of determining the prediction mode.

5. The method of claim 1 , wherein the step of merging the duplicated vertices is included in the step of encoding at least the subset of the plurality of vertices.

6. Based on the prediction mode being determined to be the inter prediction mode, encoding the vertices of the base mesh; Further comprising: The step of encoding the vertices of the base mesh comprises: determining a reference vertex in a reference frame for each of the plurality of vertices; generating inter prediction information for each of the plurality of vertices based on the corresponding reference vertex of the respective vertex in the reference frame; Including, The method of claim 1.

7. 7. The method of claim 6, wherein the reference vertex for each of the plurality of vertices is a collocated vertex in the reference frame of the respective vertex, the collocated vertex being located at the same relative position in the reference frame as the respective vertex of the base mesh in the current frame.

8. A method of mesh processing performed in an encoder, comprising the steps of: generating a base mesh from the downsampled input mesh in a current frame, the base mesh including a plurality of vertices; merging duplicate vertices of the plurality of vertices of the base mesh to generate a merged base mesh comprising a subset of the plurality of vertices; generating index information identifying the merged overlapping vertices; determining a prediction mode to be applied to the base mesh based on the merged base mesh and the generated index information, the prediction mode being an inter prediction mode or an intra prediction mode; generating inter prediction information of the base mesh according to the generated index information based on the determined prediction mode being the inter prediction mode; generating intra prediction information of the base mesh based on the determined prediction mode being the intra prediction mode; A method comprising:

9. The step of determining the prediction mode comprises: unmerging the merged overlapping vertices of the plurality of vertices based on the index information of the merged overlapping vertices; determining, for each of the vertices of the base mesh in the current frame, a reference vertex in a reference frame, the reference frame being different from the current frame; determining a first prediction difference for each of the plurality of vertices of the base mesh, the first prediction difference for each of the plurality of vertices indicative of a difference between the reference vertex in the reference frame and the respective vertex of the plurality of vertices of the base mesh; determining a second prediction difference for each of the plurality of vertices of the base mesh, the second prediction difference for each of the plurality of vertices indicating a difference between the respective vertex of the plurality of vertices and one or more neighboring vertices in the current frame of the respective vertex of the plurality of vertices; The method of claim 8 , comprising:

10. The step of determining the prediction mode comprises: determining a first cost value as a sum of the first prediction differences associated with the plurality of vertices; determining a second cost value as a sum of the second prediction differences associated with the plurality of vertices; determining the prediction mode as (i) the inter prediction mode based on the first cost value being less than or equal to the second cost value, and (ii) the intra prediction mode based on the second cost value being less than the first cost value; The method of claim 9 further comprising:

11. The step of determining the reference vertex comprises: determining, for each of the plurality of vertices, a collocated vertex in the reference frame as the reference vertex, the collocated vertex being located at the same relative position in the reference frame as the respective vertex of the base mesh in the current frame; The method of claim 9 further comprising:

12. The step of generating the inter prediction information for the base mesh comprises: generating the inter prediction information for each of the subset of vertices based on a corresponding reference vertex of the respective index in the reference frame based on the prediction mode being determined as the inter prediction mode; The method of claim 9 further comprising:

13. The step of generating the intra prediction information for the base mesh comprises: generating the intra prediction information for each of the subset of vertices based on one or more neighboring vertices of the respective vertex in the current frame based on the prediction mode being determined as the intra prediction mode; The method of claim 9 further comprising:

14. 1. An apparatus for mesh processing comprising: Generate a base mesh including a number of vertices from the downsampled input mesh in the current frame; determining a prediction mode to be applied to the base mesh, the prediction mode being an inter prediction mode or an intra prediction mode; merging duplicate vertices of the plurality of vertices in the base mesh to generate a subset of the plurality of vertices based on the prediction mode being determined as the intra prediction mode; encoding at least the subset of the plurality of vertices based on the determined intra-prediction mode to generate prediction information for at least the subset of the plurality of vertices. A processing circuit configured to An apparatus comprising:

15. the overlapping vertices in the base mesh are merged based on the prediction mode being determined as the intra prediction mode; the duplicated vertices in the base mesh are not merged based on the prediction mode being determined as being the inter prediction mode.

15. The apparatus of claim 14.

16. The apparatus of claim 14 , wherein each of the duplicated vertices is a vertex of the base mesh that shares a same reference vertex in a reference frame with another vertex in the base mesh.

17. The apparatus of claim 14 , wherein the step of merging the overlapping vertices is included in the step of determining the prediction mode.

18. The apparatus of claim 14 , wherein the step of merging the duplicated vertices is included in the step of encoding at least the subset of the plurality of vertices.

19. The processing circuitry includes: Based on the prediction mode being determined to be the inter prediction mode, determining a reference vertex in a reference frame for each of the plurality of vertices; generating inter prediction information for each of the plurality of vertices based on the corresponding reference vertex of the respective vertex in the reference frame; The apparatus of claim 14 configured as follows:

20. 20. The apparatus of claim 19, wherein the reference vertex for each of the plurality of vertices is a collocated vertex in the reference frame of the respective vertex, the collocated vertex being located at the same relative position in the reference frame as the respective vertex of the base mesh in the current frame.

Images