Method, apparatus, and computer program for reflection prediction and parallelogram cross-section prediction in polygon mesh compression
Parallelogram cross-section and reflection predictions improve vertex and UV coordinate prediction in polygon mesh compression, reducing residuals and signaling costs for more efficient 3D mesh encoding.
Patent Information
- Application Number
- JP2026525089
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-10-22
- Filing Date
- 2024-10-24
- Publication Date
- 2026-11-19
AI Technical Summary
Existing methods for compressing 3D meshes in video codecs are inefficient, particularly in predicting the position of vertices and UV coordinates, leading to high signaling costs and suboptimal compression efficiency.
Implementing parallelogram cross-section prediction and reflection prediction to accurately predict the position of vertices and UV coordinates in polygon meshes, using context modeling to determine the most efficient prediction method based on previous encoding contexts.
Reduces prediction residuals and signaling costs, enhancing the efficiency of polygon mesh compression by accurately predicting vertex positions and UV coordinates.
Smart Images

Figure 2026539062000001_ABST
Abstract
Description
TECHNICAL FIELD
[0001] Reference This application claims the benefit of priority of U.S. Patent Application No. 18 / 923,663 filed on October 22, 2024, which claims the benefit of priority of U.S. Provisional Application No. 63 / 547,964 filed on November 9, 2023. The entire disclosure of the prior application is hereby incorporated by reference in its entirety into the present specification.
[0002] Technical field The present disclosure generally describes aspects related to mesh processing.
BACKGROUND ART
[0003] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors of the present invention to the extent described in this Background section, as well as aspects of the description that may not qualify as prior art at the time of filing, are not admitted as prior art against the present disclosure, either explicitly or implicitly.
[0004] Image / video compression can help transmit image / video data across various devices, storages and networks with minimal quality degradation. In some examples, video codec techniques may compress video based on spatial and temporal redundancies. In one example, a video codec may use a technique called intra prediction that can compress images based on spatial redundancy. For example, intra prediction may use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec may use a technique called inter prediction that can compress images based on temporal redundancy. For example, inter prediction may use motion compensation to predict samples in the current picture from previously reconstructed pictures. Motion compensation may be indicated by a motion vector (MV).
[0005] Advances in three-dimensional (3D) capture, modeling, and rendering are driving the development of 3D content across various platforms and devices. For example, a baby's first steps might be captured on one continent, allowing grandparents to view (and sometimes interact with) that child on another continent, enjoying a fully immersive experience. To achieve this level of realism, models are becoming more sophisticated, with vast amounts of data linked to their creation and consumption. 3D meshes are widely used to represent such immersive content. [Overview of the Initiative] [Problems that the invention aims to solve]
[0006] Aspects of this disclosure include methods and apparatus for mesh processing. [Means for solving the problem]
[0007] Aspects of the present disclosure include a decoding method for decoding a mesh. The decoding method includes: receiving coded information, which includes syntax elements indicating one of a parallelogram cross-section prediction and one of a reflection prediction for predicting the location of one of the current vertices of a mesh and the current points of a two-dimensional (2D) map associated with the mesh; determining one of the parallelogram cross-section prediction and one of a reflection prediction based on the syntax elements for predicting the location of the one of the current vertices of a mesh and the current points of a 2D map; and using the determined one of the parallelogram cross-section prediction and the reflection prediction, predicting the location of the one of the current vertices of a mesh and the current points of a 2D map.
[0008] Aspects of this disclosure also provide apparatus for mesh decoding. The apparatus for mesh decoding includes processing circuitry configured to implement one of the described methods, which includes a decoding method for mesh processing performed in a decoder.
[0009] In one embodiment, there is a mesh coding method that encodes a mesh. The mesh coding method includes the steps of: determining one of a parallelogram cross-section prediction and a reflection prediction for predicting the position of one of the current vertices of a mesh and the current points of a two-dimensional (2D) map associated with the mesh; using the one of the parallelogram cross-section prediction and the reflection prediction, predicting the position of one of the current vertices of a mesh and the current points of a 2D map; determining a context for encoding the one of the parallelogram cross-section prediction and the reflection prediction based on whether previously encoded positions were encoded using a parallelogram cross-section prediction or a reflection prediction; and encoding a syntax element representing the one of the parallelogram cross-section prediction and the reflection prediction using context modeling with the determined context.
[0010] Aspects of this disclosure also provide apparatus for mesh coding. Apparatus for mesh coding includes processing circuitry configured to implement one of the methods for mesh processing described, which is performed on an encoder.
[0011] In one embodiment, a method for processing a mesh includes the step of processing a bitstream of the mesh according to a format rule. The bitstream includes syntax elements indicating one of a parallelogram cross-section prediction and one of a reflection prediction for predicting the position of either the current vertex of the mesh or the current point of a two-dimensional (2D) map associated with the mesh. The format rule specifies that one of the parallelogram cross-section prediction and one of the reflection prediction for predicting the position of either the current vertex of the mesh or the current point of the 2D map is determined based on the syntax elements, and that the position of either the current vertex of the mesh or the current point of the 2D map is predicted using the determined one of the parallelogram cross-section prediction and the reflection prediction.
[0012] Aspects of this disclosure also provide a non-temporary computer-readable medium that, when executed by a computer, stores instructions causing the computer to perform one of the meshing methods described.
[0013] The technical solutions of this disclosure include embodiments relating to the efficient coding of position and UV values for polygon mesh compression. In polygon mesh compression, the bitstream may include a value bitstream and a connectivity bitstream, and in some examples, the value bitstream may comprise a larger portion of the total bitstream compared to the connectivity bitstream, and thus attribute values can be coded using an efficient method. For example, the value bitstream may include information indicating the position of vertices in a mesh (e.g., a polygon mesh) and / or UV values such as UV coordinates of a 2D map associated with the mesh, and may occupy a large portion of the bitstream. In one example, it is decided whether to use parallelogram cross-section prediction or reflection prediction to predict the position of a vertex in the mesh or the current point in a 2D map (e.g., indicated by UV coordinates). Thus, one of the parallelogram cross-section prediction and the reflection prediction is determined to predict the position. For example, one of the parallelogram cross-section prediction and the reflection prediction may be selected to more accurately select the position. Therefore, when predicting a position using the more accurate prediction method among parallelogram cross-section prediction and reflection prediction, prediction residuals can be reduced, thereby lowering the signaling cost of signaling prediction residuals. Furthermore, the context for encoding either parallelogram cross-section prediction or reflection prediction may be determined based on whether the previously encoded position was encoded using parallelogram cross-section prediction or reflection prediction, and the syntax element indicating either parallelogram cross-section prediction or reflection prediction may be encoded using context modeling with the determined context. Thus, the overhead of signaling which prediction is used can be reduced. [Brief explanation of the drawing]
[0014] Further features, properties and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.
[0015] [Figure 1] FIG. 1 is a schematic diagram showing an example of a block diagram of a video processing system (100).
[0016] [Figure 2] FIG. 2 is a schematic diagram showing an example of a block diagram of a decoder.
[0017] [Figure 3] FIG. 3 is a schematic diagram showing an example of a block diagram of an encoder.
[0018] [Figure 4] FIG. 4 is a diagram illustrating an example of an encoding process for mesh processing based on an associated video codec, in accordance with one aspect of the present disclosure.
[0019] [Figure 5] FIG. 5 is a diagram illustrating an example of a decoding process for mesh processing, in accordance with one aspect of the present disclosure.
[0020] [Figure 6] FIG. 6 is a diagram illustrating an example of the vertex degree of a vertex (611) and the face degree of a face (612) of a polygon mesh (600), in accordance with one aspect of the present disclosure.
[0021] [Figure 7] FIG. 7 is a diagram illustrating an example of across-parallelogram prediction, in accordance with one aspect of the present disclosure.
[0022] [Figure 8] FIG. 8 is a diagram illustrating an example of within-parallelogram prediction, in accordance with one aspect of the present disclosure.
[0023] [Figure 9]This figure shows an example of parallelogram cross-section prediction and reflection prediction according to one aspect of the present disclosure.
[0024] [Figure 10] This figure shows a flowchart outlining the decryption process according to several aspects of this disclosure.
[0025] [Figure 11] This figure shows a flowchart outlining the encoding process according to several aspects of this disclosure.
[0026] [Figure 12] This is a schematic diagram of a computer system in one configuration. [Modes for carrying out the invention]
[0027] Figure 1 is a block diagram of a video processing system (100) in several examples. The video processing system (100) is an example of an application for the disclosed subject, a video encoder and video decoder in a streaming environment. The disclosed subject may also be equally applicable to other image and / or video-enabled applications, such as video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.
[0028] The video processing system (100) may include a capture subsystem (113) which may include a video source (101). The video source (101) may include one or more images captured by a camera and / or generated by a computer. For example, a digital camera may produce a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples taken by the digital camera. The stream of video pictures (102) is drawn as a thick line to emphasize that it is a high amount of data when compared to encoded video data (104) (or coded video bitstream) and may be processed by an electronic device (120) which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the subject matter disclosed, as will be described in more detail below. The encoded video data (104) (or encoded video bitstream) is drawn as a thin line to emphasize its low data volume compared to the stream of video picture (102) and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) may include a video decoder (110) within, for example, an electronic device (130). The video decoder (110) decodes the input copy (107) of the encoded video data to create an output stream of video picture (111) which may be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstream) may be encoded according to a specific video coding / compression standard. Examples of these standards include the ITU-T recommended H.265.For example, a video coding standard under development is informally known as VVC (Versatile Video Coding). The disclosed subject matter may be used in the context of VVC.
[0029] It should be noted that electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may also include a video encoder (not shown).
[0030] Figure 2 shows an example block diagram of a video decoder (210). The video decoder (210) may be included in an electronic device (230). The electronic device (230) may include a receiver (231). The receiver (231) may include a receiving circuit such as a network interface circuit. The video decoder (210) may be used instead of the video decoder (110) in the example in Figure 1.
[0031] The receiver (231) may receive one or more coded video sequences contained in a bitstream that is decoded by, for example, a video decoder (210). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device that stores coded video data. The receiver (231) may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective usage entities (not shown). The receiver (231) can isolate the coded video sequences from other data. To address network jitter, a buffer memory (215) may be coupled between the receiver (231) and an entropy decoder / parser (220) (hereinafter referred to as "Parser (220)"). In one application, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory (215) may be external to the video decoder (210) (not shown). In yet another case, for example to deal with network jitter, a buffer memory (not shown) may exist externally to the video decoder (210), and in addition, another buffer memory (215) may exist internally to the video decoder (210), for example to handle playback timing. When the receiver (231) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (215) may not be necessary or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (215) is required, and it may be relatively large, advantageously adaptive in size, and may be implemented at least partially within an operating system or similar element (not shown) external to the video decoder (210).
[0032] The video decoder (210) may include a parser (220) to reconstruct symbols (221) from the coded video sequence. These symbol categories may include information used to manage the operation of the video decoder (210) and potentially information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen), which is not an integral part of an electronic device (230) but may be coupled to it as shown in Figure 2. The rendering device control information may be in the form of Supplemental Extension Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (220) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-sensitive or context-insensitive arithmetic coding, etc. The parser(220) may extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder, based on at least one parameter corresponding to a group. The subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser(220) may also extract information from the coded video sequence such as transform coefficients, quantization parameter values, motion vectors, etc.
[0033] The parser (220) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (215) in order to generate symbols (221).
[0034] The reconstruction of symbol (221) may involve multiple different units, depending on the type of coded video picture or part thereof (inter and intra picture, inter and intra block, etc.) and other factors. Which units are involved and how they are involved may be controlled by subgroup control information analyzed by parser (220) from the coded video sequence. The flow of such subgroup control information between parser (220) and the following multiple units is not illustrated for clarity.
[0035] In addition to the functional blocks already described, the video decoder (210) can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated. However, for the purpose of illustrating the subject matter to be disclosed, the following conceptual subdivision into functional units is appropriate.
[0036] The first unit is the scaler / inverse unit (251). The scaler / inverse unit (251) may receive control information from the parser (220) as a symbol (221), including which transformation to use, block size, quantization coefficients, quantization scaling matrix, etc., along with the quantized transformation coefficients. The scaler / inverse unit (251) may output a block containing sample values that can be input to the aggregator (255).
[0037] In some cases, the output samples of the scaler / inverse unit (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use predictive information from previously reconstructed pictures but may use predictive information from previously reconstructed portions of the current picture. Such predictive information may be provided by the intrapicture predictive unit (252). In some cases, the intrapicture predictive unit (252) generates a block of the same size and shape as the block being reconstructed, using already reconstructed ambient information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) may, in some cases, add the predictive information generated by the intrapredictive unit (252) to the output sample information provided by the scaler / inverse unit (251) on a sample-by-sample basis.
[0038] In other cases, the output samples of the scaler / inverse unit (251) may be associated with an interconnected and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) and fetch samples to be used for prediction. After the fetched samples have been motion-compensated according to the symbols (221) associated with the block, these samples may be added by the aggregator (255) to the output of the scaler / inverse unit (251) (in this case, called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches prediction samples may be controlled by a motion vector and available to the motion-compensated prediction unit (253) in the form of a symbol (221) which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values fetched from reference picture memory (257) when the precise motion vector of a subsample is used, motion vector prediction mechanisms, etc.
[0039] The output samples of the aggregator (255) may be subjected to various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also respond to metadata obtained during decoding of earlier portions (in decoding order) of the coded picture or coded video sequence, and similarly, to previously reconstructed and loop-filtered sample values.
[0040] The output of the loop filter unit (256) may be a sample stream that is output to the rendering device (212) and can be stored in reference picture memory (257) for use in future interpicture prediction.
[0041] A particular coded picture, once fully reconstructed, may be used as a reference picture for future predictions. For example, if the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and a new current picture buffer may be reallocated before starting the reconstruction of the next coded picture.
[0042] The video decoder (210) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Rec.H.265. The coded video sequence may follow the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence follows both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool usable under that profile. Also, compliance may require that the complexity of the coded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited through the virtual reference decoder (HRD) specification and metadata for HRD buffer management signaled within the coded video sequence.
[0043] In one embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, a temporal, spatial, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction code, etc.
[0044] Figure 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in the electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) may be used instead of the video encoder (103) in the example in Figure 1.
[0045] The video encoder (303) can receive video samples from a video source (301) (which is not part of the electronic device (320) in the example in Figure 3), and the video source can capture video images to be coded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).
[0046] The video source (301) may provide a source video sequence coded by a video encoder (303) in the form of a digital video sample stream, which can be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of separate pictures that give motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, where each pixel may contain one or more samples, depending on the sampling structure, color space, etc., used. The following description will focus on samples.
[0047] In one embodiment, the video encoder (303) can encode and compress the pictures of a source video sequence into a coded video sequence (343) in real time or under any other required time constraints. Achieving an appropriate coding speed is a function of the controller (350). In some embodiments, the controller (350) controls and is functionally coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (350) may include rate control-related parameters (picture skip, quantizer, lambda value of rate distortion optimization technique, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a given system design.
[0048] In some embodiments, the video encoder (303) is configured to operate in a coding loop. In an overly simplified explanation, one example of a coding loop might include a source coder (330) (responsible for creating symbols, such as a symbol stream, based, for example, on the input and reference pictures being coded) and a (local) decoder (333) built into the video encoder (303). The decoder (333) reconstructs the symbols to create sample data in a similar manner to how a (remote) decoder would. The reconstructed sample stream (sample data) is fed into the reference picture memory (334). Since decoding the symbol stream yields bit-accurate results independent of the decoder's location (local or remote), the contents within the reference picture memory (334) are also bit-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values as the reference picture samples that the decoder "sees" when using predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, for example, due to channel errors) is also used in several related technologies.
[0049] The operation of the “local” decoder (333) may be the same as that of a “remote” decoder, such as the video decoder (210), which has already been described in detail above in relation to Figure 2. However, as also briefly referring to Figure 2, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (345) and parser (220) may be reversible, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).
[0050] In one embodiment, the decoder techniques present in the decoder, excluding analysis / entropy decoding, exist in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter disclosed focuses on the decoder operation. Descriptions of encoder techniques may be omitted, as they are the inverse of the more comprehensively described decoder techniques. More detailed descriptions are provided below in specific areas.
[0051] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (332) codes the difference between the pixel blocks of the input picture and the pixel blocks of the reference picture that may be selected as predictive references for the input picture.
[0052] A local video decoder (333) can decode coded video data of a picture that may be designated as a reference picture, based on symbols created by the source coder (330). The operation of the coding engine (332) may, advantageously, be an irreversible process. When coded video data is decoded in a video decoder (not shown in Figure 3), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that may be performed by the video decoder on the reference picture and store the reconstructed reference picture in a reference picture memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture obtained by the far-end video decoder (without transmission errors).
[0053] The predictor (335) may perform a predictive search of the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for sample data (as reference pixel blocks) or for specific metadata such as reference picture motion vectors or block shapes that can function as appropriate predictive references for the new picture. The predictor (335) can operate on a sample block-by-pixel block basis to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).
[0054] The controller (350) may manage the coding operations of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode video data.
[0055] The outputs of all the above functional units may be subject to entropy coding in the entropy coder (345). The entropy coder (345) converts the symbols generated by the various functional units into coded video sequences by applying lossless compression to the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.
[0056] The transmitter (340) may buffer the coded video sequence generated by the entropy coder (345) and prepare it for transmission over the communication channel (360), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) may merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).
[0057] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:
[0058] An intra-picture (I-picture) can be coded and decoded without using any other picture in the sequence as a prediction source. Some video codecs allow different types of intra-pictures, including, for example, independent decoder refresh ("IDR") pictures.
[0059] Predictive pictures (P-pictures) can be coded and decoded using intra-prediction or inter-prediction, with at most one motion vector and reference index used to predict the sample values of each block.
[0060] A bidirectional predictive picture (B-picture) can be coded and decoded using intra-prediction or inter-prediction, with two motion vectors and a reference index to predict the sample values for each block. Similarly, multiple predictive pictures may use two or more reference pictures and associated metadata for the reconstruction of a single block.
[0061] Aspects of this disclosure may also apply to variations of I-picture, P-picture, and B-picture, as well as their respective applications and features.
[0062] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block may be coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by referencing one previously coded reference picture, via spatial prediction or temporal prediction. Blocks of picture B may be coded predictively by referencing one or two previously coded reference pictures, via spatial prediction or temporal prediction.
[0063] The video encoder (303) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec.H.265. In these operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.
[0064] In one embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.
[0065] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is analogous to a reference block in a previously coded and still-buffered reference picture in the video, the block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.
[0066] In some embodiments, a dual prediction technique may be used for interpicture prediction. According to the dual prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, which are earlier in the video in decoding order than the current picture (but may be earlier or later in display order). Blocks in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Blocks may be predicted by a combination of the first and second reference blocks.
[0067] Furthermore, merge mode techniques can be used for interpicture prediction to improve coding efficiency.
[0068] According to some aspects of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks, such as polygon shapes or triangular blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU may be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains a Luma prediction block (PB) and two Chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a Luma prediction block as an example of a prediction block, the prediction block contains a matrix of pixel values (e.g., Luma values) such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.
[0069] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technique. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors that execute software instructions.
[0070] This disclosure includes embodiments relating to methods and apparatus for predicting vertex positions and / or texture coordinates by means of reflection, parallelogram prediction, etc., for mesh compression such as polygon mesh compression. For example, it discloses reflection prediction and / or parallelogram prediction of position and / or UV coordinates in mesh compression such as polygon mesh compression.
[0071] A mesh can contain multiple polygons (multiple polygon faces, etc.) that can describe the surface of a volumetric object. For example, the surface of a volumetric object can be approximated using a mesh. Each polygon in a mesh can be defined by the vertices of the corresponding polygon in three-dimensional (3D) space and information about how the vertices are connected (sometimes called connectivity information). In some embodiments, vertex attributes such as color, normal, and displacement can be associated with vertices (also called mesh vertices). Attributes (also called vertex attributes) can also be associated with the surface of the mesh by leveraging mapping information that parameterizes the mesh using a two-dimensional (2D) attribute map. Such mapping can be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps can be used to store high-resolution attribute information such as texture, normal, and displacement. High-resolution attribute information can be used for various purposes such as texture mapping, shading, and mesh reconstruction.
[0072] Figure 4 shows an example of an encoding process (400) for mesh processing based on an associated video codec, according to one aspect of the present disclosure. As shown in Figure 4, the encoding process (400) may include a preprocessing step (400A) and an encoding step (400B). The preprocessing step (400A) may be configured to generate a base mesh m(i) for the current frame and a displacement field d(i) for the current frame, including displacement vectors, according to an input mesh M(i) for the current frame. The encoding step (400B) may be configured to encode the base mesh m(i), the displacement field d(i), and texture information of the base mesh m(i). The displacement field d(i) for the current frame may include displacement vectors. The index i may refer to the current frame. In one aspect, a mode determination method may be performed in the encoding process (400) to determine whether intercoding (also called interframe prediction or intermode), intracoding (also called intraframe prediction or intramode), etc., is applied to the current frame. For example, a mode determination method may compare the cost of intra-mode and the cost of inter-mode, and determine the coding mode of the current frame's base mesh m(i) based on which cost is smaller. In some examples, skip mode is used to code (e.g., encode or decode) the base mesh m(i). In one example, skip mode is a special mode of inter-mode. For example, the base mesh m(i) may be intra-coded, inter-coded, or coded in SKIP mode.
[0073] Referring further to Figure 4, the preprocessing step (400A) may include a mesh decimation process (402), a parameterization process such as an atlas parameterization process (404), and a subdivision surface fitting process (406). The mesh decimation process (402) is configured to downsample the vertices of the input mesh M(i) to generate a decimated mesh dm(i) which may contain multiple decimated (or downsampled) vertices. The number of multiple decimated vertices is less than the number of vertices in the input mesh M(i). A parameterization process such as an atlas parameterization process (404) is configured to map the decimated mesh dm(i) to a planar region such as a UV atlas (or UV map) to generate a reparameterized mesh pm(i). In one example, atlas parameterization may be performed based on a video processing tool such as a UVAtlas tool. The subdivision surface fitting process (406) is configured to take a reparameterized mesh pm(i) and an input mesh M(i) as inputs and generate a base mesh m(i) along with a displacement field d(i) containing displacement vectors or a set of displacements. In one example of the subdivision surface fitting process (406), pm(i) is subdivided using a subdivision scheme such as iterative interpolation to obtain a subdivided mesh. Iterative interpolation involves inserting a new point in the middle of each edge of the reparameterized mesh pm(i) in each iteration. Any suitable subdivision scheme may be applied to subdivide pm(i). The displacement field d(i) is calculated by determining the nearest point on the surface of the input mesh M(i) for each vertex of the subdivided mesh.
[0074] The advantages of a subdivided mesh may include the fact that, while providing a faithful approximation of the input mesh, the subdivided mesh has a subdivided structure that enables efficient compression. Compression efficiency is obtained due to the following characteristics: the number of vertices in the decimated mesh dm(i) may be small, and it may be encoded and transmitted using fewer bits than the input mesh M(i) or the subdivided mesh. Referring to Figure 4, the base mesh m(i) may be generated from the decimated mesh dm(i). In one example, the base mesh m(i) is the decimated mesh dm(i). Since the subdivided mesh can be generated based on the subdivided method, the subdivided mesh may be automatically generated by the decoder when the base mesh or decimated mesh is decoded (e.g., no information other than the subdivided scheme and the subdivided iteration count is needed). On the decoder side, the displacement field d(i) may be generated by decoding the displacement vectors associated with the vertices of the subdivided mesh. In addition to enabling spatial / quality scalability, the subdivided structure enables efficient transformations such as wavelet decomposition, which can provide high compression performance.
[0075] The coding step (400B) may include base mesh coding (408), displacement coding (410), texture coding (412), etc. Base mesh coding (408) is configured to encode the geometric information of the base mesh m(i) associated with the current frame. In intra coding, the base mesh m(i) may first be quantized (e.g., using uniform quantization) and then encoded by a coding mode determined, for example, using a mode determination method. The coding mode may be inter-mode, intra-mode, skip mode, etc. The encoder used to intra-code the base mesh m(i) is sometimes called a static mesh encoder. In inter coding, a reference base mesh associated with a reference frame indicated by index j (e.g., a reconstructed quantized reference base mesh m'(j)) may be used to predict the base mesh m(i) associated with the current frame indicated by index i. Displacement coding (410) is configured to encode the displacement field d(i) generated in the preprocessing step (400A). The displacement field d(i) may include a set of displacement vectors (or displacements) associated with the vertices of the subdivided mesh. Texture coding (412) is configured to encode attribute information of the base mesh m(i). Attribute information may include texture, normals, color, etc. The attribute information may be encoded based on an appropriate codec such as High Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC).
[0076] In one embodiment, referring to Figure 4, a mesh coding process such as coding process (400) begins with preprocessing (e.g., preprocessing step (400A)). Preprocessing may convert an input mesh M(i) to a base mesh m(i) along with a displacement field d(i) containing a set of displacements (or a set of displacement vectors). Coding step (400B) may compress the output from preprocessing (e.g., m(i), d(i), etc.) to generate a compressed bitstream b(i). The compressed bitstream b(i) may include a compressed base mesh bitstream, a compressed displacement field bitstream (also called a compressed displacement bitstream), a compressed attribute bitstream, etc.
[0077] Figure 5 shows an example of a decoding process (500) for mesh processing according to one aspect of the present disclosure. The decoding process (500) may include a decoding step (505) and a post-processing step (510). A compressed bitstream b(i) may be supplied to the decoding step (505). In one example, in the case of lossless transmission, the compressed bitstream b(i) is the output b(i) from the encoding process (400). The decoding step (505) may extract various sub-bitstreams such as a compressed base mesh substream, a compressed displacement field substream, and a compressed attribute substream. The decoding step (505) may decompress the sub-bitstreams to generate the following components, namely: patch metadata indicated by metadata (i), a decoded base mesh m''(i), a decoded displacement field (including displacement) d''(i), a decoded attribute map A''(i), etc.
[0078] In one embodiment, a base mesh substream may be supplied to a mesh decoder to generate a reconstructed quantized base mesh m'(i). The decoded base mesh m''(i) may be obtained by applying inverse quantization to m'(i). A displacement field substream containing encoded, packed quantized wavelet coefficients may be decoded by a video and / or image decoder. Image unpacking and inverse quantization may be applied to the reconstructed, packed quantized wavelet coefficients to obtain unpacked inversely quantized coefficients (e.g., wavelet coefficients). An inverse wavelet transform may be applied to the unpacked inversely quantized wavelet coefficients to generate a decoded displacement field d''(i).
[0079] The decoded components (including, for example, metadata(i), m''(i), d''(i), A''(i), etc.) can be supplied to a post-processing step (510). A mesh (also called the decoded mesh) M''(i) can be generated by the post-processing step (510) based on m''(i) and d''(i). In one example, the reconstructed deformed mesh DM(i) can be obtained by subdividing m''(i) using a subdivision scheme and applying the reconstructed displacement d''(i) to the vertices of the subdivision mesh. In one example, when the encoding process (400), the decoding process (500), and the transmission are reversible, the mesh M''(i) may be identical to the input mesh M(i). When one of the encoding process (400), the decoding process (500), and the transmission is irreversible, M''(i) is different from M(i). In various examples, the difference between M''(i) and M(i), if any, is relatively small. In one example, attribute map A''(i) is also generated by the post-processing step (510).
[0080] A polygon mesh (also interchangeably called a polygonal mesh) may contain topological quantities such as vertices, edges, and faces, as well as geometric quantities such as attributes including vertex positions and face colors. Connectivity information for a polygon mesh may describe the incidents between elements and may be indicated by topology. For example, two vertices are adjacent if their edges touch those two vertices. For example, two faces are adjacent if their edges touch those two faces.
[0081] Figure 6 shows an example of the vertex degree of a vertex (611) and the face degree of a face (612) in a polygon mesh (600) according to one aspect of the present disclosure. The polygon mesh (600) may contain a plurality of faces, including face (612). The polygon mesh (600) may contain a plurality of vertices, including vertex (611). In one aspect, the vertex degree of a vertex may be interchangeably called the valence of the vertex. The vertex degree of a vertex may specify the number of edges tangent to the vertex. Referring to Figure 6, the vertex degree of vertex (611) may specify the number of edges (641) to (644) tangent to vertex (611), and the vertex degree of vertex (611) is 4. The face degree of a face may specify the number of edges tangent to the face. Referring to Figure 6, the plane degree of surface (612) can be specified by the number of edges (631) to (635) that are adjacent to surface (612), and the plane degree of surface (612) is 5.
[0082] In some embodiments, a mesh can include information such as geometry information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, geometry information is described by a set of 3D locations associated with the vertices of the mesh. In one example, (x,y,z) coordinates can be used to describe the 3D location of a vertex, also called 3D coordinates. Referring to Figure 6, an example of a vertex is vertex (611). In some examples, connectivity information includes a set of vertex indices that describe how vertices are connected to create a 3D surface. In some examples, mapping information describes how the mesh surface is mapped to a 2D region of a plane. In one example, mapping information is described by a set of UV parameters / texture coordinates (u,v) associated with the mesh vertices along with the connectivity information. In some examples, vertex attributes include scalar or vector attribute values associated with the mesh vertices. In some examples, attribute maps include attributes associated with the mesh surface and stored as a 2D image / video. In one example, the mapping between a video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.
[0083] In one embodiment, UV mapping or mesh parameterization may be used to map the faces of a mesh in a 3D region to a 2D region. In some examples, the mesh may be cut into patches (also called patch components) of the 3D region. A patch is a continuous subset of a mesh that has boundaries formed by boundary edges. Boundary edges of a patch are edges that belong to only one polygon in the patch and are not shared by two adjacent polygons in the patch. The vertices of boundary edges in a patch are called boundary vertices of the patch, and non-boundary vertices in a patch are sometimes called interior vertices of the patch in some examples.
[0084] In one embodiment, the patches are parameterized, in some examples, into 2D shapes (also called UV patches, 2D patches, or UV charts). In some examples, the 2D shapes can be packed (e.g., oriented and positioned) into a map, also called a UV atlas. In some examples, the map can be further processed using 2D image or video processing techniques.
[0085] In one example, UV mapping technology generates a UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) in 2D, corresponding to patches of a 3D mesh. The UV atlas contains the assignment of 3D vertices of the 3D mesh to 2D points within a 2D region (e.g., a rectangle). The UV atlas is a mapping between the coordinates of the 3D surface and the coordinates of the 2D region. In one example, a point in the UV atlas at 2D coordinates (u,v) has a value formed by the vertex coordinates (x,y,z) in the 3D region. In another example, the texture atlas contains the color information of the 3D mesh. For example, a point in the texture atlas at 2D coordinates (u,v) (which has the 3D value of (x,y,z) in the UV atlas) has a color that specifies the color attribute of the point at (x,y,z) in the 3D region. In some examples, the coordinates (x,y,z) in the 3D region are called 3D coordinates or xyz coordinates, and the 2D coordinates (u,v) are called uv coordinates or UV coordinates. In one example, the position of a vertex (e.g., vertex (611)) in a mesh such as polygon mesh (600) is indicated by 3D coordinates (x, y, z), and the vertex may correspond to a 2D point in a 2D map or UV map (e.g., a vertex may be mapped to a 2D point in a UV map), and the position of the 2D point may be indicated by UV coordinates (u, v).
[0086] Mesh compression may include connectivity and / or topology coding and value coding for each attribute, thus generating a value bitstream and a connectivity bitstream, respectively. In one embodiment, the value bitstream may be larger than the connectivity bitstream. Predictive schemes may be used to code the values of position and UV attributes. For example, the position value or UV coordinate of each mesh vertex in a mesh (e.g., a 3D mesh) may be predicted using a fixed value (e.g., zero or centroid), the value of the previous position or previous UV coordinate, the average of the last n positions or the average of the last n UV coordinates, parallelogram prediction, reflection prediction, etc.
[0087] Parallelogram prediction can refer to predictions based on parallelograms. According to one aspect of parallelogram prediction, the position of a point can be predicted such that it completes a parallelogram formed by three points of adjacent triangles. Referring to Figure 7, the positions of three points V1-V3 are already determined (e.g., the positions of V1-V3 are known), and the position of point V4 is predicted using parallelogram prediction. According to parallelogram prediction, the position of V4 may be predicted by completing the parallelogram (721) formed by V1-V3 and the predicted point V4', and thus the predicted position of V4 is the position of the predicted point V4'. In one example, a prediction residual showing the difference between the positions of V4 and V4' (e.g., the actual position of V4 and the predicted position of V4), such as vector (711), may be called a correction vector (711). The prediction residual (e.g., vector (711)) may be stored, encoded, and / or transmitted to a decoder. The predicted residuals can be encoded, for example, on the encoder side using entropy coding. On the decoder side, the predicted residuals can be decoded using entropy decoding. For example, the predicted residuals may include a sequence of correctors, which may spread around a zero vector and can be compressed more compactly than the sequence of positions.
[0088] Parallelogram prediction can be applied to predict the position of a point based on the positions of three points, for example, three points whose positions have already been determined. In one embodiment, the points may be vertices in a mesh, such as vertices in a polygon mesh. The mesh may be a 3D mesh. Parallelogram prediction can be applied to predict the position of a vertex in a mesh based on the positions of three vertices in the mesh that have already been determined.
[0089] In one embodiment, the points may be 2D points in a 2D map, where the 2D map is associated with a mesh, such as a 3D mesh. For example, the 2D map may be determined based on a 3D mesh. The location of the 2D points may be indicated by 2D coordinates, such as UV coordinates. Parallelogram prediction can be applied to predict the location of a 2D point in a 2D map based on the locations of three 2D points in the 2D map that have already been determined. For brevity, the explanations in some examples are given using vertices in a mesh, and the explanations allow the 2D points in the 2D map to be appropriately fitted to predict the location of the 2D points.
[0090] Parallelogram prediction may include parallelogram cross-section prediction and parallelogram intra-section prediction, as shown in Figures 7 and 8. Figure 7 shows an example of parallelogram cross-section prediction according to one aspect of the present disclosure. Figure 8 shows an example of intra-section prediction according to one aspect of the present disclosure.
[0091] Figures 7 and 8 show some examples of mesh (700). Mesh (700) may contain multiple faces (701) to (707), etc. Mesh (700) may contain multiple vertices V1 to V6, etc. For example, vertices V1 to V3 are tangent to face (701). Vertices V2, V3 and V6 are tangent to face (702). Vertices V3, V5 and V6 are tangent to face (703). Vertices V1 and V4 are tangent to face (704). Vertices V1 and V3 to V5 are tangent to face (707).
[0092] Referring to Figure 7, the black vertices V1-V3 and V6 are visited vertices; for example, vertices V1-V3 and V6 have already been processed, while the white vertices V4-V5 have not yet been visited (also called unknown vertices). In the example in Figure 7, vertex V4 has been visited; for example, the position of vertex V4 is predicted based on the positions of vertices V1-V3 using the parallelogram prediction described above. In Figure 7, vertices V1-V3, used to predict the position of V4, are tangent to face (701), and the predicted vertex V4 is tangent to face (707), which is different from face (701). Therefore, the parallelogram prediction in Figure 7 is called a parallelogram cross-section prediction, indicating, for example, that the parallelogram prediction in Figure 7 spans multiple faces such as faces (701) and (707). In one example, the multiple faces are adjacent faces. Parallelogram (721) spans faces (701) and (707).
[0093] Referring to Figure 8, the position of vertex V4 has already been determined, the black vertices V1-V4 and V6 are visited vertices, and the white vertex V5 has not yet been visited (also called an unknown vertex). In the example in Figure 8, vertex V5 has been visited; for example, the position of vertex V5 is predicted based on the positions of vertices V1, V3 and V4 using parallelogram prediction as described above. According to parallelogram prediction, the position of V5 may be predicted by completing the parallelogram (722) formed by V1, V3 and V4 and the predicted point V5', and thus the predicted position of V5 is the position of the predicted point V5'. In one example, a predicted residual such as vector (712) shows the difference between the positions of V5 and V5'. The predicted residual (e.g., vector (712)) may be stored, encoded, and / or sent to a decoder. In Figure 8, the vertices V1, V3, and V4 used to predict the position of V5 and the predicted vertex V5 are tangent to the same plane (707). Therefore, the parallelogram prediction in Figure 8 is called an in-parallelogram prediction, indicating, for example, that the parallelogram prediction lies within the same plane (707). The parallelogram (722) may also lie within the plane (707).
[0094] In polygon mesh compression, the value bitstream may occupy a larger portion of the total bitstream compared to the connectivity bitstream; therefore, attribute values can be coded using an efficient method.
[0095] Methods can be used to efficiently code vertex positions and UV values (e.g., UV coordinates) for polygon mesh compression. These methods can be applied individually or in any combination.
[0096] In one embodiment, a polygon mesh is regular (or a polygon mesh is a regular mesh) if the face degree of each face in the polygon mesh is the same and the vertex degree of each vertex in the polygon mesh is the same. Otherwise, a polygon mesh is not regular (or a polygon mesh is not a regular mesh) if the face degrees of each face in the polygon mesh are different and / or the vertex degrees of each vertex in the polygon mesh are different.
[0097] In one embodiment, if the polygon mesh is not regular, in some cases the parallelogram cross-sectional prediction of the vertex positions in the mesh or UV coordinates may be inaccurate. In one example, a triangular mesh may be obtained from the triangulation of a quad mesh (also called a quadrilateral mesh). Each face in the triangular mesh has a face degree of 3 and is identical. Each face in the quadrilateral mesh has a face degree of 4 and is identical. For triangular meshes obtained from the triangulation of a quad mesh, if the triangular mesh is not regular (for example, each vertex in the triangular mesh has a different vertex degree), in some cases the parallelogram cross-sectional prediction of the vertex positions or UV coordinates in the triangular mesh may be inaccurate.
[0098] In one embodiment, the position of a point can be predicted based on other points whose positions have already been determined, using parallelogram cross-section prediction and reflection prediction. In some situations, parallelogram cross-section prediction may be more accurate than reflection prediction, and parallelogram cross-section prediction may be used to predict the position of a point. In other situations, parallelogram cross-section prediction may be less accurate than reflection prediction, and reflection prediction may be used to predict the position of a point. The point may be a vertex in a mesh or a 2D point in a 2D map. By selecting a more accurate prediction method to predict the position, the prediction residual (e.g., vectors (711) to (712) shown in Figures 7 to 8) may be reduced, and thus the prediction residual may be closer to zero, and the prediction residual can be coded using fewer bits to reduce signaling overhead.
[0099] Figure 9 shows an example of parallelogram cross-section prediction and reflection prediction according to one aspect of the present disclosure. Figure 9 shows a portion of a mesh or 2D map (900). The mesh or 2D map (900) may include a plurality of points 1 to 12, etc. In one example, the mesh (900) is a polygon mesh, such as a 3D mesh, points 1 to 12 are vertices 1 to 12 within the mesh (900), and the mesh (900) includes faces (901) to (912), etc. In the example shown in Figure 9, the mesh is a triangular mesh, with three vertices touching each face. For example, vertices 1 to 3 touch face (901), vertices 1, 3 and 4 touch face (902), vertices 5 to 7 touch face (905), and vertices 6 to 8 touch face (911).
[0100] In some scenarios, parallelogram cross-section prediction is more accurate than reflection prediction. In Figure 9, when parallelogram cross-section prediction is applied to predict the position of vertex 4 based on the positions of vertices 1-3, the predicted vertex 4' can be determined. The first predicted position of vertex 4 is the position of predicted vertex 4', and the first predicted residual, such as vector (931), shows the difference between the positions of 4 and 4'. When reflection prediction is applied to predict the position of vertex 4 based on the positions of vertices 1-3, the predicted vertex 4'' can be determined. In reflection prediction, vertex 2 may predict vertex 4 by being reflected off a line passing through vertices 1 and 3, and therefore the line (941) passing through 2 and 4'' may be perpendicular to the line passing through vertices 1 and 3. In one example, the distances between point 2 and point 4'' to the line passing through vertices 1 and 3 are equal. The second predicted position of vertex 4 is the position of predicted vertex 4'', and the second predicted residual, such as vector (932), shows the difference between the position of 4 and the position of 4''. In the example shown in Figure 9, the amplitude of the first predicted residual, such as vector (931), is smaller than the amplitude of the second predicted residual, such as vector (932), and the predicted vertex 4' is closer to the true position of vertex 4 than the predicted point 4''. Therefore, the parallelogram cross-section prediction may be closer to the true position of vertex 4, and thus, for point 4, the parallelogram cross-section prediction is more accurate than the reflection prediction and may be chosen to predict the position of point 4.
[0101] In some scenarios, reflection prediction is more accurate than parallelogram cross-section prediction. Referring to Figure 9, if parallelogram cross-section prediction is applied to predict the position of vertex 8 based on the positions of vertices 5-7, the predicted vertex 8' can be determined. The first predicted position of vertex 8 is the position of predicted vertex 8', and the first predicted residual, such as vector (933), shows the difference between the position of 8 and the position of 8'. In this scenario, the prediction is less accurate compared to the prediction of vertex 4, for example, the amplitude of vector (933) is larger than the amplitude of vector (931).
[0102] To make a more accurate prediction in this scenario, reflection (also called reflection prediction) can be used to predict the position or UV coordinates of point 8 (e.g., vertex 8 or 2D point 8). In one example, point 5 (e.g., vertex 5) may be reflected off a line passing through points 6-7 (e.g., vertices 6 and 7) to predict vertex 8. When reflection prediction is applied to predict the position of vertex 8 based on the positions of vertices 5-7, a predicted vertex 8'' can be determined. A second predicted position of vertex 8 is the position of predicted vertex 8'', and a second predicted residual, such as vector (934), shows the difference between the positions of 8 and 8''. In the example shown in Figure 9, the amplitude of the first predicted residual, such as vector (933), is greater than the amplitude of the second predicted residual, such as vector (934), and the predicted vertex 8'' is closer to the true position of vertex 8 than the predicted vertex 8'. Therefore, the reflection prediction can be closer to the true position of vertex 8. In reflection prediction, the line (942) passing through 5 and 8'' may be perpendicular to the line passing through vertices 6 and 7.
[0103] As mentioned above, in some examples, the position of a point (e.g., point 4), such as the position of a mesh vertex or a 2D point (indicated by UV coordinates), can be predicted more accurately using parallelogram cross-section prediction rather than reflection prediction. In some examples, the reflection prediction of the position or UV coordinates of a point (e.g., vertex 8) is more accurate than the parallelogram cross-section prediction.
[0104] To determine whether to use parallelogram cross-section prediction or reflection prediction to predict vertex positions or UV coordinates, both types of predictions, such as parallelogram cross-section prediction and reflection prediction, may be performed. The more accurate prediction may be selected. In one example, the more accurate prediction may be signaled. To reduce the overhead of signaling which prediction to use, previously signaled predictions may be used to determine the context for coding (e.g., encoding or decoding) the current prediction (e.g., the current prediction may be a parallelogram cross-section prediction or a reflection prediction). In one example, the previously signaled prediction may refer to the most recent decision prior to the current prediction for selecting between parallelogram cross-section prediction and reflection prediction, e.g., the most recent or last time a decision was made between parallelogram cross-section prediction and reflection prediction. In one example, each time a selection is made between parallelogram cross-section prediction and reflection prediction, it may be remembered which predictor (parallelogram cross-section prediction or reflection prediction) is used. When the current prediction (e.g., a choice following a previous prediction such as a previously signaled prediction) is made between a parallelogram cross-section prediction and a reflection prediction, the context for the current prediction may be determined based on a previous prediction or a previous prediction such as a previously signaled prediction. Referring to Figure 9, if the most recent prediction preceding the prediction for point 8 has chosen a prediction method between a parallelogram cross-section prediction and a reflection prediction for point 5, then the most recent prediction is the prediction for point 5.
[0105] For example, two different contexts may be used for parallelogram cross-section prediction and reflection prediction, respectively. If parallelogram cross-section prediction was previously selected, the first context corresponding to parallelogram cross-section prediction may be used to code (e.g., encode or decode) the current prediction (also called the current selected prediction). If reflection prediction was previously selected, the second context corresponding to reflection prediction may be used to code (e.g., encode or decode) the current selected prediction.
[0106] In one example, a choice is made between parallelogram cross-section prediction and reflection prediction. In another example, intra-parallelogram prediction is used when possible, because, for example, intra-parallelogram prediction may be more accurate than parallelogram cross-section prediction.
[0107] In one embodiment, if parallelogram prediction is available, it is used to predict the position of a vertex or the UV coordinates of a 2D point. For example, if a face has a plane degree greater than 3 and the positions of at least three vertices tangent to the face (e.g., three consecutive vertices) are already determined or available (e.g., already encoded), then parallelogram prediction may be available to predict the remaining vertices tangent to the face, and the positions of the remaining vertices may be determined.
[0108] For example, if parallelogram-internal prediction is not available, a decision may be made to use either parallelogram-crossing prediction or reflection prediction.
[0109] In one embodiment, on the decoder side, the syntax element may indicate one of two predictions, a parallelogram cross-section prediction and a reflection prediction, which should be used to predict the position of either the current vertex of the mesh or the current point of the 2D map associated with the mesh. One of the two predictions, the parallelogram cross-section prediction and the reflection prediction, can be determined based on the syntax element to predict the position of either the current vertex of the mesh or the current point of the 2D map. The position of either the current vertex of the mesh or the current point of the 2D map can be predicted using the determined one of the two predictions, the parallelogram cross-section prediction and the reflection prediction, as illustrated, for example, in Figures 7 and 9.
[0110] For example, the context may be determined based on whether the previously signaled prediction is a parallelogram cross-section prediction or a reflection prediction. Syntax elements may be determined (e.g., decoded) using context modeling with the determined context.
[0111] For example, when the previously signaled prediction is a parallelogram prediction, the context is the first context, and when the previously signaled prediction is a reflection prediction, the context is the second context, and the first and second contexts are different.
[0112] In one example, the mesh is a 3D mesh such as mesh(600), and one of the current vertices of the mesh and the current point of the 2D map is the current vertex of the 3D mesh such as vertex(611), and the position of the current vertex is represented by 3D coordinates such as (x,y,z).
[0113] In one example, referring to Figure 9, mesh (900) is a 3D mesh, and the current vertex of the mesh (e.g., point 4) is tangent to the current face of mesh (e.g., (902)). When parallelogram cross-section prediction is determined as one of the above parallelogram cross-section prediction and reflection prediction, the position of the current vertex 4 can be predicted using a parallelogram based on three vertices (e.g., points 1-3) that are tangent to adjacent faces (e.g., face (901)) of the current face (e.g., (902)). Two of the three vertices (e.g., points 1-3) (e.g., points 1 and 3) may be tangent to adjacent faces (e.g., face (901)) and the current face (e.g., (902)).
[0114] In one example, referring further to Figure 9, mesh (900) is a 3D mesh, and the current vertex of the mesh (e.g., point 8) is tangent to the current face of the mesh (e.g., (911)). When the reflection prediction is determined as one of the above parallelogram cross-section prediction and reflection prediction, the position of the current vertex (e.g., point 8) can be predicted using the reflection of the first vertex (e.g., point 5) to a line passing through the second and third vertices (e.g., points 6-7). The first vertex (e.g., point 5) may be tangent to an adjacent face (e.g., face (905)) of the current face (e.g., (911)). The second and third vertices (e.g., points 6-7) may be tangent to an adjacent face (e.g., face (905)).
[0115] In one example, the mesh is a 3D mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh or the current points of the 2D map is the current point of the UV map, and the current point of the UV map is indicated by UV coordinates. The method described for predicting the position of a vertex can be appropriately adapted to predict the UV coordinates of a 2D point.
[0116] On the encoder side, one of either a parallelogram cross-section prediction or a reflection prediction may be determined to predict the position of either the current vertex of the mesh or the current point of the 2D map associated with the mesh. The position of either the current vertex of the mesh or the current point of the 2D map may be predicted using either the parallelogram cross-section prediction or the reflection prediction. The context for encoding either the parallelogram cross-section prediction or the reflection prediction may be determined based on whether a previously encoded position (e.g., associated with a previously encoded vertex or a previously encoded 2D point) is encoded using either the parallelogram cross-section prediction or the reflection prediction. The syntax element representing either the parallelogram cross-section prediction or the reflection prediction may be encoded using context modeling with the determined context.
[0117] For example, the encoder can perform a parallelogram cross-section prediction to obtain a first predicted position (e.g., position 4') and a first residual indicating a first correction vector (e.g., vector (931)) between the current vertex of the mesh (e.g., point 4) and the current point of the 2D map. A reflection prediction can then be performed to obtain a second predicted position (e.g., position 4'') and a second residual indicating a second correction vector (e.g., vector (932)) between the current vertex of the mesh (e.g., point 4) and the current point of the 2D map. The parallelogram cross-section prediction and the reflection prediction can be determined based on the minimum value (e.g., amplitude of vector (931)) of the first residual (e.g., vector (931)) and the second residual (e.g., vector (932)).
[0118] Figure 10 shows a flowchart outlining a process (1000) according to one aspect of the present disclosure. Process (1000) can be used in an apparatus. The apparatus may include a mesh decoder, such as a video decoder. The video decoder is configured, for example, to decode mesh connectivity information. In various aspects, process (1000) is performed by processing circuits such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), a mesh decoder, etc. In some aspects, process (1000) is implemented with software instructions, and so the processing circuit performs process (1000) when it executes the software instructions. The process starts at (S1001) and proceeds to (S1010).
[0119] In (S1010), process (1000) includes receiving coded information, which includes syntax elements indicating one of parallelogram cross-section predictions and reflection predictions for predicting the position of one of the current vertices of the mesh and the current points of a two-dimensional (2D) map associated with the mesh.
[0120] In (S1020), process (1000) includes the step of determining one of a parallelogram cross-section prediction and a reflection prediction for predicting the position of one of the current vertices of the mesh and the current points of the 2D map, based on syntax elements.
[0121] In (S1030), process (1000) includes the step of predicting the position of one of the current vertices of the mesh and the current points of the 2D map, using the determined one of the parallelogram cross-section prediction and the reflection prediction.
[0122] Next, the process proceeds to (S1099) and terminates.
[0123] Process (1000) can be appropriately adapted. Steps in process (1000) can be modified and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.
[0124] In one example, the above determination step includes determining the context based on whether the previously signaled prediction is a parallelogram cross-section prediction or a reflection prediction, and decoding the syntax elements using context modeling with the determined context.
[0125] For example, when the previously signaled prediction is a parallelogram cross-section prediction, the context is the first context, and when the previously signaled prediction is a reflection prediction, the context is the second context, and the first and second contexts are different.
[0126] In one example, the mesh is a three-dimensional (3D) mesh, and one of the following—the current vertex of the mesh or the current point of the 2D map—is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates.
[0127] In one example, the current vertex of the mesh is tangent to the current face of the mesh. When a parallelogram cross-section prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the prediction step includes predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face. Two of the three vertices may be tangent to adjacent faces and the current face.
[0128] In one example, the current vertex of the mesh is tangent to the current face of the mesh. When the reflection prediction is determined to be one of the above, a parallelogram cross-section prediction or a reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices. The first vertex may be tangent to an adjacent face of the current face, and the second and third vertices may be tangent to adjacent faces.
[0129] In one example, the mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh and the current point of the 2D map is the current point of the UV map, and the current point of the UV map is indicated by UV coordinates.
[0130] Figure 10 shows a flowchart outlining a process (1100) according to one aspect of the present disclosure. Process (1100) can be used in a device. The device may include a mesh encoder, such as a video encoder. The video encoder is configured, for example, to encode mesh connectivity information. In various aspects, process (1100) is performed by processing circuits such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), a mesh encoder, etc. In some aspects, process (1100) is implemented with software instructions, and so the processing circuit performs process (1100) when it executes the software instructions. The process starts at (S1101) and proceeds to (S1110).
[0131] In (S1110), process (1100) includes the step of determining one of a parallelogram cross-section prediction and a reflection prediction to predict the position of one of the current vertices of the mesh and the current points of a two-dimensional (2D) map associated with the mesh.
[0132] In (S1120), process (1100) includes the step of predicting the position of one of the current vertices of the mesh and the current points of the 2D map using the parallelogram cross-section prediction and the reflection prediction.
[0133] In (S1130), process (1100) includes the step of determining a context for encoding one of the above parallelogram cross-section predictions and reflection predictions, based on whether the previously encoded position was encoded using a parallelogram cross-section prediction or a reflection prediction.
[0134] In (S1140), process (1100) includes the step of encoding a syntax element that represents one of the above, a parallelogram cross-section prediction and a reflection prediction, using context modeling with the determined context.
[0135] Next, the process proceeds to (S1199) and terminates.
[0136] Process (1100) can be appropriately adapted. Steps in process (1100) can be modified and / or omitted. Additional steps can be added. Any appropriate implementation order can be used.
[0137] In one example, when a previously encoded location is encoded using parallelogram cross-section prediction, the context is the first context, and when a previously encoded location is encoded using reflection prediction, the context is the second context, and the first and second contexts are different.
[0138] In one example, the mesh is a three-dimensional (3D) mesh, and one of the current vertices of the mesh and the current point of the 2D map is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates.
[0139] In one example, the current vertex of the mesh is tangent to the current face of the mesh. When the parallelogram cross-section prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the step of predicting the position includes predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face, two of which are tangent to the adjacent face and the current face.
[0140] In one example, the current vertex of the mesh is tangent to the current face of the mesh. When the reflection prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices, wherein the first vertex is tangent to an adjacent face of the current face, and the second and third vertices are tangent to adjacent faces.
[0141] In one example, the mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current point of the UV map, and the current point of the UV map is indicated by UV coordinates.
[0142] In one example, the step of determining one of a parallelogram cross-section prediction and a reflection prediction includes: performing a parallelogram cross-section prediction to obtain a first predicted position and a first residual indicating a first correction vector between the position of the current vertex of the mesh and the current point of the 2D map; performing a reflection prediction to obtain a second predicted position and a second residual indicating a second correction vector between the position of the current vertex of the mesh and the current point of the 2D map; and determining one of the parallelogram cross-section prediction and the reflection prediction corresponding to the minimum value of the first and second residuals.
[0143] In one embodiment, a method for processing mesh data of a mesh includes processing a bitstream of the mesh according to a format rule. For example, the bitstream is a bitstream that is decoded / encoded in one of the decoding and / or encoding methods described herein. The format rule may specify one or more constraints on the bitstream and / or one or more processes performed by the decoder and / or encoder.
[0144] In one embodiment, a method for processing a mesh includes a method for processing a bitstream of the mesh according to formatting rules. The bitstream includes syntax elements indicating one of a parallelogram cross-section prediction and a reflection prediction for predicting the position of either the current vertices of the mesh or the current points of a two-dimensional (2D) map associated with the mesh.
[0145] The formatting rules specify that, based on syntax elements, one of the above-mentioned parallelogram cross-section predictions and reflection predictions is used to predict the position of either the current vertex of the mesh or the current point of the 2D map, and that the position of either the current vertex of the mesh or the current point of the 2D map is used to predict the position of either the current vertex of the mesh or the current point of the 2D map using the determined one of the above-mentioned parallelogram cross-section predictions and reflection predictions.
[0146] The methods, embodiments, and examples in this disclosure may be used separately or in any order. For example, some embodiments and / or examples performed by a decoder may be performed by an encoder, and vice versa. Each of the methods (or embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium.
[0147] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 12 shows a computer system (1200) suitable for implementing a particular aspect of the disclosed subject matter.
[0148] Computer software can be coded using any suitable machine code or computer language, which may follow similar mechanisms for creating code that includes instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or through translation, microcode execution, etc.
[0149] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game devices, and Internet of Things devices.
[0150] The components shown in Figure 12 for the computer system (1200) are illustrative and are not intended to imply any limitations on the scope of use or functionality of computer software implementing aspects of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependencies or requirements relating to any one or combination of components shown in the illustrative aspects of the computer system (1200).
[0151] The computer system (1200) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users via, for example, haptic input (keystrokes, swipes, data glove movements, etc.), audio input (voices, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), and video (2D video, 3D video including stereoscopic video, etc.).
[0152] The input human interface device may include one or more of the following (only one of each is shown): keyboard (1201), mouse (1202), trackpad (1203), touchscreen (1210), data glove (not shown), joystick (1205), microphone (1206), scanner (1207), and camera (12012).
[0153] The computer system (1200) may include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1210), data glove (not shown), or joystick (1205), although tactile feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (1209), headphones (not shown)), visual output devices (screens (1210) including CRT screens, LCD screens, plasma screens, and OLED screens, each having or not having touchscreen input functionality, each having or not having tactile feedback functionality, some of which have the ability to output 2D visual output or output beyond 3D via means such as stereoscopic image output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).
[0154] The computer system (1200) may also include human-accessible storage devices and associated media such as optical media including CD / DVD ROM / RW (1220) having CD / DVD or similar media (1221), thumb drives (1222), removable hard drives or solid-state drives (1223), legacy magnetic media such as tapes and floppy disks (not shown), and special ROM / ASIC / PLD-based devices such as security dongles (not shown).
[0155] Those skilled in the art should also understand that, when used in connection with the subject matter now disclosed, the term “computer-readable medium” does not include a transmission medium, carrier wave, or other transient signal.
[0156] The computer system (1200) may also include an interface (1254) to one or more communication networks (1255). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include Ethernet®, wireless LAN, cellular networks including GSM® and 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicle and industrial networks including CANBus, etc. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (1249) (e.g., a USB port on the computer system (1200)), while other networks are generally integrated into the core of the computer system (1200) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1200) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, such as communication to other computer systems using a local or wide-area digital network. As described above, specific protocols and protocol stacks can be used in each of these networks and network interfaces.
[0157] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (1240) of the computer system (1200).
[0158] The core (1240) may include one or more central processing units (CPUs) (1241), graphics processing units (GPUs) (1242), special programmable processing units in the form of field-programmable gate areas (FPGAs) (1243), hardware accelerators for specific tasks (1244), graphics adapters (1250), etc. These devices may be connected via a system bus (1248) along with read-only memory (ROM) (1245), random access memory (1246), internal mass storage such as an internal non-user-accessible hard drive or SSD (1247). In some computer systems, the system bus (1248) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (1248) or via a peripheral bus (1249). For example, a screen (1210) can be connected to the graphics adapter (1250). The peripheral bus architecture includes PCI, USB, etc.
[0159] The CPU (1241), GPU (1242), FPGA (1243), and accelerator (1244) can execute specific instructions that can be combined to form the aforementioned computer code. This computer code can be stored in ROM (1245) or RAM (1246). Transition data can also be stored in RAM (1246), while persistent data can be stored, for example, in internal mass storage (1247). High-speed storage and retrieval to and from any of the memory devices can be enabled through the use of cache memory that can be closely associated with one or more CPUs (1241), GPUs (1242), mass storage (1247), ROMs (1245), RAM (1246), etc.
[0160] A computer-readable medium may contain computer code for performing operations that are implemented on various computers. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.
[0161] For example, and not an limitation, a computer system having architecture (1200), particularly a core (1240), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core (1240) of a non-transient nature, such as core internal mass storage (1247) or ROM (1245). Software implementing various aspects of this disclosure can be stored in such devices and executed by the core (1240). The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core (1240), specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to execute specific processes or specific parts of specific processes as described herein, including defining data structures stored in RAM (1246) and modifying such data structures according to processes defined by the software. As an addition or alternative, a computer system may provide functionality as a result of logic hardwired or otherwise embodied within a circuit (e.g., an accelerator (1244)), which may operate in place of or with software to perform a particular process or a particular part of a particular process as described herein. References to software may, where appropriate, encompass logic, and vice versa. References to computer-readable media may, where appropriate, include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or both. This disclosure encompasses any appropriate combination of hardware and software.
[0162] The use of “at least one of” or “one of” in this disclosure is intended to include any one or combination of the enumerated elements. For example, at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; a reference to at least one of A-C is intended to include A only, B only, C only, or any combination thereof. A reference to one of A or B and one of A and B is intended to include A or B or (A and B). The use of “one of” does not exclude any combination of the enumerated elements when applicable, such as when the elements are not mutually exclusive.
[0163] While this disclosure describes examples of several embodiments, alternatives, substitutions, and various alternative equivalents exist and fall within the scope of this disclosure. Therefore, those skilled in the art will understand that numerous systems and methods, not expressly shown or described herein, can be devised to embody the principles of this disclosure and thus fall within its spirit and scope.
[0164] The above disclosure also includes the features described below. These features may be combined in various ways, and are not limited to the combinations described below.
[0165] (1) A method for decoding a mesh, the method comprising: receiving coded information including a syntax element indicating one of a parallelogram cross-section prediction and a reflection prediction for predicting the position of one of the current vertices of the mesh and the current points of a two-dimensional (2D) map associated with the mesh; determining one of the parallelogram cross-section prediction and the reflection prediction based on the syntax element for predicting the position of the one of the current vertices of the mesh and the current points of the 2D map; and using the determined one of the parallelogram cross-section prediction and the reflection prediction, predicting the position of the one of the current vertices of the mesh and the current points of the 2D map.
[0166] (2) The method of feature (1), wherein the steps of determining the context include: determining the context based on whether the previously signaled prediction is a parallelogram cross-section prediction or a reflection prediction; and decoding the syntax elements using context modeling with the determined context.
[0167] (3) When the previously signaled prediction is a parallelogram cross-section prediction, the context is the first context, and when the previously signaled prediction is a reflection prediction, the context is the second context, and the first and second contexts are different, according to the method of feature (2).
[0168] (4) The mesh is a three-dimensional (3D) mesh, and one of the above, the current vertex of the mesh and the current point of the 2D map, is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates, in any of the methods of (1) to (3).
[0169] (5) The current vertex of the mesh is tangent to the current face of the mesh, method of feature (4). When the parallelogram cross-section prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the prediction step includes predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face. Two of the three vertices may be tangent to adjacent faces and the current face.
[0170] (6) The current vertex of the mesh is tangent to the current face of the mesh, method of feature (4). When the reflection prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices. The first vertex may be tangent to an adjacent face of the current face, and the second and third vertices may be tangent to adjacent faces.
[0171] (7) The mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current point of the UV map, and the current point of the UV map is indicated by UV coordinates, in any of the methods of features (1) to (3).
[0172] (8) A method for encoding a mesh, the method comprising: determining one of a parallelogram cross-section prediction and a reflection prediction for predicting the location of one of the current vertices of a mesh and the current points of a two-dimensional (2D) map associated with the mesh; using the one of the parallelogram cross-section prediction and the reflection prediction, predicting the location of one of the current vertices of a mesh and the current points of a 2D map; determining a context for encoding the one of the parallelogram cross-section prediction and the reflection prediction, based on whether a previously encoded location was encoded using a parallelogram cross-section prediction or a reflection prediction; and encoding a syntax element representing the one of the parallelogram cross-section prediction and the reflection prediction using context modeling with the determined context.
[0173] (9) When a previously encoded position is encoded using parallelogram cross-section prediction, the context is the first context; when a previously encoded position is encoded using reflection prediction, the context is the second context, and the first and second contexts are different.
[0174] (10) The mesh is a three-dimensional (3D) mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates, in either way of feature (8) or (9).
[0175] (11) The method of feature (10), wherein the current vertex of the mesh is tangent to the current face of the mesh. When the parallelogram cross-section prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the step of predicting the position includes the step of predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face, two of the three vertices tangent to the adjacent face and the current face.
[0176] (12) The method of feature (10), wherein the current vertex of the mesh is tangent to the current face of the mesh. When the reflection prediction is determined to be one of the above parallelogram cross-section prediction and reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices, wherein the first vertex is tangent to an adjacent face of the current face, and the second and third vertices are tangent to adjacent faces.
[0177] (13) The mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and either the current vertex of the mesh or the current point of the 2D map is the current point of the UV map, and the current point of the UV map is indicated by UV coordinates, in either way of feature (8) or (9).
[0178] (14) Any method of features (8) to (13), wherein the step of determining one of a parallelogram cross-section prediction and a reflection prediction includes: performing a parallelogram cross-section prediction to obtain a first predicted position and a first residual indicating a first correction vector between the position of the current vertex of the mesh and the current point of the 2D map; performing a reflection prediction to obtain a second predicted position and a second residual indicating a second correction vector between the position of the current vertex of the mesh and the current point of the 2D map; and determining one of the parallelogram cross-section prediction and the reflection prediction corresponding to the minimum value of the first and second residuals.
[0179] (15) A method for processing a mesh, comprising the step of processing a bitstream of the mesh according to a format rule. The bitstream includes syntax elements indicating one of a parallelogram cross-section prediction and one of a reflection prediction for predicting the position of either the current vertices of the mesh or the current points of a two-dimensional (2D) map associated with the mesh. The format rule specifies that the one of the parallelogram cross-section prediction and the reflection prediction for predicting the position of either the current vertices of the mesh or the current points of the 2D map is determined based on the syntax elements, and that the position of either the current vertices of the mesh or the current points of the 2D map is predicted using the determined one of the parallelogram cross-section prediction and the reflection prediction.
[0180] (16) The method of feature (15), wherein the format rule specifies determining the context based on whether the previously signaled prediction is a parallelogram cross-section prediction or a reflection prediction, and decoding the syntax elements using context modeling with the determined context.
[0181] (17) When the previously signaled prediction is a parallelogram cross-section prediction, the context is the first context, and when the previously signaled prediction is a reflection prediction, the context is the second context, and the first context and the second context are different, the method of feature (16).
[0182] (18) The mesh is a three-dimensional (3D) mesh, and one of the above, the current vertex of the mesh and the current point of the 2D map, is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates, in any of the ways of features (15) to (17).
[0183] (19) The current vertex of the mesh is tangent to the current face of the mesh, and the format rule is that when the parallelogram cross-section prediction is determined as one of the above parallelogram cross-section prediction and reflection prediction, the position of the current vertex is predicted using a parallelogram based on three vertices tangent to adjacent faces of the current face, two of the three vertices tangent to adjacent faces and the current face, the method of feature (18).
[0184] (20) The current vertex of the mesh is tangent to the current face of the mesh, and the format rule is that when the reflection prediction is determined as one of the above parallelogram cross-sectional prediction and reflection prediction, the position of the current vertex is predicted using the reflection of the first vertex to a line passing through the second and third vertices, the first vertex is tangent to an adjacent face of the current face, and the second and third vertices are tangent to adjacent faces.
[0185] (21) An apparatus for video decoding, comprising a processing circuit configured to perform any of the methods of (1) to (7).
[0186] (22) An apparatus for video coding, comprising a processing circuit configured to perform any of the methods of (8) to (14).
[0187] (23) A non-temporary computer-readable storage medium that, when executed by at least one processor, stores instructions causing at least one processor to perform one of the methods of features (1) to (20).
Claims
1. A method for decoding a mesh, which is performed by a processor, The steps include receiving coded information, which includes syntax elements indicating one of a parallelogram cross-section prediction and a reflection prediction for predicting the position of either the current vertex of the mesh or the current point of a two-dimensional (2D) map associated with the mesh, A step of determining one of the parallelogram cross-section prediction and the reflection prediction for predicting the position of one of the current vertices of the mesh and the current points of the 2D map based on the syntax elements, A step of predicting the position of one of the current vertices of the mesh and the current point of the 2D map using the determined one of the parallelogram cross-section prediction and the reflection prediction, Methods that include...
2. The aforementioned decision-making step is: A step of determining the context based on whether the previously signaled prediction is the parallelogram cross-section prediction or the reflection prediction, The steps include: decoding the syntax elements using context modeling with the determined context; The method according to claim 1, including the method described in claim 1.
3. When the previously signaled prediction is the parallelogram cross-section prediction, the context is the first context, When the previously signaled prediction is the reflection prediction, the context is the second context, Different from the first and second contexts, The method according to claim 2.
4. The mesh is a three-dimensional (3D) mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates. The method according to claim 1.
5. The current vertex of the mesh is tangent to the current face of the mesh, When the parallelogram cross-section prediction is determined to be one of the parallelogram cross-section prediction and the reflection prediction, the prediction step includes predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face, where two of the three vertices are tangent to the adjacent face and the current face. The method according to claim 4.
6. The current vertex of the mesh is tangent to the current face of the mesh, When the reflection prediction is determined to be one of the parallelogram cross-section prediction and the reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices, wherein the first vertex is tangent to an adjacent face of the current face, and the second and third vertices are tangent to the adjacent faces. The method according to claim 4.
7. The mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current point of the UV map, and the current point of the UV map is represented by UV coordinates. The method according to claim 1.
8. A method for encoding a mesh, which is performed by a processor, The steps include determining one of a parallelogram cross-section prediction and a reflection prediction to predict the position of either the current vertex of the mesh or the current point of a two-dimensional (2D) map associated with the mesh, A step of predicting the position of one of the current vertex of the mesh and the current point of the 2D map using the parallelogram cross-section prediction and the reflection prediction, A step of determining a context for encoding one of the parallelogram cross-section prediction and the reflection prediction, based on whether the previously encoded position was encoded using the parallelogram cross-section prediction or the reflection prediction, The steps include: using the determined context to encode a syntax element representing one of the parallelogram cross-section prediction and the reflection prediction using context modeling; Methods that include...
9. When the previously encoded position is encoded using the parallelogram cross-section prediction, the context is the first context, When the previously encoded position is encoded using the reflection prediction, the context is a second context, Different from the first and second contexts, The method according to claim 8.
10. The mesh is a three-dimensional (3D) mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current vertex of the 3D mesh, and the position of the current vertex is indicated by 3D coordinates. The method according to claim 8.
11. The current vertex of the mesh is tangent to the current face of the mesh, When the parallelogram cross-section prediction is determined to be one of the parallelogram cross-section prediction and the reflection prediction, the step of predicting the position includes the step of predicting the position of the current vertex using a parallelogram based on three vertices tangent to adjacent faces of the current face, where two of the three vertices are tangent to the adjacent face and the current face. The method according to claim 10.
12. The current vertex of the mesh is tangent to the current face of the mesh, When the reflection prediction is determined to be one of the parallelogram cross-section prediction and the reflection prediction, the prediction step includes predicting the position of the current vertex using the reflection of the first vertex to a line passing through the second and third vertices, wherein the first vertex is tangent to an adjacent face of the current face, and the second and third vertices are tangent to the adjacent faces. The method according to claim 10.
13. The mesh is a three-dimensional (3D) mesh, the 2D map is a UV map associated with the 3D mesh, and one of the current vertices of the mesh and the current points of the 2D map is the current point of the UV map, and the current point of the UV map is represented by UV coordinates. The method according to claim 8.
14. The step of determining one of the parallelogram cross-section prediction and the reflection prediction is: The steps include: performing the parallelogram cross-section prediction and obtaining a first residual that indicates a first predicted position and a first correction vector between the position of one of the current vertices of the mesh and the current point of the 2D map; The steps include performing the reflection prediction and obtaining a second residual that indicates a second predicted position and a second correction vector between the position of one of the current vertices of the mesh and the current point of the 2D map, A step of determining one of the parallelogram cross-section prediction and the reflection prediction that corresponds to the minimum value of the first residual and the second residual, The method according to claim 8, including the method described in claim 8.
15. A method for processing a mesh, which is performed by a processor, The step includes processing the bitstream of the mesh according to formatting rules, The bitstream includes syntax elements indicating one of parallelogram cross-sectional prediction and reflection prediction for predicting the position of either the current vertex of the mesh or the current point of a two-dimensional (2D) map associated with the mesh. The aforementioned formatting rules are: The determination of one of the parallelogram cross-sectional prediction and the reflection prediction for predicting the position of one of the current vertices of the mesh and the current points of the 2D map, based on the syntax element, The position of one of the current vertices of the mesh and the current point of the 2D map is predicted using the determined one of the parallelogram cross-section prediction and the reflection prediction. A method for specifying this.
16. A mesh decoding apparatus comprising a processing circuit configured to perform the method described in any one of claims 1 to 7.
17. An apparatus for mesh coding, comprising a processing circuit configured to perform the method described in any one of claims 8 to 14.
18. A computer program that, when executed by a processor, causes the processor to perform the method described in any one of claims 1 to 15.
