Fixed point location coding and decoding in lossless grid compression

Abstract

Methods and apparatus, including computer code configured to cause at least one processor to obtain a mesh from a code stream, the mesh representing encoded volume data of at least one three-dimensional (3D) visual content; dividing the plurality of vertexes of the grid into a plurality of groups, at least one of the plurality of groups representing the shape of a parallelogram; and, for each of the plurality of groups, decoding the encoded volume data by predicting a plurality of vertices in the group based on a prediction mode associated with the group.

Description

[0001] This application claims priority to U.S. Application No. 63 / 458,917, filed on April 12, 2023, and U.S. Provisional Application No. 18 / 632,497, filed on April 11, 2024, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to a set of advanced video coding techniques including lossless trellis compression. Background Art

[0003] Advances in 3D capture, modeling, and rendering have led to the ubiquity of 3D content across several platforms and devices. Today, it is possible to capture a baby’s first steps on one continent, allowing grandparents on another continent to see (and perhaps interact with) the child and enjoy a fully immersive experience with the child. However, to achieve this level of realism, models are becoming increasingly complex, and large amounts of data are associated with the creation and use of these models. 3D meshes are widely used to represent this type of immersive content.

[0004] A mesh consists of several polygons that describe the surface of a volumetric object. Each polygon is defined by its vertices in 3D space and information about how these vertices are connected (called connectivity information). Optionally, vertex attributes (such as color, normals, etc.) can be associated with mesh vertices. By parameterizing the mesh using mapping information with a 2D attribute map, attributes can also be associated with the surface of the mesh. This mapping is typically described by a set of parameter coordinates (called UV coordinates or texture coordinates) associated with the mesh vertices. A 2D attribute map is used to store high-resolution attribute information, such as textures, normals, displacements, etc. This information can be used for various purposes such as texture mapping and shading.

[0005] Dynamic mesh sequences require a lot of data because it consists of a lot of information that changes over time. Therefore, in order to store and transmit such content, efficient compression techniques are required. Mesh compression standards IC, MESHGRID, FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute graphs and connectivity information. DCC (Digital Content Creation) tools usually generate such dynamic meshes. In contrast, it is a challenge for volume acquisition technology to generate constant connectivity dynamic meshes, especially under real-time constraints. Existing standards do not support this type of content. MPEG is planning to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and optional time-varying attribute graphs. The standard targets lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, AR and VR. Features such as random access and scalable / progressive coding are also considered. Therefore, for any of the above reasons, there is an urgent need for technical solutions to deal with these problems arising in video coding technology.

[0006] In lossless mesh compression, connectivity information, geometry information, mapping information, vertex attributes, and attribute maps are compressed in a lossless manner.

[0007] Mesh geometry information consists of vertex 3D coordinates, 2D texture coordinates, etc. Compression of vertex 3D coordinates (which is also called vertex position) is very important because, in most cases, it occupies a large portion of the entire geometry-related data. Summary of the invention

[0008] The present application includes a method and an apparatus, the apparatus including a memory configured to store computer program code, and at least one processor configured to read the computer program code and operate according to the computer program code. The computer program code is configured to enable the processor to implement: an acquisition code configured to enable the at least one processor to acquire a mesh from a bitstream, the mesh representing encoded volume data of at least one three-dimensional 3D visual content; a partitioning code configured to enable the at least one processor to partition a plurality of vertices of the mesh into a plurality of groups, at least one of the plurality of groups representing a parallelogram shape; and a decoding code configured to enable the at least one processor to decode the encoded volume data by predicting a plurality of vertices in the group for each of the plurality of groups based on a prediction mode associated with the group.

[0009] The decoding of the encoded volume data includes: applying parallelogram prediction, wherein the quadrilateral is divided into two triangles, and a vertex of a first one of the two triangles is used as a predictor of a vertex of a second one of the two triangles.

[0010] The first triangle of the two triangles includes vertex A, vertex B and vertex C; the second triangle of the two triangles includes vertex B, vertex C and vertex D; the parallelogram prediction includes: based on the coordinates of the vertex A, the coordinates of the vertex B and the coordinates of the vertex C, predicting the coordinates of the vertex D.

[0011] At least one of the plurality of vertices does not include a prediction candidate.

[0012] At least one of the plurality of vertices includes a prediction candidate.

[0013] At least one of the plurality of vertices includes more than one prediction candidate.

[0014] The parallelogram prediction includes predicting an average of the more than one prediction candidates as a residual.

[0015] An upper limit of a plurality of prediction candidates is signaled in the bitstream.

[0016] The upper limit of the number of prediction candidates is signaled in the codestream via the syntax mesh_position_prediction_max_parallelograms_minus1.

[0017] The syntax mesh_position_prediction_max_parallelograms_minus1 indicates the upper limit by indicating a range of values. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Other features, properties and various advantages of the disclosed subject matter will become further apparent from the following detailed description and accompanying drawings, in which:

[0019] Figure 1 is a schematic diagram of the structure according to an embodiment of the present application;

[0020] Figure 2 is a simplified block diagram according to an embodiment of the present application;

[0021] Figure 3 is a simplified schematic diagram according to an embodiment of the present application;

[0022] Figure 4 is a simplified schematic diagram according to an embodiment of the present application;

[0023] Figure 5 is a simplified schematic diagram according to an embodiment of the present application;

[0024] Figure 6 is a simplified schematic diagram according to an embodiment of the present application;

[0025] Figure 7 is a simplified schematic diagram according to an embodiment of the present application;

[0026] Figure 8 is a simplified schematic diagram according to an embodiment of the present application;

[0027] Fig. 9 is a simplified schematic diagram according to an embodiment of the present application;

[0028] Fig.10 is a simplified flow chart according to an embodiment of the present application;

[0029] Fig.11 is a simplified flow chart according to an embodiment of the present application;

[0030] Fig.12 is a simplified flow chart according to an embodiment of the present application;

[0031] Fig.13 is a simplified schematic diagram according to an embodiment of the present application;

[0032] Fig.14 is a simplified schematic diagram according to an embodiment of the present application;

[0033] Fig.15 is a simplified schematic diagram according to an embodiment of the present application;

[0034] Fig.16 is a simplified schematic diagram according to an embodiment of the present application;

[0035] Fig.17 is a simplified schematic diagram according to an embodiment of the present application;

[0036] Fig.18 is a simplified schematic diagram according to an embodiment of the present application;

[0037] Fig.19 is a simplified schematic diagram according to an embodiment of the present application;

[0038] Fig. 20 is a simplified schematic diagram according to an embodiment of the present application; and,

[0039] Fig.21 It is a simplified schematic diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The features discussed below may be used alone or in any order. In addition, the embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-transitory computer readable medium.

[0041] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment of the present application is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional transmission of data, the first terminal 103 may encode video data at a local location for transmission to another terminal 102 via the network 105. The second terminal 102 may receive the encoded video data of the other terminal from the network 105, decode the encoded data and display the recovered video data. Unidirectional data transmission may be common in media service applications and the like.

[0042] Figure 1 A second pair of terminals 101 and 104 is shown for supporting bidirectional transmission of encoded video, such as during a video conference. To transmit data bidirectionally, each terminal 101 and 104 can encode video data captured at a local location for transmission to the other terminal via a network 105. Each terminal 101 and 104 can also receive encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0043] exist Figure 1 In the embodiment of the present invention, terminals 101, 102, 103 and 104 can be shown as servers, personal computers and smart phones, but the principles of the present application are not limited to this. Embodiments of the present application are applicable to notebook computers, tablet computers, media players and / or special video conferencing equipment. Network 105 represents any number of networks that transmit encoded video data between terminals 101, 102, 103 and 104, including, for example, wired and / or wireless communication networks. Communication network 105 can exchange data in circuit switching and / or packet switching channels. Representative networks include telecommunication networks, local area networks, wide area networks and / or the Internet. For the purpose of this discussion, unless explained below, the architecture and topology of network 105 may be unimportant for the operation of the present application.

[0044] As an example, Figure 2 The video encoder and video decoder are shown in a streaming environment. The subject matter disclosed in this application is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0045] like Figure 2As shown, the streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera, for creating an uncompressed sample stream 213. In an embodiment, the sample stream 213 includes samples captured by the digital camera of the video source 201. The sample stream 213 is depicted as a thick line to emphasize the high data volume of the video picture stream compared to the encoded video code stream, and the sample stream 213 may be generated by an encoder 202 coupled to the video source 201. The encoder 202 may include hardware, software, or a combination of hardware and software to implement or implement various aspects of the disclosed subject matter as described in more detail below. Compared to the sample stream 213, the encoded video code stream 204 is depicted as a thin line to emphasize the lower data volume, which may be stored on the streaming server 205 for future use. At least one streaming client 212 and 207 may access the streaming server 205 to retrieve the copy 208 and the copy 206 of the encoded video code stream 204. The client 212 may include a video decoder 212 that decodes the encoded video bitstream 208 and generates an output video sample stream 210 that can be presented on a display 209 or another presentation device (not depicted). In some streaming systems, the video bitstreams 204, 206, and 208 may be encoded according to certain video encoding / compression standards. Examples of these standards are described above and further described in this application.

[0046] Figure 3 is a block diagram of a video decoder 300 according to an embodiment disclosed in the present application.

[0047] Receiver 302 may receive at least one encoded video sequence to be decoded by video decoder 300. In the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of the other encoded video sequences. The encoded video sequence may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive the encoded video data and other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective use entities (not shown). Receiver 302 may separate the encoded video sequence from the other data. To prevent network jitter, buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304. When receiver 302 receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isochronous network, buffer 303 may not be needed, or may be very small. For use on a best-effort packet network such as the Internet, buffer 303 may be needed, buffer 303 may be relatively large, and may advantageously have an adaptive size.

[0048] The video decoder 300 may include a parser 304 for reconstructing symbols 313 from a coded video sequence. The categories of these symbols include information for managing the operation of the video decoder 300, and potential information for controlling a display device such as a display device (312). The control information for the rendering device may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The encoding of the coded video sequence may be based on a video coding technique or standard, and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subgroup parameter set for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0049] The parser 304 may perform an entropy decoding / parsing operation on the video sequence received from the buffer 303, thereby creating the symbol 313. The parser 304 may receive the encoded data and selectively decode a specific symbol 313. In addition, the parser 304 may determine whether to provide the specific symbol 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0050] Depending on the type of the coded video picture or its portion (e.g., inter-picture vs. intra-picture, inter-block vs. intra-block) and other factors, the reconstruction of the symbol 313 may involve multiple different units. Which units are involved and how are controlled by sub-group control information parsed from the coded video sequence by the parser 304. For clarity, the flow of such sub-group control information between the parser 304 and the following multiple units is not depicted.

[0051] In addition to the functional blocks already mentioned, decoder 300 can be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and may be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, it is appropriate to conceptually subdivide into the following functional units.

[0052] The first unit is a sealer / inverse transform unit 305. The sealer / inverse transform unit 305 receives quantized transform coefficients as symbols 313 from the parser 304, as well as control information, including which transform method to use, block size, quantization factor, quantization scaling matrix, etc. The sealer / inverse transform unit 305 may output a block including sample values, which may be input into the aggregator 310.

[0053] In some cases, the output samples of the sealer / inverse transform unit 305 may belong to an intra-coded block; that is, a block that does not use predictive information from a previously reconstructed picture, but may use predictive information from a previously reconstructed portion of the current picture. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 generates surrounding blocks of the same size and shape as the block being reconstructed using reconstructed information extracted from the current (partially reconstructed) picture 309. In some cases, the aggregator 310 adds the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the sealer / inverse transform unit 305 on a per-sample basis.

[0054] In other cases, the output samples of the sealer / inverse transform unit 305 may belong to inter-frame coded and potentially motion compensated blocks. In this case, the motion compensated prediction unit 306 may access the reference picture memory 308 to extract samples for prediction. After the extracted samples are motion compensated according to the symbol 313, these samples may be added to the output of the sealer / inverse transform unit 305 (in this case referred to as residual samples or residual signals) by the aggregator 310, thereby generating output sample information. The address within the reference image memory form at which the motion compensation unit obtains the predicted samples may be controlled by a motion vector, which is provided to the motion compensation unit in the form of a symbol 313, which may have, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference picture memory when using sub-sample accurate motion vectors, motion vector prediction mechanisms, etc.

[0055] The output samples of aggregator 310 may be employed by various loop filtering techniques in loop filter unit 311. The video compression techniques may include in-loop filtering techniques controlled by parameters contained in the encoded video bitstream and provided to loop filter unit 311 as symbols 313 from parser 304, but may also be responsive to meta-information obtained during decoding of a previous (in decoding order) portion of an encoded picture or encoded video sequence, and to previously reconstructed and loop filtered sample values.

[0056] The output of the loop filter unit 311 may be a sample stream, which may be output to the rendering device 312 or may be stored in the reference picture memory 557 for future inter-picture prediction.

[0057] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future predictions. Once a coded picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 304), current reference picture 309 can become part of reference picture buffer 308, and new current picture memory can be reallocated before starting to reconstruct the next coded picture.

[0058] The video decoder 300 can perform decoding operations according to a predetermined video compression technology that can be recorded in a standard such as ITU-T Rec. H.265. The encoded video sequence can conform to the syntax specified by the video compression technology or standard used because it conforms to the syntax of the video compression technology or standard specified in the video compression technology document or standard, especially the profile document therein. Compliance also requires that the complexity of the encoded video sequence is within the range defined by the video compression technology or standard level. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (e.g., megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level can be further limited by the hypothetical reference decoder (HRD) specification and the HRD buffer management metadata emitted in the encoded video sequence.

[0059] In one embodiment, receiver 302 may receive additional (redundant) data with the encoded video. The additional data may be included as part of the encoded video sequence. Video decoder 300 may use the additional data to correctly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0060] Figure 4 is a block diagram of a video encoder 400 according to an embodiment disclosed in the present application.

[0061] Encoder 400 may receive video samples from a video source 401 (not part of the encoder), which may capture video images to be encoded by encoder 400 .

[0062] The video source 401 may provide a source video sequence to be encoded by the encoder (303) in the form of a digital video sample stream, which may be any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCB 4:2:0, Y CrCB 4:4:4). In a media service system, the video source 401 may be a storage device storing previously prepared videos. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of separate pictures that produce motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, each of which may include one or more samples depending on the sampling structure, color space, etc. used. The relationship between pixels and samples may be readily understood by those skilled in the art. The following description focuses on examples.

[0063] According to an embodiment, the video encoder 400 may encode and compress pictures of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is a function of the controller 402. In some embodiments, the controller controls other functional units as described below and is functionally coupled to these units. The parameters set by the controller may include rate control related parameters (picture skipping, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, group of pictures (group of pictures, GOP) layout, maximum motion vector search range, etc. Those skilled in the art can easily identify other functions of the controller 402 as they may be related to the video encoder 400 optimized for a specific system design.

[0064] Some video encoders operate in a manner that is readily recognizable to those skilled in the art as a "coding loop". As a simple description, the coding loop may consist of an encoding portion of an encoder 400 (hereinafter referred to as the "source encoder"), which is responsible for creating symbols based on the input picture to be encoded and the reference picture, and a (local) decoder 406 embedded in the encoder 400, which reconstructs the symbols to create sample data that will also be created by the (remote) decoder (because in the video compression techniques considered in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). This reconstructed sample stream is input to a reference picture memory 405. Since the decoding of the symbol stream results in bit-exact results that are independent of the decoder location (local or remote), the reference picture buffer contents are also bit-exact between the local encoder and the remote encoder. In other words, the reference picture samples "seen" by the prediction portion of the encoder are exactly the same sample values ​​as the decoder "sees" when using the prediction during the decoding process. The basic principles of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, for example due to channel errors) are well known to those skilled in the art.

[0065] The operation of the "local" decoder 406 can be combined with the above Figure 3 The operation of the "remote" decoder 300 described in detail is the same. However, brief reference is made to Figure 4 , since the symbols are available and the entropy encoder 408 and the parser 304 perform lossless decoding of the symbols of the encoded video sequence, the entropy decoding portion of the decoder 300, including the channel 301, the receiver 302, the buffer 303 and the parser 304, may not be fully implemented in the local decoder 406.

[0066] At this point it can be observed that, with the exception of parsing / entropy decoding present in the decoder, any decoder techniques must also be present in essentially the same functional form in the corresponding encoder. The descriptions of encoder techniques may be abbreviated, as they are the inverse of the fully described decoder techniques. A more detailed description is only required in certain areas, and is provided below.

[0067] As part of its operation, source encoder 403 may perform motion compensated predictive coding, which predictively encodes an input frame with reference to one or more previously encoded frames in a video sequence designated as "reference frames." In this manner, encoding engine 407 encodes the differences between blocks of pixels of an input frame and blocks of pixels of a reference frame that may be selected as a reference for prediction of the input frame.

[0068] The local video decoder 406 can decode the encoded video data of the frame that can be designated as the reference frame based on the symbol created by the source encoder 403. The operation of the encoding engine 407 can advantageously be a lossy process. When the encoded video data can be decoded at the video decoder ( Figure 4 When decoded at a remote video decoder (not shown), the reconstructed video sequence may generally be a copy of the source video sequence, but with some errors. The local video decoder 406 replicates the decoding process that may be performed by the video decoder on the reference frame, and may cause the reconstructed reference frame to be stored in a reference picture memory 405, which may be, for example, a cache. In this way, the encoder 400 may locally store a copy of the reconstructed reference frame having common content as the reconstructed reference frame to be obtained by the remote video decoder (without transmission errors).

[0069] The predictor 404 may perform a prediction search for the encoding engine 407. That is, for a new picture to be encoded, the predictor 404 may search the reference picture memory 405 for sample data (as a candidate reference pixel block) or certain metadata, such as reference picture motion vectors, block shapes, etc., that may serve as a suitable prediction reference for the new picture. The predictor 404 may operate on the sample blocks on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor 404, the input picture may have a prediction reference drawn from a plurality of reference pictures stored in the reference picture memory 405.

[0070] The controller 402 may manage encoding operations of the source encoder 403, including, for example, setting parameters and subgroup parameters for encoding video data.

[0071] The outputs of all the above functional units may be entropy encoded in an entropy encoder 408. The entropy encoder converts the symbols generated by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art (e.g., Huffman coding, variable length coding, arithmetic coding, etc.).

[0072] The transmitter 409 can buffer the encoded video sequence created by the entropy encoder 408 in preparation for transmission via the communication channel 411, which can be a hardware / software link to a storage device storing the encoded video data. The transmitter 409 can merge the encoded video data from the source encoder 403 with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0073] The controller 402 may manage the operation of the encoder 400. During encoding, the controller 402 may assign a specific coded picture type to each coded picture, which may affect the coding techniques that may be applied to the corresponding picture. For example, a picture may generally be assigned one of the following frame types:

[0074] An intra picture (I picture) may be a picture that can be encoded and decoded without using any other frame in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including independent decoder refresh pictures. Those skilled in the art are aware of these variations of I pictures and their respective applications and features.

[0075] A predicted picture (P picture) may be a picture that may be encoded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict sample values ​​for each block.

[0076] Bidirectional prediction pictures (B pictures) can be pictures that can be encoded and decoded using intra prediction or inter prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple prediction pictures can use more than two reference pictures and related metadata to reconstruct a single block.

[0077] The source picture may typically be spatially subdivided into a number of blocks of samples (e.g., each block of samples is 4x 4, 8x 8, 4x8, or 16x 16) and encoded block by block. Blocks may be predictively encoded with reference to other (already encoded) blocks determined by the coding assignment applied to the block's corresponding picture. For example, blocks of an I picture may be non-predictively encoded or predictively encoded (spatial prediction or intra prediction) with reference to already encoded blocks of the same picture. Blocks of pixels of a P picture may be non-predictively encoded by spatial prediction or by temporal prediction with reference to one previously encoded reference picture. Blocks of a B picture may be non-predictively encoded by spatial prediction or by temporal prediction with reference to one or two previously encoded reference pictures.

[0078] The encoder 400, which may be a video encoder, may perform encoding operations according to a predetermined video encoding technique or standard (e.g., ITU-T Rec. H.265). In its operation, the encoder 400 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in an input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard used.

[0079] In one embodiment, the transmitter 409 may send additional data along with the encoded video. The source encoder 403 may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, supplemental enhancement information (SEI) messages, visual usability information (VUI) parameter set fragments, etc.

[0080] Figure 5A simplified block workflow diagram 500 of exemplary view-port related processing in the Omnidirectional Media Application Format (OMAF) is illustrated, which may allow 360-degree virtual reality (VR360) streaming as described in OMAF.

[0081] At acquisition block 501, video data A is acquired, such as data of multiple images and audio of the same time instance, in the case where the image data may represent a scene in VR360. At processing block 503, the images Bi of the same time instance are processed, including one of the following: stitching, mapping to a projection picture relative to one or more virtual reality (VR) angles or other angles / one or more viewpoints, and region-wise packing. Additionally, metadata may be created indicating any of such processed information and other information to assist the distribution process and rendering process.

[0082] Regarding data D, at the image encoding box 505, the projected picture is encoded as data Ei and constitutes a media file, and in viewport-independent streaming, at the video encoding box 504, the video picture is encoded as data Ev as a single-layer code stream, for example, regarding data Ba, audio data can also be encoded into data Ea at the audio encoding box 502.

[0083] Data E a 、E v and E i , the entire encoded bitstream F i and / or F may be stored at a (content delivery network (CDN) / cloud) server and may typically be fully transmitted to an OMAF player 520, such as at a distribution block 507 or otherwise, and may be fully decoded by a decoder such that at least one region of the decoded picture corresponding to the current viewport is rendered to the user, at a display block 516, with respect to various metadata, file playback, and orientation / viewport metadata, such as the angle from which the user may view through the VR image device from a head / eye detection block 508 relative to the viewport specifications of the device. A unique feature of VR360 is that only a viewport may be displayed at any particular time, and such a feature may be used to improve the performance of omnidirectional video systems by selective distribution depending on the user's viewport (or any other criteria, such as recommended viewport timing metadata). For example, according to an exemplary embodiment, viewport-dependent distribution may be enabled by tile-based video encoding.

[0084] According to an exemplary embodiment, like the above-mentioned encoding frame, the OMAF player 520 may perform a iand metadata, and similarly reverses one or more aspects of such encoding. i Decode the video data E' at the video decoding block 513 v Decode and decode the image data E' at the image decoding block 514 i Decode to continue to process the data B' at the audio rendering block 511 a Audio rendering is performed and image rendering is performed on the data D' at the image rendering block 515 so as to output display data A' at the display block 516 in a VR360 format based on various metadata such as direction / viewport metadata i , and outputs audio data A' at the speaker / headphone frame 512 s Various metadata may affect some processes in the data decoding and rendering process according to various tracks, languages, qualities, views that may be selected by or for a user of the OMAF player 520, and it should be understood that the processing order described herein is presented for an exemplary embodiment and may be implemented in other orders according to other exemplary embodiments.

[0085] Figure 6 A simplified block content flow process diagram 600 of (encoded) point cloud data with view position and angle dependent processing (referred to as "V-PCC" in this application) for capturing / generating / (de)encoding / rendering / displaying 6-DOF media is illustrated. It should be understood that the features described may be used alone or in combination in any order, and that elements such as for encoding and decoding and other elements illustrated may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits), and according to an exemplary embodiment, the one or more processors may execute a program stored in a non-transitory computer-readable medium.

[0086] Diagram 600 illustrates an exemplary embodiment for streaming of encoded point cloud data according to V-PCC.

[0087] At the volume data acquisition block 601, a real-world visual scene or a computer-generated visual scene (or a combination thereof) may be captured by a set of camera devices or synthesized by a computer as volume data, and the volume data in any format may be converted to a (quantized) point cloud data format by image processing at the conversion to point cloud block 602. For example, according to an exemplary embodiment, data in the volume data may be converted to points in a point cloud region by region by pulling one or more of the following values ​​from the volume data and any related data into a desired point cloud format. According to an exemplary embodiment, the volume data may be a 3D data set of a 2D image, such as a strip of a 2D projection from which a 3D data set may be projected. According to an exemplary embodiment, the point cloud data format includes a representation of data points in one or more various spaces, and may be used to represent volume data, and may provide improvements in sampling and data compression (such as in terms of temporal redundancy), and for example, point cloud data in an x, y, z format represents a color value (e.g., RGB, etc.), brightness, intensity, etc. at each of a plurality of points in the cloud data, and may be used with progressive decoding, polygon meshing, direct rendering, and an octree 3D representation of 2D quadtree data.

[0088] At the projection to image block 603, the acquired point cloud data may be projected onto a 2D image and encoded into an image / video picture using video-based point cloud coding (V-PCC). The projected point cloud data may consist of attributes, geometry, occupancy maps, and other metadata for point cloud data reconstruction, such as using a painter's algorithm, a ray casting algorithm, a (3D) binary space partitioning algorithm, and the like.

[0089] On the other hand, at the scene generator block 609, the scene generator may generate some metadata to be used for rendering and displaying 6 degrees of freedom (DoF) media, for example, as intended by the director or as preferred by the user. Such 6DoF media may include 3D-like 360VR, where the scene may be viewed through rotational changes on the 3D axes X, Y, Z (in addition to the additional dimension), allowing forward, backward, up, down, and left, right movement in the virtual experience relative to or at least based on the point cloud encoded data. The scene description metadata defines one or more scenes consisting of the encoded point cloud data and other media data (including VR360, light field, audio, etc.), and may be provided to one or more cloud servers and / or file / segment encapsulation / decapsulation processing, such as Figure 6 and as indicated in the relevant description.

[0090] Following the video encoding block 604 and the image encoding block 605 similar to the video and image encoding described above (and as will be appreciated, audio encoding may also be provided as described above), the file / segment encapsulation block 606 processes such that the encoded point cloud data is assembled into a media file for file playback or a sequence of initialization segments and media segments for streaming according to a particular media container file format (such as one or more video container formats), and such as may be used for DASH described below, where such descriptions represent exemplary embodiments. The file container may also include scene description metadata such as from the scene generator block 1109 into the file or segment.

[0091] According to an exemplary embodiment, a file is packaged according to scene description metadata to include at least one view position, at least one or more angle views at the view position, each of the view positions at one or more times in the 6DoF media, such that the files are transmitted according to a request input by a user or creator. Further, according to an exemplary embodiment, a segment of such a file may include one or more portions of such a file, such as a portion of the 6DoF media, which indicates a single viewpoint and angle at one or more times; however, these are merely exemplary embodiments and may change according to various conditions such as network, user, creator capabilities and input.

[0092] According to an exemplary embodiment, the point cloud data is partitioned into a plurality of 2D regions / 3D regions, such as being independently encoded at one or more of the video encoding block 604 and the image encoding block 605. Each independently encoded partition of the point cloud data may then be packaged as a track in a file and / or segment at the file / segment packaging block 606. According to an exemplary embodiment, each point cloud track and / or metadata track may include some useful metadata for view position / angle related processing.

[0093] According to an exemplary embodiment, metadata useful for view position / angle related processing (such as included in a file and / or segment encapsulated with respect to a file / segment encapsulation box) includes one or more of the following: layout information of 2D / 3D partitions with indices, (dynamic) mapping information associating a 3D volume partition with one or more 2D partitions (e.g., any of tiles / tile groups / strips / sub-pictures), 3D position of each 3D partition on a 6DoF coordinate system, a representative view position / angle list, a selected view position / angle list corresponding to the 3D volume partition, an index of a 2D / 3D partition corresponding to the selected view position / angle, quality (grade) information of each 2D / 3D partition, and rendering information of each 2D / 3D partition depending on each view position / angle, for example. When a user of a V-PCC player makes a request or a content creator instructs a user of a V-PCC player to make a request, calling such metadata can allow more efficient processing of specific portions of a 6DoF media required with respect to such metadata, so that the V-PCC player can distribute images of focused portions of the 6DoF media of higher quality than other portions, rather than distributing unused portions of the media.

[0094] From the file / segment encapsulation box 606, the file or one or more segments of the file can be distributed directly to either the V-PCC player 625 and the cloud server using a distribution mechanism (e.g., via Dynamic Adaptive Streaming over HTTP (DASH)), such as at the cloud server box 607, where the cloud server can extract one or more tracks and / or one or more specific 2D / 3D partitions from the file, and can merge multiple encoded point cloud data into one data.

[0095] Based on data such as the position / view detection box 608, if the current view position and one or more view positions are defined on a 6DoF coordinate system at the client system, the view position / view position metadata can be distributed from the file / segment encapsulation box 606, or the view position / view position metadata can be processed at the cloud server box 607 based on files or segments already at the cloud server, so that the cloud server can extract the appropriate one or more partitions from one or more storage files and merge them (if necessary) based on the metadata from the client system with the V-PCC player 625, and the extracted data can be distributed to the client as files or segments.

[0096] With respect to such data, at the file / segment decapsulation box 615, the file decapsulator processes the file or received segment and extracts the encoded code stream and parses the metadata, and at the video decoding box 610 and the image decoding box 611, the encoded point cloud data is decoded and reconstructed into point cloud data at the point cloud reconstruction box 612, and the reconstructed point cloud data can be displayed at the display box 614, and / or can first be synthesized at the scene synthesis box 613 according to one or more different scene descriptions (involving scene description data according to the scene generator box 609).

[0097] In view of the above, this exemplary V-PCC process embodies advantages over the V-PCC standard, including the ability to partition multiple 2D / 3D regions described by one or more, the ability to assemble coded 2D / 3D partitions into a single standard-compliant coded video stream in the compressed domain, and the ability to extract coded 2D / 3D of a coded picture into a standard-compliant coded stream, wherein this V-PCC system support is further improved by including a container build for the VVC stream to support a mechanism for carrying one or more of the above-mentioned metadata.

[0098] In view of this and in accordance with exemplary embodiments further described below, the term "mesh" refers to a combination of one or more polygons that describe the surface of a volumetric object. Each polygon is defined by its vertices in 3D space and information about how these vertices are connected (referred to as connectivity information). Optionally, vertex attributes (such as color, normals, etc.) can be associated with mesh vertices. Attributes can also be associated with the surface of the mesh by mapping information based on parameterizing the mesh with a 2D attribute map. Such a mapping can be described by a set of parameter coordinates (referred to as UV coordinates or texture coordinates) associated with the mesh vertices. The 2D attribute map is used to store high-resolution attribute information, such as textures, normals, displacements, etc. According to exemplary embodiments, such information can be used for various purposes such as texture mapping and shading.

[0099] Nevertheless, a dynamic mesh sequence may require a large amount of data because it may consist of a large amount of information that changes over time. For example, in contrast to a "static mesh" or "static mesh sequence" (where the information of the mesh may not change from one frame to another), a "dynamic mesh" or "dynamic mesh sequence" indicates a motion in which some of the vertices represented by the mesh change from one frame to another. Therefore, in order to store and transmit such content, efficient compression techniques are required. Mesh compression standards IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute graphs and connectivity information. DCC (digital content creation) tools typically generate such dynamic meshes. In contrast, for volume acquisition technology, generating constant connectivity dynamic meshes is a challenge, especially under real-time constraints. Existing standards do not support this type of content. According to exemplary embodiments of the present application, various aspects of a new mesh compression standard are described to directly handle dynamic meshes with time-varying connectivity information and optional time-varying attribute graphs, which are lossy and lossless compression for various applications such as real-time communication, storage, free viewpoint video, AR and VR. Features such as random access and scalable / progressive coding are also considered.

[0100] Figure 7 An example framework 700 for a dynamic mesh compression such as for a method based on 2D atlas sampling is shown. Each frame of the input mesh 701 can be pre-processed by a series of operations (e.g., detection, re-meshing, parameterization, voxelization). Note that these operations can be encoder-only, which means that they may not be part of the decoding process, and this possibility can be signaled in the metadata by a flag (such as indicating that only the encoder is 0 and the others are 1). After that, a mesh with a 2D UV atlas 702 can be obtained, where each vertex of the mesh has one or more associated UV coordinates on the 2D atlas. Then, by sampling on the 2D atlas, the mesh can be converted into multiple graphs, including geometric graphs and attribute graphs. These 2D graphs can then be encoded by a video / image codec, such as HEVC, VVC, AV1, AVS3, etc. On the decoder 703 side, the mesh can be reconstructed from the decoded 2D graph. Any post-processing and filtering can also be applied to the reconstructed mesh 704. Note that other metadata can be signaled to the decoder side for the purpose of 3D mesh reconstruction. Note that the graph boundary information (including uv coordinates and xyz coordinates) of the boundary vertices can be predicted, quantized, and entropy encoded in the bitstream. The quantization step size can be configured at the encoder side to trade off between quality and bit rate.

[0101] In some embodiments, a 3D mesh may be partitioned into segments (or patches / graphs), and according to an exemplary embodiment, one or more 3D mesh segments may be considered a "3D mesh". Each segment may consist of a set of connected vertices associated with its geometry, attributes, and connectivity information. Figure 8 As shown in the example 800 of volume data in FIG. 1 , a UV parameterization process 802 for mapping from 3D mesh segments to a 2D graph (such as the 2D UV atlas 702 block described above) maps one or more mesh segments 801 to a 2D graph 803 in a 2D UV atlas 804. Each vertex (v n ) will be assigned 2D UV coordinates in the 2D UV atlas. Note that the vertices (v n ) form connected components that are their 3D counterparts. The geometry, attributes, and connectivity information of each vertex may also be inherited from its 3D counterpart. For example, information indicating that vertex v4 is directly connected to vertices v0, v5, v1, and v3 may be indicated, and similarly, information for each of the other vertices may also be indicated. In addition, according to an exemplary embodiment, such a 2D texture mesh will further indicate information (such as color information) on a patch-by-patch basis, such as by having each triangle patch, e.g., v2, v5, v3 as one "patch".

[0102] For example, in addition to Figure 8 In addition to the features of Example 800, see Fig. 9 Example 900, where 3D mesh segment 801 can also be mapped to multiple separate 2D graphs 901 and 902. In this case, a vertex in the 3D can correspond to multiple vertices in the 2D UV atlas. Fig. 9 As shown, in a 2D UV atlas, the same 3D mesh segment is mapped to multiple 2D charts instead of Figure 8 For example, 3D vertices v1 and v4 each have two 2D correspondences v1, v4, respectively. 1’ and v4, v 4’ Therefore, a general 2D UV atlas for a 3D mesh can be represented by Fig.21 The diagram is composed of multiple graphs as shown, where each graph may contain multiple (usually greater than or equal to 3) vertices associated with its 3D geometry, attributes, and connectivity information.

[0103] Fig. 9Example 903 is shown, which illustrates a derived triangulation with boundary vertices B0, B1, B2, B3, B4, B5, B6, B7 in a graph. When such information is presented, any triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampling vertices). For example, for each vertex, find the two closest vertices. Or for all vertices, continuously generate triangles until a minimum number of triangles is reached after a set number of attempts. As shown in example 903, there are various regular-shaped, repeating triangles and various odd-shaped triangles, which are usually closest to the boundary vertices and have unique sizes themselves, which may or may not be shared with any other triangles. Connectivity information can also be reconstructed by explicit signaling. According to an exemplary embodiment, if the polygon cannot be recovered by implicit rules, the encoder can signal connectivity information in the code stream.

[0104] Boundary vertices B0, B1, B2, B3, B4, B5, B6, B7 are defined in 2D UV space. Boundary edges can be determined by checking whether the edge only appears in one triangle. According to an exemplary embodiment, the following information of the boundary vertices is important and should be signaled in the codestream: geometric information, such as 3D XYZ coordinates (even if currently in 2D UV parameter form), and 2D UV coordinates.

[0105] For cases where a boundary vertex in 3D corresponds to multiple vertices in the 2D UV atlas, such as Fig. 9 As shown, the mapping from 3D XUZ to 2D UV can be one-to-many. Therefore, a UV to XYZ (or referred to as UV2XYZ) index can be signaled to indicate the mapping function. UV2XYZ can be a 1D array of indices that correspond each 2D UV vertex to a 3D XYZ vertex.

[0106] According to an exemplary embodiment, in order to efficiently represent a mesh signal, a subset of mesh vertices and connectivity information between them may be encoded first. In the original mesh, the connections between these vertices may not exist because they are subsampled from the original mesh. There are different ways to signal the connectivity information between vertices, and thus this subset is referred to as a base mesh or base vertices.

[0107] According to an exemplary embodiment, a number of methods for dynamic mesh compression are implemented, and these methods are part of the above-mentioned edge-based vertex prediction framework, where the base mesh is first encoded, and then more additional vertices are predicted based on the connectivity information of the edges from the base mesh. Note that these methods can be applied individually or in any form of combination.

[0108] For example, consider Fig.10 Vertex grouping of the prediction mode example flow chart 1001. At S101, vertices inside the mesh can be obtained, and at S102, they can be divided into different groups for prediction purposes, for example, see Fig. 9 . In one example, the partitioning is done using patch / graph partitioning at S104. In another example, the partitioning S105 is done under each patch / graph. The decision S103 whether to proceed to S104 or S105 can be signaled by a flag or the like. In the case of S105, several vertices of the same patch / graph form a prediction group and will share the same prediction mode, while several other vertices of the same patch / graph can use another prediction mode. In the present application, "prediction mode" can be considered as a specific mode used by the decoder to predict the video content including the patch, and the prediction mode can be clearly divided into intra-frame prediction mode and inter-frame prediction mode, and within each category, there can be different specific modes from which the decoder selects. According to an exemplary embodiment, each group ("prediction group") can share the same specific mode (e.g., an angle mode of a specific angle) or the same classification prediction mode (e.g., all intra-frame prediction modes, but predictions can be made at different angles) according to an exemplary embodiment. By determining the corresponding number of vertices involved in each group, such grouping at S106 can be allocated at different levels. For example, according to an exemplary embodiment, every 64, 32, or 16 vertices following the scan order within the patch / graph will be assigned the same prediction mode, and other vertices may be assigned differently. For each group, the prediction mode may be an intra-frame prediction mode or an inter-frame prediction mode. This may be signaled or assigned. According to the example flowchart 1000, if it is determined at S107 that the mesh frame or mesh strip is of intra-frame type, such as by checking whether a flag of the mesh frame or mesh strip indicates an intra-frame type, then all vertex groups within the mesh frame or mesh strip should use the intra-frame prediction mode; otherwise, at S108, the intra-frame prediction mode or the inter-frame prediction mode may be selected by group for all vertices therein.

[0109] In addition, for a set of mesh vertices using intra-frame prediction mode, its vertices can only be predicted by using previously encoded vertices within the same sub-partition of the current mesh. According to an exemplary embodiment, the sub-partition may sometimes be the current mesh itself, and according to an exemplary embodiment, for a set of mesh vertices using inter-frame prediction mode, its vertices can only be predicted by using previously encoded vertices from another mesh frame. Each of the above information can be determined and signaled by a flag, etc. The prediction feature can occur at S110, and the result of the prediction can be signaled at S111.

[0110] According to an exemplary embodiment, for each vertex in a set of vertices in the example flowchart 1000 and the flowchart 1100 described below, after prediction, the residual will be a 3D displacement vector indicating the displacement from the current vertex to its predicted value. It is necessary to further compress the residual of a set of vertices. In one example, the transformation at S111 and its signaling can be applied to the residual of the vertex group before entropy coding. The following method can be implemented to handle the coding of a set of displacement vectors. For example, in one method, a set of displacement vectors, some displacement vectors or their components are correctly signaled to have only zero values. In another embodiment, a flag is signaled for each displacement vector whether the vector has any non-zero components, and if not, the coding of all components of the displacement vector can be skipped. In addition, in another embodiment, a flag is signaled for each group of displacement vectors whether the group has any non-zero vectors, and if not, the coding of all displacement vectors of the group can be skipped. In addition, in another embodiment, a flag is signaled for each component of a set of displacement vectors whether the component of the group has any non-zero vectors, and if not, the coding of all displacement vectors of the group can be skipped. In addition, in another embodiment, a flag is signaled for each component of a set of displacement vectors whether the component of the group has any non-zero vectors, and if not, the coding of the component of all displacement vectors of the group can be skipped. Furthermore, in another embodiment, a signal may be sent to indicate whether a group of displacement vectors or components of the group of displacement vectors require a transformation, and if a transformation is not required, the transformation may be skipped and quantization / entropy coding may be applied directly to the group or components of the group. Furthermore, in another embodiment, a flag may be signaled for each group of displacement vectors as to whether the group requires a transformation, and if not, the transformation coding of all displacement vectors of the group may be skipped. Furthermore, in another embodiment, a flag may be signaled for each component of a group of displacement vectors as to whether the component of the group requires a transformation, and if not, the transformation coding of the component of all displacement vectors of the group may be skipped. The above-mentioned embodiments of the processing of vertex prediction residuals in this paragraph may also be combined and implemented in parallel on different patches respectively.

[0111] Fig.11An example flow chart 1100 is shown, wherein at S121, a grid frame encoded as an entire data unit may be obtained, which means that there may be correlations between all vertices or attributes of the grid frame. Alternatively, based on the determination at S122, the grid frame may be divided into smaller independent sub-partitions at S123, conceptually similar to strips or tiles in a 2D video or image. At S124, a prediction type may be assigned to the encoded grid frame or the encoded grid sub-partition. Possible prediction types include intra-frame coding types and inter-frame coding types. For the intra-frame coding type, at S125, only predictions from the reconstructed portion of the same frame or strip are allowed. On the other hand, at S125, in addition to the intra-frame grid frame prediction, the inter-frame prediction type will allow predictions based on previously encoded grid frames. In addition, the inter-frame prediction type may be classified into more sub-types, such as P type or B type. In the P type, only one prediction value may be used for prediction purposes, while in the B type, two prediction values ​​from two previously encoded grid frames may be used to generate a prediction value. A weighted average of two prediction values ​​may be an example. When a grid frame is encoded as a whole, the frame may be considered an intra-coded grid frame or an inter-coded grid frame. In the case of an inter-frame grid frame, the P type or B type may be further identified via signaling. Alternatively, if the grid frame is encoded within the frame with further segmentation, a prediction type is assigned to each of the sub-partitions at S124. Each of the above information may be determined and signaled by a flag, etc., and may be associated with the grid frame. Fig.10 Similar to S110 and S111 , the predicted feature may be generated at S126 , and the predicted result may be signaled at S127 .

[0112] Thus, while a dynamic mesh sequence may require a large amount of data as it may consist of a large amount of information that changes over time, efficient compression techniques are required to store and transmit such content, and the features described in this application represent such improved efficiency by at least allowing improved mesh vertex 3D position prediction, which is achieved using previously decoded vertices in the same mesh frame (intra-frame prediction) or previously decoded vertices from previously encoded mesh frames (inter-frame prediction).

[0113] In addition, exemplary embodiments may generate displacement vectors of a third layer 1303 of the mesh based on one or more reconstructed vertices of its previous layers (such as the second layer 1302 and the first layer 1301). Assuming that the index of the second layer 1302 is T, the predicted values ​​of the vertices in the third layer 1303T+1 are generated based on at least the reconstructed vertices of the current layer or the second layer 1302. Examples of such layer-based prediction structures are as follows: Fig.13 As shown in Example 1300, Fig.13Illustrated is reconstruction-based vertex prediction: progressive vertex prediction using edge-based interpolation, where predicted values are generated based on previously decoded vertices (rather than predicted-value vertices). The first layer 1301 can be a mesh bounded by a first polygon 1340 that has decoded vertices at its boundaries as its vertices and interpolated vertices along some of the lines between some of those decoded vertices. As progressive encoding proceeds from the first layer 1301 to the second layer 1302, additional polygons 1341 can be formed by displacement vectors from some of the interpolated vertices of the first layer to additional vertices of the second layer 1302, and thus, the total number of vertices in the second layer 1302 can be greater than the total number of vertices in the first layer 1301. Similarly, after advancing to the third layer 1303, the additional vertices of the second layer 1302, as well as the decoded vertices from the first layer 1301, can participate in encoding in a manner similar to how the decoded vertices advanced from the first layer 1301 to the second layer 1302 participated in encoding; that is, multiple additional polygons can be formed. As described above, refer to Example 1400 illustrated in Fig.21 where, different from Fig.13 Example 1400 illustrates that in the process of proceeding from the first layer 1401 to the second layer 1402 and then to the third layer 1403, each of the additional polygons formed can be entirely located within the polygon formed by the boundary of the first layer 1401.

[0114] According to an exemplary embodiment, for such Example 1300 and / or 1400, refer to Fig.12 Example flowchart 1200, where since the interpolated vertices on the current layer are predicted values, such values need to be reconstructed before being used to generate predicted values for vertices on the next layer. This is done by encoding the base mesh at S131, implementing vertex prediction at S132, and then adding the decoded displacement vectors of the current layer to the predicted values of vertices such as those of layer 1302 at S133. Then, the reconstructed vertices of this layer and all the decoded vertices of one or more previous layers (such as checking the additional vertex values of such layers at S134) can be used together to generate the predicted-value vertices for the next layer 1303 at S135. The process can also be summarized as follows: Let P[t](Vi) denote the predicted value of vertex Vi on layer t; Let R[t](Vi) denote the reconstructed vertex Vi on layer t; Let D[t](Vi) denote the displacement vector of vertex Vi on layer t; Let f(*) denote the predicted-value generator, specifically, it can be the average of two existing vertices. According to an exemplary embodiment, then for each layer t, there is the following:

[0115] P[t](Vi) = f(R[s|s<t](Vj), R[m|m<t](Vk)), where Vj and Vk are reconstructed vertices of previous layers

[0116] R[t](Vi)=P[t](Vi)+D[t](Vi) - Equation (1)

[0117] Then, for all vertices in one mesh frame, they are divided into layer 0 (base mesh), layer 1, layer 2, etc. Then, the reconstruction of vertices on one layer depends on the reconstruction of those vertices on the previous layer or layers. In the above, each of P, R, and D represents a 3D vector in the context of a 3D mesh representation. D is a decoded displacement vector, and quantization may or may not be applied to this vector.

[0118] According to an exemplary embodiment, vertex prediction using reconstructed vertices may be applied only to certain layers. For example, layer 0 and layer 1. For other layers, vertex prediction may still use adjacent predictor vertices without adding displacement vectors to them for reconstruction. In this way, these other layers may be processed simultaneously without waiting for the previous layer to be reconstructed. According to an exemplary embodiment, for each layer, it may be signaled whether to select vertex prediction based on reconstruction or vertex prediction based on prediction values, or it may be signaled that the layer (and its subsequent layers) not using vertex prediction based on reconstruction is used.

[0119] For displacement vectors whose vertex predictors are generated by reconstructed vertices, quantization can be applied to them without further performing a transform, such as a wavelet transform, etc. For displacement vectors whose vertex predictors are generated by other predictor vertices, a transform may be required, and quantization can be applied to the transform coefficients of those displacement vectors.

[0120] Therefore, a dynamic mesh sequence may require a lot of data because it may consist of a lot of information that changes over time. Therefore, in order to store and transmit such content, efficient compression technology is required. In the framework of the above-mentioned interpolation-based vertex prediction method, an important procedure is to compress the displacement vector, and this procedure occupies a large part in the encoded bitstream, and the focus and features of this application alleviate this problem by providing such compression.

[0121] Furthermore, similar to the other examples described above, even for those embodiments, a dynamic mesh sequence may still require a large amount of data, since it may consist of a large amount of information that changes over time, and therefore, in order to store and transmit such content, efficient compression techniques are required. In the framework of the 2D atlas sampling-based approach indicated above, important advantages can be achieved by inferring connectivity information from the sampled vertices plus the boundary vertices on the decoder side. This is a major part of the decoding process and is the focus of the additional examples described below.

[0122] According to an exemplary embodiment, connectivity information of the base mesh may be inferred (derived) from the decoded boundary vertices and sample vertices of each graph on both the encoder side and the decoder side.

[0123] As described above, any triangulation method may be applied to create connectivity between vertices (including boundary vertices and sampling vertices).According to an exemplary embodiment, the connectivity type may be signaled in a high-level syntax (such as a sequence header, a stripe header).

[0124] As described above, connectivity information can also be reconstructed through explicit signaling, for example, for irregularly shaped triangular meshes. That is, if it is determined that the polygon cannot be recovered through implicit rules, the encoder can signal connectivity information in the code stream. And according to an exemplary embodiment, the overhead of such explicit signaling can be reduced according to the boundaries of the polygon.

[0125] According to an embodiment, only the connectivity information between boundary vertices and sampling locations is determined to be signaled, while the connectivity information between the sampling locations themselves is inferred.

[0126] Furthermore, in any of the embodiments, connectivity information may be signaled by prediction, such that only the difference in inferred connectivity (as a prediction) from one mesh to another may be signaled in the codestream.

[0127] Note that according to an exemplary embodiment, the orientation of the inferred triangles (such as whether each triangle is to be inferred in a clockwise or counterclockwise manner) may be signaled for all charts in a high-level syntax (such as a sequence header, a stripe header, etc.), or may be fixed (assumed) by the encoder and decoder. The orientation of the inferred triangles may also be signaled differently for each chart.

[0128] Also note that any reconstructed mesh may have different connectivity than the original mesh. For example, the original mesh may be a triangular mesh, while the reconstructed mesh may be a polygonal mesh (e.g., a quadrilateral mesh).

[0129] According to an exemplary embodiment, connectivity information of any base vertex may not be signaled, but the same algorithm may be used at the encoder side and the decoder side to derive edges between base vertices. And according to an exemplary embodiment, interpolation of predicted vertices of additional mesh vertices may be based on the derived edges of the base mesh.

[0130] According to an exemplary embodiment, a flag may be used to signal whether connectivity information of base vertices is to be signaled or derived, and such a flag may be signaled at different levels of a codestream (eg, at a sequence level, a frame level, etc.).

[0131] According to an exemplary embodiment, the edges between the base vertices are first derived using the same algorithm on both the encoder and decoder sides. This is then compared to the original connectivity of the base mesh vertices, and the difference between the derived edges and the actual edges is signaled. Thus, after decoding the difference, the original connectivity of the base vertices can be restored.

[0132] In one example, for a derived edge, if it is determined to be erroneous when compared to an original edge, such information can be signaled in the codestream (by indicating the vertex pair that forms the edge); and for an original edge, if it is not derived, this can be signaled in the codestream (by indicating the vertex pair that forms the edge). In addition, connectivity on boundary edges and interpolation of vertices involving boundary edges can be performed separately from internal vertices and edges.

[0133] Therefore, through the exemplary embodiments described in this application, the above-mentioned technical problems can be advantageously improved through one or more of these technical solutions. For example, a dynamic grid sequence may require a large amount of data because it may consist of a large amount of information that changes over time, and therefore, the exemplary embodiments described in this application at least represent efficient compression techniques for storing and transmitting such content.

[0134] The above embodiments can be further applied to instance-based mesh coding, where the instance can be a mesh of an object or a part of an object. Fig.15 The illustrated example 1500 of 1501 illustrates a grid example 1501, in which there are various instances 1502 (a grid representing a cup), 1503 (a grid representing a spoon), and 1504 (a grid representing a plate), and they can be separated and encoded separately. And each of the instances 1501, 1502, 1503, and 1504 is illustrated in a corresponding bounding box in a bounding box, but note that the instance 1501 can be considered to be illustrated as bounded by a "grid-based bounding box", while each of the instances 1502, 1503, and 1504 can be considered to be illustrated as bounded by a corresponding bounding box in an "instance-based bounding box".

[0135] The present application embodiment also considers triangular mesh in more detail. Triangular mesh is a type of polygonal mesh, including a group of triangles connected by their common edges or common angles, which are mesh vertices. Similar methods disclosed in the present application can be extended to other polygonal meshes, such as quadrilateral meshes or pentagonal meshes.

[0136] Lossless mesh compression by Fig.16Example 1600 in is illustrated. Duplicate geometric vertices are first merged. During this process, non-manifold structures should be avoided because they will be confused with the original non-manifold structures and more processing will be spent on processing the non-manifold structures. Before using the edgebreaker to encode the connectivity, the non-manifold structure in the input mesh must be converted to a manifold structure. The connectivity information of the manifold mesh is then encoded by the Edgebreaker. The geometric information and attribute information are encoded by using a predictive encoder. The texture map is compressed by using a video codec. On the decoder side, duplicate vertices and non-manifolds are losslessly reconstructed.

[0137] In lossless mesh compression, geometric compression consists of vertex position compression, 2D texture coordinate compression, and other geometric information (such as normals, etc.) compression. And the present application discloses embodiments of a method and system for vertex position compression for lossless mesh compression. One of the creative ideas of vertex position compression according to the embodiments of the present application is parallelogram prediction. The compression algorithm introduces a new vertex of a triangle from an edge. The predicted position of the new vertex forms a parallelogram with two shared vertices and the third vertex of the opposite triangle, such as Fig.17 1700 is shown in element 1701. Whenever possible, the multi-parallelogram prediction uses the average position given by two or more parallelogram predictions. And element 1702 gives an example of a dual parallelogram prediction.

[0138] The proposed methods can be used alone or in combination in any order. In addition, each of the method (or embodiment), encoder and decoder can be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-volatile computer-readable medium.

[0139] In this application, a number of methods and systems for vertex position encoding in mesh compression are proposed. Note that these methods and systems can be applied individually or in any form of combination. It should also be noted that these methods can be applied not only to dynamic meshes, but also to static meshes, where the mesh has only one frame, or the mesh content does not change over time. In addition, the disclosed methods and systems are not limited to vertex position encoding. They can also be applied to, for example, 2D texture coordinate encoding, as a more general multi-prediction based scheme.

[0140] All triangles in a triangle mesh may be sorted. The order may be a traversal in an Edgebreaker algorithm or other algorithms. All vertices may be sorted. The order may be a traversal in an Edgebreaker algorithm or other algorithms. The order of triangles and vertices may be based on the same scheme (such as a traversal) or different schemes.

[0141] In a triangle mesh, each triangle (also called a face) has three vertices. For two triangles that share an edge, we can apply parallelogram prediction to use one of the two opposite vertices as the predicted value of the other. Fig.18 As shown in example 1800, two triangles ABC and DBC share edge BC. The present application embodiment uses the coordinates of A, B, and C (eg, assuming they have been encoded) to predict the coordinates of D.

[0142] The predicted coordinates D' will be

[0143] D' = B + C – A - Equation 2

[0144] Thus, the four vertices (D', B, A, C) form a parallelogram. Since each vertex has a 3D coordinate, the above equation calculates each coordinate component-wise. For example, assuming that the subscripts x, y, z represent 3D coordinates in xyz space, then

[0145] D x '=B x +C x –A x -Equation 3

[0146] D y '=B y +C y –A y -Equation 4

[0147] D z '=B z +C z –A z -Equation 5

[0148] If the position values ​​of A, B, and C have been encoded and can be used for prediction, the embodiment of the present application refers to triangle ABC as a prediction candidate for vertex D. A vertex can have zero, one, two, or more prediction candidates, depending on the number of existing encoded vertices and shared edges. If there is more than one prediction candidate, these prediction candidates are sorted based on the triangle order of the mesh.

[0149] According to an embodiment of the present application, the vertex position encoding calculates the average value of the available parallelogram predictions and uses the average value as the prediction value. The associated prediction error (also called prediction residual) is encoded.

[0150] In addition, according to an embodiment of the present application, there is position prediction. In a triangle mesh, all triangles are sorted. In addition, all vertices are sorted.

[0151] Given a vertex V in a triangular mesh, assume that there are N predicted values ​​V1', V2', ..., V N ', where they are sorted based on the triangle order of the mesh.

[0152] If N = 0, there is no prediction candidate. If V is the first vertex to be encoded, the prediction value V' is set to 0, and the prediction residual R is set to V: V' = 0, R = V. If V is not the first vertex to be encoded, the prediction value V' is set to the encoded value of the vertex previously encoded in vertex order, and the prediction residual R is set to the difference, R = V - V' (Equation 6).

[0153] If N=1, there is only one prediction candidate. Set V' to the predicted value, and set the prediction residual R to the difference, R=V-V' (Equation 6).

[0154] If N>=2, there are more than one prediction candidate. The average of these prediction values ​​can be used for prediction. Set the average of the prediction values ​​to V',

[0155] V'=(V1'+V2'+…+V N ') / N - Equation 7

[0156] Furthermore, an embodiment sets an upper limit M on how many prediction candidates will be considered. In this case, if N>M, then for each vertex, only the first M prediction candidates and their associated prediction values ​​V1', V2', ..., V M ',and

[0157] V'=(V1'+V2'+…+V M ') / M - Equation 8

[0158] If N=1, there is only one prediction candidate. An embodiment sets V' as the predicted value, and sets the prediction residual R as the difference, R=V-V' (Equation 6).

[0159] For example, M may be set to 4, where up to 4 prediction candidates will be considered.

[0160] The upper limit M can be signaled in the bitstream. And Fig.19Example 1900 is an example embodiment of signaling of an upper limit M, where bmesh_sequence_parameter_set_rbsp is a syntax table of a generic basemesh sequence parameter set RBSP, and where bmsps_intra_mesh_position_predicitor_limit_minus1 plus 1 indicates an upper limit on the number of vertex position prediction candidates. bmsps_intra_mesh_position_predicitor_limit_minus1 should be in the range of 0 to 7, inclusive.

[0161] According to an embodiment, the prediction residual R of the position value will be losslessly encoded using, for example, fixed-length coding, exponential Golomb coding, arithmetic coding, etc.

[0162] According to an embodiment, mesh_position_prediction_max_parallelograms_minus1 plus 1 specifies the maximum number of parallelograms used in mesh position prediction, and mesh_position_prediction_max_parallelograms_minus1 should be in the range of 0 to 15, inclusive.

[0163] According to an embodiment, “bmsps_intra_mesh_position_predicitor_limit_minus1” is replaced with “mesh_position_prediction_max_parallelograms_minus1”.

[0164] According to an embodiment, “mesh_position_prediction_max_parallelograms_minus1” is replaced with “bmsps_intra_mesh_position_predicitor_limit_minus1”.

[0165] According to an embodiment, the prediction of each vertex position attribute such as MDS23617_WG07_N00822 may be as follows. The input to the process is:

[0166] -The variable c specifies the index of the corner whose vertex position will be predicted.

[0167] The output of this process is indirect:

[0168] -It modifies the array VertexMarkingArray defined in the clause (such as clause error! Reference source not found.) and the array VertCoordValues ​​defined in the clause (such as clause error! Reference source not found.).

[0169] Let the alias O refer to the variable OppositeCornersArray.

[0170] Let the alias V refer to the variable CornerToVertexArray.

[0171] Let the alias G refer to the variable VertCoordValues.

[0172] Let the alias MV refer to the variable VertexMarkingArray.

[0173] / / maxParallelograms: mesh_position_prediction_max_parallelograms_minus1 u(4). Not yet implemented in ref soft using constants.

[0174] Let the variable maxParallelograms, which specifies the maximum number of parallelograms to be predicted, and the variable v, which specifies the index of the vertex position associated with c, be initialized as follows:

[0175] maxParallelograms=4

[0176] v=GetVertexIndex(CornerToVertexArray,c)

[0177] If MV[v] is strictly greater than 0, vertex v has already been predicted, and the procedure does nothing and returns. Otherwise, the following applies:

[0178] / / We mark the vertices

[0179] MV[v]=1

[0180] / / Search for some parallelogram estimates around the corner's vertices.

[0181] / / The triangle fan may not be complete,

[0182] / / But we know that vertices are always manifolds, so each vertex has only one fan

[0183] / / It is also possible that some relative corners are not defined due to boundaries

[0184] / / So we use the OV accessor and test the result to filter out negative values

[0185] Let the 1D array predPos of size 3 specifying the predicted position of the vertex associated with c, the variable altC specifying the corner index, and the variable nextC specifying the corner index be initialized as follows:

[0186] predPos[0]=0

[0187] predPos[1]=0

[0188] predPos[2]=0

[0189] altC=c

[0190] nextC=NextCorner(O[NextCorner(altC)])

[0191] / / Loop through the corners connected to the current vertex

[0192] / / Rotate around the sector until the boundary is found

[0193] while(nextC>=0&&nextC!=c)

[0194] {

[0195] altC=nextC

[0196] nextC=NextCorner(O[NextCorner(altC)])

[0197] }

[0198] Let the variable isBoundary specify whether nextC is on the boundary, and the variable count specify the number of valid parallelogram predictions found, and the variable startC specify the index of the extreme corner of the sector, and be initialized as follows:

[0199] isBoundary=(nextC !=c)

[0200] startC=altC

[0201] count=0

[0202] Set the variables prevV, oppoV, and nextV (specifying the indices of the vertices associated with the previous corner, the opposite corner, and the next corner, respectively) to 0.

[0203] The following applies:

[0204] / / Now we are at the rightmost corner of shared v

[0205] / / We go left and evaluate possible predictions

[0206] do

[0207] {

[0208] oppoV=GetVertexIndex(V,O[altC])

[0209] prevV=GetVertexIndex(V,PreviousCorner(altC))

[0210] nextV=GetVertexIndex(V,NextCorner(altC))

[0211] if((oppoV>-1&&prevV>-1&&nextV>-1)&&

[0212] ((MV[oppoV]>0)&&(MV[prevV]>0)&&(MV[nextV]>0)))

[0213] {

[0214] / / Parallelogram prediction estG = prevG + nextG – oppoGd

[0215] / / Accumulate parallelogram predictions

[0216] predPos[0]=predPos[0]+G[prevV][0]+G[nextV][0]-G[oppoV][0]

[0217] predPos[1]=predPos[1]+G[prevV][1]+G[nextV][1]-G[oppoV][1]

[0218] predPos[2]=predPos[2]+G[prevV][2]+G[nextV][2]-G[oppoV][2]

[0219] count++

[0220] }

[0221] / / Rotate around the triangle fan

[0222] altC=PreviousCorner(O[PreviousCorner(altC)])

[0223] / / Stop when fan is not fully or completely rotated or reaches maximum prediction}while(altC>=0&&altC!=startC&&count <maxParallelograms);

[0224] / / 1. Use parallelogram prediction when possible

[0225] if(count>0){

[0226] G[0]=mesh_position_residual[deltaIndex][0]+round(predPos[0]) / countG[1]=mesh_position_residual[deltaIndex][1]+round(predPos[1]) / countG[2]=mesh_position_residual[deltaIndex][2]+round(predPos[2]) / count

[0227] deltaIndex = deltaIndex + 1;

[0228] return

[0229] }

[0230] / / 2. Or fall back to using the increment of the available value (if any) prevV = GetVertexIndex(V, PreviousCorner(c))

[0231] nextV=GetVertexIndex(V,PreviousCorner(c))

[0232] if(prevV>-1&&MV[prevV]>-1){

[0233] G[v][0]=mesh_position_residual_coarse[deltaCoarseIndex][0]+G[prevV][0]G[v][1]=mesh_position_residual_coar se[deltaCoarseIndex][1]+G[prevV][1]G[v][2]=mesh_position_residual_coarse[deltaCoarseIndex][2]+G[prevV][2]

[0234] deltaCoarseIndex=deltaCoarseIndex+1;

[0235] return

[0236] }

[0237] if(nextV>-1&&MV[nextV]>-1){

[0238] G[v][0]=mesh_position_residual_coarse[deltaCoarseIndex][0]+G[nextV][0]

[0239] G[v][1]=mesh_position_residual_coarse[deltaCoarseIndex][1]+G[nextV][1]

[0240] G[v][2]=mesh_position_residual_coarse[deltaCoarseIndex][2]+G[nextV][2]

[0241] deltaCoarseIndex=deltaCoarseIndex+1;

[0242] return

[0243] }

[0244] Let variable b specify the index of the previous corner on the boundary.

[0245] Let the variable bV specify the index of the vertex associated with b.

[0246] / / 3. Maybe we are on the border

[0247] / / Then we can use the delta from the previous vertex on the boundary

[0248] if(isBoundary){

[0249] b=PreviousCorner(startC)

[0250] bV=GetVertexIndex(V,b)

[0251] if(MV[bv]>-1){

[0252] G[v][0]=mesh_position_residual_coarse[deltaCoarseIndex][0]+G[bV][0]

[0253] G[v][1]=mesh_position_residual_coarse[deltaCoarseIndex][1]+G[bV][1]

[0254] G[v][2]=mesh_position_residual_coarse[deltaCoarseIndex][2]+G[bV][2]

[0255] deltaCoarseIndex=deltaCoarseIndex+1;

[0256] return

[0257] }

[0258] }

[0259] / / 4. No more options, we use the absolute value (i.e. start)

[0260] G[v][0]=mesh_position_start[startIndex][0]

[0261] G[v][1]=mesh_position_start[startIndex][1]

[0262] G[v][2]=mesh_position_start[startIndex][2]

[0263] startIndex=startIndex+1

[0264] In order to show the technical advantages of the embodiments of the present application, the present application has conducted an analysis and provided its results. For example, position encoding for lossless mesh compression was evaluated in V-DMC TMM version 3.0. The experimental results show that the lossless gain of geometry exceeds 1% with no impact on computational complexity.

[0265] Lossless mesh compression, which includes lossless geometry compression and lossless texture compression, is introduced, and position coding for lossless mesh compression (more specifically, lossless geometry compression) is evaluated in V-DMC TMM version 3.0.

[0266] Position encoding for base mesh compression can be enabled in V-DMC TMM version 3.0 by setting dracoUsePosition = 1. Previously, lossy conditions (including AI and LD) have been evaluated. And position encoding for lossless mesh compression has been evaluated. As in lossy conditions, position encoding eliminates the fold calculation in base mesh compression, and therefore also reduces the computational complexity of lossless mesh compression.

[0267] Position coding for lossless compression was evaluated in V-DMC TMM Version 3.0 by setting dracoUsePosition = 1 and comparing the result with V-DMC TMM Version 3.0 (where dracoUsePosition = 0). Fig. 20 Example 2000 of element 2001, from Fig. 20 It can be seen that for lossless mesh compression, position encoding reduces the geometry code stream by more than 1%. In addition, for lossy conditions, the BD rate reduction caused by position encoding under V-DMC TMM Version 3.0 is consistent with the results reported under V-Mesh TM Version 1.0. The BD rate reduction under V-DMC TMM Version 3.0 is summarized in Element 2001. For encoder / decoder runtime, according to an embodiment, there is no negative impact by enabling position encoding for V-DMC TMM Version 3.0.

[0268] Therefore, for position coding in V-DMC TMM version 3.0 for lossless grid compression, the experimental results of the embodiments of the present application show that the lossless geometric coding gain is greater than 1%.

[0269] The above techniques may be implemented as computer software via computer-readable instructions and physically stored in at least one computer-readable storage medium. Fig.21 A computer system 2100 is shown that is suitable for implementing certain embodiments of the disclosed subject matter.

[0270] The computer software may be encoded in any suitable machine code or computer language, and may be assembled, compiled, linked, or the like to create a code comprising instructions, which may be directly executed by at least one computer central processing unit (CPU), graphics processing unit (GPU), or the like, or executed by decoding, microcode, or the like.

[0271] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablets, servers, smartphones, gaming devices, IoT devices, etc.

[0272] Fig.21The components shown for computer system 2100 are exemplary in nature and are not intended to limit the scope of use or functionality of computer software implementing embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any component or combination of components shown in the exemplary embodiment of computer system 2100.

[0273] Computer system 2100 may include certain human-computer interface input devices. Computer system 2100 may include certain human-computer interface input devices. Such human-computer interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human-computer interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0274] The human-machine interface input device may include one or more of the following (only one of each is shown): keyboard 2101 , mouse 2102 , touchpad 2103 , touch screen 2110 , joystick 2105 , microphone 2106 , scanner 2108 , camera 2107 .

[0275] The computer system 2100 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate one or more human user senses through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen 2110 or joystick 2105, but there may also be tactile feedback devices that are not input devices), audio output devices (e.g., speakers 2109, headphones (not shown)), visual output devices (screens 2110 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, some of which may output two-dimensional or more than three-dimensional visual outputs through stereo output, virtual reality glasses (not shown), holographic displays, and smoke canisters (not shown), and printer drawing).

[0276] The computer system 2100 may also include human-accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW 2120 with CD / DVD 2111 or similar media, thumb drive 2122, removable hard drive or solid state drive 2123, traditional magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), etc.

[0277] Those skilled in the art should also understand that the term "computer-readable storage media" used in connection with the disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0278] The computer system 2100 may also include an interface 2199 to one or more communication networks 2198. The network 2198 may be, for example, wireless, wired, optical. The network 2198 may also be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of the network 2198 include local area networks such as Ethernet, wireless local area networks, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicle and industrial networks including CANBus, etc. Some networks 2198 typically require an external network interface adapter connected to some general data port or peripheral bus (2150 and 2151) (e.g., a USB port of the computer system 2100; other networks are typically integrated into the computer system 2100 by connecting to the system bus (e.g., an Ethernet interface of a PC computer system or a cellular network interface of a smartphone computer system). Using any of these networks 2198, the computer system 2100 can communicate with other entities. Such communication can be one-way, receive-only (e.g., broadcast television), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way, such as other computer systems using a local area network or a wide area digital network. As described above, certain protocols and protocol stacks are available for each of these networks and network interfaces.

[0279] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces can be connected to the core 2140 of the computer system 2100.

[0280] The core 2140 may include one or more central processing units (CPUs) 2141, graphics processing units (GPUs) 2142, graphics adapters 2117, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 2143, hardware accelerators 2144 for certain tasks, and the like. These devices, as well as read-only memory (ROM) 2145, random access memory 2146, and internal mass storage (such as internal non-user accessible hard disk drives, SSDs, etc.) 2147 may be connected via a system bus 2148. In some computer systems, the system bus 2148 may be accessed in the form of one or more physical plugs to enable expansion through additional CPUs, GPUs, and the like. Peripheral devices may be connected directly to the core's system bus 2148 or via a peripheral bus 2149. The architecture of the peripheral bus includes PCI, USB, and the like.

[0281] ] The CPU 2141, GPU 2142, FPGA 2143, and accelerator 2144 may execute certain instructions, which in combination may constitute the above-mentioned computer code. The computer code may be stored in ROM 2145 or RAM 2146. Transitional data may also be stored in RAM 2146, while permanent data may be stored in, for example, internal mass storage 2147. Fast storage and retrieval of any storage device may be achieved by using a cache memory, which may be closely associated with one or more CPUs 2141, GPUs 2142, mass storage 2147, ROM 2145, RAM 2146, etc.

[0282] The computer readable storage medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0283] As an example and not limitation, a computer system having architecture 2100, in particular core 2140, can provide functionality by executing software contained in one or more tangible computer-readable media through a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such a computer-readable medium can be a medium associated with a user-accessible mass storage as described above, and certain memories of the core 2140 having a non-transient nature, such as a core internal mass storage 2147 or ROM 2145. The software implementing various embodiments of the present application can be stored in such a device and executed by the core 2140. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the core 2140, in particular the processor therein (including a CPU, GPU, FPGA, etc.) to perform a specific process or a specific part of a specific process described in the present application, including defining a data structure stored in RAM 2146 and modifying these data structures according to a software-defined process. In addition or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in a circuit (e.g., accelerator 2144) that may operate in place of or in conjunction with software to perform a specific process or a specific portion of a specific process described herein. Where appropriate, references to software may include logic and vice versa. Where appropriate, references to computer-readable media may include circuits (such as integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both. The present application includes any suitable combination of hardware and software.

[0284] Although the present application has described a number of exemplary embodiments, various changes, arrangements and various equivalent substitutions of the embodiments are within the scope of the present application. Therefore, it should be understood that those skilled in the art can design a variety of systems and methods, which, although not explicitly shown or described in the present application, embody the principles of the present application and therefore belong to the spirit and scope of the present application.

Claims

1. A video decoding method, characterized in that: Executed by at least one processor, the method includes: Obtaining a mesh from a bitstream, the mesh representing encoded volume data of at least one three-dimensional 3D visual content; Dividing the plurality of vertices of the mesh into a plurality of groups, at least one of the plurality of groups representing a parallelogram shape; and For each group in the plurality of groups, the encoded volume data is decoded by predicting a plurality of vertices in the group based on a prediction mode associated with the group.

2. The method according to claim 1, characterized in that The decoding of the encoded volume data comprises: A parallelogram prediction is applied, wherein the quadrilateral is divided into two triangles, and a vertex of a first one of the two triangles is used as a predictor for a vertex of a second one of the two triangles.

3. The method according to claim 2, characterized in that The first of the two triangles includes vertex A, vertex B and vertex C; The second triangle of the two triangles includes vertex B, vertex C and vertex D; The parallelogram prediction includes: Based on the coordinates of the vertex A, the coordinates of the vertex B, and the coordinates of the vertex C, the coordinates of the vertex D are predicted.

4. The method according to claim 2, characterized in that: At least one of the plurality of vertices does not include a prediction candidate.

5. The method according to claim 2, characterized in that: At least one of the plurality of vertices includes a prediction candidate.

6. The method according to claim 2, characterized in that At least one of the plurality of vertices includes more than one prediction candidate.

7. The method according to claim 6, characterized in that The parallelogram prediction includes: The average of the more than one prediction candidates is predicted as the residual.

8. The method according to claim 1, characterized in that An upper limit of a plurality of prediction candidates is signaled in the bitstream.

9. The method according to claim 8, characterized in that The upper limit of the number of prediction candidates is signaled in the codestream via the syntax mesh_position_prediction_max_parallelograms_minus1.

10. The method according to claim 9, characterized in that The syntax mesh_position_prediction_max_parallelograms_minus1 indicates the upper limit by indicating a range of values.

11. A video decoding device, characterized in that: include: at least one memory configured to store computer program code; At least one processor is configured to read the computer program code and operate according to the computer program code, wherein the computer program code includes: Acquisition code is configured to enable the at least one processor to acquire a grid from a bitstream, the grid representing encoded volume data of at least one three-dimensional 3D visual content; A partitioning code configured to cause the at least one processor to partition a plurality of vertices of the mesh into a plurality of groups, at least one of the plurality of groups representing a parallelogram shape; and The decoding code is configured to enable the at least one processor to decode the encoded volume data for each group in the plurality of groups by predicting a plurality of vertices in the group based on a prediction mode associated with the group.

12. The device according to claim 11, characterized in that The decoding of the encoded volume data comprises: A parallelogram prediction is applied, wherein the quadrilateral is divided into two triangles, and a vertex of a first one of the two triangles is used as a predictor for a vertex of a second one of the two triangles.

13. The device according to claim 12, characterized in that The first of the two triangles includes vertex A, vertex B and vertex C; The second triangle of the two triangles includes vertex B, vertex C and vertex D; The parallelogram prediction includes: Based on the coordinates of the vertex A, the coordinates of the vertex B, and the coordinates of the vertex C, the coordinates of the vertex D are predicted.

14. The device according to claim 12, characterized in that At least one of the plurality of vertices does not include a prediction candidate.

15. The device according to claim 12, characterized in that At least one of the plurality of vertices includes a prediction candidate.

16. The device according to claim 12, characterized in that At least one of the plurality of vertices includes more than one prediction candidate.

17. The device according to claim 16, characterized in that The parallelogram prediction includes: The average of the more than one prediction candidates is predicted as the residual.

18. The device according to claim 11, characterized in that An upper limit of a plurality of prediction candidates is signaled in the bitstream.

19. The device according to claim 18, characterized in that The upper limit of the number of prediction candidates is signaled in the codestream via the syntax mesh_position_prediction_max_parallelograms_minus1.

20. A non-volatile storage medium, characterized in that: A program is stored that causes a computer to: Obtaining a mesh from a bitstream, the mesh representing encoded volume data of at least one three-dimensional 3D visual content; Dividing the plurality of vertices of the mesh into a plurality of groups, at least one of the plurality of groups representing a parallelogram shape; and, For each group in the plurality of groups, the encoded volume data is decoded by predicting a plurality of vertices in the group based on a prediction mode associated with the group.

Images