METHOD AND APPARATUS FOR VIDEO DECODING - Patent application

Abstract

The present invention provides a method and an apparatus including computer code configured to cause one or more processors to perform the following steps: obtaining a coded bitstream including a mesh sequence of a plurality of meshes of 3D visual content and a displacement of a local coordinate system (LCS) or a global coordinate system (GCS) of at least one mesh of the plurality of meshes, where the LCS or GCS is at a vertex of the at least one mesh; determining projection coefficients of the displacements of the LCS or GCS, where the projection coefficients indicate unit tangent vectors of the LCS or GCS in a normal direction of the vertex and are based on normalization of tangent basis vectors of the LCS or GCS; and decoding the mesh sequence based on the projection coefficients of the displacements.

Description

[Technical Field]

[0001] This application claims priority to U.S. Provisional Application No. 63 / 414,346, filed October 7, 2022, and U.S. Application No. 18 / 481,757, filed October 5, 2023, the entireties of which are expressly incorporated herein by reference.

[0002] The present disclosure aims to quickly calculate a local coordinate system for displacement vectors used in mesh compression. [Background technology]

[0003] Advances in 3D capture, modeling, and rendering have facilitated the ubiquity of 3D content across multiple platforms and devices. Now, a baby's first steps can be captured in one space, while grandparents can view (and interact with) the child in another space, enjoying a fully immersive experience. Nevertheless, to achieve this realism, models are becoming increasingly sophisticated, and large amounts of data are associated with the creation and use of those models. 3D meshes can be widely used to represent such immersive content.

[0004] Dynamic mesh sequences can require large amounts of data because they can consist of a large amount of information that changes over time. Therefore, storing and transmitting such content requires efficient compression techniques. The mesh compression standards IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information. Digital content creation (DCC) tools typically generate such dynamic meshes. However, generating constant connectivity dynamic meshes, especially under real-time constraints, is challenging for volumetric acquisition techniques. This type of content is not supported by existing standards. MPEG plans to develop a new mesh compression standard that directly handles dynamic meshes with time-varying connectivity information and, optionally, time-varying attribute maps. This standard targets lossy and lossless compression for various applications, such as real-time communication, storage, free-viewpoint video, AR, and VR. Functionality such as random access and scalable / progressive coding is also envisioned. Therefore, for any of these reasons, there is a need for a technical solution to these problems that arise in video coding technology. Summary of the Invention [Means for solving the problem]

[0005] The present invention includes an apparatus and a method, the apparatus including a memory configured to store computer program code and one or more processors configured to access the computer program code and operate according to instructions of the computer program code, the computer program being configured to cause the at least one processor to implement: acquisition code configured to cause the at least one processor to acquire a coded bitstream including a mesh sequence of a plurality of meshes of 3D visual content and a displacement of a local coordinate system (LCS) or a global coordinate system (GCS) of at least one mesh of the plurality of meshes, the LCS or GCS being at a vertex of the at least one mesh; decision code configured to cause the at least one processor to determine projection coefficients of the displacement of the LCS or GCS, the projection coefficients indicating unit tangent vectors of the LCS or GCS in a normal direction of the vertex and based on normalization of tangent basis vectors of the LCS or GCS; and decoding code configured to cause the at least one processor to decode the mesh sequence based on the projection coefficients of the displacement.

[0006] According to one aspect of the present disclosure, the projection coefficients may be based on a plurality of tangent basis vectors, including tangent basis vectors of the LCS or GCS.

[0007] According to one aspect of the present disclosure, the tangent basis vector may include three components: a first component, a second component, and a third component.

[0008] According to one aspect of the present disclosure, the first component may have a size greater than both the second component and the third component.

[0009] According to one aspect of the present disclosure, the tangent basis vectors of the LCS or GCS may be specified as t=[0, −n3, n2], where t represents the tangent basis vector, −n3 represents the third component, and n2 represents the second component.

[0010] According to one aspect of the present disclosure, if the values ​​of the second component and the third component are both 0, the tangent basis vector of the LCS or GCS may be specified as t=[0,1,0], where t represents the tangent basis vector.

[0011] According to one aspect of the present disclosure, determining a component of the displacement of the LCS may include determining another unit tangent vector that represents a cross product of the unit tangent vector and a normal basis vector from which the tangent basis vector is generated.

[0012] According to one aspect of the present disclosure, the projection coefficient of the displacement of the LCS may represent another unit tangent vector that is a cross product of a normal vector and the unit tangent vector.

[0013] According to one aspect of the present disclosure, the tangent basis vectors of the LCS or GCS may represent those obtained by subtracting the part of at least one component of a standard basis vector that is in the normal direction from the at least one component.

[0014] According to one aspect of the present disclosure, the standard basis vector may be pre-specified and may be one of three standard basis vectors pre-specified for the LCS or GCS. [Brief explanation of the drawings]

[0015] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings.

[0016] [Figure 1] FIG. 1 is a schematic diagram of a diagram according to an embodiment. [Figure 2] FIG. 1 is a simplified block diagram according to an embodiment. [Figure 3] FIG. 1 is a simplified diagram according to an embodiment. [Figure 4] FIG. 1 is a simplified diagram according to an embodiment. [Figure 5] FIG. 1 is a simplified diagram according to an embodiment. [Figure 6] FIG. 1 is a simplified diagram according to an embodiment. [Figure 7] FIG. 1 is a simplified diagram according to an embodiment. [Figure 8] FIG. 1 is a simplified diagram according to an embodiment. [Figure 9] FIG. 1 is a simplified diagram according to an embodiment. [Figure 10] FIG. 1 is a simplified flow diagram according to an embodiment. [Figure 11] FIG. 1 is a simplified flow diagram according to an embodiment. [Figure 12] FIG. 1 is a simplified flow diagram according to an embodiment. [Figure 13] FIG. 1 is a simplified diagram according to an embodiment. [Figure 14] FIG. 1 is a simplified diagram according to an embodiment. [Figure 15] FIG. 1 is a simplified diagram according to an embodiment. [Figure 16] FIG. 1 is a simplified diagram according to an embodiment. [Figure 17] FIG. 1 is a simplified diagram according to an embodiment. [Figure 18] FIG. 1 is a simplified diagram according to an embodiment. [Figure 19] FIG. 1 is a simplified diagram according to an embodiment. [Figure 20] FIG. 1 is a simplified diagram according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] The proposed features described below may be used alone or in any combination in any order. Furthermore, the embodiments may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0018] 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The communication system 100 may include at least two terminals 102, 103 interconnected via a network 105. In the case of unidirectional transmission of data, a first terminal 103 may locally encode video data for transmission to another terminal 102 via the network 105. The second terminal 102 may receive the coded video data of the other terminal from the network 105, decode the coded data, and display the recovered video data. The unidirectional transmission of data may be common in media serving applications, for example.

[0019] 1 shows a second pair of terminals 101, 104 provided to support two-way transmission of coded video, such as might occur during a video conference. For two-way transmission of data, each terminal 101 and / or 104 can code captured video data at a local location for transmission to another terminal over network 105. Each terminal 101, 104 can also receive coded video data transmitted by another terminal, decode the coded data, and display the recovered video data on a local display device.

[0020] In FIG. 1 , terminal devices 101, 102, 103, and 104 may be depicted as servers, personal computers, and smartphones, although the principles of the present disclosure are not limited thereto. Embodiments of the present disclosure apply to laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 105 represents any number of networks that transmit coded video data between terminals 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 105 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0021] 2 shows the arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter, which is equally applicable to other video-supported applications including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0022] The streaming system may include a capture subsystem 203, which may include a video source 201, such as a digital camera, that creates an uncompressed video sample stream 213. The sample stream 213 may be emphasized as a high data volume compared to an encoded video bitstream and may be processed by an encoder 202 coupled to the video source 201, which may be a camera as described above. The encoder 202 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream 204, which may be emphasized as a lower data volume compared to the sample stream, may be stored on a streaming server 205 for future use. One or more streaming clients 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video bitstream 204. Client 212 may include a video decoder 211 that can decode an incoming copy of encoded video bitstream 208 and create an outgoing video sample stream 210 that can be rendered on a display 209 or other rendering device (not shown). In some streaming systems, video bitstreams 204, 206, and 208 may be encoded according to a particular video coding / compression standard. Examples of those standards are mentioned above and further described herein.

[0023] FIG. 3 may be a functional block diagram of a video decoder 300 according to one embodiment of the present invention.

[0024] Receiver 302 may receive one or more coded video sequences to be decoded by decoder 300. In the same or another embodiment, receiver 302 may receive one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences may be received from channel 301, which may be a hardware / software link to a storage device that stores the encoded video data. Receiver 302 may receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, that may be forwarded to a respective using entity (not shown). Receiver 302 may separate the coded video sequences from other data. To prevent network jitter, buffer memory 303 may be coupled between receiver 302 and entropy decoder / parser 304 (hereinafter "parser"). When receiver 302 receives data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, buffer 303 may not be needed or may be small. For use in a best effort packet network such as the Internet, buffer 303 may be necessary and may be relatively large, and may advantageously be adaptively sized.

[0025] The video decoder 300 may include a parser 304 that reconstructs symbols 313 from the entropy-coded video sequence. These symbol categories include information used to manage the operation of the decoder 300 and potentially include information to control a rendering device, such as a display 312, that is not an integral part of the decoder but may be coupled to the decoder. The information to control the rendering device may be in the form of a supplemental enhancement information (SEI) message or a video usability information (VUI) parameter set fragment (not shown). The parser 304 may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be aligned with a video coding technique or standard and may follow principles well known to those skilled in the art, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a set of subgroup parameters for at least one subgroup of pixels within the video decoder from the coded video sequence based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may also extract information such as transform coefficients, quantization parameter values, motion vectors, etc. from the coded video sequence.

[0026] The parser 304 may perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbols 313. The parser 304 may receive encoded data and selectively decode particular symbols 313. Furthermore, the parser 304 may determine whether to provide the particular symbols 313 to the motion compensated prediction unit 306, the scaler / inverse transform unit 305, the intra prediction unit 307, or the loop filter 311.

[0027] The reconstruction of symbols 313 may involve several different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by parser 304. The flow of such subgroup control information between parser 304 and the following units is not shown for clarity.

[0028] In addition to the functional blocks already mentioned, decoder 300 may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0029] The first unit is a scalar / inverse transform unit 305. The scalar / inverse transform unit 305 receives control information from the parser 304, including the transform to use, block size, quantization factor, quantization scaling matrix, etc., as well as the quantized transform coefficients as symbols 313. The scalar / inverse transform unit 305 can output blocks containing sample values ​​that can be input to an aggregator 310.

[0030] In some cases, the output samples of the scaler / inverse transform unit 305 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 generates blocks of the same size and shape as the block being reconstructed using surrounding already reconstructed information taken from the current (partially reconstructed) picture 309. In some cases, the aggregator 310 adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305.

[0031] In other cases, the output samples of the scalar / inverse transform unit 305 may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion-compensated prediction unit 306 may access the reference picture memory 308 to retrieve samples used for prediction. After motion-compensating the retrieved samples according to symbols 313 related to the block, these samples may be added to the output of the scalar / inverse transform unit by the aggregator 310 to generate output sample information (in this case, referred to as residual samples or residual signals). The addresses in the reference picture memory from which the motion compensation unit retrieves prediction samples may be controlled by motion vectors available to the motion compensation unit in the form of symbols 313, which may have, for example, X, Y, and reference picture components. Motion compensation may include interpolation of sample values ​​retrieved from the reference picture buffer when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.

[0032] The output samples of aggregator 310 may be applied by various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filter techniques that may be controlled by parameters contained in the coded video bitstream and made available to loop filter unit 311 as symbols 313 from parser 304, and may also depend on previously reconstructed loop-filtered sample values ​​as well as on meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence.

[0033] The output of the loop filter unit 311 may be a sample stream that can be output to a rendering device 312 and also stored in a reference picture memory 557 for use in future inter-picture prediction.

[0034] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. Once a coded picture is fully reconstructed and identified as a reference picture (e.g., by parser 304), current picture 309 can become part of reference picture buffer 308, and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0035] Video decoder 300 may perform decoding operations according to a predetermined video compression technique, which may be described in a standard such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to the syntax of the video compression technique or standard, as specified in the video compression technique document or standard, particularly the profile document therein. Compliance also requires that the complexity of the coded video sequence be within the range limited by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the level may be further limited by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled in the coded video sequence.

[0036] In one embodiment, the receiver 302 can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data can be used by the video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0037] FIG. 4 may be a functional block diagram of a video encoder 400 according to one embodiment of the present disclosure.

[0038] The encoder 400 may receive video samples from a video source 401 (not part of the encoder) that may capture video images that are coded by the encoder 400 .

[0039] The video source 401 may provide a source video sequence to be coded by the encoder 303 in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source 401 may be a storage device that stores previously prepared video. In a video conferencing system, the video source 401 may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that, when viewed in sequence, give the appearance of motion. The pictures themselves may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion will focus on samples.

[0040] According to one embodiment, the encoder 400 can encode and compress pictures of a source video sequence into a coded video sequence 410 in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one of the functions of the controller 402. The controller controls and is operatively coupled to other functional units, as described below. For clarity, coupling is not shown. Parameters set by the controller may include rate control-related parameters (picture skip, quantization, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of the controller 402 that may be relevant to optimizing the video encoder 400 for a particular system design.

[0041] Some video encoders operate in what those skilled in the art would readily recognize as a "coding loop." As an overly simplified explanation, the coding loop may include an encoding portion of an encoder 400 (hereinafter, "source coder") (responsible for creating symbols based on the input picture to be coded and reference pictures) and a (local) decoder 406 embedded in the encoder 400. This (local) decoder 406 reconstructs the symbols to create sample data that is also created by a (remote) decoder (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream is input to a reference picture memory 405. Because decoding the symbol stream yields bit-exact results regardless of the location (local or remote) of the decoder, the contents of the reference picture buffer are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronism (and the resulting drift when synchronism cannot be maintained, eg, due to channel errors) is well known to those skilled in the art.

[0042] The operation of the "local" decoder 406 may be the same as that of the "remote" decoder 300, already described in detail above in connection with Figure 3. However, briefly referring also to Figure 4, the entropy decoding portion of the decoder 300, including the channel 301, receiver 302, buffer 303 and parser 304, may not be fully implemented in the local decoder 406, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder 408 and parser 304 may be lossless.

[0043] As can be seen, any decoder technology other than parsing / entropy decoding present in a decoder must necessarily be present in the corresponding encoder in substantially the same functional form. A description of the encoder technology can be omitted, as it is the inverse of the decoder technology described generically. Only in certain areas is a more detailed description required, which is provided below.

[0044] As part of its operation, the source coder 403 may perform motion-compensated predictive coding, which predictively codes an input frame with reference to one or more previously coded frames from the video sequence, designated as “reference frames.” In this manner, the coding engine 407 codes differences between pixel blocks of the input frame and pixel blocks of reference frames that may be selected as prediction references for the input frame.

[0045] The local video decoder 406 can decode coded video data of frames that may be designated as reference frames based on symbols created by the source coder 403. The operation of the coding engine 407 can advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 4), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder 406 can replicate the decoding process that may be performed by the video decoder on the reference frames and store the reconstructed reference frames in the reference picture memory 405, which may be, for example, a cache. In this way, the encoder 400 can locally store copies of reconstructed reference frames that have content in common (no transmission errors) with the reconstructed reference frames obtained by the far-end video decoder.

[0046] The predictor 404 may perform the prediction search of the coding engine 407. That is, for a new frame to be coded, the predictor 404 may search the reference picture memory 405 for sample data (as candidate reference pixel blocks) or specific metadata that may serve as suitable prediction criteria for the new picture, such as the reference picture's motion vectors, block shapes, etc. The predictor 404 may operate on a sample block / pixel block basis to find a suitable prediction criteria. In some cases, as determined by the search results obtained by the predictor 404, the input picture may have prediction criteria drawn from multiple reference pictures stored in the reference picture memory 405.

[0047] The controller 402 may manage the coding operations of the source coder 403, which may be, for example, a video coder, including, for example, setting parameters and subgroup parameters used to encode the video data.

[0048] The output of all the aforementioned functional units may be entropy coded in entropy coder 408. The entropy coder converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing these symbols according to techniques well known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0049] The transmitter 409 may buffer the coded video sequence created by the entropy coder 408 in preparation for transmission over a communication channel 411, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 409 may merge the coded video data from the source coder 403 with other data to be transmitted, such as coded audio data and / or ancillary data streams (sources not shown).

[0050] A controller 402 may manage the operation of the encoder 400. During coding, the controller 402 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as any of the following frame types:

[0051] An intra picture (I-picture) may be one that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0052] A predictive picture (P picture) may be coded and decoded by intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0053] Bidirectionally predicted pictures (B-pictures) may be coded and decoded by intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0054] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be nonpredictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P-picture may be nonpredictively coded via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be nonpredictively coded via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0055] Encoder 400, which may be, for example, a video coder, may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In doing so, encoder 400 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in an input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0056] In one embodiment, the transmitter 409 can transmit additional data along with the encoded video. The source coder 403 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, Supplemental Enhancement Information (SEI) messages, Visual Usability Information (VUI) parameter set fragments, etc.

[0057] FIG. 5 shows a simplified block-style workflow diagram 500 of exemplary viewpoint-dependent processing in Omnidirectional Media Application Format (OMAF) that may enable 360-degree virtual reality (VR360) streaming as described in OMAF.

[0058] In acquisition block 501, video data A, such as multiple image and audio data of the same time instance, is acquired if the image data can represent a scene in VR360. In processing block 503, the images Bi of the same time instance are processed by one or more of stitching, mapping to projected pictures with respect to one or more virtual reality (VR) angles or other angles / viewpoints, and region-by-region packing. Furthermore, metadata indicating any of this processed information and other information can be created to assist in the distribution and rendering process.

[0059] With respect to the data D, the image encoding block 505 encodes the projected pictures into data Ei and configures them as a media file with viewpoint-independent streaming, the video encoding block 504 encodes the video pictures into data Ev, for example as a single-layer bitstream, and with respect to the data Ba, the audio encoding block 502 may also encode audio data into data Ea.

[0060] The data Ea, Ev, and Ei, as well as the entire coded bitstreams F and / or F, can be stored on a (content delivery network (CDN) / cloud) server and typically transmitted in full to an OMAF player 520, such as in a distribution block 507, and then fully decoded by a decoder, such that at least one region of the decoded picture corresponding to the current viewpoint is rendered to the user in a display block 516, in relation to various metadata from a head / eye tracking block 508, such as the angle at which the user is looking through the VR image device relative to the viewpoint specifications of the device, file playback, and orientation / viewpoint metadata. A distinctive feature of VR360 is the ability to display only one viewpoint at any particular time; such a feature can be exploited to improve the performance of omnidirectional video systems by selectively delivering the images according to the user's viewpoint (or any other criteria, such as timed metadata of a preferred viewpoint). For example, according to an exemplary embodiment, viewpoint-dependent delivery can be enabled by tile-based video coding.

[0061] Similar to the encoding blocks described above, the OMAF player 520 according to example embodiments similarly reverses one or more aspects of such encoding with respect to decapsulating the files / segments of one or more data F′ and / or F′i and metadata, decoding audio data E′i in audio decoding block 510, video data E′v in video decoding block 513, and image data E′i in image decoding block 514, and proceeding with audio rendering of data B′a in audio rendering block 511 and image rendering of data D′ in image rendering block 515, thereby outputting display data A′i in display block 516 and audio data A′s in loudspeaker / headphone block 512 in VR360 format according to various metadata, such as orientation / perspective metadata. The various metadata may affect one of the data decoding and rendering processes according to various tracks, languages, qualities, and views that may be selected by or for the user of the OMAF player 520. It should be understood that the order of the processes described herein is shown for the exemplary embodiment and that the processes may be implemented in other orders according to other exemplary embodiments.

[0062] 6 shows a simplified block-style content flow process diagram 600 for (encoding) point cloud data with view position and angle dependent processing of point cloud data (herein "V-PCC") for six degrees of freedom media capture / generation / (decode)coding / rendering / display. It should be understood that, according to example embodiments, the described features can be used individually or in any order and in combination, and that the illustrated elements, particularly those for encoding and decoding, can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits), and that the one or more processors can execute a program stored on a non-transitory computer-readable medium.

[0063] Diagram 600 illustrates an exemplary embodiment of streaming coded point cloud data according to V-PCC.

[0064] In volumetric data acquisition block 601, a real-world visual scene or a computer-generated visual scene (or a combination thereof) can be captured by a set of camera devices or synthesized by a computer as volumetric data, and the volumetric data, which can be in any format, can be converted into a (quantized) point cloud data format through image processing in point cloud conversion block 602. For example, according to an exemplary embodiment, data from the volumetric data is united in area data that is converted into several points of a point cloud by subtracting one or more of the values ​​described below from the volumetric data and any associated data into the desired point cloud format. According to an exemplary embodiment, the volumetric data can be a 3D dataset of a 2D image, such as a slice from which a 2D projection of the 3D dataset can be made. According to example embodiments, point cloud data formats include representations of data points in one or more various spaces and can be used to represent volumetric data and can provide improvements in terms of sampling and data compression, e.g., with respect to temporal redundancy; for example, point cloud data in x, y, z format represents color values ​​(e.g., RGB, etc.), brightness, intensity, etc., at each of multiple points in the point cloud data and can be used with progressive decoding, polygon meshes, direct rendering, and octree 3D representations of 2D quadtree data.

[0065] In the projection to image block 603, the acquired point cloud data can be projected onto a 2D image and encoded as an image / video picture using video-based point cloud coding (V-PCC). The projected point cloud data may consist of attributes, geometry, occupancy maps, and other metadata used to reconstruct the point cloud data by painter's algorithms, ray casting algorithms, (3D) binary space partitioning algorithms, etc.

[0066] Meanwhile, in the scene generator block 609, the scene generator can generate several metadata used for rendering and displaying six degrees of freedom (DoF) media, for example, according to the director's intentions or user preferences. Such 6DoF media may include VR360, such as 3D viewing of a scene with rotational changes on 3D axes (X, Y, Z), in addition to additional dimensions that allow for forward / backward, up / down, and left / right movement relative to the virtual experience within or at least in accordance with the coded point cloud data. Scene description metadata defines one or more scenes composed of coded point cloud data and other media data, including VR360, light field, audio, etc., and can be provided to one or more cloud servers and / or file / segment encapsulation / deencapsulation processes, as shown in FIG. 6 and the related description.

[0067] When such description represents an exemplary embodiment, after a video encoding block 604 and an image encoding block 605, similar to the video and image encoding described above (it will be understood that audio encoding may also be provided as described above), a file / segment encapsulation block 606 processes the coded point cloud data into a media file for file playback or a sequence of initialization segments and media segments for streaming according to a particular media container file format, such as one or more video container formats, for use in connection with DASH, among other things, as described below. The file container may also include scene description metadata in the files or segments, such as from a scene generator block 1109.

[0068] According to exemplary embodiments, a file is encapsulated containing one or more views of 6DoF media at at least one view position and one or more angular views at that / those view positions according to scene description metadata, as transmitted upon request according to user or creator input. Further, according to exemplary embodiments, a segment of such a file may include one or more portions of such a file, such as a portion of 6DoF media showing one or more angles at a single viewpoint, although these are merely exemplary embodiments and may vary according to various conditions, such as network, user, creator capabilities and input.

[0069] According to an example embodiment, the point cloud data is partitioned into multiple 2D / 3D regions that are independently coded, such as in one or more of the video encoding block 604 and the image encoding block 605. Then, in the file / segment encapsulation block 606, each independently coded partition of the point cloud data can be encapsulated as a track within a file and / or segment. According to an example embodiment, each point cloud track and / or metadata track may include some metadata useful for view position / angle dependent processing.

[0070] According to an example embodiment, metadata useful for view position / angle dependent processing, such as metadata included in a file and / or segment encapsulated with respect to a file / segment encapsulation block, includes one or more of the following: layout information of indexed 2D / 3D partitions; (dynamic) mapping information relating a 3D volume partition to one or more 2D partitions (e.g., tiles / tile groups / slices / subpictures); 3D position of each 3D partition on the 6DoF coordinate system; a representative view position / angle list; a selected view position / angle list corresponding to the 3D volume partition; an index of the 2D / 3D partition corresponding to the selected view position / angle; quality (rank) information of each 2D / 3D partition; e.g., rendering information of each 2D / 3D partition according to each view position / angle. Invoking such metadata, when requested by a user of the V-PCC player or directed by a content creator for a user of the V-PCC player, can enable more efficient processing of specific portions of 6DoF media that are desirable with respect to such metadata, thereby allowing the V-PCC player to focus on specific portions of the 6DoF media and deliver a higher quality image than other portions, rather than delivering unused portions of the 6DoF media.

[0071] For example, the cloud server block 607 can deliver a file or one or more segments of the file directly from the file / segment encapsulation block 606 to either the V-PCC player 625 or the cloud server using a delivery mechanism (e.g., via Dynamic Adaptive Streaming over HTTP (DASH)), and the cloud server block 607 can extract one or more tracks and / or one or more specific 2D / 3D partitions from the file and merge multiple coded point cloud data into one data.

[0072] If the current viewing position and angle is defined in a 6DoF coordinate system on the client system according to data such as the Position / View Angle Tracking block 608, then the Cloud Server block 607 can deliver the view position / angle metadata from the File / Segment Encapsulation block 606 or otherwise process it from files or segments already on the cloud server, so that the cloud server can extract the appropriate partitions from the store files and merge them (if necessary) depending on the metadata from the client system, for example, with the V-PCC Player 625, and the extracted data can be delivered to the client as files or segments.

[0073] For such data, in the file / segment decapsulation block 615, the file decapsulator processes the file or received segment, extracts the coded bitstream, parses the metadata, and then the coded point cloud data is decoded in the video decoding block 610 and image decoding block 611 and reconstructed in the point cloud reconstruction block 612, and the reconstructed point cloud data can be displayed in the display block 614 and / or initially constructed according to one or more different scene descriptions in the scene construction block 613 in relation to the scene description data from the scene generator block 609.

[0074] In view of the above, such an exemplary V-PCC flow represents an advantage over the V-PCC standard, which includes one or more of the described partitioning capabilities for multiple 2D / 3D areas, compressed domain assembly of coded 2D / 3D partitions into a single conformal coded video bitstream, and bitstream extraction of coded 2D / 3D pictures into a conformal coded bitstream, and support for such a V-PCC system is further improved by including a container formation for VVC bitstreams that supports a metadata-containing mechanism for carrying one or more of the above-mentioned metadata.

[0075] In this regard, and in accordance with exemplary embodiments described further below, the term "mesh" refers to a configuration of one or more polygons that describe the surface of a volumetric object. Each polygon is defined by vertices in 3D space and information about how the vertices are connected, called connectivity information. Optionally, vertex attributes, such as color, normals, etc., can be associated with mesh vertices. Attributes can also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh in a 2D attribute map. Such mapping can be described by a set of parametric coordinates, called UV coordinates or texture coordinates, that are associated with the mesh vertices. The 2D attribute map is used to store high-resolution attribute information, such as texture, normals, and displacement. In accordance with exemplary embodiments, such information can be used for various purposes, such as texture mapping, shading, and mesh reconstruction.

[0076] Nevertheless, dynamic mesh sequences can require large amounts of data because they may consist of a large amount of information that changes over time. For example, in contrast to a “static mesh” or “static mesh sequence,” in which the mesh information remains constant from frame to frame, a “dynamic mesh” or “dynamic mesh sequence” exhibits motion in which some vertices represented by the mesh change from frame to frame. Therefore, storing and transmitting such content requires efficient compression techniques. The mesh compression standards IC, MESHGRID, and FAMC were previously developed by MPEG to address dynamic meshes with constant connectivity, time-varying geometry, and vertex attributes. However, these standards do not take into account time-varying attribute maps and connectivity information. Typically, digital content creation (DCC) tools generate such dynamic meshes. However, generating constant connectivity dynamic meshes, especially under real-time constraints, is challenging for volumetric acquisition techniques. This type of content is not supported by existing standards. According to exemplary embodiments herein, aspects of a new mesh compression standard are described that directly handles dynamic meshes with time-varying connectivity information and, optionally, time-varying attribute maps, and covers lossy and lossless compression for a variety of applications such as real-time communication, storage, free-viewpoint video, AR, VR, etc. Functionalities such as random access and scalable / progressive coding are also contemplated.

[0077] Figure 7 illustrates an example framework 700 for dynamic mesh compression, such as a 2D atlas sampling-based method. Each frame of an input mesh 701 can be preprocessed by a series of operations, such as tracking, remeshing, parameterization, and voxelization. Note that these operations may be performed solely by the encoder and not part of the decoding process; this possibility can be signaled in the metadata, e.g., by a flag indicating 0 for encoder-only and 1 for other cases. A mesh 702 with a 2D UV atlas can then be obtained, where each vertex of the mesh has one or more associated UV coordinates in the 2D atlas. Next, by sampling in the 2D atlas, the mesh can be converted into multiple maps, including a geometry map and an attribute map. These 2D maps can then be coded by a video / image codec, such as HEVC, VVC, AV1, or AVS3. At the decoder 703 side, a mesh can be reconstructed from the decoded 2D maps. Any post-processing and filtering can also be applied to the reconstructed mesh 704. Note that other metadata may be signaled to the decoder side for the purposes of 3D mesh reconstruction: chart boundary information, including uv and xyz coordinates of boundary vertices, can be predicted, quantized, and entropy coded in the bitstream; the quantization step size can be configured on the encoder side to trade off between quality and bitrate.

[0078] According to an exemplary embodiment, in some implementations, a 3D mesh can be divided into segments (or patches / charts), and one or more 3D mesh segments can be considered a "3D mesh." Each segment consists of a set of connected vertices, each of which has associated geometry, attribute, and connectivity information. As shown in the volumetric data example 800 in FIG. 8 , the UV parameterization process 802 of mapping a 3D mesh segment to a 2D chart, such as the 2D UV atlas 702 block described above, maps one or more mesh segments 801 to a 2D chart 803 in a 2D UV atlas 804. Each vertex (vn) in a mesh segment is assigned a 2D UV coordinate in the 2D UV atlas. Note that the vertices (vn) in the 2D chart form the connected component of their corresponding 3D chart. The geometry, attribute, and connectivity information of each vertex can also be inherited from their corresponding 3D vertices. For example, information can be indicated that vertex v4 is directly connected to vertices v0, v5, v1, and v3, and similar information can be indicated for each of the other vertices. Furthermore, according to an exemplary embodiment, such a 2D texture mesh would further indicate information such as color information for each patch, e.g., for each patch of each triangle (v2, v5, v3) as one "patch."

[0079] For example, see example 900 in FIG. 9, which, in addition to the features of example 800 in FIG. 8, also maps 3D mesh segment 801 to multiple separate 2D charts 901 and 902. In this case, a 3D vertex may correspond to multiple vertices in the 2D UV atlas. As shown in FIG. 9, in the 2D UV atlas, the same 3D mesh segment is mapped to multiple 2D charts rather than a single chart as in FIG. 8. For example, 3D vertices v1 and v4 each have two 2D correspondences: v1, v1' and v4, v4'. Thus, a typical 2D UV atlas for a 3D mesh may be composed of multiple charts, as shown in FIG. 14, where each chart may contain multiple vertices (typically three or more) associated with 3D geometry, attributes, and connectivity information.

[0080] FIG. 9 shows example 903 illustrating a derived triangulation in a chart with boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7. Given this information, any triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). For example, for each vertex, find the two closest vertices. Alternatively, for every vertex, continuously generate triangles until a minimum number of triangles is reached after a set number of attempts. As shown in example 903, there are various regular, repeating triangles and various irregular triangles. The irregular triangles are typically closest to the boundary vertex and have their own unique dimensions, which may or may not be the same as any other triangle. Connectivity information can also be reconstructed by explicit signaling. According to an exemplary embodiment, if a polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream.

[0081] Boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7 are defined in 2D UV space. Boundary edges can be determined by checking whether an edge appears in only one triangle. According to an exemplary embodiment, the geometry information of the bounding vertices, e.g., 3D XYZ coordinates, currently in 2D UV parametric form, and 2D UV coordinates, is important and should be signaled in the bitstream.

[0082] For example, as shown in Figure 9, if a 3D bounding vertex corresponds to multiple vertices in the 2D UV atlas, the mapping from 3D XYZ to 2D UV can be one-to-many. Therefore, a UV-to-XYZ (also called UV2XYZ) index can be signaled to indicate the mapping function. UV2XYZ can be a 1D array of indices that map each 2D UV vertex to a 3D XYZ vertex.

[0083] According to an exemplary embodiment, to efficiently represent a mesh signal, a subset of mesh vertices can be first coded along with connectivity information between them. In the original mesh, the connections between these vertices may not exist because they are subsampled from the original mesh. Because there are various ways to signal connectivity information between vertices, such a subset is called a base mesh or base vertices.

[0084] According to an exemplary embodiment, multiple methods are implemented for dynamic mesh compression and are part of the edge-based vertex prediction framework described above, which first encodes a base mesh and then predicts a larger number of additional vertices based on connectivity information from the edges of the base mesh. Note that these can be applied individually or in any combination.

[0085] For example, consider the example flowchart 1001 of vertex grouping for prediction modes in Figure 10. Referring to Figure 9, for example, vertices in a mesh can be obtained in S101 and divided into different groups for prediction purposes in S102. In one example, the division is performed using patch / chart partitioning in S104. In another example, the division is performed in each patch / chart S105. The decision S103 on whether to proceed to S104 or S105 can be signaled by a flag or the like. For S105, some vertices of the same patch / chart form a prediction group and share the same prediction mode, while some other vertices of the same patch / chart can use a different prediction mode. As used herein, a "prediction mode" can be considered to be a particular mode that a decoder uses to make predictions for video content including patches, and prediction modes can be divided categorically into intra-prediction modes and inter-prediction modes, and within each category, there can be different specific modes for the decoder to select from. According to an exemplary embodiment, in each group, the "prediction groups" can share the same specific mode (e.g., angular mode at a specific angle) or the same categorical prediction mode (e.g., all intra prediction mode, but can be predicted at different angles). Such grouping in S106 can be assigned at different levels by determining the respective number of vertices involved per group. According to an exemplary embodiment, for example, every 64, 32, or 16 vertices following the scan order in the patch / chart can be assigned the same prediction mode, and other vertices can be assigned different prediction modes. For each group, the prediction mode can be an intra prediction mode or an inter prediction mode. This can be signaled or assigned.According to the exemplary flowchart 1000, if it is determined in S107 that the mesh frame or mesh slice is of intra type, for example by checking whether the flag of the mesh frame or mesh slice indicates intra type, then all groups of vertices in the mesh frame or mesh slice must use intra prediction mode; otherwise, in S108, an intra prediction mode or an inter prediction mode can be selected for each group of vertices in the mesh frame or mesh slice.

[0086] Furthermore, if a group of mesh vertices uses intra prediction mode, the vertices can only be predicted using previously coded vertices within the same subpartition of the current mesh. According to an example embodiment, in some cases, the subpartition may be the current mesh itself; according to an example embodiment, if a group of mesh vertices uses inter prediction mode, the vertices can only be predicted using previously coded vertices from another mesh frame. Each of the above information can be determined and signaled by a flag or the like. The prediction characteristics can occur in S110, and the results and signaling of the prediction can occur in S111.

[0087] According to an exemplary embodiment, for each vertex in a group of vertices in exemplary flowchart 1000 and flowchart 1100 described below, after prediction, the residual becomes a 3D displacement vector, indicating the shift from the current vertex to its predictor. The residual for a group of vertices needs to be further compressed. In one example, the transform in S111, along with its signaling, can be applied to the residual for a group of vertices before entropy coding. The following method can be implemented to handle the coding of a group of displacement vectors. For example, one method appropriately signals the group of displacement vectors, or the situation where some displacement vectors or their components have only zero values. In another embodiment, a flag can be signaled for whether each displacement vector has a non-zero component, and if not, the coding of all components of this displacement vector can be skipped. Furthermore, in another embodiment, a flag can be signaled for whether each group of displacement vectors has a non-zero component, and if not, the coding of all displacement vectors in this group can be skipped. Furthermore, in another embodiment, a flag may be signaled as to whether each component of each group of displacement vectors has a non-zero vector, and if not, coding of this component of all displacement vectors in this group may be skipped. Furthermore, in another embodiment, a situation in which a transformation needs to be performed on a group of displacement vectors or on components of this group of displacement vectors may be signaled, and if not, the transformation may be skipped and quantization / entropy coding may be applied directly to this group or on components of this group. Furthermore, in another embodiment, a flag may be signaled as to whether a transformation needs to be performed on each group of displacement vectors, and if not, transform coding of all displacement vectors in this group may be skipped. Furthermore, in another embodiment, a flag may be signaled as to whether a transformation needs to be performed on each component of each group of displacement vectors, and if not, transform coding of this component of all displacement vectors in this group may be skipped.The above-described embodiments in this paragraph regarding the processing of vertex prediction residuals can also be combined and implemented in parallel, each on a different patch.

[0088] FIG. 11 shows an exemplary flowchart 1100. In step S121, a mesh frame coded as an entire data unit can be obtained, meaning that all vertices or attributes of the mesh frame may have correlations between them. Alternatively, depending on the determination in step S122, the mesh frame can be divided into smaller, independent subpartitions in step S123, similar in concept to slices or tiles of a 2D video or image. In step S124, a prediction type can be assigned to the coded mesh frame or coded mesh subpartition. Possible prediction types include an intra-coded type and an inter-coded type. In the case of the intra-coded prediction type, in step S125, only prediction based on a reconstructed portion of the same frame or slice is allowed. Meanwhile, in the case of the inter-prediction type, in step S125, prediction based on a previously coded mesh frame is allowed in addition to intra-mesh frame prediction. Furthermore, the inter-prediction type can be classified into more subtypes, such as a P type or a B type. In the P type, only one predictor can be used for prediction purposes, while in the B type, two predictors from two previously coded mesh frames can be used to generate a predictor. A weighted average of two predictors may be an example. If a mesh frame is coded as a whole, the frame can be considered an intra- or inter-coded mesh frame. For an inter-mesh frame, the P or B type can be further identified via signaling. Alternatively, if a mesh frame is further divided within the frame and coded, a prediction type can be assigned to each subpartition in S124. Each of the above information can be determined and signaled by a flag or the like; similar to S110 and S111 of FIG. 10, the prediction characteristics can occur in S126, and the results and signaling of the prediction can occur in S127.

[0089] Dynamic mesh sequences may therefore require large amounts of data as they may consist of large amounts of information that change over time, and therefore storing and transmitting such content requires efficient compression techniques, and the features described herein represent such efficiency improvements by enabling improved 3D position prediction of at least mesh vertices by using previously decoded vertices within the same mesh frame (intra prediction) or vertices from a previous coded mesh frame (inter prediction).

[0090] Further, exemplary embodiments may generate displacement vectors for a third layer 1303 of the mesh based on one or more reconstructed vertices of a previous layer, such as the second layer 1302 and the first layer 1301. Assuming the index of the second layer 1302 is T, a predictor T+1 for a vertex in the third layer 1303 is generated based on at least the reconstructed vertices of the current layer or the second layer 1302. An example of such a layer-based prediction structure is shown in example 1300 of FIG. 13 , which illustrates reconstruction-based vertex prediction, i.e., progressive vertex prediction using edge-based interpolation, in which predictors are generated based on previously decoded vertices rather than predictor vertices. The first layer 1301 may be a mesh bounded by a first polygon 1340 having as vertices the decoded vertices at its boundary and interpolated vertices along some lines between those decoded vertices. As the progressive coding proceeds from the first layer 1301 to the second layer 1302, additional polygons 1341 can be formed by displacement vectors from some of the interpolated vertices of the first layer to additional vertices of the second layer 1302, so that the total number of vertices of the second layer 1302 may be greater than the total number of vertices of the first layer 1301. Similarly, when proceeding to the third layer 1303, the additional vertices of the second layer 1302, along with the decoded vertices from the first layer 1301, can be used for coding in a manner similar to that used for the decoded vertices when proceeding from the first layer 1301 to the second layer 1303, i.e., multiple additional polygons can be formed. As a note, see example 1400 in Figure 14 illustrating such progressive coding; unlike Figure 13, example 1400 shows that when progressing from the first layer 1401 to the second layer 1403 and then to the third layer 1403, each of the additionally formed polygons can lie entirely within the polygon formed by the boundary of the first layer 1401.

[0091] According to an exemplary embodiment, for such examples 1300 and / or 1400, referring to the exemplary flowchart 1200 of FIG. 12, because the interpolated vertices on the current layer are predicted values, such values ​​need to be reconstructed before being used to generate predictors for vertices on the next layer. This is done by encoding a base mesh in S131, implementing the vertex prediction itself in S132, and then adding the decoded displacement vectors of the current layer to the predictors of vertices of layer 1302, etc., in S133. Then, in S135, the reconstructed vertices of this layer and all decoded vertices of the previous layer (e.g., confirming the addition of the vertex values ​​of these layers in S134) can be used to generate and signal predictor vertices for the next layer 1303. This process can also be summarized as follows: Let P[t](Vi) denote the predictor of vertex Vi on layer t, R[t](Vi) denote the reconstructed vertex Vi on layer t, D[t](Vi) denote the displacement vector of vertex Vi on layer t, and f(*) denote the predictor generator, which in particular can represent the average of two existing vertices. Then, according to an exemplary embodiment, for each layer t, we have the following equation:

[0092]

number

[0093] where Vj and Vk are the reconstructed vertices of the previous layer.

[0094]

number

[0095] Next, we split all vertices in one mesh frame into layer 0 (base mesh), layer 1, layer 2, etc. Then, the reconstruction of vertices on one layer depends on the reconstruction of vertices on the previous layer. In the above equation, P, R, and D each represent a 3D vector in the context of a 3D mesh representation. D is the decoded displacement vector, and quantization may or may not be applied to this vector.

[0096] According to an example embodiment, vertex prediction using reconstructed vertices may be applied only to certain layers, e.g., layer 0 and layer 1. For other layers, vertex prediction may continue to use adjacent predictor vertices without adding displacement vectors for reconstruction, allowing these other layers to be processed simultaneously without waiting for the previous layer to be reconstructed. According to an example embodiment, for each layer, it may be signaled whether to select reconstruction-based vertex prediction or predictor-based vertex prediction, or it may be signaled that the layer (and subsequent layers) do not use reconstruction-based vertex prediction.

[0097] If the vertex predictor is a displacement vector generated by a reconstructed vertex, quantization can be applied without further transformation, such as a wavelet transform. If the vertex predictor is a displacement vector generated by other predictor vertices, which may require a transformation, quantization can be applied to the transform coefficients of those displacement vectors.

[0098] Therefore, dynamic mesh sequences may require large amounts of data because they may consist of a large amount of information that changes over time. Therefore, storing and transmitting such content requires efficient compression techniques. In the framework of the interpolation-based vertex prediction method described above, one key step is compressing displacement vectors, which occupy a major portion of the coded bitstream and are the focus of this disclosure. Features of this disclosure alleviate this problem by providing such compression.

[0099] Additionally, as with the other examples described above, and in further embodiments thereof, dynamic mesh sequences may require large amounts of data because they may consist of a large amount of information that changes over time, and therefore, storing and transmitting such content requires efficient compression techniques. Within the framework of the 2D atlas sampling-based methods described above, significant advantages can be achieved by inferring connectivity information from sampled vertices and boundary vertices at the decoder side. This is a key part of the decoding process and is the focus of further examples described below.

[0100] According to an exemplary embodiment, the connectivity information of the base mesh can be inferred (derived) from the decoded boundary vertices and sampled vertices of each chart, both on the encoder and decoder side.

[0101] Also as mentioned above, any triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). According to an example embodiment, the type of connectivity can be signaled in a high-level syntax such as a sequence header, slice header, etc.

[0102] As mentioned above, connectivity information can also be reconstructed by explicit signaling, e.g., for irregularly shaped triangular meshes. That is, if it is determined that a polygon cannot be reconstructed by implicit rules, the encoder can signal connectivity information in the bitstream. According to exemplary embodiments, the overhead of such explicit signaling can be reduced depending on the polygon boundary.

[0103] According to an embodiment, what is decided to be signaled is only the connectivity information between the boundary vertices and the sampled locations, but the connectivity information between the sampled locations themselves is inferred.

[0104] Also, in any embodiment, connectivity information can be signaled predictively so that only the difference from the inferred connectivity (as a prediction) from one mesh to another can be signaled in the bitstream.

[0105] Note that, according to an example embodiment, the inferred triangle orientation (e.g., inferred clockwise or counterclockwise for each triangle) can be signaled for all charts in a high-level syntax such as a sequence header, slice header, etc., or can be fixed (assumed) by the encoder and decoder. The inferred triangle orientation can also be signaled differently for each chart.

[0106] As a further note, any reconstructed mesh may have a different connectivity than the original mesh, for example, the original mesh may be a triangle mesh, while the reconstructed mesh may be a polygon mesh (e.g., a quad mesh).

[0107] According to example embodiments, connectivity information for any base vertices may not be signaled; instead, the same algorithm can be used on both the encoder and decoder sides to derive edges between base vertices. According to example embodiments, predicted vertex interpolation of additional mesh vertices may be based on the derived edges of the base mesh.

[0108] According to an exemplary embodiment, a flag can be used to signal whether the connectivity information of the base vertex is signaled or derived, and such a flag can be signaled at different levels of the bitstream, such as the sequence level, the frame level, etc.

[0109] According to an exemplary embodiment, first, the same algorithm is used on both the encoder and decoder sides to derive edges between base vertices. Then, the difference between the derived edges and the actual edges is signaled by comparing them with the original connectivity of the base mesh vertices. Therefore, after decoding the difference, the original connectivity of the base vertices can be restored.

[0110] In one example, for a derived edge, if it is determined to be incorrect when compared to the original edge, such information can be signaled in the bitstream (by indicating the vertex pairs that form this edge), and for an original edge, if it is not derived, it can be signaled in the bitstream (by indicating the vertex pairs that form this edge). Furthermore, connectivity on and vertex interpolation for boundary edges can be performed separately from interior vertices and edges.

[0111] Alternatively, the exemplary embodiments described herein may advantageously ameliorate the above-mentioned technical problems with one or more of these technical solutions. For example, dynamic mesh sequences may require large amounts of data, as they may consist of large amounts of information that change over time, and therefore the exemplary embodiments described herein represent at least an efficient compression technique for storing and transmitting such content.

[0112] The above-described embodiments may also be applied to instance-based mesh coding, where an instance may be a mesh of an object or a portion of an object. For example, illustrated example 1500 in Figure 15 shows an example mesh 1501 in which various instances 1502 (representing a mesh of a cup), 1503 (representing a mesh of a spoon), and 1504 (representing a mesh of a plate) exist and can each be coded separately. Each of instances 1501, 1502, 1503, and 1504 is depicted in a respective bounding box, as described further below, with the caveat that instance 1501 can be considered to be depicted as being bounded by a "mesh-based bounding box," while each of instances 1502, 1503, and 1504 can be considered to be depicted as being bounded by a respective "instance-based bounding box."

[0113] 16, an example 1600 illustrating an example distance-based displacement coding of a 3D mesh is described, in accordance with an exemplary embodiment, for a near-lossless displacement coding of a 3D mesh (which may be considered lossless herein) based on the selection of a 3D coding. For example, if it is determined that lossy coding is not selected, a vertex point z4 is predicted based on neighboring vertices z1, z2, and z3 in the base mesh. As in the 2D case of example 1601, the distance h h If is known, point z4 can be predicted based on point z'4. On the other hand, point z'4 is located at a distance h t and h s (depending on the rate and distortion cost) at point z n or point z' n In total, to signal point z4, three distances h s , h t , h his used with an index indicating which edge is used for prediction. That is, points z1, z2, z3 may be base mesh vertices, point z4 may be a remaining vertex, point z'4 may be a projected vertex, and point z n and point z' n may be derived adjacent vertices.

[0114] Turning to example 1603 illustrating subdivision and distance-based mesh coding, such exemplary embodiment similarly introduces displacement coding of the lossy 3D mesh selected in S2008 based on distance and surface subdivision. That is, similar to example 1602, in example 1603, the projected vertex of point x4 lies on point x'4 of the base mesh surface, and is at a distance d' h is sufficient to encode point x4. In this embodiment, we first subdivide the surface at level L. The nearest subdivision point to point x'4 (in this example, x n ) and then move the distance d h Point x^4 (translation comment: ^ is a symbol that replaces the symbol directly above x, and so on) n The point x^4 can be considered as a non-invertible version of the point x4. Finally, the distance d h and the subdivided point x n Although example 1603 shows a triangular subdivision, other polygon shapes may be used as described herein. That is, points x1, x2, x3 may be base mesh vertices, point x4 may be a remaining vertex, point x'4 may be a projected vertex, and point x n may be the nearest subdivision and point x^4 is the predicted vertex.

[0115] As discussed above for example 1601, example 1603 also represents an additional advantageous improvement over example 1602, since it simplifies the computational complexity compared to situations where some points z4 and z'4 may not have integer values ​​(for the sake of description, points z4 and z'4 correspond to points x4 and x'4, respectively). That is, if point x4 is selected as the closest point to point x4 (from the vertices of the polygon regularly divided within the entire polygon formed by vertices x1, x2, and x3), n By finding the point x n is more likely to have an integer value than point x'4, and therefore point x^4 as the vertex predicted from it is also likely to have an integer value, thereby reducing the computational complexity compared to point x4, which is less likely to have such an integer value.

[0116] The embodiments described herein can be further applied to a subdivision scheme to advantageously utilize dynamic meshes for efficient compression, a 1D diagram of which is shown in example 1700 of FIG. 17, where an original curve 1701 is first decimated by a decimation curve 1702 and then subdivided by a subdivision curve 1703.

[0117] 17 shows an original curve 1701, a decimated curve 1702, a subdivision curve 1703, and a displacement curve 1704. The decimated curve 1702 is decimated compared to the original curve 1701. The subdivision curve 1703 is subdivided compared to the decimated curve 1702. The displacement curve 1704 is displaced compared to the subdivision curve 1703.

[0118] The original curve 1701 includes mesh points such as point 1710, point 1711, point 1712, point 1713, point 1714, point 1715, point 1716, point 1717, point 1718, point 1719, point 1720, point 1721, point 1722, point 1723, and point 1724, which may be vertex points.

[0119] Decimated curve 1702 may be a decimated version of original curve 1782 and may include only points 1710, 1712, 1713, 1716, 1720, and 1724, thereby simplifying it compared to original curve 1701. The decimation may be based on the complexity of original curve 1701, such that a minimal number of, or at least a smaller number of, straight lines approximate original curve 1701, as shown by example decimated curve 1702.

[0120] The subdivision curve 1703 is a subdivision version of the decimate curve 1703 and includes not only points 1710, 1712, 1713, 1716, 1720, and 1724, but also subdivision points between them, such as points 1731′, 1732′, 1733′, 1734′, 1735′, 1736′, 1737′, 1738′, 1739′, 1740′, 1741′, 1742′, 1743′, 1744′, and 1745′. The subdivision points can be added as a predetermined number of points, such as three, between each point of the decimate curve 1702. The number of points may be set differently. Each of the subdivision points, point 1731', point 1732', point 1733', point 1734', point 1735', point 1736', point 1737', point 1738', point 1739', point 1740', point 1741', point 1742', point 1743', point 1744' and point 1745', potentially serves as an anchor on decimation curve 1702 to which a displacement may be applied, as represented by subdivision curve 1703.

[0121] Displaced curve 1704 is a displaced version of subdivision curve 1703 and includes not only points 1710, 1712, 1713, 1716, 1720, and 1724, but also displaced points therebetween, such as points 1731'', 1732'', 1733'', 1737'', 1738'', 1739'', 1740'', 1741'', 1742'', 1743'', 1744'', and 1745'', as well as subdivision points 1734', 1735', and 1736', which are not displaced compared to subdivision curve 1703 because they already accurately reflect their respective portions of original curve 1701. Points 1731'', 1732'', 1733'', 1737'', 1738'', 1739'', 1740'', 1741'', 1742'', 1743'', 1744'', and 1745'' are displaced compared to points 1731', 1732', 1733', 1737', 1738', 1739', 1740', 1741', 1742', 1743', 1744', and 1745', thereby reflecting the original curve 1701. Such reflection may be non-reversible or reversible, depending on the various embodiments described herein.

[0122] The subdivision polyline from the subdivision curve 1703 is then transformed into a displacement curve 1704 to obtain a better approximation of the original curve 1701. More precisely, a displacement vector, if any, is calculated for each vertex of the subdivision mesh such that the shape of the displacement curve 1704 approximates as closely as possible the shape of the original curve 1701, or as desired according to one or more of the aspects provided herein. A key advantage of the subdivision curve 1703 is that it has a subdivision structure that allows for efficient compression while providing a faithful approximation of the original curve 1701. In general, prediction-based mesh vertex coding methods require a displacement vector that indicates the difference between a vertex prediction and its intended position in 3D space.

[0123] The proposed methods or method steps may be used alone or in any combination in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0124] In this disclosure, we propose several methods that improve upon existing approaches for compressing mesh vertex displacements. These methods can be applied individually or in any combination. Furthermore, the proposed methods can be generalized to the case of mesh vertex coding with or without subdivision.

[0125] Subdivision-based methods can compute displacements per vertex with respect to a global coordinate system (GCS) or a local coordinate system (LCS). Computing displacements in a GCS is less complex, but coding efficiency is typically lower than with an LCS. Using an LCS may be preferable to a GCS in terms of coding efficiency because an LCS produces smaller magnitude displacements, but the LCS must be computed per vertex at each vertex position. Therefore, a low-complexity algorithm for computing an LCS is desirable, especially for meshes with a large number of vertices (and therefore a large number of LCSs).

[0126] When computing an LCS, one typically needs to choose one basis vector of the LCS at a vertex as the unit vector (denoted as n) normal to the vertex, and compute the other two basis vectors in the tangential directions (described below).

[0127] In one embodiment, the standard basis e1, e2, e3, etc. 3 We can specify the basis of each standard basis vector e i onto the normal vector n by taking the dot product between them.

[0128]

number

[0129] where c i is, e i is the coefficient that projects n onto n. Then, as a candidate vector (denoted as e) for one tangent basis vector of the LCS, we select the vector with the smallest magnitude |c i e with | i In other words, e can be selected from all e i This is the basis vector that is "most perpendicular" to n among (i=1, 2, 3).

[0130] However, since e may not be perfectly perpendicular to n, we need to subtract the component of e that is in the normal direction to produce a single tangent basis vector t that is perpendicular to n.

[0131]

number

[0132] After obtaining the tangent basis vector t, we can normalize t by its norm to obtain a unit vector (further denoted as t), and then take the cross product of n and t to obtain another unit tangent vector.

[0133] In another embodiment, two tangent basis vectors are generated directly from the normal basis vector n. First, the three components of n are divided into n i (i=1, 2, 3) and assume, without loss of generality, that n1 is the component with the largest magnitude. Then, one tangent basis vector e can be specified as

[0134]

number

[0135] Specify t as t = [0,1,0] if n2 = n3 = 0. Since the dot product of t and n is 0, we know that t is a vector perpendicular to n, and therefore t is indeed a tangent vector. Similarly, we can calculate t where n2 or n3 is the component with the largest magnitude. As above, after calculating one tangent basis vector t, we can normalize t to get a unit vector, and then take the cross product of t and n to get another unit tangent vector. Note that criteria other than maximum magnitude can be used to select the components of n to be set to 0.

[0136] If only the normal component of the displacement needs to be coded, there is no need to compute the LCS: in this case, we can simply project the displacement vector onto the unit normal vector n at each vertex and encode the projection coefficients.

[0137] According to an embodiment, when calculating an LCS, one basis vector of the LCS at a vertex can be selected as a unit vector (denoted as n) in the normal direction of the vertex, and two other basis vectors in the tangential direction can be calculated (described below).

[0138] According to an embodiment, FIG. 18 shows an example 1800, in which in S1801, R 3 Next, in S1802, each standard basis vector e i can be projected onto the normal vector n by taking the dot product between them. See equation (1) above.

[0139] Then, in S1803, the vector with the smallest magnitude |c i e with | i In other words, e is selected from all e i(i=1, 2, 3), is the basis vector that is "most perpendicular" to n. However, since e may not be perfectly perpendicular to n, this is determined in S1804, and the component of e in the normal direction is subtracted in S1805 to generate one tangent basis vector t that is perpendicular to n. See equation (2) above.

[0140] For an LCS, after obtaining the tangent basis vector t in S1806, in an embodiment, t may be normalized by its norm in S1807 to obtain a unit vector (further denoted as t), and then in an embodiment, n may be cross-producted with t in S1808 to obtain another unit tangent vector, and the process may be repeated for other LCSs or encoded and transmitted for decoding processing.

[0141] According to an embodiment, Fig. 19 shows an example 1900, in which in S1901, two tangent basis vectors are directly generated from a normal basis vector n. That is, in the embodiment, first, in S1901, three components of n are converted into n i (i=1, 2, 3), and without loss of generality, n i It can be assumed that is the component with the largest magnitude. Then, one tangent basis vector t can be specified as: Refer to equation (3) above.

[0142] If it is determined in S1902 that n2 = n3 = 0, then in S1903, t is designated as t = [0, 1, 0]. Because the dot product of t and n is 0, it can be seen that t is a vector perpendicular to n, and therefore t is indeed a tangent vector. Similarly, in embodiments, it is possible to calculate t where n2 or n3 is the component with the largest magnitude. As in S1804 to S1808 above and thereafter, after calculating one tangent basis vector t in S1903, in embodiments, t is normalized to obtain a unit vector, and then the cross product of t and n is taken to obtain another unit tangent vector. Note that in embodiments, criteria other than maximum magnitude can be used to select the component of n to be set to 0.

[0143] If only the normal component of the displacement needs to be coded (which can be set in advance), then embodiments may not need to calculate the LCS. In this case, embodiments may simply project the displacement vector onto the unit normal vector n of each vertex and encode the projection coefficients.

[0144] Therefore, embodiments herein more quickly calculate the local coordinate system for the displacement vectors used for mesh compression. See the comparative example below.

[0145] As shown below, there is a comparative example comparing the number of arithmetic operations performed to calculate an LCS at one vertex in an existing method, "reference software," with those embodiments herein, which are summarized in Table 1. As shown in Table 1, it can be seen that the number of additions / subtractions and / or multiplications / divisions according to the embodiments herein is about 29% and about 42% of the number of additions / subtractions and / or multiplications / divisions in the reference software.

[0146] [Table 1]

[0147] Evaluations were also performed for all test cases for the embodiments herein, and performance was compared with the reference software, as shown in Table 2. As can be seen from Table 2, the difference between the coding performance of the methods in terms of BD-TotalRate may be negligible. However, the computation time of the LCS according to the embodiments herein was consistently reduced by approximately 30% (±1%) across all test cases compared to the reference software.

[0148] [Table 2]

[0149] Therefore, embodiments herein represent a computational simplification of the LCS and are evaluated in terms of coding performance compared to reference software. Analysis and testing results show that embodiments herein can significantly reduce the computational complexity of the LCS without sacrificing rate-distortion performance.

[0150] Furthermore, according to embodiments, the features of the embodiments described herein relating to the LCS also apply to the GCS as at least part of the GCS.

[0151] The techniques described above may be implemented as computer software using computer-readable instructions, physically stored on one or more computer-readable media, or implemented by one or more specially configured hardware processors. For example, Figure 20 illustrates a computer system 2000 suitable for implementing certain embodiments of the disclosed subject matter.

[0152] The computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly configured to create code containing instructions that can be executed directly by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or by interpretation, microcode execution, etc.

[0153] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0154] 20 for computer system 2000 are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system 2000.

[0155] The computer system 2000 may include certain human interface input devices that can respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0156] The human interface input devices may include one or more of a keyboard 2001, a mouse 2002, a trackpad 2003, a touchscreen 2010, a joystick 2005, a microphone 2006, a scanner 2007, and a camera 2008 (only one of each is shown).

[0157] The computer system 2000 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 2010 or joystick 2005, but may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers 2009, headphones (not shown), etc.), visual output devices (e.g., screens 2010, including CRT screens, LCD screens, plasma screens, OLED screens, etc., each of which may or may not have touchscreen input and haptic feedback capabilities, and some of which may output two-dimensional visual output or three- or more-dimensional output via means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0158] The computer system 2000 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 2020 or similar media with CD / DVD 2011, thumb drives 2022, and removable hard drives or solid state drives 2023, legacy magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0159] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0160] The computer system 2000 may also include an interface 2099 to one or more communications networks 2098. The network 2098 may be, for example, a wireless, wired, or optical network. The network 2098 may further be a local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant network, etc. Examples of the network 2098 include local area networks such as Ethernet and wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; vehicular and industrial networks including CAN bus; and the like. A particular network 2098 typically requires an external network interface adapter connected to a particular general-purpose data port or peripheral bus (2050 and 2051) (e.g., a USB port on the computer system 2000). Other networks are typically integrated into the core of the computer system 2000 by connection to the system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system), as described below. Using any of these networks 2098, computer system 2000 can communicate with other entities. Such communications may be unidirectional receive only (e.g., broadcast TV), unidirectional transmit only (e.g., from a CAN bus to a particular CAN bus device), or bidirectional with other computer systems using, for example, local or wide area digital networks. As noted above, specific protocols and protocol stacks may be used for each of these networks and network interfaces.

[0161] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to core 2040 of computer system 2000 .

[0162] The core 2040 may include one or more central processing units (CPUs) 2041, graphics processing units (GPUs) 2042, graphics adapters 2017, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 2043, hardware accelerators 2044 for specific tasks, etc. These devices may be connected via a system bus 2048, along with read-only memory (ROM) 2045, random access memory 2046, and internal mass storage device 2047, such as an internal hard drive or SSD, that is not user accessible. In some computer systems, the system bus 2048 is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus 2048 or via a peripheral bus 2049. Peripheral bus architectures include PCI, USB, etc.

[0163] The CPU 2041, GPU 2042, FPGA 2043, and accelerator 2044 may combine to execute specific instructions that may constitute the aforementioned computer code. That computer code may be stored in ROM 2045 or RAM 2046. Transient data may also be stored in RAM 2046, while permanent data may be stored, for example, in internal mass storage device 2047. Cache memory, which may be closely associated with one or more of the CPU 2041, GPU 2042, mass storage device 2047, ROM 2045, RAM 2046, etc., may be used to enable fast storage and retrieval of any memory device.

[0164] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0165] By way of example only and not limitation, architecture 2000, and specifically a computer system having core 2040, may provide functionality as a result of processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media associated with user-accessible mass storage devices, as introduced above, as well as specific storage devices of the core 2040 that are non-transitory in nature, such as the core's internal mass storage device 2047 or ROM 2045. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core 2040. The computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause the core 2040, and specifically the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM 2046 and modifying such data structures according to software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 2044) that can operate in place of or together with software to perform particular processes or portions of particular processes described herein. References to software may, where appropriate, include logic, and vice versa. References to computer-readable media may, where appropriate, include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0166] While this disclosure has described some exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It should thus be understood that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure, which are within its spirit and scope.

Claims

1. 1. A method for video decoding executed by at least one processor, comprising: obtaining a coded bitstream comprising a mesh sequence of a plurality of meshes of 3D visual content and a displacement of a local coordinate system (LCS) or a global coordinate system (GCS) of at least one mesh of the plurality of meshes, the LCS or GCS being at a vertex of the at least one mesh; determining projection coefficients of the displacements of the LCS or GCS, the projection coefficients representing unit tangent vectors of the LCS or GCS in the normal direction of the vertex and based on normalization of tangent basis vectors of the LCS or GCS; and decoding the mesh sequence based on projection coefficients of the displacements.

2. The method of claim 1 , wherein the projection coefficients are based on a plurality of tangent basis vectors, including tangent basis vectors of the LCS or GCS.

3. The method of claim 1 , wherein the tangent basis vectors include three components: a first component, a second component, and a third component.

4. The method of claim 3 , wherein the first component comprises a magnitude greater than both the second component and the third component.

5. The tangent basis vectors of the LCS or GCS are t=[0, −n ３ , n ２ ], where t represents the tangent basis vector, and −n ３ represents the third component, and n ２ The method of claim 3 , wherein: represents the second component.

6. 4. The method of claim 3, wherein if the values ​​of the second and third components are both 0, then the tangent basis vectors of the LCS or GCS are specified as t = [0, 1, 0], where t represents the tangent basis vectors.

7. 4. The method of claim 3, wherein determining a component of the displacement of the LCS or GCS comprises determining another unit tangent vector that represents a cross product of the unit tangent vector and a normal basis vector from which the tangent basis vector is generated.

8. The method of claim 1 , wherein the projection coefficient of the LCS or GCS displacement further indicates another unit tangent vector that is a cross product of a normal vector and the unit tangent vector.

9. The method of claim 1 , wherein the tangent basis vectors of the LCS or GCS represent at least one component of a standard basis vector minus a portion of the at least one component in the normal direction.

10. The method of claim 7 , wherein the standard basis vectors are pre-specified and are one of three pre-specified standard basis vectors for the LCS or GCS.

11. at least one memory configured to store computer program code; at least one processor configured to access the computer program code and to operate according to instructions of the computer program code, the computer program code comprising: acquisition code configured to cause the at least one processor to acquire a coded bitstream comprising a mesh sequence of a plurality of meshes of 3D visual content and a displacement of a local coordinate system (LCS) or a global coordinate system (GCS) of at least one mesh of the plurality of meshes, the LCS or GCS being at a vertex of the at least one mesh; and decision code configured to cause the at least one processor to determine projection coefficients of displacements of the LCS or GCS, the projection coefficients representing unit tangent vectors of the LCS or GCS in a normal direction of the vertex and based on normalization of tangent basis vectors of the LCS or GCS; and a decoding code configured to cause the at least one processor to decode the mesh sequence based on the displacement projection coefficients.

12. The apparatus of claim 11 , wherein the projection coefficients are based on a plurality of tangent basis vectors, including tangent basis vectors of the LCS or GCS.

13. The apparatus of claim 11 , wherein the tangent basis vector includes three components: a first component, a second component, and a third component.

14. 14. The apparatus of claim 13, wherein the first component comprises a magnitude greater than both the second component and the third component.

15. The tangent basis vectors of the LCS or GCS are t=[0, −n ３ , n ２ ], where t represents the tangent basis vector, and −n ３ represents the third component, and n ２ The apparatus of claim 13 , wherein: represents the second component.

16. 14. The apparatus of claim 13, wherein if the values ​​of the second component and the third component are both 0, then a tangent basis vector of the LCS or GCS is specified as t = [0, 1, 0], where t represents the tangent basis vector.

17. 14. The apparatus of claim 13, wherein determining a component of the displacement of the LCS or GCS comprises determining another unit tangent vector that represents a cross product of the unit tangent vector and a normal basis vector from which the tangent basis vector is generated.

18. The apparatus of claim 11 , wherein the projection coefficient of the displacement of the LCS or GCS further indicates another unit tangent vector that is a cross product of a normal vector and the unit tangent vector.

19. The apparatus of claim 11 , wherein the tangent basis vectors of the LCS or GCS are represented by at least one component of a standard basis vector minus a portion of the at least one component in the normal direction.

20. A non-transitory computer-readable medium storing a program, the program comprising: obtaining a coded bitstream comprising a mesh sequence of a plurality of meshes of 3D visual content and a displacement of a local coordinate system (LCS) or a global coordinate system (GCS) of at least one mesh of the plurality of meshes, the LCS or GCS being at a vertex of the at least one mesh; determining projection coefficients of the displacements of the LCS or GCS, the projection coefficients representing unit tangent vectors of the LCS or GCS in the normal direction of the vertex and based on normalization of tangent basis vectors of the LCS or GCS; and decoding the mesh sequence based on projection coefficients of the displacements.

Images