Method and apparatus for video coding, method and apparatus for storing and transmitting video bitstream, and computer readable storage medium

Abstract

Method and apparatus for video coding, method and computer readable storage medium for storing and transmitting video bitstream, method for video coding includes: obtaining a mesh, the mesh represents volume data of at least one three-dimensional (3D) visual content; mapping adjacent normals of octahedron normal vectors (ONVs) of the mesh to a two-dimensional plane to predict the ONVs based on a reference vector, the reference vector is calculated according to a partial adjacent normal, and encoding the volume data according to the predicted ONVs.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Patent Application No. 19 / 464,834, filed January 30, 2026, and U.S. Provisional Application No. 63 / 773,512, filed March 18, 2025, the disclosures of which are incorporated herein by reference in their entirety. Technical Field

[0002] This application discloses methods and apparatus for predicting octahedral normal vectors, and more particularly for video encoding and decoding methods, methods for storing and transmitting video streams, and computer-readable storage media. Background Technology

[0003] Advances in 3D capture, modeling, and rendering have facilitated the ubiquity of 3D content across multiple platforms and devices. Today, it's possible to capture the first steps of a baby on one continent and allow grandparents on another continent to see (and interact with) them, creating an immersive experience for the child. However, to achieve this realism, the models have become increasingly complex, and vast amounts of data are associated with their creation and application. 3D meshes are widely used to represent this immersive content.

[0004] A mesh consists of several polygons that describe the surface of a volumetric object. Each polygon is defined by its vertices in three-dimensional space and information about how those vertices are connected (called connectivity information). Optionally, vertex attributes (such as color, normals, etc.) can be associated with mesh vertices. Attributes can also be associated with the mesh's surface using mapping information that parameterizes the mesh as a 2D attribute graph. This mapping is typically described by a set of parametric coordinates (called UV coordinates or texture coordinates) associated with the mesh vertices. The two-dimensional attribute graph is used to store high-resolution attribute information such as texture, normals, displacement, etc. Such information can be used for various purposes, such as texture mapping and shading.

[0005] Dynamic mesh sequences can require large amounts of data because they may consist of a wealth of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Mesh compression standards IC, MESHGRID, and FAMC, previously developed by MPEG, are used to handle dynamic meshes with constant connectivity and time-varying geometry and vertex properties. However, these standards do not consider time-varying property graphs and connectivity information. Digital Content Creation (DCC) tools typically generate such dynamic meshes. On the other hand, generating constant connectivity dynamic meshes using volumetric (3D) capture techniques is challenging, especially under real-time constraints. Existing standards do not support this type of content. MPEG is planning to develop a new mesh compression standard to directly handle dynamic meshes with time-varying connectivity information and optional time-varying property graphs. This standard offers lossy and lossless compression for various applications such as real-time communication, storage, free-viewpoint video, augmented reality (AR), and virtual reality (VR). The standard also considers features such as random access and scalable / progressive coding.

[0006] While it may be necessary to use octahedral normals to represent normal vectors, these octahedral normals need to be predicted before encoding and decoding. Therefore, there is an urgent need to propose technical solutions to this type of problem in video coding technology. Summary of the Invention

[0007] A method for video decoding is provided, the method being executed by at least one processor, and the method comprising: obtaining a mesh from a bitstream, the mesh representing encoded volume data of at least one three-dimensional (3D) visual content; mapping adjacent normals of an octahedral normal vector (ONV) of the mesh onto a two-dimensional plane based on a reference vector, the reference vector being calculated based on a subset of adjacent normals; and decoding the encoded volume data based on the predicted ONV.

[0008] A method for video encoding is provided, the method being executed by at least one processor, and the method comprising: obtaining a mesh representing volume data of at least one three-dimensional (3D) visual content; mapping adjacent normals of an octahedral normal vector (ONV) of the mesh onto a two-dimensional plane based on a reference vector, the reference vector being calculated based on a subset of adjacent normals; and encoding the volume data based on the predicted ONV.

[0009] An apparatus for video decoding is provided, the apparatus being configured to perform the aforementioned method for video decoding.

[0010] An apparatus for video encoding is provided, the apparatus being configured to perform the aforementioned method for video encoding.

[0011] A method for storing a video stream is provided, comprising performing the aforementioned method for video encoding to generate the video stream; and storing the video stream.

[0012] A method for transmitting a video stream is provided, comprising performing the aforementioned method for video encoding to generate the video stream; and transmitting the video stream.

[0013] A computer-readable storage medium is provided that stores a video stream, the video encoding being generated by the aforementioned method for video encoding.

[0014] The ONV can be obtained by mapping a 2-norm unit sphere to a 1-norm unit sphere in an octahedron and projecting and unfolding the 1-norm unit sphere into a unit square using any of the above methods.

[0015] The prediction of the ONV can be determined by any of the above factors, depending on the bit depth at which the normal is quantized.

[0016] The reference vector can be calculated as any of the adjacent normals using any of the above methods.

[0017] By any of the above methods, the reference vector can be calculated as the average of the adjacent normals.

[0018] By any of the above methods, the reference vector can be calculated as the average of a subset of the adjacent normals.

[0019] Predicting the ONV using any of the above methods may include: calculating the 2D residual. Attached Figure Description

[0020] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which: Figure 1 This is a block diagram of a communication system according to an embodiment; Figure 2 This is a block diagram of a video encoder and decoder in a video streaming environment according to an embodiment; Figure 3 This is a block diagram of a video decoder according to an embodiment; Figure 4 This is a block diagram of a video encoder according to an embodiment; Figure 5 This is a block diagram of the content streaming process in a video streaming environment according to an embodiment; Figure 6 This is a block diagram of the content streaming process in a video streaming environment according to an embodiment; Figure 7 This is a block diagram of a video encoder and decoder in a video streaming environment according to an embodiment; Figure 8 This is a simplified illustration of mapping 3D mesh fragments to 2D tiles according to an embodiment; Figure 9 This is a simplified illustration of derived triangulation performed in a block with boundary vertices according to an embodiment. Figure 10 This is a simplified flowchart of the features of the prediction pattern according to the embodiments; Figure 11 This is a simplified flowchart of media processing according to an embodiment; Figure 12 This is a simplified flowchart of media processing according to an embodiment; Figure 13 This is a simplified illustration of vertex prediction according to an embodiment; Figure 14 This is a simplified illustration of progressive coding according to an embodiment; Figure 15 These are simplified illustrations of various instances of grid representation according to the embodiments; Figure 16 This is a simplified illustration of converting a normal vector on a 2-norm unit sphere into an ONV according to an embodiment; Figure 17 This is a simplified flowchart for ONV prediction according to an embodiment; Figures 18 to 23 A comparative example is shown, illustrating the improvements achieved according to the embodiments; and Figure 24 This is a simplified diagram of computer features according to an embodiment. Detailed Implementation

[0021] The features discussed below can be used individually or in any combination in any order. Furthermore, embodiments can be implemented by processing circuitry (e.g., at least one processor or at least one integrated circuit). In one example, at least one processor executes a program stored in a non-transitory computer-readable medium.

[0022] Figure 1 A simplified block diagram of a communication system 100 according to an embodiment disclosed in this application is shown. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. For unidirectional data transmission, the first terminal 103 may encode video data locally for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal from the network 105, decode the encoded data, and display the recovered video data. Unidirectional data transmission is common in media service applications, etc.

[0023] Figure 1A second pair of terminals 101 and 104 is shown, which are provided to support bidirectional transmission of encoded video that may occur, for example, during a video conference. For bidirectional data transmission, each terminal 101 and 104 can encode video data acquired at a local location for transmission to the other terminal via network 105. Each terminal 101 and 104 can also receive encoded video data sent by the other terminal, can decode the encoded data, and can display the recovered video data on a local display device.

[0024] exist Figure 1 In this application, terminals 101, 102, 103, and 104 may be shown as servers, personal computers, and smartphones, but the principles disclosed herein are not limited thereto. The embodiments disclosed herein are applicable to laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 refers to any number of networks, including, for example, wired and / or wireless communication networks, that transmit encoded video data between terminals 101, 102, 103, and 104. Communication network 105 may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), and / or the Internet. For the purposes of this discussion, unless explained below, the architecture and topology of network 105 may be irrelevant to the operation of this application.

[0025] As an example of an application used for the disclosed topic, Figure 2 The arrangement of video encoders and decoders in a streaming environment is illustrated. The disclosed subject matter can be equally applied to other video-enabled applications, including, for example, video conferencing, digital TV, storing compressed video on digital media including CDs, DVDs, Memory Sticks, etc.

[0026] The streaming system may include an acquisition subsystem 203, which may include a video source 201, such as a digital camera, that creates, for example, an uncompressed video sample stream 213. This sample stream 213 may be emphasized as having a high data volume compared to an encoded video stream and may be processed by an encoder 202 coupled to the video source 201, which may be, for example, a camera as described above. The encoder 202 may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. The encoded video stream 204 may be emphasized as having a lower data volume compared to the sample stream and may be stored on a streaming server 205 for future use. At least one streaming client 212 and 207 may access the streaming server 205 to retrieve copies 208 and 206 of the encoded video stream 204. Client 212 may include a video decoder 211 that decodes an input copy of the encoded video stream 208 and produces an output video sample stream 210 that can be rendered on a display 209 or other rendering device (not depicted). In some streaming systems, video streams can be encoded using 204, 206, and 208 bitrates according to certain video coding / compression standards. Implementations of these standards are mentioned above and further described herein.

[0027] Figure 3 This can be a functional block diagram of a video decoder 300 according to an embodiment of the present invention.

[0028] Receiver 302 may receive at least one codec video sequence to be decoded by decoder 300; in the same or another embodiment, one encoded video sequence is received at a time, wherein the decoding of each encoded video sequence is independent of other encoded video sequences. Encoded video sequences may be received from channel 301, which may be a hardware / software link to a storage device storing the encoded video data. Receiver 302 may receive encoded video data as well as other data, such as encoded audio data and / or auxiliary data streams that may be forwarded to their respective user entities (not depicted). Receiver 302 may separate the encoded video sequences from other data. To prevent network jitter, buffer memory 303 may be coupled between receiver 302 and entropy decoder / resolver 304 (hereinafter referred to as the "resolver"). Buffer 303 may be unnecessary or can be made small when receiver 302 receives data from a store / forward device with sufficient bandwidth and controllability or from a synchronous network. Buffer 303 may also be required for use on packet-switched networks such as the Internet, and the buffer may be relatively large and advantageously have an adaptive size.

[0029] The video decoder 300 may include a parser 304 to reconstruct symbols 313 based on an entropy-encoded video sequence. These symbols may include information for managing the operation of the decoder 300, and potential information for controlling a display device (such as a display 312), which is not part of the decoder but may be coupled to it. Control information for the display device may be in the form of Supplementary Enhancement Information (SEI) messages or fragments of Video Usability Information (VUI) parameter sets (not depicted). The parser 304 may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 304 may extract a subset of parameters from at least one pixel subset of pixels in the video decoder based on at least one parameter corresponding to a group. Subgroups can include group of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The entropy decoder / parser can also extract information such as transform coefficients, quantizer parameter values, and motion vectors from the encoded video sequence.

[0030] The parser 304 can perform entropy decoding / parsing operations on the video sequence received from the buffer 303 to create symbol 313. The parser 304 can receive encoded data and selectively decode specific symbols 313. Furthermore, the parser 304 can determine whether to provide specific symbols 313 to the motion compensation prediction unit 306, the scaler / inverse transform unit 305, the intra-frame prediction unit 307, or the loop filter 311.

[0031] Depending on the type of encoded video frames or a subset of encoded video frames (e.g., inter-frame and intra-frame frames, inter-frame and intra-frame blocks) and other factors, the reconstruction of symbol 313 may involve multiple different units. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by parser 304. For brevity, the flow of such subgroup control information between parser 304 and the various units described below is not described.

[0032] In addition to the functional blocks already mentioned, the decoder 300 can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and can be integrated with one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0033] The first unit is the scaler / inverse transform unit 305. The scaler / inverse transform unit 305 receives quantization transform coefficients and control information from the parser 304, including which transform to use, block size, quantization factor, quantization scaling matrix, etc., as symbols 313. It can output blocks containing sample values, which can be input into the aggregator 310.

[0034] In some cases, the output samples of the scaler / inverse transform unit 305 may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit 307. In some cases, the intra-picture prediction unit 307 uses surrounding reconstructed information extracted from the current (partially reconstructed) image 309 to generate blocks of the same size and shape as the blocks being reconstructed. In some cases, the aggregator 310 adds the predictive information generated by the intra-picture prediction unit 307 to the output sample information provided by the scaler / inverse transform unit 305 based on each sample.

[0035] In other cases, the output samples of the scaler / inverse transform unit 305 may belong to inter-frame coded and potentially motion-compensated blocks. In this case, the motion-compensated prediction unit 306 can access the reference image memory 308 to extract samples for prediction. After motion compensation of the extracted samples according to the block-related symbol 313, these samples can be added by the aggregator 310 to the output of the scaler / inverse transform unit (referred to as residual samples or residual signals in this case) to generate output sample information. The address in the reference image memory form from which the motion compensation unit extracts the predicted samples can be controlled by motion vectors, which can be provided to the motion compensation unit in the form of symbol 313, which may have, for example, X, Y, and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0036] The output samples of aggregator 310 can undergo various loop filtering techniques in loop filter unit 311. Video compression techniques may include in-loop filtering techniques, which are controlled by parameters included in the encoded video bitstream and are available to loop filter unit 311 as symbols 313 from parser 304, but may also be in response to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and in response to previously reconstructed and loop-filtered sample values.

[0037] The output of the loop filter unit 311 can be a sample stream, which can be output to the rendering device 312 and stored in the reference image memory 557 for future inter-image prediction.

[0038] Once a certain encoded image has been fully reconstructed, it can be used as a reference image for future predictions. Once a encoded image has been fully reconstructed and has been identified as a reference image (e.g., by parser 304), the current reference image 309 can become part of the reference image buffer 308, and a new current image memory can be reallocated before starting the reconstruction of the next encoded image.

[0039] The video decoder 300 can perform decoding operations according to a predetermined video compression technique that may be documented in standards such as ITU-T H.265. An encoded video sequence is considered to conform to the syntax specified by the video compression technique or standard (especially its configuration document) in the sense that the encoded video sequence follows the syntax defined in the video compression technique or standard document. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further defined by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0040] In this embodiment, receiver 302 may receive additional (redundant) data along with the encoded video. The additional data may be a portion of the encoded video sequence. The additional data may be used by video decoder 300 to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0041] Figure 4 This may be a functional block diagram of a video encoder 400 according to an embodiment disclosed in this application.

[0042] Encoder 400 can receive video samples from video source 401 (which is not part of the encoder), and video source 401 can capture video images that will be encoded by encoder 400.

[0043] Video source 401 can provide a source video sequence in the form of a digital video sample stream encoded by video encoder 400. This digital video sample stream can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, video source 401 can be a storage device storing previously prepared video. In a video conferencing system, video source 401 can be a camera capturing local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed sequentially. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include at least one sample, depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0044] According to an embodiment, encoder 400 can encode and compress images of a source video sequence into an encoded video sequence 410 in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of controller 402. The controller controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller may include rate control related parameters (image skipping, quantizer, λ value of rate distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. Other functions of controller 402 can be readily identified by those skilled in the art, as they may be related to video encoder 400 optimized for a particular system design.

[0045] Some video encoders operate in a manner readily recognizable to those skilled in the art as an "encoding loop." In a simplified description, an encoding loop may include the encoding portion of encoder 400 (hereinafter referred to as the "source encoder") responsible for creating symbols based on the input image to be encoded and (multiple) reference images (a plurality of) images) and a (local) decoder 406 embedded in encoder 400, which reconstructs the symbols to create sample data in a manner similar to that of the (remote) decoder (because in the video compression techniques considered in this application, any compression between the symbols and the encoded video stream is lossless). This reconstructed sample stream is input to reference image memory 405. Since the decoding of the symbol stream results in bit-accurate results regardless of the decoder's location (local or remote), the contents of the reference image buffer are also bit-accurately corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction portion are exactly the same sample values ​​that the decoder will "see" during prediction. The basic principles of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) are well known to those skilled in the art.

[0046] The operation of the "local" decoder 406 can be combined with the above. Figure 3 The operation of the "remote" decoder 300 is the same as described in the detailed description. However, please refer to the following brief reference: Figure 4 When symbols are available and the entropy encoder 408 and the parser 304 are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding part of the decoder 300 (including the channel 301, receiver 302, buffer 303 and parser 304) may not be fully implemented in the local decoder 406.

[0047] It can then be observed that any decoder technique other than parsing / entropy decoding, which exists in the decoder, must also exist in the corresponding encoder in essentially the same functional form. The description of encoder techniques can be simplified because encoder techniques are inverses of the fully described decoder techniques. More detailed descriptions are only required in certain areas, and are provided below.

[0048] As part of its operation, the source encoder 403 can perform motion-compensated predictive coding. The motion-compensated predictive coding predictively encodes the input frame with reference to at least one previously encoded frame from the video sequence designated as a "reference frame." In this manner, the encoding engine 407 encodes the differences between pixel blocks of the input frame and pixel blocks of the reference frame, which can be selected as a prediction reference for the input frame.

[0049] The local video decoder 406 can decode encoded video data of frames that can be designated as reference frames based on symbols created by the source encoder 403. The operation of the encoding engine 407 can advantageously be a lossy process. When encoded video data can be decoded by the video decoder (… Figure 4 When decoded at a location (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder 406 replicates the decoding process, which can be performed by the video decoder on the reference frame, and allows the reconstructed reference frame to be stored in a reference image memory 405, which can be, for example, a cache. In this way, the encoder 400 can locally store a copy of the reconstructed reference frame that shares common content (no transmission errors) with the reconstructed reference frame to be obtained by the remote video decoder.

[0050] Predictor 404 can perform a prediction search against encoding engine 407. That is, for a new frame to be encoded, predictor 404 can search the reference image memory 405 for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. Predictor 404 can operate pixel-by-pixel based on the samples to find appropriate prediction references. In some cases, based on the search results obtained by predictor 404, it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory 405.

[0051] The controller 402 can manage the encoding operations of the source encoder 403 (which may be, for example, a video encoder), including, for example, setting parameters and subgroup parameters for encoding video data.

[0052] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder 408. The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by lossless compression according to techniques known to those skilled in the art (e.g., Huffman coding, variable-length coding, arithmetic coding, etc.).

[0053] Transmitter 409 can buffer the encoded video sequence created by entropy encoder 408, thereby preparing it for transmission via communication channel 411, which may be a hardware / software link to a storage device that will store the encoded video data. Transmitter 409 can combine encoded video data from source encoder 403 with other data to be transmitted (e.g., encoded audio data and / or auxiliary data streams (sources not shown)).

[0054] The controller 402 manages the operation of the encoder 400. During encoding, the controller 402 can assign a specific encoded image type to each encoded image, but this may affect the encoding techniques applicable to the corresponding images. For example, images can typically be assigned to one of the following frame types: An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other frames in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are familiar with variations of I-pictures and their corresponding applications and characteristics.

[0055] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0056] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0057] Source images are typically spatially subdivided into multiple sample blocks (e.g., 4×4, 8×8, 4×8, or 16×16 sample blocks), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignment of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial prediction or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be non-predictively coded with reference to a previously coded reference image via spatial prediction or temporal prediction. Blocks of a B-image can be non-predictively coded with reference to one or two previously coded reference images via spatial prediction or temporal prediction.

[0058] Encoder 400 (which may be, for example, a video encoder) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, encoder 400 can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0059] In this embodiment, transmitter 409 may transmit additional data while transmitting encoded video. Source encoder 403 may include such data as part of the encoded video sequence. Additional data may include temporal / spatial / SNR enhancement layers, redundant images and slices, other forms of redundant data, supplementary enhancement information (SEI) messages, video availability information (VUI) parameter set fragments, etc.

[0060] Figure 5 A simplified block diagram 500 illustrates an exemplary viewport-related processing workflow in the Omnidirectional Media Application Format (OMAF) that supports 360-degree virtual reality (VR360) streaming as described in OMAF.

[0061] At acquisition box 501, if the image data can represent a scene in VR360, video data A is acquired, such as data from multiple images and audio from instances at the same time. At processing box 503, image B at the same time is processed. i Perform one or more of the following processes: stitching, mapping to the projected image relative to at least one virtual reality (VR) angle or other angle / viewpoint, and packing by region. Additionally, metadata can be created to indicate any such processing information and other details to assist in the transport and rendering process.

[0062] Regarding data D, at image encoding frame 505, the projected image is encoded as data E. i And they are combined into a media file, and in viewport-independent streaming, and at video encoding frame 504, the video image is encoded into data E. v For example, a single-layer bitstream, and regarding data B a Audio data can also be encoded into data E at audio encoding frame 502. a .

[0063] Data E a E v and E i The entire encoded bitstream F iAnd / or F can be stored at a (Content Delivery Network (CDN) / cloud) server and can typically be fully transmitted to the OMAF player 520, for example, at delivery frame 507 or otherwise, and can be fully decoded by a decoder, such that at display frame 516, at least the area of ​​the decoded image corresponding to the current viewport is presented to the user relative to various metadata, file playback, and orientation / viewport metadata from head / eye tracking frame 508 (e.g., the angle at which the user can view the device relative to the viewport specification of the VR imaging device). A unique feature of VR360 is that only the viewport can be displayed at any given time, and this feature can be used to improve the performance of omnidirectional video systems through selective delivery depending on the user's viewport (or any other standard, such as recommended viewport timing metadata). For example, viewport-dependent delivery can be implemented through tile-based video encoding according to an exemplary embodiment.

[0064] Similar to the encoding frame described above, the OMAF player 520 according to the exemplary embodiment can specify the data F' and / or F' i Decapsulate at least one executable file / fragment in the metadata to reverse execute at least one step of the above encoding, and process the audio data E' at audio decoding frame 510. i Decode the video data E' at point 513 in the video decoding frame. v Decode the image data E' at image decoding frame 514. i Decode the data, and then process the data B' at point 511 in the audio rendering frame. a The audio rendering, and the image rendering of data D' at image rendering frame 515, to output display data A' in VR360 format at display frame 516 according to various metadata (such as orientation / viewport metadata). i And output audio data A' at speaker / headphone frame 512. s Various metadata can influence the corresponding stages of the data decoding and rendering process based on various tracks, languages, qualities, and views that can be selected by or for the user of the OMAF player 520. It should be understood that the processing order described herein is presented for exemplary embodiments and can be implemented in other orders according to other exemplary embodiments.

[0065] Figure 6A simplified framework for (encoded) point cloud data is illustrated in Content Flow Process Diagram 600, where acquisition / generation / (de)encoding / rendering / display of 6-DOF media employs point cloud data processing based on viewpoint position and angle correlation (referred to herein as "V-PCC"). It should be understood that the described features can be used individually or in combination in any order, and the elements shown, such as those for encoding and decoding, can be implemented by processing circuitry (e.g., at least one processor or at least one integrated circuit), and according to an exemplary embodiment, at least one processor can execute a program stored in a non-transitory computer-readable medium.

[0066] Figure 600 illustrates an exemplary embodiment for streaming encoded point cloud data according to V-PCC.

[0067] At volume data acquisition frame 601, a real-world visual scene or a computer-generated visual scene (or a combination thereof) can be acquired by a set of camera devices or synthesized by a computer into volume data, and the volume data can have any format. The volume data can be converted into a (quantized) point cloud data format through image processing at conversion to point cloud frame 602. For example, according to an exemplary embodiment, the data from the volume data can be region data, which is converted into one of the points of the point cloud by extracting at least one of the values ​​described below from the volume data and any associated data into the desired point cloud format. According to an exemplary embodiment, the volume data can be a 3D dataset of a 2D image, such as a slice, from which a 2D projection of the 3D dataset can be projected, for example. According to an exemplary embodiment, the point cloud data format includes a representation of at least one data point in various spaces and can be used to represent volume data. It can also provide improvements regarding sampling and data compression, such as regarding temporal redundancy, and point cloud data in, for example, x, y, z formats, where the point cloud data represents color values ​​(e.g., RGB, etc.), brightness, intensity, etc., at each of the multiple points of the cloud data. It can also be used with progressive decoding, polygon meshing, direct rendering, and octree 3D representations of 2D quadtree data.

[0068] At frame 603 projected onto the image, the acquired point cloud data can be projected onto the 2D image and encoded into an image / video image using video-based point cloud coding (V-PCC). The projected point cloud data can consist of attributes, geometry, occupancy map, and other metadata for point cloud data reconstruction, such as using Painter algorithms, ray casting algorithms, (3D) binary space segmentation algorithms, etc.

[0069] On the other hand, at scene generator box 609, the scene generator can generate metadata for rendering and displaying 6-DOF media, for example, based on director's intent or user preferences. Such 6DOF media can include 3D viewing scenes like those in 360VR, which, in addition to being able to rotate around the 3D X, Y, Z axes, also allow movement forward / backward, up / down, and left / right relative to or based on the virtual experience within the point cloud encoded data. The scene description metadata defines at least one scene composed of encoded point cloud data and other media data including VR360, light fields, audio, etc., and the scene description metadata can be provided to at least one cloud server and / or such as... Figure 6 And the file / fragment encapsulation / decapsulation processing indicated in the relevant description.

[0070] After video encoding frame 604 and image encoding frame 605 complete processing similar to that described above for video and image encoding (and as will be understood, audio encoding can also be implemented in the same manner), file / segment encapsulation frame 606 performs corresponding processing to combine the encoded point cloud data into a media file for file playback or a sequence of initial segments and media segments for streaming according to a specific media container file format (such as at least one video container format, e.g., DASH may also be applicable). The above description is merely an exemplary embodiment. The file container may also incorporate scene description metadata, such as from scene generator frame 1109, into the file or segment.

[0071] According to an exemplary embodiment, a file is encapsulated based on scene description metadata such that the file includes at least one viewpoint position and at least one view angle at at least one point in time within the 6DoF media, enabling the file to be transmitted upon user or creator input request. Furthermore, according to an exemplary embodiment, fragments of the file may include at least one portion of the file, such as a portion of the 6DoF media indicating a single viewpoint and angle at at least one point in time; however, these are merely exemplary embodiments and may vary depending on various conditions such as network, user, creator capabilities, and input.

[0072] According to an exemplary embodiment, point cloud data is divided into multiple 2D / 3D regions, which are independently encoded, for example, at at least one of the video encoding frame 604 and the image encoding frame 605. Each independently encoded partition of the point cloud data can then be encapsulated as a track within a file and / or segment at a file / segment encapsulation frame 606. According to an exemplary embodiment, each point cloud track and / or metadata track may include some useful metadata for viewpoint position / angle-related processing.

[0073] According to an exemplary embodiment, the metadata contained in the file and / or fragment encapsulated in the file / fragment encapsulation frame that can be used for viewpoint position / angle related processing includes one or more of the following metadata useful for viewpoint position / angle related processing, such as metadata included in the file and / or fragment encapsulated relative to the file / fragment encapsulation frame including at least one of the following: layout information of indexed 2D / 3D partitions, (dynamic) mapping information that associates 3D volume partitions with at least one 2D partition (e.g., any one of tile / tile group / slice / sub-picture), 3D position of each 3D partition in the 6DoF coordinate system, a representative viewpoint position / angle list, a selected viewpoint position / angle list corresponding to the 3D volume partition, an index of the 2D / 3D partition corresponding to the selected viewpoint position / angle, quality (rank) information of each 2D / 3D partition, and rendering information of the 2D / 3D partition, for example, depending on each viewpoint position / angle. Invoking such metadata when a request is received (such as one initiated by the user of the V-PCC player or an instruction set by the content creator for the user of the V-PCC player) allows for more efficient processing of specific portions of the 6DoF media relative to the metadata expected. This enables the V-PCC player to deliver higher quality images of the portions of the 6DoF media that are of interest than the rest, without transmitting unused portions of the media.

[0074] According to file / segment encapsulation box 606, a file or at least one segment of a file can be directly transmitted to either the V-PCC player 625 or a cloud server (such as at cloud server box 607) using a transmission mechanism (e.g., Dynamic Adaptive Streaming over HTTP (DASH)). At cloud server box 607, the cloud server can extract at least one track and / or at least one specific 2D / 3D partition from the file and can merge multiple encoded point cloud data into a single data set.

[0075] Based on data such as position / view tracking box 608, if the current view position and angle are defined in the 6DoF coordinate system on the client system, the view position and angle can be sent from file / fragment encapsulation box 606, or otherwise processed at cloud server box 607 based on files or fragments already stored on the cloud server, so that the cloud server can extract appropriate partitions from the stored files and merge them if necessary based on metadata from the client system (e.g., a client with V-PCC player 625), and can transmit the extracted data as files or fragments to the client.

[0076] Regarding this type of data, at file / fragment decapsulation box 615, the file decapsulator processes the file or received fragment and extracts the encoded bitstream and parses the metadata. At video decoding box 610 and image decoding box 611, the encoded point cloud data is decoded. Subsequently, at point cloud reconstruction box 612, this data is decoded and reconstructed into point cloud data. The reconstructed point cloud data can be displayed at display box 614 and / or can be composited at scene compositing box 613 based on at least one of various scene descriptions according to the scene description data in scene generator box 609, and then displayed.

[0077] In view of the foregoing, this exemplary V-PCC stream presents advantages over the V-PCC standard, including at least one of the following capabilities: the ability to partition multiple 2D / 3D regions as described, the ability to assemble compressed domains of encoded 2D / 3D partitions into a single compliant encoded video stream, and the ability to extract encoded 2D / 3D from encoded images to form an encoded stream, wherein such V-PCC system support is further improved by including a container format for VVC streams to support mechanisms that include metadata (carrying at least one of the aforementioned metadata).

[0078] In this context, and according to the exemplary embodiments further described below, the term "mesh" refers to a combination of at least one polygon describing the surface of a volumetric object. Each polygon is defined by its vertices in three-dimensional space and information about how those vertices are connected (referred to as connectivity information). Optionally, vertex attributes (such as color, normals, etc.) can be associated with mesh vertices. Attributes can also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh using a two-dimensional attribute map. This mapping can be described by a set of parametric coordinates (referred to as UV coordinates or texture coordinates) associated with the mesh vertices. The two-dimensional attribute map is used to store high-resolution attribute information, such as texture, normals, displacement, etc. According to the exemplary embodiments, such information can be used for various purposes, such as texture mapping and shading.

[0079] Nevertheless, dynamic mesh sequences may require a large amount of data because they can consist of a vast amount of information that changes over time. For example, unlike "static meshes" or "static mesh sequences" (where the information about the mesh may not change between frames), "dynamic meshes" or "dynamic mesh sequences" embody motion, meaning that the vertices in the individual vertices represented by the mesh change between frames. Therefore, efficient compression techniques are needed to store and transmit such content. MPEG has previously developed mesh compression standards IC, MESHGRID, and FAMC for handling dynamic meshes with constant connectivity and time-varying geometry and vertex properties. However, these standards do not account for time-varying attribute graphs and connectivity information. DCC (Digital Content Creation) tools typically generate such dynamic meshes. On the other hand, generating constant connectivity dynamic meshes using volumetric acquisition techniques is challenging, especially under real-time constraints. Existing standards do not support this type of content. Based on exemplary embodiments herein, aspects of a new mesh compression standard are described that directly handle dynamic meshes with time-varying connectivity information and optionally time-varying property graphs. This standard is designed to provide lossy and lossless compression techniques for a variety of applications, such as real-time communications, storage, free-viewpoint video, AR, and VR. Features such as random access and scalable / progressive coding are also considered.

[0080] Figure 7 An example framework 700 for dynamic mesh compression, such as one based on 2D atlas sampling, is shown. Each frame of the input mesh 701 can be preprocessed through a series of operations (e.g., tracking, remeshing, parameterization, voxelization). Note that these operations can be encoder-only, meaning they may not be part of the decoding process, and this possibility can be indicated by flags sent in the metadata, such as 0 for encoder-only and 1 for other cases. A mesh 702 with a 2D UV atlas is then obtained, where each vertex of the mesh has at least one associated UV coordinate on the 2D atlas. The mesh can then be converted into multiple mappings, including geometry and attribute maps, by sampling on the 2D atlas. These 2D mappings can then be encoded using a video / image codec (such as HEVC, VVC, AV1, AVS3, etc.). On the decoder 703 side, the mesh can be reconstructed from the decoded 2D maps. Any post-processing and filtering can also be applied to the reconstructed mesh 704. Note that additional metadata can be signaled to the decoder side for 3D mesh reconstruction purposes. Note that tile boundary information, including the uv and xyz coordinates of boundary vertices, can be predicted, quantized, and entropy encoded within the bitstream. The quantization step size can be configured on the encoder side to strike a tradeoff between quality and bit rate.

[0081] In some implementations, a 3D mesh can be divided into segments (or patches / tiles), and according to an exemplary implementation, at least one 3D mesh segment can be considered a "3D mesh". Each segment consists of a set of connected vertices, which are associated with their respective geometry, properties, and connectivity information. Figure 8 As shown in example 800 of the volume data, the UV parameterization process 802, which maps 3D mesh fragments to 2D tiles (such as reaching the aforementioned 2D UV atlas 702 box), maps at least one mesh fragment 801 onto a 2D tile 803 in the 2D UV atlas 804. Each vertex (v) in the mesh fragment will be assigned a UV parameter in the 2D UV atlas. n Specify a 2D UV coordinate system. Note that the vertices (v...) in a 2D tile... n This forms a connection component that serves as its 3D counterpart. The geometry, attributes, and connectivity information of each vertex can also be inherited from its 3D counterpart. For example, information indicating that vertex v4 is directly connected to vertices v0, v5, v1, and v3 can be provided, and similar information for each of the other vertices can be provided in the same way. Furthermore, according to an exemplary embodiment, such a 2D textured mesh will further indicate information on a block-by-block basis, such as color information, such as the facets of each triangle, for example, treating v2, v5, and v3 as a single "facet".

[0082] For example, regarding Figure 8 For the features of Example 800, see [link / reference]. Figure 9 Example 900 shows that a 3D mesh fragment 801 can also be mapped to multiple individual 2D tiles 901 and 902. In this case, a vertex in 3D can correspond to multiple vertices in the 2D UV atlas. Figure 9 As shown, in a 2D UV atlas, the same 3D mesh fragment is mapped to multiple 2D tiles, rather than as... Figure 8 A single tile in the graph. For example, 3D vertices v1 and v4 each have two 2D correspondences v1 and v4. 1' and v4, v 4' Therefore, a typical 2D UV atlas for a 3D mesh can include, for example: Figure 14 The diagram shows multiple tiles, each of which may contain multiple (usually more than or equal to 3) vertices associated with their respective 3D geometry, properties, and connectivity information.

[0083] Figure 9Example 903 is shown, illustrating a derived triangulation in a tile with boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7. When presenting such information, any derived triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). For example, for each vertex, the two nearest vertices are found. Alternatively, for all vertices, triangles are generated continuously until a minimum number of triangles is reached after a set number of attempts. As shown in Example 903, there are various regular-shaped repeating triangles that are usually closest to the boundary vertices, as well as various odd-shaped triangles with their own unique dimensions, which may or may not be shared with any other triangle. Connectivity information can also be reconstructed via explicit signaling. According to an exemplary embodiment, if the polygon cannot be recovered via implicit rules, the encoder can signal connectivity information in the bitstream.

[0084] Boundary vertices B0, B1, B2, B3, B4, B5, B6, and B7 are defined in 2D UV space. Boundary edges can be determined by checking whether they appear in only one triangle. According to an exemplary embodiment, the following information about the boundary vertices is important and should be signaled in the bitstream: geometric information (e.g., 3D XYZ coordinates even though currently in 2D UV parametric form), and 2D UV coordinates.

[0085] For example Figure 9 The illustrated case shows a boundary vertex in 3D corresponding to multiple vertices in a 2D UV atlas, where the mapping from 3D XYZ to 2D UV can be one-to-many. Therefore, a UV-to-XYZ (or UV2XYZ) index can be signaled to indicate the mapping function. UV2XYZ can be a 1D array of indices that map each 2D UV vertex to a 3D XYZ vertex.

[0086] According to an exemplary embodiment, in order to effectively represent mesh signals, a subset of mesh vertices and their connectivity information can first be encoded. In the original mesh, these vertices may not have any connections because they are obtained by secondary sampling from the original mesh. Different methods exist for signaling the connectivity information between vertices; therefore, such a subset is referred to as the base mesh or base vertices.

[0087] According to exemplary embodiments, numerous methods are implemented for dynamic mesh compression and are part of the aforementioned edge-based vertex prediction framework, wherein a base mesh is first encoded, and then additional vertices are predicted based on connectivity information from the edges of the base mesh. Note that they can be applied individually or in any form of combination.

[0088] For example, consider Figure 10Example flowchart 1001 for vertex grouping for prediction patterns. At S101, the vertices within the grid can be obtained, and at S102 they can be divided into different groups for prediction purposes, see, for example... Figure 9 In one example, segmentation is performed at S104 using patch / tile partitioning. In another example, segmentation is performed at S105 under each patch / tile. Whether S104 or S105 has been reached can be signaled at S103 via a flag or similar means. In the case of S105, several vertices of the same patch / tile form a prediction group and share the same prediction mode, while several other vertices of the same patch / tile may use a different prediction mode. In this document, a "prediction mode" can be considered a specific mode used by the decoder to predict video content including patches. Prediction modes can be categorized into intra-frame prediction modes and inter-frame prediction modes, and within each category, different specific modes can exist from which the decoder selects. According to an exemplary embodiment, each group, i.e., a "prediction group," can share the same specific mode (e.g., an angular mode at a specific angle) or the same categorical prediction mode (e.g., all intra-frame prediction modes, but predictions can be performed at different angles). This grouping at S106 can be allocated at different levels by determining the corresponding number of vertices involved in each group. For example, according to an exemplary embodiment, every 64, 32, or 16 vertices in the scan order within a patch / tile will be assigned the same prediction mode, and different assignments may be made for other vertices. For each group, the prediction mode may be an intra-frame prediction mode or an inter-frame prediction mode. This can be signaled or assigned. According to example flowchart 1000, if it is determined at S107 that a mesh frame or mesh slice is of intra-frame type (e.g., by checking whether a flag of the mesh frame or mesh slice indicates an intra-frame type), then all vertex groups within that mesh frame or mesh slice should use the intra-frame prediction mode; otherwise, intra-frame prediction (S108) or inter-frame prediction mode may be selected for all vertices in that group.

[0089] Furthermore, for a set of grid vertices using intra-frame prediction mode, its vertices can only be predicted by using previously encoded vertices within the same sub-partition of the current grid. According to an exemplary embodiment, sometimes the sub-partition can be the current grid itself. According to an exemplary embodiment, for a set of grid vertices using inter-frame prediction mode, its vertices can only be predicted by using previously encoded vertices from another grid frame. Each of the above information can be determined by flags, etc., and signaled. The prediction features can occur at S110, and the result of the prediction and signaling can occur at S111.

[0090] According to an exemplary embodiment, for each vertex in a set of vertices in example flowchart 1000 and flowchart 1100 described below, after prediction, the residual will be a 3D displacement vector indicating the offset of the current vertex from its predicted value. Further compression of the residuals of the set of vertices is required. In one example, the transformation at S111, along with signaling, can be applied to the residuals of the vertex set before entropy encoding. The encoding of a set of displacement vectors can be handled in the following ways: For example, in one method, a set of displacement vectors is appropriately signaled, indicating that some displacement vectors or their components have only zero values. In another embodiment, for each displacement vector, a flag is signaled indicating whether the vector has any non-zero components; if not, encoding of all components of that displacement vector can be skipped. Furthermore, in another embodiment, for each set of displacement vectors, a flag is signaled indicating whether the set has any non-zero vectors; if not, encoding of all displacement vectors in that set can be skipped. Additionally, in another embodiment, for each component of a set of displacement vectors, a flag is signaled indicating whether that component of the set has any non-zero vectors; if not, encoding of that component of all displacement vectors in that set can be skipped. Furthermore, in another embodiment, there may be a signal indicating that a set of displacement vectors or components of a set of displacement vectors need to be transformed. If no transformation is required, the transformation can be skipped, and quantization / entropy coding can be directly applied to that set or its components. Furthermore, in another embodiment, a flag indicating whether a transformation is required can be signaled for each set of displacement vectors. If not, the transformation coding for all displacement vectors in that set can be skipped. Furthermore, in another embodiment, for each component of a set of displacement vectors, a flag indicating whether that component in the set needs to be transformed can be signaled. If not, the transformation coding for that component of all displacement vectors in that set can be skipped. Although the embodiments described above relate to the processing of vertex prediction residuals, they can also be combined and implemented in parallel in different aspects.

[0091] Figure 11Example flowchart 1100 is shown, where, at S121, a grid frame is obtained, and the grid frame is encoded as a whole data unit, meaning that all vertices or attributes of the grid frame can be correlated among them. Alternatively, depending on the determination result at S122, the grid frame can be divided into smaller, independent sub-partitions at S123, conceptually similar to slices or tiles in a 2D video or image. At S124, a prediction type can be assigned to the encoded grid frame or the encoded grid sub-partition. Possible prediction types include intra-frame coding type and inter-frame coding type. For intra-frame coding type, at S125, only predictions from reconstructed portions of the same frame or slice are allowed. On the other hand, in addition to allowing intra-frame prediction of grid frames, predictions for previously encoded grid frames can also be allowed using inter-frame prediction type. Furthermore, inter-frame prediction type can be classified into more subtypes, such as P type or B type. In P type, only one predictor is available for prediction purposes, while in B type, two predictors from two previously encoded grid frames can be used to generate predicted values. A weighted average of two predictions can be an example. When a grid frame is encoded as a whole, the frame can be considered as a grid frame that is either intra-coded or inter-coded. In the case of an inter-grid frame, the P or B type can be further identified via signaling. Alternatively, if the grid frame is further segmented intra-frame, in which case the grid frame is encoded, the prediction type is specified for each sub-segment at S124. Each of the above pieces of information can be determined and signaled by flags, etc., and is similar to... Figure 10 The features of the prediction can be performed at S126 in S110 and S111, and the results of the prediction and signaling can be performed at S127.

[0092] Therefore, although dynamic mesh sequences may require large amounts of data due to the vast amounts of information they can contain that change over time, storing and transmitting such content still requires efficient compression techniques. The features described in this paper demonstrate the efficiency of this improvement by allowing the use of either previously decoded vertices within the same mesh frame (intra-frame prediction) or previously encoded mesh frames (inter-frame prediction).

[0093] Furthermore, an exemplary embodiment can generate the displacement vector of the third layer 1303 of the mesh based on at least one reconstructed vertex of a previous layer (such as the second layer 1302 and the first layer 1301). Assuming the index of the second layer 1302 is T, the predicted value T+1 of the vertices in the third layer 1303 is generated based on at least the reconstructed vertices of the current layer or the second layer 1302. An example of this layer-based prediction structure is... Figure 13Example 1300 is shown, illustrating reconstruction-based vertex prediction: progressive vertex prediction using edge-based interpolation, where the predictor is generated based on previously decoded vertices rather than on predictor vertices. A first layer 1301 can be a mesh defined by a first polygon 1340 having decoded vertices at its boundaries and interpolated vertices along lines between those decoded vertices as its vertices. As progressive encoding progresses from the first layer 1301 to the second layer 1302, another polygon 1341 can be formed by displacement vectors from the interpolated vertices of the first layer to additional vertices of the second layer 1302; thus, the total number of vertices in the second layer 1302 can be greater than the total number of vertices in the first layer 1301. Similarly, moving to the third layer 1303, the additional vertices of the second layer 1302, along with the decoded vertices from the first layer 1301, can participate in encoding in a similar manner to the decoded vertices that played when moving from the first layer 1301 to the second layer 1303; that is, multiple additional polygons can be formed. See also... Figure 14 Example 1400 of this progressive coding is shown in the figure (with Figure 13 (Different from the previous examples) Example 1400 shows that each of the additionally formed polygons can be completely within the polygon formed by the boundary of the first layer 1401 as one travels from the first layer 1401 to the second layer 1402 and then to the third layer 1403.

[0094] For such examples 1300 and / or 1400, see the exemplary embodiments. Figure 12 Example flowchart 1200 shows that, since the interpolated vertices on the current layer are predicted values, such values ​​need to be reconstructed before being used to generate predicted values ​​for vertices on the next layer. This is accomplished by encoding the base mesh in S131, performing vertex prediction in S132, and then adding the decoded displacement vectors of the current layer to the predictor of the vertices (such as layer 1302) in S133. The reconstructed vertices of this layer, along with all the decoded vertices of the previous layers (such as checking the additional vertex values ​​of such layers in S134), can then be used to generate the predictor vertices of the next layer 1303 in S135 and signal these predictor vertices. This process can also be summarized as follows: Let P[t](Vi) denote the predictor of vertex Vi on layer t; let R[t](Vi) denote the reconstructed vertex Vi on layer t; let D[t](Vi) denote the displacement vector of vertex Vi on layer t; let f(*) denote the predictor generator, which in particular can be the average of two existing vertices. Then, for each layer t, according to the exemplary embodiment, the following exists: Where Vj and Vk are the reconstructed vertices of the previous layer. - Equation (1) Then, for all vertices in a mesh frame, they are divided into layer 0 (base mesh), layer 1, layer 2, and so on. The reconstruction of vertices on a layer then depends on the reconstruction of vertices on previous layers. Above, each of P, R, and D represents a 3D vector in the context of the 3D mesh representation. D is the decoded displacement vector, which may or may not be quantized.

[0095] According to an exemplary embodiment, vertex prediction using reconstructed vertices can be applied only to certain layers, such as layers 0 and 1. For other layers, vertex prediction can still use adjacent predictor vertices without adding displacement vectors to them for reconstruction. Therefore, these other layers can be processed simultaneously without waiting for a previous layer to be reconstructed. According to an exemplary embodiment, for each layer, a signal can be used to indicate whether reconstruction-based vertex prediction or predictor-based vertex prediction is selected, or a signal can be used to indicate which layers (and their subsequent layers) do not use reconstruction-based vertex prediction.

[0096] For vertex predictors that generate displacement vectors from reconstructed vertices, quantization can be applied without further transformations (such as wavelet transforms). For vertex predictors that generate displacement vectors from vertices of other predictors, transformations may be required, and quantization can be applied to the transformation coefficients of those displacement vectors.

[0097] Therefore, since dynamic mesh sequences can contain a large amount of information that changes over time and may require a large amount of data, efficient compression techniques are needed to store and transmit such content. Within the framework of the aforementioned interpolation-based vertex prediction method, a crucial process is the compression of displacement vectors, which constitutes a major portion of the encoded bitstream and is the focus of this application. The features disclosed in this application alleviate this problem by providing such compression.

[0098] Furthermore, similar to the other examples described above, even using those embodiments, dynamic mesh sequences still require a large amount of data because they may consist of a large amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content. Within the framework of the 2D atlas-based sampling method described above, a significant advantage can be achieved by adding boundary vertices to the sampled vertices on the decoder side to infer connectivity information. This is the main part of the decoding process and the focus of the other examples described below.

[0099] According to an exemplary embodiment, connectivity information of the underlying mesh can be inferred (derived) from the decoding boundary vertices and sampling vertices of each tile on the encoder and decoder sides.

[0100] Similarly, any derived triangulation method can be applied to create connectivity between vertices (including boundary vertices and sampled vertices). According to an exemplary embodiment, the connectivity type can be signaled in a high-level syntax (such as a sequence header or slice header).

[0101] As mentioned above, connectivity information can also be reconstructed by explicitly signaling, for example, for irregularly shaped triangular meshes. That is, if it is determined that the polygon cannot be recovered by implicit rules, the encoder can signal the connectivity information in the bitstream. And according to an exemplary embodiment, the overhead of such explicit signaling can be reduced based on the boundaries of the polygon.

[0102] According to the embodiment, only the connectivity information between the boundary vertices and the sampling positions to be sent by signal is determined, while the connectivity information between the sampling positions themselves is derived.

[0103] Furthermore, in any embodiment, connectivity information can be signaled by prediction, such that the inferred connectivity (as a prediction) difference from one grid to another can be signaled only in the bitstream.

[0104] Note that, according to the exemplary embodiment, the orientation of the derived triangles (such as whether each triangle is derived clockwise or counterclockwise) can be signaled for all tiles in a high-level syntax (such as sequence headers, slice headers, etc.), or it can be pre-fixed (conventional) by the encoder and decoder. The orientation of the derived triangles can be indicated individually for each tile by a signal.

[0105] It should also be noted that any reconstructed mesh can have different connectivity than the original mesh. For example, the original mesh can be a triangular mesh, while the reconstructed mesh can be a polygonal mesh (e.g., a quadrilateral mesh).

[0106] According to an exemplary embodiment, connectivity information between any base vertices can be derived without signaling, using the same algorithm on both the encoder and decoder sides. Furthermore, according to an exemplary embodiment, interpolation of predicted vertices for additional mesh vertices can be based on the derived edges of the base mesh.

[0107] According to an exemplary embodiment, flags can be used to signal whether the connectivity information of the underlying vertices is to be signaled or derived, and such flags can be signaled at different levels of the bitstream (such as at the sequence level, frame level, etc.).

[0108] According to an exemplary embodiment, the same algorithm is first used on both the encoder and decoder sides to derive the edges between the base vertices. Then, the derived edges are compared with the original connectivity of the base mesh vertices, and the difference between the derived edges and the actual edges is represented by a signal. Therefore, after decoding the difference, the original connectivity of the base vertices can be recovered.

[0109] In one example, for a derived edge, if it is determined to be incorrect when compared to the original edge, such information can be signaled in the bitstream (by indicating the pair of vertices forming the edge); and for the original edge, if it is not derived, it can be signaled in the bitstream (by indicating the pair of vertices forming the edge). Furthermore, connectivity on boundary edges and vertex interpolation processing involving boundary edges can be performed separately from internal vertices and edges.

[0110] Therefore, the aforementioned technical problems can be advantageously improved by at least one of these technical solutions through the exemplary embodiments described herein. For example, since dynamic grid sequences can contain a large amount of information that changes over time and may therefore require a large amount of data, the exemplary embodiments described herein at least represent efficient compression techniques for storing and transmitting such content.

[0111] The above embodiments can also be applied to instance-based mesh coding, where an instance can be a mesh of objects or a portion of an object. For example, Figure 15 Illustrative example 1500 shows a mesh example 1501, in which various instances 1502 (a mesh representing a cup), 1503 (a mesh representing a spoon), and 1504 (a mesh representing a plate) exist, and these instances can be separated and encoded separately. Each of instances 1501, 1502, 1503, and 1504 is shown in its respective bounding box; however, it should be noted that instance 1501 can be considered as being defined by a "mesh-based bounding box," while each of instances 1502, 1503, and 1504 can be considered as being defined by its respective "instance-based bounding box."

[0112] The embodiments described herein apply to octahedral normal vectors (ONV). According to the embodiments described herein, such as... Figure 16 and 17 As shown in Examples 1600 and 1700, the octahedral normal vector (ONV) is obtained by first mapping a 2-norm unit sphere (ordinary sphere) to a 1-norm unit sphere (octahedron), and then projecting the 1-norm unit sphere and unfolding it into a unit square. See Example 1600.

[0113] According to an embodiment, the following provides an example of predicting octahedral normals before encoding and decoding. Example 1700 illustrates an example algorithm for predicting ONV. First, in S1701, halfMax=2 is calculated. qn-1 -1, where qn is the bit depth used to quantize the normal. Then, the prediction and encoding for each ONV n0 includes the following 6 steps, or, according to an embodiment, consists of the following 6 steps.

[0114] 1. At S1702, assume there exist N adjacent ONVs of type n0 (denoted as n1, ..., n). N If so, then we need to find a reference vector ref such that n1, ..., n are mapped according to this ref. N Mapped to the same 2D plane. According to one embodiment, n1, ..., n can be selected. N Any one of them can be used as the ref. Alternatively, according to one embodiment, n1, ..., n can be used. N The average value is used as the reference. Alternatively, according to one embodiment, n1, ..., n can be used. N The average of a subset (e.g., {n1, n2}) is used as the ref.

[0115] 2. In S1703, after calculating ref, n1, ..., n can be determined based on ref. N Mapped to the same 2D plane (denoted as m1, ..., m) N The 2D plane corresponds to the plane containing the two smaller components of the ref.

[0116] 3. At S1704, calculate m1, ..., m N A linear combination of (denoted as q). For example, given N=3, q=m1-m2,+m3.

[0117] 4. In S1705, the process in step 2 is reversed by mapping the linear combination q back to 3D space (denoted as p) according to the ref.

[0118] 5. In S1706, similar to step 2, p is used as the ref and n0 is mapped to the 2D plane (denoted as t) according to p.

[0119] 6. In S1707, p is used as the prediction of t, and the 2D residual is calculated and encoded: r = tp.

[0120] Therefore, embodiments of this paper provide methods such as predicting octahedral normals prior to encoding and decoding.

[0121] The document also provides quantization and dequantization of ONV according to embodiments thereof. For example, ONV may need to be quantized before encoding and dequantized after decoding.

[0122] According to the embodiments in this paper, the following algorithm is provided for quantizing ONV. First, calculate halfMax=2. qn-1 -1, where qn is the bit depth used to quantize the normal. Then, each original normal vector n0 can be quantized to ONV as follows:

[0123]

[0124] Where n2 is the quantized ONV.

[0125] For the inverse quantization of ONV n2, the embodiments in this paper can be simply derived from... n Subtract from 2 halfMax :

[0126] Where n deq It is the ONV of inverse quantization.

[0127] Therefore, quantization and dequantization of ONV according to embodiments herein are also provided.

[0128] According to one embodiment, a modification to the coding octahedral normal is provided. A first variation simplifies octahedral quantization and dequantization of the normal without altering coding performance, while a second variation improves the prediction of the octahedral normal and demonstrates coding gain.

[0129] For example, in the context of simplified octahedral quantization and inverse quantization according to embodiments of this document, the following encoded fragment is provided as the main part of octahedral quantization:

[0130] Where, value is the original normal vector; maxVec[0] is the maximum length of all normals in the mesh, maxDelta=2*maxVec[0].

[0131] By performing the following algebraic operations:

[0132] The embodiments in this article can simplify the above code as follows:

[0133] In this way, octahedral quantization can be performed without calculating maxVec[0], maxDelta, and scale. Furthermore, according to the embodiment, the inverse quantization of the octahedral normal is as follows:

[0134] Furthermore, note that `iscale*halfMax / l2` is a constant for each dequantized octahedral normal value. Therefore, the second `for` loop scales each component of the value by the same amount, which does not change the direction of the normal vector. Consequently, the embodiment advantageously eliminates scaling, and the above code is simplified to:

[0135] An improved prediction of octahedral normals according to embodiments herein is also provided. To predict octahedral normals, their neighboring normals are first mapped to a 2D plane, where the prediction is performed in the 2D plane. A reference vector ref is used to determine the 2D plane. The reference vector ref is calculated based on ap0[0], which is the first neighboring normal in the first neighbor set, as follows:

[0136] Instead of using the ref calculated based on ap0[0], the embodiments described herein can calculate the ref based on the first two adjacent normals of all available adjacent sets. For example, that is, for the “SIX_NEIGHBORS” prediction configuration, the ref is calculated as follows:

[0137] For the “THREE_NEIGHBORS” and “TWO_NEIGHBORS” prediction configurations, the ref is calculated as follows:

[0138] The variable, function, and file named “CornerAttribute*Predictor” will also be renamed to the corresponding “CornerVertexAttribute*Predictor”, because these predictors are used not only to predict corner attribute values ​​but also vertex attribute values.

[0139] To evaluate the quantization and dequantization of the simplified octahedral normals, a comparison test was run using the modified default lossless configuration file “T0DefaultL3.bin.cfg”, generating five configurations (“nbits”=6, 7, 8, 9, 10) corresponding to the five normal quantization levels, while keeping all other parameters in “T0DefaultL3.bin.cfg” unchanged.

[0140] By adopting the PMC (Programme for the Visual Media Consortium) work group's VVM-2025-022i (Programme for the Visual Media Consortium) The distortion metric described in the paper "Ectophedral Unit Normal Quantization Integration in v7.0" uses TM v8.1 as a benchmark, calculating the "BD attribute rate" and "BD total rate" for each mesh with normals of categories C0, C1, and C2, and regarding... Figure 18 , Figure 19 , Figure 20 , Figure 21 , Figure 22 , Figure 23 Examples 1800, 1900, 2000, 2100, 2200, and 2300 illustrate the results. Examples 2000 and 2100 show the overall results of the embodiments described herein under the lossless default configuration, and examples 2200 and 2300 show the overall results of the embodiments described herein under the per-mesh lossless configuration.

[0141] The above-described techniques can be implemented as computer software using computer-readable instructions and physically stored in at least one computer-readable medium or implemented by at least one specially configured hardware processor. For example, Figure 24 A computer system 2400 suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0142] Computer software can be encoded using any suitable machine code or computer language, which can be assembled, compiled, linked, or similarly processed to create code, which includes instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), or through interpretation, microcode execution, or other means.

[0143] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0144] Figure 24 The components shown for computer system 2400 are exemplary in nature and are not intended to imply any limitation on the scope or functionality of computer software implementing the embodiments disclosed herein. The configuration of the components should also not be construed as having any dependency or requirement related to any one or a combination of the components shown in the exemplary embodiments of computer system 2400.

[0145] Computer system 2400 may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from at least one human user via, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), and olfactory input (not depicted). The human-machine interface device can also be used to acquire certain media that are not necessarily directly related to conscious human input, such as audio (such as voice, music, ambient sound), images (such as scanned images, photographic images obtained from still image cameras), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0146] The input human-machine interface device may include at least one of the following (only one of each is depicted): keyboard 2401, mouse 2402, touchpad 2403, touch screen 2410, joystick 2405, microphone 2406, scanner 2408, and camera 2407.

[0147] Computer system 2400 may also include certain human-machine interface (HMI) output devices. Such HMI output devices can stimulate the senses of at least one human user through, for example, tactile output, sound, light, and smell / taste. These HMI output devices may include tactile output devices (e.g., tactile feedback from touchscreen 2410 or joystick 2405, but tactile feedback devices not used as input devices may also exist), audio output devices (such as speakers 2409, headphones (not depicted)), visual output devices (such as screen 2410, including CRT screens, LCD screens, plasma screens, OLED screens, each screen may or may not have touchscreen input capability, each screen may or may not have tactile feedback capability—some of which are capable of outputting two-dimensional or more than three-dimensional visual output via devices such as stereoscopic output; virtual reality glasses (not shown), holographic displays and smoke canisters (not shown), and printers (not shown).

[0148] Computer system 2400 may also include human-accessible storage devices and their associated media, such as optical media, including CD / DVD ROM / RW 2420 or similar media with CD / DVD 2411, thumb drives 2422, removable hard disk drives or solid-state drives 2423, conventional magnetic media such as magnetic tapes and floppy disks (not shown), devices based on dedicated ROM / ASIC / PLD, such as security dongles (not shown), etc.

[0149] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter currently disclosed in this application does not cover transmission media, carrier waves, or other transient signals.

[0150] Computer system 2400 may also include an interface 2499 connected to at least one communication network 2498. Network 2498 may be, for example, wireless, wired, or optical. Network 2498 may also be local, wide area, metropolitan area, vehicular, industrial, real-time, latency-tolerant, etc. Examples of network 2498 include local area networks (such as Ethernet), wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial networks including CANbus, etc. Some networks 2498 typically require external network interface adapters (e.g., USB ports of computer system 2400) to connect to certain general-purpose data ports or peripheral buses (2450 and 2451); other network interface adapters are typically integrated into the core of computer system 2400 by connecting to system buses as described below (e.g., Ethernet interfaces connected to PC computer systems, or cellular network interfaces connected to smartphone computer systems). By using any of these networks 2498, computer system 2400 can communicate with other entities. This communication can be unidirectional, receive-only (e.g., broadcasting TV), transmit-only (e.g., to CANbus of certain CANbus devices), or bidirectional, such as connecting to other computer systems using local area networks (LANs) or wide area networks (WANs). Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0151] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core 2440 of the computer system 2400.

[0152] Core 2440 may include at least one central processing unit (CPU) 2441, graphics processing unit (GPU) 2442, graphics adapter 2417, dedicated programmable processing unit in the form of field-programmable gate array (FPGA) 2443, hardware accelerators for certain tasks 2444, etc. These devices, as well as read-only memory (ROM) 2445, random access memory 2446, and internal mass storage 2447 such as internal non-user-accessible hard disk drives (SD drives) or SSDs, can be connected via system bus 2448. In some computer systems, system bus 2448 may be accessible in the form of at least one physical connector to allow for the expansion of additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus 2448 or attached to the core's system bus 2448 via peripheral bus 2449. Peripheral bus architectures include PCI, USB, etc.

[0153] The CPU 2441, GPU 2442, FPGA 2443, and accelerator 2444 can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM 2445 or RAM 2446. Transient data can also be stored in RAM 2446, while permanent data can be stored, for example, in internal mass storage 2447. Fast storage and retrieval of any memory device can be achieved by using a cache memory, which can be closely associated with at least one CPU 2441, GPU 2442, mass storage 2447, ROM 2445, RAM 2446, etc.

[0154] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be those specifically designed and constructed for the purposes disclosed in this application, or they may be of a type well known and available to those skilled in the art of computer software.

[0155] By way of example and not limitation, a computer system having architecture 2400 (especially core 2440) can provide functionality by having a processor (including CPU, GPU, FPGA, accelerator, etc.) execute software contained in one or more tangible computer-readable media. Such a computer-readable medium may be a medium associated with certain storage of the user-accessible mass storage described above and the non-transitory nature of core 2440, such as core-internal mass storage 2447 or ROM 2445. Software implementing the various embodiments disclosed herein may be stored in such a device and executed by core 2440. Depending on specific needs, the computer-readable medium may include at least one memory device or chip. The software may cause core 2440, and in particular its processor (including CPU, GPU, FPGA, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM 2446 and modifying such data structures according to software-defined processes. Additionally or alternatively, the computer system may provide functionality through hard-wired logic embodied in circuitry or other means (e.g., accelerator 2444), which may replace or operate with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry storing the software (such as integrated circuits (ICs)), circuitry embodying the execution logic, or both. This application disclosure covers any suitable combination of hardware and software.

[0156] While this application has described several exemplary embodiments, there are variations, arrangements, and various alternative equivalents that fall within the scope of this application. Therefore, it should be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of this application and are therefore within its spirit and scope.

Claims

1. A method for video decoding, the method being performed by at least one processor, the method comprising: The method includes: A grid is obtained from the video stream, the grid representing encoded volume data of at least one three-dimensional (3D) visual content; The adjacent normals of the octahedral normal vector (ONV) of the mesh are mapped onto a two-dimensional plane based on a reference vector, which is calculated based on a subset of the adjacent normals; and the ONV is predicted. Decode the encoded data based on the predicted ONV.

2. The method according to claim 1, characterized in that, The ONV is obtained by mapping a 2-norm unit sphere to a 1-norm unit sphere in an octahedron, and then projecting and unfolding the 1-norm unit sphere into a unit square.

3. The method according to claim 1, characterized in that, The prediction of the ONV depends on the bit depth of the normal being quantized.

4. The method according to claim 1, characterized in that, The reference vector is calculated as any one of the adjacent normals.

5. The method according to claim 1, characterized in that, The reference vector is calculated as the average of the adjacent normals.

6. The method according to claim 1, characterized in that, The reference vector is calculated as the average of a subset of the adjacent normals.

7. The method according to claim 1, characterized in that, Predicting the ONV includes: calculating the 2D residual.

8. A method for video encoding, the method being performed by at least one processor, the method comprising: The method includes: Obtain a mesh, which represents volume data of at least one three-dimensional (3D) visual content; The adjacent normals of the octahedral normal vector (ONV) of the mesh are mapped onto a two-dimensional plane based on a reference vector, which is calculated based on a subset of the adjacent normals; and the ONV is predicted. The volume data is encoded based on the predicted ONV.

9. The method according to claim 8, characterized in that, The ONV is obtained by mapping a 2-norm unit sphere to a 1-norm unit sphere in an octahedron, and then projecting and unfolding the 1-norm unit sphere into a unit square.

10. The method according to claim 8, characterized in that, The prediction of the ONV depends on the bit depth of the normal being quantized.

11. The method according to claim 8, characterized in that, The reference vector is calculated as any one of the adjacent normals.

12. The method according to claim 8, characterized in that, The reference vector is calculated as the average of the adjacent normals.

13. The method according to claim 8, characterized in that, The reference vector is calculated as the average of a subset of the adjacent normals.

14. The method according to claim 8, characterized in that, Predicting the ONV includes: calculating the 2D residual.

15. An apparatus for video decoding, the apparatus comprising: The apparatus is configured to perform the method according to any one of claims 1 to 7.

16. An apparatus for video coding, the apparatus comprising: The apparatus is configured to perform the method of any one of claims 8 to 14.

17. A method of storing a video bitstream, the method comprising: The video stream is generated by performing the method according to any one of claims 8 to 14; and the video stream is stored.

18. A method of transmitting a video bitstream, the method comprising: The method of any one of claims 8 to 14 is used to generate the video stream; and the video stream is transmitted.

19. A computer-readable storage medium, characterized in that, The system stores a video stream, which is generated by the video encoding method according to any one of claims 8 to 14.

Images