Method, electronic device, and storage medium for mesh decompression and compression

Abstract

Aspects of the disclosure provide methods, electronic devices, and storage media for mesh decompression and compression. The decompression method includes decoding, from a bitstream carrying a first three-dimensional (3D) mesh frame, 3D coordinates of vertices and connectivity information of the vertices in the first 3D mesh frame. The first 3D mesh frame represents a surface of an object with polygons. The processing circuitry derives texture coordinates associated with the vertices and decodes a texture map of the first 3D mesh frame from the bitstream. The texture map includes a first one or more 2D maps of 2D vertices with the texture coordinates. The processing circuitry reconstructs the first 3D mesh frame based on the 3D coordinates of the vertices, the connectivity information of the vertices, the texture map, and the texture coordinates.

Description

[0001] References merged

[0002] This application claims priority to U.S. Patent Application No. 17 / 969,570, filed October 19, 2022, entitled "Mesh Compression with Deduced Texture Coordinates," which claims priority to U.S. Provisional Patent Application No. 63 / 298,106, filed January 10, 2022, entitled "Mesh Compression with Deduced Texture Coordinates." The disclosures of both prior applications are incorporated herein by reference in their entirety. Technical Field

[0003] This application generally relates to mesh encoding and decoding, and in particular to methods, electronic devices, and storage media for mesh decompression and compression. Background Technology

[0004] The background description provided herein is intended to generally present the context of this application. To the extent described in this background section, the work of the currently named inventors, and aspects of the description that may not have qualified as prior art at the time of filing of this application, are neither expressly nor implicitly considered as prior art to this application.

[0005] Various technologies have been developed to capture and represent the real world in 3D space, such as real-world objects and environments. 3D representations of the real world enable more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used for 3D representations of the real world. Summary of the Invention

[0006] This disclosure provides methods and apparatus for mesh encoding and / or decoding. In some examples, the apparatus for mesh encoding and decoding includes a processing circuit system. The processing circuit system decodes 3D coordinates and vertex connectivity information of vertices in a first 3D mesh frame from a bitstream carrying a first three-dimensional (3D) mesh frame. The first 3D mesh frame represents the surface of an object having polygons. The processing circuit system derives texture coordinates associated with the vertices and decodes a texture map of the first 3D mesh frame from the bitstream. The texture map includes a first or more 2D graphs of 2D vertices with texture coordinates. The processing circuit system reconstructs the first 3D mesh frame based on the 3D coordinates of the vertices, the vertex connectivity information, the texture map, and the texture coordinates.

[0007] In some examples, the processing circuitry uses a lossless codec to decode the 3D coordinates and connectivity information of the vertices. In other examples, the processing circuitry uses a lossy codec to decode the 3D coordinates and connectivity information of the vertices.

[0008] In some examples, in order to derive the texture coordinates associated with a vertex, the processing circuitry performs parameterization based on the vertex's 3D coordinates and connectivity information to determine the texture coordinates associated with the vertex.

[0009] In some examples, to perform parameterization, the processing circuitry partitions the polygons of a first 3D mesh frame into one or more first 2D graphs and packs these first or more 2D graphs into a 2D map. The processing circuitry performs temporal alignment, aligning the first or more 2D graphs with the second or more 2D graphs associated with the second 3D mesh frame. The first and second 3D mesh frames are frames in a 3D mesh sequence. The processing circuitry determines texture coordinates from the temporally aligned first or more 2D graphs.

[0010] In some examples, in order to partition a polygon, the processing circuitry partitions the polygon based on the normal values ​​associated with it.

[0011] In some examples, the processing circuitry performs time alignment based on at least one of a scaling invariant, rotation invariant, translation invariant, and / or affine transformation invariant invariant.

[0012] In some examples, the processing circuitry performs time alignment based on at least one of the following: the center of the chart, the average depth of the chart, the weighted average texture value of the chart, and / or the weighted average attribute value of the chart, calculated based on the 3D coordinates associated with the chart.

[0013] In some examples, the processing circuitry decodes a flag that enables texture coordinate derivation. This flag is at least one of a sequence-level flag, a set of frame-level flags, and a frame-level flag.

[0014] In some examples, in order to derive the texture coordinates associated with a vertex, the processing circuitry decodes a flag indicating the inheritance of the texture coordinates, the index of the 3D mesh frame selected from the list of decoded 3D mesh frames, and inherits the texture coordinates from the selected 3D mesh frame.

[0015] In some examples, the processing circuitry decodes a flag associated with a portion of the texture map, indicating whether the UV coordinates associated with that portion of the texture map are inherited from an already decoded mesh frame or derived through parameterization.

[0016] In some examples, the processing circuit system decodes the indexes of a set of key vertices in the vertices, and parameterization begins from that set of key vertices.

[0017] In some examples, the processing circuit system decodes the index of the parameterized method selected from the list of parameterized method candidates.

[0018] This disclosure also provides non-volatile computer-readable media storing instructions that, when executed by a computer, cause the computer to perform any or a combination of methods for grid encoding / decoding. Attached Figure Description

[0019] Other features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0020] Figure 1 Block diagrams of communication systems in some examples are shown.

[0021] Figure 2 Block diagrams of streaming systems in some examples are shown.

[0022] Figure 3 A block diagram of an encoder for encoding point cloud frames is shown in some examples.

[0023] Figure 4 A block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame is shown in some examples.

[0024] Figure 5 A block diagram of a video decoder is shown in some examples.

[0025] Figure 6 A block diagram of a video encoder is shown in some examples.

[0026] Figure 7 A block diagram of an encoder for encoding point cloud frames is shown in some examples.

[0027] Figure 8 A block diagram of a decoder used in some examples to decode a compressed bitstream carrying point cloud frames is shown.

[0028] Figure 9 The illustration shows a schematic diagram of how a grid is mapped to an atlas in some examples.

[0029] Figure 10 A schematic diagram of a mesh compression framework according to some embodiments of the present disclosure is shown.

[0030] Figure 11 A schematic diagram of a mesh compression framework according to some embodiments of the present disclosure is shown.

[0031] Figure 12 The syntax table is shown in some examples.

[0032] Figure 13 A flowchart is shown as an example of an overview process in some examples.

[0033] Figure 14 A flowchart is shown as an example of an overview process in some examples.

[0034] Figure 15 These are schematic diagrams of computer systems in some examples. Detailed Implementation

[0035] This disclosure provides techniques in the field of three-dimensional (3D) media processing.

[0036] The development of 3D media processing technologies, such as advancements in 3D capture, 3D modeling, and 3D rendering, has facilitated the ubiquitous availability of 3D media content across various platforms and devices. In one example, capturing a baby's first steps on one continent allows grandparents on another continent to view (and perhaps interact with) and immerse themselves in the baby's experience. According to one aspect of this disclosure, to enhance the immersive experience, 3D models are becoming increasingly complex, and the creation and consumption of 3D models consume significant data resources, such as data storage and data transmission resources.

[0037] According to some aspects of this disclosure, point clouds and meshes can be used as 3D models to represent immersive content.

[0038] A point cloud typically refers to a set of points in 3D space, each point having associated attributes such as color, material properties, texture information, intensity, reflectivity, motion-related attributes, shape attributes, and various other attributes. Point clouds can be used to reconstruct objects or scenes as combinations of such points.

[0039] An object's mesh (also called a mesh model) can include polygons that describe the object's surfaces. Each polygon can be defined by its vertices in 3D space and information about how those vertices connect to form the polygon. The information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include properties associated with the vertices, such as color and normals.

[0040] According to some aspects of this disclosure, some encoding and decoding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be re-meshed to generate a new mesh, and the connectivity information of the new mesh can be inferred. The vertices of the new mesh and the properties associated with the vertices of the new mesh can be treated as points in the point cloud, and they can be compressed using a PCC codec.

[0041] Point clouds can be used to reconstruct objects or scenes as a combination of points. These points can be acquired using multiple cameras, depth sensors, or LiDAR in various settings and can consist of thousands to billions of points to realistically represent the reconstructed scene or object. A patch can typically refer to a continuous subset of surfaces described by a point cloud. In the example, a patch consists of points whose surface normal vectors deviate from each other by less than a threshold amount.

[0042] PCC can be implemented using various schemes, such as a geometry-based scheme called G-PCC and a video codec-based scheme called V-PCC. According to some aspects of this disclosure, G-PCC directly encodes 3D geometry and is a completely geometry-based method with little sharing with video codecs, while V-PCC is largely based on video codecs. For example, V-PCC can map points in a 3D cloud to pixels in a 2D raster (image). The V-PCC scheme can utilize general-purpose video codecs for point cloud compression. The PCC codec (encoder / decoder) in this disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0043] According to one aspect of this disclosure, the V-PCC scheme can compress the geometry, occupancy, and texture of a point cloud into three separate video sequences using existing video codecs. The additional metadata required to interpret these three video sequences is compressed separately. A small portion of the entire bitstream is metadata, which, in this example, can be efficiently encoded / decoded using software. This substantial amount of information is handled by the video codec.

[0044] Figure 1 A block diagram of a communication system (100) in some examples is shown. The communication system (100) includes a plurality of terminal devices that can communicate with each other via, for example, a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). Figure 1 In the example, the first pair of terminal devices (110) and (120) can perform unidirectional transmission of point cloud data. For example, terminal device (110) can compress a point cloud (e.g., points representing structures) acquired by a sensor (105) connected to terminal device (110). The compressed point cloud can be transmitted, for example, as a bitstream via a network (150) to another terminal device (120). Terminal device (120) can receive the compressed point cloud from the network (150), decompress the bitstream to reconstruct the point cloud, and display the reconstructed point cloud appropriately. Unidirectional data transmission is common in media service applications, etc.

[0045] exist Figure 1In the examples, terminal device (110) and terminal device (120) may be illustrated as a server and a personal computer, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure are applicable to laptop computers, tablet computers, smartphones, gaming terminals, media players, and / or dedicated three-dimensional (3D) equipment. Network (150) refers to any number of networks transmitting compressed point clouds between terminal device (110) and terminal device (120). Network (150) may include, for example, wired (wired line) and / or wireless communication networks. Network (150) may exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0046] Figure 2 A block diagram of a streaming system (200) is shown in some examples. The streaming system (200) is an application using point clouds. The disclosed subject matter can be equally applied to other applications that support point clouds, such as 3D telepresence applications, virtual reality applications, etc.

[0047] The streaming system (200) may include an acquisition subsystem (213). The acquisition subsystem (213) may include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates an uncompressed point cloud in software, and the like that that generates, for example, an uncompressed point cloud (202). In the example, the point cloud (202) includes points acquired by a 3D camera. The point cloud (202) is depicted as thick lines to emphasize its higher data volume compared to a compressed point cloud (204) (the bitstream of the compressed point cloud). The compressed point cloud (204) may be generated by an electronics device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to enable or implement various aspects of the disclosed subject matter, which are described in more detail below. The compressed point cloud (204) (or the bitstream of the compressed point cloud (204)) is depicted as a thin line to emphasize its lower data volume compared to the bitstream of the point cloud (202). The compressed point cloud can be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as Figure 2 The client subsystems (206) and (208) can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) may include, for example, a decoder (210) in an electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and creates an output stream of the reconstructed point cloud (211) that can be rendered on the rendering device (212).

[0048] Please note that electronic devices (220) and (230) may include other components (not shown). For example, electronic device (220) may include a decoder (not shown), and electronic device (230) may also include an encoder (not shown).

[0049] In some streaming systems, compressed point clouds (204), (207), and (209) (e.g., the bitstream of a compressed point cloud) can be compressed according to specific standards. In some examples, video coding standards can be used for point cloud compression. Examples of these standards include High Efficiency Video Coding (HEVC), Universal Video Coding (VVC), etc.

[0050] Figure 3 A block diagram of a V-PCC encoder (300) for encoding point cloud frames according to some embodiments is shown. In some embodiments, the V-PCC encoder (300) can be used in a communication system (100) and a streaming system (200). For example, an encoder (203) can be configured to operate in a manner similar to the V-PCC encoder (300).

[0051] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream corresponding to the compressed point cloud frames. In some embodiments, the V-PCC encoder (300) can receive point cloud frames from a point cloud source such as a point cloud source (201).

[0052] exist Figure 3 In the example, the V-PCC encoder (300) includes a patch generation module (306); a patch packing module (308); a geometric image generation module (310); a texture image generation module (312); a patch information module (304); an occupancy map module (314); a smoothing module (336); image filling modules (316) and (318); a group expansion module (320); video compression modules (322), (323) and (332); an auxiliary patch information compression module (338); an entropy compression module (334); and a multiplexer (324).

[0053] According to one aspect of this disclosure, a V-PCC encoder (300) converts a 3D point cloud frame into an image-based representation along with some metadata (e.g., occupancy map and patch information) used to convert the compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) can convert a 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometry image is a 2D image in which pixel-filled geometric values ​​are associated with points projected onto the pixels; pixels filled with geometric values ​​can be called geometry samples. A texture image is a 2D image in which pixel-filled texture values ​​are associated with points projected onto the pixels; pixels filled with texture values ​​can be called texture samples. An occupancy map is a 2D image in which pixel-filled values ​​indicate whether patches are occupied or not.

[0054] The patch generation module (306) segments the point cloud into a set of patches that may overlap or not overlap (e.g., a patch is defined as a continuous subset of surfaces described by the point cloud), such that each patch can be described by a depth field relative to a plane in 2D space. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction errors.

[0055] In some examples, the patch information module (304) can collect patch information indicating the size and shape of the patches. In some examples, the patch information can be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0056] In some examples, the patch packing module (308) is configured to map extracted patches onto a 2D grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a unique patch. Effective patch packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.

[0057] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) store the geometry and texture of the point cloud as images using the 3D-to-2D mapping calculated in the above-described packing process. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images called layers. In the example, the geometry image is represented by a monochrome frame of WxH in YUV420-8-bit format. To generate the texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to calculate the color associated with the resampled points.

[0058] The occupancy map module (314) can generate an occupancy map that describes the fill information for each cell. For example, the occupancy image includes a binary map that indicates whether each cell of the raster belongs to empty space or to a point cloud. In one example, the occupancy map uses binary information to describe whether each pixel is filled. In another example, the occupancy map uses binary information to describe whether each pixel block is filled.

[0059] The occupancy map generated by the occupancy map module (314) can be compressed using either lossless or lossy codecs. When using lossless codecs, the entropy compression module (334) is used to compress the occupancy map. When using lossy codecs, the video compression module (332) is used to compress the occupancy map.

[0060] Note that the patch packing module (308) may leave some blank space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the blank space (called padding) to generate an image frame suitable for 2D video and image codecs. Image padding, also known as background padding, fills unused space with redundant information. In some examples, good background padding increases the bit rate minimally while avoiding the introduction of noticeable coding distortion around patch boundaries.

[0061] The video compression modules (322), (323), and (332) can encode 2D images (such as filled geometric images, filled texture images, and occupancy maps) based on a suitable video coding standard (such as HEVC, VVC, etc.). In one example, the video compression modules (322), (323), and (332) are separate components that operate independently. Note that in another example, the video compression modules (322), (323), and (332) can be implemented as a single component.

[0062] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry. The smoothed image can be provided to the texture image generation module (312). The texture image generation module (312) can then adjust the generation of the texture image based on the reconstructed geometry. For example, when the patch shape (such as geometry) has slight distortion during encoding and decoding, this distortion can be taken into account when generating the texture image to correct the distortion in the patch shape.

[0063] In some embodiments, group expansion (320) is configured to fill pixels around object boundaries with redundant low-frequency content in order to improve encoding / decoding gain and visual quality of the reconstructed point cloud.

[0064] The multiplexer (324) can multiplex the compressed geometric image, the compressed texture image, the compressed occupancy map, and the compressed auxiliary patch information into the compressed bitstream.

[0065] Figure 4 A block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame is shown in some examples. In some examples, the V-PCC decoder (400) can be used in a communication system (100) and a streaming system (200). For example, a decoder (210) can be configured to operate in a similar manner to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0066] exist Figure 4 In the example, the V-PCC decoder (400) includes a demultiplexer (432), a video decompression module (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0067] The demultiplexer (432) can receive the compressed bitstream and separate it into a compressed texture image, a compressed geometric image, a compressed occupancy map, and compressed auxiliary patch information.

[0068] Video decompression modules (434) and (436) can decode compressed images according to appropriate standards (e.g., HEVC, VVC, etc.) and output decompressed images. For example, video decompression module (434) decodes compressed texture images and outputs decompressed texture images; video decompression module (436) decodes compressed geometric images and outputs decompressed geometric images.

[0069] The occupancy map decompression module (438) can decode the compressed occupancy map according to a suitable standard (e.g., HEVC, VVC, etc.) and output the decompressed occupancy map.

[0070] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0071] The geometry reconstruction module (444) can receive the decompressed geometry image and generate the reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.

[0072] The smoothing module (446) can smooth out inconsistencies at the edges of the patch. The smoothing process aims to mitigate discontinuities that may occur at the patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located at the patch boundaries to mitigate distortion that may be caused by compression / decompression.

[0073] The texture reconstruction module (448) can determine the texture information of points in the point cloud based on the decompressed texture image and the smoothed geometry.

[0074] The color smoothing module (452) can smooth out inconsistencies in shading. Non-adjacent faces in 3D space are often packed together as adjacent to each other in 2D video. In some examples, pixel values ​​from non-adjacent faces can be mixed by a block-based video codec. The purpose of color smoothing is to reduce visible artifacts that appear at face boundaries.

[0075] Figure 5 Block diagrams of some examples of video decoders (510) are shown. The video decoder (510) can be used in the V-PCC decoder (400). For example, video decompression modules (434) and (436) and occupancy map decompression module (438) can be similarly configured as video decoders (510).

[0076] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from compressed images such as encoded video sequences. The categories of these symbols include information for managing the operation of the video decoder (510). The parser (520) may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters from the encoded video sequence for at least one subgroup of pixels in the subgroups used in the video decoder, based on at least one parameter corresponding to a group. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) can also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0077] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory to create symbols (521).

[0078] Depending on the type of encoded video picture or portion of encoded video picture (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors, the reconstruction of the symbol (521) may involve multiple different units. Which units are involved and how they are involved can be controlled by the subgroup control information parsed from the encoded video sequence by the parser (520). For the sake of brevity, the flow of such subgroup control information between the parser (520) and the various units described below is not described.

[0079] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In practical embodiments operating under commercial constraints, many of these units interact closely with each other and may at least be partially integrated with each other. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the functional units described below.

[0080] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantization transform coefficients as symbols (521) and control information from the parser (520), including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

[0081] In some cases, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images but can use predictive information from previously reconstructed portions of the current image. Such predictive information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses surrounding reconstructed information extracted from the current picture buffer (558) to generate blocks of the same size and shape as the block being reconstructed. For example, the current picture buffer (558) buffers portions that have reconstructed the current image and / or completely reconstructed the current image. In some cases, the aggregator (555) adds the predictive information generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) based on each sample.

[0082] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coding and latent motion compensation blocks. In this case, the motion compensation prediction unit (553) may access the reference image memory (557) to extract samples for prediction. After motion compensation of the extracted samples according to the symbols (521) belonging to the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (referred to in this case as residual samples or residual signals) to generate output sample information. The motion compensation prediction unit (553) may obtain the predicted samples from the address in the reference image memory (557) under motion vector control, and the motion vector is available to the motion compensation prediction unit (553) in the form of the symbols (521), which may include, for example, X, Y and reference image components. Motion compensation may also include interpolation of sample values ​​extracted from the reference image memory (557) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0083] The output samples of the aggregator (555) can be employed by various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream), and these parameters can be used as symbols (521) from the parser (520) in the loop filter unit (556), but may also be responsive to metadata obtained during decoding of a previous (in decoding order) portion of the encoded picture or encoded video sequence, and to previously reconstructed and loop-filtered sample values.

[0084] The output of the loop filter unit (556) can be a sample stream, which can be output to a display device and stored in a reference image memory (557) for subsequent inter-frame image prediction.

[0085] Once fully reconstructed, some of the encoded images can be used as reference images for future predictions. For example, once the encoded images corresponding to the current image have been fully reconstructed and the encoded images (by, for example, the parser (520)) are identified as reference images, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0086] The video decoder (510) can perform decoding operations according to a predetermined video compression technique as specified in standards such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the configuration file recorded in the video compression technique or standard. Specifically, the configuration file may select certain tools from all available tools in the video compression technique or standard as the only tools available under said configuration file. For compliance, the complexity of the encoded video sequence is also required to be within the range defined by the hierarchy of the video compression technique or standard. In some cases, the hierarchy limits the maximum image size, maximum frame rate, maximum reconstruction sampling rate (measured in megasamples per second, for example), maximum reference image size, etc. In some cases, the limitations set by the hierarchy can be further limited by the Hypothetical Reference Decoder (HRD) specification and the metadata managed by the HRD buffer, which is represented by signals in the encoded video sequence.

[0087] Figure 6 A block diagram of a video encoder (603) according to an embodiment disclosed in this application is shown. The video encoder (603) can be used as a V-PCC encoder (300) for compressing point clouds. In one example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).

[0088] The video encoder (603) can receive 2D images, such as filled geometric images, filled texture images, etc., and generate compressed images.

[0089] According to an embodiment, the video encoder (603) can encode and compress images of a source video sequence (images) into an encoded video sequence (compressed images) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units described below. For simplicity, coupling is not shown in the figures. Parameters set by the controller (650) may include rate control related parameters (image skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), image size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be used with other suitable functions related to the video encoder (603) optimized for a particular system design.

[0090] In some embodiments, the video encoder (603) operates in an encoding loop. As a simplified description, in one example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols, such as a symbol stream, based on the input image to be encoded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols in a manner similar to how the (remote) decoder creates sample data, to create sample data (because in the video compression techniques considered in the subject matter disclosed in this application, any compression between the symbols and the encoded video stream is lossless). The reconstructed sample stream (sample data) is input to a reference image memory (634). Since decoding of the symbol stream produces bit-precise results independent of the decoder's location (local or remote), the contents of the reference image memory (634) are also bit-precisely corresponding between the local encoder and the remote encoder. In other words, the reference image samples "seen" by the encoder's prediction section are exactly the same sample values ​​that the decoder will "see" when using the prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related technologies.

[0091] The operation of the “local” decoder (633) can be combined with, for example, the above-described... Figure 5 The video decoder (510) is described in detail as the same as the "remote" decoder. However, a further brief reference is provided. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode the symbols into the encoded / decoded video sequence without loss, the entropy decoding part of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0092] During operation, in some examples, the source encoder (630) may perform motion-compensated predictive coding, predictively encoding the input image with reference to one or more previously encoded images from the video sequence designated as "reference images." In this manner, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of the reference image, which may be selected as a predictive reference for the input image.

[0093] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture, based on symbols created by the source encoder (630). The operation of the encoding engine (632) can be a lossy process. When the encoded video data can be decoded by the video decoder (633), Figure 6 During decoding (not shown), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) replicates the decoding process, which can be performed by the video decoder on the reference image, and allows the reconstructed reference image to be stored in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares the same content (no transmission errors) as the reconstructed reference image to be obtained by the remote video decoder.

[0094] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, based on the search results obtained by the predictor (635), it can be determined that the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0095] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0096] The outputs of all the above-mentioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) performs lossless compression on the symbols generated by the various functional units according to techniques such as Huffman coding, variable length coding, and arithmetic coding, thereby converting the symbols into an encoded video sequence.

[0097] The controller (650) manages the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, which may affect the encoding techniques applicable to the corresponding image. For example, images can typically be assigned to any of the following image types:

[0098] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand variations of I-pictures and their corresponding applications and characteristics.

[0099] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and a reference index to predict sample values ​​for each block.

[0100] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata to reconstruct a single block.

[0101] Source images are typically spatially subdivided into multiple sample blocks (e.g., each with 4×4, 8×8, 4×8, or 16×16 samples), and encoded block by block. These blocks can be predictively coded with reference to other (already coded) blocks, determined based on the coding assignments of the corresponding images applied to the blocks. For example, blocks of an I-image can be non-predictively coded, or the blocks can be predictively coded (spatial or intra-frame prediction) with reference to already coded blocks of the same image. Pixel blocks of a P-image can be predictively coded with reference to a previously coded reference image via spatial or temporal prediction. Blocks of a B-image can be predictively coded with reference to one or two previously coded reference images via spatial or temporal prediction.

[0102] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0103] Video can be a series of source images presented in time. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In an embodiment, a specific image being encoded / decoded is segmented into blocks, referred to as the current image. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. This motion vector points to the reference block in the reference image, and when multiple reference images are used, the motion vector may have a third dimension that identifies the reference image.

[0104] In some embodiments, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, for example, a first reference image and a second reference image, both preceding the current image in the video in decoding order (but possibly past and future in display order). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. The block can be predicted using a combination of the first and second reference blocks.

[0105] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0106] According to some embodiments disclosed in this application, predictions such as inter-frame image prediction and intra-frame image prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video image sequence are segmented into coding tree units (CTUs) for compression. The CTUs in the images have the same size, for example, 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Furthermore, each CTU can be further subdivided into one or more coding units (CUs) using a quadtree. For example, a 64×64 pixel CTU can be subdivided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In embodiments, each CU is analyzed to determine the prediction type used for the CU, for example, inter-frame prediction or intra-frame prediction. Furthermore, depending on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in encoding (encoding / decoding) are performed on a per-prediction-block basis. Taking a luma prediction block as an example, a prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0107] Figure 7 A block diagram of a G-PCC encoder (700) in some examples is shown. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In an embodiment, the G-PCC encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic encoding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing the reconstructed attribute values.

[0108] like Figure 7As shown, an input point cloud (701) can be received at a G-PCC encoder (700). The position (e.g., 3D coordinates) of the point cloud (701) is provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. A duplicate point removal module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform an octree-based encoding process to generate an octet code sequence describing the 3D voxel grid. The octet code is provided to an arithmetic encoding module (770).

[0109] The attribute transfer module (720) is configured to receive attributes of the input point cloud and, when multiple attribute values ​​are associated with individual voxels, perform an attribute transfer process to determine the attribute value for each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize these points into different LODs. LOD information is supplied to the attribute prediction module (750).

[0110] The attribute prediction module (750) processes these points according to the LOD-based order indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction value for the current point based on the reconstructed attribute values ​​of the set of neighboring points stored in memory (790). The prediction residual can then be obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction value. When candidate indices are used in the corresponding attribute prediction process, the index corresponding to the selected prediction candidate can be provided to the arithmetic encoding module (770).

[0111] The residual quantization module (760) is configured to receive the prediction residuals from the attribute prediction module (750) and perform quantization to generate quantized residuals. The quantized residuals are then provided to the arithmetic coding module (770).

[0112] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate the reconstructed prediction residual by performing the inverse operation of the quantization operation performed at the residual quantization module (760). The addition module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the corresponding attribute prediction value from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute prediction value, a reconstructed attribute value is generated and stored in the memory (790).

[0113] The arithmetic coding module (770) is configured to receive occupancy codes, candidate indices (if used), quantized residuals (if generated), and other information, and to perform entropy coding to further compress the received values ​​or information. As a result, a compressed bitstream (702) carrying the compressed information can be generated. The bitstream (702) can be transmitted or otherwise provided to a decoder that decodes the compressed bitstream, or it can be stored in a storage device.

[0114] Figure 8 A block diagram of a G-PCC decoder (800) according to an embodiment is shown. The G-PCC decoder (800) can be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In an embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.

[0115] like Figure 8 As shown, a compressed bitstream (801) can be received at the arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain the quantized residual (if generated) and occupancy code of the point cloud. The octree decoding module (830) is configured to determine the reconstructed positions of points in the point cloud based on the occupancy code. The LOD generation module (840) is configured to reorganize these points into different LODs based on the reconstructed positions and determine the LOD-based order. The inverse residual quantization module (820) is configured to generate a reconstructed residual based on the quantized residual received from the arithmetic decoding module (810).

[0116] The attribute prediction module (850) is configured to perform an attribute prediction process according to a LOD-based order to determine the attribute prediction value of the point. For example, the attribute prediction value of the current point can be determined based on the reconstructed attribute values ​​of the current point's neighboring points stored in memory (860). In some examples, the attribute prediction value can be combined with the corresponding reconstruction residual to generate the reconstructed attribute value of the current point.

[0117] In one example, the sequence of reconstructed attribute values ​​generated from the attribute prediction module (850) and the reconstructed locations generated from the octree decoding module (830) correspond together with the decoded point cloud (802) output from the G-PCC decoder (800). Furthermore, the reconstructed attribute values ​​are also stored in memory (860) and can subsequently be used to derive attribute prediction values ​​for subsequent points.

[0118] In various embodiments, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented in hardware, software, or a combination thereof. For example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented using a processing circuitry system such as one or more integrated circuits (ICs) that operate with or without software, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. In another example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented as software or firmware comprising instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. These instructions, when executed by a processing circuitry system such as one or more processors, cause the processing circuitry system to perform the functions of the encoder (300), decoder (400), encoder (700), and / or decoder (800).

[0119] Please note that the attribute prediction module (750) and attribute prediction module (850) configured to implement the attribute prediction technology disclosed in this application may be included in modules that can have the same characteristics as those disclosed in this application. Figure 7 and Figure 8 Other decoders or encoders with similar or different structures as shown. Furthermore, in various examples, the encoder (700) and decoder (800) may be included in the same device or separate devices.

[0120] According to some aspects of this disclosure, mesh compression can use encoding and decoding tools different from PCC encoding and decoding tools, or it can use PCC encoding and decoding tools, such as the PCC (e.g., G-PCC, V-PCC) encoders and decoders described above.

[0121] An object's mesh (also called a mesh model or mesh frame) can comprise polygons describing the object's surfaces. Each polygon can be defined by its vertices in 3D space and the edges that connect the vertices to form the polygon. Information about how the vertices are connected (e.g., edge information) is called connectivity information. In some examples, the object's mesh is formed by connected triangles describing the object's surfaces. Two triangles sharing an edge are called two connected triangles. In other examples, the object's mesh is formed by connected quadrilaterals. Two quadrilaterals sharing an edge can be called two connected quadrilaterals. Note that the mesh can be formed by other suitable polygons.

[0122] In some examples, the mesh can also include attributes associated with the vertices, such as color, normals, etc. By using a 2D attribute map to parameterize the mesh's mapping information, attributes can be associated with the mesh's surface. Map information is typically described by a set of parametric coordinates associated with the mesh vertices, known as UV coordinates or texture coordinates. A 2D attribute map (in some examples, a texture map) is used to store high-resolution attribute information, such as texture, normals, displacement, etc. This information can be used for various purposes, such as texture mapping and painting.

[0123] In some embodiments, the mesh may include components referred to as geometric information, connectivity information, mapping information, vertex attributes, and an attribute map. In some examples, geometric information is described by a set of 3D positions associated with mesh vertices. In examples, (x,y,z) coordinates may be used to describe the 3D position of a vertex, also referred to as 3D coordinates. In some examples, connectivity information includes a set of vertex indices describing how vertices are connected to create a 3D surface. In some examples, mapping information describes how the mesh surface is mapped to a 2D region of a plane. In examples, mapping information is described by a set of UV parameter / texture coordinates (u,v) associated with mesh vertices, along with connectivity information. Texture coordinates associated with mesh vertices may define the position of the mapped mesh vertices in 2D space, such as a 2D map, UV atlas, etc. In some examples, texture coordinates are also referred to as UV coordinates or 2D coordinates. In some examples, vertex attributes include scalar or vector attribute values ​​associated with mesh vertices. In some examples, the attribute map includes attributes associated with the mesh surface and stored as a 2D image / video. In the example, the mapping between the video (e.g., 2D image / video) and the mesh surface is defined by mapping information.

[0124] According to one aspect of this disclosure, techniques known as UV mapping or mesh parameterization are used to map the surfaces of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is partitioned into patches in the 3D domain. A patch is a contiguous subset of the mesh whose boundaries are formed by boundary edges. A boundary edge of a patch is an edge that belongs to only one polygon in the patch and is not shared with two adjacent polygons in the patch. In some examples, the vertices of the boundary edges in a patch are called the boundary vertices of the patch, while non-boundary vertices in the patch may be called the interior vertices of the patch.

[0125] In some examples, the mesh of an object consists of connected triangles, and the mesh can be divided into patches, each patch being a subset of connected triangles. A patch's boundary edge is an edge that belongs only to one triangle within the patch and is not shared with adjacent triangles within the patch. In some examples, the vertices of the boundary edges in a patch are called the patch's boundary vertices, while non-boundary vertices in a patch can be called the patch's interior vertices. A boundary loop consists of a sequence of boundary vertices; the boundary edges formed by this sequence of boundary vertices can form a loop known as a boundary loop.

[0126] According to one aspect of this disclosure, in some examples, patches are parameterized as 2D shapes (also known as UV patches, 2D patches, or 2D graphs). In some examples, the 2D shapes can be packaged (e.g., oriented and placed) into a map, also known as an atlas. In some examples, the map can be further processed using 2D image or video processing techniques.

[0127] In one example, UV mapping generates a 2D UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) corresponding to patches of a 3D mesh. The UV atlas comprises 2D points that assign the 3D vertices of the 3D mesh to 2D points within a 2D domain (e.g., a rectangle). The UV atlas is a mapping between the coordinates of the 3D surface and the coordinates of the 2D domain. In the example, the value of a point at 2D coordinates (u,v) in the UV atlas is formed by the coordinates (x,y,z) of the vertex in the 3D domain. In the example, the texture atlas includes color information for the 3D mesh. For example, a point at 2D coordinates (u,v) in the texture atlas (whose 3D value in the UV atlas is (x,y,z)) has a color that specifies the color attribute of the point at (x,y,z) in the 3D domain. In some examples, the coordinates (x,y,z) in the 3D domain are referred to as 3D coordinates or xyz coordinates, and the 2D coordinates (u,v) are referred to as uv coordinates or UV coordinates.

[0128] According to some aspects of this disclosure, grid compression can be achieved by representing a grid using one or more 2D maps (also referred to as 2D atlases in some examples), and then encoding the 2D maps using an image or video codec. Different techniques can be used to generate 2D maps.

[0129] Figure 9 The illustration shows a schematic diagram of mapping a 3D mesh (910) to a 2D atlas (920) in some examples. Figure 9 In the example, the 3D mesh (910) comprises four vertices 1 to 4, forming four faces A to D. Each face has a set of vertices and associated attribute information. For example, face A is formed by vertices 1, 2, and 3 connected to form a triangle; face B is formed by vertices 1, 3, and 4 connected to form a triangle; face C is formed by vertices 1, 2, and 4 connected to form a triangle; and face D is formed by vertices 2, 3, and 4 connected to form a triangle. In some examples, vertices 1, 2, 3, and 4 may have corresponding attributes, and the triangles formed by vertices 1, 2, 3, and 4 may each have their own attributes.

[0130] In the example, patches A, B, C, and D in 3D are mapped to a 2D domain, such as a 2D atlas (920) also called a UV atlas (920) or map (920). For example, patch A is mapped to a 2D shape (also called a UV patch) A' in the map (920), patch B is mapped to a 2D shape (also called a UV patch) B' in the map (920), patch C is mapped to a 2D shape (also called a UV patch) C' in the map (920), and patch D is mapped to a 2D shape (also called a UV patch) D' in the map (920). In some examples, the coordinates in the 3D domain are called (x,y,z) coordinates, and the coordinates in the 2D domain (such as the map (920)) are called UV coordinates. Vertices in the 3D mesh can have corresponding UV coordinates in the map (920).

[0131] The map (920) can be a geometric map with geometric information, or a texture map with color, normal, texture or other attribute information, or an occupancy map with occupancy information.

[0132] Although Figure 9 In the examples, each facet is represented by a triangle, but it's important to note that a facet can include any suitable number of vertices that connect to form a contiguous subset of the mesh. In some examples, the vertices in a facet are connected to form triangles. It's also important to note that the vertices in a facet can be connected using other suitable shapes.

[0133] In the example, the geometric information of the vertices can be stored in a 2D geometry map. For instance, the 2D geometry map stores the (x, y, z) coordinates of the sampled points at the corresponding points in the 2D geometry map. For example, the point at position (u, v) in the 2D geometry map has a vector value with three components, corresponding to the x, y, and z values ​​of the corresponding sampled point in the 3D mesh.

[0134] According to one aspect of this disclosure, areas in a map do not necessarily need to be fully occupied. For example, in Figure 9 In the map, the regions outside the 2D shapes A', B', C', and D' are undefined. After decoding, sample values ​​from these regions can be discarded. In some cases, the occupancy map is used to store additional information for each pixel, such as storing binary values ​​to identify whether a pixel belongs to a patch or is undefined.

[0135] According to one aspect of this disclosure, a dynamic mesh is a mesh in which at least one component (geometric information, connectivity information, mapping information, vertex attributes, and attribute map) changes over time. A dynamic mesh can be described by a sequence of meshes (also called mesh frames). In some examples, mesh frames in a dynamic mesh can be representations of an object's surface at different times, each mesh frame being a representation of the object's surface at a specific time (also called a temporal instance). Dynamic meshes may require large amounts of data because they may contain a significant amount of information that changes over time. Mesh compression techniques can efficiently store and transmit media content within a mesh representation.

[0136] In some examples, dynamic meshes can possess constant connectivity information, time-varying geometry, and time-varying vertex properties. Digital content creation tools often generate dynamic meshes with time-varying attribute maps and connectivity information. Volumetric acquisition techniques are used to generate dynamic meshes, particularly under real-time constraints.

[0137] Dynamic mesh sequences may require large amounts of data because dynamic meshes can contain a significant amount of information that changes over time. In particular, the bits used to signal the UV coordinates are a crucial part of the bitstream. Some aspects of this disclosure provide mesh compression techniques where texture coordinates (also known as UV coordinates) are derived instead of encoded. The derivation of the texture coordinates of the current mesh frame means that the texture coordinates of the current mesh frame are not directly encoded and signaled, but rather obtained from other suitable sources, such as derivation from the 3D coordinates and connectivity of the current mesh frame, or inheritance of texture coordinates from previously encoded mesh frames. Therefore, mesh encoding efficiency can be improved. Mesh compression techniques are based on UV coordinate derivation in mesh compression. Mesh compression techniques can be applied alone or in combination in any form. It should also be noted that mesh compression techniques can be applied to both dynamic and static meshes. A static mesh consists of a single mesh frame.

[0138] In some examples, trellis compression techniques can derive UV coordinates at the decoder end, rather than signaling the UV coordinates in the bitstream from the encoder. Therefore, a portion of the bit rate representing the UV coordinates can be saved. In some examples, the trellis codec (encoder / decoder) can be configured to perform lossless trellis encoding / decoding. In some examples, the trellis codec (encoder / decoder) can be configured to perform lossy trellis encoding / decoding. Some aspects of this disclosure provide trellis compression techniques for use with lossless trellis codecs (encoders / decoders), while some aspects of this disclosure provide trellis compression techniques for use with lossy trellis codecs (encoders / decoders).

[0139] Figure 10 A schematic diagram of a framework (1000) for a lossless mesh codec according to some embodiments of the present disclosure is shown. The framework (1000) includes a mesh encoder (1010) and a mesh decoder (1050). The mesh encoder (1010) encodes an input mesh (1005) (mesh frames in the case of dynamic meshes) into a bitstream (1045), and the mesh decoder (1050) decodes the bitstream (1045) to generate a reconstructed mesh (1095) (mesh frames in the case of dynamic meshes).

[0140] The mesh encoder (1010) can be any suitable device, such as a computer, server computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The mesh decoder (1050) can be any suitable device, such as a computer, client computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The bitstream (1045) can be transmitted from the mesh encoder (1010) to the mesh decoder (1050) via a network (not shown).

[0141] exist Figure 10 In the example, the mesh encoder (1010) includes a parameterization module (1020), a texture conversion module (1030), and multiple encoders, such as a lossless encoder (1040), an image / video encoder (1041), an auxiliary encoder (1042), etc.

[0142] In the example, the input mesh (1005) can include the 3D coordinates of the vertices (represented by XYZ), the raw texture coordinates of the vertices (also called raw UV coordinates, represented by raw UV), the vertex connectivity information (represented by connectivity), and the raw texture map. The raw texture map is associated with the raw texture coordinates of the vertices. Figure 10In the diagram, the 3D coordinates of the vertices and the connectivity information of the vertices are shown by XYX and connectivity (1021) and are provided to the parameterization module (1020) and the lossless encoder (1040). The original texture map and original UVs shown by (1031) are provided to the texture conversion module (1030).

[0143] The parameterization module (1020) is configured to perform parameterization, generating new texture coordinates (also known as new UV coordinates of vertices) of the vertices using the XYZ and connectivity (1021) of the input mesh (1005), as shown in the new UV (1025).

[0144] The texture conversion module (1030) can receive the original texture map and the original UVs (1031), and can also receive the new UVs (1025), and convert the original texture map into a new texture map (1022) associated with the new UVs (1025). The new texture map (1022) is provided to the image / video encoder (1041) for encoding.

[0145] XYZ and connectivity (1021) can include geometric information of vertices, such as (x,y,z) coordinates describing the vertex position in 3D space, and vertex connectivity information (e.g., also known as the definition of a polygon's face). In some examples, XYZ and connectivity (1021) also include vertex attributes, such as normals, color reflectivity, etc. A lossless encoder (1040) can use lossless encoding techniques to encode XYZ and connectivity (1021) into a bitstream (1045).

[0146] The new texture map (1022) (also referred to as the new attribute map in some examples) includes attributes associated with the mesh surface, which are linked to the new texture coordinates of the vertices. In some examples of dynamic mesh processing, the new texture map (1022) of a sequence of mesh frames can form a video sequence. The new texture map (1022) can be encoded using appropriate image and / or video encoding techniques by an image / video encoder (1041).

[0147] exist Figure 10 In the example, the grid encoder (1010) can generate auxiliary data (1023), which includes auxiliary information such as flags, indexes, etc. The auxiliary data encoder (1042) receives the auxiliary data (1023) and encodes the auxiliary data (1023) into the bitstream (1045).

[0148] exist Figure 10 In the example, the encoded outputs from the lossless encoder (1040), the image / video encoder (1041), and the auxiliary data encoder (1042) are mixed (e.g., multiplexed) into a bitstream (1045) carrying encoded information of the input grid (1005).

[0149] exist Figure 10 In the example, the grid decoder (1050) can demultiplex the bitstream (1045) into fragments and decode these fragments separately by multiple decoders, such as lossless decoder (1060), image / video decoder (1061) and auxiliary data decoder (1062).

[0150] In the example, the lossless decoder (1060) corresponds to the lossless encoder (1040) and can decode segments of the bitstream (1045) encoded by the lossless encoder (1040). The lossless encoder (1040) and the lossless decoder (1060) can perform lossless encoding and decoding. The lossless decoder (1060) can output the 3D coordinates of the decoded vertices and the vertex connectivity information, as shown by the decoded XYZ and connectivity (1065). The decoded XYZ and connectivity (1065) can be the same as the XYZ and connectivity (1021).

[0151] In some examples, the parameterization module (1070) includes the same parameterization algorithm as the parameterization module (1020). The parameterization module (1070) is configured to perform parameterization, generating new texture coordinates for vertices using the decoded XYZ and connectivity (1065), as shown in the new UV (1075). It should be noted that in some examples, the decoded XYZ and connectivity (1065) are the same as those for XYZ and connectivity (1021), therefore the new UV (1075) can be the same as the new UV (1025).

[0152] In the example, the image / video decoder (1061) corresponds to the image / video encoder (1041) and can decode segments of the bitstream (1045) encoded by the image / video encoder (1041). The image / video decoder (1061) can generate a decoded new texture map (1066). The decoded new texture map (1066) is associated with new UVs (1025), which can be identical to the new UVs (1075) at the mesh decoder (1050). The decoded new texture map (1066) is provided to the mesh reconstruction module (1080).

[0153] In the example, the auxiliary data decoder (1062) corresponds to the auxiliary data encoder (1042) and can decode segments of the bitstream (1045) encoded by the auxiliary data encoder (1042). The auxiliary data decoder (1062) can generate decoded auxiliary data (1067). The decoded auxiliary data (1067) is provided to the mesh reconstruction module (1080).

[0154] The mesh reconstruction module (1080) receives the decoded XYZ and connectivity (1065), new UVs (1075), decoded new texture map (1066), and decoded auxiliary data (1067), and generates the reconstructed mesh (1095) accordingly.

[0155] It should be noted that the components in the mesh encoder (1010), such as the parameterization module (1020), texture conversion module (1030), lossless encoder (1040), image / video encoder (1041), and auxiliary data encoder (1042), can be implemented using various techniques. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0156] It should be noted that the components in the mesh decoder (1050), such as the lossless decoder (1060), the parameterization module (1070), the mesh reconstruction module (1080), the image / video decoder (1061), and the auxiliary data decoder (1062), can be implemented using various techniques. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0157] According to one aspect of this disclosure, the parameterization modules (1020) and (1070) can be implemented using any suitable parameterization algorithm. In some examples, parameterization algorithms for generating new texture coordinates aligned between mesh frames in a 3D mesh sequence can be implemented in the parameterization modules (1020) and (1070). Therefore, the new texture maps of mesh frames in a 3D mesh sequence can have a high correlation between mesh frames, and the new texture maps of mesh frames can be encoded using inter-frame prediction, which can further improve encoding efficiency.

[0158] In some examples, when using the parameterization module (1020) to illustrate the parameterization algorithm, the parameterization module (1020) can partition the faces (also called polygons) of a mesh frame into different graphs (also called 2D patches, 2D shapes, UV patches, etc.) based on the normal values ​​of the faces. These graphs can be packed into a texture space (e.g., a UV map, a 2D UV atlas). In some examples, the parameterization module (1020) can process individual mesh frames sequentially and generate multiple UV maps packed with the graphs. The UV maps are used for mesh frames in the sequence. The parameterization module (1020) can perform temporal alignment of the graphs between the UV maps of the mesh frames.

[0159] In some examples, alignment can use a metric calculated from the charts. For example, the metric could be a scale-invariant metric, rotation-invariant metric, translation-invariant metric, or affine transformation-invariant metric. In one example, the metric could include the center of the 3D coordinates of each chart. In another example, the metric could include the average depth of each chart. In yet another example, the metric could include the weighted average texture value or attribute value of each chart. It is important to note that the metric used for alignment can include one or more of the metrics listed above, or it can include other metrics with similar properties.

[0160] In some examples, the parameterization module (1020) can use one or more metrics calculated from the graph to perform time alignment of the graph between grid frames in the sequence. The UV coordinates obtained after time alignment are the new UVs.

[0161] It should be noted that the image / video encoder (1041) and image / video decoder (1061) can be implemented using any suitable image and / or video codecs. In the example of a static mesh with a single mesh frame, the image / video encoder (1041) and image / video decoder (1061) can be implemented using an image codec such as a JPEG or PNG codec to encode and decode the new texture map of the single mesh frame. In the example of a dynamic mesh with a sequence of mesh frames, the image / video encoder (1041) and image / video decoder (1061) can be implemented using a video codec, for example, according to H.265, to encode and decode the new texture map of the sequence of mesh frames.

[0162] It should be noted that in some examples, the lossless encoder (1040) and lossless decoder (1060) can be implemented using a lossless mesh codec. The lossless mesh codec is configured to encode and decode the XYZ and connectivity information of the mesh frame, and skip the encoding and decoding of the UV coordinates. In embodiments, the lossless encoder (1040) and lossless decoder (1060) can be implemented by including the SC3DMC codec for lossless encoding of the mesh; the SC3DMC codec is MPEG reference software for static mesh compression.

[0163] Figure 11 A schematic diagram of a framework (1100) for a lossy mesh codec according to some embodiments of the present disclosure is shown. The framework (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) encodes an input mesh (1105) (mesh frames in the case of dynamic meshes) into a bitstream (1145), and the mesh decoder (1150) decodes the bitstream (1145) to generate a reconstructed mesh (1195) (mesh frames in the case of dynamic meshes).

[0164] The mesh encoder (1110) can be any suitable device, such as a computer, server computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The mesh decoder (1150) can also be any suitable device, such as a computer, client computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The bitstream (1145) can be transmitted from the mesh encoder (1110) to the mesh decoder (1150) via a network (not shown).

[0165] exist Figure 11 In the example, the mesh encoder (1110) includes a parameterization module (1120), a texture conversion module (1130), and multiple encoders, such as a lossy encoder (1140), a video encoder (1141), an auxiliary encoder (1142), etc. Furthermore, the mesh encoder (1110) includes a lossy decoder (1145) corresponding to the lossy encoder (1140).

[0166] In the example, the input mesh (1105) can include the 3D coordinates of the vertices (represented by XYZ), the raw texture coordinates of the vertices (also called raw UV coordinates, represented by raw UV), the vertex connectivity information (represented by connectivity), and the raw texture map. The raw texture map is associated with the raw texture coordinates of the vertices. Figure 11 In the diagram, the 3D coordinates of the vertices and the connectivity information of the vertices are shown by XYX and connectivity (1121) and are provided to the lossy encoder (1140). The original texture map and original UVs shown by (1131) are provided to the texture conversion module (1130).

[0167] XYZ and connectivity (1121) may include geometric information of vertices, such as (x,y,z) coordinates describing the vertex position in 3D space, and vertex connectivity information (e.g., also known as the definition of a polygon's face). In some examples, XYZ and connectivity (1121) may also include vertex attributes, such as normals, color reflectivity, etc. A lossy encoder (1140) may encode XYZ and connectivity (1121) into a bitstream (1145) using lossy encoding techniques. Further, the encoded XYZ and connectivity shown by (1144) is also provided to a lossy decoder (1145). In the example, the lossy decoder (1145) corresponds to the lossy encoder (1140) and may decode the encoded XYZ and connectivity (1144) encoded by the lossy encoder (1140). The lossy encoder (1140) and the lossy decoder (1145) may perform lossy encoding and decoding. Note that in some examples, the lossy encoder (1140) and lossy decoder (1145) can be implemented using a lossy mesh codec. The lossy mesh codec is configured to encode and decode the XYZ and connectivity information of the mesh frame, and skip the encoding and decoding of the UV coordinates. In an embodiment, the lossy encoder (1140) and lossy decoder (1145) can be implemented to include Draco for lossy mesh encoding and decoding. The lossy decoder (1145) can output the decoded 3D coordinates of the vertices and the vertex connectivity information, as shown in the decoded XYZ and connectivity (1146).

[0168] The parameterization module (1120) is configured to perform parameterization, generating new texture coordinates for vertices using the decoded XYZ and connectivity (1146), as shown by the new UV (1125).

[0169] The texture conversion module (1130) can receive the original texture map and the original UVs (1131), and can also receive the new UVs (1125), and convert the original texture map into a new texture map (1122) associated with the new UVs (1125). The new texture map (1122) is provided to the image / video encoder (1141) for encoding.

[0170] The new texture map (1122) (also referred to as the new attribute map in some examples) includes attributes associated with the mesh surface, which are linked to the new texture coordinates of the vertices. In some examples of dynamic mesh processing, the new texture map (1122) of a sequence of mesh frames can form a video sequence. The new texture map (1122) can be encoded using appropriate image and / or video encoding techniques by an image / video encoder (1141).

[0171] exist Figure 11In the example, the grid encoder (1110) can generate auxiliary data (1123), which includes auxiliary information such as flags, indexes, etc. The auxiliary data encoder (1142) receives the auxiliary data (1123) and encodes the auxiliary data (1123) into the bitstream (1145).

[0172] exist Figure 11 In the example, the encoded outputs from the lossy encoder (1140), the image / video encoder (1141), and the auxiliary data encoder (1142) are mixed (e.g., multiplexed) into a bitstream (1145) carrying encoded information of the input grid (1105).

[0173] exist Figure 11 In the example, the grid decoder (1150) can demultiplex the bitstream (1145) into segments and decode these segments separately by multiple decoders, such as a lossy decoder (1160), an image / video decoder (1161), and an auxiliary data decoder (1162).

[0174] In the example, the lossy decoder (1160) corresponds to the lossy encoder (1140) and can decode segments of the bitstream (1145) encoded by the lossy encoder (1140). The lossy encoder (1140) and the lossy decoder (1160) can perform lossy encoding and decoding. The lossy decoder (1160) can output the 3D coordinates of the decoded vertices and the vertex connectivity information, as shown by the decoded XYZ and connectivity (1165).

[0175] In some examples, the lossy decoder (1145) in the mesh encoder (1110) and the lossy decoder (1165) in the mesh decoder (1150) use the same decoding technique, so the decoded XYZ and connectivity (1165) can have the same decoded 3D coordinates and vertex connectivity information of the vertices as the decoded XYZ and connectivity (1146). It should be noted that, due to lossy encoding and decoding, the decoded XYZ and connectivity (1165) and the decoded XYZ and connectivity (1146) can be different from XYZ and connectivity (1121).

[0176] In some examples, the parameterization module (1170) includes the same parameterization algorithm as the parameterization module (1120). The parameterization module (1170) is configured to perform parameterization, generating new texture coordinates for vertices using the decoded XYZ and connectivity (1165), as shown in the new UV (1175). It should be noted that in some examples, the decoded XYZ and connectivity (1165) are the same as the decoded XYZ and connectivity (1146), so the new UV (1175) can be the same as the new UV (1125).

[0177] In the example, the image / video decoder (1161) corresponds to the image / video encoder (1141) and can decode segments of the bitstream (1145) encoded by the image / video encoder (1141). The image / video decoder (1161) can generate a decoded new texture map (1166). The decoded new texture map (1166) is associated with a new UV (1125), which can be the same as the new UV (1175) at the mesh decoder (1150). The decoded new texture map (1166) is provided to the mesh reconstruction module (1180).

[0178] In the example, the auxiliary data decoder (1162) corresponds to the auxiliary data encoder (1142) and can decode segments of the bitstream (1145) encoded by the auxiliary data encoder (1142). The auxiliary data decoder (1162) can generate decoded auxiliary data (1167). The decoded auxiliary data (1167) is provided to the mesh reconstruction module (1180).

[0179] The mesh reconstruction module (1180) receives the decoded XYZ and connectivity (1165), new UVs (1175), decoded new texture map (1166), and decoded auxiliary data (1167), and generates the reconstructed mesh (1195) accordingly.

[0180] It should be noted that the components in the mesh encoder (1110), such as the parameterization module (1120), texture conversion module (1130), lossy encoder (1140), lossy decoder (1145), image / video encoder (1141), and auxiliary data encoder (1142), can be implemented using various techniques. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0181] It should be noted that the components in the mesh decoder (1150), such as the lossy decoder (1160), the parameterization module (1170), the mesh reconstruction module (1180), the image / video decoder (1161), and the auxiliary data decoder (1162), can be implemented using various techniques. In one example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0182] According to one aspect of this disclosure, the parameterization modules (1120) and (1170) can be implemented using any suitable parameterization algorithm. In some examples, parameterization algorithms for generating new texture coordinates aligned between mesh frames in a 3D mesh sequence can be implemented in the parameterization modules (1120) and (1170). Therefore, the new texture maps of mesh frames in a 3D mesh sequence can have a high correlation between mesh frames, and the new texture maps of mesh frames can be encoded using inter-frame prediction, which can further improve encoding efficiency.

[0183] In some examples, when using a parameterization module (1120) to illustrate the parameterization algorithm, the parameterization module (1120) can partition the faces (also called polygons) of a mesh frame into different graphs (also called 2D patches, 2D shapes, UV patches, etc.) based on the normal values ​​of the faces. These graphs can be packed into a texture space (e.g., a UV map, a 2D UV atlas). In some examples, the parameterization module (1120) can process individual mesh frames sequentially and generate multiple UV maps packed with the graphs. The UV maps are used for mesh frames in the sequence. The parameterization module (1120) can perform temporal alignment of the graphs between the UV maps of the mesh frames.

[0184] In some examples, alignment can use a metric calculated from the charts. For example, the metric could be a scale-invariant metric, rotation-invariant metric, translation-invariant metric, or affine transformation-invariant metric. In one example, the metric could include the center of the 3D coordinates of each chart. In another example, the metric could include the average depth of each chart. In yet another example, the metric could include the weighted average texture value or attribute value of each chart. It is important to note that the metric used for alignment can include one or more of the metrics listed above, or it can include other metrics with similar properties.

[0185] In some examples, the parameterization module (1120) can use one or more metrics calculated from the graph to perform time alignment of the graph between grid frames in the sequence. The UV coordinates obtained after time alignment are the new UVs.

[0186] It should be noted that the image / video encoder (1141) and image / video decoder (1161) can be implemented using any suitable image and / or video codecs. In the example of a static mesh with a single mesh frame, the image / video encoder (1141) and image / video decoder (1161) can be implemented using an image codec such as a JPEG or PNG codec to encode and decode the new texture map of the single mesh frame. In the example of a dynamic mesh with a sequence of mesh frames, the image / video encoder (1141) and image / video decoder (1161) can be implemented using a video codec, for example, according to H.265, to encode and decode the new texture map of the sequence of mesh frames.

[0187] According to one aspect of this disclosure, the mesh encoding side may choose to signal the texture coordinates of the vertices in the bitstream (so that the mesh decoder can directly decode the texture coordinates from the bitstream), or derive the texture coordinates on the mesh decoding side. In some examples, a high-level indicator flag may be used to signal this choice and notify the mesh decoder.

[0188] In an embodiment, a sequence-level flag can be used to indicate whether the vertex texture coordinates are derived on the mesh decoding side or signaled in the bitstream. The entire sequence of mesh frames associated with the sequence-level flag follows the indication of the sequence-level flag.

[0189] In another embodiment, a frame-level flag can be used to indicate whether texture coordinates are derived on the mesh decoding side or signaled in the bitstream. The entire mesh frame sequence associated with the frame-level flag follows the indication of the frame-level flag.

[0190] In another embodiment, a group of mesh frames can use a group-level flag to indicate whether the texture coordinates are derived on the mesh decoding side or signaled in the bitstream. The entire mesh frame group follows the indication of the group-level flag. The concept of a mesh frame group is similar to a picture group in video coding. For example, a picture group may include pictures referencing the same Picture Parameter Set (PPS) or pictures within the same random access point; a mesh frame group may include mesh frames referencing the same Picture Parameter Set (PPS) or mesh frames within the same random access point.

[0191] In another embodiment, as described above, when a sequence-level flag is present, the frame-level flag or group-level flag is conditionally signaled to indicate that texture coordinate derivation can be performed. Otherwise, it is not necessary to signal the frame-level flag or group-level flag. Instead, a default value is assigned to the frame-level flag and group-level flag (indicating that texture coordinates are signaled in the bitstream).

[0192] According to another aspect of this disclosure, additional information can be signaled in the bitstream to assist the mesh decoder in deriving texture coordinates. This additional information can be signaled at various levels. In one example, the additional information can be signaled using high-level syntax, such as in the sequence header, frame header, or group header (for a group of mesh frames). In another example, the additional information can be signaled using lower-level syntax, such as for each patch, each graph, each stripe, etc.

[0193] In some embodiments, the bitstream may include a flag indicating whether the texture coordinates of the current mesh frame are inherited from previously decoded mesh frames. Furthermore, when the flag indicates that the texture coordinates of the current mesh frame are inherited from previously decoded mesh frames, an index may be further signaled to indicate a reference frame in a set of previously decoded frames. The texture coordinates of the current frame can then be directly inherited from the indicated mesh frame, skipping the entire parameterization process of the mesh decoder.

[0194] In some embodiments, the mesh encoder can determine key vertices in the mesh frame vertices used to initiate parameterization, and the indices of the key vertices can be associated with the mesh frame and signaled. The mesh decoder can then decode the indices of the key vertices and begin parameterization starting from the key vertices, partitioning the mesh frame into, for example, multiple graphs / patches / strips.

[0195] In some embodiments, a flag can be signaled for the current portion of a mesh frame, such as for each graph, each patch, or each stripe associated with the mesh frame, to indicate whether the texture coordinates of the current portion (e.g., current graph, current patch, current stripe) are inherited from previously decoded frames or derived through a parameterized method. When the flag indicates that the texture coordinates of the current portion are inherited from previously decoded frames, the index of the selected previously decoded frame and the decoded portion (e.g., graph, patch, stripe) can be signaled in the bitstream.

[0196] In some embodiments, the trellis encoder may select a specific parameterization method from a set of candidate parameterization methods and may include an index in the bitstream to indicate the selection of a particular parameterization method. It is important to note that the index may be signaled at different levels, such as sequence headers, frame headers, picture groups, associated with patches, associated with stripes (e.g., in strip headers), etc.

[0197] Figure 12 An example of the syntax table (1200) is shown in some examples.

[0198] Syntax table (1200) uses multiple flags to indicate whether the texture coordinates of the vertices are signaled in the bitstream (so that the mesh decoder can directly decode the texture coordinates from the bitstream) or whether the texture coordinates are derived on the mesh decoder side. For example, the multiple flags include a first flag (1210) at the sequence level and a second flag (1220) at the frame level. Syntax table (1200) also includes a third flag (1230) indicating whether the texture coordinates of the current mesh frame are inherited from previously decoded mesh frames, and an index (1240) indicating a reference frame from a set of previously decoded frames.

[0199] Specifically, in Figure 12 In the example, the first flag (1210) is represented by ps_derive_uv_enabled_flag. A first flag (1210) of 0 indicates that texture coordinate derivation is not enabled for mesh frames in the sequence. A first flag (1210) of 1 indicates that texture coordinates can be derived for mesh frames in the sequence. Whether the texture coordinates of the current mesh are derived can then be determined by the second flag (1220) associated with the current mesh. In the example, the first flag (1210) can be set to 0 when it is not present in the bitstream.

[0200] exist Figure 12 In the example, the second flag (1220) is represented by ph_derive_uv_flag. A second flag (1220) of 0 indicates that the texture coordinates of the current mesh frame are not derived; a second flag (1220) of 1 indicates that the texture coordinates of the current mesh frame are derived (e.g., by performing parameterization). If the second flag (1220) does not exist, it is set to 0. When the second flag (1220) is 0, the texture coordinates of the current mesh frame can be inherited or signaled. Whether the texture coordinates of the current mesh frame are inherited or signaled depends on the third flag (1230).

[0201] exist Figure 12 In the example, the third flag (1230) is represented by ph_inherit_uv_flag. A third flag (1230) of 0 indicates that the texture coordinates are not inherited from previously decoded mesh frames of the current mesh frame, but are signaled in the bitstream; a third flag (1230) of 1 indicates that the texture coordinates are inherited from previously decoded mesh frames of the current mesh frame. When the third flag (1230) does not exist, the third flag (1230) is set to 0.

[0202] When the third flag (1230) indicates that texture coordinates are inherited from a previously decoded mesh frame, the index (1240), represented by inherit_idx, is signaled to specify the frame index of the decoded mesh frame from the list of decoded mesh frames. The texture coordinates of the current mesh frame are inherited from the decoded mesh frames. When index (1240) does not exist, in this example, index (1240) is set to 0.

[0203] Figure 13 A flowchart of an overview process (1300) according to an embodiment of the present disclosure is shown. The process (1300) may be used during the encoding process of one or more grid frames. In various embodiments, the process (1300) is executed by a processing circuitry system. In some embodiments, the process (1300) is implemented with software instructions, so that the processing circuitry system executes the process (1300) when the software instructions are executed. The process begins at (S1301) and proceeds to (S1310).

[0204] At (S1310), new texture coordinates associated with vertices in the first 3D mesh frame are determined. The first 3D mesh frame represents the surface of an object with polygons.

[0205] At (S1320), the original texture map of the first 3D mesh frame is transformed into a new texture map associated with the new texture coordinates of the vertices. The original texture map is associated with the original texture coordinates of the vertices.

[0206] At (S1330), the 3D coordinates of the vertices, the connectivity information of the vertices, and the new texture map are encoded into the bitstream carrying the first 3D mesh frame.

[0207] In some examples, the encoding of the 3D coordinates and connectivity information of the vertices is performed according to a lossless codec.

[0208] In some examples, encoding of the 3D coordinates and connectivity information of vertices is performed according to a lossy codec.

[0209] In some examples, to derive new texture coordinates associated with a vertex, the encoded 3D coordinates and connectivity information can be decoded to generate decoded 3D coordinates and connectivity information for the vertex. Parameterization is then performed based on the decoded 3D coordinates and connectivity information of the vertex to determine the new texture coordinates associated with it.

[0210] In some examples, to perform parameterization, the polygons of the first 3D mesh frame are partitioned into first or more graphs in a 2D map based on the decoded 3D coordinates of the vertices and decoded connectivity information. Temporal alignment is performed, aligning the first or more graphs with second or more graphs associated with the second 3D mesh frame. The first and second 3D mesh frames are frames in a 3D mesh sequence. New texture coordinates are determined from the first or more graphs that have been temporally aligned.

[0211] In some examples, the polygon is partitioned based on the normal values ​​associated with it.

[0212] In some examples, time alignment is performed based on at least one of the scale-invariant, rotation-invariant, translation-invariant, and affine transformation-invariant indices.

[0213] In one example, time alignment is performed based on the center of the chart, calculated from the 3D coordinates associated with the chart. In another example, time alignment is performed based on the average depth of the chart. In yet another example, time alignment is performed based on the weighted average texture value of the chart. In yet another example, time alignment is performed based on the weighted average attribute value of the chart.

[0214] In some examples, a flag indicating that texture coordinate derivation is enabled is encoded into the bitstream. This flag is at least one of a sequence-level flag, a set of frame-level flags, and a frame-level flag.

[0215] In some examples, to derive the texture coordinates associated with a vertex, a flag indicating the inheritance of texture coordinates is encoded into the bitstream. A specific encoded 3D mesh frame is selected from the list of encoded 3D mesh frames to inherit texture coordinates from that specific encoded 3D mesh frame. The index indicating the selection of a specific encoded 3D mesh frame from the list of encoded 3D mesh frames is encoded into the bitstream.

[0216] In some examples, a flag associated with a portion of the texture map is encoded into the bitstream, indicating whether the texture coordinates associated with that portion of the texture map are inherited from an encoded mesh frame or derived through parameterization.

[0217] In some examples, a set of key vertices is identified, and parameterization begins with this set of key vertices. The indices of this set of key vertices are then encoded into the bitstream.

[0218] In some examples, the selected parameterization method is determined from a list of candidate parameterization methods, and the index indicating the parameterization method selected from the list of candidate parameterization methods is encoded into the bitstream.

[0219] Then, the process proceeds to (S1399) and terminates.

[0220] The process (1300) can be modified as appropriate. One or more steps in the process (1300) can be improved and / or omitted. One or more additional steps can be added. Any suitable implementation order can be used.

[0221] Figure 14 A flowchart of an overview process (1400) according to an embodiment of the present disclosure is shown. The process (1400) may be used during the decoding process of one or more grid frames. In various embodiments, the process (1400) is executed by a processing circuitry system. In some embodiments, the process (1400) is implemented as software instructions, so that the processing circuitry system executes the process (1400) when the software instructions are executed. The process begins at (S1401) and proceeds to (S1410).

[0222] At (S1410), the 3D coordinates of the vertices and the connectivity information of the vertices in the first 3D mesh frame are decoded from the bitstream carrying the first 3D mesh frame. The first 3D mesh frame represents the surface of an object with polygons.

[0223] At (S1420), the texture coordinates associated with the vertex are derived. In some examples, the bitstream does not include the encoded texture coordinates.

[0224] At (S1430), the texture map of the first 3D mesh frame is decoded from the bitstream. The texture map includes a first one or more 2D graphs of 2D vertices with texture coordinates.

[0225] At (S1440), the first 3D mesh frame is reconstructed based on the 3D coordinates of the vertices, the connectivity information of the vertices, the texture map, and the texture coordinates.

[0226] In some examples, a lossless codec is used to encode and decode the 3D coordinates and connectivity information of the vertices. In other examples, a lossy codec is used to encode and decode the 3D coordinates and connectivity information of the vertices.

[0227] In some examples, to derive the texture coordinates associated with a vertex, parameterization is performed based on the vertex's 3D coordinates and connectivity information to determine the texture coordinates associated with the vertex. For example, the polygons of a first 3D mesh frame are partitioned into one or more first graphs, and the first or more graphs are packaged into a 2D map. Temporal alignment is performed to align the first or more graphs with a second or more graphs associated with a second 3D mesh frame. The first and second 3D mesh frames are frames in a 3D mesh sequence. Texture coordinates are determined from the first or more graphs that have been temporally aligned.

[0228] In some examples, the polygon is partitioned into a first or one other graph based on the normal values ​​associated with it.

[0229] In some examples, time alignment is performed based on at least one of the scale-invariant, rotation-invariant, translation-invariant, and affine transformation-invariant indices.

[0230] In one example, time alignment is performed based on the center of the chart, calculated from the 3D coordinates associated with the chart. In another example, time alignment is performed based on the average depth of the chart. In yet another example, time alignment is performed based on the weighted average texture value of the chart. In yet another example, time alignment is performed based on the weighted average attribute value of the chart.

[0231] In some examples, a flag indicating that texture coordinate derivation is enabled is decoded. This flag is at least one of a sequence-level flag, a set of frame-level flags, and a frame-level flag.

[0232] In some examples, to derive the texture coordinates associated with a vertex, a flag indicating the inherited texture coordinates is decoded. Then, the index indicating the 3D mesh frame selected from the list of decoded 3D mesh frames is decoded. The texture coordinates are then inherited from the selected 3D mesh frame.

[0233] In some examples, a flag associated with a portion of the texture map is decoded. This flag indicates whether the texture coordinates associated with that portion of the texture map are inherited from an already decoded mesh frame or derived through parameterization.

[0234] In some examples, the indices of a set of key vertices are decoded. Parameterization begins with this set of key vertices.

[0235] In some examples, the index indicating the parameterization method to be selected from the list of candidate parameterization methods is decoded. Then, parameterization is performed according to the selected parameterization method.

[0236] Then, the process proceeds to (S1499) and terminates.

[0237] The process (1400) can be modified as appropriate. One or more steps in the process (1400) can be improved and / or omitted. One or more additional steps can be added. Any suitable implementation order can be used.

[0238] The techniques disclosed herein can be used individually or in any combination in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders can be implemented via a processing circuit system (e.g., one or more processors or one or more integrated circuits). In some examples, one or more processors execute a program stored on a non-volatile computer-readable medium.

[0239] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 15 A computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0240] The computer software can be encoded using any suitable machine code or computer language, which can be subjected to assembly, compilation, linking or similar mechanisms to create code including instructions that can be executed directly or by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., through interpretation, microcode execution, etc.

[0241] The instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablets, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0242] Figure 15 The components shown for the computer system (1500) are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing embodiments of this application. Nor should the configuration of the components be construed as having any dependency or requirement on any one or combination of components shown in the exemplary embodiments of the computer system (1500).

[0243] The computer system (1500) may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users via, for example, tactile input (e.g., key presses, swipes, data glove movements), audio input (e.g., voice, taps), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0244] The input human-machine interface device may include one or more of the following (only one of each is depicted): keyboard (1501), mouse (1502), trackpad (1503), touch screen (1510), data glove (not shown), joystick (1505), microphone (1506), scanner (1507), and camera (1508).

[0245] The computer system (1500) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback from a touchscreen (1510), a data glove (not shown), or a joystick (1505), but tactile feedback devices that do not act as input devices may also exist), audio output devices (e.g., speakers (1509), headphones (not depicted)), visual output devices (e.g., screens (1510), including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which are capable of outputting two-dimensional or greater than three-dimensional visual outputs in a manner such as stereoscopic flat image output; virtual reality glasses (not depicted), holographic displays and smoke boxes (not depicted), and printers (not depicted).

[0246] The computer system (1500) may also include human-accessible storage devices and associated media of the storage devices, such as optical media, including CD / DVD ROM / RW (1520) having media such as CD / DVD (1521), thumb drives (1522), removable hard disk drives or solid-state drives (1523), legacy magnetic media such as magnetic tapes and floppy disks (not depicted), dedicated devices based on ROM / Application-Specific Integrated Circuits (ASICs) / Programmable Logic Devices (PLDs), such as security protection devices (not depicted), etc.

[0247] Those skilled in the art should also understand that the term "computer-readable medium" used in connection with the currently disclosed subject matter does not cover transmission media, carrier waves, or other transient signals.

[0248] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The network may be, for example, wireless, wired, or optical. The network may also be local, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks include, for example, local area networks such as Ethernet and wireless LANs; cellular networks including Global System for Mobile Communications (GSM), 3G, 4G, 5G, and LTE; wired or wireless wide area digital TV networks including cable TV, satellite TV, and terrestrial broadcast TV; and vehicular and industrial networks including Controller Area Network Bus (CANBus), etc. Some networks typically require external network interface adapters attached to certain general-purpose data ports or peripheral buses (1549) (e.g., the Universal Serial Bus (USB) port of the computer system (1500)); others are typically integrated into the core of the computer system (1500) by attaching to system buses as described below (e.g., integrated into a PC computer system via an Ethernet interface, or integrated into a smartphone computer system via a cellular network interface). By using any of these networks, the computer system (1500) can communicate with other entities. Such communication can be one-way receiving (e.g., broadcasting TV), one-way transmitting (e.g., a CANBus connected to a CANBus device), or bidirectional, such as connecting to other computer systems using a local area digital network or a wide area digital network. Certain protocols and protocol stacks can be used on each of those networks and network interfaces described above.

[0249] The aforementioned human-machine interface device, human-accessible storage device, and network interface can be attached to the core (1540) of the computer system (1500).

[0250] The core (1540) may include one or more central processing units (CPU) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units (1543) in the form of field-programmable gate areas (FPGAs), hardware accelerators (1544) for certain tasks, graphics adapters (1550), and so on. These devices, along with read-only memory (ROM) (1545), random access memory (1546), internal mass storage devices such as internal non-user-accessible hard disk drives, solid-state drives (SSDs), etc. (1547), may be connected via a system bus (1548). In some computer systems, the system bus (1548) may be accessed via one or more physical connectors to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly or via a peripheral bus (1549) to the core's system bus (1548). In one example, a screen (1510) may be connected to a graphics adapter (1550). Architectures for the peripheral bus include peripheral interconnect (PCI), USB, and so on.

[0251] The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions, which, when combined, constitute the aforementioned computer code. The computer code can be stored in ROM (1545) or RAM (1546). Transient data can also be stored in RAM (1546), while permanent data can be stored, for example, in an internal mass storage device (1547). Fast storage and retrieval of any memory device can be achieved using a cache memory, which can be closely associated with one or more CPUs (1541), GPUs (1542), mass storage devices (1547), ROMs (1545), RAMs (1546), etc.

[0252] The computer-readable medium may contain computer code for performing various computer-implemented operations. The medium and computer code may be designed and constructed specifically for the purposes of this application, or may belong to a class well-known and available to those skilled in the art of computer software.

[0253] For example, but not as a limitation, a computer system having an architecture (1500), and particularly a core (1540), can provide functionality resulting from the execution of software embodied in one or more tangible computer-readable media by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be media associated with certain storage devices of a non-transitory nature (e.g., internal core storage device (1547) or ROM (1545)) of the user-accessible mass storage devices described above and the core (1540). Software implementing various embodiments of this application can be stored in such devices and executed by the core (1540). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core (1540), and more specifically, the processors therein (including CPUs, GPUs, FPGAs, etc.), to perform specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1546) and modifying such data structures according to processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality generated by logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1544)), which may operate in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. Where appropriate, references to software may cover logic, and vice versa. Where appropriate, references to computer-readable media may cover circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This application covers any suitable combination of hardware and software.

[0254] Although this application describes several exemplary embodiments, various modifications, arrangements, and alternative equivalents are possible within the scope of this application. Therefore, it should be understood that those skilled in the art can design various systems and methods, though not expressly shown or described herein, that embody the principles of this application, within the spirit and scope of the application.

Claims

1. A method for mesh decompression, characterized in that, include: Decode the 3D coordinates of vertices and the connectivity information of the vertices in the first 3D mesh frame from the bitstream carrying the first 3D mesh frame, the first 3D mesh frame representing the surface of an object with polygons; Derive the texture coordinates associated with the vertex; Decode the texture map of the first 3D mesh frame from the bitstream, the texture map comprising a first or more 2D graphs of 2D vertices having the texture coordinates; as well as The first 3D mesh frame is reconstructed based on the 3D coordinates of the vertex, the connectivity information of the vertex, the texture map, and the texture coordinates; The process of deriving the texture coordinates associated with the vertex includes: performing parameterization based on the 3D coordinates and connectivity information of the vertex to determine the texture coordinates associated with the vertex; Performing parameterization based on the 3D coordinates and connectivity information of the vertex to determine the texture coordinates associated with the vertex further includes: The polygons of the first 3D mesh frame are partitioned into the first or more 2D charts, and the first or more 2D charts are packaged into the 2D map; Perform time alignment to align the first or more 2D graphs with the second or more 2D graphs associated with the second 3D mesh frame, wherein the first 3D mesh frame and the second 3D mesh frame are frames in a 3D mesh sequence; and The texture coordinates are determined from the first or more 2D graphs that are time-aligned.

2. The method according to claim 1, characterized in that, The 3D coordinates of the vertex and the connectivity information of the vertex are decoded using at least one of a lossless codec and a lossy codec.

3. The method according to claim 1, characterized in that, Partitioning and packaging the polygon further includes: The polygon is partitioned based on the normal values ​​associated with it.

4. The method according to claim 1, characterized in that, Performing the time alignment further includes: Perform the time alignment based on at least one of the following: The proportion remains unchanged; Rotationally invariant index; Translation-invariant index; Affine transformation invariant index; The center of the chart is calculated based on the 3D coordinates associated with the chart; The average depth of the chart; The weighted average texture value of the chart; and / or The weighted average attribute value of the chart.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The decoding instruction enables a flag for texture coordinate derivation, wherein the flag is at least one of a sequence-level flag, a set of frame-level flags, and a frame-level flag.

6. The method according to any one of claims 1 to 4, characterized in that, Deriving the texture coordinates associated with the vertex includes: The decoding instruction inherits the flag of the texture coordinates; The decoding instruction specifies the index of the 3D mesh frame selected from the list of decoded 3D mesh frames; and Inherit the texture coordinates from the selected 3D mesh frame.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Decode a flag associated with a portion of the texture map, the flag indicating that the UV coordinates associated with the portion of the texture map are inherited from a decoded mesh frame or derived through parameterization.

8. The method according to claim 1, characterized in that, The method further includes: Decode the indexes of a set of key vertices in the vertices, the parameterization starting from the set of key vertices.

9. The method according to claim 1, characterized in that, The method further includes: The decoder indicates the index of the parameterized method selected from the list of candidate parameterized methods.

10. A method for grid compression, characterized in that, include: Determine new texture coordinates associated with vertices in a first 3D mesh frame, which represents the surface of an object with polygons; The original texture map of the first 3D mesh frame is converted into a new texture map associated with the new texture coordinates of the vertex, wherein the original texture map is associated with the original texture coordinates of the vertex; The 3D coordinates of the vertex, the connectivity information of the vertex, and the new texture map are encoded into a bitstream carrying the first 3D mesh frame; The step of determining the new texture coordinates associated with the vertices in the first 3D mesh frame includes: Decode the encoded 3D coordinates and connectivity information of the vertex to generate the decoded 3D coordinates and decoded connectivity information of the vertex; Parameterization is performed based on the decoded 3D coordinates of the vertex and the decoded connectivity information to determine the new texture coordinates associated with the vertex; The execution parameterization includes: Based on the decoded 3D coordinates and decoded connectivity information of the vertices, the polygons of the first 3D mesh frame are partitioned into the first or more 2D graphs in the 2D map; Perform time alignment to align the first or more 2D graphs with the second or more 2D graphs associated with the second 3D mesh frame, wherein the first 3D mesh frame and the second 3D mesh frame are frames in a 3D mesh sequence; The new texture coordinates are determined from the first or more 2D graphs that are time-aligned.

11. The method according to claim 10, characterized in that, The method further includes: A flag indicating the enablement of texture coordinate derivation is encoded into the bitstream, wherein the flag is at least one of a sequence-level flag, a set of frame-level flags, and a frame-level flag.

12. The method according to claim 10, characterized in that, Deriving new texture coordinates associated with the vertex includes: The flag indicating the inherited texture coordinates is encoded into the bitstream; Select a specific encoded 3D mesh frame from the list of encoded 3D mesh frames to inherit texture coordinates from that specific encoded 3D mesh frame; The index of the specific coded 3D mesh frame selected from the list of coded 3D mesh frames is encoded into the bitstream.

13. The method according to claim 10, characterized in that, The method further includes: A flag associated with a portion of the texture map is encoded into the bitstream, the flag indicating that the texture coordinates associated with the portion of the texture map are inherited from an encoded mesh frame or derived through a parameterization method.

14. The method according to claim 10, characterized in that, The method further includes: Identify a set of critical vertices from the vertices, wherein parameterization begins from this set of critical vertices; The indexes of the key vertices in this group are encoded into the bitstream.

15. An electronic device, characterized in that, It includes a memory for storing computer-readable instructions; and a processor for reading the computer-readable instructions and performing the method as described in any one of claims 1 to 14 as instructed by the computer-readable instructions.

16. A computer-readable storage medium having instructions stored thereon, which, when executed by a point cloud encoding and / or decoding computer, cause the computer to perform the method as described in any one of claims 1 to 14.

Images