Method, computer device and apparatus for mesh decompression, and storage medium

Abstract

Aspects of the disclosure provide methods and apparatuses for mesh coding (e.g., compression and decompression). In some examples, an apparatus for mesh coding includes processing circuitry. The processing circuitry decodes, from a bitstream carrying a three-dimensional (3D) mesh frame comprising a plurality of patches, an attribute array. The attribute array corresponds to vertices of the 3D mesh frame. The vertices of the 3D mesh frame are arranged into subsets that respectively belong to the plurality of patches. The processing circuitry determines a first portion of the attribute array corresponding to a first subset of vertices of the 3D mesh frame. The first subset belongs to a first patch of the plurality of patches. The processing circuitry determines first connectivity information of the first subset of vertices, and reconstructs the first patch of the 3D mesh frame based on the first portion of the attribute array and the first connectivity information.

Description

[0001] References merged

[0002] This application claims priority to U.S. Patent Application No. 17 / 903,505, filed September 6, 2022, entitled “VERTEXREORDERING FOR MESH COMPRESSION,” which claims priority to U.S. Provisional Application No. 63 / 241,876, filed September 8, 2021, entitled “Vertex Reordering for Mesh Compression,” and U.S. Provisional Application No. 63 / 241,883, filed September 8, 2021, entitled “Vertex Reordering for Mesh Compression.” The entire disclosure of the earlier applications is incorporated herein by reference. Technical Field

[0003] This disclosure describes implementations of grid coding in general, and more particularly relates to a method, computer device, apparatus, and non-transitory computer-readable storage medium for grid decompression. Background Technology

[0004] The background description provided herein is for the purpose of presenting the general context of this disclosure. The work of the currently attributed inventors, to the extent described in this background section, and aspects of the description that may not be otherwise limited to prior art at the time of filing, are neither expressly nor implicitly acknowledged as prior art to this disclosure.

[0005] Various techniques have been developed to capture and represent the world in 3D space, such as objects and environments within the world. A 3D representation of the world enables more immersive forms of interaction and communication. In some examples, point clouds and meshes can be used as 3D representations of the world. Methods based on UV atlas sampling can introduce problems such as undersampling, oversampling, compressing more vertices, and inconsistent sampling data between frames. These issues can negatively impact the compression ratio. Summary of the Invention

[0006] According to an embodiment, a method for mesh decompression is provided, characterized in that the method includes: decoding an attribute array from a bitstream carrying a three-dimensional (3D) mesh frame comprising a plurality of facets, the attribute array corresponding to the vertices of the 3D mesh frame, the vertices of the 3D mesh frame being arranged into subsets respectively belonging to the plurality of facets; determining a first portion of the attribute array corresponding to a first subset of the vertices of the 3D mesh frame, the first subset belonging to a first facet; determining first connectivity information of the first subset of the vertices; and reconstructing a first facet of the 3D mesh frame based on the first portion of the attribute array and the first connectivity information.

[0007] According to an embodiment, a computer device is provided. The computer device includes: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and execute the above-described method for grid decompression as instructed by the computer program code.

[0008] According to an embodiment, an apparatus for mesh decompression is provided, characterized in that the apparatus includes a processing circuit configured to perform the above-described method for mesh decompression.

[0009] According to an embodiment, a non-transitory computer-readable storage medium is provided, characterized in that the non-transitory computer-readable storage medium stores computer program code, and when the computer program code is executed by at least one processor, the at least one processor executes the above-described method for grid decompression.

[0010] The present invention provides a method, computer device, apparatus, and non-transitory computer-readable storage medium for mesh decompression, which offers a vertex rearrangement technique for mesh compression. The vertices of the mesh are rearranged according to specific rules to increase the correlation between adjacent vertices, thus allowing the use of predictive coding methods to increase coding efficiency. Attached Figure Description

[0011] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings, in which:

[0012] Figure 1 Block diagrams of communication systems in some examples are shown.

[0013] Figure 2 Block diagrams of some examples of streaming systems are shown.

[0014] Figure 3 Block diagrams of encoders for encoding point cloud frames are shown in some examples.

[0015] Figure 4 Block diagrams of decoders for decoding compressed bitstreams corresponding to point cloud frames are shown in some examples.

[0016] Figure 5 Block diagrams of video decoders in some examples are shown.

[0017] Figure 6 Block diagrams of video encoders are shown in some examples.

[0018] Figure 7 Block diagrams of encoders for encoding point cloud frames are shown in some examples.

[0019] Figure 8 Block diagrams of decoders for decoding compressed bitstreams carrying point cloud frames are shown in some examples.

[0020] Figure 9 The diagram shows a mapping from grid to atlas in some examples.

[0021] Figure 10 Two UV atlas frames are shown as examples.

[0022] Figure 11 A diagram of a framework for mesh compression according to some embodiments of this disclosure is shown.

[0023] Figure 12 A diagram of a framework for mesh compression according to some other embodiments of this disclosure is shown.

[0024] Figure 13 A flowchart outlining a process for rearranging vertices in a mesh according to some embodiments of this disclosure is shown.

[0025] Figure 14 A flowchart outlining some of the processing examples is shown.

[0026] Figure 15 A flowchart outlining processing examples from some other examples is shown.

[0027] Figure 16 These are schematic diagrams of computer systems in some examples. Detailed Implementation

[0028] This disclosure provides various aspects of technology in the field of three-dimensional (3D) media processing.

[0029] Technological advancements in 3D media processing, such as progress in 3D capture, 3D modeling, and 3D rendering, have facilitated the widespread availability of 3D media content across multiple platforms and devices. In one example, a baby's first steps could be captured on one continent, and media technologies could allow grandparents to view (and potentially interact with) the baby on another continent and enjoy an immersive experience. According to aspects of this disclosure, to improve the immersive experience, 3D models are becoming increasingly complex, and the creation and consumption of 3D models consume significant data resources, such as data storage and data transmission resources.

[0030] Based on some aspects of this disclosure, point clouds and meshes can be used as 3D models to represent immersive content.

[0031] A point cloud typically refers to a collection of points in 3D space, each point possessing associated attributes such as color, material properties, texture information, intensity, reflectivity, motion-related attributes, morphological attributes, and various other properties. Point clouds can be used to reconstruct objects or scenes as combinations of such points.

[0032] An object's mesh (also known as a mesh model) can include polygons that describe the object's surfaces. Each polygon can be defined by its vertices in 3D space and information about how the vertices connect to form the polygon. Information about how the vertices are connected is called connectivity information. In some examples, the mesh may also include properties associated with the vertices, such as color and normals.

[0033] Based on some aspects of this disclosure, some encoding tools used for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be re-meshed to generate a new mesh, and the connectivity information of the new mesh can be inferred. The vertices of the new mesh and the properties associated with the vertices of the new mesh can be treated as points in the point cloud and compressed using a PCC codec.

[0034] Point clouds can be used to reconstruct objects or scenes as combinations of points. These points can be captured using multiple cameras, depth sensors, or LiDAR in various settings, and can consist of thousands to billions of points to realistically represent the reconstructed scene or object. A patch can typically refer to a continuous subset of a surface described by a point cloud. In the example, a patch includes points whose surface normals are offset from each other by less than a threshold amount.

[0035] Point cloud compression (PCC) can be performed using various schemes, such as geometry-based schemes called G-PCC (Geometry-based Point Cloud Coding) and video-based schemes called V-PCC (Video-based Point Cloud Coding). According to some aspects of this disclosure, G-PCC directly encodes 3D geometry and is a purely geometry-based method with little connection to video coding, while V-PCC is largely based on video coding. For example, V-PCC can map points in a 3D cloud to pixels in a 2D raster (image). The V-PCC scheme can utilize general-purpose video codecs for point cloud compression. The PCC codec (encoder / decoder) in this disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.

[0036] According to aspects of this disclosure, the V-PCC scheme can use existing video codecs to compress the geometry, occupancy, and texture of a point cloud into three separate video sequences. The additional metadata required to interpret these three video sequences is compressed separately. A small portion of the overall bitstream is metadata, which, in this example, can be efficiently encoded / decoded using a software implementation. The majority of the information is processed by the video codec.

[0037] Figure 1 Block diagrams of communication systems (100) in several examples are shown. The communication system (100) includes multiple terminal devices that can communicate with each other via, for example, a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). Figure 1 In the example, the first pair of terminal devices (110) and (120) can perform unidirectional transmission of point cloud data. For example, terminal device (110) can compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to terminal device (110). The compressed point cloud can be sent, for example, as a bitstream via a network (150) to the other terminal device (120). Terminal device (120) can receive the compressed point cloud from the network (150), decompress the bitstream to reconstruct the point cloud, and display the reconstructed point cloud appropriately. Unidirectional data transmission can be common in media service applications, etc.

[0038] exist Figure 1In the example, terminal devices (110) and (120) may be shown as a server and a personal computer, but the principles of this disclosure are not limited thereto. Implementations of this disclosure are applicable to laptop computers, tablet computers, smartphones, gaming terminals, media players, and / or dedicated three-dimensional (3D) devices. Network (150) refers to any number of networks transmitting compressed point clouds between terminal devices (110) and (120). Network (150) may include, for example, cable (wired) and / or wireless communication networks. Network (150) may exchange data in circuit-switched channels and / or packet-switched channels. Representative networks include telecommunications networks, local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0039] Figure 2 Block diagrams of some examples of streaming systems (200) are shown. The streaming system (200) is an application using point clouds. The disclosed subject matter can be equally applied to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, etc.

[0040] The streaming system (200) may include a capture subsystem (213). The capture subsystem (213) may include a point cloud source (201) that generates, for example, an uncompressed point cloud (202), such as a Light Detection and Ranging (LIDAR) system, a 3D camera device, a 3D scanner, a graphics generation component that generates the uncompressed point cloud in software, etc. In the example, the point cloud (202) includes points captured by a 3D camera device. The point cloud (202) is depicted as thick lines to emphasize the high data volume compared to a compressed point cloud (204) (a bitstream of the compressed point cloud). The compressed point cloud (204) may be generated by an electronic device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) may include hardware, software, or a combination thereof to implement or enforce aspects of the disclosed subject matter as described in more detail below. Compressed point cloud (204) (or a bitstream of compressed point cloud (204) depicted as thin lines to emphasize its lower data volume when compared to the stream of point cloud (202) can be stored on a streaming server (205) for future use. One or more streaming client subsystems, for example Figure 2 The client subsystems (206) and (208) can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) may include a decoder (210), for example, in an electronic device (230). The decoder (210) decodes the incoming copy (207) of the compressed point cloud and creates an outgoing stream of the reconstructed point cloud (211) that can be rendered on a rendering device (212).

[0041] Note that electronic devices (220) and (230) may include other components (not shown). For example, electronic device (220) may include a decoder (not shown), and electronic device (230) may also include an encoder (not shown).

[0042] In some streaming systems, compressed point clouds (204), (207), and (209) (e.g., a bitstream of the compressed point cloud) can be compressed according to certain standards. In some examples, video coding standards are used in the compression of the point cloud. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc.

[0043] Figure 3 A block diagram of a V-PCC encoder (300) for encoding point cloud frames according to some embodiments is shown. In some embodiments, the V-PCC encoder (300) can be used in a communication system (100) and a streaming system (200). For example, an encoder (203) can be configured and operated in a similar manner to the V-PCC encoder (300).

[0044] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream corresponding to the compressed point cloud frames. In some implementations, the V-PCC encoder (300) can receive point cloud frames from a point cloud source, such as a point cloud source (201).

[0045] exist Figure 3 In the example, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image filling modules (316) and (318), a group expansion module (320), video compression modules (322), (323) and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).

[0046] According to aspects of this disclosure, a V-PCC encoder (300) converts a 3D point cloud frame along with some metadata (e.g., occupancy map and patch information) into an image-based representation used to convert the compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) can convert the 3D point cloud frame into a geometry image, a texture image, and an occupancy map, and then encode the geometry image, texture image, and occupancy map into a bitstream using video coding techniques. Typically, a geometry image is a 2D image having pixels filled with geometric values ​​associated with points projected onto the pixel, and pixels filled with geometric values ​​may be referred to as geometry samples. A texture image is a 2D image having pixels filled with texture values ​​associated with points projected onto the pixel, and pixels filled with texture values ​​may be referred to as texture samples. An occupancy map is a 2D image having pixels filled with values ​​indicating whether a patch is occupied or not.

[0047] The patch generation module (306) divides the point cloud into a set of patches that may overlap or not overlap (e.g., patches are defined as a continuous subset of surfaces described by the point cloud), such that each patch can be described by a depth field relative to a plane in 2D space. In some implementations, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while also minimizing reconstruction errors.

[0048] In some examples, the patch information module (304) can collect patch information indicating the size and shape of the patches. In some examples, the patch information can be packaged into image frames and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.

[0049] In some examples, the patch packing module (308) is configured to map the extracted patches onto a 2D grid while minimizing unused space and ensuring that each M×M (e.g., 16x16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.

[0050] The geometry image generation module (310) generates a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) generates a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) store the geometry and texture of the point cloud as images using a 3D-to-2D mapping computed during the packing process. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images (called layers). In the example, the geometry image is represented by a monochrome frame of WxH in YUV420-8-bit format. To generate the texture image, the texture generation process utilizes the reconstructed / smoothed geometry to compute the color to be associated with the resampled points.

[0051] The occupancy map module (314) can generate an occupancy map describing the fill information at each cell. For example, the occupancy map includes a binary map indicating whether each cell of the raster belongs to empty space or to the point cloud. In one example, the occupancy map uses binary information describing whether a pixel is filled for each pixel. In another example, the occupancy map uses binary information describing whether a pixel block is filled for each pixel block.

[0052] The occupancy map generated by the occupancy map module (314) can be compressed using either lossless or lossy encoding. When lossless encoding is used, the occupancy map is compressed using the entropy compression module (334). When lossy encoding is used, the occupancy map is compressed using the video compression module (332).

[0053] Note that the patch packing module (308) may leave some empty spaces between the packed 2D patches in the image frame. The image padding modules (316) and (318) can fill in the empty spaces (referred to as padding) to generate image frames that can be adapted to 2D video and image codecs. Image padding, also known as background padding, fills unused spaces with redundant information. In some examples, good background padding increases the bit rate to a minimum without introducing significant coding distortion around patch boundaries.

[0054] The video compression modules (322), (323), and (332) can encode 2D images, such as filled geometric images, filled texture images, and occupancy maps, based on suitable video coding standards such as HEVC, VVC, etc. In this example, the video compression modules (322), (323), and (332) are separate components that operate independently. Note that in another example, the video compression modules (322), (323), and (332) can be implemented as a single component.

[0055] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) can then adjust the generation of the texture image based on the reconstructed geometry. For example, when the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion can be taken into account when generating the texture image to correct for the distortion of the patch shape.

[0056] In some implementations, group expansion (320) is configured to fill pixels around object boundaries with redundant low-frequency content to improve coding gain and visual quality of the reconstructed point cloud.

[0057] The multiplexer (324) can multiplex the compressed geometric image, the compressed texture image, the compressed occupancy map, and the compressed auxiliary patch information into a compressed bit stream.

[0058] Figure 4 Block diagrams of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame are shown in some examples. In some examples, the V-PCC decoder (400) can be used in a communication system (100) and a streaming system (200). For example, a decoder (210) can be configured to operate in a similar manner to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.

[0059] exist Figure 4 In the example, the V-PCC decoder (400) includes a demultiplexer (432), a video decompression module (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).

[0060] The demultiplexer (432) can receive the compressed bit stream and separate it into a compressed texture image, a compressed geometric image, a compressed occupancy map, and compressed auxiliary patch information.

[0061] Video decompression modules (434) and (436) can decode compressed images and output decompressed images according to suitable standards (e.g., HEVC, VVC, etc.). For example, video decompression module (434) decodes compressed texture images and outputs decompressed texture images; and video decompression module (436) decodes compressed geometric images and outputs decompressed geometric images.

[0062] The occupancy map decompression module (438) can decode the compressed occupancy map according to a suitable standard (e.g., HEVC, VVC, etc.) and output the decompressed occupancy map.

[0063] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to a suitable standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.

[0064] The geometry reconstruction module (444) can receive a decompressed geometry image and generate a reconstructed point cloud geometry based on a decompressed occupancy map and decompressed auxiliary patch information.

[0065] The smoothing module (446) can smooth out inconsistencies at the edges of the patch. The smoothing process is designed to mitigate potential discontinuities that may occur at patch boundaries due to compression distortion. In some implementations, a smoothing filter can be applied to pixels located at patch boundaries to mitigate distortion that may be caused by compression / decompression.

[0066] The texture reconstruction module (448) can determine the texture information of points in the point cloud based on the decompressed texture image and smooth geometry.

[0067] The color smoothing module (452) can smooth out inconsistencies in shading. Non-adjacent faces in 3D space are typically packed adjacent to each other in 2D video. In some examples, pixel values ​​from non-adjacent faces may be mixed by a block-based video codec. The purpose of color smoothing is to reduce visible artifacts that appear at face boundaries.

[0068] Figure 5 Block diagrams of some examples of video decoders (510) are shown. The video decoder (510) can be used in a V-PCC decoder (400). For example, video decompression modules (434) and (436), and occupancy map decompression module (438) can be similarly configured as video decoders (510).

[0069] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a compressed image, such as an encoded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510). The parser (520) may perform parsing / entropy decoding on the received encoded video sequence. The encoding of the encoded video sequence may be based on video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters from the encoded video sequence, based on at least one parameter corresponding to a group, for use in the video decoder. Subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0070] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory to create symbols (521).

[0071] The reconstruction of the symbol (521) can involve multiple different units, depending on the type of the encoded video picture or a portion thereof (e.g., inter-frame and intra-frame pictures, inter-frame and intra-frame blocks) and other factors. Which units are involved and how they are involved can be controlled by subgroup control information parsed from the encoded video sequence by the parser (520). For brevity, the flow of such subgroup control information between the parser (520) and the multiple units described below is not depicted.

[0072] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units interact closely with each other and can be at least partially integrated into one another. However, for the purposes of describing the disclosed subject matter, it is appropriate to conceptually subdivide it into the following functional units.

[0073] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives from the parser (520) quantization transform coefficients as (one or more) symbols (521) and control information, including the transform method used, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values, which can be input into the aggregator (555).

[0074] In some cases, the output samples of the scaler / inverse transform (551) may belong to intra-coded blocks; that is, blocks that do not use predictive information from previously reconstructed images, but can use predictive information from previously reconstructed portions of the current image. Such predictive information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the blocks in the reconstruction using surrounding reconstructed information obtained from the current picture buffer (558). For example, the current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the prediction information already generated by the intra-picture prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0075] In other cases, the output samples of the scaler / inverse transform unit (551) may belong to inter-frame coded and possibly motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference image memory (557) to obtain samples for prediction. After motion compensation of the obtained samples according to the block-related symbols (521), these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The address in the reference image memory (557) from which the motion compensation prediction unit (553) obtains the predicted samples can be controlled by motion vectors, which are provided to the motion compensation prediction unit (553) in the form of symbols (521), which may have, for example, X components, Y components, and reference image components. Motion compensation may also include interpolation of sample values ​​obtained from the reference image memory (557) when using subsample precise motion vectors, motion vector prediction mechanisms, etc.

[0076] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique controlled by parameters included in the encoded video sequence (also referred to as the encoded video bitstream), which are available as symbols (521) from the parser (520) to the loop filter unit (556). However, the in-loop filtering technique may also respond to metadata obtained during decoding of a previous portion of the encoded picture or encoded video sequence (in the order of decoding) and to previously reconstructed and loop-filtered sample values.

[0077] The output of the loop filter unit (556) can be a sample stream, which can be output to a rendering device and stored in a reference image memory (557) for future inter-frame image prediction.

[0078] Once a certain encoded image is fully reconstructed, it can be used as a reference image for future predictions. For example, once the encoded image corresponding to the current image has been fully reconstructed and the encoded image (by, for example, the parser (520)) is identified as the reference image, the current image buffer (558) can become part of the reference image memory (557), and a new current image buffer can be reallocated before the reconstruction of subsequent encoded images begins.

[0079] The video decoder (510) can perform decoding operations according to a predetermined video compression technique as specified in a standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence follows the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile can select certain tools from all available tools in the video compression technique or standard as tools usable only under said profile. For compliance, the complexity of the encoded video sequence is also required to be within the limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sampling rate (measured, for example, megasamples per second), maximum reference picture size, etc. In some cases, the limitations set by the level can be further limited by the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management indicated by signals in the encoded video sequence.

[0080] Figure 6A block diagram of a video encoder (603) according to an embodiment of the present disclosure is shown. The video encoder (603) can be used in a V-PCC encoder (300) for compressing point clouds. In the example, video compression modules (322) and (323) and video compression module (332) are configured similarly to the encoder (603).

[0081] The video encoder (603) can receive images such as filled geometric images, filled texture images, etc., and generate compressed images.

[0082] According to the implementation, the video encoder (603) can encode and compress images (pictures) of a source video sequence into an encoded video sequence (compressed images) in real time or under any other time constraints required by the application. Implementing an appropriate encoding rate is a function of the controller (650). In some implementations, the controller (650) controls and is functionally coupled to other functional units as described below. Coupling is not depicted for brevity. Parameters set by the controller (650) may include rate control related parameters (picture skipping, quantizer, λ value of rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a particular system design.

[0083] In some implementations, the video encoder (603) is configured to operate in an encoding loop. As a highly simplified description, in this example, the encoding loop may include a source encoder (630) (e.g., responsible for creating symbols such as a symbol stream based on the input picture to be encoded and one or more reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to how a (remote) decoder would also create sample data (since any compression between the symbols and the encoded video bitstream is lossless in the video compression or techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream yields a bit-accurate result independent of the decoder's location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the reference picture samples "seen" by the encoder's prediction section are exactly the same sample values ​​that the decoder will "see" when using prediction during decoding. This fundamental principle of reference image synchronization (and the drift that occurs when synchronization cannot be maintained, for example, due to channel errors) is also used in some related techniques.

[0084] The operation of the "local" decoder (633) can be combined with that already mentioned above. Figure 5 The operation of a "remote" decoder, such as a video decoder (510), is the same as described in the detailed description. However, a brief reference is also provided. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are able to encode / decode the symbols into an encoded video sequence without loss, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633).

[0085] During operation, in some examples, the source encoder (630) may perform motion-compensated predictive coding, which predictively codes the input image with reference to one or more previously encoded images designated as "reference images" from the video sequence. In this way, the encoding engine (632) encodes the differences between pixel blocks of the input image and pixel blocks of one or more reference images (which may be selected as prediction references to the input image).

[0086] The local video decoder (633) can decode encoded video data of a picture that can be designated as a reference picture based on symbols created by the source encoder (630). The operation of the encoding engine (632) can advantageously be lossy. When the encoded video data can be decoded by the video decoder (633), Figure 6 When the reference image (not shown) is decoded, the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process performed on the reference image by the video decoder, and can store the reconstructed reference image in a reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that shares common content (no transmission errors) with the reconstructed reference image that will be obtained by the remote video decoder.

[0087] The predictor (635) can perform a prediction search against the encoding engine (632). That is, for a new image to be encoded, the predictor (635) can search in the reference image memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference image motion vectors, block shapes, etc., that can be used as suitable prediction references for the new image. The predictor (635) can operate pixel-by-pixel based on the sample blocks to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references obtained from multiple reference images stored in the reference image memory (634).

[0088] The controller (650) can manage the encoding operations of the source encoder (630), including, for example, setting parameters and subgroup parameters for encoding video data.

[0089] The outputs of all the aforementioned functional units can be entropy encoded in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0090] The controller (650) can manage the operation of the video encoder (603). During encoding, the controller (650) can assign a specific encoded image type to each encoded image, which may affect the encoding techniques that can be applied to the corresponding image. For example, images can typically be assigned to one of the following image types:

[0091] An intra-frame picture (I-picture) is a picture that can be encoded and decoded without using any other pictures in the sequence as a prediction source. Some video codecs allow different types of intra-frame pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will understand those variations of I-pictures and their corresponding applications and characteristics.

[0092] A predictive image (P-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0093] A bidirectional predictive image (B-image) can be an image that can be encoded and decoded using intra-frame prediction or inter-frame prediction, which uses at most two motion vectors and a reference index to predict sample values ​​for each block. Similarly, multiple predictive images can use more than two reference images and associated metadata for the reconstruction of a single block.

[0094] Source images are typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and encoded block by block. These blocks can be predictively encoded with reference to other (already encoded) blocks, which are determined by the coding assignments of the corresponding images applied to the blocks. For example, blocks of image I can be nonpredictively encoded, or blocks of image I can be predictively encoded (spatial prediction or intra-frame prediction) with reference to already encoded blocks of the same image. Pixel blocks of image P can be predictively encoded with reference to a previously encoded reference image via spatial prediction or temporal prediction. Blocks of image B can be predictively encoded with reference to one or two previously encoded reference images via spatial prediction or temporal prediction.

[0095] The video encoder (603) can perform encoding operations according to a predetermined video coding technique or standard, such as ITU-T H.265 Recommendation. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations utilizing temporal and spatial redundancy in the input video sequence. Therefore, the encoded video data can conform to the syntax specified by the video coding technique or standard used.

[0096] Video can be presented as a series of multiple source images over time. Intra-frame image prediction (often simplified to intra-frame prediction) utilizes spatial correlations within a given image, while inter-frame image prediction utilizes (temporal or other) correlations between images. In the example, the specific image being encoded / decoded, referred to as the current image, is segmented into blocks. When a block in the current image resembles a reference block in a previously encoded and still buffered reference image in the video, the block in the current image can be encoded using a vector called a motion vector. The motion vector points to the reference block in the reference image, and in the case of multiple reference images, the motion vector can have a third dimension that identifies the reference images.

[0097] In some implementations, bidirectional prediction techniques can be used in inter-frame image prediction. According to bidirectional prediction, two reference images are used, such as a first reference image and a second reference image that both precede the current image in the video in decoding order (but may be in the past and future in display order, respectively). A block in the current image can be encoded using a first motion vector pointing to a first reference block in the first reference image and a second motion vector pointing to a second reference block in the second reference image. A block can be predicted using a combination of the first and second reference blocks.

[0098] In addition, merging mode techniques can be used in inter-frame image prediction to improve coding efficiency.

[0099] According to some embodiments of this disclosure, predictions such as inter-frame picture prediction and intra-frame picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, images in a video picture sequence are segmented into Coding Tree Units (CTUs) for compression, with CTUs in the images having the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU comprises three Coding Tree Blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively divided into one or more Coding Units (CUs) using a quadtree. For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In the example, each CU is analyzed to determine the prediction type used for the CU, such as inter-frame prediction or intra-frame prediction. Based on temporal and / or spatial predictability, the CU is divided into one or more prediction units (PUs). Typically, each PU includes a luminance prediction block (PB) and two chrominance PBs. In implementations, prediction operations in decoding (encoding / decoding) are performed on a per-prediction-block basis. Using a luminance prediction block as an example, a prediction block comprises a matrix of pixel values ​​(e.g., luminance values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0100] Figure 7 Block diagrams of some examples of a G-PCC encoder (700) are shown. The G-PCC encoder (700) can be configured to receive point cloud data and compress the point cloud data to generate a bitstream carrying the compressed point cloud data. In an implementation, the G-PCC encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a Level of Detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic encoding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing the reconstructed attribute values.

[0101] As shown, an input point cloud (701) can be received at a G-PCC encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. A duplicate point removal module (712) is configured to receive the quantized positions and perform filtering to identify and remove duplicate points. An octree encoding module (730) is configured to receive the filtered positions from the duplicate point removal module (712) and perform octree-based encoding to generate a sequence of occupancy codes describing a 3D raster of voxels. The occupancy codes are provided to an arithmetic encoding module (770).

[0102] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform attribute transfer processing to determine the attribute value for each voxel when multiple attribute values ​​are associated with corresponding voxels. Attribute transfer processing can be performed on the rearranged points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the rearranged points output from the octree encoding module (730) and reorganize these points into different LODs. LOD information is provided to the attribute prediction module (750).

[0103] The attribute prediction module (750) processes points according to the LOD-based order indicated by LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of a set of neighboring points stored in memory (790). Subsequently, a prediction residual can be obtained based on the original attribute values ​​received from the attribute transfer module (720) and the locally generated attribute prediction. When a candidate index is used in the corresponding attribute prediction process, the index corresponding to the selected prediction candidate can be provided to the arithmetic coding module (770).

[0104] The residual quantization module (760) is configured to receive the prediction residuals from the attribute prediction module (750) and perform quantization to generate quantized residuals. The quantized residuals are then provided to the arithmetic coding module (770).

[0105] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate the reconstructed prediction residual by performing the inverse operation of the quantization operation performed at the residual quantization module (760). The addition module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and the corresponding attribute prediction from the attribute prediction module (750). By combining the reconstructed prediction residual and the attribute prediction, a reconstructed attribute value is generated and stored in the memory (790).

[0106] The arithmetic coding module (770) is configured to receive occupancy codes, candidate indices (if used), quantized residuals (if generated), and other information, and to perform entropy coding to further compress the received values ​​or information. As a result, a compressed bitstream (702) carrying the compressed information can be generated. The bitstream (702) can be transmitted to a decoder that decodes the compressed bitstream, or otherwise provided to a decoder that decodes the compressed bitstream, or stored in a storage device.

[0107] Figure 8 A block diagram of a G-PCC decoder (800) according to an embodiment is shown. The G-PCC decoder (800) can be configured to receive a compressed bitstream and perform point cloud data decompression to decompress the bitstream to generate decoded point cloud data. In an embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing the reconstructed attribute values.

[0108] As shown, a compressed bitstream (801) can be received at the arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain the occupancy code of the point cloud and the quantized residual (if generated). The octree decoding module (830) is configured to determine the reconstructed position of the points in the point cloud based on the occupancy code. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine the LOD-based order. The inverse residual quantization module (820) is configured to generate the reconstructed residual based on the quantized residual received from the arithmetic decoding module (810).

[0109] The attribute prediction module (850) is configured to perform attribute prediction processing according to LOD-based order to determine the attribute prediction of a point. For example, the attribute prediction of the current point can be determined based on the reconstructed attribute values ​​of the current point's neighboring points stored in memory (860). In some examples, the attribute prediction can be combined with the corresponding reconstruction residuals to generate the reconstructed attributes of the current point.

[0110] In one example, the sequence of reconstructed attributes generated from the attribute prediction module (850), along with the reconstructed locations generated from the octree decoding module (830), corresponds to the decoded point cloud (802) output from the G-PCC decoder (800). Additionally, the reconstructed attributes are stored in memory (860) and can be subsequently used to derive attribute predictions for subsequent points.

[0111] In various implementations, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented in hardware, software, or a combination thereof. For example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented using processing circuitry such as one or more integrated circuits (ICs) operating with or without software, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. In another example, the encoder (300), decoder (400), encoder (700), and / or decoder (800) can be implemented as software or firmware comprising instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. These instructions, when executed by processing circuitry such as one or more processors, cause the processing circuitry to perform the functions of the encoder (300), decoder (400), encoder (700), and / or decoder (800).

[0112] Note that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in modules that can have the same characteristics as those described herein. Figure 7 and Figure 8 The decoder or encoder shown may be in other decoders or encoders with similar or different structures. Additionally, in various examples, the encoder (700) and decoder (800) may be included in the same device or separate devices.

[0113] According to some aspects of this disclosure, mesh compression can use encoding tools different from PCC encoding tools, or it can use PCC decoding tools such as the PCC (e.g., G-PCC, V-PCC) encoders and decoders mentioned above.

[0114] An object's mesh (also known as a mesh model or mesh frame) can include polygons describing the object's surfaces. Each polygon can be defined by its vertices in 3D space and information about how these vertices are connected to form the polygon. This information about how vertices are connected is called connectivity information. In some examples, the mesh can also include attributes associated with the vertices, such as color and normals. These attributes can be associated with the mesh's surfaces using mapping information parameterized by a 2D attribute map. This mapping information is typically described by a set of parametric coordinates associated with the mesh vertices, called UV coordinates or texture coordinates. A 2D attribute map (in some examples, a texture map) is used to store high-resolution attribute information, such as texture, normals, and displacement. Such information can be used for various purposes, such as texture mapping and shading.

[0115] In some implementations, the mesh may include components referred to as geometric information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, geometric information is described by a set of 3D positions associated with the vertices of the mesh. In examples, (x,y,z) coordinates may be used to describe the 3D positions of the vertices and are also referred to as 3D coordinates. In some examples, connectivity information includes a set of vertex indices describing how vertices are connected to create a 3D surface. In some examples, mapping information describes how the mesh surface is mapped to a 2D region of a plane. In examples, mapping information is described by a set of UV parameter / texture coordinates (u,v) associated with mesh vertices and connectivity information. In some examples, vertex attributes include scalar or vector attribute values ​​associated with mesh vertices. In some examples, the attribute map includes attributes associated with the mesh surface and stored as a 2D image / video. In examples, the mapping between the video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.

[0116] According to aspects of the invention, techniques known as UV mapping or mesh parameterization are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is divided into patches in the 3D domain. A patch is a contiguous subset of a mesh having boundaries formed by boundary edges. A boundary edge of a patch is an edge belonging to only one polygon of the patch and not shared by two adjacent polygons in the patch. In some examples, the vertices of the boundary edges in a patch are called boundary vertices of the patch, and non-boundary vertices in a patch may be called interior vertices of the patch.

[0117] In some examples, the mesh of an object is formed by connected triangles, and the mesh can be divided into patches, each patch being a subset of connected triangles. A patch's boundary edge is an edge that belongs to only one triangle within the patch and is not shared by adjacent triangles within the patch. In some examples, the vertices of the boundary edges in a patch are called the patch's boundary vertices, and non-boundary vertices in a patch can be called the patch's interior vertices.

[0118] According to aspects of this disclosure, in some examples, patches are parameterized as 2D shapes (also referred to as UV patches). In some examples, the 2D shapes can be packaged (e.g., oriented and placed) into a mapping graph, also referred to as an atlas. In some examples, 2D image or video processing techniques can be used to further process the mapping graph.

[0119] In the example, UV mapping generates one or more 2D texture atlases (also called texture maps) and UV atlases (also called UV maps) corresponding to patches of a 3D mesh. A UV atlas includes the assignment of 3D vertices of the 3D mesh to 2D points in a 2D domain (e.g., a rectangle). A UV atlas is a mapping between coordinates of a 3D surface and coordinates of a 2D domain. In the example, a point in the UV atlas at 2D coordinates (u,v) has a value formed by the coordinates (x,y,z) of the vertices in the 3D domain. In the example, the texture atlas includes color information for the 3D mesh. For example, a point in the texture atlas at 2D coordinates (u,v) (which has 3D values ​​(x,y,z) in the UV atlas) has a color that specifies the color attribute of the point at (x,y,z) in the 3D domain. In some examples, the coordinates (x,y,z) in the 3D domain are referred to as 3D coordinates or xyz coordinates, and the 2D coordinates (u,v) are referred to as uv coordinates.

[0120] According to some aspects of this disclosure, mesh compression can be performed by representing a mesh using one or more 2D maps (also referred to as 2D atlases in some examples) and then encoding the 2D maps using an image or video codec.

[0121] Figure 9 A diagram illustrating the mapping from a 3D mesh (910) to a 2D atlas (920) in some examples is shown. Figure 9In the example, the 3D mesh (910) includes four vertices 1 to 4 forming four patches A to D. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 connected to form a triangle; patch B is formed by vertices 1, 3, and 4 connected to form a triangle; patch C is formed by vertices 1, 2, and 4 connected to form a triangle; and patch D is formed by vertices 2, 3, and 4 connected to form a triangle. In some examples, vertices 1, 2, 3, and 4 may have corresponding attributes, and the triangle formed by vertices 1, 2, 3, and 4 may have corresponding attributes.

[0122] In the example, faces A, B, C, and D in 3D are mapped to a 2D domain, such as a 2D atlas (920) also referred to as a mapping graph (920). For example, face A is mapped to a 2D shape (also referred to as a UV face) A' in the mapping graph (920), face B is mapped to a 2D shape (also referred to as a UV face) B' in the mapping graph (920), face C is mapped to a 2D shape (also referred to as a UV face) C' in the mapping graph (920), and face D is mapped to a 2D shape (also referred to as a UV face) D' in the mapping graph (920). In some examples, coordinates in the 3D domain are referred to as (x,y,z) coordinates, while coordinates in the 2D domain such as the mapping graph (920) are referred to as UV coordinates. Vertices in the 3D mesh can have corresponding UV coordinates in the mapping graph (920).

[0123] The mapping map (920) can be a geometric mapping map with geometric information, or a texture mapping map with color, fabric or other attribute information, or an occupancy mapping map with occupancy information.

[0124] Although Figure 9 In the examples, each facet is represented by a triangle; however, note that a facet can include any suitable number of vertices connected to form a contiguous subset of the mesh. In some examples, the vertices in a facet are connected to form triangles. Note that other suitable shapes can be used to connect the vertices in a facet.

[0125] In the example, the geometric information of the vertices can be stored in a 2D geometry map. For example, the geometry map stores the (x, y, z) coordinates of a sample point at a corresponding point in the 2D geometry map. For example, a point at position (u, v) in the geometry map has vector values ​​for three components, corresponding to the x, y, and z values ​​of the corresponding sample point in the 3D mesh.

[0126] According to aspects of this disclosure, areas in the mapping diagram may not be fully occupied. For example, in Figure 9In the code, regions outside of the 2D shapes A', B', C', and D' are undefined. Sample values ​​in regions outside of these 2D shapes can be discarded after decoding. In some cases, occupancy maps are used to store additional information for each pixel, such as binary values ​​used to identify whether a pixel belongs to a patch or is undefined.

[0127] According to aspects of this disclosure, a dynamic mesh is a mesh in which at least one of its components (geometric information, connectivity information, mapping information, vertex attributes, and attribute mapping graphs) changes over time. A dynamic mesh can be described by a sequence of meshes (also known as mesh frames). Dynamic meshes may require large amounts of data because they can include a significant amount of information that changes over time. Mesh compression techniques can allow for the efficient storage and transmission of media content within a mesh representation.

[0128] In some examples, dynamic meshes can possess constant connectivity information, time-varying geometry, and time-varying vertex properties. In other examples, dynamic meshes can also possess time-varying connectivity information. In these examples, digital content creation tools typically generate dynamic meshes with time-varying property maps and time-varying connectivity information. In some examples, volumetric acquisition techniques are used to generate dynamic meshes. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.

[0129] Several techniques are used for mesh compression. In some examples, UV atlas sampling and V-PCC can be used for mesh compression. For example, UV atlases are sampled on a regular raster to generate a geometric image with regular raster samples. The connectivity of the regular raster samples can be inferred. Regular raster samples can be treated as points in a point cloud and therefore can be encoded using a V-PCC codec, such as PCC encoding. Methods based on UV atlas sampling and V-PCC are based on sampling the original mesh vertices, and the number of regular raster samples generated by the sampling is usually much larger than the number of original vertices. Moreover, when using a constant sampling step, regions with dense vertices may be undersampled, while regions with sparse vertices may be oversampled. Furthermore, with UV atlas sampling methods, the generated geometric images may be inconsistent between frames because patches in the UV atlas may be arranged differently in different frames.

[0130] Figure 10 Two UV atlas frames (1010) and (1020) are shown as examples. In these examples, UV atlas frames (1010) and (1020) are consecutive frames in a sequence. Figure 10As shown, patches are not uniformly assigned across different UV atlas frames. For example, patch (1011) in UV atlas frame (1010) and patch (1021) in UV atlas frame (1020) may correspond to the same part of an object at different times, but are assigned to different regions in UV atlas frames (1010) and (1020) and are oriented differently.

[0131] UV atlas-based sampling methods can introduce problems such as undersampling, oversampling, compression of more vertices, and inconsistent sampling data between frames. These issues can negatively impact the compression ratio.

[0132] This disclosure provides vertex rearrangement techniques for mesh compression. The vertices of the mesh are rearranged according to specific rules to increase the correlation between adjacent vertices, thus allowing predictive coding methods to be used to increase coding efficiency. In one example, vertices are rearranged to increase the correlation between adjacent vertices in a sequence. In another example, the vertices of the mesh are rearranged and rearranged within a 2D frame to increase the correlation between adjacent vertices in local regions of the 2D frame, thus allowing the 2D frame to be encoded more efficiently by an image / video codec, for example, using intra-frame prediction techniques. In yet another example, the vertices of different mesh frames are rearranged and rearranged within a 2D frame to increase the correlation between 2D frames, thus allowing the 2D frame to be encoded more efficiently by a video codec, for example, using inter-frame prediction techniques.

[0133] Figure 11 A diagram of a frame (1100) for mesh compression according to some embodiments of the present disclosure is shown. The frame (1100) includes a mesh encoder (1110) and a mesh decoder (1150). The mesh encoder (1110) encodes an input mesh (1105) (mesh frames in the case of dynamic mesh processing) into a bitstream (1145), and the mesh decoder (1150) decodes the bitstream (1145) to generate a reconstructed mesh (1195) (reconstructed mesh frames in the case of dynamic mesh processing).

[0134] The mesh encoder (1110) can be any suitable device, such as a computer, server computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The mesh decoder (1150) can also be any suitable device, such as a computer, client computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The bitstream (1145) can be sent from the mesh encoder (1110) to the mesh decoder (1150) via any suitable communication network (not shown).

[0135] exist Figure 11 In the example, the mesh encoder (1110) includes a vertex rearrangement module (1120), a 1D / 2D encoder (1130), and an auxiliary data encoder (1140) coupled together. The vertex rearrangement module (1120) receives an input mesh (1105) and performs a rearrangement to generate mesh information (1125) of rearranged vertices. In some examples, the input mesh (1105) includes 3D position information of vertices of the original mesh in sequence, vertex connectivity information, mapping information that parameterizes the mesh to 2D (e.g., UV atlas), and other 2D attribute maps (e.g., 2D color maps). The vertex rearrangement module (1120) can rearrange and / or rearrange vertices to increase the attribute correlation of adjacent vertices (e.g., attribute correlation of adjacent vertices in a sequence (e.g., a 1D array), attribute correlation of adjacent vertices in a region in a 2D array, attribute correlation of adjacent frames).

[0136] The vertex rearrangement module (1120) outputs mesh information (1125) of the rearranged vertices. In some examples, the mesh information (1125) of the rearranged vertices does not include connectivity information. In the example, the mesh information (1125) of the rearranged vertices includes vertex information of the vertices in the rearranged sequence. For example, the vertex information of each vertex may include various attributes such as 3D spatial information in the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), etc.

[0137] In another example, the mesh information (1125) of the rearranged vertices includes vertex information of the vertices in the form of a 2D image. For example, the rearranged vertices are rearranged into 2D samples (pixels) (e.g., based on raster scan lines, etc.), and the mesh information of the rearranged vertices can form one or more 2D images.

[0138] A 1D / 2D encoder (1130) is configured to encode the mesh information (1125) of rearranged vertices into a bitstream (1145). When the mesh information (1125) of rearranged vertices includes vertex information of vertices in a rearranged sequence (1D), the 1D / 2D encoder (1130) can encode the vertex information of vertices in the rearranged sequence using 1D encoding techniques. When the mesh information (1125) of rearranged vertices includes a 2D image, the 1D / 2D encoder (1130) can encode the 2D image using image encoding and / or video encoding techniques (e.g., using an image codec or a video codec).

[0139] The vertex rearrangement module (1120) also generates auxiliary data (1127) including auxiliary information. The auxiliary data encoder (1140) receives the auxiliary data (1127) and encodes it into a bitstream (1145). For example, the vertex rearrangement module (1120) can rearrange vertices face-by-face. The vertex rearrangement module (1120) can provide a value in the auxiliary data (1127) indicating the number of vertices in each face. Furthermore, in this example, the vertex rearrangement module (1120) can rearrange the boundary vertices preceding the non-boundary vertices for each face. The vertex rearrangement module (1120) can provide a value in the auxiliary data (1127) indicating the number of boundary vertices in each face.

[0140] In some examples, the auxiliary data encoder (1140) can receive reconstructed information (1131) from the 1D / 2D encoder (1130). In the example, lossy compression is used in the 1D / 2D encoder (1130). The 1D / 2D encoder (1130) can provide reconstructed vertex information, such as reconstructed boundary coordinates, in the reconstructed information (1131). In the example, the auxiliary data encoder (1140) can determine the boundary coordinate residual based on the original boundary coordinates and the reconstructed boundary coordinates, and encode the boundary coordinate residual into a bitstream (1145).

[0141] The operation of the vertex rearrangement module (1120), the 1D / 2D encoder (1130), and the auxiliary data encoder (1140) will be further described in this disclosure.

[0142] exist Figure 11 In the example, the bitstream (1145) is provided to the grid decoder (1150). The grid decoder (1150) includes, for example, Figure 11 The diagram shows a 1D / 2D decoder (1160), a connectivity determination module (1180), an auxiliary data decoder (1170), and a mesh reconstruction module (1190) coupled together. In this example, the 1D / 2D decoder (1160) corresponds to the 1D / 2D encoder (1130) and can decode the portion of the bitstream (1145) encoded by the 1D / 2D encoder (1130) to generate decoded information (1165).

[0143] exist Figure 11 In the example, the auxiliary data decoder (1170) corresponds to the auxiliary data encoder (1140) and can decode the portion of the bitstream (1145) encoded by the auxiliary data encoder (1140) to generate decoded auxiliary data (1175).

[0144] exist Figure 11In the example, the connectivity determination module (1180) can determine (infer) the connectivity information (1185) based on the decoded information (1165).

[0145] exist Figure 11 In the example, the decoded information (1165), the decoded auxiliary data (1175), and the connectivity information (1185) are provided to the mesh reconstruction module (1190). The mesh reconstruction module (1190) generates a reconstructed mesh (1195) based on the decoded information (1165), the decoded auxiliary data (1175), and the connectivity information (1185). The operation of the 1D / 2D decoder (1160), the auxiliary data decoder (1170), and the connectivity determination module (1180) will be further described in this disclosure.

[0146] Note that components in the mesh encoder (1110), such as the vertex rearrangement module (1120), the auxiliary data encoder (1140), and the 1D / 2D encoder (1130), can be implemented using various techniques. In this example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0147] Note that components in the mesh decoder (1150), such as the 1D / 2D decoder (1160), auxiliary data decoder (1170), connectivity determination module (1180), and mesh reconstruction module (1190), can be implemented using various techniques. In this example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0148] Figure 12 A diagram of a frame (1200) for mesh compression according to some embodiments of the present disclosure is shown. The frame (1200) includes a mesh encoder (1210) and a mesh decoder (1250). The mesh encoder (1210) encodes an input mesh (1205) (mesh frames in the case of dynamic mesh processing) into a bitstream (1245), and the mesh decoder (1250) decodes the bitstream (1245) to generate a reconstructed mesh (1295) (reconstructed mesh frames in the case of dynamic mesh processing).

[0149] The mesh encoder (1210) can be any suitable device, such as a computer, server computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The mesh decoder (1250) can also be any suitable device, such as a computer, client computer, desktop computer, laptop computer, tablet computer, smartphone, gaming device, AR device, VR device, etc. The bitstream (1245) can be sent from the mesh encoder (1210) to the mesh decoder (1250) via any suitable communication network (not shown).

[0150] exist Figure 12 In the example, the mesh encoder (1210) includes a vertex rearrangement module (1220), a 1D / 2D encoder (1230), an auxiliary data encoder (1240), and a connectivity encoder (1235) coupled together. The vertex rearrangement module (1220) operates similarly to the vertex arrangement module (1120). The vertex rearrangement module (1220) receives an input mesh (1205) and performs a rearrangement to generate mesh information (1225) of rearranged vertices. In some examples, the input mesh (1205) includes 3D position information of vertices of the original ordered mesh, vertex connectivity information, mapping information that parameterizes the mesh to 2D (e.g., UV atlas), and a 2D attribute map. The vertex rearrangement module (1220) can rearrange and / or rearrange vertices to increase the correlation of adjacent vertices (e.g., sequential adjacency, spatial adjacency, temporal adjacency).

[0151] The vertex rearrangement module (1220) outputs mesh information (1225) of the rearranged vertices. In some examples, the mesh information (1225) of the rearranged vertices does not include connectivity information. In the example, the mesh information (1225) of the rearranged vertices includes vertex information of the vertices in the rearranged sequence. For example, the vertex information of each vertex may include various attributes such as 3D spatial information in the mesh (e.g., xyz coordinates), mapping information to 2D (e.g., uv coordinates), color information (e.g., RGB values), etc.

[0152] In another example, the mesh information (1225) of the rearranged vertices includes vertex information of the vertices in the form of a 2D image. For example, the rearranged vertices are rearranged into 2D samples (pixels), and the mesh information of the rearranged vertices can form one or more 2D images.

[0153] A 1D / 2D encoder (1230) is configured to encode the mesh information (1225) of rearranged vertices into a bitstream (1245). When the mesh information (1225) of rearranged vertices includes vertex information of vertices in a rearranged sequence (1D), the 1D / 2D encoder (1230) can encode the vertex information of vertices in the rearranged sequence using 1D encoding techniques. When the mesh information (1225) of rearranged vertices includes a 2D image, the 1D / 2D encoder (1230) can encode the 2D image using image encoding and / or video encoding techniques (e.g., using an image codec or a video codec).

[0154] In addition, Figure 12 In the example, the connectivity encoder (1235) can encode connectivity information in the input grid (1205) into a bitstream (1245). The connectivity encoder (1235) can use any suitable technique to encode the connectivity information.

[0155] The vertex rearrangement module (1220) also generates auxiliary data (1227) including auxiliary information. The auxiliary data encoder (1240) receives the auxiliary data (1227) and encodes it into a bitstream (1245). For example, the vertex rearrangement module (1220) can rearrange vertices face-by-face. The vertex rearrangement module (1220) can provide a value in the auxiliary data (1227) indicating the number of vertices in each face. Furthermore, in this example, the vertex rearrangement module (1220) can rearrange the boundary vertices preceding the non-boundary vertices for each face. The vertex rearrangement module (1220) can provide a value in the auxiliary data (1227) indicating the number of boundary vertices in each face.

[0156] In some examples, the auxiliary data encoder (1240) can receive reconstructed information (1231) from the 1D / 2D encoder (1230). In the example, lossy compression is used in the 1D / 2D encoder (1230). The 1D / 2D encoder (1230) can provide reconstructed vertex information, such as reconstructed boundary coordinates, in the reconstructed information (1231). In the example, the auxiliary data encoder (1240) can determine the boundary coordinate residual based on the original boundary coordinates and the reconstructed boundary coordinates, and encode the boundary coordinate residual into a bitstream (1245).

[0157] The operation of the vertex rearrangement module (1220), the auxiliary data encoder (1240), and the 1D / 2D encoder (1230) is similar to that of the vertex rearrangement module (1120), the auxiliary data encoder (1140), and the 1D / 2D encoder (1130), and will be further described in this disclosure.

[0158] exist Figure 12 In the example, the bitstream (1245) is provided to the grid decoder (1250). The grid decoder (1250) includes, for example, Figure 12 The diagram shows a 1D / 2D decoder (1260), an auxiliary data decoder (1270), a connectivity decoder (1280), and a mesh reconstruction module (1290) coupled together. In this example, the 1D / 2D decoder (1260) corresponds to the 1D / 2D encoder (1230) and can decode the portion of the bitstream (1245) encoded by the 1D / 2D encoder (1230) to generate decoded information (1265).

[0159] exist Figure 12 In the example, the auxiliary data decoder (1270) corresponds to the auxiliary data encoder (1240) and can decode the portion of the bitstream (1245) encoded by the auxiliary data encoder (1240) to generate decoded auxiliary data (1275).

[0160] exist Figure 12 In the example, the connectivity decoder (1280) corresponds to the connectivity encoder (1235) and can decode the portion of the bitstream (1245) encoded by the connectivity encoder (1235) to generate the decoded connectivity (1285).

[0161] exist Figure 12 In the example, the decoded information (1265), the decoded auxiliary data (1275), and the decoded connectivity (1285) are provided to the mesh reconstruction module (1290). The mesh reconstruction module (1290) generates a reconstructed mesh (1295) based on the decoded information (1265), the decoded auxiliary data (1275), and the decoded connectivity (1285). The operation of the 1D / 2D decoder (1260) and the connectivity decoder (1280) will be further described in this disclosure.

[0162] Note that components in the mesh encoder (1210), such as the vertex rearrangement module (1220), auxiliary data encoder (1240), 1D / 2D encoder (1230), and connectivity encoder (1235), can be implemented using various techniques. In this example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0163] Note that components in the mesh decoder (1250), such as the 1D / 2D decoder (1260), auxiliary data decoder (1270), connectivity decoder (1280), and mesh reconstruction module (1290), can be implemented using various techniques. In this example, the components are implemented using integrated circuits. In another example, the components are implemented using software that can be executed by one or more processors.

[0164] Based on some aspects of this disclosure, the vertices of a mesh can be rearranged face-by-face based on the spatial characteristics of the vertices in the mesh.

[0165] Figure 13 A flowchart outlining a process (1300) for rearranging vertices in a mesh (a mesh frame for dynamic mesh processing) according to some embodiments of this disclosure is shown. In some examples, the process (1300) is performed by a vertex rearrangement module such as a vertex rearrangement module (1120), a vertex rearrangement module (1220), etc. In various embodiments, the process (1300) is performed by processing circuitry. In some embodiments, the process (1300) is implemented as software instructions, so that the processing circuitry executes the process (1300) when the software instructions are executed. The process begins at (S1301) and proceeds to (S1310).

[0166] At (S1310), the starting vertex in the mesh is determined, and the current patch including the starting vertex is determined. In some examples, the mesh includes multiple vertices, and these vertices may be connected by edges. In some examples, the mesh includes triangles formed by edges. If two triangles share an edge, then the two triangles are connected. The mesh is divided into patches in the 3D domain. A patch is a contiguous subset of a mesh with connected triangles. In the examples, when two adjacent triangles of a patch do not share an edge in the patch, for example, if an edge belongs to only one triangle of the patch, then the edge is a boundary edge of the patch, and the vertex of the boundary edge is a boundary vertex. Vertices in a patch can be classified as boundary vertices and non-boundary vertices.

[0167] In some implementations, vertex attributes with specific spatial characteristics are used to select the starting vertex, such that for each frame in a group of mesh frames in a dynamic mesh, vertex traversal begins at almost the same location. In the example, the group of mesh frames has a concept similar to a group of pictures (GOP) in 2D.

[0168] In some examples, the starting vertex is selected based on its coordinates in 3D. In one example, the vertex with the largest vertical coordinate (e.g., the largest z-coordinate) is chosen as the starting vertex, making it the tallest vertex. Note that other properties such as Gaussian curvature, local vertex density, color information, etc., can be used to select the starting vertex.

[0169] After selecting the starting vertex, the corresponding face including the starting vertex is determined as the first face to be processed. Therefore, the first face is the current face to be processed, and the starting vertex is also the starting vertex of the current face.

[0170] At (S1320), traverse the vertices of the current face starting from the starting vertex. Various techniques can be used to traverse the vertices of the current face. In one example, a breadth-first search can be performed for traversal. In another example, a depth-first search can be performed for traversal. In yet another example, the boundary vertices of the current face are traversed first, followed by the interior vertices (also called non-boundary vertices). Therefore, boundary vertices are better grouped together. In some examples, while lossy compression can be used to encode vertex attributes, additional processing can be applied to boundary vertices, thus allowing compression of their coordinates without information loss (e.g., lossless compression).

[0171] At (S1330), if there are unvisited faces (vertex traversal has not yet been performed), the process proceeds to (S1340); otherwise, all faces have been processed (traversed), and the process proceeds to (S1399) and terminates.

[0172] At (S1340), the next face to be traversed after the current face (e.g., an old current face whose vertices have been traversed) is determined, and the determined next face is set as the current face to be processed (e.g., a new current face to be processed for further traversal). For example, after traversing all vertices in the first face, the next face to be traversed after the first face is determined. Various techniques can be used to determine the next face. In the example, the next face is selected as the face that includes the highest vertex among the unvisited vertices (thresholds that have not yet been traversed). For example, when the first face is selected as the face that contains the highest vertex, the next face of the first face can be determined as the face that contains the vertex with the largest vertical coordinate among the unvisited vertices.

[0173] In some examples, the face closest to the current face (e.g., with the smallest distance) is selected as the next face to be traversed. The distance between a previous face (e.g., a previous face with traversed vertices) and another face can be defined by various techniques. In the example, for an unvisited face, the 3D Euclidean distance is calculated between each of the unvisited vertices and the last visited vertex in the current face (e.g., a previous face with traversed vertices). Then, among the 3D Euclidean distances, the minimum 3D Euclidean distance is determined as the distance between the current face (the previous face) and the unvisited face.

[0174] In another example, the average 3D coordinates (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in the unvisited face are calculated. Then, the 3D Euclidean distance between the point with the average 3D coordinates of the unvisited face and the last visited vertex in the current face (e.g., the old current face) is calculated and defined as the distance between the current face (the old current face) and the unvisited face.

[0175] In another example, the first average 3D coordinates (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in the current face (e.g., the old current face) are calculated, and the second average 3D coordinates (average x-coordinate, average y-coordinate, average z-coordinate) of all vertices in the unvisited face are calculated. Then, the distance between a first point with the first average 3D coordinates and a second point with the second average 3D coordinates is calculated and defined as the distance between the current face and the unvisited face.

[0176] At (S1350), the starting vertex in the current face (e.g., the new current face) to be traversed is determined. Various techniques can be used to select the starting vertex in the current face (e.g., the new current face). In one example, the highest vertex (e.g., the one with the largest z-coordinate) within the current face (e.g., the new current face) is selected as the starting vertex. In another example, the vertex in the current face (e.g., the new current face) that is closest to the last visited vertex (in the old current face) is selected as the starting vertex. In yet another example, the highest boundary vertex in the current face (e.g., the new current face) is selected as the starting vertex, and the boundary vertices in the current face (e.g., the new current face) are traversed before the interior vertices (e.g., non-boundary vertices) in the current face (e.g., the new current face). In another example, the boundary vertex that is closest to the last visited vertex (in the old current face) (in the new current face) is selected as the starting vertex, and the boundary vertices in the current face (e.g., the new current face) are traversed before the interior vertices (e.g., non-boundary vertices) in the current face (e.g., the new current face). Then, the process (S1300) returns to (S1320).

[0177] According to one aspect of this disclosure, after traversing the vertices in the mesh, for example according to the process (1300), the vertex attribute information of the vertices in the mesh, such as xyz coordinates (also known as 3D coordinates), uv coordinates, RGB color, normals, etc., is recorded in, for example, rearranged mesh information (1125) or rearranged mesh information (1225) according to the traversal order. In some examples, after visiting each face, the number of vertices in the face and the number of boundary vertices in the face are recorded in, for example, auxiliary data (1127), auxiliary data (1227), etc.

[0178] In some examples, the rearranged mesh information (1125) or rearranged mesh information (1225) includes an array of vertex attributes, such as an array of xyz coordinates, an array of uv coordinates, and an array of other attributes (e.g., normals, colors, etc.). In the example, the array of vertex attributes is a 1D array in the traversal order of the vertices. The 1D array can be encoded by a 1D / 2D encoder (1130) or a 1D / 2D encoder (1230).

[0179] In another example, vertices arranged in traversal order can be rearranged into a 2D form, for example, according to a raster scan line pattern. The vertex attributes can then be rearranged into a 2D image based on the rearranged 2D form. In some examples involving multiple mesh frames (e.g., dynamic mesh compression), after accessing mesh frames or groups of mesh frames in a sequence, the number of vertices in these frames (in the sequence or group) can be determined. The maximum number of vertices is then determined based on the number of vertices in the mesh frame. Based on the maximum number of vertices, the width (denoted by m) and height (denoted by n) of the 2D image can be determined. In the example, the width is the number of pixels in the horizontal direction, and the height is the number of pixels in the vertical direction, and the product of the width and height is equal to or greater than the maximum number of vertices in the mesh frame group. For each mesh frame, when the number of vertices in the mesh frame is less than the product of the width (m) and height (n), padding vertices (with padding attribute values) can be added appropriately.

[0180] According to one aspect of this disclosure, the attributes of the vertices in each mesh frame, such as xyz coordinates, uv coordinates, normals, and colors, are reshaped into an m×n 2D image. Therefore, the attributes of the mesh frame, such as xyz coordinates and uv coordinates, are stored in a 2D image of the same size, and the 2D image can be encoded using a 2D video codec. For example, the xyz coordinates of the vertices in the mesh frame can be reshaped into an m×n×3 2D image (3-channel); the uv coordinates of the vertices in the mesh frame can be reshaped into an m×n×2 2D image (2-channel).

[0181] In one example, the xyz coordinates of vertices in a mesh frame can be stored as color components in a 2D image, for example, using an RGB 4:4:4 color format. In another example, the uv coordinates of vertices in a mesh frame can be segmented and stored as two sets of 2D images in one channel, for example, using an RGB 4:0:0 color format. For instance, the u coordinates of vertices in the mesh frame are stored in the first set of 2D images in one channel, and the v coordinates of vertices in the mesh frame are stored in the second set of 2D images in one channel.

[0182] In another example, a transform function (e.g., a 2-tap Haar transform) can be applied to a 2D image with uv coordinates to decorrelate the two channels. The decorrelated 2D image can then be segmented into two sets of 2D images with a single channel. The two sets of 2D images can then be encoded separately.

[0183] In another example, the uv coordinates of vertices in a mesh frame can be stored as color components (three channels) in a 2D image, for example, using an RGB 4:4:4 format. For example, the u coordinates can be stored as the first channel, the v coordinates as the second channel, and the third channel can be filled with arbitrary numbers, and the third channel can be discarded on the decoder side.

[0184] In some examples, lossy compression is used in encoders such as 1D / 2D encoder (1130) and 1D / 2D encoder (1230) to encode the vertex attributes of the mesh. To preserve more information about the boundary vertices of the mesh (e.g., to avoid gaps between patches), the reconstructed coordinates (e.g., xyz coordinates, uv coordinates) of the boundary vertices (e.g., in the reconstructed information (1131), reconstructed information (1231)) can be determined based on the encoded vertex attributes, and the difference between the reconstructed coordinates of the boundary vertices and the original vertex coordinates can be calculated. The difference between the reconstructed coordinates of the boundary vertices and the original vertex coordinates is represented as the boundary coordinate residual. The boundary coordinate residual can be encoded into a bitstream carrying the mesh, such as bitstream (1145), bitstream (1245), etc., for example, by auxiliary data encoder (1140), auxiliary data encoder (1240), etc.

[0185] According to some aspects of this disclosure, the bitstream carrying the 3D mesh (e.g., bitstream (1145), bitstream (1245)) includes vertex attributes such as xyz coordinates, uv coordinates, etc. in the form of a 1D array or 2D image, and includes auxiliary data for assisting in the correct decoding and reconstruction of the 3D mesh.

[0186] In some examples, for each grid frame, attributes such as xyz coordinates, uv coordinates, etc., can be decoded from the bitstream by a 2D video codec, or from other compressed formats, such as using a 1D / 2D decoder (1160), a 1D / 2D decoder (1260), etc. For example, each pixel of a 2D frame output from a 2D video decoder represents a vertex attribute, such as xyz coordinates, uv coordinates, RGB color, or normal. In the examples, a 4:4:4 image format can be used to represent the xyz coordinates of all vertices in the grid frame; and two 1-channel 4:0:0 format images can be used to represent the uv coordinates of all vertices in the grid frame.

[0187] In some examples, the auxiliary data is decoded from the bitstream, for example, by an auxiliary data decoder (1170), an auxiliary data decoder (1270), etc. In one example, the auxiliary data includes the number of vertices in each face and the number of boundary vertices. In another example, the auxiliary data includes the number of vertices in each face and the number of boundary vertices, and includes boundary coordinate residuals.

[0188] In some examples, the decoded vertices of a mesh frame can be grouped face-by-face based on the number of vertices in each facet and the number of boundary vertices. Typically, the encoded vertices in the image are arranged face-by-face or in any order that the decoder can recover. Then, for each mesh frame, mesh reconstruction can be performed face-by-face.

[0189] According to one aspect of this disclosure, for a facet, the boundary vertices in the facet can be identified based on the number of vertices in the facet and the number of boundary vertices. The xyz and uv coordinates of the vertices in the facet can be determined based on decoding information output from, for example, a 1D / 2D decoder (1160), a 1D / 2D decoder (1260), etc. In the example, both the encoder and decoder sides agree to place all boundary vertices in front of all other internal facet vertices (e.g., non-boundary vertices) for each facet. Therefore, the boundary vertices of each facet can be identified among all decoded vertices of the facet.

[0190] In some examples using lossy compression, the boundary coordinate residuals are decoded (e.g., by the auxiliary data decoder (1170), by the auxiliary data decoder (1270)) and added (which is the difference between the original coordinates and the decoded coordinates) to the decoded boundary coordinates, for example by the mesh reconstruction module (1190), mesh reconstruction module (1290), etc., to obtain the original xyz coordinates and uv coordinates of the boundary vertices.

[0191] In some examples, for lossy compression, duplicate vertices with the same xyz or uv coordinates may exist in the decoded information, and these duplicate vertices are removed. Furthermore, in some examples, some decoded vertices corresponding to the interior vertices of a patch (the original interior vertices in the original mesh) may be outside the patch due to lossy compression (e.g., outside the boundary formed by the patch's boundary edges), and decoded vertices outside the patch can be removed.

[0192] In some examples, to obtain connectivity, triangulation operations such as constrained Delaunay triangulation are performed using the uv coordinates of all vertices and boundary vertices of a face to obtain (inferred) the connectivity of the face. In other examples, the connectivity information of each face can be explicitly indicated by a signal in the bitstream, allowing the decoder to directly reconstruct the topology of the mesh without performing any triangulation algorithm.

[0193] Figure 14A flowchart outlining a process (1400) according to an embodiment of this disclosure is shown. The process (1400) can be used during mesh encoding. In various embodiments, the process (1400) is executed by processing circuitry. In some embodiments, the process (1400) is implemented as software instructions, so that the processing circuitry executes the process (1400) when the software instructions are executed. The process begins at (S1401) and proceeds to (S1410).

[0194] At (S1410), the vertex traversal order of the 3D mesh frame is determined. The 3D mesh frame is divided into patches, and the traversal order is rearranged patch by patch. For example, the vertices are arranged into subsets that belong to patches.

[0195] At (S1420), an array of vertex attributes is formed according to the traversal order of the vertices of the 3D mesh frame.

[0196] At (S1430), the attribute array is encoded into a bitstream carrying 3D mesh frames.

[0197] In some implementations, to determine the traversal order, a starting vertex is determined from the vertices of the 3D mesh frame, and a first patch of a first subset of the vertices is determined. The starting vertex is a vertex in the first subset of the vertices. In some examples, the starting vertex is determined based on the spatial characteristics of the 3D mesh frames maintained in the sequence of 3D mesh frames. Therefore, the starting vertex of each 3D mesh frame in the sequence of 3D mesh frames is in almost the same position in each 3D mesh frame. In one example, the starting vertex has the maximum coordinate value. In another example, the starting vertex has the minimum coordinate value. In another example, the starting vertex has the maximum Gaussian curvature. In another example, the starting vertex has the minimum Gaussian curvature. In another example, the starting vertex has the maximum local vertex density. In another example, the starting vertex has the minimum local vertex density. In another example, the starting vertex has the maximum value of color information. In another example, the starting vertex has the minimum value of color information.

[0198] In some examples, the first subset of vertices is traversed starting from the initial vertex of the 3D mesh frame, before traversing vertices outside the first subset. In some examples, the boundary vertices of the first face are traversed before traversing the interior vertices (non-boundary) of the first face.

[0199] In some examples, after traversing the last vertex of the previous face (also known as the old current face), the next face to be traversed (also known as the new current face) is determined based on the spatial relationship between the untraversed face and the previous face. Then, the starting vertex in the next face is determined.

[0200] In some examples, the next patch is the untraversed patch that is closest to the previous patch. In one example, to calculate the distance between the untraversed patch and the previous patch, the 3D Euclidean distance between the last vertex traversed in the previous patch and each vertex in the untraversed patch is calculated, and the minimum of the 3D Euclidean distances is determined as the distance between the untraversed patch and the previous patch. In another example, to calculate the distance between the untraversed patch and the previous patch, the average 3D coordinates of multiple vertices in the untraversed patch are calculated, and the distance between the untraversed patch and the previous patch is determined based on the distance between the point with the average 3D coordinates and the last vertex traversed in the previous patch. In another example, to calculate the distance between an untraversed face and a previous face, the first average 3D coordinates of a first plurality of vertices in the previous face are calculated, the second average 3D coordinates of a plurality of vertices in the untraversed face are calculated, and the distance between the untraversed face and the previous face is calculated based on the distance between a first point with the first average 3D coordinates and a second point with the second average 3D coordinates.

[0201] To determine the starting vertex in the next patch, in one example, the vertex with the largest coordinate value in the next patch is selected as the starting vertex. In another example, the vertex with the smallest coordinate value in the next patch is selected as the starting vertex. In yet another example, the vertex in the next patch that is closest to the last vertex traversed in the previous patch is selected as the starting vertex. In yet another example, the boundary vertex with the largest coordinate value in the next patch is selected as the starting vertex. In yet another example, the boundary vertex in the next patch that is closest to the last vertex traversed in the previous patches is selected as the starting vertex.

[0202] According to one aspect of this disclosure, in order to form an attribute array, in some examples, the maximum number of vertices in a mesh frame is determined after vertex traversal of the sequence of mesh frames. Based on the maximum number of vertices, the 2D image size can be determined. The attributes of the vertices in each of the mesh frames can then be reshaped into a 2D array of the 2D image size.

[0203] To encode the attribute arrays, in one example, a 2D image corresponding to the 3D coordinate array of the vertices is encoded into a bitstream. In another example, a 2D image corresponding to the uv coordinate array of the vertices is encoded into a bitstream. In yet another example, a first 2D image corresponding to a first array of the u coordinates of the vertices and a second 2D image corresponding to a second array of the v coordinates of the vertices are encoded into a bitstream. In some examples, 2D images corresponding to arrays of other vertices' attributes (e.g., color information, normals, etc.) are encoded into a bitstream.

[0204] In some examples, for each face, a first value indicating the number of vertices in the face is encoded into a bitstream. Additionally, in some examples, for each face, a second value indicating the number of boundary vertices in the face is encoded into a bitstream.

[0205] In some examples, the attributes of the boundary vertices in the face are encoded into a bitstream before the attributes of other vertices in the face (e.g., non-boundary vertices).

[0206] In some examples, lossy compression is used. On the encoder side, for a patch, the reconstructed properties of the patch's boundary vertices are determined based on the encoded vertex properties. Then, the boundary coordinate residuals of the patch's boundary vertices are determined based on the reconstructed coordinates and the original coordinates of the boundary vertices. The boundary coordinate residuals are encoded into a bitstream. The coordinates of the boundary vertices can be the 3D coordinates of the boundary vertices and / or the uv coordinates of the boundary vertices.

[0207] In some examples, connectivity information of vertices in each facet is encoded into a bitstream.

[0208] Then, the process proceeds to (S1499) and terminates.

[0209] The process (1400) can be adjusted as appropriate. Steps in the process (1400) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0210] Figure 15 A flowchart outlining a process (1500) according to an embodiment of this disclosure is shown. The process (1500) can be used during the decoding process of the mesh. In various embodiments, the process (1500) is executed by a processing circuitry system. In some embodiments, the process (1500) is implemented as software instructions, so that the processing circuitry executes the process (1500) when the software instructions are executed. The process begins at (S1501) and proceeds to (S1510).

[0211] At (S1510), an attribute array is decoded from the bitstream carrying a 3D mesh frame comprising multiple faces. The attribute array corresponds to the vertices of the 3D mesh frame arranged face by face. For example, the vertices are arranged into subsets belonging to multiple faces respectively.

[0212] At (S1520), the first part of the attribute array corresponding to the first subset of vertices of the 3D mesh frame is determined. The first subset belongs to the first face among a plurality of faces.

[0213] At (S1530), the first connectivity information of the first subset of vertices is determined.

[0214] At (S1540), the first facet of the 3D mesh frame is reconstructed based on the first part of the attribute array and the first connectivity information.

[0215] To decode the attribute arrays, in one example, the 2D image corresponding to the 3D coordinate array of the vertices is decoded. In another example, the 2D image corresponding to the uv coordinate array of the vertices is decoded. In yet another example, a first 2D image corresponding to a first array of vertices' u coordinates and a second 2D image corresponding to a second array of vertices' v coordinates are decoded.

[0216] To determine the first portion of the attribute array, in this example, a first value indicating the number of vertices in a first subset of the vertices of the 3D mesh frame is decoded from the bitstream. The first portion of the attribute array is selected based on the first value. In some examples, a second value indicating the number of boundary vertices in the first subset is decoded from the bitstream. The attributes of the boundary vertices are then determined from the first portion of the attribute array based on the second value.

[0217] In some examples, the properties of the boundary vertices of the first face are ranked before the properties of the other vertices in the first subset of the first face.

[0218] In some examples, lossy compression is used to encode vertex attributes such as vertex coordinates. In one example, the boundary coordinate residuals of the boundary vertices are decoded from the bitstream. The decoded coordinates of the boundary vertices are combined with their boundary coordinate residuals to obtain the recovered coordinates of the boundary vertices. The recovered coordinates of the boundary vertices can be equal to the original coordinates of the boundary vertices on the encoder side. The coordinates of the boundary vertices can be either 3D coordinates or UV coordinates.

[0219] In some examples, due to lossy compression, duplicate vertices (with the same 3D coordinates or the same UV coordinates) may exist depending on the decoding properties. In the example, a first vertex and a second vertex with the same coordinates are detected, and one of the first vertex and the second vertex is removed from the reconstruction of the 3D mesh frame.

[0220] In some examples, due to lossy compression, non-boundary vertices of a patch may be outside the patch's boundary. In the example, non-boundary vertices of the patch that are outside the boundary formed by the patch's boundary vertices are removed from the reconstruction of the 3D mesh frame.

[0221] In some examples, the first connectivity information of the first subset of vertices of the first facet is inferred based on the first subset of vertices of the 3D mesh frame.

[0222] In some examples, the first connectivity information of the first subset of vertices is decoded from the bitstream.

[0223] Then, the process proceeds to (S1599) and terminates.

[0224] The process (1500) can be adjusted as appropriate. Steps in the process (1500) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0225] The techniques disclosed in this disclosure can be used individually or in any combination in any order. Furthermore, each of the techniques (e.g., methods, implementations), encoders, and decoders can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, one or more processors execute a program stored on a non-transitory computer-readable medium.

[0226] The above-described technology can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 16 A computer system (1600) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0227] Computer software can be coded using any suitable machine code or computer language, which can be subjected to mechanisms such as assembly, compilation, and linking to create code including instructions that can be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), or through interpretation, microcode execution, etc.

[0228] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0229] Figure 16 The components shown for the computer system (1600) are exemplary in nature and are not intended to impose any limitation on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should also not be construed as having any dependency or requirement relating to any one or combination of the components shown in the exemplary embodiments of the computer system (1600).

[0230] The computer system (1600) may include certain human-machine interface input devices. Such human-machine interface input devices can respond to input from one or more human users via, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., speech, tapping), visual input (e.g., gestures), and olfactory input (not depicted). The human-machine interface device can also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image capturing devices), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0231] Input human-machine interface devices may include one or more of the following (only one of each is depicted): keyboard (1601), mouse (1602), touchpad (1603), touch screen (1610), data glove (not shown), joystick (1605), microphone (1606), scanner (1607), and camera device (1608).

[0232] The computer system (1600) may also include certain human-machine interface output devices. Such human-machine interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-machine interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen (1610), data gloves (not shown), or joystick (1605), but tactile feedback devices that are not used as input devices may also exist), audio output devices (e.g., speakers (1609), headphones (not depicted)), visual output devices (e.g., screens (1610), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without tactile feedback capability—some of which may be able to output two-dimensional visual output or more than three-dimensional output by means such as stereoscopic image output, virtual reality glasses (not depicted), holographic displays, and smoke generators (not depicted)), and printers (not depicted).

[0233] The computer system (1600) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1620) having media such as CD / DVD (1621), thumb drives (1622), removable hard disk drives or solid-state drives (1623), conventional magnetic media such as magnetic tapes and floppy disks (not depicted), devices based on dedicated ROM / ASIC / PLD such as security dongles (not depicted), etc.

[0234] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transient signals.

[0235] The computer system (1600) may also include interfaces (1654) to one or more communication networks (1655). Networks may be, for example, wireless, wired, or optical. Networks may also be local area, wide area, metropolitan area, vehicular and industrial, real-time, latency-tolerant, etc. Examples of networks include: local area networks, such as Ethernet and wireless LANs; cellular networks, including GSM, 3G, 4G, 5G, LTE, etc.; cable or wireless wide area digital television networks, including cable television, satellite television, and terrestrial broadcast television; and vehicular and industrial networks, including CANbus, etc. Some networks typically require external network interface adapters attached to certain general-purpose data ports or peripheral buses (1649) (e.g., USB ports of the computer system (1600); other networks are typically integrated into the core of the computer system (1600) via system buses attached to systems such as Ethernet interfaces in PC computer systems or cellular network interfaces in smartphone computer systems. Using any of these networks, the computer system (1600) can communicate with other entities. Such communication can be one-way receiving (e.g., broadcast television), one-way transmitting (e.g., to a CANBus device), or two-way (e.g., using a local area digital network or a wide area digital network to other computer systems). Specific protocols and protocol stacks can be used on each of these networks and network interfaces as described above.

[0236] The human-machine interface devices, human-accessible storage devices and network interfaces mentioned above can be attached to the core (1640) of the computer system (1600).

[0237] The core (1640) may include one or more Central Processing Units (CPU) (1641), Graphics Processing Units (GPUs) (1642), Dedicated Programmable Processing Units (DPSUs) in the form of Field Programmable Gate Areas (FPGAs) (1643), hardware accelerators (1644) for certain tasks, graphics adapters (1650), etc. These devices, along with Read-Only Memory (ROM) (1645), Random Access Memory (1646), and internal mass storage devices such as internal non-user-accessible hard disk drives, Solid-State Drives (SSDs), etc. (1647), can be connected via the system bus (1648). In some computer systems, the system bus (1648) can be accessed in the form of one or more physical plugs to allow for expansion by additional CPUs, GPUs, etc. Peripheral devices may be directly attached to the core's system bus (1648) or may be attached to the core's system bus (1648) via a peripheral bus (1649). In the example, the screen (1610) can be connected to the graphics adapter (1650). Peripheral bus architectures include PCI (Peripheral Component Interconnect / Interface), USB (Universal Serial Bus), etc.

[0238] The CPU (1641), GPU (1642), FPGA (1643), and accelerator (1644) can execute certain instructions, which, when combined, constitute the aforementioned computer code. This computer code can be stored in ROM (1645) or RAM (1646). Transient data can also be stored in RAM (1646), while permanent data can be stored, for example, in an internal mass storage device (1647). Fast storage and retrieval of any storage device within the storage system can be achieved using a cache memory, which can be closely associated with one or more CPUs (1641), GPUs (1642), mass storage devices (1647), ROMs (1645), RAMs (1646), etc.

[0239] Computer-readable media may have computer code thereon for performing operations of various computer implementations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type known and available to those skilled in the art of computer software.

[0240] By way of example and not limitation, a computer system (1600) having an architecture, particularly a core (1640), can be functionalized by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as described above, as well as certain storage devices of the core (1640) having non-transitory characteristics, such as a mass storage device (1647) or ROM (1645) within the core. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core (1640). Depending on specific needs, the computer-readable media may include one or more memory devices or chips. The software can cause the core (1640), particularly its processors (including CPUs, GPUs, FPGAs, etc.), to execute specific processes or specific portions of specific processes described herein, including defining data structures stored in RAM (1646) and modifying such data structures according to the processes defined by the software. Alternatively or as an alternative, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., an accelerator (1644)), which may replace or operate with software to perform the specific process or a specific portion of the specific process described herein. Where appropriate, references to software may include logic, and conversely, references to logic may include software. Where appropriate, references to computer-readable media may encompass circuitry (e.g., integrated circuits, ICs) storing software for execution, circuitry implementing logic for execution, or both. This disclosure includes any suitable combination of hardware and software.

[0241] While several exemplary embodiments have been described in this disclosure, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it will be appreciated that those skilled in the art will be able to conceive of numerous systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

Claims

1. A method for mesh decompression, characterized in that, The method includes: Decode an attribute array from a bitstream carrying a three-dimensional (3D) mesh frame comprising multiple facets, the attribute array corresponding to the vertices of the 3D mesh frame, the vertices of the 3D mesh frame being arranged into subsets belonging to the multiple facets respectively; Determine a first portion of the attribute array corresponding to a first subset of the vertices of the 3D mesh frame, the first subset belonging to a first patch; Determine the first connectivity information of the first subset of the vertices; and The first patch of the 3D mesh frame is reconstructed based on the first portion of the attribute array and the first connectivity information; Determining the first portion of the attribute array corresponding to a first subset of the vertices of the 3D mesh frame includes: Decode a first value from the bitstream indicating the number of vertices in a first subset of the vertices of the 3D mesh frame; and select a first portion of the attribute array based on the first value; Decode a second value from the bitstream indicating the number of boundary vertices in the first subset; and determine the attributes of the boundary vertices from a first portion of the attribute array based on the second value.

2. The method according to claim 1, characterized in that, Decoding the attribute array further includes at least one of the following steps: Decode the two-dimensional (2D) image corresponding to the three-dimensional coordinate array of the vertices; Decode the 2D image corresponding to the horizontal and vertical UV coordinate arrays of the vertex; and / or Decode the first 2D image and the second 2D image corresponding to the first array of U coordinates of the vertex and the second array of V coordinates of the vertex, respectively.

3. The method according to claim 1, characterized in that, The method further includes: The attributes of the boundary vertices are selected from the first portion of the attribute array, and the attributes of the boundary vertices are ranked before the attributes of other vertices in the first subset.

4. The method according to claim 1, characterized in that, The attribute includes coordinates, and the method further includes: Decode the boundary coordinate residuals of the boundary vertices; and The coordinates of the boundary vertex are combined with the boundary coordinate residual of the boundary vertex to obtain the restored coordinates of the boundary vertex.

5. The method according to claim 4, characterized in that, The coordinates of the boundary vertex include at least one of the three-dimensional (3D) coordinates of the boundary vertex and / or the UV coordinates of the boundary vertex, and the boundary coordinate residuals of the boundary vertex include at least one of the boundary 3D coordinate residuals and / or the boundary UV coordinate residuals.

6. The method according to claim 4, characterized in that, The method further includes: The first and second vertices were detected to have the same coordinates; and Remove one of the first vertex and the second vertex from the reconstruction.

7. The method according to claim 4, characterized in that, The method further includes: Detecting that the non-boundary vertices of the first patch are outside the boundary formed by the boundary vertices of the first patch; and Reconstruct the first facet of the 3D mesh frame without the non-boundary vertices.

8. The method according to any one of claims 1 to 2, characterized in that, The first connectivity information for determining the first subset of the vertices also includes: Infer the first connectivity information of the first subset of the vertices.

9. The method according to any one of claims 1 to 2, characterized in that, The first connectivity information for determining the first subset of the vertices also includes: Decode the first connectivity information of the first subset of the vertices from the bitstream.

10. A computer device, characterized in that, The computer device includes: One or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 9 as instructed by the computer program code.

11. An apparatus for mesh decompression, characterized in that, The apparatus includes a processing circuit configured to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer program code, which, when executed by at least one processor, performs the method according to any one of claims 1 to 9.

13. A method for grid compression, characterized in that, The method includes: Arrange the vertices of a 3D mesh frame into subsets that belong to multiple faces; An attribute array is generated based on the arranged vertices, and the attribute array corresponds to the vertices of the 3D mesh frame; For the first facet among the plurality of faces, determine the first subset of vertices belonging to the first facet; The first value indicating the number of vertices in the first subset, the second value indicating the number of boundary vertices in the first subset, and the attribute array are encoded into a bitstream. Wherein, the first value is used to select a first part corresponding to the first subset from the attribute array at the decoding end, and the second value is used to determine the attributes of the boundary vertex from the first part at the decoding end.

14. A method for storing a bit stream, characterized in that, Generate a bit stream by performing the method of claim 13; and store the bit stream.

15. A method for transmitting a bit stream, characterized in that, The method of claim 13 is used to generate a bit stream; and the bit stream is transmitted.

16. A computer-readable storage medium storing a computer program / instructions and a bit stream thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 13 to generate the bit stream.

Images