Encoding Patch Temporal Registration for Mesh Compression
Patent Information
- Application Number
- JP2023555501
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-09-23
- Filing Date
- 2022-09-27
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-09-27
Smart Images

Figure 0007589904000001 
Figure 0007589904000002 
Figure 0007589904000003
Abstract
Description
[Technical field]
[0001] [Related Applications] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 264,421, entitled "Encoding of Patch Temporal Alignment for Mesh Compression," filed November 22, 2021, which claims the benefit of priority to U.S. Provisional Patent Application No. 17 / 951,942, entitled "ENCODING OF PATCH TEMPORAL ALIGNMENT FOR MESH COMPRESSION," filed September 23, 2022. The disclosures of the aforementioned applications are hereby incorporated by reference in their entireties.
[0002] [Technical field] This disclosure generally describes embodiments related to mesh coding. [Background technology]
[0003] The background description provided herein is to provide an overview of the context of the present disclosure. The work of the presently named inventors is not admitted explicitly or implicitly as prior art to the present disclosure, to the extent that the work described in this background section, as well as aspects of the description that may not be considered prior art at the time of filing, are not admitted explicitly or implicitly as prior art to the present disclosure.
[0004] Various techniques have been developed to capture and represent the world: objects in the world, environments in the world, and the like in three-dimensional (3D) space. A 3D representation of the world allows for more immersive interaction and communication. In some instances, a point cloud or mesh can be used as the 3D representation of the world. Summary of the Invention
[0005] Aspects of the present disclosure provide a method and apparatus for mesh coding (encoding and / or decoding). In some examples, the apparatus for coding a mesh includes a processing circuit. The processing circuit receives at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance. The first mesh frame and the second mesh frame represent a surface of an object having a polygon. The first mesh frame includes a twelfth dimensional (2D) map having a first patch with three-dimensional (3D) information mapped to the 2D. The second mesh frame includes a second 2D map having a second patch with 3D information mapped to the 2D. The processing circuit identifies that a first patch of one of the first patches is a reference matching patch of a second patch of one of the second patches. Furthermore, the processing circuit determines a first remap transformation parameter for a temporal alignment of the second patch relative to the first patch in the first 2D map, and generates a new second 2D map corresponding to the second 2D map. The new 2D map includes a transformed second patch that is transformed from the second patch according to the first remap transformation parameters.
[0006] To identify the first patch as the reference matching patch relative to the second patch, in some examples, the processing circuitry comprises: determining a first match metric for each second patch relative to the first patch; Select the first patch as the best matching patch from the first patch according to the first match metric. Further, in the example, the processing circuitry determining a second match metric for each of the first patches relative to the second patch; determining that the first patch is a reference matching patch for the second patch in response to the second patch being the best matching patch according to the second match metric;
[0007] To determine the first match metric of each second patch relative to the first patch, in an example, the processing circuitry determines the match metric of the second patch relative to the particular patch of the first patch in response to matched feature points of the second patch and the particular patch. In another example, the processing circuitry determines the match metric of the second patch relative to the particular patch of the first patch in response to a difference between a center of 3D coordinates of the second patch and a center of 3D coordinates of the particular patch. In another example, the processing circuitry determines the match metric of the second patch relative to the particular patch of the first patch in response to a Hausdorff distance in 3D coordinates between the second patch and the particular patch.
[0008] In some examples, to determine a first remapping transformation parameter for the temporal alignment of the second patch relative to the first patch, the processing circuitry determines at least one of a rotation parameter, a reflectance parameter, and a spatial translation parameter.
[0009] In some examples, the processing circuitry encodes the first remap transformation parameters of the second patch into a bitstream that transmits at least the first mesh frame and the second mesh frame. In some examples, the processing circuitry updates the geometry 2D map of the second mesh frame according to the first remap transformation parameters.
[0010] In some embodiments, the processing circuit classifies the second patches into a first class and a second class, where the first class includes a first subset of the second patches that have a corresponding reference matching patch identified in the first patch, and the second class includes a second subset of the second patches that do not have a reference matching patch identified in the first patch.
[0011] In some examples, the processing circuitry sorts the first subset of the second patches by size into a size-ordered first patch sequence and determines remap transformation parameters for each of the first subset of the second patches according to the first patch sequence.
[0012] In some examples, to determine the remap transformation parameters for each of the patches in the first patch sequence, the processing circuitry selects a second patch from the first patch sequence according to a size order. The first patch sequence includes at least a third patch that is larger than the second patch by size, a second remap transformation parameter for the third patch is determined, and the third patch transformed according to the second remap transformation parameter is added to the packed patch list. The processing circuitry determines a first rotation angle for generating a rotated second patch that is most similar to the first patch. Then, the processing circuitry determines a first spatial translation value for moving the rotated second patch to avoid overlap with the transformed patch in the packed patch list. In examples, the processing circuitry generates a transformed second patch of the second patch according to the first rotation angle and the first spatial translation value, and adds the transformed second patch to the packed patch list. In examples, the processing circuitry also determines whether to reflect the second patch based on a similarity comparison between the highest similarity and a second highest similarity of the reflected second patch and the first patch.
[0013] In some examples, the processing circuitry determines to reclassify the second patch into the second class in response to allowing the spatial transform values within the predefined window to be non-overlapping.
[0014] In some examples, the processing circuitry sorts the second patch sequence in size order by a second subset size of the second patches and determines remap transformation parameters for each of the second subset of the second patches according to the second patch sequence.
[0015] For example, the size of the patch is determined based on the number of pixels in the patch including the boundary of the patch. In another example, the size of the patch is determined based on the number of pixels in the patch excluding the boundary of the patch. In another example, the size of the patch is determined based on the area of the smallest rectangular bounding box that comprises the patch.
[0016] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform any one or combination of the methods for mesh coding. [Brief description of the drawings]
[0017] Further features, characteristics, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.
[0018] [Figure 1] 1 illustrates a block diagram of a communication system in accordance with some examples.
[0019] [Diagram 2] 1 shows a block diagram of a streaming system, according to some examples.
[0020] [Diagram 3] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.
[0021] [Figure 4] 1 illustrates a block diagram of a decoder for decoding a compressed bitstream corresponding to a point cloud frame in some examples.
[0022] [Diagram 5] 1 shows a block diagram of a video decoder, according to some examples.
[0023] [Figure 6] 1 shows a block diagram of a video encoder, according to some examples.
[0024] [Figure 7] 1 shows a block diagram of an encoder for encoding point cloud frames in some examples.
[0025] [Figure 8]1 illustrates a block diagram of a decoder for decoding a compressed bitstream carrying point cloud frames in some examples.
[0026] [Figure 9] 1 shows a diagram illustrating mapping of a mesh to an atlas in some examples.
[0027] [Figure 10] 13 shows the texture map at two time instances in the example.
[0028] [Figure 11] 1 shows a flowchart outlining an example process in some examples.
[0029] [Figure 12] 1 shows a flowchart outlining an example process in some examples.
[0030] [Figure 13] FIG. 1 is a schematic diagram of a computer system according to some examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0031] Aspects of the present disclosure provide techniques in the field of three-dimensional (3D) media processing.
[0032] Technological developments in three-dimensional (3D) media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, have facilitated the ubiquitous presence of 3D media content across several platforms and devices. As an example, a baby's first steps can be captured on one continent, and media technology allows grandparents to enjoy an immersive experience watching (and possibly interacting with) the baby on another continent. According to aspects of the present disclosure, to enhance the immersive experience, 3D models have become increasingly sophisticated, and the creation and consumption of 3D models occupies a significant amount of data resources, such as data storage, data transmission resources, etc.
[0033] According to some aspects of the present disclosure, point clouds and meshes can be used as 3D models to represent immersive content.
[0034] A point cloud may generally refer to a collection of points in 3D space with associated attributes such as color, material properties, texture information, intensity attributes, reflectance attributes, motion-related attributes, modality attributes, and various other attributes. A point cloud can be used to reconstruct an object or scene as a composition of such points.
[0035] An object's mesh (also called a mesh model) can contain polygons that describe the object's surface. Each polygon can be defined by its vertices in 3D space and information about how the vertices are connected to the polygon. The information about how the vertices are connected is called connectivity information. In some examples, the mesh can also include attributes such as color, normals, etc. associated with the vertices.
[0036] According to some aspects of the present disclosure, some coding tools for point cloud compression (PCC) can be used for mesh compression. For example, a mesh can be remeshed to generate a new mesh and connectivity information for the new mesh can be estimated. The vertices of the new mesh and the attributes associated with the vertices of the new mesh can be viewed as points in the point cloud and compressed using the PCC codec.
[0037] A point cloud can be used to reconstruct an object or scene as a composition of points. The points can be captured using multiple cameras, depth sensors, or Lidar in various settings, and may consist of thousands to billions of points to realistically represent a scene or object. A patch may generally refer to a contiguous subset of the surface described by a point cloud. In an example, a patch includes points with surface normal vectors that deviate from each other by less than a threshold value.
[0038] PCC can be performed according to various schemes, such as a geometry-based scheme called G-PCC, a video coding-based scheme called V-PCC, etc. According to some aspects of the present disclosure, G-PCC is a pure geometry-based approach that directly encodes 3D geometry and does not share much with video coding, while V-PCC is heavily based on video coding. For example, V-PCC can map points of a 3D cloud to pixels of a 2D grid (image). The V-PCC scheme can utilize a general-purpose video codec for point cloud compression. The PCC codec (encoder / decoder) in the present disclosure can be a G-PCC codec (encoder / decoder) or a V-PCC codec.
[0039] According to aspects of the present disclosure, the V-PCC scheme can compress the geometry, occupancy, and texture of the point cloud as three separate video sequences using an existing video codec. Additional metadata required to interpret the three video sequences is compressed separately. A small portion of the overall bitstream is the metadata, which in the example could be efficiently encoded / decoded using a software implementation. The majority of the information is handled by the video codec.
[0040] FIG. 1 illustrates a block diagram of a communication system (100) in some examples. The communication system (100) includes a plurality of terminal devices that can communicate with each other, for example, via a network (150). For example, the communication system (100) includes a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform unidirectional transmission of point cloud data. For example, the terminal device (110) may compress a point cloud (e.g., points representing structures) captured by a sensor (105) connected to the terminal device (110). The compressed point cloud may be transmitted to another terminal device (120) via the network (150), for example, in the form of a bit stream. The terminal device (120) may receive the compressed point cloud from the network (150), decompress the bit stream to reconstruct the point cloud, and appropriately display the reconstructed point cloud. Unidirectional data transmission may be common in media serving applications and the like.
[0041] In the example of FIG. 1, the terminal devices (110) and (120) may be shown as a server and a personal computer, although the principles of the present disclosure are not so limited. The embodiments of the present disclosure may be applied to laptop computers, tablet computers, smart phones, gaming consoles, media players, and / or dedicated three-dimensional (3D) devices. The network (150) represents any number of networks that transmit compressed point clouds between the terminal devices (110) and (120). The network (150) may include, for example, wireline and / or wireless communication networks. The network (150) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include electronic communication networks, local area networks, wide area networks, the Internet, and the like.
[0042] 2 illustrates a block diagram of a streaming system 200 in some example applications using point clouds. The disclosed subject matter is equally applicable to other point cloud-enabled applications, such as 3D telepresence applications, virtual reality applications, etc.
[0043] The streaming system (200) can include a capture subsystem (213). The capture subsystem (213) can include a point cloud source (201), such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a graphics generation component that generates the uncompressed point cloud in software, etc., that generates the uncompressed point cloud (202). In one example, the point cloud (202) includes points captured by a 3D camera. The point cloud (202) is shown in bold to emphasize the high amount of data compared to the compressed point cloud (204) (compressed point cloud bitstream). The compressed point cloud (204) can be generated by an electronic device (220) that includes an encoder (203) coupled to the point cloud source (201). The encoder (203) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The compressed point cloud (204) (or video bitstream of the compressed point cloud (204)), shown with thin lines to emphasize its lower data volume when compared to the point cloud stream (202), can be stored on a streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, can access the streaming server (205) to retrieve copies (207) and (209) of the compressed point cloud (204). The client subsystem (206) can include a decoder (210), for example within the electronic device (230). The decoder (210) decodes the incoming copy of the compressed point cloud (207) and creates an outgoing stream of a reconstructed point cloud (211) that can be rendered by a rendering device (212).
[0044] It is noted that electronic devices 220 and 230 may include other components (not shown). For example, electronic device 220 may include a decoder (not shown) and electronic device 230 may also include an encoder (not shown).
[0045] In some streaming systems, the compressed point clouds 204, 207, and 209 (e.g., compressed point cloud bitstreams) may be compressed according to a particular standard. In some examples, a video coding standard is used to compress the point cloud. Examples of these standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc.
[0046] 3 shows a block diagram of a V-PCC encoder for encoding point cloud frames according to some embodiments. In some embodiments, a V-PCC encoder (300) can be used in the communication system (100) and the streaming system (200). For example, the encoder (203) can be configured and operate in a manner similar to the V-PCC encoder (300).
[0047] The V-PCC encoder (300) receives point cloud frames as uncompressed input and generates a bitstream that corresponds to the compressed point cloud frames. In some embodiments, the V-PCC encoder (300) can receive the point cloud frames from a point cloud source, such as the point cloud source (201).
[0048] In the example of FIG. 3, the V-PCC encoder (300) includes a patch generation module (306), a patch packing module (308), a geometry image generation module (310), a texture image generation module (312), a patch information module (304), an occupancy map module (314), a smoothing module (336), image padding modules (316) and (318), a group expansion module (320), video compression modules (322), (323), and (332), an auxiliary patch information compression module (338), an entropy compression module (334), and a multiplexer (324).
[0049] According to disclosed aspects, the V-PCC encoder (300) converts the 3D point cloud frames into an image-based representation along with some metadata (e.g., occupancy map and patch information) that is used to convert the compressed point cloud back into a decompressed point cloud. In some examples, the V-PCC encoder (300) can convert the 3D point cloud frames into a geometry image, a texture image, and an occupancy map, which can then be encoded into a bitstream using video coding techniques. In general, a geometry image is a 2D image with pixels filled with geometry values associated with points projected onto pixels, where the pixels filled with geometry values can be referred to as geometry samples. A texture image is a 2D image with pixels filled with texture values associated with points projected onto pixels, where the pixels filled with texture values can be referred to as texture samples. An occupancy map is a 2D image with pixels filled with values that indicate whether they are occupied or not occupied by a patch.
[0050] The patch generation module (306) segments the point cloud into a set of patches (e.g., a patch is defined as a contiguous subset of the surface described by the point cloud), such that each patch is described by a depth field relative to a plane in 2D space, regardless of whether they overlap. In some embodiments, the patch generation module (306) aims to decompose the point cloud into a minimum number of patches with smooth boundaries while minimizing reconstruction error.
[0051] In some examples, the patch information module (304) can collect patch information indicative of the size and shape of the patch, and in some examples, the patch information can be packed into an image frame and then encoded by the auxiliary patch information compression module (338) to generate compressed auxiliary patch information.
[0052] In some examples, a patch packing module (308) is configured to map the extracted patches onto a two-dimensional (2D) grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a unique patch. Efficient patch packing can directly impact compression efficiency by minimizing unused space or ensuring temporal consistency.
[0053] The geometry image generation module (310) can generate a 2D geometry image associated with the geometry of the point cloud at a given patch location. The texture image generation module (312) can generate a 2D texture image associated with the texture of the point cloud at a given patch location. The geometry image generation module (310) and the texture image generation module (312) use the 3D to 2D mapping calculated during the packing process to store the geometry and texture of the point cloud as images. To better handle the case where multiple points are projected onto the same sample, each patch is projected onto two images called layers. In the example, the geometry image is represented by a W×H monochromatic frame in YUV420-8bit format. To generate the texture image, the texture generation procedure uses the reconstructed / smoothed geometry to calculate the colors associated with the resampled points.
[0054] The occupancy map module (314) can generate an occupancy map that describes the padding information for each unit. For example, the occupancy image includes a binary map for each cell of the grid that indicates whether the cell belongs to empty space or to the point cloud. In an example, the occupancy map uses binary information that describes for each pixel whether the pixel is padded or not. In another example, the occupancy map uses binary information that describes for each block of pixels whether the block of pixels is padded or not.
[0055] The occupancy map generated by the occupancy map module (314) can be compressed using lossless or lossy coding. If lossless coding is used, the occupancy map is compressed using an entropy compression module (334). If lossy coding is used, the occupancy map is compressed using a video compression module (332).
[0056] Note that the patch packing module (308) may leave some empty space between the 2D patches packed into the image frame. The image padding modules (316) and (318) may fill the empty space (called padding) to generate an image frame suitable for 2D video and image codecs. Image padding, also called background filling, may fill unused space with redundant information. In some instances, good background filling minimizes bitrate increase but does not introduce significant coding artifacts near patch boundaries.
[0057] The video compression modules (322), (323), (332) can encode 2D images, such as padded geometry images, padded texture images, occupancy maps, etc., based on an appropriate video coding standard, such as HEVC, VVC, etc. In some examples, the video compression modules (322), (323), (332) are individual components operating independently. However, in other examples, the video compression modules (322), (323), (332) can be implemented as a single component.
[0058] In some examples, the smoothing module (336) is configured to generate a smoothed image of the reconstructed geometry image. The smoothed image can be provided to the texture image generation (312). The texture image generation (312) can then adjust the generation of the texture image based on the reconstructed geometry image. For example, if the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating the texture image to compensate for the distortion of the patch shape.
[0059] In some embodiments, the group expansion (320) is configured to pad pixels around object boundaries with redundant low-frequency content to improve coding gain as well as visual quality of the reconstructed point cloud.
[0060] A multiplexer (324) can multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.
[0061] 4 shows a block diagram of a V-PCC decoder (400) for decoding a compressed bitstream corresponding to a point cloud frame in some examples. In some examples, the V-PCC encoder (400) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured and operated in a manner similar to the V-PCC decoder (400). The V-PCC decoder (400) receives the compressed bitstream and generates a reconstructed point cloud based on the compressed bitstream.
[0062] In the example of FIG. 4, the V-PCC decoder (400) includes a demultiplexer (432), video decompression modules (434) and (436), an occupancy map decompression module (438), an auxiliary patch information decompression module (442), a geometry reconstruction module (444), a smoothing module (446), a texture reconstruction module (448), and a color smoothing module (452).
[0063] A demultiplexer (432) can receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.
[0064] The video compression modules (434) and (436) can decode compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (434) decodes compressed texture images and outputs decompressed texture images. The video decompression module (436) decodes compressed geometry images and outputs decompressed geometry images.
[0065] [The occupancy map compression module (438) can decode the compressed occupancy map according to an appropriate standard (e.g., HEVC, VVC, etc.) and output a decompressed occupancy map.
[0066] The auxiliary patch information decompression module (442) can decode the compressed auxiliary patch information according to an appropriate standard (e.g., HEVC, VVC, etc.) and output the decompressed auxiliary patch information.
[0067] A geometry reconstruction module (444) can receive the decompressed geometry image and generate a reconstructed point cloud geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.
[0068] The smoothing module (446) can smooth out discrepancies at the edges of the patches. The smoothing procedure aims to mitigate potential discontinuities that may occur at the patch boundaries due to compression artifacts. In some embodiments, a smoothing filter can be applied to pixels located at the patch boundaries to mitigate distortions that may occur due to compression / decompression.
[0069] A texture reconstruction module (448) can determine texture information for points in the point cloud based on the decompressed texture image and the smoothed geometry.
[0070] The color smoothing module (452) can smooth out coloring discrepancies. Non-adjacent patches in 3D space are often packed adjacent to each other in 2D video. In some instances, pixel values of non-adjacent patches can be confused by block-based video codecs. The goal of color smoothing is to reduce visible artifacts that appear at patch boundaries.
[0071] 5 shows a block diagram of a video decoder (510) according to some examples. The video decoder (510) may be used within the V-PCC (400). For example, the video decompression modules (434) and (436) and the occupancy map decompression module (438) may be configured similarly to the video decoder (510).
[0072] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a compressed image, such as a coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser (520) may extract a set of subgroup parameters from the coded video sequence for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract information from the encoded video sequence, such as transform coefficients, quantization parameter values, motion vectors, and so on.
[0073] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory to generate symbols (521).
[0074] The reconstruction of the symbols (521) may include a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not shown for clarity.
[0075] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.
[0076] The first unit is a scalar / inverse transform unit 551. The scalar / inverse transform unit (551) receives quantized transform coefficients and control information, including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc., as symbols (521) from the parser (520). The scalar / inverse transform unit (551) can output blocks containing sample values that can be input to an aggregator (555).
[0077] In some examples, the output samples of the scaler / inverse transform unit (551) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558), for example, buffers the reconstructed current picture partially and / or completely. The aggregator (555) adds the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.
[0078] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to an inter-coded, possibly motion-compensated, block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches prediction samples may be controlled by the available motion vectors of the motion compensated prediction unit (553), e.g., in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may include interpolation of sample values fetched from the reference picture memory (557) when sub-sample accurate motion vectors are in use, motion vector prediction mechanisms, etc.
[0079] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). The video compression techniques are controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also include in-loop filter techniques that are responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and that may also be responsive to previously reconstructed loop filtered sample values.
[0080] The output of the loop filter unit (556) may be a sample stream that can be output to a render device and stored in a reference picture memory (557) for use in future inter-picture prediction.
[0081] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.
[0082] The video decoder (510) may perform decoding operations according to a given video compression technique of a standard, such as ITU-T Rec. H.265. The coding video sequence may conform to a syntax specified by the video compression technique or standard in use, in the sense that the coding video sequence conforms to both the video compression technique or standard and a profile documented in the video compression technique or standard. In particular, a profile may select certain tools from the full set of tools available in the video compression technique or standard as tools that are only available under the profile. Also, a requirement for compliance may be that the complexity of the coding video sequence is within the limits defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may in some cases be further restricted through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coding video sequence.
[0083] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video decoder (603) can be used within the V-PCC encoder (300) to compress the point cloud. In the example, the video compression modules (322) and (323) and the video compression module (332) are configured similarly to the encoder (603).
[0084] The video encoder (603) can receive images, such as padded geometry images, padded texture images, etc., and generate compressed images.
[0085] According to an embodiment, the video encoder (603) may code and compress pictures of a source video sequence (images) into a coded video sequence (compressed images) in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units, which are described below, and are not shown for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.
[0086] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a very simplified description, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and reference pictures) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that generated by a (remote) decoder (when any compression between the symbols and the coding bitstream is lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). When the decoding of the symbol stream produces bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, e.g., due to channel errors) is used in several related techniques as well.
[0087] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510) detailed above in connection with Figure 5. Referring also briefly to Figure 5, however, the entropy decoding portion of the video decoder (510), including the parser (520), may not be fully implemented in the local decoder (633), because symbols are available and the encoding / decoding of symbols into the encoded video sequence by the entropy coder (645) and parser (520) may be lossless.
[0088] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this method, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture.
[0089] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (630). The operation of the coding engine (632) may advantageously be lossy. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures, resulting in reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) may store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end video decoder (in the absence of transmission errors).
[0090] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be encoded, the predictor (635) may search the reference picture memory (634) for sample data (such as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc. that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-pixel block basis to find a suitable prediction reference. In some examples, the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).
[0091] The control unit (650) may manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used for encoding the video data.
[0092] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.
[0093] A controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coding picture type to each coding picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:
[0094] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different kinds of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art will recognize the variations of I-pictures and their respective applications and characteristics.
[0095] A predictive picture (P-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, in most cases using one motion vector and reference index to predict the sample values of each block.
[0096] A Bi-directionally Predictive Picture (B Picture) may be a picture that can be coded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values of each block. Similarly, a multi-predictive picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.
[0097] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to each picture of the block. For example, blocks of I pictures may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of B pictures may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.
[0098] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coding video data may therefore conform to a syntax specified by the video coding technique or standard being used.
[0099] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (sometimes abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded is called a current picture and is partitioned into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block within a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are in use.
[0100] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are before the current picture in the video in decoding order (but may be in the past and future, respectively, in display order). A block in the current picture can be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first and second reference blocks.
[0101] Furthermore, to improve coding efficiency, merge mode techniques can be used in inter-picture prediction.
[0102] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a video picture sequence are partitioned into coding tree units (CTUs) for compression. The CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Typically, a CTU includes three coding tree blocks (CTBs), i.e., one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Typically, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and so on.
[0103] 7 illustrates a block diagram of a G-PCC encoder (700), according to some examples. The G-PCC encoder (700) may be configured to receive point cloud data, compress the point cloud data, and generate a bitstream that transmits the compressed point cloud data. In an embodiment, the G-PCC encoder (700) may include a position quantization module (710), a duplicate point removal module (712), an octree encoding module (730), an attribute transfer module (720), a level of detail (LOD) generation module (740), an attribute prediction module (750), a residual quantization module (760), an arithmetic coding module (770), an inverse residual quantization module (780), an addition module (781), and a memory (790) for storing reconstructed attribute values.
[0104] As shown, an input point cloud (701) can be received at a G-PCC encoder (700). The positions (e.g., 3D coordinates) of the point cloud (701) are provided to a quantization module (710). The quantization module (710) is configured to quantize the coordinates to generate quantized positions. The remove duplicate points module (712) is configured to receive the quantized positions and perform a filtering process to identify and remove duplicate points. The octree encoding module (730) is configured to receive the filtered positions from the remove duplicate points module (712) and perform an octree-based encoding process to generate a sequence of occupancy codes that describe a 3D grid of voxels. The occupancy codes are provided to an arithmetic coding module (770).
[0105] The attribute transfer module (720) is configured to receive attributes of the input point cloud and perform an attribute transfer process to determine an attribute value for each voxel if multiple attribute values are associated with each voxel. The attribute transfer process can be performed on the reordered points output from the octree encoding module (730). The attributes after the transfer operation are provided to the attribute prediction module (750). The LOD generation module (740) is configured to operate on the reordered points output from the octree encoding module (730) and reorganize the points into different LODs. The LOD information is provided to the attribute prediction module (750).
[0106] The attribute prediction module (750) processes the points according to an order based on the LOD indicated by the LOD information from the LOD generation module (740). The attribute prediction module (750) generates an attribute prediction for the current point based on the reconstructed attributes of a set of neighboring points of the current point stored in the memory (790). A prediction residual can then be obtained based on the original attribute values received from the attribute transfer module (720) and the locally generated attribute prediction. If a candidate index is used in each attribute prediction process, an index corresponding to the selected prediction candidate can be provided to the arithmetic coding module (770).
[0107] The residual quantization module (760) is configured to receive the prediction residual from the attribute prediction module (750) and perform quantization to generate a quantized residual. The quantized residual is provided to the arithmetic coding module (770).
[0108] The inverse residual quantization module (780) is configured to receive the quantized residual from the residual quantization module (760) and generate a reconstructed prediction residual by performing the inverse of the quantization operation performed in the residual quantization module (760). The summation module (781) is configured to receive the reconstructed prediction residual from the inverse residual quantization module (780) and receive each attribute prediction from the attribute prediction module (750). Combining the reconstructed prediction residual and the attribute prediction generates a reconstructed attribute value that is stored in the memory (790).
[0109] The arithmetic coding module (770) is configured to receive the occupation codes, the candidate index (if used), the quantized residual (if generated), and other information, and perform entropy coding to further compress the received values or information. As a result, a compressed bitstream (702) carrying the compressed information may be generated. The bitstream (702) may be transmitted or provided to a decoder that decodes the compressed bitstream, or may be stored in a storage device.
[0110] 8 illustrates a block diagram of a G-PCC (800), according to an embodiment. The G-PCC decoder (800) may be configured to receive a compressed bitstream and perform decompression of the point cloud data to decompress the bitstream and generate decoded point cloud data. In an embodiment, the G-PCC decoder (800) may include an arithmetic decoding module (810), an inverse residual quantization module (820), an octree decoding module (830), an LOD generation module (840), an attribute prediction module (850), and a memory (860) for storing reconstructed attribute values.
[0111] As shown, a compressed bitstream (801) can be received by an arithmetic decoding module (810). The arithmetic decoding module (810) is configured to decode the compressed bitstream (801) to obtain quantized residuals (if generated) and an occupied code of the point cloud. The octree decoding module (830) is configured to determine reconstructed positions of points in the point cloud according to the occupied codes. The LOD generation module (840) is configured to reorganize the points into different LODs based on the reconstructed positions and determine an order based on the LODs. The inverse residual quantization module (820) is configured to generate a reconstructed residual based on the quantized residuals received from the arithmetic decoding module (810).
[0112] The attribute prediction module (850) is configured to perform an attribute prediction process to determine attribute predictions for the points according to an order based on the LOD. For example, the attribute predictions for the current point may be determined based on reconstructed attribute values of neighboring points of the current point stored in the memory (860). In some examples, the attribute predictions may be combined with respective reconstructed residuals to generate reconstructed attributes for the current point.
[0113] The sequence of reconstructed attributes generated from the attribute prediction module (850), together with the reconstructed positions generated from the octree decoding module (830), corresponds in one example to the decoded point cloud (802) output from the G-PCC decoder (800). Additionally, the reconstructed attributes are also stored in memory (860) and can then be used to derive attribute predictions for subsequent points.
[0114] In various embodiments, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in hardware, software, or a combination thereof. For example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented in a processing circuit, such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and the like. In another example, the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800) may be implemented as software or firmware that includes instructions stored on a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuit, such as one or more processors, cause the processing circuit to perform the functions of the encoder (300), the decoder (400), the encoder (700), and / or the decoder (800).
[0115] It should be noted that the attribute prediction modules (750) and (850) configured to implement the attribute prediction techniques disclosed herein may be included in other decoders or encoders that may have similar or different structures to those shown in Figures 7 and 8. Additionally, the encoder (700) and decoder (800) may be included in the same device or in separate devices in various examples.
[0116] According to some aspects of the disclosure, mesh compression may use a coding tool different from a PCC coding tool, or may use a PCC coding tool such as a PCC (e.g., G-PCC, V-PCC) encoder or a PCC (e.g., G-PCC, V-PCC) decoder as described above.
[0117] An object's mesh (also called a mesh model or mesh frame) may contain polygons that describe the object's surface. Each polygon may be defined by the polygon's vertices in 3D space and the edges that connect the vertices to the polygon. The information about how the vertices are connected (e.g., the edge information) is called the connectivity information. In some instances, an object's mesh is formed by connected triangles that describe the object's surface. Two triangles that share an edge are called two connected triangles. In other instances, an object's mesh is formed by connected quadrilaterals. Two quadrilaterals that share an edge may be called two connected quadrilaterals. Note that a mesh may be formed by any other suitable polygons.
[0118] In some instances, meshes may also include attributes such as color, normals, etc. associated with the vertices. Attributes can be associated with the surface of a mesh using mapping information that parameterizes the mesh in a 2D attribute map. The mapping information is typically described by a set of parameter coordinates, called UV coordinates or texture coordinates, associated with the vertices of the mesh. 2D attribute maps (sometimes called texture maps) are used to store high-resolution attribute information such as texture, normal, and displacement. Such information can be used for various purposes such as texture mapping and shading.
[0119] In some embodiments, a mesh may include components called geometry information, connectivity information, mapping information, vertex attributes, and attribute maps. In some examples, the geometry information is described by a set of 3D positions associated with the vertices of the mesh. In examples, (x,y,z) coordinates may be used to describe the 3D positions of the vertices, also referred to as 3D coordinates. In some examples, the connectivity information includes a set of vertex indices that describe how to connect the vertices to create a 3D surface. In some examples, the mapping information describes how to map the mesh surface to a planar 2D region. In examples, the mapping information is described by a set of UV parameters / texture coordinates (u,v) associated with the mesh vertices along with the connectivity information. In some examples, the vertex attributes include scalar or vector attribute values associated with the mesh vertices. In some examples, the attribute maps include attributes associated with the mesh surface and are stored as a 2D image / video. In examples, the mapping between a video (e.g., a 2D image / video) and the mesh surface is defined by the mapping information.
[0120] According to disclosed aspects, several techniques, called UV mapping or mesh parameterization, are used to map the surface of a mesh in a 3D domain to a 2D domain. In some examples, the mesh is partitioned into patches in the 3D domain. A patch is a contiguous subset of the mesh with a boundary formed by boundary edges. A boundary edge of a patch is an edge that belongs to only one polygon of the patch and is not shared by two adjacent polygons in the patch. The vertices of the boundary edges in a patch are referred to as boundary vertices of the patch, and the non-boundary vertices in a patch are sometimes referred to as interior vertices of the patch.
[0121] In some instances, the mesh of an object is formed by connected triangles, and the mesh can be partitioned into patches, where each patch is a subset of the connected triangles. A boundary edge of a patch is an edge that belongs to only one triangle in the patch and is not shared by adjacent triangles in the patch. The vertices of a boundary edge in a patch are called boundary vertices of the patch, and the non-boundary vertices in a patch are sometimes called interior vertices of the patch. A boundary loop contains a sequence of boundary vertices, and the boundary edges formed by a sequence of boundary vertices can form a loop, called a boundary loop.
[0122] According to disclosed aspects, in some examples, the patches are each parameterized into a 2D shape (also called a UV patch). The 2D shapes can be packed (e.g., oriented and arranged) into a map, also called an atlas, in some examples. In some examples, the map can be further processed using 2D image or video processing techniques.
[0123] In an example, a UV mapping technique generates a 2D UV atlas (also called a UV map) and one or more texture atlases (also called texture maps) that correspond to patches of a 3D mesh. The UV atlas contains the assignment of 3D vertices of the 3D mesh to 2D points in a 2D domain (e.g., a rectangle). The UV atlas is a mapping from 3D surface coordinates to 2D domain coordinates. In an example, a point at 2D coordinate (u,v) in the UV atlas has a value formed by the coordinate (x,y,z) of the vertex in the 3D domain. In an example, a texture atlas contains color information for a 3D mesh. For example, a point at 2D coordinate (u,v) in the texture atlas (which has a 3D value of (x,y,z) in the UV atlas) has a color that specifies the color attributes of the point at (x,y,z) in the 3D domain. In some examples, the coordinates (x,y,z) in the 3D domain are called D coordinates or xyz coordinates, and the 2D coordinates (u,v) are called UV coordinates or UV coordinates.
[0124] According to some aspects of the disclosure, mesh compression can be performed by representing the mesh using one or more 2D maps (also referred to in some examples as 2D atlases) and then encoding the 2D maps using an image or video codec. Various techniques can be used to generate the 2D maps.
[0125] FIG. 9 shows a diagram illustrating the mapping of a 3D mesh (910) to a 2D atlas (920) in some examples. In the example of FIG. 9, the 3D mesh (910) includes four vertices 1-4 that form four patches A-D. Each patch has a set of vertices and associated attribute information. For example, patch A is formed by vertices 1, 2, and 3 that are connected to a triangle. Patch B is formed by vertices 1, 3, and 4 that are connected to a triangle. Patch C is formed by vertices 1, 2, and 4 that are connected to a triangle. Patch D is formed by vertices 2, 3, and 4 that are connected to a triangle. In some examples, vertices 1, 2, 3, and 4 can have their own attributes, and the triangle formed by vertices 1, 2, 3, and 4 can have their own attributes.
[0126] In examples, 3D patches A, B, C, and D are mapped to a 2D domain, such as a 2D atlas (920), also referred to as a UV atlas (920) or map (920). For example, patch A is mapped to a 2D shape (also referred to as a UV patch) A' in the map (920), patch B is mapped to a 2D shape (also referred to as a UV patch) B' in the map (920), patch C is mapped to a 2D shape (also referred to as a UV patch) C' in the map (920), and patch D is mapped to a 2D shape (also referred to as a UV patch) D' in the map (920). In some examples, coordinates in the 3D domain are referred to as (x,y,z) coordinates, and coordinates in a 2D domain, such as the map (920), are referred to as UV coordinates. Vertices in a 3D mesh can have corresponding UV coordinates in the map (920).
[0127] The map (920) can be a geometry map, containing geometry information; a texture map, containing color, normal, textile, or other attribute information; or an occupancy map, having occupancy information.
[0128] In the example of Figure 9, each patch is represented by a triangle, however, it is noted that a patch may include any suitable number of vertices that are connected to form a contiguous subset of the mesh. In some examples, the vertices in a patch are connected by triangles. It is noted that the vertices in a patch may be connected using other suitable shapes.
[0129] In an example, the geometry information of vertices can be stored in a 2D geometry map, which stores the (x,y,z) coordinates of the sampling points at corresponding points in the 2D geometry map, e.g., a point at (u,v) location in the 2D geometry map is a three-component vector value that respectively corresponds to the x, y, and z values of the corresponding sampling point in the 3D mesh.
[0130] According to aspects of the disclosure, regions in the map may not be fully occupied. For example, in FIG. 9, the regions outside the 2D shapes A', B', C', D' are undefined. Sample values in the regions outside the 2D shapes A', B', C', D' after decoding can be discarded. In some cases, the occupancy map is used to store some additional information per pixel, such as storing a binary value that identifies whether the pixel belongs to a patch or is undefined.
[0131] According to disclosed aspects, a dynamic mesh is a mesh in which at least one of the components (geometry information, connectivity information, mapping information, vertex attributes, attribute maps) changes over time. A dynamic mesh can be described by a set of meshes (also called mesh frames). In some examples, the mesh frames of a dynamic mesh are representations of an object's surface at different times, with each mesh frame being a representation of the object's surface at a particular time (also called a time instance). Dynamic meshes may contain a large amount of information that changes over time, and therefore may require a large amount of data. Mesh compression techniques allow for efficient storage and transmission of media content in mesh representations.
[0132] In some examples, the dynamic mesh can have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some examples, the dynamic mesh can have time-varying connectivity information. In examples, digital content creation tools typically generate dynamic meshes with time-varying attribute maps and time-varying connectivity information. In some examples, volumetric acquisition techniques are used to generate dynamic meshes. Volumetric acquisition techniques can generate dynamic meshes with time-varying connectivity information, especially under real-time constraints.
[0133] According to the disclosed aspects, texture mapping (e.g., texture map, attribute map) consumes a large portion of the mesh representation, and efficient compression of texture mapping can be one of the important tasks in mesh compression. In some examples, the texture mapping at a time instance includes an image (e.g., texture map, attribute map) that is composed of 2D shapes corresponding to patches, also called UV patches. Each UV patch represents texture information (e.g., red, green, blue color values) of a surface region of the mesh at the time instance. Pixel coordinates of the image correspond to UV coordinates.
[0134] FIG. 10 shows texture maps at two time instances in an example. In some examples, texture mappings at different time instances may be generated independently, and the associated UV patches in the texture maps of different time instances may be arranged in different ways. FIG. 10 shows the texture maps at the first time t 0 The first texture map (1010) at time t 1 10 shows a second texture map (1020) at a first time t 0 The second texture map (1020) contains a UV patch (1011) corresponding to the face of the person at time t 1 The UV patch (1011) and the UV patch (1021) are related. As can be seen in Figure 10, the UV patch (1011) and the UV patch (1021) are placed in each texture map with different orientations and different UV positions.
[0135] Some aspects of the disclosure provide encoding techniques for temporally aligning patches, e.g., UV patches in a 2D map, such as a texture map, an attribute map, a geometry map, etc. For example, if the UV patches in a 2D map are temporally aligned, the 2D map can be compressed more efficiently, e.g., using inter-prediction techniques. Note that in some examples, a mesh can be represented using a 2D map, and the term "patch" in this disclosure may refer to a UV patch in the context of a 2D map.
[0136] FIG. 11 shows a diagram of a process (1100) for temporally aligning patches according to some embodiments of the present disclosure. The process (1100) can be used during the encoding process of a mesh. In various embodiments, the process (1100) is performed by a processing circuit. In some embodiments, the process (1100) is implemented with software instructions, and thus, the processing circuit performs the process (1100) when the processing circuit executes the software instructions. The process starts at (S1101) and includes patch analysis (S1110), patch classification (S1120), remapping analysis (S1130), and remapping (S1140).
[0137] At (S1110), a patch analysis is performed. The patch analysis can analyze characteristics of the patch. In some examples, during the patch analysis, feature points of each patch in a 2D map (e.g., attribute map, texture map, etc.) are identified. The feature points can be identified by any suitable technique, such as scale-invariant feature transform (SIFT), speedup robust features (SURF), gradient location and orientation histogram (GLOH), histogram of oriented gradients (HOG), etc.
[0138] In some examples, a metric is calculated for each patch during patch analysis. The metric may be a scale invariant metric, a rotation invariant metric, a translation invariant metric, an affine transformation invariant metric, or the like. The following description describes some example metrics. In some examples, the example metrics may be used individually as a metric for patch analysis. In some examples, two or more of the example metrics may be combined to form a composite metric for patch analysis. In some examples, other suitable metrics that indicate similar characteristics of the patches may be used in the patch analysis.
[0139] In some examples, the center of the 3D coordinates of a patch can be calculated as a metric for the patch. For example, the center of the 3D coordinates of a patch can be calculated by a weighted sum of a subset of the 3D coordinates of the points (e.g., vertices) in the patch. In an example, the center of the 3D coordinates of a patch can be calculated by averaging the 3D coordinates of all points (e.g., vertices) in the patch, including the boundary points (e.g., including the boundary vertices). In another example, the center of the 3D coordinates of a patch is calculated by averaging the 3D coordinates of all points (e.g., vertices) in the patch excluding the boundary points (e.g., excluding the boundary vertices). In another example, the center of the 3D coordinates of a patch is calculated as the average of the 3D coordinates of all the boundary vertices of the patch.
[0140] In some examples, the UV center of a patch can be calculated as a metric for the patch. For example, the UV center of a patch can be calculated by a weighted sum of a subset of the UV coordinates of the points (e.g., vertices) in the patch. In an example, the UV center of a patch can be calculated by averaging the UV coordinates of all points (e.g., vertices) in the patch, including the border points (e.g., border vertices). In another example, the UV center of a patch is calculated by averaging the UV coordinates of all points (e.g., vertices) in the patch, excluding the border points (e.g., border vertices). In another example, the UV center of a patch is calculated by taking the average of the UV coordinates of all border vertices of the patch.
[0141] In some examples, a weighted average texture or attribute value of a subset of points in a patch can be calculated as a metric for the patch. In an example, the weighted average texture or attribute value can be calculated by averaging the texture or attribute values (e.g., red, green, blue channel values, etc.) of all points (e.g., vertices) in the patch, including the boundary points (e.g., boundary vertices). In another example, the weighted average texture or attribute value is calculated by averaging the texture or attribute values (e.g., red, green, blue channel values, etc.) of all points (e.g., vertices) in the patch, excluding the boundary points (e.g., boundary vertices). In another example, the weighted average texture or attribute value is calculated by taking the average of the texture or attribute values (e.g., red, green, blue channel values, etc.) of all boundary vertices of the patch.
[0142] At 1120, patch classification is performed. In some examples, during patch classification, patches at a given time instance can be classified into different categories based on frame-to-frame consistency or other temporal / spatial characteristics.
[0143] In some embodiments, patches of a 2D map (e.g., a geometry map, a texture map, or an attribute map) at a given time instance can be classified into two classes, called class M and class N. Class M includes patches for which a reference match patch was found at a past time instance, and class N includes patches for which a reference match patch was not found at a past time instance. In some examples, patch and reference match patch of patch are representations of portions of a surface of an object at different time instances. For example, patch represents a particular surface portion of an object at a current time, and reference match patch represents a particular surface portion of an object at a past time instance.
[0144] During patch classification, in some examples, for each patch, a reference matching technique is used against a 2D map (e.g., a geometry map, a texture map, an attribute map) of a past time instance to determine whether a reference matching patch can be found at the past time instance.
[0145] In some examples, the technique may determine, for a patch at a given time instance, a reference matching patch at a past time instance based on feature matching, e.g., if the patch and the reference matching patch have sufficient matching feature points. In an embodiment, a cross-feature matching technique may be utilized. For example, a cross-feature matching technique may be used to determine the reference matching patch at a given time instance (t) based on extracted feature points. i,t Compare the patch P with all patches at past time instances (tk). i,t The patch that best matches (e.g., based on the number of matching feature points) is PP i0,t-k Next, let us consider a patch P i0,t-k matches all patches at a time instance (t) and finds the best matching patch in the future. P i0,t-k The best future matching patch for P i,t When P i0,t-k and P i,t is a cross-match according to the cross feature matching technique. And the patch P i,t For P, there exists a reference-matching patch at a past time instance, i,t is classified as class M. In all other cases (e.g., P i0,t-k The optimal future match of P i,t ), Pi,t is classified into class N.
[0146] In some examples, during patch classification, a metric matching technique is used against a 2D map (e.g., a geometry map, a texture map, an attribute map) at a past time instance to determine if a reference matching patch is found at the past time instance. In an example, the center of the 3D coordinates is used to find the location of the patch P at time instance (t). i,t is matched against all patches at past time instances (tk), where k is a positive number. Among all patches at past time instances (tk), patch Pi,t The patch with the smallest difference in the center of the 3D coordinates to i0,t-k The difference between the centers of the 3D coordinates is l 1 Norm, l 2 The distance can be measured by the distance norm, or any other distance metric. Then, the best matching patch P i0,t-k Match P with all patches at time instance (t) i0,t-k Find the best matching patch for the future for patch P i0,t-k Let P be the patch of the past time instance (tk). i,t The difference between the center of the 3D coordinates of the patch P is the smallest. i,t Among the patches at time instance (t), patch P i0,t-k The difference between the center of the 3D coordinates of the patch P i,t P i0,t-k If P i0,t-k and P i,t is cross-matched by cross-matching the centers of 3D coordinates, and P i,t is classified as class M. In all other cases (e.g., P i0,t-k The optimal future match of P i,t ), Pi,t is classified into class N.
[0147] In some instances, more than one metric can be used in the metric matching technique. In the example, i,t , Cross-match patch of past time instances P i0,t-k are detected according to the center of 3D coordinates, and the difference between another metric (e.g., average texture value) or two metrics (e.g., average texture value, center of UV coordinates) of the two cross-match patches is small such that it is less than a given threshold (or two given thresholds in case of two metrics), P i,t is classified into class M. Otherwise (e.g., the difference between other metrics is greater than or equal to a threshold), P i,t is classified as Class N.
[0148] In some examples, the Hausdorff distance is used in the metric matching. For example, given a patch P i,t matches all patches at past time instances (tk), where k is a positive number according to the Hausdorff distance in 3D coordinates.
[0149] In some examples, the Hausdorff distance is combined with one, two, or more calculated metrics (e.g., average texture value, and / or center of 3D coordinates, and / or center of UV coordinates, etc.) in metric matching.
[0150] Although the patch classification in the above description performs reference matching of a given time instance to past time instances, patch classification can also be applied using reference matching of a given time instance to future time instances. For example, the patches of a 2D map (e.g., a geometry map, a texture map, or an attribute map) at a given time instance are classified into two classes, called class M and class N. Class M includes patches for which a reference match patch is found at a future time instance, and class N includes patches for which a reference match patch is not found at a future time instance.
[0151] In step S1130, a remapping analysis is performed. In some examples, during the remapping analysis, remapping transformation parameters such as rotation parameters, reflection parameters, and translation parameters are calculated for each patch to better align the patches in the time domain.
[0152] In some instances, remap transformation parameters can be derived based on matching local feature points.
[0153] In some instances, the remapping analysis begins with a patch of class M. After the remapping analysis is completed with the patch of class M, the remapping analysis can be performed with the patch of class N.
[0154] In an embodiment, for the remapping analysis, the packed patch list L is initialized to empty. Then, the patches of class M are ordered by size. The remapping analysis is performed on the patches of class M according to the size order. For example, a large patch of class M is analyzed before a small patch of class M. In an example, the patch size can be calculated by the number of pixels in the patch including the boundary. In another example, the patch size can be calculated by the number of pixels in the patch excluding the boundary. In another example, the patch size can be the area of the smallest rectangular bounding box that contains the patch.
[0155] In some examples, a rotation angle is determined during the remapping analysis of each patch to achieve the highest similarity with the reference matching patch. i,t has a reference matching patch at time instance (tk), where k is a positive number, and the reference matching patch is patch P i0,t-k In one example, for a patch, the angle r i,t A patch P with i,t After rotation of R(P i,t ) is the rotation patch indicated by the reference P i0,t-k The rotation angle r is set so that the image has the highest similarity (the similarity value is called the first highest similarity value) to i,t In some cases, the rotation angle r i,t is a value between 0 and 2π, 0≦r i,t <2π. In some cases, the similarity can be measured by the IoU (intersection over union) value. In some cases, the rotated patch R(P i,t ) and the reference matching patch P i0,t-k The cross-correlation between them is a measure of similarity.
[0156] In some examples, patch P i,t is reflected along the horizontal axis to form a reflection patch FP i,t is generated, and a reflection patch FP i,t Rotation angle f i,t But the standard P i0,t-kThen, the patch P is determined to have the highest similarity (the value of similarity is called the second highest similarity value) to the first highest similarity value. i,t r i,t or use the reflective patch FP i,t r i,t In one example, if the first similarity value is higher, the final rotation angle is r i,t If the second similarity value is higher, the final rotation angle r i,t HA-F i,t The final rotation angle r i,t The range is -2π <r i,t < 2π, and negative angles are in the patch P i,t This indicates that reflection has been added to the
[0157] Some examples use a 1-bit reflection flag to indicate whether reflection has been applied. When using a 1-bit reflection flag, the rotation angle ranges between 0 and 2π.
[0158] In some cases, the rotation angle r i,t After determining the rotated patch R(P i,t ) spatial translation value (u i,t ,v i,t For example, the spatial translation value (u i,t ,v i,t ) is the spatially translated patch R(P i,t )'s UV coordinate center (called the spatially translated patch, T(R(P i,t ) is a reference matching patch P i0,t-k The initial value is set to be equal to the center of the UV coordinates of the spatially translated patch T(R(P i,t )) is a spatially translated patch T(R(P i,t )) is checked to see if it overlaps with any patch already in the packed patch list L. i,t )) does not overlap with other patches in the packed patch list L, then the spatially translated patch T(R(P i,t)) is added to the packed patch list L. The remapping analysis can then continue with the next patch in class M in size order.
[0159] In some cases, the spatial translation value (u i,t ,v i,t ) is the rotation of the patch R(P i,t ) and the reference matching patch P i0,t-k The value in the window that can give the highest cross-correlation value is the spatial translation value (u i,t ,v i,t ) can be used as the initialization value for
[0160] In some cases, during overlap checking, the spatially translated patch T(R(P i,t ) overlaps with a patch already in the packed patch list L, and then the spatial translation value (u i,t ,v i,t ) is gradually modified in a range centered on the initialization value, and the spatial translation patch T(R(P i,t )) to find the spatial translation patch T(R(P i,t )) overlaps with any patch in the packed patch list L. i,t )) does not overlap with any patch in the packed patch list L, then the spatial translation patch T(R(P i,t )) is added to the packed patch list L. Then the remapping analysis continues with the next patch in class M, ordered by size.
[0161] However, in the example, all spatial translation values (u i,t ,v i,t ) for the spatial translation patch TT(R(P i,t If there is an overlap between patch P and some patch in the packed patch list L, i,t is removed from class M and added to class N. Then the remapping analysis continues with the next patch in class M, ordered by size.
[0162] Once the remapping analysis is complete for all patches in class M, the remapping analysis continues for patches in class N.
[0163] In some examples, the patches of class N are sorted by size. The remap analysis follows the size order. For example, large patches of class N are remap analyzed before small patches of class N. In some examples, the patch size is calculated by the number of pixels in the patch including the border. In other examples, the patch size can be calculated by the number of pixels in the patch excluding the border. In other examples, the patch size is the area of the smallest rectangular bounding box that contains the patch.
[0164] In some examples, patch P of class N i,t For the spatial translation value (u i,t ,v i,t ) can be determined. In the example, the spatial translation value (u i,t ,v i,t ) is initialized to (0,0), indicating no spatial translation. Next, the patch P i,t A patch P is considered to overlap with a patch already in the packed patch list L. i,t Check the patch P i,t If is non-overlapping (e.g., does not overlap with any patch in the packed patch list L), then patch P i,t is added to the packed patch list L. Then the remapping analysis continues with the next patch of class N in size order.
[0165] In some examples, patch P i,t If overlaps with a patch in the packed patch list L, the spatial translation value (u i,t ,v i,t ) is gradually modified around (0,0) and overlap-checked to obtain T(P i,t ) does not overlap with any patch in the packed patch list L.i,t ) are not overlapping, the spatially translated patch T(P i,t ) is added to the packed patch list L. Then the remapping analysis continues with the next patch of class N in size order.
[0166] In some cases, all spatial translation values (u i,t ,v i,t ), T(P i,t If an overlap occurs between t and some patch in the packed patch list L, the remapping analysis stops (called early stopping). Early stopping indicates that remapping is not feasible for the given search range. In the example, the remapping step (S1140) is skipped. As a result, no remapping is performed at the given time instance (t).
[0167] In some cases, if the remapping analysis with the patch for class N is completed without early stopping, remapping (S1140) can be performed.
[0168] Remap analysis is performed at (S1140).
[0169] In some examples, during remapping, one or more new 2D maps, such as a new texture map or a new attribute map, are generated for a time instance (t). In examples, each pixel in the original 2D map (e.g., the original texture map or the original attribute map) can be checked to determine whether the pixel belongs to the patch.
[0170] Pixel patch P i,t A pixel is an occupied pixel if it belongs to patch P i,t For an occupied pixel of, the pixel location is patch P i,t By applying the rotation angle and spatial translation value of the patch P, a new position in the new 2D map (e.g., a new geometry map, a new texture map, or a new attribute map) can be determined. i,tis in class N, so patch P i,t Note that the remapping transformation parameters of may not include a rotation angle parameter in t, and thus no rotation is applied to the pixel. After deriving the new position of the pixel, the pixel value of the original 2D map (e.g., geometry map, texture map, attribute value) is assigned to the value of the new position in the new 2D map. The remapping then continues to the next pixel.
[0171] If the pixel does not belong to any patch, it is an unoccupied pixel in the original 2D map (e.g. geometry map, texture map, attribute map) and the remapping continues with the next pixel.
[0172] After the pixel values of all occupied pixels in the original 2D map have been remapped to the new 2D map, there may be some pixels that do not have a value assigned to them in the new map. These are the unoccupied pixels in the new 2D map.
[0173] For unoccupied pixels of the new 2D map (e.g., new geometry map, new texture map, new attribute map), any suitable value may be assigned. In an example, the unoccupied pixels may be derived from the values of their occupied spatial / temporal neighbors to improve coding efficiency. In another example, the unoccupied pixels may be assigned a constant value, which may be the average or median value of all occupied pixels. In another example, a boundary extension technique is used to extend the boundary of the patch and fill the unoccupied pixels with the boundary value. In another example, a mipmap technique is used to fill the unoccupied pixels.
[0174] In some embodiments, for each 2D map, such as a geometry map, a texture map, an attribute image, etc., a temporal alignment of the patches is applied to generate a new 2D map, such as a new geometry map, a new texture map, a new attribute map, etc. However, in case of early stopping during the remap analysis, the new 2D map may be an exact copy of the original 2D map.
[0175] In some examples, for a 2D map (e.g., a geometry map, texture map, or attribute map) at the beginning of a mesh sequence, such as the very first 2D map, the corresponding new 2D map can be an exact copy of the original 2D map. In some examples, a 2D map of a future time instance can be used as a reference frame to generate a new 2D map for the very first 2D map of the mesh sequence.
[0176] In some examples, a new 2D map (e.g., a new geometry map, a new texture map, a new attribute map) may be generated for every time instance to form a new 2D map sequence, and compression may be applied to the new 2D map sequence.
[0177] In some examples, to restore a 2D map (e.g., a texture map or an attribute map) to its original UV coordinates at the decoder side, remap transformation parameters for each patch are communicated between the encoder and decoder, allowing the decoder to restore the original 2D map.
[0178] According to one aspect of the disclosure, the remap transformation parameters can be coded by various techniques. In some examples, for each patch, a one-bit transformation flag can be signaled to indicate whether the remap transformation is a non-trivial transformation. A trivial transformation is a remap transformation with no reflections, a rotation angle of 0 degrees, and a spatial translation value of (0.0). In some examples, the rotation angle can be quantized before coding. In examples, the quantization of the rotation angle can be done with a quantization step of π / 2, π / 4, or other quantization steps. In some examples, the spatial translation values can be quantized before coding. The quantization of the spatial translation values can be done with a quantization step of integer values, floating point values, or other quantization steps.
[0179] In some embodiments, the UV coordinates of the mesh representation (such as a 3D geometry representation like a Wavefront obj file) are updated. For example, the geometry map is updated to a new geometry map. After the UV coordinates are updated, there is no need to encode the remap transformation parameters.
[0180] According to another aspect of the disclosure, adaptive sampling may be applied at the encoder side. In some examples, a patch temporal alignment method may be utilized in conjunction with the adaptive sampling method.
[0181] Adaptive sampling adaptively applies sampling steps to a geometry, texture, or attribute map that can reduce the bandwidth required to represent the information. After adaptive sampling, patches of a 2D map, such as a geometry, texture, or attribute map, can be aligned in time. Note that the patches may have been sampled at different sampling rates during adaptive sampling.
[0182] FIG. 12 shows a flow chart outlining a process (1200) according to one embodiment of the present disclosure. The process (1200) can be used during the encoding process of a mesh. In various embodiments, the process (1200) is performed by a processing circuit. In some embodiments, the process (1200) is implemented by software instructions, and thus the processing circuit performs the process (1200) when the processing circuit executes the software instructions. The process starts at (S1201) and proceeds to (S1210).
[0183] At (S1210), at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance are received. The first mesh frame and the second mesh frame represent a surface of an object having a polygon. The first mesh frame includes a first 2D map having a first patch where the 3D information is mapped to 2D at the first time instance. The second mesh frame includes a second 2D map having a second patch where the 3D information is mapped to 2D at the second time instance.
[0184] At (S1220), a first patch of one of the first patches is identified as a reference matching patch of a second patch of one of the second patches.
[0185] At (S1230), first remapping transformation parameters for temporal alignment of the second patch to the first patch of the first 2D map are determined.
[0186] At (S1240), a new second 2D map corresponding to the second 2D map is generated, the new 2D map including a transformed second patch that is transformed from the second patch according to the first remap transformation parameters.
[0187] In some examples, a cross-match technique is used to identify a first patch as a reference match patch of a second patch. In examples, a first match metric of each second patch is determined relative to the first patch. The first patch is selected as a best match patch from the first patch according to the first match metric. Then, a second match metric of each first patch is determined relative to the second patch. The first patch is determined to be a reference match patch of the second patch in response to the second patch being the best match patch according to the second match metric.
[0188] In an example, a matching metric of a second patch relative to a particular patch of a first patch is determined according to matching feature points of the second patch and the particular patch. In another example, a matching metric of a second patch relative to a particular patch of a first patch is determined according to a difference between a center of 3D coordinates of the second patch and a center of 3D coordinates of the particular patch. In another example, a matching metric of a second patch relative to a particular patch of a first patch is determined according to a Hausdorff distance of 3D coordinates between the second patch and the particular patch.
[0189] In some examples, the first remapping transformation parameters for temporal alignment include at least one of a rotation parameter, a reflectance parameter, and a spatial translation parameter.
[0190] In some embodiments, the first remap transformation parameters of the second patch are encoded into a bitstream transmitting at least the first mesh frame and the second mesh frame.
[0191] In some embodiments, the geometric 2D map of the second mesh frame is updated according to the first remap transformation parameters.
[0192] In some embodiments, the second patches are classified into a first class and a second class, where the first class includes a first subset of the second patches that have a corresponding reference matching patch identified in the first patch, and the second class includes a second subset of the second patches that do not have a reference matching patch identified in the first patch.
[0193] In some examples, the first subset of the second patches are sorted by size into a first patch sequence in size order, and remap transformation parameters for each of the first subset of the second patches are determined according to the first patch sequence. In examples, the second patches are selected from the first patch sequence according to size order. The first patch sequence includes at least a third patch that is larger in size than the second patch. Second remap transformation parameters for the third patch are determined, and the third patch transformed according to the second remap transformation parameters is added to the packed patch list. A first rotation angle is determined to generate a rotated second patch that is most similar to the first patch. And a first spatial translation value is determined to move the rotated second patch to avoid overlap with the transformed patch in the packed patch list. The second patch is transformed according to the first rotation angle and the first spatial translation value to generate a transformed second patch. The transformed second patch is added to the packed patch list.
[0194] In some examples, whether to reflect the second patch is determined based on a similarity comparison of the highest similarity between the reflected second patch and a second highest similarity between the first patch.
[0195] In some examples, in response to the absence of spatial translation values that allow for no overlap in the predefined window, the second patch is removed from the first class and added to the second class (e.g., classified as a second class).
[0196] In some examples, the second subset of the second patches are sorted by size in a size-ordered second patch sequence. Remap transformation parameters for each of the second subset of the second patches are then determined according to the second patch sequence.
[0197] Next, the process proceeds to (S1299) and ends.
[0198] The process 1200 may be adapted as appropriate. Steps of the process 1200 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.
[0199] The techniques disclosed in this disclosure may be used separately or combined in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.
[0200] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 13 illustrates a computer system (1300) suitable for implementing certain embodiments of the subject matter of this disclosure.
[0201] The computer software can be coded using any suitable machine code or computer language that can be processed by mechanisms such as assembly, compilation, linking, etc. to generate code including instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly or through interpretation, microcode execution, etc.
[0202] The instructions may be executed in a variety of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.
[0203] 13 of the computer system (1300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. Furthermore, the configuration of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (1300).
[0204] The computer system (1300) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, through sensory input (e.g., keystrokes, swipes, data grabbing actions), audio input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a digital camera), and video (including, for example, two-dimensional video, three-dimensional video, stereoscopic video).
[0205] The input human interface devices may include one or more of a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a data grab (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308) (only one of which is shown).
[0206] The computer system (1300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through sensory output, sound, light, and smell / taste. Such human interface output devices may include sensory output devices (e.g., sensory feedback via a touch screen (1310), data grab (not shown), or joystick (1305), although there may also be sensory feedback devices that do not function as input devices), audio output devices (e.g., speakers (1309), headphones (not shown), visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without sensory feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output through such means as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown)).
[0207] The computer system (1300) may also include human accessible storage and associated media such as optical media including CD / DVD ROM / RW (1320) with media such as CDs / DVDs (1321), thumb drives (1322), removable hard drives or solid state drives (1323), legacy magnetic media such as tape and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.
[0208] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.
[0209] The computer system (1300) may also include an interface (1354) to one or more communication networks (1355). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan area, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide area digital networks including cable TV, satellite TV, terrestrial broadcast TV, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface that is attached to a specific general purpose data port or peripheral bus (1349) (e.g., a USB port on the computer system (1300)). Others are generally integrated into the core of the computer system (1300) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using these networks, the computer system (1300) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems, for example, using local or wide area digital networks. Specific protocols and protocol stacks may be used with each of the above-mentioned networks and network interfaces.
[0210] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (1340) of the computer system (1300).
[0211] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units (1343) in the form of FPGAs, hardware accelerators (1344) for specific tasks, graphics adapters (1350), etc. These devices may be connected through a system bus (1348), along with read-only memory (ROM) (1345), random access memory (1346), internal mass storage devices (1347) such as internal non-user-accessible hard drives, SSDs, etc. In some computer systems, the system bus (1348) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or through a peripheral bus (1349). In an example, a screen (1310) may be connected to the graphics adapter (1350). Peripheral bus architectures include PCI, USB, and the like.
[0212] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) may execute certain instructions that may be combined to generate the aforementioned computer code. The computer code may be stored in ROM (1345) or RAM (1346). Temporary data may also be stored in RAM (1346), while permanent data may be stored, for example, in an internal mass storage device (1347). Rapid storage and retrieval from any of the memory devices may be made possible through the use of cache memory, which may be closely associated with one or more of the CPU (1341), GPU (1342), mass storage device (1347), ROM (1345), RAM (1346), etc.
[0213] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.
[0214] As an example and not by way of limitation, the computer system (1300) having the architecture, and specifically the core (1340), can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be specific storage of the core (1340) of a non-transitory nature, such as the core's internal mass storage (1347) or ROM (1345), and media associated with user-accessible mass storage as described above. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (1340). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (1340) and specifically the processor therein (including CPU, GPU, FPGA, etc.) to perform certain operations or certain portions of certain operations described herein, including defining and modifying data structures stored in RAM (1346) according to software-defined operations. Additionally or alternatively, the computer system may provide functionality as a result of implementation in hardwired or other circuitry (e.g., accelerator (1344)) that can operate in conjunction with or in place of software to perform certain processes or certain portions of certain processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that implements logic for execution, or both. The present disclosure includes any appropriate combination of hardware and software.
[0215] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which are encompassed within the scope of this disclosure. It will be apparent to those skilled in the art that numerous systems and methods can be devised that, although not explicitly shown or described herein, embody the principles of the disclosure and thus are encompassed within the spirit and scope of the disclosure.
Claims
1. 1. A method of mesh encoding, comprising: receiving at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance, the first mesh frame and the second mesh frame representing a surface of an object having a polygon, the first mesh frame including a first 2D map having a first patch having three-dimensional (3D) information mapped into two dimensions (2D), and the second mesh frame including a second 2D map having a second patch having 3D information mapped into 2D; identifying a first patch of the first patches as a reference matching patch of a second patch of the second patches; determining first remapping transformation parameters for a temporal alignment of the second patch with respect to the first patch of the first 2D map; generating a new second 2D map corresponding to the second 2D map, the new second 2D map having a transformed second patch transformed from the second patch according to the first remap transformation parameters; The method includes:
2. The step of identifying the first patch as the reference matching patch for the second patch further comprises: determining a first match metric for each of the second patches relative to the first patch; selecting the first patch as a best matching patch from the first patches according to the first match metric; The method of claim 1 , comprising:
3. determining a second match metric for each of the first patches relative to the second patches; determining that the first patch is the reference matching patch of the second patch in response to the second patch being the best matching patch according to the second match metric; The method of claim 2 further comprising:
4. The step of determining the first match metric of each of the second patches relative to the first patch further comprises: determining a matching metric of the second patch to the particular patch of the first patch according to matching feature points of the second patch and the particular patch; determining a match metric of the second patch relative to the particular patch of the first patch according to a difference between a center of 3D coordinates of the second patch and a center of 3D coordinates of the particular patch; determining a match metric of the second patch relative to the particular patch of the first patch according to a Hausdorff distance in 3D coordinates between the second patch and the particular patch; The method of claim 2 , comprising at least one of:
5. The step of determining the first remapping transformation parameters for temporal alignment of the second patch to the first patch comprises: The method of claim 1 , further comprising determining at least one of a rotation parameter, a reflectance parameter, and a spatial translation parameter.
6. encoding the first remap transformation parameters for the second patch into a bitstream transmitting at least the first mesh frame and the second mesh frame; The method of claim 1 further comprising:
7. The method of claim 1 , further comprising: updating a geometry 2D map of the second mesh frame according to the first remap transformation parameters.
8. 2. The method of claim 1, further comprising: classifying the second patches into a first class and a second class, the first class including a first subset of the second patches that have a corresponding reference matching patch identified in the first patch, and the second class including a second subset of the second patches that do not have a reference matching patch identified in the first patch.
9. sorting the first subset of the second patches by size into a size-ordered first sequence of patches; determining remap transformation parameters for each of the first subset of the second patches according to the first patch sequence; The method of claim 8 further comprising:
10. The step of determining each of the remapping transformation parameters for the patches of the first sequence of patches further comprises: selecting the second patch from the first patch sequence according to the size order, the first patch sequence including at least a third patch having a size larger than the second patch, determining second remap transformation parameters for the third patch, and adding the third patch transformed according to the second remap transformation parameters to a packed patch list; determining a first rotation angle to generate a rotated second patch that is most similar to the first patch; determining a first spatial translation value for moving the rotated second patch to avoid overlap with transformed patches in the packed patch list; 10. The method of claim 9, comprising:
11. determining a transformed second patch of the second patch according to the first rotation angle and the first spatial translation value; adding the transformed second patch to the packed patch list; The method of claim 10 further comprising:
12. The method of claim 10 , further comprising: determining whether to reflect the second patch based on a similarity comparison between the highest similarity and a second highest similarity of the reflected second patch and the first patch.
13. The method of claim 10 , further comprising: determining to reclassify the second patch into the second class in response to allowing spatial translation values within a predefined window to be non-overlapping.
14. sorting the second subset of the second patches by size into a second size-ordered sequence of patches; determining remap transformation parameters for each of the second subset of the second patches according to the second patch sequence; The method of claim 9 further comprising:
15. the number of pixels in the patch, including the boundary of the patch; the number of pixels in the patch excluding the boundary of the patch; the area of the smallest rectangular bounding box that contains said patch; The method of claim 9 , further comprising determining a size of the patch according to at least one of:
16. 1. A mesh processing apparatus, comprising: a processing circuit, the processing circuit comprising: receiving at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance, the first mesh frame and the second mesh frame representing a surface of an object having a polygonal shape, the first mesh frame including a first 2D map having a first patch having three-dimensional (3D) information mapped into two dimensions (2D), and the second mesh frame including a second 2D map having a second patch having 3D information mapped into the 2D; identifying a first patch of the first patches as a reference matching patch of a second patch of the second patches; determining first remapping transformation parameters that align the second patch with respect to the first patch of the first 2D map; generating a new second 2D map having a transformed second patch that is transformed from the second patch according to the first remapping transformation parameters; The device is configured to:
17. The processing circuitry includes: determining a first match metric for each of the second patches relative to the first patch; selecting the first patch as a best matching patch from the first patches according to the first match metric; determining a second match metric for each of the first patches relative to the second patches; determining that the first patch is the reference matching patch of the second patch in response to the second patch being the best matching patch according to the second match metric; 17. The device of claim 16, configured to:
18. The processing circuitry includes: classifying the second patches into a first class and a second class, the first class including a first subset of the second patches having corresponding reference matching patches identified in the first patches, and the second class including a second subset of the second patches having no reference matching patches identified in the first patches; 17. The device of claim 16, configured to:
19. The processing circuitry includes: sorting the first subset of the second patches by size into a first size-ordered sequence of patches; determining remap transformation parameters for each of the first subset of the second patches according to the first patch sequence; 20. The device of claim 18, configured to:
20. The processing circuitry includes: selecting the second patch from the first patch sequence according to the size order, the first patch sequence including at least a third patch having a size larger than the second patch, determining second remap transformation parameters for the third patch, and adding the third patch transformed according to the second remap transformation parameters to a packed patch list; determining a first rotation angle for generating a rotated second patch that is most similar to the first patch; determining a first spatial translation value for moving the rotated second patch to avoid overlap with transformed patches in the packed patch list; 20. The device of claim 19, configured to:
21. 1. A method of mesh decoding, comprising: determining at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance, the first mesh frame and the second mesh frame representing a surface of a polygonal object, the first mesh frame including a first 2D map having a first patch with three-dimensional (3D) information mapped into two dimensions (2D), and the second mesh frame including a second 2D map having a second patch with 3D information mapped into 2D; identifying a first patch of the first patches as a reference matching patch of a second patch of the second patches; determining first remapping transformation parameters for a temporal alignment of the second patch with respect to the first patch of the first 2D map; generating a new second 2D map corresponding to the second 2D map, the new second 2D map having a transformed second patch transformed from the second patch according to the first remap transformation parameters; The method includes:
22. 1. A mesh processing apparatus, comprising: a processing circuit, the processing circuit comprising: determining at least a first mesh frame associated with a first time instance and a second mesh frame associated with a second time instance, the first mesh frame and the second mesh frame representing a surface of an object having a polygon, the first mesh frame including a first 2D map having a first patch having three-dimensional (3D) information mapped into two dimensions (2D), and the second mesh frame including a second 2D map having a second patch having 3D information mapped into the 2D; identifying a first patch of the first patches as a reference matching patch of a second patch of the second patches; determining first remapping transformation parameters that align the second patch with respect to the first patch of the first 2D map; generating a new second 2D map having a transformed second patch that is transformed from the second patch according to the first remapping transformation parameters; The device is configured to:
Citation Information
Patent Citations
Mesh-based video compression using domain transform
JP2009545931A
Coding and decoding patch data units for point cloud coding - Patents.com
JP2022525599A
