Merging Multiple Attribute Maps

Abstract

A method, apparatus, and system for merging multiple attribute maps for mesh compression may be provided. The process may include obtaining multiple attribute maps associated with a mesh that includes one or more texture maps. The multiple attribute maps may be concatenated into a single connectivity map, and concatenated UV coordinates for each of the multiple attribute maps may be generated based on recalculating the original UV coordinates of each of the multiple attribute maps within the single connectivity map.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 408,994, filed September 22, 2022, and U.S. Patent Application No. 18 / 314,307, filed May 9, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This disclosure is directed to a set of advanced video coding techniques, and more particularly, to dynamic mesh compression based on merging multiple attribute maps. [Background technology]

[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive interaction and communication. To achieve the sense of realism of the 3D representations, 3D models are becoming ever more sophisticated, and a significant amount of data is associated with the creation and consumption of these 3D models. 3D meshes are widely used to model immersive content.

[0004] A 3D mesh may contain several polygons that describe the surface of a volumetric object. Each polygon is defined by its vertices in 3D space and information about how the vertices are connected, called connectivity information. In some embodiments, vertex attributes, such as color and normals, can be associated with mesh vertices. Attributes can also be associated with the surface of a mesh by using mapping information that parameterizes the mesh with a 2D attribute map. Such mapping can typically be described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps are used to store high-resolution attribute information, such as texture, normals, and displacements. Such information can be used for various purposes, such as texture mapping and shading. Summary of the Invention [Problem to be solved by the invention]

[0005] Dynamic mesh sequences can require large amounts of data because they can contain a significant amount of information that changes over time, thus requiring efficient compression techniques to store and transmit such content.

[0006] Mesh compression standards IC, MESHGRID, and FAMC have previously been developed to deal with dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not take into account time-varying attribute maps and connectivity information.

[0007] Although common test conditions are used to evaluate the performance of compression schemes, the evaluation may not be consistent, at least for static meshes. Therefore, methods and systems need to be consistently adapted to the evaluation system. [Means for solving the problem]

[0008] According to an embodiment, a method for merging multiple attribute maps for mesh compression may be provided. The method may include obtaining multiple attribute maps associated with a mesh, where the multiple attribute maps include two or more texture maps associated with the mesh, generating a single concatenated map based on concatenating the multiple attribute maps, and generating concatenated UV coordinates for each of the multiple attribute maps based on recalculating original UV coordinates for each of the multiple attribute maps in the single concatenated map.

[0009] According to an embodiment, an apparatus for merging multiple attribute maps for mesh compression may be provided. The apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include: retrieval code configured to cause the at least one processor to retrieve multiple attribute maps associated with a mesh, the multiple attribute maps including two or more texture maps associated with the mesh; first generation code configured to cause the at least one processor to generate a single concatenated map based on concatenating the multiple attribute maps; and second generation code configured to cause the at least one processor to generate concatenated UV coordinates for each of the multiple attribute maps based on recalculating original UV coordinates for each of the multiple attribute maps in the single concatenated map.

[0010] According to an embodiment, a non-transitory computer-readable medium may be provided that stores computer instructions including one or more instructions that, when executed by one or more processors of a device for merging multiple attribute maps for mesh compression, cause the one or more processors to obtain multiple attribute maps associated with a mesh, the multiple attribute maps including two or more texture maps associated with the mesh, generate a single concatenation map based on concatenating the multiple attribute maps, and generate concatenated UV coordinates for each of the multiple attribute maps based on recalculating original UV coordinates for each of the multiple attribute maps in the single concatenation map.

[0011] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1]FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] 1 is an exemplary proposed framework of general text conditions for lossy mesh compression according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a flow diagram illustrating a texture map merging pipeline according to an embodiment of the present disclosure. [Figure 6] 1 is an exemplary connectivity texture map illustrating connectivity and pixel padding according to an embodiment of the present disclosure. [Figure 7A] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 7B] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 8] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 9A] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 9B] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 10A] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 10B] 1 is an exemplary texture map illustrating texture map concatenation according to an embodiment of the present disclosure. [Figure 11] FIG. 10 is an exemplary flow diagram illustrating merging multiple attribute maps according to an embodiment of the present disclosure. [Figure 12]FIG. 1 is a diagram of a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] A mesh may contain several polygons that describe the surface of a volumetric object. Information about its vertices in 3D space and how the vertices are connected may define each polygon, referred to as connectivity information. Optionally, vertex attributes, such as color and normals, may be associated with mesh vertices. Attributes may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping may be defined using a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. 2D attribute maps may be used to store high-resolution attribute information, such as texture, normals, and displacement. The high-resolution attribute information may be used for various purposes, such as texture mapping and shading.

[0014] As mentioned above, 3D meshes or dynamic meshes may require a large amount of data because they may consist of a significant amount of information that changes over time. Existing standards do not consider time-varying attribute maps and connectivity information. Existing standards also do not support volume acquisition techniques that generate constantly connected dynamic meshes, especially under real-time conditions.

[0015] Although common test conditions are used to evaluate the performance of compression schemes, the evaluations can be inconsistent, at least for static meshes, and therefore methods and systems need to be consistently adapted to the evaluation system.

[0016] Therefore, to consistently fit the evaluation system, especially for meshes with multiple attribute maps, the attribute maps, e.g., texture maps, are concatenated into a single attribute map. The UV coordinates of each attribute map can be recalculated to the corresponding size and position within the single concatenated attribute map. Attribute map merging, e.g., texture map merging, can be considered a preprocessing step for mesh compression for meshes with multiple attribute maps and multiple texture maps.

[0017] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium are provided for merging attribute maps during dynamic mesh compression. Embodiments of the present disclosure may also be applied to static meshes.

[0018] 1 and 2, one embodiment of the present disclosure for implementing the encoding and decoding structures of the present disclosure is described.

[0019] 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected through a network 150. In the case of unidirectional data transmission, a first terminal 110 may encode video data, which may include mesh data, at a local location for transmission to another terminal 120 through the network 150. A second terminal 120 may receive the other terminal's coded video data from the network 150, decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media delivery applications, for example.

[0020] 1 illustrates a second pair of terminals 130, 140 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 can code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 can also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0021] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0022] 2 shows the arrangement of a video encoder and decoder in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter can be used in other video-enabled applications, including, for example, video conferencing, digital TV, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0024] The video source 201 may, for example, create a stream 202 including a 3D mesh and metadata associated with the 3D mesh. The video source 201 may, for example, include a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., digital camera(s)) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202 may have a larger amount of data than an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. As described in more detail below, the encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoder 203 may also generate an encoded video bitstream 204. The encoded video bitstream 204 may have a smaller amount of data than the uncompressed stream 202 and may be stored on a streaming server 205 for future use. One or more streaming clients 206 can access the streaming server 205 to obtain a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0025] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an input copy of the encoded video bitstream 204, and generate an output video sample stream 211 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0026] FIG. 3 is an exemplary diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and decoder.

[0027] 3, framework 300 may include an encoder 301 and a decoder 351. Encoder 301 may include one or more input meshes 305, one or more meshes with UV atlases 310, an occupancy map 315, a geometry map 320, an attribute map 325, and metadata 330. Decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, decoded metadata 350, and a reconstructed mesh 360.

[0028] As previously mentioned, attributes can be associated with the vertices or surfaces of a mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mappings, called UV coordinates or texture coordinates, can be defined using a set of parametric coordinates associated with the mesh vertices. 2D attribute maps can be used to store high-resolution attribute information such as texture, normals, and displacements. The high-resolution attribute information can be used for various purposes, such as texture mapping and shading. 3D attribute maps can be used to store multiple attributes of nodes within a mesh.

[0029] According to one aspect of the present disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate the mesh 310 with a UV atlas. By way of example, the pre-processing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may be performed only on the encoder side and not on the decoder side.

[0030] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh with UV atlas may be a mesh in which each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed and converted into multiple maps based on sampling. As an example, the UV atlas 310 may be processed and converted into an occupancy map, a geometry map, and an attribute map based on sampling the 2D mesh with UV atlas. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and transmitted to the decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.

[0031] According to one aspect, decoder 351 can receive the encoded occupancy map, geometry map, and attribute map from the encoder. Decoder 351 can use appropriate techniques and methods for decoding the occupancy map, geometry map, and attribute map in addition to the embodiments described herein. In one embodiment, decoder 351 may generate decoded occupancy map 335, decoded geometry map 340, decoded attribute map 345, and decoded metadata 350. Input mesh 305 can be reconstructed into reconstructed mesh 360 using one or more reconstruction filters and techniques based on decoded occupancy map 335, decoded geometry map 340, decoded attribute map 345, and decoded metadata 350. In some embodiments, metadata 330 can be transmitted directly to decoder 351, and decoder 351 can use the metadata to generate reconstructed mesh 360 based on decoded occupancy map 335, decoded geometry map 340, and decoded attribute map 345. Post-filtering techniques may be applied to the reconstructed mesh 360, including but not limited to remeshing, parameterization, tracking, voxelization, etc.

[0032] An input mesh with a 2D UV atlas can have vertices, and each vertex of the mesh can have associated UV coordinates on the 2D atlas. Occupancy maps, geometry maps, and attribute maps can be generated by sampling one or more points / locations on the UV atlas. Each sample location can be occupied or unoccupied if the location is inside a polygon defined by the mesh vertices. For each occupancy sample, its corresponding 3D geometry coordinates and attributes can be calculated by interpolating from the associated polygon vertices.

[0033] According to one aspect of the present disclosure, the sampling rate may be consistent throughout the 2D atlas. In some embodiments, the sampling rates for the u-axis and v-axis may be different, allowing for anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, and each such region may have a different sampling rate.

[0034] According to one aspect of the present disclosure, the sampling rate for each region (or the entire 2D atlas) may be signaled in a high-level syntax, including, but not limited to, a sequence header, a frame header, a slice header, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be selected from a pre-established set of rates assumed by both the encoder and the decoder. Because the pre-established set of rates is known by both the encoder and the decoder, signaling one particular sampling rate only requires signaling an index within the pre-established rate set. Examples of such a pre-established set may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) of a mesh frame may be predicted from the pre-established rate set, from previously used sampling rates in other already-coded regions of the same frame, or from previously used sampling rates in other already-coded mesh frames.

[0035] In some embodiments, the sampling rate for each region (or the entire 2D atlas) can be based on some characteristic of each region (or the entire 2D atlas). As one example, the sample rate can be based on activity, and for richly textured regions (or the entire 2D atlas) or regions with high activity (or the entire 2D atlas), the sample rate can be set high. As another example, for smooth regions (or the entire 2D atlas) or regions with low activity (or the entire 2D atlas), the sample rate can be set low.

[0036] In some embodiments, the sampling rate for each region of the mesh frame (or the entire 2D atlas) may be signaled in such a way that it may be possible to combine prediction and direct signaling. Syntax may be configured to indicate whether the sampling rate is predicted or directly signaled. If predicted, which predictor sampling rate should be used may be further signaled. If directly signaled, syntax to represent the value of the rate may be signaled.

[0037] FIG. 4 shows a framework 400 for common text conditions for lossy mesh compression. In the prior art, the process for common text conditions for lossy mesh compression may include triangulation 410, voxelization 420, and encoding 430 during encoding. Decoding may include decoding 440 and voxel de-voxelization 450. The process may also include generating mesh point clouds 460 and 470. The generated mesh point clouds 460 and 470 may be used to determine an error 480. The error 480 may include D1 PSNR, D2 PSNR, UV PSNR, and YCrCb PSNR. However, this framework results in inconsistent evaluation of mesh compression.

[0038] To fit the evaluation system consistently, for meshes with multiple texture maps, the texture maps can be concatenated into a single texture map. The UV coordinates of each texture map can be recalculated to the corresponding size and position within the single concatenated texture map. According to one embodiment of the present disclosure, this concatenation and recalculation of texture or attribute maps can be a pre-processing step for framework 400, improving the consistency of the evaluation system. This pre-processing step is shown in framework 400 as texture merging or attribute merging 405.

[0039]

[0003] Embodiments of the present disclosure relate to a method for merging attributes in a mesh. It can be understood by those skilled in the art that the attributes in a mesh can be normals or textures, or any other attributes associated with a mesh. As an example, embodiments of the present disclosure can be used for texture attribute merging. Embodiments of the present disclosure can also be applied to other attributes (i.e., normals).

[0040] Multiple texture maps in a mesh may be of different sizes, and the UV coordinates of each texture map may be in the range [0,1]; this disclosure relates to concatenating these maps into a single map at their original size and calculating new UV coordinates in the concatenated map.

[0041] In some embodiments, some UV coordinates in a mesh are out of range [0,1] or cross the boundaries of a texture map. This disclosure also provides specific methods and processes for merging texture maps and calculating corresponding new UV coordinates regarding these properties.

[0042] The attribute merging pipeline, consisting of the main steps, is shown in Figure 5. As an example, we will explain the texture map merging pipeline. Mesh faces that cross the boundary of a texture map may be signaled. If there are faces that cross the boundary of a texture map, the texture map can be modified and modified UV coordinates can be calculated. After this is done for all texture maps, the texture maps, including the modulation texture map, may be concatenated and pixel padded. The mesh UV coordinates can be recalculated for the concatenated maps. If there are no faces that cross the boundary of a texture map, the texture maps can be merged directly by concatenation and pixel padding without modifying the texture map.

[0043] FIG. 5 is a flow diagram illustrating a texture map synthesis process 500 according to an embodiment of the present disclosure.

[0044] As shown in process 500, faces of the mesh that cross the boundary of the texture map may be signaled in operation 505. If there are faces that cross the boundary of the texture map, the texture map may be modified in operation 510 and modified UV coordinates may be calculated in operation 515. After this has been done for all texture maps, the texture maps, including the modified texture map, may be concatenated in operation 520 and pixel padded in operation 525. The UV coordinates of the mesh may be recalculated for the connectivity map in operation 530.

[0045] However, in embodiments where there are no faces that cross the boundaries of the texture maps, the texture maps may be merged directly in operation 530 after pixel padding in operations 520 and 525 without modifying the texture maps in operations 510-515.

[0046] According to one embodiment, multiple texture maps are concatenated in two dimensions (U and V) as a square. Any suitable shape may be used. The number of texture maps is used to calculate the width and length of the concatenated texture map as follows:

[0047] N length ,N width =max(floor(log2N),2),ceil(log2N)…Equation (1)

[0048] Here, N is the total number of texture maps, and N_length and N_width are the number of texture maps in the vertical (length) and horizontal (width) directions, respectively.

[0049] Texture maps may be concatenated at their original size. For texture maps with different sizes, pixel padding can be used to fill the gaps between these maps. Pixel padding can be zero padding or non-zero padding. Zero padding can be padding pixels with zero values ​​in the RGB ("red, green, blue") channels. Non-zero padding can be padding pixels with non-zero values ​​in the RGB channels. Figure 6 includes texture maps with zero padding. Non-zero padding can also be used.

[0050] 6 is an exemplary concatenated texture map 600 illustrating concatenation and pixel padding according to an embodiment of the present disclosure. A concatenated texture map of the mesh "Winter Girl" is provided to illustrate an example of texture map concatenation and pixel padding, as seen in concatenated texture map 600. In concatenated texture map 600, 12 texture maps are concatenated as a 4x3 concatenated texture map. The empty spaces between texture maps of different sizes are padded with zeros.

[0051] The original UV coordinates in a texture map are usually in the range [0,1]. After concatenating texture maps into a single texture map, the original UV coordinates (or modified UV coordinates of texture maps with intersecting planes) reflecting their relative positions in the individual texture maps may be recalculated to fit the range [0,1] reflecting their relative positions in the newly merged texture map. Therefore, the new UV coordinates may be calculated according to the size and position of the corresponding individual texture map in the concatenation map. The UV coordinates may be normalized by the ratio of the individual texture map size to the overall concatenation map size, and then shifted to the corresponding position in the concatenation map. The calculation of the U and V coordinates may be as follows:

[0052]

number

[0053]

number

[0054] where (U original ,V original ) are the old UV coordinates that reflect their relative positions within the original individual texture maps. offset ,V offset ) are the positions of the individual texture maps in the new merged map in the U and V dimensions, respectively. The position of the point (0,0) in the individual texture maps can be made to be that position in the new merged texture map.

[0055] 7A-7B are exemplary texture maps 700 and 750, respectively, illustrating concatenation of texture maps according to an embodiment of the present disclosure.

[0056] As can be seen from FIG. 7A, FIG. 7A includes a large texture map of 1×1 and a small texture map of 0.5×0.5. The UV coordinates of the black dot in the small texture map are (0.75, 0.25). FIG. 7B shows a concatenated texture map 750 obtained by concatenating the two texture maps of FIG. 7A. As shown in FIG. 7B, the new U coordinate of the black dot is calculated as (0.75×0.5 + 1) / 1.5 = 0.917, and U offset is 1. The new V coordinate of the black dot is calculated as (0.25×0.5 + 0.5) / 1 = 0.625, and V offset is 0.5.

[0057] In some embodiments, the mesh can have coordinates less than 0 or greater than 1. Such coordinates indicate repetition in the map. As an example, FIG. 8 shows a texture map 800 with a repeating background.

[0058] When U (or V) is some parameter where 0 < U < 1 (or 0 < V < 1), U + 1, U + 2 (or V + 1, V + 2), etc. are the same positions within the texture map. Thus, the UV coordinates can be expressed as follows.

[0059] UV = remainder(UV / 1.0) + floor(UV / 1.0)… Equation (4)

[0060] To calculate the new UV coordinates in the concatenated texture map, the remainder of UV / 1.0 is normalized and shifted, and then the floor of UV / 1.0 is added to maintain the texture repetition property at the UV coordinates.

[0061] The color of each face in a mesh can be calculated by interpolating the sampled colors of its UV coordinates in a texture map. Figure 8 shows the color mapping of a texture map to faces with different UV coordinates. If a face does not cross the boundary of the texture map, the color mapping can be the same as the color mapping of the remaining UV coordinates. As shown in Figure 8, the UV residual of the right gray triangle is the same as the UV coordinate of the left gray triangle. The two gray triangles have the same color mapping on the face of the mesh. However, if a face crosses the boundary of the texture map as a black triangle, the color sampled by the remaining UV coordinates is changed to the color of the dashed black triangle, resulting in incorrect color mapping results.

[0062] Therefore, the UV coordinates of faces that cross the boundary of the texture map cannot be expressed as the remainder of UV / 1.0 for UVs greater than 1 or less than 0.

[0063] To address this issue, faces in a mesh that cross the boundary of a texture map are signaled. If there are any crossing faces, the texture map containing such faces is modified, and modified UV coordinates are calculated to ensure that all faces are within the texture map and that no faces in the modified texture map cross the boundary. If there are no crossing faces in the original texture map, the texture map modification and UV coordinate recalculation procedures can be skipped.

[0064] 9A-9B are exemplary texture maps illustrating concatenation of texture maps according to an embodiment of the present disclosure.

[0065] Figure 9A shows the original texture map and the UV coordinates crossing the texture map boundary, while Figure 9B shows the modified texture map and the recalculated UV coordinates.

[0066] 9A, the UV coordinates that cross the right boundary of the texture map are the same color as the UV coordinates that are closer to the left boundary of the texture map. Therefore, the left region of the texture map is copied over a width corresponding to the UV coordinates that cross the boundary and pasted onto the right side of the texture map. The modified UV coordinates of the modified texture map can be calculated based on the position and size of the copied region.

[0067] In one embodiment, if the copied region is on the right side,

[0068]

number

[0069] In one embodiment, if the copied region is on the left side,

number

[0070] In one embodiment, if the copied area is on the top surface:

[0071] V new =V original / (length original +length copied_area )…Formula (7)

[0072] In one embodiment, if the copied area is below: V new =(V original +length copied_area )+ / (length original +length copied_area )…Formula (8)

[0073] Figure 9B shows the copied and pasted area in the black dashed line in the transformed texture map. As shown in Figure 9B, the copied area is to the right of the original texture map and its width is 0.125. The three new UV coordinates can be calculated as (0.75 / 1.125, 0), (1.125 / 1.125, 0), and (0.938 / 1.125, 0.5), which are equal to (0.667, 0), (1, 0), and (0.833, 0.5). After calculating the new UV coordinates, there are no faces in the mesh that cross the boundary of the texture map. Therefore, all texture maps can be merged using the remaining part of the outer UV [0, 1] UV / 1.

[0074] 10A-10B are exemplary texture maps illustrating concatenation of texture maps according to an embodiment of the present disclosure.

[0075] 10A shows an example where both the U and V coordinates cross the boundary of a texture map. Due to the repeating nature of the UV coordinates, the bottom right point outside the texture map is equal to the top left point within the texture map. As described in this disclosure, the cross-boundary region is copied and pasted to modify the texture map and recalculate the UV coordinates.

[0076] FIG. 10B shows the modified texture map with three new UV coordinates: (0.75 / 1.125, (-1.125+1.125) / 1.125), (1.125 / 1.125, (-1.125+1.125) / 1.125), (0.938 / 1.125, (0.375+0.125) / 1.125), which are equal to (0.667,0), (1,0), and (0.833,0.444) in the modified texture map.

[0077] FIG. 11 shows an exemplary process 1100 for merging multiple attribute maps for mesh compression.

[0078] In an operation 1105, multiple attribute maps associated with the mesh may be obtained.

[0079] In some embodiments, the plurality of attribute maps includes two or more texture maps associated with the mesh. The plurality of maps may include any suitable type of attribute map.

[0080] At operation 1110, a single concatenated map may be generated based on concatenating multiple attribute maps.

[0081] In some embodiments, multiple 2D attribute maps may be concatenated, and the length and width of the single connectivity map may be based on the number of attribute maps included in the multiple connectivity map. In some embodiments, empty pixels between multiple attribute maps in a single connectivity map may be padded with a padding value. In some embodiments, padding may include assigning RGB channels with zero or non-zero values ​​to the empty pixels.

[0082] At operation 1115, connected UV coordinates for each of the multiple attribute maps may be generated based on recalculating the original UV coordinates for each of the multiple attribute maps in the single connected map.

[0083] In some embodiments, generating concatenated UV coordinates may include determining original UV coordinates for each of the multiple attribute maps in the single concatenation map and determining an offset of the original UV coordinates for each of the multiple attribute maps in the single concatenation map. The operations may also include generating concatenated UV coordinates for each of the multiple attribute maps in the single concatenation map based on the original UV coordinates and the offset of the original UV coordinates.

[0084] In some embodiments, when a texture map among the multiple attribute maps has a repeating pattern, the concatenated UV coordinates of the first texture map may be generated based on a normalized shift of the remaining portion of the original UV coordinates of the first texture map.

[0085] In some embodiments, prior to operation 1110, the process may include determining that one or more surfaces of at least one texture map of the plurality of attribute maps cross a boundary of the at least one texture map, and then signaling the one or more surfaces that cross the boundary of the at least one texture map. A respective region of the at least one texture map corresponding to a width or height of each respective surface of the one or more surfaces that cross the boundary of the at least one texture map may be repeated.

[0086] A modified texture map of the at least one texture map can be generated using modified UV coordinates based on the respective positions of the respective repeating regions and the respective sizes of the respective repeating regions. Finally, a single concatenation map can be generated based on concatenating the modified texture map and the multiple attribute maps.

[0087] It may be appreciated that the process 1100 may be performed on any type of map used in mesh compression, including attribute maps, texture maps, etc.

[0088] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 12 illustrates a computer system 1200 suitable for implementing certain embodiments of the present disclosure.

[0089] Computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0090] The instructions may be executed by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0091] 12 for computer system 1200 are examples and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be construed as having any dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 1200.

[0092] The computer system 1200 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), and olfactory input (not shown). The human interface devices may further be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (speech, music, ambient sounds, etc.), images (scanned images, photographic images obtained from a still camera, etc.), and video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0093] The input human interface devices may include one or more of a keyboard 1201, a mouse 1202, a trackpad 1203, a touchscreen 1210, a data glove, a joystick 1205, a microphone 1206, a scanner 1207, and a camera 1208 (only one of each is shown).

[0094] Additionally, computer system 1200 may include specific human interface output devices. Such human interface output devices may stimulate one or more of a human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via touchscreen 1210, data gloves, or joystick 1205, although haptic feedback devices that do not function as input devices may also be present). For example, such devices may include audio output devices (such as speakers 1209, headphones (not shown)), visual output devices (such as screens 1210, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as through means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0095] Additionally, computer system 1200 may include human-accessible storage devices and their associated media, such as optical media such as CD / DVD ROM / RW 1220 having media 1221 such as CD / DVD, thumb drive 1222, removable hard drive or solid state drive 1223, legacy magnetic media such as tape and floppy disk (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0096] Those skilled in the art will further appreciate that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves or other transitory signals.

[0097] Computer system 1200 may also include interfaces to one or more communications networks. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial television; vehicular and industrial networks including CANBus; and the like. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 1249 (e.g., a USB port on computer system 1200); others are generally built into the core of computer system 1200 by connection to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 1200 can communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Such communications may include communications to cloud computing environment 1255. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as previously described.

[0098] The aforementioned human interface devices, human-accessible storage devices, and network interface 1254 may be attached to core 1240 of computer system 1200 .

[0099] Core 1240 may include one or more central processing units (CPUs) 1241, graphics processing units (GPUs) 1242, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 1243, hardware accelerators 1244 for specific tasks, etc. These devices, along with read-only memory (ROM) 1245, random access memory 1246, and internal mass storage 1247 such as an internal non-user-accessible hard drive or SSD, may be connected through a system bus 1248. In some computer systems, system bus 1248 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 1248 or via a peripheral bus 1249. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 1250 may also be included in core 1240.

[0100] The CPU 1241, GPU 1242, FPGA 1243, and accelerator 1244 can execute specific instructions that, in combination, may constitute the aforementioned computer code, which may be stored in ROM 1245 or RAM 1246. Temporary data may also be stored in RAM 1246, while permanent data may be stored, for example, in internal mass storage 1247. Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 1241, GPU 1242, mass storage 1247, ROM 1245, and RAM 1246, etc.

[0101] The computer-readable medium may bear computer code for performing various computer-implemented operations. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0102] By way of example and not limitation, a computer system having the architecture of computer system 1200, and in particular core 1240, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of core 1240 that is non-transitory in nature, such as core internal mass storage 1247 or ROM 1245. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by core 1240. Computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 1240, and in particular the processors within the core (including a CPU, GPU, FPGA, etc.), to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM 1246 and modifying such data structures in accordance with the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic circuitry, hardwired or otherwise embodied in circuitry (e.g., accelerator 1244), that can operate in place of or cooperate with software to perform particular processes or portions of particular processes described herein. References to software may, where appropriate, encompass logic circuitry, and vice versa. References to computer-readable media may, where appropriate, encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0103] While this disclosure describes several non-limiting embodiments, there are modifications, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0104] 100 Communication system, 110 Terminal, 120 Terminal, 130 Terminal, 140 Terminal, 150 Network, 200 Streaming system, 201 Video source, 202 Sample stream, 203 Encoder, 204 Video bitstream, 205 Streaming server, 206 Streaming client, 209 Video bitstream, 210 Video decoder, 211 Video sample stream, 212 Display, 213 Capture subsystem, 300 Framework, 301 Encoder, 305 Input mesh, 310 Mesh with UV atlas, 315 Occupancy map, 320 Geometry map, 325 Attribute map, 330 Metadata, 335 Decoded occupancy map, 340 Decoded geometry map, 345 Decoded attribute map, 350 Decoded metadata, 351 Decoder, 360 Reconstructed mesh, 400 Framework, 405 Texture merging or attribute merging, 410 Triangulation, 420 Voxelization, 430 Encoder, 440 Decoder, 450 Voxel Decomposition, 460 Mesh Point Cloud, 470 Mesh Point Cloud, 480 Error, 500 Texture Map Synthesis Process, 600 Connected Texture Map, 700 Texture Map, 750 Connected Texture Map, 800 Texture Map, 1100 Process, 1200 Computer System, 1201 Keyboard, 1202 Mouse, 1203 Trackpad, 1205 Joystick, 1206 Microphone, 1207 Scanner, 1208 Camera, 1209 Speaker, 1210 Touchscreen, 1221 Media, 1222 Thumbdrive, 1223 Solid State Drive, 1240 Core, 1241 Computer Central Processing Unit (CPU), 1242 Graphics Processing Unit (GPU), 1243 Field Programmable Gate Array (FPGA), 1244 Hardware accelerator, 1245 read-only memory (ROM), 1246 random access memory, 1247 internal mass storage, 1248 system bus, 1249 peripheral bus, 1250 graphics adapter, 1254 network interface, 1255 cloud computing environment

Claims

1. 1. A method for merging multiple attribute maps for mesh compression, the method being executed by at least one processor, the method comprising: obtaining a plurality of attribute maps associated with a mesh, the plurality of attribute maps including two or more texture maps associated with the mesh; generating a single concatenated map based on concatenating the plurality of attribute maps; generating concatenated UV coordinates for each of the plurality of attribute maps based on recalculating original UV coordinates for each of the plurality of attribute maps within the single concatenated map.

2. The step of generating the single connectivity map comprises: concatenating the plurality of attribute maps in 2D, wherein the length and width of the single concatenated map is based on the number of attribute maps included in the plurality of attribute maps; padding empty pixels between the multiple attribute maps in the single connectivity map with a padding value.

3. 3. The method of claim 2, wherein the padding value is zero, and the padding step comprises assigning red-green-blue (RGB) channels in the empty pixels with a zero value.

4. The method of claim 2 , wherein the padding value is non-zero, and the padding step includes assigning RGB channels in the empty pixels with non-zero values.

5. The step of generating the concatenated UV coordinates comprises: determining the original UV coordinates for each of the plurality of attribute maps within the single connectivity map; determining an offset in the original UV coordinates for each of the plurality of attribute maps within the single connectivity map; generating the concatenated UV coordinates for each of the plurality of attribute maps in the single concatenated map based on the original UV coordinates and the offsets in the original UV coordinates.

6. 6. The method of claim 5, wherein, based on determining that a first texture map of the plurality of attribute maps has a repeating pattern, generating the concatenated UV coordinates for the first texture map is based on a normalized shift of a remaining portion of the original UV coordinates in the first texture map.

7. Prior to generating the single connectivity map, the method further comprises: signaling, based on one or more surfaces of the at least one texture map of the plurality of attribute maps that cross the boundary of the at least one texture map, the one or more surfaces that cross the boundary of the at least one texture map; repeating a region of each of the at least one texture map corresponding to a width or height of each respective face of the one or more faces that cross the boundary of the at least one texture map; generating a modified texture map of the at least one texture map having modified UV coordinates based on the respective positions of the respective repeating regions and the respective sizes of the respective repeating regions; 2. The method of claim 1, further comprising: generating the single concatenated map based on concatenating the modified texture map and the plurality of attribute maps, wherein the plurality of attribute maps do not include the at least one texture map used to generate the modified texture map.

8. 1. An apparatus for merging multiple attribute maps for mesh compression, the apparatus comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to operate as instructed by said program code, said program code comprising: retrieval code configured to cause the at least one processor to retrieve a plurality of attribute maps associated with a mesh, the plurality of attribute maps including two or more texture maps associated with the mesh; and first generation code configured to cause the at least one processor to generate a single concatenated map based on concatenating the plurality of attribute maps; and second generation code configured to cause the at least one processor to generate concatenated UV coordinates for each of the plurality of attribute maps based on recalculating original UV coordinates for each of the plurality of attribute maps within the single concatenated map.

9. The first generated code is a first concatenation code configured to cause the at least one processor to concatenate the plurality of attribute maps in 2D, wherein a length and a width of the single concatenation map are based on a number of attribute maps included in the plurality of attribute maps; and padding code configured to cause the at least one processor to pad empty pixels between the plurality of attribute maps in the single connectivity map with a padding value.

10. 10. The apparatus of claim 9, wherein the padding value is zero, and the padding comprises assigning red-green-blue (RGB) channels in the empty pixels with zero values.

11. The apparatus of claim 9 , wherein the padding values ​​are non-zero, and the padding comprises assigning RGB channels in the empty pixels with non-zero values.

12. The second generated code is first determination code configured to cause the at least one processor to determine the original UV coordinates for each of the plurality of attribute maps in the single connectivity map; second decision code configured to cause the at least one processor to determine an offset in the original UV coordinates for each of the plurality of attribute maps in the single connectivity map; and third generation code configured to cause the at least one processor to generate the concatenated UV coordinates for each of the plurality of attribute maps in the single concatenated map based on the original UV coordinates and the offsets in the original UV coordinates.

13. 13. The apparatus of claim 12, wherein generating the concatenated UV coordinates for a first texture map of the plurality of attribute maps based on determining that the first texture map has a repeating pattern is based on a normalized shift of a remaining portion of the original UV coordinates in the first texture map.

14. The method comprises: signaling code configured to cause the at least one processor to signal, based on one or more surfaces of the at least one texture map of the plurality of attribute maps that cross a boundary of the at least one texture map, the one or more surfaces that cross the boundary of the at least one texture map; iteration code configured to cause the at least one processor to iterate over each region of the at least one texture map corresponding to a width or height of each respective face of the one or more faces that cross the boundary of the at least one texture map; fourth generation code configured to cause the at least one processor to generate a modified texture map of the at least one texture map having modified UV coordinates based on the respective positions of the respective repeat areas and the respective sizes of the respective repeat areas; 9. The apparatus of claim 8, further comprising: fifth generation code configured to cause the at least one processor to generate the single concatenated map based on concatenating the modified texture map and the plurality of attribute maps, wherein the plurality of attribute maps do not include the at least one texture map used to generate the modified texture map.

15. 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device for merging multiple attribute maps for mesh compression, cause the one or more processors to: obtaining a plurality of attribute maps associated with a mesh, the plurality of attribute maps including two or more texture maps associated with the mesh; generating a single concatenated map based on concatenating the plurality of attribute maps; a non-transitory computer-readable medium comprising one or more instructions for generating concatenated UV coordinates for each of the plurality of attribute maps based on recalculating original UV coordinates for each of the plurality of attribute maps within the single concatenated map;

16. generating the single connectivity map concatenating the plurality of attribute maps in 2D, wherein the length and width of the single concatenated map are based on the number of attribute maps included in the plurality of attribute maps; and padding empty pixels between the plurality of attribute maps in the single connectivity map with a padding value.

17. 17. The non-transitory computer-readable medium of claim 16, wherein the padding value is 0, and the padding comprises assigning the empty pixels red-green-blue (RGB) channel values ​​of 0.

18. generating the concatenated UV coordinates determining original UV coordinates for each of the plurality of attribute maps within the single connectivity map; determining an offset in the original UV coordinates for each of the plurality of attribute maps within the single connectivity map; and generating the concatenated UV coordinates for each of the plurality of attribute maps in the single concatenated map based on the original UV coordinates and the offset in the original UV coordinates.

19. 20. The non-transitory computer-readable medium of claim 18, wherein generating the concatenated UV coordinates for the first texture map based on determining that a first texture map of the plurality of attribute maps has a repeating pattern is based on a normalized shift of a remaining portion of the original UV coordinates in the first texture map.

20. Prior to generating the single linkage map, the one or more instructions, when executed by the one or more processors, further cause the one or more processors to: signaling, based on one or more surfaces of the at least one texture map of the plurality of attribute maps that cross the boundary of the at least one texture map, the one or more surfaces that cross the boundary of the at least one texture map; repeating a region of each of the at least one texture map corresponding to a width or height of each respective face of the one or more faces that cross the boundary of the at least one texture map; generating a modified texture map of the at least one texture map having modified UV coordinates based on the respective positions of the respective repeating regions and the respective sizes of the respective repeating regions; 16. The non-transitory computer-readable medium of claim 15, wherein the single concatenated map is generated based on concatenating the modified texture map and the plurality of attribute maps, the plurality of attribute maps not including the at least one texture map used to generate the modified texture map.

Images