Method, device and computer program for generating a temporally correlated UV atlas

Abstract

A method, apparatus, and system for generating a temporally correlated UV atlas are provided. The process may include generating a plurality of consistent UV charts based on partition information associated with one or more previous frames, removing one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more entrance faces associated with the one or more non-manifold vertices to corresponding UV charts, merging two or more of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with one or more previous frames, and generating a temporally correlated UV atlas for a current frame based on aligning the plurality of consistent UV charts with one or more reference charts associated with one or more previous frames.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 323885, filed March 25, 2022, and U.S. Patent Application No. 18 / 185033, filed March 16, 2023, the disclosures of which are incorporated by reference in their entireties herein.

[0002] This disclosure relates to a set of advanced video coding techniques, and more particularly, to video-based dynamic mesh alignment and compression. [Background technology]

[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive interaction and communication. To achieve the realism of the 3D representations, 3D models are becoming ever more sophisticated, and a significant amount of data is tied to the creation and consumption of these 3D models. 3D meshes are widely used in 3D model immersive content.

[0004] A 3D mesh may be composed of several polygons that describe the surface of a volumetric object. A dynamic mesh sequence may require a large amount of data since it may contain a significant amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content.

[0005] Mesh compression standards IC, MESHGRID, and FAMC have been previously developed to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information.

[0006] Furthermore, it is also difficult for volume acquisition techniques to generate always-on dynamic meshes, especially under real-time constraints, and this type of dynamic mesh content is not supported by existing standards.

[0007] In addition to the above, many alignment methods, especially temporal alignment, are based on sequential techniques. In other words, the processing of one frame depends on other frames, such as using the result of the chart assignment from the previous frame as the basis for assigning the chart of the current frame. The dependency makes those methods unsuitable for parallelization and therefore slow to encode / decode the mesh. Therefore, a method that eliminates this dependency is needed to enable parallelization and improve encoding and decoding efficiency. Summary of the Invention [Means for solving the problem]

[0008] According to an embodiment, a method for generating a temporally correlated UV atlas includes generating a plurality of consistent UV charts associated with a current frame based on partition information associated with one or more previous frames; removing one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more entrance faces associated with the one or more non-manifold vertices to corresponding UV charts among the plurality of consistent UV charts; merging two or more of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with the one or more previous frames; and generating a temporally correlated UV atlas for the current frame based on aligning the plurality of consistent UV charts with one or more reference charts associated with the one or more previous frames, where the aligning is based on packing information associated with the one or more reference charts.

[0009] According to an embodiment, a device for generating a temporally correlated UV atlas includes at least one memory configured to store program code and at least one processor configured to read the program code and operate as instructed by the program code, the program code including: a first generation code configured to cause the at least one processor to generate a plurality of consistent UV charts associated with a current frame based on partition information associated with one or more previous frames; a first removal code configured to cause the at least one processor to remove one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more entrance faces associated with the one or more non-manifold vertices to corresponding UV charts among the plurality of consistent UV charts; a first merging code configured to cause the at least one processor to merge two or more of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with the one or more previous frames; and a second generation code configured to cause the at least one processor to generate a temporally correlated UV atlas of the current frame based on aligning the plurality of consistent UV charts with one or more reference charts associated with the one or more previous frames, the aligning being based on packing information associated with the one or more reference charts.

[0010] According to an embodiment, a non-transitory computer-readable medium stores instructions including one or more instructions that, when executed by one or more processors of a device for generating a temporally correlated UV atlas, cause the one or more processors to generate a plurality of consistent UV charts associated with a current frame based on partition information associated with one or more previous frames; remove one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more entrance faces associated with the one or more non-manifold vertices to corresponding UV charts in the plurality of consistent UV charts; merge two or more of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with the one or more previous frames; and generate a temporally correlated UV atlas for the current frame based on aligning the plurality of consistent UV charts with one or more reference charts associated with the one or more previous frames, wherein the aligning is based on packing information associated with the one or more reference charts.

[0011] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system, according to an embodiment of the present disclosure. [Diagram 3] 1 is a schematic diagram of a simplified block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is an exemplary diagram of a projection of a 3D chart onto a 2D UV plane, according to an embodiment of the present disclosure. [Diagram 5]FIG. 13 is an illustration of filling "holes" in a chart according to an embodiment of the present disclosure. [Figure 6] 1 is an example flow diagram illustrating a process for generating a temporally correlated UV atlas according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A mesh may contain a number of polygons that describe the surface of a volumetric object. The vertices of the mesh in 3D space and information on how the vertices are connected may define each polygon, referred to as connectivity information. Optionally, vertex attributes such as color and normal may be associated with the mesh vertices. Attributes may also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping may be defined using a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normal, displacement, etc. The high-resolution attribute information may be used for various purposes, such as texture mapping and shading.

[0014] As mentioned above, a 3D mesh or dynamic mesh may require a large amount of data since it may consist of a significant amount of information that changes over time. Existing standards do not consider time-varying attribute maps and connectivity information. Existing standards also do not support volume acquisition techniques that generate a constantly connected dynamic mesh, especially under real-time conditions.

[0015] Therefore, a new mesh compression standard is needed to directly process dynamic meshes with time-varying connectivity information and, optionally, time-varying attribute maps. The embodiments of the present disclosure enable efficient compression techniques for storing and transmitting such dynamic meshes. The embodiments of the present disclosure enable lossy and / or lossless compression for various applications such as real-time communication, storage, free viewpoint video, AR and VR, etc.

[0016] To achieve efficient compression, embodiments of the present disclosure employ spatial and temporal alignment to obtain intra-frame and / or inter-frame correlation. In the prior art, many of the alignment methods, especially the temporal alignment, are based on sequential techniques. In other words, the processing of one frame depends on other frames, such as using the result of the chart assignment from the previous frame as the basis for assigning the chart of the current frame. The dependency makes those methods unsuitable for parallelization and therefore slow to encode / decode meshes. Therefore, a method to eliminate this dependency is needed to enable parallelization and improve encoding and decoding efficiency.

[0017] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium are provided for parallel processing of dynamic mesh compression. The embodiments of the present disclosure may also be applied to static meshes.

[0018] 1 and 2, one embodiment of the present disclosure for implementing the encoding and decoding structures of the present disclosure is described.

[0019] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected through a network 150. In the case of a unidirectional transmission of data, the first terminal 110 may encode video data, which may include mesh data, at a local location for transmission to the other terminal 120 through the network 150. The second terminal 120 may receive the encoded video data of the other terminal from the network 150, decode the encoded data, and display the restored video data. Unidirectional data transmission may be common in media provision applications, etc.

[0020] 1 shows a second pair of terminals 130, 140 provided to support two-way transmission of coded video, such as may occur during a video conference. For two-way transmission of data, each terminal 130, 140 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0021] In FIG. 1, terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or other types of terminals. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communications network 150 may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure unless described herein below.

[0022] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter may be used in other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0024] The video source 201 may create a stream 202 including, for example, a 3D mesh and metadata associated with the 3D mesh. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., digital camera(s)) and a computational device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202 may have a large amount of data compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. As described in more detail below, the encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoder 203 may further generate an encoded video bitstream 204. The encoded video bitstream 204, which may have a small amount of data compared to the uncompressed stream 202, may be stored in a streaming server 205 for later use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0025] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an incoming copy of the encoded video bitstream 204, and create an outgoing video sample stream 211 that can be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0026] FIG. 3 is an example diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and a decoder.

[0027] 3, framework 300 may include an encoder 301 and a decoder 351. Encoder 301 may include one or more input meshes 305, one or more meshes with UV atlas 310, occupancy map 315, geometry map 320, attribute map 325, and metadata 330. Decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, decoded metadata 350, and a reconstructed mesh 360.

[0028] According to one aspect of the disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate a mesh with UV atlas 310. By way of example, the pre-processing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may be performed only on the encoder side and not on the decoder side.

[0029] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh with UV atlas may be a mesh where each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed and converted into multiple maps based on sampling. As an example, the UV atlas 310 may be processed and converted into an occupancy map, a geometry map, and an attribute map based on sampling the 2D mesh with UV atlas. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and transmitted to the decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.

[0030] According to one aspect, the decoder 351 may receive the encoded occupancy map, geometry map, and attribute map from the encoder. The decoder 351 may use appropriate techniques and methods to decode the occupancy map, geometry map, and attribute map in addition to the embodiments described herein. In one embodiment, the decoder 351 may include the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. The input mesh 305 may be reconstructed into the reconstructed mesh 360 using one or more reconstruction filters and techniques based on the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. In some embodiments, the metadata 330 may be sent directly to the decoder 351, and the decoder 351 may use the metadata to generate the reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, and the decoded attribute map 345. Post-filtering techniques may also be applied to the reconstructed mesh 360, including, but not limited to, remeshing, parameterization, tracking, voxelization, etc.

[0031] An input mesh with a 2D UV atlas may have vertices, and each vertex of the mesh may have an associated UV coordinate on the 2D atlas. The occupancy map, geometry map, and attribute map may be generated by sampling one or more points / locations on the UV atlas. Each sample location may be occupied or unoccupied if the location is inside a polygon defined by the mesh vertices. For each occupancy sample, the corresponding 3D geometry coordinates and attributes of the sample may be calculated by interpolating from the associated polygon vertices.

[0032] According to one aspect of the present disclosure, the sampling rate may be consistent throughout the 2D atlas. In some embodiments, the sampling rates of the u and v axes may be different, allowing for anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, and each such region may have a different sampling rate.

[0033] According to one aspect of the present disclosure, the sampling rate of each region (or the entire 2D atlas) may be signaled in a high level syntax, including but not limited to a sequence header, a frame header, a slice header, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) may be selected from a set of pre-established rates that are assumed by both the encoder and the decoder. Because of the set of pre-established rates that are known by both the encoder and the decoder, signaling of one particular sampling rate only requires signaling an index within the pre-established rate set. Examples of such pre-established sets may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) of a mesh frame may be predicted from a pre-established rate set, from previously used sampling rates in other already coded regions of the same frame, or from previously used sampling rates in other already coded mesh frames.

[0034] In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be based on some characteristic of each region (or the entire 2D atlas). As an example, the sample rate may be based on activity, and for rich textured regions (or the entire 2D atlas), or for regions of high activity (or the entire 2D atlas), the sample rate may be set high. As another example, for smooth regions (or the entire 2D atlas), or for regions of low activity (or the entire 2D atlas), the sample rate may be set low.

[0035] In some embodiments, the sampling rate of each region of the mesh frame (or the entire 2D atlas) may be signaled so that it may be possible to combine prediction and direct signaling. Syntax may be configured to indicate whether the sampling rate is predicted or directly signaled. If predicted, which predictor sampling rate should be used may be further signaled. If directly signaled, syntax to represent the value of the rate may be signaled.

[0036] FIG. 4 is an example diagram 400 illustrating projecting a 3D chart onto a 2D UV plane (2D UV chart) according to an embodiment of the present disclosure.

[0037] As seen in Figure 4, a 3D mesh can be divided into several charts, each of which can be projected onto a 2D plane. On the 2D UV plane, the mesh's attribute information, such as texture information, can find a correspondence with the 3D vertices.

[0038] As seen in Figure 4, the UV coordinate information can be used to find texture information for 3D positions (such as vertices) on the 2D UV plane. In an embodiment, the division process of the chart may not be the same, resulting in various shapes of the chart and allocation on the 2D plane across different mesh frames.

[0039] As mentioned above, dynamic mesh sequences may require a large amount of data since they may consist of a significant amount of information that changes over time. In particular, compression of texture maps or geometry images may be inefficient if the correlation between UV atlases of different mesh frames is low. Therefore, a highly correlated UV atlas across different mesh frames is desirable for dynamic mesh compression.

[0040] The embodiments of the present disclosure relate to generating a temporally correlated UV atlas for dynamic mesh compression. The embodiments may be applied individually or in any combination.

[0041] The framework for creating a mesh UV atlas may include five steps: mesh preprocessing, partitioning, parameterization, merging and packing.

[0042] According to embodiments, mesh pre-processing may be required to meet the input requirements of the mesh parameterization algorithm. In embodiments, the mesh may first be cleaned by removing overlapping, disjointed, or degenerate faces or vertices. Then, depending on the smoothness of the mesh, a smoothing filter may be applied to the cleaned mesh such that the number of UV charts generated is reduced. In some embodiments, the connectivity of the mesh is invariant under the smoothing operation.

[0043] According to an embodiment, to generate a highly correlated UV atlas, a consistent UV chart may first be generated. The consistent UV chart may include charts in which some of the charts in the current frame are similar to charts in the previous frame in terms of 3D position, shape, size, etc. The consistent UV chart may be obtained using partition information from the previous frame.

[0044] In one embodiment, the representative vertices employed in the previous frame for mesh partitioning may be used as reference vertices. The representative vertices for partitioning in the current frame may be selected from the landmark vertices that are closest to those reference vertices. Since partitioning may be based on the geodesic distance between the faces and the representative vertices, similar representative vertices will result in similar partitioning and thus consistent charts.

[0045] In another embodiment, some of the charts of the previous frame may be used as reference charts for splitting in the current frame. For example, the reference charts may be selected based on the largest chart in terms of number of vertices, number of faces, or 2D or 3D chart area, etc. After the reference charts of the previous frame are determined, those information may be used to find the corresponding charts of the current frame. For example, the centroid of each face in the reference chart may be used to find the closest face in the current frame that becomes a face in the corresponding chart.

[0046] In some embodiments, the generated chart may be validated. In one example, the corresponding chart obtained from the reference chart may not be a manifold, or may simply not be connected (by the standard definition), and therefore non-manifold vertices may need to be removed and / or "holes" in the chart may need to be filled. Non-manifold vertices may be detected and removed by assigning all incidence faces to the corresponding chart. To detect "holes" in the chart, the Euler characteristic of the chart may be calculated.

number

[0047] where k 0 , k 1 , k 2 is the number of vertices, edges, and faces in the chart, and χ=b 0 -b 1 and b 0 is the number of connected components, and b1 is the number of "holes" enclosed by the inner bounding loop. So, for a single chart, b 0 =1, the number of "holes" in the chart can be calculated as follows: b 1 =b 0 -χ=1-k 0 +k 1 -k 2 Formula (2)

[0048] In some embodiments, "holes" may be detected by finding all interior boundary loops in a chart. After detecting a "hole," it may be filled by assigning connected components at the "hole" to a corresponding chart or by iteratively relabeling faces that are incident on the interior boundary as part of the corresponding chart.

[0049] As an example, Figure 5 illustrates filling a "hole" in a chart according to an embodiment of the present disclosure, where the left and right panels show the chart before and after filling the "hole", respectively.

[0050] After verifying the topology of the corresponding chart, it may be necessary to verify whether the corresponding chart has become too large due to the face reassignment operation described above. As an example, it may be determined whether the chart area or the number of faces in the verified chart is significantly different from the reference chart. If so, the chart as inferred from the reference chart may be dropped.

[0051] In future embodiments, charts may have to be merged. During merging, it is desirable to maintain the similarity between the reference chart in the previous frame and the corresponding chart in the current frame. Therefore, some restrictions are required on the merge chart so that the similarity is not lost by the merge. As one non-limiting example, a specific policy may be adopted to merge charts. As a first matter, a unique merge ID may be assigned to each corresponding chart inferred from the reference chart. An exception is when the reference chart was merged in the previous frame, then the corresponding charts may have the same merge ID since they may also be allowed to be merged. All other charts that are not inferred from the reference chart may have the same merge ID that is different from the corresponding chart. In some embodiments, all descendant charts may inherit the same merge ID as their ancestor chart. Then, only charts with the same merge ID may be merged. An exception may be made when merging a large corresponding chart with a small chart that does not change the corresponding chart much, for example in terms of number of faces or number of chart areas, requiring the same merge ID. Then, merging may be allowed regardless of the merge ID.

[0052] The final step to generate a UV mesh atlas may be packing charts. In some embodiments, a reference chart from a previous frame and a corresponding chart in the current frame may be aligned. First, the packing information of the reference chart, such as position, orientation, etc., may be used to pack the corresponding chart. Then, charts that are not inferred from the reference chart may be packed.

[0053] FIG. 6 is a flow diagram illustrating a process 600 for generating a temporally correlated UV atlas according to an embodiment of the present disclosure.

[0054] In operation 605, the current frame may be preprocessed by removing duplicate, disjoint, or degenerate faces or vertices.

[0055] In some embodiments, after pre-processing, a smoothing filter can be optionally applied to the cleaned mesh so that the number of UV charts generated is reduced, depending on the smoothness of the mesh.

[0056] In operation 610, one or more consistent UV charts associated with the current frame may be generated based on partition information associated with one or more previous frames.

[0057] In some embodiments, the partition information may include at least one of a reference vertex from one or more previous frames and a reference chart from one or more previous frames. In some embodiments, a representative vertex for dividing the current frame may be selected from among landmark vertices in the current frame that are geodesically close to the reference vertex from one or more previous frames. In some embodiments, a representative chart for dividing the current frame may be selected from among reference charts associated with one or more previous frames, and the selection may be based on at least one of the number of vertices in the reference chart, the number of faces in the reference chart, or the number of chart regions in the reference chart.

[0058] In operation 615, one or more non-manifold vertices from the one or more consistent UV charts may be removed based on assigning one or more incidence faces associated with the one or more non-manifold vertices to corresponding UV charts in the one or more consistent UV charts.

[0059] In operation 620, more than one of the one or more consistent UV charts may be merged based on a similarity between the one or more consistent UV charts and one or more reference charts associated with one or more previous frames.

[0060] In some embodiments, the merging may include assigning a merge ID to each UV chart associated with the current frame inferred from the one or more reference charts, where the same merge ID may be assigned to merges of UV charts in the one or more consistent UV charts associated with the current frame based on a reference chart in the one or more reference charts that have been merged in one or more previous frames. In some embodiments, the merging may include merging two or more of the one or more consistent UV charts that have the same merge ID.

[0061] In some embodiments, the merging may include merging a larger UV chart in the one or more consistent UV charts with a smaller UV chart in the one or more consistent UV charts, where merging the larger UV chart with the smaller UV chart may not change the number of faces of the merged UV chart, and the larger UV chart may have a different merge ID than the smaller UV chart.

[0062] In operation 625, a temporally correlated UV atlas for the current frame may be generated based on aligning one or more consistent UV charts with one or more reference charts associated with one or more previous frames, where the alignment is based on packing information associated with the one or more reference charts.

[0063] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the present disclosure.

[0064] The computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0065] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0066] 7 for computer system 700 are examples and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be construed as having a dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 700.

[0067] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0068] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touch screen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, a camera 708 (only one of each is shown).

[0069] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 710, data gloves, or joystick 705, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (such as speakers 709, headphones (not shown)), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as through means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0070] The computer system 700 may also include human accessible storage devices and media associated with the storage devices such as optical media including CD / DVD ROM / RW 720 with media 721 such as CD / DVDs, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0071] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0072] The computer system 700 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television and terrestrial television, vehicular and industrial including CANBus, and the like. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 749 (e.g., a USB port of the computer system 700, etc.), while others are generally integrated into the core of the computer system 700 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), transmit only unidirectional (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Such communications may include communications to a cloud computing environment 755. Particular protocols and protocol stacks may be used in each of those networks and network interfaces, as described above.

[0073] The aforementioned human interface devices, human accessible storage devices, and network interface 754 may be connected to the core 740 of the computer system 700 .

[0074] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748, along with read only memory (ROM) 745, random access memory 746, internal mass storage such as internal non-user accessible hard drives, SSDs, etc. 747. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0075] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage 747. Rapid storage and retrieval in any of the memory devices may be enabled by the use of cache memories that may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0076] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0077] By way of example and not limitation, a computer system having an architecture corresponding to computer system 700, and in particular core 740, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with a particular storage of core 740 that is non-transitory in nature, such as core internal mass storage 747 and ROM 745. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 740. Computer-readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause core 740, and in particular the processors in the core (including CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0078] While this disclosure describes several non-limiting embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods not explicitly shown or described herein, but which embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0079] 100 communication system, 110-140 terminal, 150 communication network, 200 streaming system, 201 video source, 202 sample stream, 203 encoder, 204 video bitstream, 205 streaming server, 206 streaming client, 209 video bitstream, 210 video decoder, 211 outgoing video sample stream, 212 display, 213 capture subsystem, 300 framework, 301 encoder, 305 input mesh, 310 UV atlas, 315 occupancy map, 320 geometry map, 325 attribute map, 330 metadata, 335 decoded occupancy map, generated occupancy map, 340 decoded geometry map, 345 decoded attribute map, 350 decoded metadata, 351 decoder, 360 reconstructed mesh, 400 exemplary diagram, 600 process, 700 computer system, 701 keyboard, 702 mouse, 703 track pad, 705 joystick, 706 microphone, 707 scanner, 708 camera, 709 speaker, 710 touch screen, 720 CD / DVD ROM / RW, 721 CD / DVD or other medium, 722 thumb drive, 723 removable hard drive or solid state drive, 740 core, 741 central processing unit, CPU, 742 graphics processing unit, GPU, 743 field programmable gate area, FPGA, 744 hardware accelerator, 745 read only memory, ROM, 746 random access memory, RAM, 747 core internal mass storage, 748 system bus, 749 peripheral bus, 750 graphics adapter, 754 network interface, 755 cloud computing environment

Claims

1. 1. A method for generating a temporally correlated UV atlas, the method being executed by at least one processor, the method comprising: generating a plurality of consistent UV charts associated with a current frame based on partition information associated with one or more previous frames; removing one or more non-manifold vertices from the plurality of consistent UV charts based on assigning one or more faces of incidence associated with the one or more non-manifold vertices to corresponding UV charts in the plurality of consistent UV charts; merging two or more of the plurality of consistent UV charts based on a similarity between the plurality of consistent UV charts and one or more reference charts associated with the one or more previous frames; generating a temporally correlated UV atlas of the current frame based on aligning the plurality of consistent UV charts with the one or more reference charts associated with the one or more previous frames, the aligning being based on packing information associated with the one or more reference charts; A method comprising:

2. The method of claim 1 , wherein the partition information includes at least one of a reference vertex from the one or more previous frames and a reference chart from the one or more previous frames.

3. The method of claim 2 , wherein representative vertices for dividing the current frame are selected from among landmark vertices in the current frame that are geodesically close to the reference vertices from the one or more previous frames.

4. 3. The method of claim 2, wherein a representative chart for dividing the current frame is selected from among the reference charts associated with the one or more previous frames, the selection being based on at least one of a number of vertices in the reference chart, a number of faces in the reference chart, or a number of chart regions in the reference chart.

5. The step of merging further comprises: assigning a merge ID to each UV chart associated with the current frame inferred from the one or more reference charts; assigning the same merge ID to merges of UV charts among the plurality of consistent UV charts associated with the current frame based on a reference chart among the one or more reference charts that have been merged in the one or more previous frames; The method of claim 1.

6. The method of claim 5 , wherein the merging step further comprises merging two or more of the plurality of consistent UV charts having the same merge ID.

7. 7. The method of claim 6, wherein the merging step further comprises merging a large UV chart in the plurality of consistent UV charts with a small UV chart in the plurality of consistent UV charts, wherein merging the large UV chart with the small UV chart does not change a number of faces of the merged UV chart, and the large UV chart has a different merge ID than the small UV chart.

8. 1. A device for generating a temporally correlated UV atlas, the device comprising: at least one memory configured to store program code; and at least one processor configured to read said program code and to operate as instructed by said program code, said program code causing said at least one processor to perform the method of any one of claims 1 to 7.

9. A computer program comprising one or more instructions that, when executed by one or more processors of a device for generating a temporally correlated UV atlas, cause the one or more processors to perform a method according to any one of claims 1 to 7.

Images