A parallel approach to dynamic mesh alignment.

JP2024515737A5Active Publication Date: 2024-11-06TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023565287
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-03-16
Filing Date
2023-03-17
Publication Date
2024-11-06
Estimated Expiration
2043-03-17

AI Technical Summary

Technical Problem

Existing mesh compression standards do not effectively handle dynamic meshes with time-varying connectivity and attribute maps, and sequential alignment methods hinder parallelization, leading to inefficient encoding and decoding.

Method used

Implementing a method for parallel dynamic mesh alignment that includes determining temporal and intra-frame alignments to enable independent processing of frames, using spatial and temporal correlation techniques to achieve efficient compression.

Benefits of technology

Enables efficient compression and decoding of dynamic meshes by allowing parallel processing, improving encoding and decoding efficiency and supporting real-time applications like AR and VR.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A method, apparatus, and system for parallel dynamic mesh alignment are provided. The process may include determining that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons that describe a surface of a volumetric object, and then determining an intra-frame alignment scheme for spatially aligning charts in the frames. The process may also include applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for the inter-frame alignment, and processing one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 323886, filed March 25, 2022, and U.S. Patent Application No. 18 / 184957, filed March 16, 2023, the disclosures of which are incorporated by reference in their entireties herein.

[0002] This disclosure relates to a set of advanced video coding techniques, and more particularly, to video-based dynamic mesh alignment and compression. [Background technology]

[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive interaction and communication. To achieve the realism of the 3D representations, 3D models are becoming ever more sophisticated, and a significant amount of data is tied to the creation and consumption of these 3D models. 3D meshes are widely used in 3D model immersive content.

[0004] A 3D mesh may contain several polygons that describe the surface of a volumetric object. A dynamic mesh sequence may require a large amount of data since it may contain a significant amount of information that changes over time. Therefore, efficient compression techniques are needed to store and transmit such content.

[0005] Mesh compression standards IC, MESHGRID, and FAMC have been previously developed to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information.

[0006] Furthermore, it is also difficult for volume acquisition techniques to generate always-on dynamic meshes, especially under real-time constraints, and this type of dynamic mesh content is not supported by existing standards.

[0007] In addition to the above, many alignment methods, especially temporal alignment, are based on sequential techniques. In other words, the processing of one frame depends on other frames, such as using the result of the chart assignment from the previous frame as the basis for assigning the chart of the current frame. The dependency makes those methods unsuitable for parallelization and therefore slow to encode / decode the mesh. Therefore, a method that eliminates this dependency is needed to enable parallelization and improve encoding and decoding efficiency. Summary of the Invention [Means for solving the problem]

[0008] According to an embodiment, a method for parallel dynamic mesh alignment may be provided, which may include the steps of determining that a temporal alignment exists between one or more frames in a received input mesh including a plurality of polygons describing a surface of a volumetric object, determining an intra-frame alignment scheme for spatially aligning charts within a frame based on the determination that a temporal alignment exists between the one or more frames, applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for the inter-frame alignment, and processing one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

[0009] According to an embodiment, an apparatus for parallel dynamic mesh alignment may be provided. The apparatus may comprise at least one memory configured to store a program code, and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include a first decision code configured to cause the at least one processor to determine that a temporal alignment exists between one or more frames in a received input mesh including a plurality of polygons describing a surface of a volumetric object, a second decision code configured to cause the at least one processor to determine an intra-frame alignment scheme for spatially aligning charts within a frame based on a determination that a temporal alignment exists between the one or more frames, a first application code configured to cause the at least one processor to apply the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for the inter-frame alignment, and a first processing code configured to cause the at least one processor to process one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

[0010] According to an embodiment, a non-transitory computer-readable medium may be provided that stores computer instructions. The instructions may include one or more instructions that, when executed by one or more processors of a device for parallel dynamic mesh alignment, cause the one or more processors to determine that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons describing a surface of a volumetric object, determine an intra-frame alignment scheme for spatially aligning charts within a frame based on the determination that a temporal alignment exists between the one or more frames, apply the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for the inter-frame alignment, and process one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

[0011] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system, according to an embodiment of the present disclosure. [Diagram 3] 1 is a schematic diagram of a simplified block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is an exemplary diagram of a projection of a 3D chart onto a 2D UV plane, according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 illustrates an exemplary hierarchical grouping or subgrouping of frames, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is an exemplary flow diagram illustrating parallel dynamic mesh alignment, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] A mesh may contain a number of polygons that describe the surface of a volumetric object. The vertices of the mesh in 3D space and information on how the vertices are connected may define each polygon, referred to as connectivity information. Optionally, vertex attributes such as color and normal may be associated with the mesh vertices. Attributes may also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping may be defined using a set of parametric coordinates, referred to as UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normal, displacement, etc. The high-resolution attribute information may be used for various purposes, such as texture mapping and shading.

[0014] As mentioned above, a 3D mesh or dynamic mesh may require a large amount of data since it may consist of a significant amount of information that changes over time. Existing standards do not consider time-varying attribute maps and connectivity information. Existing standards also do not support volume acquisition techniques that generate a constantly connected dynamic mesh, especially under real-time conditions.

[0015] Therefore, a new mesh compression standard is needed to directly process dynamic meshes with time-varying connectivity information and optionally time-varying attribute maps. The embodiments of the present disclosure enable efficient compression techniques for storing and transmitting such dynamic meshes. The embodiments of the present disclosure enable lossy and / or lossless compression for various applications such as real-time communication, storage, free viewpoint video, AR and VR, etc.

[0016] To achieve efficient compression, embodiments of the present disclosure employ spatial and temporal alignment to obtain intra-frame and / or inter-frame correlation. In the prior art, many of the alignment methods, especially the temporal alignment, are based on sequential techniques. In other words, the processing of one frame depends on other frames, such as using the result of the chart assignment from the previous frame as the basis for assigning the chart of the current frame. The dependency makes those methods unsuitable for parallelization and therefore slow to encode / decode meshes. Therefore, a method to eliminate this dependency is needed to enable parallelization and improve encoding and decoding efficiency.

[0017] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium are provided for parallel processing of dynamic mesh compression. The embodiments of the present disclosure may also be applied to static meshes.

[0018] 1 and 2, one embodiment of the present disclosure for implementing the encoding and decoding structures of the present disclosure is described.

[0019] FIG. 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of unidirectional transmission of data, the first terminal 110 may code video data, which may include mesh data, at a local location for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the restored video data. The unidirectional data transmission may be common in media serving applications, etc.

[0020] 1 shows a second pair of terminals 130, 140 arranged to support two-way transmission of coded video, such as may occur during a video conference. For two-way transmission of data, each terminal 130, 140 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0021] In FIG. 1, terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or other types of terminals. For example, terminals (110-140) may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communications network 150 may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure unless described herein below.

[0022] 2 shows an arrangement of video encoders and decoders in a streaming environment as an example of an application of the disclosed subject matter. The disclosed subject matter may be used in other video-enabled applications including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0024] The video source 201 may create a stream 202 including, for example, a 3D mesh and metadata associated with the 3D mesh. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., digital camera(s)) and a computational device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202 may have a large amount of data compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. As described in more detail below, the encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoder 203 may further generate an encoded video bitstream 204. The encoded video bitstream 204, which may have a small amount of data compared to the uncompressed stream 202, may be stored in a streaming server 205 for later use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0025] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an incoming copy of the encoded video bitstream 204, and create an outgoing video sample stream 211 that can be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0026] FIG. 3 is an example diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and a decoder.

[0027] 3, framework 300 may include an encoder 301 and a decoder 351. Encoder 301 may include one or more input meshes 305, one or more meshes with UV atlases 310, an occupancy map 315, a geometry map 320, an attribute map 325, and metadata 330. Decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, decoded metadata 350, and a reconstructed mesh 360.

[0028] According to one aspect of the disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate the mesh with UV atlas 310. By way of example, the pre-processing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may be performed only on the encoder side and not on the decoder side.

[0029] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh with UV atlas may be a mesh in which each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed and converted into multiple maps based on sampling. As an example, the UV atlas 310 may be processed and converted into an occupancy map, a geometry map, and an attribute map based on sampling the 2D mesh with UV atlas. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and transmitted to the decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.

[0030] According to one aspect, the decoder 351 may receive the encoded occupancy map, geometry map, and attribute map from the encoder. The decoder 351 may use appropriate techniques and methods to decode the occupancy map, geometry map, and attribute map in addition to the embodiments described herein. In one embodiment, the decoder 351 may include the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. The input mesh 305 may be reconstructed into the reconstructed mesh 360 using one or more reconstruction filters and techniques based on the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. In some embodiments, the metadata 330 may be sent directly to the decoder 351, and the decoder 351 may use the metadata to generate the reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, and the decoded attribute map 345. Post-filtering techniques may also be applied to the reconstructed mesh 360, including, but not limited to, remeshing, parameterization, tracking, voxelization, etc.

[0031] An input mesh with a 2D UV atlas may have vertices, and each vertex of the mesh may have an associated UV coordinate on the 2D atlas. The occupancy map, geometry map, and attribute map may be generated by sampling one or more points / locations on the UV atlas. Each sample location may be occupied or unoccupied if the location is inside a polygon defined by the mesh vertices. For each occupancy sample, the corresponding 3D geometry coordinates and attributes of the sample may be calculated by interpolating from the associated polygon vertices.

[0032] According to one aspect of the present disclosure, the sampling rate may be consistent throughout the 2D atlas. In some embodiments, the sampling rates of the u and v axes may be different, allowing for anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, and each such region may have a different sampling rate.

[0033] According to one aspect of the present disclosure, the sampling rate of each region (or the entire 2D atlas) may be signaled in a high level syntax, including but not limited to a sequence header, a frame header, a slice header, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) may be selected from a set of pre-established rates that are assumed by both the encoder and the decoder. Because of the set of pre-established rates that are known by both the encoder and the decoder, signaling of one particular sampling rate only requires signaling an index within the pre-established rate set. Examples of such pre-established sets may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) of a mesh frame may be predicted from a pre-established rate set, from previously used sampling rates in other already coded regions of the same frame, or from previously used sampling rates in other already coded mesh frames.

[0034] In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be based on some characteristic of each region (or the entire 2D atlas). As an example, the sample rate may be based on activity, and for rich textured regions (or the entire 2D atlas), or for regions of high activity (or the entire 2D atlas), the sample rate may be set high. As another example, for smooth regions (or the entire 2D atlas), or for regions of low activity (or the entire 2D atlas), the sample rate may be set low.

[0035] In some embodiments, the sampling rate of each region of the mesh frame (or the entire 2D atlas) may be signaled so that it may be possible to combine prediction and direct signaling. Syntax may be configured to indicate whether the sampling rate is predicted or directly signaled. If predicted, which predictor sampling rate should be used may be further signaled. If directly signaled, syntax to represent the value of the rate may be signaled.

[0036] FIG. 4 is an example diagram 400 illustrating projecting a 3D chart onto a 2D UV plane (2D UV chart) according to an embodiment of the present disclosure.

[0037] As seen in Figure 4, a 3D mesh can be divided into several charts, each of which can be projected onto a 2D plane. On the 2D UV plane, the mesh's attribute information, such as texture information, can find a correspondence with the 3D vertices.

[0038] As seen in Figure 4, the UV coordinate information can be used to find texture information for 3D positions (such as vertices) on the 2D UV plane. Note that those skilled in the art should understand that for different mesh frames, the division process of the chart may not be the same, resulting in various shapes of the chart and allocation on the 2D plane across different mesh frames.

[0039] As mentioned above, dynamic mesh sequences may require a large amount of data since it may consist of a significant amount of information that changes over time. To achieve efficient compression, embodiments of the present disclosure employ spatial and temporal alignment to obtain intra-frame and / or inter-frame correlation. In the prior art, many of the alignment methods, especially the temporal alignment, are based on sequential techniques. In other words, the processing of one frame depends on other frames, such as using the result of the chart assignment from the previous frame as the basis for assigning the chart of the current frame. The dependency makes those methods unsuitable for parallelization and therefore slow to encode / decode the meshes. Therefore, a method that eliminates this dependency is needed to enable parallelization and improve the encoding and decoding efficiency.

[0040] Embodiments of the present disclosure relate to aligning mesh texture maps and / or geometry images at different levels of parallelism.

[0041] In one embodiment, maximum temporal parallelism of alignment is achieved. In this embodiment, each frame may be processed independently while maintaining temporal alignment between frames. In this embodiment, an intra-frame alignment scheme may be determined to spatially align charts within a frame. Intra-frame alignment may be applied to all frames. Assuming consecutive frames are similar, the aligned charts obtained from those frames are similar as well. Thus, both intra-frame alignment and inter-frame alignment may be achieved, and frames may be processed independently and in parallel. In some embodiments, intra-frame alignment refers to charts at nearby locations within a frame that have similar attributes, such as color values. In some embodiments, inter-frame alignment refers to charts at similar locations across different frames that have similar attributes, such as color values, shapes, etc.

[0042] In some embodiments, the geometric and / or texture information of the chart may be used to design an intra-frame alignment scheme.

[0043] When spatially aligning charts in a geometry image, the charts may be spatially aligned using any information of the chart, such as the chart's 3D position, size, normal, and shape. As an example, the position of a chart may be represented by its centroid, which may be calculated by averaging the 3D coordinates of all vertices in the chart, or may be approximated by averaging only the 3D coordinates of the chart bounds vertices. The charts may then be ordered and rasterized by their position by using the Morton code / Z-order of their centroid coordinates.

[0044] In addition to the geometric information mentioned above, texture information such as the color or histogram of the chart can also be taken into account when spatially aligning the chart if a texture map is available.

[0045] When spatially aligning charts in a texture map, both geometric and texture information may be used. The rationale for using geometric information to align texture charts is that if the geometric change between consecutive frames is small, the textures of geometrically nearby charts from those frames will be similar. For texture information, in addition to the histograms mentioned above, the average RGB / YUV pixel values ​​of the chart, which represent its color information, may be used. In some embodiments, the average RGB / YUV pixel value of a chart may be calculated by averaging all pixel values ​​in the chart, or may be approximated by only averaging those pixel values ​​on the chart boundary. The charts may then be ordered and rasterized by the Morton code / Z-order of their average RGB / YUV pixel values. In some embodiments, the charts may be sorted and rasterized by other orders of the average RGB / YUV pixel values, such as using corresponding positions in the visible spectrum.

[0046] In some embodiments, the temporal alignment algorithm for placing a chart on a 2D image for either geometry or texture information may allow for referencing of the results of previous frames. However, certain restrictions may be placed on such referencing so that parallel processing is still possible to some extent. In one embodiment, frames may be divided into subgroups that can be processed in parallel.

[0047] FIG. 5 illustrates an example hierarchical coding structure with several layers or levels illustrating dependencies between groups or subgroups for limited parallel processing of frames, according to an embodiment of the present disclosure.

[0048] As seen in FIG. 5, the structure 500 includes I0 and B16 at the lowest level, and B1, B3, B5, B7, B9, B11, B13, B15 at the highest level. In one embodiment, frames at higher levels may only refer to frames at lower levels for temporal alignment results, and may not refer to frames at the same level or higher levels. As an example, B8 may refer to the temporal alignment result of I0. Thus, some parallel processing is possible for frames at the same level.

[0049] In some embodiments, a mesh frame may be divided into independent groups of pictures (GoPs) (e.g., B1, B2, ..., B16, I0) such that each GoP may be processed in parallel. Within a GoP, the prediction structures described above (either hierarchical or linear) (e.g., structure 500) may be used.

[0050] As another embodiment, the first mesh frame may be processed independently, and all other frames may be processed based on the first frame, so only one frame latency may be introduced.

[0051] FIG. 6 is a flow diagram illustrating a process 600 for parallel dynamic mesh alignment, according to an embodiment of the present disclosure.

[0052] In operation 605, it may be determined that there is a temporal alignment between one or more frames. In some embodiments, the temporal alignment is determined between frames in a received input mesh that includes a plurality of polygons that describe a surface of a volumetric object.

[0053] In some embodiments, determining that there is a temporal alignment between one or more frames is based on a temporal alignment result of one or more previous frames.

[0054] At operation 610, based on a determination that there is temporal alignment between one or more frames, an intra-frame alignment scheme for spatially aligning the chart within the frame may be determined.

[0055] In some embodiments, determining the intra-frame alignment may include spatially aligning the charts within the frame based on at least one of a 3D position of each chart, a size of each chart, and a shape of each chart. In some embodiments, determining the intra-frame alignment may include spatially aligning the texture charts within the frame based on geometry information associated with the charts and at least one of a histogram of each chart, an average RGB pixel value of each chart, or an average YUV pixel value of each chart.

[0056] In operation 615, an intra-frame alignment scheme may be applied to one or more corresponding charts in one or more frames for inter-frame alignment.

[0057] In some embodiments, one or more corresponding charts in one or more frames may be charts that are similarly positioned across one or more frames and have at least one of similar color values ​​or similar shapes.

[0058] In operation 620, one or more frames may be processed in parallel based on the intra-frame and inter-frame alignment.

[0059] In some embodiments, processing one or more frames in parallel may include dividing one or more frames into hierarchical subgroups for parallel processing, where hierarchical subgroups at the same level may be processed in parallel and frames at higher levels may only refer to frames at lower levels for temporal alignment results.

[0060] In some embodiments, processing one or more frames in parallel may include a first frame being processed independently of the one or more frames, where the one or more frames are processed in parallel based on the first frame.

[0061] In some embodiments, processing the one or more frames in parallel may include dividing the one or more frames into one or more groups of pictures (GoPs) and processing each GoP in the one or more GoPs in parallel. In some embodiments, the one or more frames may be divided into one or more hierarchical GoP structures.

[0062] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the present disclosure.

[0063] The computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0064] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0065] 7 for computer system 700 are examples and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be construed as having a dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 700.

[0066] The computer system 700 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users via, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface devices may be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0067] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touch screen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, a camera 708 (only one of each is shown).

[0068] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 710, data gloves, or joystick 705, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (such as speakers 709, headphones (not shown)), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may output two-dimensional visual output or output in more than three dimensions, such as through means of stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0069] The computer system 700 may also include human accessible storage devices and media associated with the storage devices such as optical media including CD / DVD ROM / RW 720 with media 721 such as CD / DVDs, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0070] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0071] The computer system 700 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television and terrestrial television, vehicular and industrial including CANBus, and the like. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 749 (e.g., a USB port of the computer system 700, etc.), while others are generally integrated into the core of the computer system 700 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 may communicate with other entities. Such communications may be unidirectional, receive only (e.g., broadcast television), transmit only unidirectional (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Such communications may include communications to a cloud computing environment 755. Particular protocols and protocol stacks may be used in each of those networks and network interfaces, as described above.

[0072] The aforementioned human interface devices, human accessible storage devices, and network interface 754 may be connected to the core 740 of the computer system 700 .

[0073] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748, along with read only memory (ROM) 745, random access memory 746, internal mass storage such as internal non-user accessible hard drives, SSDs, etc. 747. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0074] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage 747. Rapid storage and retrieval in any of the memory devices may be enabled by the use of cache memories that may be closely associated with one or more of the CPU 741, GPU 742, mass storage 747, ROM 745, RAM 746, etc.

[0075] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0076] By way of example and not limitation, a computer system having the architecture of computer system 700, and in particular core 740, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with a particular storage of core 740 that is non-transitory in nature, such as core internal mass storage 747 and ROM 745. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 740. Computer-readable media may include one or more memory devices or chips, depending on the particular need. The software may cause core 740, and in particular the processors in the core (including CPU, GPU, FPGA, etc.) to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. Where appropriate, references to software may encompass logic and vice versa. Where appropriate, references to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0077] While this disclosure describes several non-limiting embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods not explicitly shown or described herein, but which embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0078] 100 Systems 110 First Terminal 120 Second Terminal 130 Second Terminal Pair 140 Second Terminal Pair 150 Network 200 Streaming System 201 Video Sources 202 Sample Stream 203 Encoder 204 encoded video bitstream 205 Streaming Server 206 Streaming Client 209 Video Bitstream 210 Video Decoder 211 outgoing video sample stream 212 Display 213 Capture Subsystem 301 Encoder 305 Input Mesh 310 Meshes with UV atlas 315 Occupied Map 320 Geometry Map 325 Attribute Map 330 Metadata 335 Decoded Occupancy Map 340 Decoded Geometry Map 345 Decoded Attribute Map 350 Decoded Metadata 351 Decoder 360 Reconstructed Mesh 700 Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Mike 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 720 CD / DVD ROM / RW 721 CD / DVD and other media 722 Thumb Drive 723 Removable Hard Drive or Solid State Drive 740 cores 741 Central Processing Unit (CPU) 742 Graphics Processing Unit (GPU) 743 Field Programmable Gate Area (FPGA) 744 Hardware accelerators for specific tasks 745 Read-Only Memory (ROM) 746 Random Access Memory (RAM) 747 Internal large capacity storage such as SSD 748 System Bus 749 Surrounding Bus 750 Graphics Adapter 754 Network Interface 755 Cloud Computing Environment

Claims

1. 1. A method for parallel dynamic mesh alignment, the method being executed by at least one processor, the method comprising: determining that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons describing a surface of a volumetric object; determining an intra-frame alignment scheme for spatially aligning a chart within a frame based on a determination that the temporal alignment exists between the one or more frames; applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for inter-frame alignment; and processing the one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

2. The method of claim 1 , wherein the one or more corresponding charts in the one or more frames are charts that are similarly positioned across the one or more frames and have at least one of similar color values ​​or similar shapes.

3. The step of determining an intra-frame alignment scheme comprises:

2. The method of claim 1, comprising spatially aligning the charts within the frame based on at least one of a three-dimensional (3D) position of each chart, a size of the each chart, and a shape of the each chart.

4. The step of determining an intra-frame alignment scheme comprises:

2. The method of claim 1, comprising spatially aligning texture charts within the frame based on geometry information associated with the charts and at least one of a histogram of the respective chart, an average RGB pixel value of the respective chart, or an average YUV pixel value of the respective chart.

5. determining that there is a temporal alignment between the one or more frames is based on a temporal alignment result of one or more previous frames; 2. The method of claim 1, wherein the step of processing the one or more frames in parallel includes a step of dividing the one or more frames into hierarchical subgroups for parallel processing, the hierarchical subgroups at the same level being processed in parallel, and for the temporal alignment result, frames at a higher level refer only to frames at a lower level.

6. The method of claim 1 , wherein a first frame is processed independently of the one or more frames, and the one or more frames are processed in parallel based on the first frame.

7. The step of processing the one or more frames in parallel comprises: dividing the one or more frames into one or more Groups of Pictures (GoPs); and processing each GoP in the one or more GoPs in parallel.

8. The method of claim 7 , wherein the one or more frames are divided into one or more hierarchical GoPs.

9. 1. A device for parallel dynamic mesh alignment, the device comprising: at least one memory configured to store program code; at least one processor configured to read the program code and to operate as instructed by the program code, the program code comprising: a first decision code configured to cause the at least one processor to determine that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons describing a surface of a volumetric object; second decision code configured to cause the at least one processor to determine an intra-frame alignment scheme for spatially aligning charts within a frame based on a determination that the temporal alignment exists between the one or more frames; and a first applying code configured to cause the at least one processor to apply the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for inter-frame alignment; and first processing code configured to cause the at least one processor to process the one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

10. 10. The device of claim 9, wherein the one or more corresponding charts in the one or more frames are charts that are similarly positioned across the one or more frames and have at least one of similar color values ​​or similar shapes.

11. The first decision code is 10. The device of claim 9, further comprising: first spatial alignment code configured to cause the at least one processor to spatially align the charts within the frame based on at least one of a three-dimensional (3D) position of each chart, a size of the each chart, and a shape of the each chart.

12. The first decision code is 10. The device of claim 9, further comprising: first spatial alignment code configured to cause the at least one processor to spatially align texture charts within the frame based on geometry information associated with the charts and at least one of a histogram of the respective chart, an average RGB pixel value of the respective chart, or an average YUV pixel value of the respective chart.

13. determining that the temporal alignment between the one or more frames exists is based on a temporal alignment result of one or more previous frames; 10. The device of claim 9, wherein processing the one or more frames in parallel includes dividing the one or more frames into hierarchical subgroups for parallel processing, the hierarchical subgroups at the same level being processed in parallel, and for the temporal alignment result, frames at a higher level refer only to frames at a lower level.

14. 10. The device of claim 9, wherein a first frame is processed independently of the one or more frames, and the one or more frames are processed in parallel based on the first frame.

15. The processing code is a first partitioning code configured to cause the at least one processor to partition the one or more frames into one or more Groups of Pictures (GoPs); and second processing code configured to cause the at least one processor to process each GoP in the one or more GoPs in parallel.

16. 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device for parallel dynamic mesh alignment, cause the one or more processors to: determining that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons describing a surface of a volumetric object; determining an intra-frame alignment scheme for spatially aligning a chart within a frame based on a determination that the temporal alignment exists between the one or more frames; applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for inter-frame alignment; A non-transitory computer-readable medium comprising one or more instructions that cause the one or more frames to be processed in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

17. Determining the intra-frame alignment scheme comprises:

17. The non-transitory computer-readable medium of claim 16, comprising spatially aligning the charts within the frame based on at least one of a three-dimensional (3D) position of each chart, a size of the each chart, and a shape of the each chart.

18. Determining the intra-frame alignment scheme comprises:

17. The non-transitory computer-readable medium of claim 16, comprising spatially aligning texture charts within the frame based on geometry information associated with the charts and at least one of a histogram of the respective chart, an average RGB pixel value of the respective chart, or an average YUV pixel value of the respective chart.

19. determining that the temporal alignment between the one or more frames exists is based on a temporal alignment result of one or more previous frames; 17. The non-transitory computer-readable medium of claim 16, wherein processing the one or more frames in parallel includes dividing the one or more frames into hierarchical subgroups for parallel processing, the hierarchical subgroups at the same level being processed in parallel, and for the temporal alignment result, frames at a higher level refer only to frames at a lower level.

20. Processing the one or more frames in parallel includes: Dividing the one or more frames into one or more Groups of Pictures (GoPs); and processing each GoP in the one or more GoPs in parallel.