A parallel approach to dynamic mesh alignment

Abstract

A method, apparatus, and system for parallel dynamic mesh alignment are provided. The process may include determining that a temporal alignment exists between one or more frames in a received input mesh that includes a plurality of polygons that describe a surface of a volumetric object, and then determining an intra-frame alignment scheme for spatially aligning charts in the frames. The process may also include applying the intra-frame alignment scheme to one or more corresponding charts in the one or more frames for the inter-frame alignment, and processing one or more frames in parallel based on the intra-frame alignment scheme and the inter-frame alignment.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 323,886, filed Mar. 25, 2022, and U.S. Patent Application No. 18 / 184,957, filed Mar. 16, 2023, the disclosures of which are hereby incorporated by reference in their entireties.

[0002] This disclosure relates to a set of advanced video coding techniques. More particularly, this disclosure relates to video - based dynamic mesh alignment and compression.

Background Art

[0003] The world's advanced three - dimensional (3D) representations enable more immersive interactions and communications. To achieve the sense of presence of 3D representations, 3D models have become more sophisticated than ever, and a significant amount of data is associated with the creation and consumption of these 3D models. 3D meshes are widely used in 3D model - immersive content.

[0004] A 3D mesh can include several polygons that describe the surface of a volumetric object. A dynamic mesh sequence may require a large amount of data since it can have a significant amount of information that changes over time. Therefore, an efficient compression technique is needed to store and transmit such content.

[0005] Mesh compression standards IC, MESHGRID, FAMC were previously developed to handle dynamic meshes with constant connectivity and time - varying geometry and vertex attributes. However, these standards do not consider time - varying attribute maps and connectivity information.

[0006] Furthermore, it is also difficult for volume acquisition techniques to generate constant - connectivity dynamic meshes, especially under real - time constraints. This type of dynamic mesh content is not supported by existing standards.

[0007] In addition to the above, many alignment methods, especially temporal alignment, are based on sequential techniques. In other words, processing one frame depends on other frames, such as using the results of chart assignment from the previous frame as the basis for assigning the chart in the current frame. This dependency makes these methods unsuitable for parallelization and therefore slows down the encoding / decoding of the mesh. Thus, methods to eliminate this dependency are needed to enable parallelization and improve encoding and decoding efficiency. [Overview of the Initiative] [Means for solving the problem]

[0008] According to embodiments, a method for parallel dynamic mesh alignment may be provided. The method may include the steps of: determining that a temporal alignment exists between one or more frames in a received input mesh including a plurality of polygons describing the surface of a volume object; determining an in-frame alignment scheme for spatially aligning charts within frames based on the determination that a temporal alignment exists between one or more frames; applying the in-frame alignment scheme to one or more corresponding charts in one or more frames for inter-frame alignment; and processing one or more frames in parallel based on the in-frame alignment scheme and inter-frame alignment.

[0009] According to one embodiment, a device for parallel dynamic mesh alignment may be provided. The device may comprise at least one memory configured to store program code, and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include a first determination code configured to cause at least one processor to determine that a temporal alignment exists between one or more frames in a received input mesh comprising a plurality of polygons describing the surface of a volume object; a second determination code configured to cause at least one processor to determine an in-frame alignment scheme for spatially aligning charts within frames, based on the determination that a temporal alignment exists between one or more frames; a first application code configured to cause at least one processor to apply the in-frame alignment scheme to one or more corresponding charts in one or more frames for inter-frame alignment; and a first processing code configured to cause at least one processor to process one or more frames in parallel based on the in-frame alignment scheme and inter-frame alignment.

[0010] According to the embodiment, a non-temporary computer-readable medium for storing computer instructions may be provided. When executed by one or more processors of the device for parallel dynamic mesh alignment, the instructions may include one or more instructions causing one or more processors to determine that there is a temporal alignment between one or more frames in a received input mesh which includes a plurality of polygons describing the surface of a volume object; to determine an in-frame alignment scheme for spatially aligning charts within frames based on the determination that there is a temporal alignment between one or more frames; to apply the in-frame alignment scheme to one or more corresponding charts in one or more frames for inter-frame alignment; and to process one or more frames in parallel based on the in-frame alignment scheme and inter-frame alignment.

[0011] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description and accompanying drawings below. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] This is a schematic diagram of a simplified block diagram of a streaming system according to an embodiment of the present disclosure. [Figure 3] This is a schematic diagram of a simplified block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] This is an exemplary figure illustrating the projection of a 3D chart onto a 2D UV plane according to an embodiment of the present disclosure. [Figure 5] This figure shows an example of hierarchical grouping or subgrouping of frames according to an embodiment of the present disclosure. [Figure 6] This is an exemplary flowchart illustrating parallel dynamic mesh alignment according to an embodiment of the present disclosure. [Figure 7] This is a diagram of a computer system suitable for implementing the embodiment. [Modes for carrying out the invention]

[0013] A mesh can contain several polygons that describe the surface of a volume object. Information about the vertices of the mesh in 3D space and how those vertices are connected can define each polygon, known as connectivity information. Optionally, vertex attributes such as color and normals can be associated with the mesh vertices. Attributes can also be associated with the surface of the mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping, called UV coordinates or texture coordinates, can be defined using a set of parametric coordinates associated with the mesh vertices. 2D attribute maps can be used to store high-resolution attribute information such as texture, normals, and displacements. High-resolution attribute information can be used for various purposes, such as texture mapping and shading.

[0014] As mentioned above, 3D meshes or dynamic meshes can require large amounts of data because they can consist of a significant amount of information that changes over time. Existing standards do not take into account time-varying attribute maps and connectivity information. Existing standards also do not support volume acquisition techniques that generate always-on-a-connectivity dynamic meshes, especially under real-time conditions.

[0015] Therefore, a new mesh compression standard is needed for directly processing dynamic meshes that have time-varying connectivity information and optionally time-varying attribute maps. Embodiments of the present disclosure enable efficient compression techniques for storing and transmitting such dynamic meshes. Embodiments of the present disclosure enable lossy and / or lossless compression for various applications such as real-time communications, storage, free-viewpoint video, AR, and VR.

[0016] To achieve efficient compression, embodiments of the present disclosure employ spatial and temporal alignment to obtain intra-frame and / or inter-frame correlations. In the prior art, many alignment methods, particularly temporal alignment, are based on sequential techniques. In other words, processing one frame depends on other frames, such as using the results of chart assignments from previous frames as a basis for assigning charts to the current frame. This dependency makes these methods unsuitable for parallelization and therefore slows down the encoding / decoding of meshes. Thus, methods that eliminate this dependency are needed to enable parallelization and improve encoding and decoding efficiency.

[0017] According to one aspect of this disclosure, a method, system, and non-temporary storage medium for parallel processing of dynamic mesh compression are provided. Embodiments of this disclosure may also be applied to static meshes.

[0018] Referring to Figures 1 and 2, one embodiment of the present disclosure for implementing the coding and decoding structures of the present disclosure will be described.

[0019] Figure 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of unidirectional data transmission, the first terminal 110 may encode video data, which may include mesh data at its local location, for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the encoded video data from the other terminal via the network 150, decode the encoded data, and display the restored video data. Unidirectional data transmission may be common in applications such as media serving.

[0020] FIG. 1 shows a second terminal pair 130, 140 provided to support bidirectional transmission of encoded video that can be performed, for example, during a video conference. In the case of bidirectional transmission of data, each terminal 130, 140 can encode video data captured at a local position for transmission to other terminals via network 150. Each terminal 130, 140 can also receive encoded video data transmitted by other terminals, decode the encoded data, and display the restored video data on a local display device.

[0021] In FIG. 1, terminals 110-140 can be, for example, a server, a personal computer, and a smartphone, and / or other types of terminals. For example, the terminals (110-140) may be a laptop computer, a tablet computer, a media player, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 can exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of network 150 may not be important for the operation of the present disclosure unless described hereinafter in this document.

[0022] FIG. 2 shows the arrangement of video encoders and decoders in a streaming environment as an example of the use of the disclosed subject matter. The disclosed subject matter can be used in other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media such as CDs, DVDs, memory sticks, etc.

[0023] As shown in FIG. 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0024] The video source 201 can create a stream 202 that includes, for example, a 3D mesh and metadata associated with the 3D mesh. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera(s)) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or the 3D imaging technology. The sample stream 202 may have a larger data volume compared to an encoded video bitstream and can be processed by an encoder 203 coupled to the video source 201. As will be described in detail below, the encoder 203 can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter. The encoder 203 may further generate an encoded video bitstream 204. The encoded video bitstream 204, which may have a smaller data volume compared to the uncompressed stream 202, can be stored at the streaming server 205 for later use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 that may be a copy of the encoded video bitstream 204.

[0025] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 can, for example, decode a video bitstream 209, which is an incoming copy of an encoded video bitstream 204, and create an outgoing video sample stream 211 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a specific video coding / compression standard.

[0026] Figure 3 is an illustrative diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and decoder.

[0027] As shown in Figure 3, the framework 300 may include an encoder 301 and a decoder 351. The encoder 301 may include one or more input meshes 305, one or more meshes 310 with UV atlases, an occupancy map 315, a geometry map 320, an attribute map 325, and metadata 330. The decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, a decoded metadata 350, and a reconstructed mesh 360.

[0028] According to one aspect of the present disclosure, the input mesh 305 may include one or more frames, each of which may be preprocessed by a series of operations to generate a mesh 310 with a UV atlas. For example, the preprocessing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the preprocessing operations may be performed only on the encoder side and not on the decoder side.

[0029] The mesh 310 with a UV atlas may be a 2D mesh. The 2D mesh with a UV atlas may be a mesh in which each vertex of the mesh can be associated with a UV coordinate on the 2D atlas. The mesh 310 with a UV atlas may be processed based on sampling and converted into multiple maps. For example, the UV atlas 310 may be processed based on sampling the 2D mesh with the UV atlas and converted into an occupancy map, a geometry map, and an attribute map. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and sent to a decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be sent to the decoder.

[0030] In one embodiment, the decoder 351 may receive an encoded occupancy map, geometry map, and attribute map from the encoder. In addition to the embodiments described herein, the decoder 351 may use appropriate techniques and methods to decode the occupancy map, geometry map, and attribute map. In one embodiment, the decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, and decoded metadata 350. The input mesh 305 can be reconstructed into a reconstructed mesh 360 using one or more reconstruction filters and techniques based on the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350. In some embodiments, metadata 330 may be transmitted directly to the decoder 351, and the decoder 351 may use the metadata to generate a reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, and the decoded attribute map 345. Post-filtering techniques, including but not limited to remeshing, parameterization, tracking, and voxelization, may also be applied to the reconstructed mesh360.

[0031] An input mesh with a 2D UV atlas may have vertices, and each vertex of the mesh may have associated UV coordinates on the 2D atlas. Occupancy maps, geometry maps, and attribute maps can be generated by sampling one or more points / locations on the UV atlas. Each sample location may be occupied or unoccupied if its location lies inside a polygon defined by the mesh vertices. For each occupied sample, the corresponding 3D geometry coordinates and attributes of that sample can be calculated by interpolation from the associated polygon vertices.

[0032] According to one aspect of this disclosure, the sampling rate may be consistent throughout the entire 2D atlas. In some embodiments, the sampling rates for the u-axis and v-axis may be different, enabling anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, each of which may have a different sampling rate.

[0033] According to one aspect of this disclosure, the sampling rate of each region (or the entire 2D atlas) may be signaled in high-level syntax, including, but not limited to, sequence headers, frame headers, slice headers, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) may be selected from a set of pre-established rates assumed by both the encoder and the decoder. Because of the set of pre-established rates known by both the encoder and the decoder, signaling a particular sampling rate requires only sending an index within the pre-established rate set in the signal. Examples of such pre-established sets may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate of each region (or the entire 2D atlas) of a mesh frame may be predicted from the set of pre-established rates, from previously used sampling rates in other already coded regions of the same frame, or from previously used sampling rates in other already coded mesh frames.

[0034] In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be based on some characteristic of each region (or the entire 2D atlas). For example, the sampling rate may be based on activity, with a higher sampling rate for richly textured regions (or the entire 2D atlas) or high-activity regions (or the entire 2D atlas). As another example, a lower sampling rate may be set for smooth regions (or the entire 2D atlas) or low-activity regions (or the entire 2D atlas).

[0035] In some embodiments, the sampling rate for each region of a mesh frame (or the entire 2D atlas) may be signaled in such a way that prediction and direct signaling can be combined. The syntax may be configured to indicate whether the sampling rate is predicted or directly signaled. If predicted, the predictor sampling rate to be used may be further signaled. If directly signaled, the syntax to represent the rate value may be signaled.

[0036] Figure 4 is an illustrative figure 400 showing the projection of a 3D chart onto a 2D UV plane (2D UV chart) according to an embodiment of the present disclosure.

[0037] As shown in Figure 4, a 3D mesh can be divided into several charts, each of which can be projected onto a 2D plane. On the 2D UV plane, the correspondence between the mesh's attribute information, such as texture information, and the 3D vertices can be found.

[0038] As shown in Figure 4, UV coordinate information can be used to find texture information for 3D positions (such as vertices) on a 2D UV plane. Those skilled in the art should note that the chart subdivision process does not have to be the same for different mesh frames, resulting in various shapes of charts and assignments on the 2D plane across different mesh frames.

[0039] As described above, dynamic mesh sequences can require large amounts of data because they can consist of a considerable amount of information that changes over time. To achieve efficient compression, embodiments of this disclosure employ spatial and temporal alignment to obtain intra-frame and / or inter-frame correlations. In the prior art, many alignment methods, particularly temporal alignment, are based on sequential techniques. In other words, processing one frame depends on other frames, such as using the results of chart assignment from previous frames as a basis for assigning charts to the current frame. This dependency makes these methods unsuitable for parallelization and therefore slows down the encoding / decoding of meshes. Thus, a method to eliminate this dependency is needed to enable parallelization and improve encoding and decoding efficiency.

[0040] Embodiments of this disclosure relate to aligning mesh texture maps and / or geometry images with different levels of parallelization.

[0041] In one embodiment, maximum temporal parallelism of alignment is achieved. In this embodiment, each frame can be processed independently while maintaining temporal alignment between frames. In this embodiment, an intra-frame alignment scheme can be determined to spatially align the chart within a frame. Intra-frame alignment may be applied to all frames. Assuming that consecutive frames are similar, the aligned chart obtained from those frames will also be similar. Thus, both intra-frame and inter-frame alignment can be achieved, and frames can be processed independently and in parallel. In some embodiments, intra-frame alignment is, Multiple charts located in the same neighborhood within the frame Similar attributes such as color values Being This refers to. In some embodiments, interframe alignment is Multiple charts located in similar positions across different frames Similar attributes such as color value and shape Being It refers to.

[0042] In some embodiments, the geometric information and / or texture information of the chart may be used to design an in-frame alignment scheme.

[0043] When spatially aligning charts in a geometric image, the charts can be spatially aligned using any information about the chart, such as its 3D position, size, normals, and shape. For example, the position of a chart can be calculated by averaging the 3D coordinates of all vertices in the chart, or approximated by averaging only the 3D coordinates of the chart boundary vertices, and the position can be represented by its centroid. The charts can then be ordered and rasterized by their positions using the Morton code / Z-order of their centroid coordinates.

[0044] In addition to the geometric information mentioned above, texture information such as the chart's color or histogram can also be taken into consideration when spatially aligning the chart, if a texture map is available.

[0045] When spatially aligning charts in a texture map, both geometric and texture information may be used. The rationale for using geometric information to align texture charts is that if the geometric changes between consecutive frames are small, the textures of geometrically nearby charts will be similar from those frames. For texture information, in addition to the histogram described above, the average RGB / YUV pixel values ​​of the chart representing its color information may be used. In some embodiments, the average RGB / YUV pixel values ​​of the chart may be calculated by averaging all pixel values ​​in the chart, or may be approximated only by averaging those pixel values ​​on the chart boundaries. The charts can then be ordered and rasterized by the Morton code / Z order of their average RGB / YUV pixel values. In some embodiments, the charts can be sorted and rasterized by other orders of the average RGB / YUV pixel values, such as using the corresponding positions in the visible spectrum.

[0046] In some embodiments, a temporal alignment algorithm for positioning a chart on a 2D image for either geometric or texture information may allow referencing the results of a previous frame. However, certain limitations may be imposed on such referencing so that parallel processing is still possible to some extent. In one embodiment, a frame may be divided into subgroups that can be processed in parallel.

[0047] Figure 5 illustrates an exemplary hierarchical coding structure with several layers or levels illustrating inter-group or sub-group dependencies for limited parallel processing of frames according to an embodiment of the present disclosure.

[0048] As shown in Figure 5, the structure 500 includes I0 and B16 at the lowest level, and B1, B3, B5, B7, B9, B11, B13, and B15 at the highest level. In one embodiment, the higher level frames are temporally aligned. of For the result to,Only lower-level frames Can be referenced , frames of the same or higher level Can be referenced No. For example, B8 is the temporal alignment result of I0. Can be referenced Therefore, they are at the same level. multiple frame of, parallel to process It is possible .

[0049] In some embodiments, the mesh frame may be divided into independent groups of images (GoPs) (e.g., B1, B2, ..., B16, I0) so that each GoP can be processed in parallel. Within each GoP, the aforementioned prediction structure (either hierarchical or linear) (e.g., structure 500) may be used.

[0050] In another embodiment, the first mesh frame may be processed independently, and all other frames may be processed based on the first frame. Thus, only a single-frame latency may be introduced.

[0051] Figure 6 is a flowchart showing a process 600 for parallel dynamic mesh alignment according to an embodiment of the present disclosure.

[0052] In operation 605, it may be determined that a temporal alignment exists between one or more frames. In some embodiments, the temporal alignment is determined between frames in the received input mesh, which includes multiple polygons describing the surface of a volume object.

[0053] In some embodiments, determining that a temporal alignment exists between one or more frames is based on the temporal alignment results of one or more previous frames.

[0054] In operation 610, based on the determination that a temporal alignment exists between one or more frames, an in-frame alignment scheme for spatially aligning the chart within the frame may be determined.

[0055] In some embodiments, determining in-frame alignment may include spatially aligning charts within a frame based on at least one of the 3D position of each chart, the size of each chart, or the shape of each chart. In some embodiments, determining in-frame alignment may include spatially aligning texture charts within a frame based on geometric information associated with the charts, and at least one of the histogram of each chart, the average RGB pixel value of each chart, or the average YUV pixel value of each chart.

[0056] In operation 615, an intra-frame alignment scheme may be applied to one or more corresponding charts in one or more frames for inter-frame alignment.

[0057] In some embodiments, one or more corresponding charts in one or more frames may be charts arranged similarly across one or more frames and having at least one of similar color values ​​or similar shapes.

[0058] In operation 620, one or more frames may be processed in parallel based on intra-frame alignment and inter-frame alignment.

[0059] In some embodiments, processing one or more frames in parallel may involve dividing one or more frames into hierarchical subgroups for parallel processing, where hierarchical subgroups at the same level may be processed in parallel, and higher-level frames may refer only to lower-level frames with respect to the temporal alignment results.

[0060] In some embodiments, processing one or more frames in parallel may include a first frame that is processed independently of the one or more frames, and the one or more frames are processed in parallel based on the first frame.

[0061] In some embodiments, processing one or more frames in parallel may include dividing one or more frames into one or more groups of images (GoPs), and processing each of the GoPs in parallel. In some embodiments, one or more frames may be divided into one or more hierarchical GoP structures.

[0062] The techniques described above can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 7 shows a computer system 700 suitable for implementing a particular embodiment of the present disclosure.

[0063] Computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly, to create code that contains instructions that can be executed directly or via interpretation, microcode execution, etc., by a computer central processing unit (CPU), graphics processing unit (GPU), etc.

[0064] Instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, and Internet of Things devices.

[0065] The components of the computer system 700 shown in Figure 7 are examples and are not intended to imply any limitations on the scope or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as dependent on or requiring any one component or combination of components shown in any non-limiting embodiment of the computer system 700.

[0066] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users via, for example, tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, applause, etc.), visual input (gestures, etc.), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographic images acquired from a still image camera, etc.), or video (2D video, 3D video including stereoscopic video, etc.).

[0067] The input human interface device may include one or more of the following (only one of each is shown): a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708.

[0068] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 710, data glove, or joystick 705, although tactile feedback devices that do not function as input devices may also exist). For example, such devices may include audio output devices (e.g., speaker 709, headphones (not shown)), visual output devices (e.g., screen 710 including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touchscreen input functionality, each with or without tactile feedback functionality, some of which may output two-dimensional visual output or output beyond three dimensions through means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0069] The computer system 700 may also include human-accessible storage devices and media associated with storage devices such as CD / DVD ROM / RW 720 with media such as CD / DVD 721, thumb drives 722, removable hard drives or solid-state drives 723, legacy magnetic media such as tapes and floppy disks (not shown), and optical media such as dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0070] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0071] The computer system 700 may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. Networks can further be local, wide-area, metropolitan, automotive, and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including Wi-Fi, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks for television including cable television, satellite television, and terrestrial television, and automotive and industrial networks including CANBus. Certain networks generally require an external network interface adapter connected to a specific general-purpose data port or peripheral bus 749 (e.g., a USB port on the computer system 700), while others are generally integrated into the core of the computer system 700 by connections to the system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications can be unidirectional, receive only (e.g., broadcast television), transmit only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using local or wide-area digital networks. Such communications can include communications to cloud computing environments. Specific protocols and protocol stacks can be used on each of those networks and network interfaces, as described above.

[0072] The aforementioned human interface device, human-accessible storage device, and network interface 754 can be connected to the core 740 of the computer system 700.

[0073] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field-programmable gate areas (FPGAs) 743, hardware accelerators 744 for specific tasks, and so on. These devices may be connected via a system bus 748, along with internal mass storage units 747 such as read-only memory (ROM) 745, random access memory 746, and internal non-user-accessible hard drives, SSDs, etc. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may be included in the core 740.

[0074] The CPU 741, GPU 742, FPGA 743, and accelerator 744 can execute specific instructions that, in combination, can constitute the aforementioned computer code. This computer code can be stored in ROM 745 or RAM 746. Transient data can also be stored in RAM 746, while persistent data can be stored, for example, in the internal mass storage unit 747. High-speed storage and retrieval to any of the memory devices can be enabled by using cache memory that can be closely associated with one or more CPUs 741, GPUs 742, mass storage units 747, ROM 745, RAM 746, etc.

[0075] A computer-readable medium may contain computer code for performing various computer implementation operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0076] For example, but not limited to, a computer system having the architecture of computer system 700, in particular core 740, may provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of core 740 that are non-temporary in nature, such as the core internal mass storage 747 and ROM 745. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 740. The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause core 740, specifically the processor (including CPU, GPU, FPGA, etc.) in the core, to execute specific processes or specific parts of specific processes as described herein, including defining data structures stored in RAM 746 and modifying such data structures according to processes defined by the software. In addition, or as an alternative, a computer system may also provide functionality resulting from logic (e.g., accelerator 744) wired or otherwise embodied in a circuit, which can operate in place of or with software to perform a particular process or a particular part of a particular process described herein. Where necessary, references to software may encompass logic and vice versa. References to computer-readable media may, where necessary, encompass circuits that store software for execution (such as integrated circuits (ICs)), circuits that embody logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software.

[0077] While this disclosure describes several non-limiting embodiments, there are many variations, permutations, and alternative equivalents that fall within the scope of this disclosure. Those skilled in the art will therefore understand that numerous systems and methods not expressly shown or described herein can be devised to embody the principles of this disclosure and thus fall within its spirit and scope. [Explanation of Symbols]

[0078] 100 Systems 110 First terminal 120 Second terminal 130 Second terminal 140 Second terminal 150 Networks 200 Streaming Systems 201 Video Sources 202 Sample Streams 203 Encoder 204 encoded video bitstream 205 Streaming Servers 206 Streaming Clients 209 video bitstreams 210 Video Decoders 211 Outgoing video sample stream 212 displays 213 Capture Subsystem 301 Encoder 305 Input Mesh Mesh with 310 UV Atlas 315 Occupation Map 320 Geometry Maps 325 Attribute Map 330 Metadata 335 Decrypted Occupation Map 340 decoded geometry maps 345 Decrypted attribute map 350 decrypted metadata 351 Decoder 360 Reconstructed Mesh 700 Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Mike 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 720 CD / DVD ROM / RW 721 CD / DVD and other media 722 Thumb Drive 723 Removable hard drive or solid state drive 740 cores 741 Central Processing Unit (CPU) 742 Graphics Processing Unit (GPU) 743 Field-Programmable Gate Area (FPGA) 744 Hardware accelerators for specific tasks 745 Read-only memory (ROM) 746 Random Access Memory (RAM) 747 SSD and other internal high-capacity storage units 748 System Bus 749 Local Buses 750 Graphics Adapter 754 Network Interfaces 755 Cloud Computing Environment

Claims

1. A method for dynamic mesh alignment, wherein the method is performed by at least one processor, The steps include receiving an input mesh containing a plurality of polygons describing the surface of a volume object, wherein the input mesh includes charts representing the polygons, and each of the plurality of charts obtained from the input mesh constitutes a reference frame and a current frame that follow a time series. A first determination step of determining whether the chart at the same position in the reference frame and the current frame has similar attributes, If the first determination step determines that the chart has similar attributes, the first determination step determines to apply an inter-frame alignment scheme that determines the method of assigning the chart at the same position in the reference frame as the method of assigning the chart at the same position in the current frame, In the current frame, the first assigned chart obtained in the first determination step is determined to be the chart of the reference block, and a second determination step is performed to determine whether the current block is adjacent to the reference block and has similar attributes to the reference block. If the second determination step determines that the reference block has the same attributes as the current block, the step of applying an in-frame alignment scheme determines that the first assigned chart is the chart of the current block. A method that includes this.

2. Having similar attributes means that at least one of the color value and the shape of the chart is the same, The method according to claim 1, wherein the in-frame alignment scheme is applied based on geometric information including at least one of the three-dimensional (3D) position of the chart, the size of the chart, and the shape of the chart.

3. The method according to claim 2, wherein the in-frame alignment scheme is based on the geometry information and is applied based on at least one of the histogram of the chart, the average RGB pixel values ​​of the chart, and the average YUV pixel values ​​of the chart.

4. The process further includes dividing multiple frames into multiple layers, each containing one or more frames, The method according to claim 1, wherein the first decision step determines whether or not to apply the interframe alignment scheme to frames included in the hierarchy by referring only to frames included in a hierarchy at a lower level than the hierarchy.

5. The steps include dividing multiple frames into groups of images (GoPs), each containing multiple frames, The method further includes dividing the multiple frames included in the Group of P into multiple layers, each containing one or more frames, The method according to claim 1, wherein the first decision step determines whether or not to apply the interframe alignment scheme to frames included in the hierarchy by referring only to frames included in a hierarchy at a lower level than the hierarchy.

6. The method according to claim 4, wherein a plurality of frames included in the hierarchy of the same level are processed in parallel by the inter-frame alignment scheme.

7. The method according to claim 5, wherein a plurality of frames included in the hierarchy of the same level are processed in parallel by the inter-frame alignment scheme.

8. A device for dynamic mesh alignment, wherein the device is At least one memory configured to store program code, A device comprising: at least one processor configured to read the program code and to be instructed by the program code to perform the method according to any one of claims 1 to 7.

9. A computer program, which, when executed by one or more processors of a device for dynamic mesh alignment, includes one or more instructions that cause the one or more processors to perform the method according to any one of claims 1 to 7.

Images