Submesh Zipper Ring

Abstract

This document describes Zippering SEI, a new SEI message for the V-DMC standard. Decoders can use Zippering SEI messages for mesh reconstruction, and in the case of multiple submeshes, Zippering SEI provides a way to reduce common artifacts such as holes and cracks on the mesh surface that result from independent submesh coding.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application Serial No. 63 / 269,911, filed on March 25, 2022, entitled "MESH ZIPPERING", and is a continuation - in - part of U.S. Patent Application No. 17 / 987,847, filed on November 15, 2022, entitled "MESH ZIPPERING", and also claims priority under 35 U.S.C. § 119 to U.S. Provisional Patent Application Serial No. 63 / 513,305, filed on July 12, 2023, entitled "SUB - MESH ZIPPERING". All of these documents are hereby incorporated by reference in their entirety for all purposes.

[0002] The present invention relates to 3D graphics. Specifically, the present invention relates to the encoding of 3D graphics.

Background Art

[0003] In recent years, new methods for compressing volumetric content, such as point clouds, based on projections from 3D to 2D are being standardized. This method, also known as V3C (Visual Volumetric - based Video Compression), maps 3D volumetric data into a plurality of 2D patches, further places the patches within an atlas image, and then encodes them with a video encoder. The atlas image corresponds to the geometry of the points, their respective textures, and an occupancy map indicating which positions should be considered for point - cloud reconstruction.

[0004] In 2017, MPEG issued a call for proposals (CfP) for point cloud compression. After evaluating several proposals, MPEG is currently considering two different point cloud compression techniques: a 3D native encoding technique (based on octrees and similar encoding methods) or one that performs conventional video encoding after projection from 3D to 2D. For dynamic 3D scenes, MPEG uses experimental model software (TMC2) based on patch surface modeling, projection of patches from 3D to 2D images, and encoding of the 2D images using a video encoder such as HEVC. This method has been proven to be more efficient than native 3D encoding and to achieve competitive bitrates with acceptable quality.

[0005] Following the success of projection-based 3D point cloud coding (also known as video-based or V-PCC), this standard is expected to include further 3D data, such as 3D meshes, in future versions. However, the current version of this standard is only suitable for transmitting a series of unconnected points and therefore lacks a mechanism for transmitting the point connectivity required for 3D mesh compression.

[0006] Methods have also been proposed to extend the functionality of V-PCC to meshes. One possible method is to encode vertices using V-PCC and then encode connectivity using a mesh compression method such as TFAN or Edgebreaker. The limitation of this method is that the original mesh must be dense so that the point cloud generated from the vertices can be efficiently encoded after projection without becoming low-density. Furthermore, since the order of vertices affects the encoding of connectivity, different methods for reorganizing mesh connectivity have been proposed. Another method for encoding sparse meshes is to encode vertex positions in 3D using RAW patch data. Since RAW patches directly encode (x,y,z), in this method all vertices are encoded as RAW data, while connectivity is encoded by a similar mesh compression method as described above. Note that RAW patches can transmit vertices in any preferred order, and therefore the order resulting from connectivity encoding can be used. While this method can encode sparse point clouds, RAW patches are inefficient at encoding 3D data, and further data such as triangular face attributes may be lost from this method. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] U.S. Patent Application No. 17 / 161,300 [Patent Document 2] U.S. Provisional Patent Application No. 62 / 991,128 [Overview of the Initiative]

[0008] This document describes Zippering SEI, a new SEI message for the V-DMC standard. Decoders can use Zippering SEI messages for mesh reconstruction, and in the case of multiple submeshes, Zippering SEI provides a way to reduce common artifacts such as holes and cracks on the mesh surface that result from independent submesh coding.

[0009] In one embodiment, a method programmed into the device's non-temporary memory includes determining one or more boundary points in one or more submeshes, selecting one zippering implementation from a plurality of mesh zippering implementations, and merging each of the one or more boundary points with a corresponding point based on the selected mesh zippering implementation. The plurality of mesh zippering implementations include a predetermined search distance implementation, a maximum distance implementation for all submeshes and all frames, a maximum distance implementation per frame, a maximum distance implementation per submesh, the maximum distance for each boundary vertex, and a matching index between two boundary vertices. The predetermined search distance implementation uses a user-defined search distance. The predetermined search distance implementation uses a computer-generated search distance using artificial intelligence and machine learning. The predetermined search distance implementation, the maximum distance implementation for all submeshes and all frames, the maximum distance implementation per frame, the maximum distance implementation per submesh, and the maximum distance implementation for each boundary vertex each include limiting the search range of points based on distance, respectively. The distance is the radius of the spherical search area. Determining one or more boundary points in one or more submeshes involves determining each point on an edge that is not connected to two triangles.

[0010] In another embodiment, the device includes non-temporary memory that stores an application for determining one or more boundary points in one or more submeshes, selecting one zippering implementation from a plurality of mesh zippering implementations, and merging each of the one or more boundary points with a corresponding point based on the selected mesh zippering implementation, and a processor coupled to the memory and configured to process the application. The plurality of mesh zippering implementations include a predetermined search distance implementation, a maximum distance implementation in all submeshes and all frames, a maximum distance implementation per frame, a maximum distance implementation per submesh, a maximum distance for each boundary vertex, and a matching index between two boundary vertices. The predetermined search distance implementation uses a user-defined search distance. The predetermined search distance implementation uses a computer-generated search distance using artificial intelligence and machine learning. The predetermined search distance implementation, the maximum distance implementation in all submeshes and all frames, the maximum distance implementation per frame, the maximum distance implementation per submesh, and the maximum distance implementation for each boundary vertex each include limiting the search range of points based on distance, respectively. The distance is the radius of the spherical search area. Determining one or more boundary points in one or more submeshes involves determining each point on an edge that is not connected to two triangles.

[0011] In another embodiment, the system includes an encoder configured to encode content, and a decoder configured to determine one or more boundary points in one or more submeshes, select one zippering implementation from a plurality of mesh zippering implementations, and merge each of the one or more boundary points with a corresponding point based on the selected mesh zippering implementation. The plurality of mesh zippering implementations include a predetermined search distance implementation, a maximum distance implementation for all submeshes and all frames, a maximum distance implementation per frame, a maximum distance implementation per submesh, the maximum distance for each boundary vertex, and a matching index between two boundary vertices. The predetermined search distance implementation uses a user-defined search distance. The predetermined search distance implementation uses a computer-generated search distance using artificial intelligence and machine learning. The predetermined search distance implementation, the maximum distance implementation for all submeshes and all frames, the maximum distance implementation per frame, the maximum distance implementation per submesh, and the maximum distance implementation for each boundary vertex each include limiting the search range of points based on distance, respectively. The distance is the radius of the spherical search area. Determining one or more boundary points in one or more submeshes involves determining each point on an edge that is not connected to two triangles. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows flowcharts of mesh zippering methods according to several embodiments. [Figure 2] This figure shows images of zippering configurations according to several embodiments. [Figure 3] This figure shows images illustrating the advantages and disadvantages of various zipper ring implementations according to several embodiments. [Figure 4] This is a block diagram of an exemplary computer device configured to implement a mesh zippering method according to several embodiments. [Figure 5] This is a diagram of a V-DMC V3C decoder according to several embodiments. [Figure 6]This figure shows images for discovering boundary vertices according to several embodiments. [Figure 7] This figure shows images of distance zippering method 0 according to several embodiments. [Figure 8] This figure shows images of Method 1 of distance zippering according to several embodiments. [Figure 9] This figure shows images of the distance zippering method 2 according to several embodiments. [Figure 10] This figure shows images of the distance zippering method 3 according to several embodiments. [Figure 11] This figure shows images of distance zippering method 4 according to several embodiments. [Figure 12] This figure shows images of the distance zippering method 5 according to several embodiments. [Figure 13] This figure shows the results of the zippering algorithm according to several embodiments. [Modes for carrying out the invention]

[0013] This specification describes a method, also known as zippering, that improves mesh reconstruction by correcting the position of vertices at patch boundaries to ensure that no gaps exist between adjacent patches. Six different methods for performing post-processing operations, as well as syntactic elements and semantics for sending filter parameters, are disclosed. Hierarchical methods indicate geometry distortions that may cause gaps between patches. Values ​​are sent per frame, per patch, or per boundary object. The number of bits used to encode the values ​​also depends on the previous geometry distortion. Some methods send index matches instead of geometry distortions. Matching indices are sent for each boundary vertex, but methods are also implemented that send only one index per pair.

[0014] As described in U.S. Patent Application No. 17 / 161,300, “Projection-Based Mesh Compression,” filed on 28 January 2021, and U.S. Provisional Patent Application No. 62 / 991,128, “Projection-Based Mesh Compression,” filed on 18 March 2020, zippering addresses the problem of vertex displacement, and these documents are incorporated herein by reference in their entirety for all purposes.

[0015] Figure 1 shows a flowchart of a method of mesh zippering according to some embodiments. At step 100, boundary points are discovered. The boundary points can be discovered by any method. After the boundary points are found, mesh zippering is performed. Mesh zippering includes determining the neighborhood of the boundary vertices and merging specific neighboring boundary vertices. Mesh zippering can be performed using one or more different implementations. To eliminate gaps in the mesh, mesh zippering is utilized to find matching points / vertices. To find the matching points, the search is performed by looking for neighboring points of the points in 3D space. The search range can be limited (e.g., based on a fixed value such as a maximum distance of 5 or based on the maximum distortion). Thus, if the distance is greater than 5, the point will not find its match. The search can also be limited based on the maximum distortion. The maximum distortion of each point can be different. Mesh zippering for each sequence can limit the search using distance or maximum distortion. Searching based on the maximum distortion may take too much time or computational cost for the entire sequence, so it may be better to search frame by frame. For example, most frames are searched based on a fixed value (e.g., the maximum distance), while one specific frame is searched based on the maximum distortion. The maximum distortion can be implemented for each patch. For example, there may be large patches with less distortion. In another example, there may be small patches with more distortion. The distortion can be sent for each case of boundary / boundary point. In this implementation, the search is not performed and the distortion is applied when received. However, more distortion information is sent, and thus the bitrate is higher, but the mesh reconstruction becomes better (e.g., more accurate).

[0016] In step 102, perform zippering for each frame. As explained, zippering performs a search for each point within a frame using the maximum distortion. By performing zippering for each frame instead of the entire sequence, some processing is performed without using distortion information, and only frames with high distortion use zippering based on the maximum distortion. In step 104, perform zippering for each patch. By performing zippering for each patch, some processing is performed without using distortion information, and only patches with high distortion use zippering based on the maximum distortion. In step 106, perform zippering for each boundary point. In zippering for each boundary point, no search is performed, and the distortion is applied when received. However, the amount of distortion information transmitted increases, and thus the bitrate increases, but the mesh reconstruction becomes better (e.g., more accurate). In step 108, perform zippering border point match. Indices that match each other are transmitted. The decoder determines where the patch extends in 3D space based on the matching vertices (e.g., averaging the distance between two points and selecting one of the points). The zippering implementation can be selected in any way such as programming based on a detected set of criteria (e.g., detecting that a frame or patch contains a distortion amount higher than a threshold) or adaptively selecting.

[0017] In step 110, merge the vertices. The merging of vertices can be performed in any way. In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of steps is changed. The zippering implementation is performed on the decoder side.

[0018] Figure 2 shows images of zippering in several embodiments. Image 200 shows that gaps may exist between boundary points. In Image 202, zippering is applied to boundary vertices to narrow or eliminate gaps. As described, zippering involves classifying vertices as boundary vertices or non-boundary vertices, determining the neighbors of boundary vertices, and merging neighboring boundary vertices. Image 204 shows a decoded image without gaps using zippering.

[0019] Figure 3 shows images illustrating the advantages and disadvantages of various zippering implementations according to several embodiments. Image 300 is the original image. Image 302 shows 12.172 Mbps without zippering. Image 304 shows 12.222 Mbps with zippering. Image 306 shows 13.253 Mbps with zippering. Image 308 shows 13.991 Mbps with zippering. Zippering can fill gaps in areas such as the face, hair, and ears.

[0020] This section explains the latest zippering syntax. JPEG2026524897000002.jpg79160 JPEG2026524897000003.jpg199151 JPEG2026524897000004.jpg201154 JPEG2026524897000005.jpg36170gs_zippering_max_match_distance[k] specifies the value of zipperingMaxMatchDistance[k], which is a variable used to process the current mesh frame of the geometry smoothing instance at index k when the zippering filtering process is used. If gs_zippering_send_border_point_match[k] is equal to 1, it specifies that zippering will be applied to the boundary point of the geometry smoothing instance at index k by sending the matching index. If gs_zippering_send_border_point_match[k] is equal to 0, it specifies that zippering by sending a matching index will not be applied to the boundary point of the geometry smoothing instance at index k. The default value for gs_zippering_send_border_point_match[k] is 0. gs_zippering_number_of_patches[k] indicates the number of patches being filtered by the current SEI message. The value of gs_zippering_number_of_patches must be within the range of 0 to MaxNumPatches[frameIdx]. The default value of gs_zippering_number_of_patches is 0. gs_zippering_number_of_border_points[k][p] represents numBorderPoints[p], which is the number of border points of the patch at index p. gs_zippering_border_point_match_patch_index[k][p][b] specifies the value of zipperingBorderPointMatchPatchIndex[k][p][b], which is a variable used to process the current boundary point of index b in the current patch of index p in the current mesh frame of the geometry smoothing instance of index k when the zippering filtering process is used. gs_zippering_border_point_match_border_point_index[k][p][b] specifies the value of zipperingBorderPointMatchBorderPointIndex[k][p][b], which is a variable used to process the current boundary point of index b in the current patch of index p in the current mesh frame of the geometry smoothing instance of index k when the zippering filtering process is used. If gs_zippering_send_distance_per_patch[k] is equal to 1, it specifies that zippering is applied to the boundary point of the geometry smoothing instance at index k by sending the distance per matching patch. If gs_zippering_send_distance_per_patch[k] is equal to 0, it specifies that zippering by sending the distance per matching patch is not applied to the boundary point of the geometry smoothing instance at index k. The default value for gs_zippering_send_distance_per_patch[k] is 0. If gs_zippering_send_distance_per_border_point[k] is equal to 1, it specifies that zippering is applied to the boundary points of the geometry smoothing instance at index k by sending the distance for each matching boundary point. If gs_zippering_send_distance_per_border_point[k] is equal to 0, it specifies that zippering by sending the distance for each matching boundary point is not applied to the boundary point of the geometry smoothing instance at index k. The default value for gs_zippering_send_distance_per_border_point[k] is 0. gs_zippering_max_match_distance_per_patch[k] specifies the value of zipperingMaxMatchDistancePerPatch[k][p], which is a variable used to process the current patch at index p within the current mesh frame of the geometry smoothing instance at index k when the zippering filtering process is used. gs_zippering_border_point_distance[k][p][b] specifies the value of zipperingMaxMatchDistancePerBorderPoint[k][p][b], which is a variable used to process the current boundary point of index b in the current patch of index p in the current mesh frame of the geometry smoothing instance of index k when the zippering filtering process is used.

[0021] As explained, trade-offs can be achieved by selecting different zippering methods. Sending a single distance for the entire sequence uses only one SEI message, while sending distances per frame, patch, or boundary distance involves sending an SEI message per frame. However, the choice of zippering method can make or break gaps, so subjective influence is considered significant.

[0022] Figure 4 is a block diagram of an exemplary computer device configured to implement a mesh zippering method according to several embodiments. The computer device 400 can be used for acquiring, storing, calculating, processing, communicating and / or displaying information such as images and videos, including 3D content. The computer device 400 can implement any form of encoding / decoding. Generally, a hardware structure suitable for implementing the computer device 400 includes a network interface 402, memory 404, a processor 406, (one or more) I / O devices 408, a bus 410, and a storage device 412. The choice of processor is not critical as long as a suitable processor of sufficient speed is selected. The memory 404 can be any conventional computer memory known in the art. The storage device 412 can include a hard drive, CD-ROM, CDRW, DVD, DVDRW, high-definition disk / drive, Ultra HD drive, flash memory card, or any other storage device. The computer device 400 can include one or more network interfaces 402. Examples of network interfaces include a network card connected to Ethernet or other types of LANs. The (single or duplicate) I / O devices 408 may include one or more of the following: a keyboard, mouse, monitor, screen, printer, modem, touchscreen, button interface, and other devices. The storage devices 412 and memory 404 likely store the (single or duplicate) mesh zippering application 430 used to implement mesh zippering, and process it as the application would normally be processed. The computer device 400 may also include more or fewer components than those shown in Figure 4. In some embodiments, mesh zippering hardware 420 is included. The computer device 400 in Figure 4 includes the application 430 and hardware 420 for mesh zippering implementation, but the mesh zippering method may also be implemented in the computer device in hardware, firmware, software, or a combination of these.For example, in some embodiments, the mesh zippering application 430 is programmed in memory and executed using a processor. In another example, in some embodiments, the mesh zippering hardware 420 is programmed hardware logic including gates specifically designed to implement the mesh zippering method.

[0023] In some embodiments, the (single or multiple) mesh zippering application 430 includes multiple applications and / or modules. In some embodiments, a module also includes one or more submodules. In some embodiments, it may include fewer or more modules.

[0024] Examples of suitable computer devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants (PDAs), cellular / mobile phones, smart home appliances, game consoles, digital cameras, digital camcorders, camera phones, smartphones, portable music players, tablet computers, mobile devices, video players, video disc writers / players (DVD writers / players, high-definition disc writers / players, ultra-high-definition disc writers / players, etc.), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smartwatches), vehicles (e.g., autonomous vehicles), or any other suitable computer device.

[0025] Since submeshes are processed individually, the reconstructed mesh may have gaps between submeshes due to quantization and encoding artifacts of vertex positions, resulting in mismatches between submesh boundaries and potentially leading to visual artifacts that include gaps between submeshes. However, if we assume that the reconstructed mesh is a manifold, gaps should not exist. This is a common problem in mesh generation using range images, and a zippering algorithm that merges triangular vertices belonging to submesh boundaries is the solution.

[0026] As explained, zippering modifies the vertex positions of boundary vertices to match the edges of triangles that are encoded separately. Zippering can be applied to various patches that are generated. The same concept can be applied to submeshes. Furthermore, since this is a 3D reconstruction problem as shown in Figure 5, and many other zippering methods can be implemented, new SEI messages can be applied for zippering operations, similar to the geometry smoothing concept of V-PCC.

[0027] Zippering The zippering method searches for boundary point / vertex matches (e.g., within a 3D sphere) from different submeshes according to a given geometry distortion distance, which can be defined for each sequence, frame, submesh, or boundary vertex. Furthermore, to reduce the complexity of the search, the transmission of explicit matches between boundary vertex indices has been added. The encoder can choose from the following six different zippering methods: (zipperingMethod=0) User-defined search distance for the entire sequence, (zipperingMatchMaxDistance=5(default)), (zipperingMethod=1) A single search distance obtained from the maximum geometric distortion of all boundary vertices in all submeshes and all frames, (zipperingMethod=2) Frame-by-frame search distance obtained from the maximum geometric distortion of all boundary vertices within all submeshes, (zipperingMethod=3) A set of search distances per submesh per frame obtained from the maximum geometric distortion of all boundary vertices. (zipperingMethod=4) Geometric distortion of all boundary vertices, (zipperingMethod=5) The matching index between two boundary vertices.

[0028] User-defined searches are also called sequence-based searches, where the search values ​​are fixed (for example, searching with a 3D radius of 2).

[0029] The maximum geometry distortion search first performs an analysis of the sequence to determine the geometry distortion for each frame (e.g., 1, 4, 3, etc.), and then uses the maximum value (e.g., 4) for the entire sequence.

[0030] The implementation of maximum geometry distortion per frame involves determining the distance for each boundary vertex within a single frame and using the maximum value.

[0031] The maximum geometric distortion per submesh is similar to the per-frame implementation, but instead is based on each individual submesh.

[0032] In the boundary point implementation, the distance from each boundary vertex to another vertex is sent for each boundary point.

[0033] In the boundary pair implementation, instead of sending distance, matching boundary pairs are sent (for example, vertex 1 matches vertex 3 in submesh 4).

[0034] Figure 6 shows images for discovering boundary vertices according to several embodiments. Boundary vertices (e.g., 600, 602, and 604) are discovered by determining each vertex on an edge that is not connected to two triangles.

[0035] Figure 7 shows images of distance zippering method 0 according to several embodiments. In distance zippering method 0, a user-defined distance is used for all frames / subframes. For example, a radius of 5 is used. As shown in the figure, a radius of 5 706 can be large enough for some points to connect to boundary vertices. For example, point 704 is within a radius of 5 706 from its nearest boundary vertex 604. However, some points may not be within a radius of 5 to connect to boundary vertices. For example, points 700 and 702 are not within a radius of 5 706 from their respective nearest boundary vertices 600 and 602. In some embodiments, the user-defined distance is determined by artificial intelligence (AI) / machine learning (ML).

[0036] Figure 8 shows images of distance zippering method 1 according to several embodiments. In distance zippering method 1, the maximum distance for all frames / submeshes is determined. For example, the distance from boundary vertex 600 to boundary vertex 700 is 7, the distance from boundary vertex 602 to boundary vertex 702 is 10, and the distance from boundary vertex 604 to boundary vertex 704 is 5. Therefore, since 10 is the longest distance (e.g., the maximum distance), a radius of 706' of 10 is used. Using a radius of 706' of 10, all three boundary vertices 600, 602, and 604 find matching vertices 700, 702, and 704, respectively. In some cases, there may be multiple matching vertex candidates (e.g., vertices 704 and 708), and therefore one of the vertex candidates is selected (e.g., the closest vertex is selected, or the selection is based on other criteria).

[0037] Figure 9 shows images of distance zippering method 2 according to several embodiments. In distance zippering method 2, the maximum distance for each frame / submesh is determined. Method 2 is similar to method 1 except that a new maximum distance is determined for each frame / submesh instead of using the same maximum distance for all frames / submesh.

[0038] Figure 10 shows images of distance zippering method 3 according to several embodiments. In distance zippering method 3, the maximum distance for each submesh is determined. Method 3 is similar to method 2, except that a new maximum distance is determined for each submesh.

[0039] As described above, in the first submesh, for example, the maximum distances from vertices 600, 602, and 604 to vertices 700, 702, and 704 are 10. In the second submesh, for example, the maximum distances from vertices 1000, 1002, and 1004 to vertices 1010, 1012, and 1014 are 4, and as a result the radius 1006 becomes 4. Therefore, the smaller the radius, the smaller the search area of ​​the second submesh.

[0040] Figure 11 shows images of distance zippering method 4 according to several embodiments. In distance zippering method 4, the maximum distance to each boundary point is determined. In other words, the distance from each boundary vertex to the nearest boundary vertex is determined. For example, the distance from boundary vertex 600 to boundary vertex 700 is 7, and therefore the maximum distance or radius 706'' is 7. The distance from boundary vertex 602 to boundary vertex 702 is 10, and therefore the radius 706' is 10. The distance from boundary vertex 604 to boundary vertex 704 is 5, and therefore the radius 706''' is 5. Similarly, the distance from boundary vertex 1000 to boundary vertex 1010 is 3, and therefore the maximum distance or radius 1006' is 3. The distance from boundary vertex 1002 to boundary vertex 1012 is 2, and therefore the radius 1006'' is 2. The distance from boundary vertex 1004 to boundary vertex 1014 is 4, and therefore the radius 1006 is 4. All these separate distances are sent to another device that provides more accurate results while increasing the bitrate.

[0041] Figure 12 shows images of distance zippering method 5 according to several embodiments. In distance zippering method 5, matches between boundary points are determined and transmitted. Pairs of matching points are indicated by submesh index and boundary point index. For example, pairs 1200, 1202, 1204, 1210, 1212, and 1214 are shown. As described herein, these pairs can be determined by determining the nearest neighbor vertex to the boundary vertex based on distance.

[0042] After distance information, vertex information, and / or pair information have been determined, this information can be transmitted to another device (e.g., a decoder).

[0043] High-level syntax The V3C standard allows for the generation of a new SEI message called a zippering message with a payload equal to 68, and this is still available. JPEG2026524897000006.jpg199165 JPEG2026524897000007.jpg197156 JPEG2026524897000008.jpg204162 The JPEG2026524897000009.jpg13162SEI message has the following syntax: JPEG2026524897000010.jpg178168 JPEG2026524897000011.jpg200170 JPEG2026524897000012.jpg184169

[0044] Semantics SEI messages specify the recommended zippering method and associated parameters that can be used for vertex processing after the reconstruction of the current mesh frame in order to improve the quality of the reconstructed geometry.

[0045] Up to 256 (or another number) different zippering instances can be specified to be used with each mesh frame. These instances are indicated using the array ZipperingMethod. The zippering instances that the decoder can select to operate are outside the scope of this document. At the start of each sequence, ZipperingMethod[i] is set to equal to 0, where i corresponds to the zippering instance index, ranging from 0 to 255, including both end values. ZipperingMethod[i] being equal to 0 means that no zippering filter is specified for the zippering instance at index [i]. zp_persistence_flag specifies the persistence of zippering SEI messages for the current layer. zp_persistence_flag being equal to 0 specifies that zippering SEI messages apply only to the current decoded atlas frame.

[0046] Let aFrmA be the current atlas frame. A zp_persistence_flag equal to 1 specifies that the zippering SEI message persists in output order for the current layer until one of the following conditions is met: a new CAS starts, the bitstream ends, or an atlas frame aFrmB in the current layer of the coded atlas access unit is output, containing a zippering SEI message applicable to the current layer with the same value of zp_persistence_flag, and AtlasFrmOrderCnt(aFrmB) is greater than AtlasFrmOrderCnt(aFrmA), where AtlasFrmOrderCnt(aFrmB) and AtlasFrmOrderCnt(aFrmA) are the AtlasFrmOrderCntVal values ​​of aFrmB and aFrmA immediately after the decoding process starts, respectively, for the atlas frame order count of aFrmB.

[0047] If zp_reset_flag is equal to 1, all entries in the ZipperingMethod array are reset to 0, and all parameters associated with this SEI message are set to their default values.

[0048] zp_instances_updated specifies the number of zippering instances to be updated within the current zippering SEI message.

[0049] zp_instance_index[i] indicates the index of the i-th zippering instance in the array ZipperingMethod, which should be updated by the current SEI message.

[0050] A zp_instance_cancel_flag[k] equal to 1 indicates that the value of ZipperingMethod[k] should be set to 0, and all parameters related to the zippering instance at index k should be set to their default values.

[0051] zp_method_type[k] indicates the ZipperingMethod[k], which is the zippering method that can be used to process the current mesh frame as specified in Table 1 for the zippering instance at index k. JPEG2026524897000013.jpg26137 Table 1 - Definition of zp_method_type[k]

[0052] Values ​​of zp_method_type[k] greater than 2 are reserved for future use by ISO / IEC. As a bitstream conformance requirement, bitstreams conforming to this version of this document must not contain such zp_method_type[k] values. The decoder must ignore zippering SEI messages containing reserved values ​​for zp_method_type[k]. The default value for zp_method_type[k] is equal to 0.

[0053] zp_zippering_max_match_distance[k] specifies the value of zipperingMaxMatchDistance[k], which is a variable used to process the current mesh frame of the zippering instance at index k when the zippering filtering process is used.

[0054] A value of 1 for zp_zippering_send_distance_per_submesh[k] specifies that zippering is applied to the boundary points of the zippering instance at index k by sending the distance for each matching submesh. A value of 0 for zp_zippering_send_distance_per_submesh[k] specifies that zippering is not applied to the boundary points of the zippering instance at index k by sending the distance for each matching submesh. The default value for zp_zippering_send_distance_per_submesh[k] is 0.

[0055] zp_zippering_number_of_submeshes[k] indicates the number of submeshes that are scheduled to be zippered by the current SEI message. The value of zp_zippering_number_of_submeshes ranges from 0 to MaxNumSubmeshes[frameIdx]. The default value of zp_zippering_number_of_submeshes is 0.

[0056] zp_zippering_max_match_distance_per_submesh[k][p] specifies the value of zipperingMaxMatchDistancePerPatch[k][p], which is a variable used to process the current submesh at index p within the current mesh frame of the zippering instance at index k when the zippering process is used. The length of the zp_zippering_max_match_distance_per_submesh[k][p] syntax element is Ceil(Log2(zp_zippering_max_match_distance[k])) bits.

[0057] A value of 1 for zp_zippering_send_distance_per_border_point[k] specifies that zippering is applied to the boundary points of the zippering instance at index k by sending the distance for each matching boundary point. A value of 0 for zp_zippering_send_distance_per_border_point[k] specifies that zippering is not applied to the boundary points of the zippering instance at index k by sending the distance for each matching boundary point. The default value for zp_zippering_send_distance_per_border_point[k] is 0.

[0058] zp_zippering_number_of_border_points[k][p] represents numBorderPoints[p], which is the number of boundary points of the submesh at index p in the current mesh frame of the zippering instance at index k when the zippering process is used.

[0059] zp_zippering_border_point_distance[k][p][b] specifies the value of zipperingMaxMatchDistancePerBorderPoint[k][p][b], which is a variable used to process the current boundary point of index b in the current submesh of index p in the current mesh frame of the zippering instance of index k when the zippering process is used. The length of the zp_zippering_border_point_distance[k][p][b] syntax element is Ceil(Log2(zp_zippering_max_match_distance_per_submesh[k][p])) bits.

[0060] zp_zippering_border_point_match_submesh_index[k][p][b] specifies the value of zipperingBorderPointMatchSubmeshIndex[k][p][b], which is a variable used to process the current boundary point of index b in the current submesh of index p in the current mesh frame of the zippering instance of index k when the zippering process is used. The length of the zp_zippering_border_point_match_submesh_index[k][p][b] syntax element is Ceil(Log2(zp_zippering_number_of_submesh[k])) bits.

[0061] zp_zippering_border_point_match_border_point_index[k][p][b] specifies the value of zipperingBorderPointMatchBorderPointIndex[k][p][b], which is a variable used to process the current boundary point of index b in the current patch of index p for the zippering instance of index k when the zippering filtering process is used. The length of the zp_zippering_border_point_match_border_point_index[k][p][b] syntax element is Ceil(Log2(zp_zippering_number_of_border_points[k][zp_zippering_border_point_match_submesh_index[k][p][b])) bits.

[0062] Reconstruction The boundary vertices of each submesh are detected. If the SEI indicates a distance match, the system searches for a match between two boundary vertices by checking the distance between them, given a certain threshold (which can be defined by the sequence / frame / submesh / boundary), and fills the structure with boundary vertex pairs. On the other hand, if the SEI message indicates an index match, the boundary vertex pair structure is obtained directly from the SEI message. The matched vertices are then merged. Vertex boundary detection From the following code: void zippering_find_borders(std::vector <TriangleMesh <meshtype>>& submeshes, std::vector<std::vector<int8_t>>& isBoundaryVertex, std::vector<size_t>& numBoundaries, TriangleMesh <meshtype>(& boundary vertices) Discovery of vertex match based on distance From the following code: void zippering_find_matches(std::vector <TriangleMesh <meshtype>>& submeshes, TriangleMesh <meshtype>& boundaryVertices, std::vector<size_t> & numBoundaries, std::vector <std::vector<int64_t> >& zipperingDistanceBorderPoint, std::vector <std::vector<Vec2<size_t> >>& zipperingMatchedBorderPoint) Matching apex zipper From the following code: void zippering_fuse_border(std::vector <TriangleMesh <meshtype>>& submeshes, std::vector <std::vector<int8_t> >& isBoundaryVertex, std::vector <std::vector<Vec2<size_t> >>& zipperingMatchedBorderPoint)

[0063] Figure 13 shows the results of the zippering algorithm. The crack has been corrected using the method described herein. A fixed threshold is selected for the frame, and the added bitrate is only a few bytes for SEI message transmission. The impact of SEI transmission is further reduced if the same threshold can be used for other frames.

[0064] To utilize the mesh zippering method, the device acquires or receives 3D content (e.g., point cloud content). The mesh zippering method can be performed with or without user assistance.

[0065] The mesh zippering method enables more efficient and accurate 3D content decoding during operation compared to previous implementations.

[0066] Several embodiments of submesh zippering 1. A method programmed into the non-temporary memory of the device, Determining one or more boundary points in one or more submeshes, Selecting one zippering implementation from multiple mesh zippering implementations, A method comprising merging each of one or more boundary points with a corresponding point, based on a selected mesh zippering implementation.

[0067] 2. Multiple mesh zippering implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The method described in Clause 1, including the method described in Clause 1.

[0068] 3. The implementation of a specified search distance is as described in Clause 2, using a user-defined search distance.

[0069] 4. The implementation of the specified search distance is as described in Clause 2, using computer-generated search distances with artificial intelligence and machine learning.

[0070] 5. The method described in Clause 2, wherein the implementation of a specified search distance, the implementation of the maximum distance in all submeshes and all frames, the implementation of the maximum distance per frame, the implementation of the maximum distance per submesh, and the implementation of the maximum distance for each boundary vertex includes limiting the search range of points based on distance, respectively.

[0071] 6. The distance is the radius of the spherical search area, as described in Clause 5.

[0072] The method according to Clause 1, wherein determining one or more boundary points in 7.1 or two or more submeshes includes determining each point on an edge that is not connected to two triangles.

[0073] 8. Determine one or more boundary points in 8.1 or 2 or more submeshes, Select one zippering implementation from multiple mesh zippering implementations. Non-temporary memory for storing an application to merge each of one or more boundary points with the corresponding point, based on the selected mesh zippering implementation, A device comprising a processor coupled to memory and configured to process applications.

[0074] 9. Multiple mesh zippering implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The apparatus described in Clause 8, including the apparatus described in Clause 8.

[0075] 10. The implementation of the specified search distance is the device described in Clause 9, which uses a user-defined search distance.

[0076] 11. The implementation of the specified search distance is the device described in Clause 9, which uses a computer-generated search distance using artificial intelligence and machine learning.

[0077] 12. The apparatus described in Clause 9, which includes limiting the search range of points based on distance, for each of the following: implementation of a specified search distance, implementation of the maximum distance in all submeshes and all frames, implementation of the maximum distance per frame, implementation of the maximum distance per submesh, and implementation of the maximum distance for each boundary vertex.

[0078] 13. The apparatus as described in Clause 12, where the distance is the radius of the spherical search area.

[0079] The apparatus described in Clause 8 for determining one or more boundary points in 14.1 or two or more submeshes, which includes determining each point on an edge that is not connected to two triangles.

[0080] 15. An encoder configured to encode content, Determine one or more boundary points in one or more submeshes, Select one zippering implementation from multiple mesh zippering implementations. Based on the selected mesh zippering implementation, merge each of one or more boundary points with its corresponding point. A system comprising a decoder configured in such a way.

[0081] 16. Multiple mesh zippering implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The systems described in Clause 15, including those mentioned above.

[0082] 17. The implementation of the specified search distance is the system described in Clause 16, which uses a user-defined search distance.

[0083] 18. The implementation of the specified search distance is the system described in Clause 16, which uses computer-generated search distances using artificial intelligence and machine learning.

[0084] 19. The implementation of a specified search distance, the implementation of the maximum distance in all submeshes and all frames, the implementation of the maximum distance per frame, the implementation of the maximum distance per submesh, and the implementation of the maximum distance for each boundary vertex, respectively, includes limiting the search range of points based on distance, as described in Clause 16.

[0085] 20. The distance is the radius of the spherical search area, as described in Clause 19 of the system.

[0086] 21.1 The system described in Clause 15, which determines one or more boundary points in two or more submeshes, including determining each point on an edge that is not connected to two triangles.

[0087] The present invention has been described in relation to specific embodiments, including details, to facilitate understanding of its structure and operating principles. Such references to specific embodiments and details thereof are not intended to limit the claims appended herein. Those skilled in the art will readily see that various other modifications can be made to the embodiments selected for illustrative purposes without departing from the spirit and scope of the invention as defined by the claims. [Explanation of symbols]

[0088] 100 boundary points discovered 102 Zippering per frame 104 Zippering per patch 106 Zippering at each boundary point 108 Boundary Point Matching Zippering Merge 110 vertices< / meshtype> < / meshtype> < / meshtype> < / meshtype> < / meshtype>

Claims

1. A method of programming into the device's non-temporary memory, Determining one or more boundary points in one or more submeshes, Selecting one zippering implementation from multiple mesh zippering implementations, Based on the selected mesh zippering implementation, each of the one or more boundary points is merged with the corresponding point. A method that includes this.

2. The aforementioned multiple mesh zipper implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The method according to claim 1, including the method described in claim 1.

3. The implementation of the aforementioned search distance uses a user-defined search distance. The method according to claim 2.

4. The implementation of the aforementioned search distance uses a computer-generated search distance using artificial intelligence and machine learning. The method according to claim 2.

5. The implementation of the specified search distance, the implementation of the maximum distance in all submeshes and all frames, the implementation of the maximum distance for each frame, the implementation of the maximum distance for each submesh, and the implementation of the maximum distance for each boundary vertex each includes limiting the search range of the points based on distance, The method according to claim 2.

6. The aforementioned distance is the radius of the spherical search area. The method according to claim 5.

7. Determining the one or more boundary points in the one or more submesh includes determining each point on an edge that is not connected to two triangles. The method according to claim 1.

8. Determine one or more boundary points in one or more submeshes, Select one zippering implementation from multiple mesh zippering implementations. Based on the selected mesh zippering implementation, each of the one or more boundary points is merged with the corresponding point. Non-temporary memory for storing applications, A processor coupled to the memory and configured to process the application, A device equipped with the following features.

9. The aforementioned multiple mesh zipper implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The apparatus according to claim 8, including the apparatus described in claim 8.

10. The implementation of the aforementioned search distance uses a user-defined search distance. The apparatus according to claim 9.

11. The implementation of the aforementioned search distance uses a computer-generated search distance using artificial intelligence and machine learning. The apparatus according to claim 9.

12. The implementation of the specified search distance, the implementation of the maximum distance in all submeshes and all frames, the implementation of the maximum distance for each frame, the implementation of the maximum distance for each submesh, and the implementation of the maximum distance for each boundary vertex each includes limiting the search range of the points based on distance, The apparatus according to claim 9.

13. The aforementioned distance is the radius of the spherical search area. The apparatus according to claim 12.

14. Determining the one or more boundary points in the one or more submesh includes determining each point on an edge that is not connected to two triangles. The apparatus according to claim 8.

15. An encoder configured to encode content, Determine one or more boundary points in one or more submeshes, Select one zippering implementation from multiple mesh zippering implementations. Based on the selected mesh zippering implementation, each of the one or more boundary points is merged with the corresponding point. A decoder configured as follows, A system equipped with these features.

16. The aforementioned multiple mesh zipper implementations are, Implementation of a defined search distance, Implementation of maximum distance for all submeshes and all frames, Implementation of the maximum distance per frame, Implementation of the maximum distance for each submesh, The maximum distance for each boundary vertex, The matching index between two boundary vertices, The system according to claim 15, including the system described in claim 15.

17. The implementation of the aforementioned search distance uses a user-defined search distance. The system according to claim 16.

18. The implementation of the aforementioned search distance uses a computer-generated search distance using artificial intelligence and machine learning. The system according to claim 16.

19. The implementation of the specified search distance, the implementation of the maximum distance in all submeshes and all frames, the implementation of the maximum distance for each frame, the implementation of the maximum distance for each submesh, and the implementation of the maximum distance for each boundary vertex each includes limiting the search range of the points based on distance, The system according to claim 16.

20. The aforementioned distance is the radius of the spherical search area. The system according to claim 19.

21. Determining the one or more boundary points in the one or more submesh includes determining each point on an edge that is not connected to two triangles. The system according to claim 15.

Images