Efficient mapping coordinate creation and transmission

Abstract

A method is disclosed that generates (u, v) coordinates on the decoder side by using parameters of an orthographic projection function transmitted via an atlas bitstream. Using the parameters of orthographic projection, the decoder is able to efficiently generate (u, v) coordinates and avoid their expensive coding.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application Serial No. 63 / 378,547, filed on October 6, 2022, entitled “EFFICIENT MAPPING COORDINATE CREATION AND TRANSMISSION,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present invention relates to three-dimensional graphics. More particularly, the present invention relates to mapping coordinates. Background Art

[0004] Recently, new methods for compressing volumetric content such as point clouds based on projections from 3D to 2D are being standardized. This method, also known as V3C (Visual Volumetric Video-based Compression), maps 3D volumetric data into several 2D patches, which are then further arranged into atlas images that are subsequently encoded with a video encoder. The atlas image corresponds to the geometry of the points, the corresponding texture, and an occupancy map that indicates which of the locations are to be considered for point cloud reconstruction.

[0005] In 2017, MPEG published a Call for Proposals (CfP) for the compression of point clouds. After evaluating several proposals, MPEG is currently considering two different techniques for point cloud compression: 3D native coding techniques (based on octrees and similar coding methods), or 3D to 2D projection followed by traditional video coding. In the case of dynamic 3D scenes, MPEG is using a test model software (TMC2) based on surfel surface modeling, the projection of surfels from 3D images to 2D images, and encoding of 2D images with a video encoder such as HEVC. This approach has been shown to be more efficient than native 3D coding and to achieve competitive bitrates with acceptable quality.

[0006] Due to the success of projection-based methods (also called video-based methods or V-PCC) for encoding 3D point clouds, it is expected that the standard will include more 3D data, such as 3D meshes, in future versions. However, the current version of the standard is only suitable for the transmission of a set of disconnected points, so there is no mechanism to send the connectivity of the points, which is required in 3D mesh compression.

[0007] Methods to extend the functionality of V-PCC to meshes have also been proposed. One possible approach is to encode the vertices using V-PCC and then encode the connectivity using a mesh compression method such as TFAN or Edgebreaker. A limitation of this approach is that the original mesh must be dense so that the point cloud generated from the vertices is not sparse and can be efficiently encoded after projection. In addition, the order of the vertices affects the encoding of the connectivity, so different methods of reorganizing the connectivity of the mesh have been proposed. An alternative way to encode sparse meshes is to encode the vertex positions in 3D using RAW surface data. Since RAW surfaces encode (x, y, z) directly, in this approach all vertices are encoded as RAW data, while connectivity is encoded by a similar mesh compression method as described above. Note that in RAW surfaces, the vertices can be sent in any preferred order, so the order resulting from the connectivity encoding can be used. This method can encode sparse point clouds, however, RAW surfaces are not efficient for encoding 3D data, and more data such as the attributes of triangle faces may be lost from this method.

[0008] UV atlas from Microsoft is the most advanced automatic texture map generation, but it takes a lot of time and is only optimized for local frames. V-PCC uses orthographic projection to generate surfels, but only for point clouds, so it does not solve the problem of surfel generation for meshes. Summary of the invention

[0009] A method is disclosed for generating (u, v) coordinates at the decoder side by using the parameters of the orthographic projection function transmitted via an atlas bitstream. Using the parameters of the orthographic projection, the decoder can efficiently generate the (u, v) coordinates and avoid their expensive encoding.

[0010] In one aspect, a method for programming in a non-temporary memory of a device includes receiving facet identification information and mapping function parameters, and generating (u, v) coordinates based on the facet identification and mapping function parameters. The method also includes encoding a 3D mesh to generate facet identification information and mapping function parameters. The mapping function parameters are encoded on an atlas sub-bitstream. The mapping function parameters include 3D to 2D mapping function parameters. Encoding the 3D mesh includes: generating facets from dynamic mesh information, and packing the facets on a texture atlas using an orthographic projection. Generating (u, v) coordinates based on facet identification and mapping function parameters includes: generating (u, v) coordinates based on facet identification and mapping function parameters using a function, wherein the mapping function parameters correspond to the facet identification. The (u, v) coordinates are generated based on a transformation using a bounding box size, an occupied resolution, and scaling information. The method also includes reconstructing the 3D mesh based on the (u, v) coordinates.

[0011] In another aspect, a device includes a non-temporary memory for storing an application and a processor coupled to the memory, the application being used to: receive facet identification information and mapping function parameters, and generate (u, v) coordinates based on the facet identification and mapping function parameters, the processor being configured to process the application. The application is configured to encode a 3D mesh to generate facet identification information and mapping function parameters. The mapping function parameters are encoded on an atlas sub-bitstream. The mapping function parameters include 3D to 2D mapping function parameters. Encoding the 3D mesh includes: generating facets from dynamic mesh information, and packing the facets on a texture atlas using an orthographic projection. Generating (u, v) coordinates based on facet identification and mapping function parameters includes: generating (u, v) coordinates based on the facet identification and mapping function parameters using a function, wherein the mapping function parameters correspond to the facet identification. The (u, v) coordinates are generated based on a transformation using a bounding box size, an occupied resolution, and scaling information. The application is configured to reconstruct a 3D mesh based on the (u, v) coordinates.

[0012] In another aspect, a system includes an encoder and a decoder, the encoder being configured to encode a 3D mesh to generate facet identification information and mapping function parameters, the decoder being configured to: receive facet identification information and mapping function parameters, and generate (u, v) coordinates based on the facet identification and mapping function parameters. The mapping function parameters are encoded on an atlas sub-bitstream. The mapping function parameters include 3D to 2D mapping function parameters. Encoding the 3D mesh includes: generating facets from dynamic mesh information, and packing the facets on a texture atlas using an orthographic projection. Generating (u, v) coordinates based on facet identification and mapping function parameters includes: generating (u, v) coordinates based on facet identification and mapping function parameters using a function, wherein the mapping function parameters correspond to the facet identification. The (u, v) coordinates are generated based on a transformation using a bounding box size, an occupied resolution, and scaling information. The decoder is configured to reconstruct the 3D mesh based on the (u, v) coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Diagram illustrating texture mapping in accordance with some embodiments.

[0014] Figure 2 A diagram illustrating a decoding implementation in accordance with some embodiments.

[0015] Figure 3 A block diagram illustrating an exemplary computing device configured to implement efficient mapping coordinate generation and transmission methods in accordance with some embodiments. DETAILED DESCRIPTION

[0016] A mesh consists of a set of polygons that typically describe the surface of a volume. An efficient way to describe the surface properties of a mesh (e.g., its color characteristics) is to generate a texture atlas that maps the properties of the 3D surface onto a 2D surface. The result of the mapping function is stored in (u, v) coordinates and added to the mesh data, which is then further encoded using a mesh compression method. However, the presence of (u, v) coordinates can significantly increase the size of the compressed mesh.

[0017] In the latest international point cloud compression standard, a depth map image is generated for a point cloud using orthographic projection. The parameters of the projection are then encoded into a metadata bitstream called an atlas bitstream, so the decoder only receives these parameters and applies a mapping function to each (u, v) coordinate of the depth map to then reconstruct the 3D information.

[0018] A method is disclosed for transmitting parameters for generating mapping coordinates for a mesh using orthographic projection, similar to that used in the V-PCC standard. Using these parameters, a decoder can generate (u, v) coordinates and additionally reduce the size of the compressed mesh.

[0019] Figure 1 A diagram illustrating texture mapping according to some embodiments. In step 100, surfel generation is performed. Position information and connectivity information are received and used to generate surfels. Surfel generation involves the f function with position information (x, y, z) and encoding parameters (C).

[0020] In step 102, bin packing is implemented. Bin packing involves a g-function based on the f-function and the encoding parameter P. The mapping (f-function and g-function) is determined at the encoder side using orthographic projection. Bin generation and bin packing are described in U.S. Patent Application No. *** Attorney Docket No. Sony-76000***, entitled ORTHOATLAS: TEXTURE MAP GENERATION FOR DYNAMIC MESHES USING ORTHOGRAPHIC PROJECTIONS, which is incorporated herein by reference in its entirety for all purposes.

[0021] In step 104, the 3D->2D mapping function parameters (C1 and P1...C N and P N ) is sent in an atlas sub-bitstream (e.g., to a decoder).

[0022] In step 106, instead of sending u,v coordinates, a patchId is sent. For example, the patchID (pid) is sent to Draco, which encodes and decodes the facet ID. The patchID (pid) indicates which parameters are used for the facet. For example, pid 1 corresponds to encoding parameters C1 and P1, pid 2 corresponds to encoding parameters C2 and P2, and so on. Using the appropriate parameters, the 2D projection can be recovered. The triangles in 3D space are known, so the encoding parameters (e.g., C1 and P1) can be used to determine where to find the texture coordinates on the 2D surface.

[0023] In some embodiments, fewer or additional steps are implemented. In some embodiments, the order of the steps is modified.

[0024] Figure 2 A diagram illustrating a decoding implementation according to some embodiments is shown. A V3C sub-bitstream is received. For example, a base grid sub-bitstream 200, an atlas sub-bitstream 202, a displacement sub-bitstream 204, and an attribute sub-bitstream 206 are received. The sub-bitstreams are decoded by a grid codec 210, a first video codec 212, or a second video codec 214. For example, the base grid sub-bitstream 200 is decoded by the grid codec 210; the displacement sub-bitstream 204 is decoded by the first video codec 212; and the attribute sub-bitstream 206 is decoded by the second video codec 214. The displacement sub-bitstream 204 is decoded by the video codec 212 and the displacement decoder 222.

[0025] The trellis codec 210 decodes the base trellis sub-bitstream 200 and outputs a decoded base trellis 220 including a bin ID.

[0026] The atlas mapping process 230 is an added implementation. The atlas mapping process 230 receives (patchID, 0) information based on the decoded base grid. The atlas mapping process 230 also receives parameters (e.g., C, P) from the atlas sub-bitstream 202 in the form of bins to regenerate u, v coordinates.

[0027] The generated u, v coordinates are used for connectivity processing 232 and vertex processing 234. Based on this processing, the object / mesh is reconstructed 236. In addition, the texture map 238 is reconstructed based on the decoded attribute sub-bitstream 206.

[0028] Texture map coordinates can be derived by applying the following transformation:

[0029]

[0030] in:

[0031]

[0032] in:

[0033]

[0034] By using homogeneous coordinates, all operations can be combined into a single 4x4 homography transformation matrix.

[0035] These transforms are used at the decoder to generate u, v coordinates based on bounding box size, projection, occupied resolution, scaling, and / or other information / parameters.

[0036] Example syntax includes:

[0037] Sequence Header

[0038]

[0039]

[0040] Frame Header

[0041]

[0042] For example, in the syntax, in the sequence header, a flag can be sent to indicate that the texture coordinates will be derived from 3D positions. Additionally, the size of the gutter and the occupied resolution can be sent. In the frame header, the number of bins is sent. Additionally, for each bin, the projection in 2D space, orientation, position and size, and scale are sent on the bitstream.

[0043] Figure 3The block diagram of an exemplary computing device configured to implement efficient mapping coordinate generation and transmission methods according to some embodiments is illustrated. The computing device 300 can be used to obtain, store, calculate, process, transmit and / or display information such as images and videos including 3D content. The computing device 300 can implement any aspect of encoding / decoding. Generally, the hardware structure suitable for implementing the computing device 300 includes a network interface 302, a memory 304, a processor 306, (one or more) I / O devices 308, a bus 310 and a storage device 312. The choice of processor is not important, as long as a suitable processor with sufficient speed is selected. The memory 304 can be any conventional computer memory known in the art. The storage device 312 can include a hard disk drive, a CDROM, a CDRW, a DVD, a DVDRW, a high-definition optical disk / drive, an ultra-high-definition drive, a flash memory card or any other storage device. The computing device 300 may include one or more network interfaces 302. Examples of network interfaces include network cards connected to Ethernet or other types of LANs. The I / O device(s) 308 may include one or more of the following: a keyboard, a mouse, a monitor, a screen, a printer, a modem, a touch screen, a button interface, and other devices. The efficient mapping coordinate generation and transmission application(s) 330 for implementing the efficient mapping coordinate generation and transmission implementation may be stored in the storage device 312 and the memory 304 and processed in the manner in which applications are normally processed. The computing device 300 may include Figure 3 More or fewer components may be shown in FIG. In some embodiments, efficient mapping coordinate generation and transmission hardware 320 is included. Although Figure 3 The computing device 300 in the embodiment includes an application 330 and hardware 320 for efficient mapping coordinate generation and transmission implementation, but the texture efficient mapping coordinate generation and transmission method can be implemented on the computing device in hardware, firmware, software, or any combination thereof. For example, in some embodiments, the efficient mapping coordinate generation and transmission application 330 is programmed in a memory and executed using a processor. In another example, in some embodiments, the efficient mapping coordinate generation and transmission hardware 320 is programmed hardware logic including gates specifically designed to implement the efficient mapping coordinate generation and transmission method.

[0044] In some embodiments, (one or more) efficient mapping coordinate generation and transmission application 330 includes several applications and / or modules. In some embodiments, a module also includes one or more sub-modules. In some embodiments, fewer or additional modules may be included.

[0045] Examples of suitable computing devices include personal computers, laptop computers, computer workstations, servers, mainframe computers, handheld computers, personal digital assistants, cellular / mobile phones, smart appliances, game consoles, digital cameras, digital video cameras, camera phones, smart phones, portable music players, tablet computers, mobile devices, video players, video disc recorders / players (e.g., DVD recorders / players, high-definition disc recorders / players, ultra-high-definition disc recorders / players), televisions, home entertainment systems, augmented reality devices, virtual reality devices, smart jewelry (e.g., smart watches), vehicles (e.g., self-driving vehicles), or any other suitable computing device.

[0046] In order to utilize the efficient mapping coordinate generation and transmission method, the device acquires or receives 3D content (eg, point cloud content). The efficient mapping coordinate generation and transmission method can be implemented with the help of a user, or can be implemented automatically without user participation.

[0047] In operation, the orthoAtlas(u,v) generation function parameters are sent to the decoder: Texture map generation using orthographic projection and exported on the decoder side. Defines the elements that together generate a mapping between 3D vertex coordinates (x,y,z) to 2D positions (u,v) in the atlas. The parameters used to generate the mapping function are efficiently encoded using the atlas bitstream. The result is a more efficient implementation that transmits less data.

[0048] As mentioned above, the transmission of (u, v) coordinates takes up a lot of space in the compressed mesh representation. Today, texture map generation relies on complex optimizations to reduce mapping distortion and texture seams, and the resulting (u, v coordinates) are encoded as part of the mesh representation. However, even with methods that efficiently use the corresponding (x, y, z) values ​​and triangle structure, these coordinates require a large number of bits to encode. Efficient mapping coordinate generation and transmission methods enable the generation of (u, v) coordinates on the decoder side by using the parameters of the orthographic projection function transmitted via the atlas bitstream. Using the parameters of the orthographic projection, the decoder is able to efficiently generate the (u, v) coordinates and avoid their expensive encoding.

[0049] Some embodiments of efficient mapping coordinate creation and transmission

[0050] 1. A method of programming in a non-transitory memory of a device, comprising:

[0051] receiving facet identification information and mapping function parameters; and

[0052] Based on the bin identity and mapping function parameters, (u, v) coordinates are generated.

[0053] 2. The method according to clause 1 also includes encoding the 3D mesh to generate facet identification information and mapping function parameters.

[0054] 3. A method as claimed in clause 2, wherein the mapping function parameters are encoded on the atlas sub-bitstream.

[0055] 4. The method of clause 2, wherein the mapping function parameters include 3D to 2D mapping function parameters.

[0056] 5. The method of clause 2, wherein encoding the 3D mesh comprises:

[0057] generating facets from the dynamic mesh information; and

[0058] Pack the polygons on the texture atlas using orthographic projection.

[0059] 6. A method according to clause 1, wherein generating (u, v) coordinates based on face element identification and mapping function parameters comprises: using a function to generate (u, v) coordinates according to the face element identification and mapping function parameters, wherein the mapping function parameters correspond to the face element identification.

[0060] 7. The method of clause 1, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

[0061] 8. The method according to clause 1, further comprising reconstructing a 3D mesh based on (u, v) coordinates.

[0062] 9. A device comprising:

[0063] Non-transitory memory for storing applications for:

[0064] receiving facet identification information and mapping function parameters; and

[0065] Generate (u, v) coordinates based on the bin identity and mapping function parameters; and

[0066] A processor is coupled to the memory, the processor being configured to process the application.

[0067] 10. The apparatus of clause 9, wherein the application is configured to encode the 3D mesh to generate bin identification information and mapping function parameters.

[0068] 11. Apparatus according to clause 10, wherein mapping function parameters are encoded on the atlas sub-bitstream.

[0069] 12. The apparatus of clause 10, wherein the mapping function parameters comprise 3D to 2D mapping function parameters.

[0070] 13. The apparatus of clause 10, wherein encoding the 3D mesh comprises:

[0071] generating facets from the dynamic mesh information; and

[0072] Pack the polygons on the texture atlas using orthographic projection.

[0073] 14. An apparatus according to clause 9, wherein generating (u, v) coordinates based on the face element identifier and mapping function parameters comprises: generating (u, v) coordinates according to the face element identifier and mapping function parameters using a function, wherein the mapping function parameters correspond to the face element identifier.

[0074] 15. The apparatus of clause 9, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

[0075] 16. The apparatus of clause 9, wherein the application is configured to reconstruct a 3D mesh based on (u, v) coordinates.

[0076] 17. A system comprising:

[0077] an encoder configured to encode the 3D mesh to generate surfel identification information and mapping function parameters; and

[0078] A decoder configured to:

[0079] receiving facet identification information and mapping function parameters; and

[0080] Generate (u, v) coordinates based on the surfel identity and mapping function parameters.

[0081] 18. A system according to clause 17, wherein mapping function parameters are encoded on the atlas sub-bitstream.

[0082] 19. The system of clause 17, wherein the mapping function parameters comprise 3D to 2D mapping function parameters.

[0083] 20. The system of clause 17, wherein encoding the 3D mesh comprises:

[0084] generating facets from the dynamic mesh information; and

[0085] Pack the polygons on the texture atlas using orthographic projection.

[0086] 21. A system according to clause 17, wherein generating (u, v) coordinates based on the face element identification and mapping function parameters includes: using a function to generate (u, v) coordinates according to the face element identification and mapping function parameters, wherein the mapping function parameters correspond to the face element identification.

[0087] 22. The system of clause 17, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

[0088] 23. The system of clause 17, wherein the decoder is configured to reconstruct a 3D mesh based on (u, v) coordinates.

[0089] The present invention has been described by specific embodiments, which contain details that facilitate understanding of the principles of construction and operation of the present invention. Such reference to specific embodiments and details thereof herein is not intended to limit the scope of the claims appended hereto. It will be apparent to those skilled in the art that various other modifications may be made in the embodiments selected for illustration without departing from the spirit and scope of the present invention as defined by the claims.

Claims

1. A method of programming in a non-transitory memory of a device, comprising: Receiving facet identification information and mapping function parameters; as well as Based on the bin identity and mapping function parameters, (u, v) coordinates are generated.

2. The method according to claim 1 also includes encoding the 3D mesh to generate surface element identification information and mapping function parameters.

3. The method of claim 2, wherein mapping function parameters are encoded on an atlas sub-bitstream.

4. The method of claim 2, wherein the mapping function parameters comprise 3D to 2D mapping function parameters.

5. The method of claim 2, wherein encoding the 3D mesh comprises: Generate facets from dynamic mesh information; and Pack the polygons on the texture atlas using orthographic projection.

6. The method of claim 1, wherein generating (u, v) coordinates based on the bin identifier and the mapping function parameters comprises: A function is used to generate (u, v) coordinates according to the bin identifier and mapping function parameters, wherein the mapping function parameters correspond to the bin identifier.

7. The method of claim 1, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

8. The method of claim 1 further comprising reconstructing a 3D mesh based on (u, v) coordinates.

9. An apparatus comprising: Non-transitory memory for storing applications for: Receiving facet identification information and mapping function parameters; as well as generating (u, v) coordinates based on the bin identification and the mapping function parameters; and a processor coupled to the memory, the processor configured to process the application.

10. The apparatus of claim 9, wherein the application is configured to encode the 3D mesh to generate surfel identification information and mapping function parameters.

11. The apparatus of claim 10, wherein mapping function parameters are encoded on an atlas sub-bitstream.

12. The apparatus of claim 10, wherein the mapping function parameters comprise 3D to 2D mapping function parameters.

13. The apparatus of claim 10, wherein encoding the 3D mesh comprises: Generate facets from dynamic mesh information; and Pack the polygons on the texture atlas using orthographic projection.

14. The apparatus of claim 9, wherein generating (u, v) coordinates based on the bin identifier and the mapping function parameters comprises: A function is used to generate (u, v) coordinates according to the bin identifier and mapping function parameters, wherein the mapping function parameters correspond to the bin identifier.

15. The apparatus of claim 9, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

16. The apparatus of claim 9, wherein the application is configured to reconstruct a 3D mesh based on (u, v) coordinates.

17. A system comprising: An encoder configured to encode the 3D mesh to generate bin identification information and mapping function parameters; and A decoder configured to: receiving facet identification information and mapping function parameters; and Generate (u, v) coordinates based on the surfel identity and mapping function parameters.

18. The system of claim 17, wherein mapping function parameters are encoded on the atlas sub-bitstream.

19. The system of claim 17, wherein the mapping function parameters include 3D to 2D mapping function parameters.

20. The system of claim 17, wherein encoding the 3D mesh comprises: Generate facets from dynamic mesh information; and Pack the polygons on the texture atlas using orthographic projection.

21. The system of claim 17, wherein generating (u, v) coordinates based on the bin identification and mapping function parameters comprises: A function is used to generate (u, v) coordinates according to the bin identifier and mapping function parameters, wherein the mapping function parameters correspond to the bin identifier.

22. The system of claim 17, wherein the (u, v) coordinates are generated based on a transformation using bounding box size, occupied resolution, and scaling information.

23. The system of claim 17, wherein the decoder is configured to reconstruct a 3D mesh based on (u, v) coordinates.

Images