Texture coordinate compression using chart partitioning

Abstract

A method, apparatus, and system for mesh compression are provided. The process may include generating a texture plane associated with a mesh including three-dimensional coordinates using a mesh parameterization. The process may include, for a triangle in the texture plane, predicting texture coordinates of a first one of the vertices of the triangle based on encoding statuses of texture coordinates of a second vertex and a third vertex of the triangle. The process may further include determining a residual between the predicted texture coordinate of the first vertex and the actual texture coordinate of the first vertex and encoding the residual using entropy coding.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 418,399, filed October 21, 2022, and U.S. Patent Application No. 18 / 490,325, filed October 19, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This disclosure is directed to a set of advanced video coding techniques, and more particularly, to a method and system for texture coordinate prediction using chart partitioning in mesh compression. [Background technology]

[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive interaction and communication. To achieve a sense of presence in 3D representations, 3D models are becoming ever more sophisticated, and a significant amount of data is tied to the creation and consumption of these 3D models. 3D meshes are widely used to model immersive content.

[0004] A mesh may be composed of several polygons that describe the surface of a volumetric object. Each polygon is defined by its vertices in 3D space and information about how the vertices are connected, called connectivity information. Such mapping is typically described by a set of parametric coordinates, called XYZ coordinates. In some embodiments, vertex attributes such as color, normals, etc. may also be associated with mesh vertices. Attributes may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping is typically described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with the mesh vertices. The 2D attribute map may be used to store high-resolution attribute information such as texture, normals, displacement, etc. Such information may also be used for various purposes, such as texture mapping and shading.

[0005] Dynamic mesh sequences can require large amounts of data because they can contain a significant amount of information that changes over time, and therefore efficient compression techniques are needed to store and transmit such content.

[0006] Mesh compression standards IC, MESHGRID, and FAMC have previously been developed to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not take into account time-varying attribute maps and connectivity information.

[0007] Furthermore, it is also difficult for volume acquisition techniques to generate always-on dynamic meshes, especially under real-time constraints, and this type of dynamic mesh content is not supported by existing standards. Summary of the Invention [Means for solving the problem]

[0008] According to embodiments, a method, apparatus, and a non-transitory computer-readable medium storing computer instructions for process video encoding may be provided.

[0009] According to one embodiment, a method may include: partitioning a mesh into a plurality of charts using a mesh parameterization, each chart associated with a local texture domain; generating a texture plane based on the plurality of charts, the texture plane including a triangle; determining whether texture coordinates of a second vertex and a third vertex of the triangle are coded, where the first vertex, the second vertex, and the third vertex are different vertices of the triangle in the texture plane; predicting texture coordinates of a first vertex of the triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle are coded; and entropy encoding a residual between the predicted texture coordinates of the first vertex and the actual texture coordinates of the first vertex.

[0010] According to one embodiment, an apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as instructed by the program code. The program code may include: a first partitioning code configured to cause at least one processor to partition a mesh into a plurality of charts using a mesh parameterization, each chart associated with a local texture domain; a first generation code configured to cause the at least one processor to generate a texture plane based on the plurality of charts, the texture plane including a triangle; a first decision code configured to cause the at least one processor to determine whether texture coordinates of a second vertex and a third vertex of the triangle have been coded, the first vertex, the second vertex, and the third vertex being different vertices of the triangle within the texture plane; a first prediction code configured to cause the at least one processor to predict a texture coordinate of a first vertex of the triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle have been coded; and a first encoding code configured to cause the at least one processor to entropy encode a residual between the predicted texture coordinate of the first vertex and the actual texture coordinate of the first vertex.

[0011] According to one embodiment, a non-transitory computer-readable medium having stored thereon instructions may include one or more instructions that, when executed by one or more processors of a device for video encoding, cause the one or more processors to partition a mesh into a plurality of charts using a mesh parameterization, each chart associated with a local texture domain, generate a texture plane based on the plurality of charts, determine whether the texture plane includes a triangle and texture coordinates of a second vertex and a third vertex of the triangle are coded, predict texture coordinates of a first vertex of the triangle based on whether the first vertex, the second vertex, and the third vertex are different vertices of a triangle in the texture plane and texture coordinates of the second vertex and the third vertex of the triangle are coded, and entropy encode a residual between the predicted texture coordinate of the first vertex and the actual texture coordinate of the first vertex.

[0012] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system, according to an embodiment of the present disclosure. [Figure 3] 1 is a schematic diagram of a simplified block diagram of a video encoder and a video decoder, according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is an exemplary diagram of a process for vertex position compression, according to an embodiment of the present disclosure. [Figure 5] FIG. 2 is an exemplary flow diagram illustrating a process for video encoding, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] A mesh may include several polygons that describe the surface of a volumetric object. As mentioned above, mesh geometry information consists of vertex connectivity information, 3D coordinates, texture coordinates, etc.

[0015] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium are provided for parallel processing of dynamic mesh compression. Embodiments of the present disclosure may also be applied to static meshes.

[0016] 1-2, one embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure is described.

[0017] 1 illustrates a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of unidirectional data transmission, a first terminal 110 may locally code video data, which may include mesh data, for transmission to the other terminal 120 via the network 150. The second terminal 120 may receive the coded video data of the other terminal from the network 150, decode the coded data, and display the recovered video data. Unidirectional data transmission may be common in media serving applications, etc.

[0018] 1 shows a second pair of terminals 130, 140 provided to support bidirectional transmission of coded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 may code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 may also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0019] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that convey coded video data between terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 150 may not be important to the operation of the present disclosure unless described herein below.

[0020] 2 illustrates the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter can be used in other video-enabled applications including, for example, video conferencing, digital television, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0022] The video source 201 may, for example, create a stream 202 including a 3D mesh and metadata associated with the 3D mesh. The video source 201 may, for example, include a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., digital camera(s)) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202 may have a high amount of data compared to an encoded video bitstream and may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoder 203 may also further generate an encoded video bitstream 204. The encoded video bitstream 204 may have a low amount of data compared to the uncompressed stream 202 and may be stored on a streaming server 205 for later use. One or more streaming clients 206 can access the streaming server 205 to retrieve a video bitstream 209 , which may be a copy of the encoded video bitstream 204 .

[0023] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is a copy of the input encoded video bitstream 204, and create an output video sample stream 211 that can be rendered on the display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0024] FIG. 3 is an example diagram of a framework 300 for dynamic mesh compression and mesh reconstruction using an encoder and decoder.

[0025] 3, framework 300 may include encoder 301 and decoder 351. Encoder 301 may include one or more input meshes 305, a mesh 310 with one or more UV atlases, an occupancy map 315, a geometry map 320, an attribute map 325, and metadata 330. Decoder 351 may include a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, decoded metadata 350, and a reconstructed mesh 360.

[0026] According to one aspect of the present disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate the mesh with UV atlas 310. By way of example, the pre-processing operations may include, but are not limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may occur only on the encoder side and not on the decoder side. Mesh parameterization may be the process of mapping textures onto the surfaces of 2D and 3D meshes.

[0027] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh with UV atlas may be a mesh in which each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed based on sampling and converted into multiple maps. As an example, the UV atlas 310 may be processed based on sampling of the 2D mesh with UV atlas and converted into an occupancy map, a geometry map, and an attribute map. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and transmitted to the decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.

[0028] According to one aspect, the decoder 351 may receive the encoded occupancy map, geometry map, and attribute map from the encoder. The decoder 351 may use suitable techniques and methods, in addition to the embodiments described herein, to decode the occupancy map, geometry map, and attribute map. In one embodiment, the decoder 351 may generate a decoded occupancy map 335, a decoded geometry map 340, a decoded attribute map 345, and decoded metadata 350. The input mesh 305 may be reconstructed into a reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, the decoded attribute map 345, and the decoded metadata 350 using one or more reconstruction filters and techniques. In some embodiments, the metadata 330 may be sent directly to the decoder 351, and the decoder 351 may use the metadata to generate the reconstructed mesh 360 based on the decoded occupancy map 335, the decoded geometry map 340, and the decoded attribute map 345. Post-filtering techniques, including but not limited to remeshing, parameterization, tracking, voxelization, etc., may also be applied to the reconstructed mesh 360 .

[0029] An input mesh with a 2D UV atlas may have vertices, and each vertex of the mesh may have associated UV coordinates on the 2D atlas. Occupancy maps, geometry maps, and attribute maps may be generated by sampling one or more points / locations on the UV atlas. Each sample location may be occupied or unoccupied if the location is inside a polygon defined by the mesh vertices. For each occupancy sample, the sample's corresponding 3D geometry coordinates and attributes may be calculated by interpolating from the associated polygon vertices.

[0030] According to one aspect of the present disclosure, the sampling rate may be consistent throughout the 2D atlas. In some embodiments, the sampling rates for the u-axis and v-axis may be different, allowing for anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, and each such region may have a different sampling rate.

[0031] According to one aspect of the present disclosure, the sampling rate for each region (or the entire 2D atlas) may be signaled in a high-level syntax, including, but not limited to, a sequence header, a frame header, a slice header, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be selected from a pre-established set of rates assumed by both the encoder and the decoder. Because the pre-established set of rates is known by both the encoder and the decoder, signaling one particular sampling rate simply requires signaling an index in the pre-established set of rates. Examples of such pre-established sets may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) of a mesh frame may be predicted from the pre-established set of rates, from previously used sampling rates in other already-coded regions of the same frame, or from previously used sampling rates within other already-coded mesh frames.

[0032] In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be based on some characteristic of each region (or the entire 2D atlas). As an example, the sample rate may be based on activity, and the sample rate may be set higher for regions (or the entire 2D atlas) with rich texture or high activity. As another example, the sample rate may be set lower for smooth regions (or the entire 2D atlas) or low activity regions (or the entire 2D atlas).

[0033] In some embodiments, the sampling rate for each region of the mesh frame (or the entire 2D atlas) may be signaled in a manner that may allow a combination of prediction and direct signaling. The syntax may be structured to indicate whether the sampling rate is predicted or directly signaled. If predicted, which predictor sampling rate should be used may be further signaled. If directly signaled, a syntax representing the value of the rate may be signaled.

[0034] In this disclosure, exemplary polygons such as triangular meshes are considered in more detail. A triangular mesh is a type of polygon mesh that includes a set of triangles connected by common edges or corners. The methods disclosed in this disclosure may be extended to other polygonal meshes, such as quadrilateral meshes, pentagonal meshes, etc.

[0035] In the related art, the similarity of triangles from 3D coordinates to texture coordinates is used to compress texture coordinates by the Draco method. Suppose point C, point N, and point P are three vertices of a triangle in a texture plane (e.g., a 2D plane), and C is the next vertex to be encoded.

[0036] Draco texture coordinate compression may include:

[0037] Step 1: If either N or P is after C in the encoding order list, compress the texture coordinates of C using delta coding.

[0038] Step 2: If the coding order of both N and P is before C, encode C using the predictive coding method: CX is perpendicular to NP. X and

number

number

[0039] Step 3: Calculate the Euclidean distance between C and the two candidates, and select the smaller one as the predicted coordinate of C. A flag is used to mark which is the predicted coordinate.

[0040] Step 4: Calculate the residual between the original coordinates and the predicted coordinates. Compress the residual and flags by using entropy coding.

[0041] The proposed embodiments may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0042] In this disclosure, several methods and systems for texture coordinate prediction in mesh compression are proposed. It should be noted that the methods and systems can be applied individually or in any combination. It should also be noted that the methods can be applied not only to dynamic meshes, but also to static meshes where there is only one frame of the mesh or the mesh content does not change over time.

[0043] Chart Segments Mesh parameterization is the process of one-to-one mapping between two surfaces, e.g., 2D or 3D mesh surfaces. For a mesh, given 3D vertex position values ​​along with connectivity information, a mesh parameterization algorithm can generate a texture atlas. A texture atlas is a non-overlapping map of 2D textures that can be mapped onto a mesh. In this disclosure, we use one mesh parameterization algorithm called isochart to illustrate our method. Isochart-based mesh parameterization involves performing an initial parameterization. Stretch optimization can then be performed in one or more iterations to reduce mesh distortion. Spectral clustering can then be performed to partition the surface into charts, which are recursively split until a predetermined stretch criterion is met. The disclosed method can also be applied to other mesh parameterization algorithms.

[0044] In isocharts, a mesh is partitioned into charts. Each chart is parameterized into a local texture domain. The local texture domain is a mapping of the chart's texture generated using the isocharts. All charts are then packed into a single texture plane to form a texture atlas. The texture plane is a flattened and / or planar representation of the chart in two dimensions.

[0045] In this disclosure, a method and system are disclosed that enables texture coordinate compression based on chart partitioning.

[0046] Texture Coordinate Prediction For a mesh, after both the position values ​​and connectivity information of 3D vertices are coded, chart partitioning is used to reconstruct the position values ​​and connectivity information. The chart partitioning can be based on an isochart or other mesh parameterization algorithm. After the chart partitioning, a chart with a parameterized local texture domain is generated. For example, the disclosed method can partition a mesh into multiple charts using mesh parameterization, each chart associated with a local texture domain. The disclosed method can then generate a texture plane based on the multiple charts, where the texture plane includes a triangle. Furthermore, the disclosed method can determine whether texture coordinates of a second vertex and a third vertex of a triangle are coded, where the first vertex, the second vertex, and the third vertex are different vertices of a triangle in the same texture plane. Next, the disclosed method can predict the texture coordinate of a first vertex of a triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle are coded. If at least one of the texture coordinates of the second vertex and the third vertex of the triangle is not coded, the disclosed method can compress the texture coordinate of the first vertex through delta coding. If the texture coordinates of both the second and third vertices of the triangle have been coded, the disclosed method predicts the texture coordinates of the first vertex based on a determination of whether the first vertex, the second vertex, and the third vertex are associated with the same local texture domain. The disclosed method then entropy encodes the residual between the predicted texture coordinates of the first vertex and the actual texture coordinates of the first vertex.

[0047] Figure 5 shows a more detailed example of the method described above. As shown in Figure 5, points C, N, and P are three vertices of a triangle in the texture plane, and C is the next vertex to be encoded. Texture coordinate compression, according to an embodiment, may work as follows:

[0048] If either N or P is not coded, the texture coordinates of C can be compressed with delta coding.

[0049] If both N and P are coded, check whether N, P, and C are vertices of a triangle in the local texture domain.

[0050] If N, P, and C are the vertices of a triangle in the local texture domain, then the local coordinates of N, P, and C are N, P, and C, respectively. local , P local , and C local In the local texture domain, C local From side N local P local Draw a perpendicular line to X and make the intersection point local Let X local , N local , P local , and C local is used to identify a point X in the texture plane, where X is on the edge NP or on an extension of NP, and preserves the ratio between the lengths |NX| and |XP| in the local texture domain, i.e., |NX| / |XP|=|N local X local | / |X local P local In the texture plane, draw a perpendicular line from point X to side NP, and the length is |CX| / |NP|=|C local X local | / |N local P local Choose a point C' on the perpendicular line such that |C' can then be used as a prediction of C in texture coordinates.

[0051] In the same or another embodiment, if N, P, and C do not belong to the same triangle in any local texture domain, then the Draco method is applied to predict the texture coordinates of C.

[0052] The process may include calculating the residual between the original coordinates and the predicted coordinates, and compressing the residual by using entropy coding.

[0053] Alternatively, an embodiment may include selecting a texture prediction from the local texture domain or the Draco method. Texture coordinate compression works as follows.

[0054] If either N or P is not coded, the texture coordinates of C can be compressed with delta coding.

[0055] If both N and P are coded, check whether N, P, and C are vertices of a triangle in the local texture domain.

[0056] If N, P, and C are the vertices of a triangle in the local texture domain, estimate the coding cost of texture coordinate prediction using the local texture domain and the coding cost of texture coordinate prediction using Draco, and signal the one with the smaller coding cost. Then, apply the texture coordinate prediction with the smaller coding cost.

[0057] If N, P, and C do not belong to the same triangle in any local texture domain, then we apply the Draco method to predict the texture coordinate of C.

[0058] The process may include calculating the residual between the original coordinates and the predicted coordinates, and compressing the residual and flags by using entropy coding.

[0059] Texture coordinate compression for preprocessing meshes In an embodiment, preprocessing may be applied to the mesh to improve mesh compression efficiency. For a mesh that is preprocessed by isochart, texture coordinate compression may work as follows:

[0060] If either N or P is not coded, the texture coordinates of C can be compressed with delta coding.

[0061] If both N and P are coded, C may be reconstructed from the local texture domain and therefore C may not be coded.

[0062] Those skilled in the art will recognize that the proposed methods may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0063] In this disclosure, a method and system for texture coordinate prediction in mesh compression are proposed. Those skilled in the art will recognize that the methods and systems can be applied individually or in any combination. It should also be noted that the methods can be applied not only to dynamic meshes, but also to static meshes where there is only one frame of the mesh or the mesh content does not change over time. Furthermore, the disclosed methods and systems are not limited to texture coordinate compression. For example, the disclosed methods and systems can also be applied to 3D coordinate compression.

[0064] As mentioned above, this disclosure uses triangular meshes to illustrate the disclosure in detail, but those skilled in the art will know that mesh polygons of any shape may be used.

[0065] FIG. 5 is a flow diagram illustrating a process 500 for mesh compression.

[0066] In operation 505, the mesh parameterization may be used to partition the mesh into multiple charts, in an embodiment, each chart being associated with a local texture domain.

[0067] In operation 510, a texture plane associated with the mesh may be generated based on the plurality of charts. The texture plane may be triangular in shape in an exemplary embodiment. In embodiments, the texture plane may be a shape other than triangular.

[0068] In one embodiment, generating a texture plane may include receiving, for a mesh, three-dimensional (3D) coordinates of a plurality of vertices and connectivity information associated with the 3D coordinates; partitioning the received mesh into two or more charts; generating texture domains for each of the two or more charts based on the 3D coordinates and the connectivity information; and generating a texture plane by combining the respective texture domains.

[0069] In operation 515, for a triangle in the texture plane, it may be determined whether the texture coordinates of the second vertex and the third vertex of the triangle are coded, where the first vertex, the second vertex, and the third vertex are different vertices of the triangle in the texture plane.

[0070] In an embodiment, operation 515 may also include predicting the texture coordinates of a first one of the vertices of the triangle based on whether the texture coordinates of a second and third vertex of the triangle are coded.

[0071] In an embodiment, the prediction includes determining whether the first vertex, the second vertex, and the third vertex are vertices of the same triangle in the local texture domain based on determining that the texture coordinates of the second vertex and the texture coordinates of the third vertex of the triangle in the texture plane have already been encoded; and determining a local intersection point on a line extending between the second local vertex and the third local vertex using a first perpendicular from the first local vertex to a line extending between the second local vertex and the third local vertex based on determining that the first vertex, the second vertex, and the third vertex are vertices of the same triangle in the local texture domain, and a third local vertex by a first ratio, where the first local vertex, the second local vertex, and the third local vertex are points in the local texture domain corresponding to the first vertex, the second vertex, and the third vertex in the texture plane; determining an intersection point in the texture plane corresponding to the local intersection point, where the intersection point divides the line extending between the second vertex and the third vertex by the first ratio; determining a first point in the texture plane located on a second perpendicular line from the intersection point to the line extending between the second vertex and the third vertex; and predicting texture coordinates of the first vertex based on the position of the first point in the texture plane.

[0072] In some embodiments, each of the respective texture domains is triangular in shape.

[0073] In some embodiments, based on determining that the first vertex, the second vertex, and the third vertex are not vertices of the same triangle in the local texture domain, the texture coordinates of the first vertex may be encoded using Draco coding.

[0074] In some embodiments, based on determining that at least one of the second vertex and the third vertex of a triangle in a texture plane is not encoded, the texture coordinates of the first vertex may be encoded using delta coding.

[0075] In operation 520, the determined residual between the predicted texture coordinates of the first vertex and the actual texture coordinates of the first vertex may be encoded using entropy coding.

[0076] In embodiments, the connectivity information may include coordinate information, edge information, shape information, texture information, etc. associated with the mesh. In embodiments, the mesh may be partitioned into two or more charts. In some embodiments, one or more charts may be isocharts. The isocharts may include isomaps or charts generated using any technique known to those skilled in the art.

[0077] In an embodiment, the texture plane may include a two-dimensional (2D) representation of a volumetric object represented in a mesh, and the texture coordinates may include 2D coordinates of vertices in the texture plane. In an embodiment, the texture domain may include a 2D representation of one or more charts.

[0078] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 6 illustrates a computer system 600 suitable for implementing certain embodiments of the present disclosure.

[0079] Computer software can be coded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly, or via interpretation, microcode execution, etc.

[0080] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0081] 6 for computer system 600 are examples and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the non-limiting embodiment of computer system 600.

[0082] The computer system 600 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images, still image captured from a camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0083] The input human interface devices may include one or more of a keyboard 601, a mouse 602, a trackpad 603, a touchscreen 610, a data glove, a joystick 605, a microphone 606, a scanner 607, and a camera 608 (only one of each is shown).

[0084] The computer system 600 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 610, data gloves, or joystick 605, although there may also be haptic feedback devices that do not function as input devices). For example, such devices may be audio output devices (such as speakers 609, headphones (not shown)), visual output devices (such as screens 610 including CRT screens, LCD screens, plasma screens, OLED screens, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), each with or without touchscreen input capability and each with or without haptic feedback capability, some of which may be capable of outputting output in more than three dimensions by means of two-dimensional visual output or stereoscopic output, etc.), and printers (not shown).

[0085] The computer system 600 may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 620 with CD / DVD or similar media 621, a thumb drive 622, a removable hard drive or solid state drive 623, legacy magnetic media such as tape or floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0086] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0087] The computer system 600 may also include interfaces to one or more communications networks. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet; cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc.; wired or wireless wide area digital networks for television including cable, satellite, and terrestrial broadcast television; vehicular and industrial networks including CANBus; and the like. Particular networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 649 (e.g., a USB port of computer system 600), while other networks are generally integrated into the core of computer system 600 by connection to the system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 600 can communicate with other entities. Such communications can be unidirectional receive only (e.g., broadcast television), unidirectional transmit only (e.g., a CANbus to a particular CANbus device), or bidirectional to other computer systems using, for example, local or wide-area digital networks. Such communications can include communications to a cloud computing environment 655. Particular protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0088] The aforementioned human interface devices, human-accessible storage devices, and network interface 654 may be connected to core 640 of computer system 600 .

[0089] The core 640 may include one or more central processing units (CPUs) 641, graphics processing units (GPUs) 642, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 643, hardware accelerators 644 for specific tasks, etc. These devices may be connected via a system bus 648, along with read-only memory (ROM) 645, random access memory 646, internal mass storage such as an internal non-user-accessible hard drive, SSD, etc. 647. In some computer systems, the system bus 648 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected to the core's system bus 648 directly or via a peripheral bus 649. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 650 may also be included in the core 640.

[0090] The CPU 641, GPU 642, FPGA 643, and accelerator 644 can execute specific instructions that, in combination, can constitute the aforementioned computer code. That computer code can be stored in ROM 645 or RAM 646. Persistent data can be stored, for example, in internal mass storage 647, while temporary data can also be stored in RAM 646. Fast storage and retrieval in any of the memory devices can be made possible by the use of cache memory, which can be closely associated with one or more of the CPU 641, GPU 642, mass storage 647, ROM 645, RAM 646, etc.

[0091] The computer-readable medium may bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0092] By way of example and not limitation, computer system 600 having the architecture, and specifically core 640, may provide functionality as a result of processor(s) (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices, as described above, as well as media associated with specific storage of core 640 that is non-transitory in nature, such as core internal mass storage device 647 or ROM 645. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by core 640. Computer-readable media may include one or more memory devices or chips, depending on particular needs. The software may cause core 640, and specifically the processor(s) therein (including CPU, GPU, FPGA, etc.), to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 646 and modifying such data structures according to the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 644), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software may encompass logic, where appropriate, and vice versa. References to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0093] While this disclosure describes several non-limiting embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art can devise numerous systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are thus within its spirit and scope. [Explanation of symbols]

[0094] 100 communication system, 110 terminal, 120 terminal, 130 terminal, 140 terminal, 150 network, 200 streaming system, 201 video source, 202 stream, 203 encoder, 204 encoded video bitstream, 205 streaming server, 206 streaming client, 209 video bitstream, 210 video decoder, 211 output video sample stream, 212 display, 213 capture subsystem, 300 framework, 301 encoder, 305 input mesh, 310 mesh with UV atlas, 315 occupancy map, 320 geometry map, 325 attribute map, 330 metadata, 335 decoded occupancy map, 340 decoded geometry map, 345 decoded attribute map, 350 decoded metadata, 351 decoder, 360 reconstructed mesh, 500 process for mesh compression, 600 computer system, 601 Keyboard, 602 Mouse, 603 Trackpad, 605 Joystick, 606 Microphone, 607 Scanner, 608 Camera, 609 Speaker, 610 Touchscreen, 620 CD / DVD ROM / RW, 621 Media such as CD / DVD, 622 Thumb drive, 623 Removable hard drive or solid state drive, 640 Core, 641 Central Processing Unit (CPU), 642 Graphics Processing Unit (GPU), 643 Field Programmable Gate Area (FPGA), 644 Hardware Accelerator, 645 Read Only Memory (ROM), 646 Random Access Memory, 647 Internal Mass Storage, 648 System Bus, 649 Peripheral Bus, 650 Graphics Adapter, 654 Network Interface, 655 Cloud Computing Environment, C Triangle Vertex, N Triangle Vertex, P Triangle Vertex

Claims

1. 1. A method for mesh compression, the method being executed by at least one processor, the method comprising: partitioning the mesh into a plurality of charts using a mesh parameterization, each chart being associated with a local texture domain; generating a texture plane based on the plurality of charts, the texture plane including triangles; determining whether texture coordinates of a second vertex and a third vertex of the triangle are coded, wherein the first vertex, the second vertex, and the third vertex are different vertices of the triangle in the texture plane; predicting texture coordinates of the first vertex of the triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle are coded; and entropy encoding a residual between the predicted texture coordinate of the first vertex and the actual texture coordinate of the first vertex.

2. The step of predicting the texture coordinate of the first vertex includes: determining whether the first vertex, the second vertex, and the third vertex are vertices of the same triangle in a local texture domain based on determining that the texture coordinates of the second vertex and the texture coordinates of the third vertex of the triangle in the texture plane have already been coded; based on determining that the first vertex, the second vertex, and the third vertex are vertices of the same triangle in the local texture domain, determining a local intersection point on the line extending between the second local vertex and the third local vertex using a first perpendicular to the line extending from the first local vertex to the line extending between the second local vertex and the third local vertex; the local intersection divides the line extending between the second local vertex and the third local vertex by a first ratio; the first local vertex, the second local vertex, and the third local vertex are points in the local texture domain corresponding to the first vertex, the second vertex, and the third vertex in the texture plane; determining an intersection point in the texture plane corresponding to the local intersection point, the intersection divides a line extending between the second vertex and the third vertex in the first ratio; and determining a first point in the texture plane located on a second perpendicular to the line extending from the intersection point between the second vertex and the third vertex; and predicting the texture coordinates of the first vertex based on a position of the first point in the texture plane.

3. The method comprises:

3. The method of claim 2, further comprising: encoding the texture coordinates of the first vertex using Draco coding based on determining that the first vertex, the second vertex, and the third vertex are not vertices of the same triangle in the local texture domain.

4. The step of predicting the texture coordinate of the first vertex includes:

2. The method of claim 1, further comprising: encoding the texture coordinates of the first vertex using delta coding based on determining that at least one of the second vertex and the third vertex of the triangle in the texture plane is not coded.

5. The step of generating a texture plane comprises: receiving, for the mesh, three-dimensional (3D) coordinates of a plurality of vertices and connectivity information associated with the 3D coordinates; Segmenting the received mesh into two or more charts; generating a local texture domain for each of the two or more charts based on the 3D coordinates and the connectivity information; and generating the texture plane by combining the respective local texture domains.

6. The method of claim 5 , wherein the two or more charts can be isocharts.

7. The method of claim 5 , wherein each of the respective local texture domains is triangular in shape.

8. 1. An apparatus for mesh compression, said apparatus comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to operate as instructed by said program code, said program code comprising: first partitioning code configured to cause the at least one processor to partition a mesh using a mesh parameterization into a plurality of charts, each chart associated with a local texture domain; and first generation code configured to cause the at least one processor to generate a texture plane based on the plurality of charts, the texture plane including triangles; and a first decision code configured to cause the at least one processor to determine whether texture coordinates of a second vertex and a third vertex of the triangle are coded, wherein the first vertex, the second vertex, and the third vertex are different vertices of the triangle in the texture plane; and a first prediction code configured to cause the at least one processor to predict texture coordinates of the first vertex of the triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle are coded; and and first encoding code configured to cause the at least one processor to entropy encode residuals between the predicted texture coordinates of the first vertex and actual texture coordinates of the first vertex.

9. The first predictive code comprises: second decision code configured to cause the at least one processor to determine, based on determining that the texture coordinate of the second vertex and the texture coordinate of the third vertex of the triangle in the texture plane have already been encoded, whether the first vertex, the second vertex, and the third vertex are vertices of the same triangle in a local texture domain; third decision code configured to cause the at least one processor, based on determining that the first vertex, the second vertex, and the third vertex are vertices of the same triangle in the local texture domain, to determine a local intersection on the line extending between the second local vertex and the third local vertex using a first perpendicular to the line extending from the first local vertex to the line extending between the second local vertex and the third local vertex; the local intersection divides the line extending between the second local vertex and the third local vertex by a first ratio; a third decision code, wherein the first local vertex, the second local vertex, and the third local vertex are points in the local texture domain corresponding to the first vertex, the second vertex, and the third vertex in the texture plane; fourth decision code configured to cause the at least one processor to determine an intersection point in the texture plane corresponding to the local intersection point, a fourth decision code, wherein the intersection divides a line extending between the second vertex and the third vertex by the first ratio; fifth decision code configured to cause the at least one processor to determine a first point in the texture plane located on a second perpendicular to the line extending from the intersection point between the second vertex and the third vertex; and second prediction code configured to cause the at least one processor to predict the texture coordinate of the first vertex based on a position of the first point in the texture plane.

10. The program code 10. The apparatus of claim 9, further comprising: sixth encoding code configured to cause the at least one processor to encode the texture coordinate of the first vertex using Draco coding based on determining that the first vertex, the second vertex, and the third vertex are not vertices of the same triangle in the local texture domain.

11. The first predictive code comprises:

9. The apparatus of claim 8, further comprising: third encoding code configured to cause the at least one processor to encode the texture coordinate of the first vertex using delta coding based on determining that at least one of the second vertex and the third vertex of the triangle in the texture plane is not encoded.

12. The first generated code is first receiving code configured to cause the at least one processor to receive, for the mesh, three-dimensional (3D) coordinates of a plurality of vertices and connectivity information associated with the 3D coordinates; second partitioning code configured to cause the at least one processor to partition the received mesh into two or more charts; seventh generation code configured to cause the at least one processor to generate a texture domain for each of the two or more charts based on the 3D coordinates and the connectivity information; and and eighth generation code configured to cause the at least one processor to generate the texture plane by combining the respective texture domains.

13. The apparatus of claim 12 , wherein the two or more charts can be isocharts.

14. The apparatus of claim 12 , wherein each texture domain of the respective texture domains is triangular in shape.

15. 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device for mesh compression, cause the one or more processors to: Using mesh parameterization, the mesh is partitioned into multiple charts, each chart being associated with a local texture domain; generating a texture plane based on the plurality of charts, the texture plane including triangles; determining whether texture coordinates of a second vertex and a third vertex of the triangle are coded, the first vertex, the second vertex, and the third vertex being different vertices of the triangle in the texture plane; predicting texture coordinates of the first vertex of the triangle based on whether the texture coordinates of the second vertex and the third vertex of the triangle are coded; a non-transitory computer-readable medium comprising one or more instructions that entropy encode a residual between the predicted texture coordinate of the first vertex and an actual texture coordinate of the first vertex;

16. Predicting the texture coordinate of the first vertex includes: determining whether the first vertex, the second vertex, and the third vertex are vertices of the same triangle in a local texture domain based on a determination that the texture coordinates of the second vertex and the texture coordinates of the third vertex of the triangle in the texture plane have already been encoded; based on determining that the first vertex, the second vertex, and the third vertex are vertices of the same triangle in the local texture domain, determining a local intersection point on the line extending between the second local vertex and the third local vertex using a first perpendicular to the line extending from the first local vertex to the line extending between the second local vertex and the third local vertex; the local intersection divides the line extending between the second local vertex and the third local vertex by a first ratio; the first local vertex, the second local vertex, and the third local vertex are points in the local texture domain corresponding to the first vertex, the second vertex, and the third vertex in the texture plane; determining an intersection point in the texture plane corresponding to the local intersection point, the intersection divides a line extending between the second vertex and the third vertex in the first ratio; and determining a first point in the texture plane located on a second perpendicular to the line extending from the intersection point between the second vertex and the third vertex; and predicting the texture coordinates of the first vertex based on a position of the first point in the texture plane.

17. the one or more processors; 17. The non-transitory computer-readable medium of claim 16, further comprising instructions to encode the texture coordinates of the first vertex using Draco coding based on determining that the first vertex, the second vertex, and the third vertex are not vertices of the same triangle in the local texture domain.

18. Predicting the texture coordinate of the first vertex includes:

16. The non-transitory computer-readable medium of claim 15, further comprising: encoding the texture coordinate of the first vertex using delta coding based on determining that at least one of the second vertex and the third vertex of the triangle in the texture plane is not encoded.

19. Generating the texture plane comprises: receiving, for the mesh, three-dimensional (3D) coordinates of a plurality of vertices and connectivity information associated with the 3D coordinates; Segmenting the received mesh into two or more charts; generating a texture domain for each of the two or more charts based on the 3D coordinates and the connectivity information; and generating the texture plane by combining the respective texture domains.

20. 16. The non-transitory computer-readable medium of claim 15, wherein the two or more charts can be isocharts.

Images