Texture coordinate prediction in mesh compression

Abstract

A method, apparatus, and system for predicting texture coordinates to compress a mesh are provided. The process may include receiving, for the mesh, coordinates of a first vertex and coordinates of a predicted candidate vertex in three-dimensional (3D) space. The process may include determining a stretched normal distance associated with the first vertex and the predicted candidate vertex, where the stretched normal distance is based on a transformation of the 3D space to a two-dimensional (2D) space, and determining 2D texture coordinates of the first vertex based on the stretched normal distance associated with the first vertex and the predicted candidate vertex. The process may also include determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex, and compressing the mesh based on entropy coding the residual.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 404,399, filed September 7, 2022, and U.S. Patent Application No. 18 / 345,027, filed June 30, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This disclosure relates to a set of advanced video coding techniques, and more particularly to texture coordinate prediction in mesh compression. [Background technology]

[0003] Advanced three-dimensional (3D) representations of the world are enabling more immersive interaction and communication. To achieve the sense of realism of 3D representations, 3D models are becoming ever more sophisticated, and a significant amount of data is tied to the creation and consumption of these 3D models. 3D meshes are widely used in 3D model immersive content.

[0004] A 3D mesh may be composed of several polygons that describe the surface of a volumetric object. Each polygon may be defined by its vertices in 3D space and information about how the vertices are connected, called connectivity information. Such mapping is typically described by a set of parametric coordinates, called XYZ coordinates. In some embodiments, vertex attributes, such as color and normals, may also be associated with mesh vertices. Attributes may also be associated with the surface of a mesh by utilizing mapping information that parameterizes the mesh with a 2D attribute map. Such mapping is typically described by a set of parametric coordinates, called UV coordinates or texture coordinates, associated with mesh vertices. 2D attribute maps may be used to store high-resolution attribute information, such as texture, normals, and displacement. Such information can be used for various purposes, such as texture mapping and shading.

[0005] Dynamic mesh sequences can require large amounts of data because they may contain a significant amount of information that changes over time, and therefore efficient compression techniques are needed to store and transmit such content. Summary of the Invention [Problem to be solved by the invention]

[0006] Mesh compression standards IC, MESHGRID, and FAMC have previously been developed to address dynamic meshes with constant connectivity and time-varying geometry and vertex attributes. However, these standards do not consider time-varying attribute maps and connectivity information.

[0007] Furthermore, it is also difficult for volumetric acquisition techniques to generate constant connectivity dynamic meshes, especially under real-time constraints. This type of dynamic mesh content is not supported by existing standards. [Means for solving the problem]

[0008] According to embodiments, methods, apparatus, and non-transitory computer-readable media storing computer instructions for a process for predicting texture coordinates for compressing meshes may be provided.

[0009] According to one embodiment, the process may include receiving texture coordinates of a first vertex and texture coordinates of a predicted candidate vertex in three-dimensional (3D) space for a mesh, where the predicted candidate vertex is a common edge vertex of the first vertex and the first vertex and the common edge vertex are on opposite sides of two polygonal meshes that share an edge; determining a stretched normal distance associated with the first vertex and the predicted candidate vertex, where the stretched normal distance is based on a transformation of the 3D space into a two-dimensional (2D) space; and determining 2D texture coordinates of the first vertex based on the stretched normal distance associated with the first vertex and the predicted candidate vertex. The process may also include determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex, and compressing the mesh based on entropy coding the residual.

[0010] According to one embodiment, an apparatus may include at least one memory configured to store program code and at least one processor configured to read the program code and operate as directed by the program code. The program code may include: receiving code configured to cause the at least one processor to receive texture coordinates of a first vertex and texture coordinates of a predicted candidate vertex in three-dimensional (3D) space for a mesh, where the predicted candidate vertex is a common edge vertex of the first vertex and the first vertex and the common edge vertex are on opposite sides of two polygonal meshes that share an edge; first decision code configured to cause the at least one processor to determine a stretched normal distance associated with the first vertex and the predicted candidate vertex, where the stretched normal distance is based on a transformation of the 3D space to a two-dimensional (2D) space; and second decision code configured to cause the at least one processor to determine 2D texture coordinates of the first vertex based on the stretched normal distance associated with the first vertex and the predicted candidate vertex. The program code may include third determining code configured to cause the at least one processor to determine a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex, and first compressing code configured to cause the at least one processor to compress the mesh based on entropy coding the residual.

[0011] According to one embodiment, a non-transitory computer-readable medium storing instructions may include one or more instructions that, when executed by one or more processors of a device for predicting texture coordinates to compress a mesh, cause the one or more processors to: receive, for the mesh, texture coordinates of a first vertex and texture coordinates of a predicted candidate vertex in three-dimensional (3D) space, where the predicted candidate vertex is a common edge vertex of the first vertex and the first vertex and the common edge vertex are on opposite sides of two polygonal meshes that share an edge; determine a stretched vertical distance associated with the first vertex and the predicted candidate vertex, where the stretched vertical distance is based on a transformation of the 3D space to a two-dimensional (2D) space; and determine 2D texture coordinates of the first vertex based on the stretched vertical distance associated with the first vertex and the predicted candidate vertex. The one or more instructions may further cause the one or more processors to determine a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex and compress the mesh based on entropy coding the residual.

[0012] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of a communication system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a schematic diagram of a simplified block diagram of a streaming system, according to an embodiment of the present disclosure. [Figure 3] 1 is a schematic diagram of a simplified block diagram of a video encoder and decoder according to an embodiment of the present disclosure. [Figure 4] 1 is an exemplary diagram of texture coordinate prediction in the related art, in accordance with an embodiment of the present disclosure. [Figure 5A]FIG. 2 is a diagram of exemplary common edge vertices and candidate predicted vertices, according to an embodiment of the present disclosure. [Figure 5B] FIG. 1 is a diagram of texture coordinate prediction for compressing meshes, according to an embodiment of the present disclosure. [Figure 6] 1 is an exemplary flow diagram illustrating a process for generating a temporally correlated UV atlas according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a diagram of a computer system suitable for implementing embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0014] A mesh may contain several polygons that describe the surface of a volumetric object. As mentioned above, mesh geometry information consists of vertex connectivity information, 3D coordinates, texture coordinates, etc.

[0015] According to one aspect of the present disclosure, a method, system, and non-transitory storage medium for parallel processing of dynamic mesh compression are provided. Embodiments of the present disclosure may be applied to static meshes.

[0016] 1-2, one embodiment of the present disclosure for implementing the encoding and decoding structure of the present disclosure will be described.

[0017] 1 shows a simplified block diagram of a communication system 100 according to one embodiment of the present disclosure. The system 100 may include at least two terminals 110, 120 interconnected via a network 150. In the case of unidirectional data transmission, a first terminal 110 may encode video data, which may include mesh data, at a local location for transmission to another terminal 120 via the network 150. A second terminal 120 may receive the other terminal's coded video data from the network 150, decode the coded data, and display the decoded video data. Unidirectional data transmission may be common in media serving applications, for example.

[0018] 1 shows a second pair of terminals 130, 140 provided to support two-way transmission of coded video, such as may occur during a video conference. For two-way transmission of data, each terminal 130, 140 can code video data captured at a local location for transmission to the other terminal over network 150. Each terminal 130, 140 can also receive coded video data transmitted by the other terminal, decode the coded data, and display the recovered video data on a local display device.

[0019] In FIG. 1 , terminals 110-140 may be, for example, servers, personal computers, and smartphones, and / or any other type of terminal. For example, terminals 110-140 may be laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 150 represents any number of networks that convey coded video data among terminals 110-140, including, for example, wired and / or wireless communication networks. Communication network 150 may exchange data over circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 150 may not be important to the operation of the present disclosure, unless otherwise described herein below.

[0020] 2 illustrates the placement of a video encoder and decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter can be used in other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] 2, the streaming system 200 may include a capture subsystem 213 that includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0022] The video source 201 may, for example, create a stream 202 including a 3D mesh and metadata associated with the 3D mesh. The video source 201 may, for example, include a 3D sensor (e.g., a depth sensor) or 3D imaging technology (e.g., a digital camera) and a computing device configured to generate a 3D mesh using data received from the 3D sensor or 3D imaging technology. The sample stream 202, which may have a large amount of data compared to an encoded video bitstream, may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoder 203 may also generate an encoded video bitstream 204. The encoded video bitstream 204, which may have a smaller amount of data compared to the uncompressed stream 202, may be stored on a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to retrieve a video bitstream 209, which may be a copy of the encoded video bitstream 204.

[0023] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video bitstream 209, which is an input copy of the encoded video bitstream 204, and create an output video sample stream 211 that can be rendered on a display 212 or another rendering device (not shown). In some streaming systems, the video bitstreams 204, 209 may be encoded according to a particular video coding / compression standard.

[0024] FIG. 3 is an example diagram of a framework 300 for dynamic mesh compression and mesh decompression using an encoder and decoder.

[0025] 3, framework 300 may include encoder 301 and decoder 351. Encoder 301 may include one or more input meshes 305, one or more meshes with UV atlases 310, occupancy map 315, geometry map 320, attribute map 325, and metadata 330. Decoder 351 may include decoded occupancy map 335, decoded geometry map 340, decoded attribute map 345, decoded metadata 350, and reconstructed mesh 360.

[0026] According to one aspect of the present disclosure, the input mesh 305 may include one or more frames, each of which may be pre-processed by a series of operations and used to generate the mesh 310 with a UV atlas. By way of example, the pre-processing operations may include, but may not be limited to, tracking, parameterization, remeshing, voxelization, etc. In some embodiments, the pre-processing operations may be performed only on the encoder side and not on the decoder side.

[0027] The mesh with UV atlas 310 may be a 2D mesh. The 2D mesh with UV atlas may be a mesh in which each vertex of the mesh may be associated with a UV coordinate on the 2D atlas. The mesh with UV atlas 310 may be processed and converted into multiple maps based on sampling. As an example, the UV atlas 310 may be processed and converted into an occupancy map, a geometry map, and an attribute map based on sampling the 2D mesh with UV atlas. The generated occupancy map 335, geometry map 340, and attribute map 345 may be encoded using an appropriate codec (e.g., HVEC, VVC, AV1, etc.) and transmitted to a decoder. In some embodiments, metadata (e.g., connectivity information, etc.) may also be transmitted to the decoder.

[0028] According to one aspect, decoder 351 may receive the encoded occupancy map, geometry map, and attribute map from the encoder. Decoder 351 may use appropriate techniques and methods, in addition to the embodiments described herein, to decode the occupancy map, geometry map, and attribute map. In one embodiment, decoder 351 may generate decoded occupancy map 335, decoded geometry map 340, decoded attribute map 345, and decoded metadata 350. Input mesh 305 may be reconstructed into reconstructed mesh 360 based on decoded occupancy map 335, decoded geometry map 340, decoded attribute map 345, and decoded metadata 350 using one or more reconstruction filters and techniques. In some embodiments, metadata 330 may be sent directly to decoder 351, and decoder 351 may generate reconstructed mesh 360 using the metadata based on decoded occupancy map 335, decoded geometry map 340, and decoded attribute map 345. Post-filtering techniques may also be applied to the reconstructed mesh 360, including but not limited to remeshing, parameterization, tracking, voxelization, etc.

[0029] An input mesh with a 2D UV atlas may have vertices, and each vertex of the mesh may have associated UV coordinates on the 2D atlas. Occupancy maps, geometry maps, and attribute maps may be generated by sampling one or more points / locations on the UV atlas. Each sample location may be occupied or unoccupied if it is inside the polygon defined by the mesh vertices. For each occupied sample, its corresponding 3D geometry coordinates and attributes can be calculated by interpolating from the associated polygon vertices.

[0030] According to one aspect of the present disclosure, the sampling rate may be consistent throughout the 2D atlas. In some embodiments, the sampling rates for the u-axis and v-axis may be different, allowing for anisotropic remeshing. In some embodiments, the entire 2D atlas may be divided into multiple regions, such as slices or tiles, and each such region may have a different sampling rate.

[0031] According to one aspect of the present disclosure, the sampling rate for each region (or the entire 2D atlas) may be signaled within a high-level syntax, including, but not limited to, a sequence header, a frame header, a slice header, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be selected from a pre-established set of rates assumed by both the encoder and the decoder. For a pre-established set of rates known by both the encoder and the decoder, signaling one particular sampling rate only requires signaling an index within the pre-established set of rates. An example of such a pre-established set may be every 2 pixels, every 4 pixels, every 8 pixels, etc. In some embodiments, the sampling rate for each region (or the entire 2D atlas) of a mesh frame may be predicted from the pre-established set of rates, from sampling rates previously used in other already-coded regions of the same frame, or from sampling rates previously used in other already-coded mesh frames.

[0032] In some embodiments, the sampling rate for each region (or the entire 2D atlas) may be based on some characteristic of each region (or the entire 2D atlas). As an example, the sample rate may be based on activity, and for richly textured regions (or the entire 2D atlas), or for regions with high activity (or the entire 2D atlas), the sample rate may be set high. As another example, for smooth regions (or the entire 2D atlas), or for regions with low activity (or the entire 2D atlas), the sample rate may be set low.

[0033] In some embodiments, the sampling rate for each region of the mesh frame (or the entire 2D atlas) may be signaled in a way that allows for a combination of predicted and direct signaling. Syntax may be configured to indicate whether the sampling rate is predicted or directly signaled. When predicted, which predictor sampling rate should be used may be further signaled. When directly signaled, syntax representing the value of the rate may be signaled.

[0034] In this disclosure, exemplary polygons such as triangular meshes are considered in more detail. A triangular mesh is a type of polygonal mesh that includes a set of triangles connected by their common edges or corners. The methods disclosed in this disclosure may be extended to other polygonal meshes, such as quadrilateral meshes or pentagonal meshes.

[0035] In the related art, the similarity of triangles from 3D coordinates to texture coordinates is used by the Draco method to compress texture coordinates. As shown in Figure 4, points C, N, and P are three vertices of a triangle in a texture plane (e.g., a 2D planar mapping texture), and C is the next vertex to be coded. The 3D coordinates may include characteristics of a 3D mesh in three dimensions, and the texture coordinates may include characteristics of each 3D coordinate in two dimensions.

[0036] Draco texture coordinate compression may include:

[0037] Step 1: If either N or P is after C in the encoding order list, compress the texture coordinates of C with delta coding.

[0038] Step 2: If the coding order of both N and P is before C, use the predictive coding method to code C: CX is perpendicular to NP. X and

number

number

number

[0039] Step 3: Calculate the Euclidean distance between C and the two candidates, and select the smaller one as the predicted coordinate of C. A flag is used to mark which is the predicted coordinate.

[0040] Step 4: Calculate the residual between the original coordinates and the predicted coordinates. Compress the residual and flag by using entropy coding.

[0041] According to one embodiment of the present disclosure, a better way to capture the projection from 3D space onto the texture plane is to consider post-projection distance stretching. In the present disclosure, a method and system for texture coordinate prediction in mesh compression uses post-projection distance stretching.

[0042] Those skilled in the art will know that the proposed methods may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0043] This disclosure proposes methods and systems for texture coordinate prediction in mesh compression. Those skilled in the art will recognize that they can be applied individually or in any combination. It should also be noted that the methods can be applied not only to dynamic meshes, but also to static meshes, where the mesh has only one frame or the mesh content does not change over time. Furthermore, the disclosed methods and systems are not limited to texture coordinate compression. They can also be applied to, for example, 3D coordinate compression.

[0044] As mentioned above, the present disclosure uses triangular meshes to describe the disclosure in detail, but those skilled in the art will know that mesh polygons of any shape may be used.

[0045] The triangles in a triangle mesh may be ordered. The triangle order may be based on a particular order (e.g., traversal in the Edgebreaker algorithm). The vertices may be ordered. The vertex order may be based on a particular order (e.g., traversal in the Edgebreaker algorithm). The triangle and vertex orders may be based on the same scheme, such as traversal, or different schemes.

[0046] As seen in FIG. 5A, mesh 500 may include two triangle meshes with common edge vertices and predicted candidate vertices.

[0047] As can be seen in Figure 5A, an example of a common edge vertex is shown. For a vertex V, if there exists another vertex W such that V and W are opposite vertices of two triangles that share an edge in the texture plane, then W is a common edge vertex of V. During texture coordinate compression, if the texture coordinates of W and the texture coordinates of the two vertices of the common edge (which are vertices P and Q in Figure 5A) have already been coded, then W is a texture coordinate prediction candidate for V.

[0048] According to one embodiment, the texture coordinate of V may be encoded such that the vertical distance extension after projection from 3D space onto the texture plane is calculated and used to predict the texture coordinate accordingly.

[0049] FIG. 5B illustrates texture coordinate prediction for compressing a mesh from 3D coordinates in 3D space to texture coordinates in 2D space.

[0050] As can be seen in FIG. 5B, a diagram 550 shows the 3D and texture coordinates of V, respectively. xyz and V uv As can be seen in diagram 550, W is a texture coordinate compression prediction candidate for V.

[0051] In one embodiment, as seen in diagram 550, V xyz and W xyz From Edge P xyz Q xyz A perpendicular line to may be drawn in 3D space. The intersection vertex is E, as in figure 550. xyz and F xyz is.

[0052] In one embodiment, vertical prediction may be used.

[0053] In the texture plane, vertex E uv may be determined, where E uv Edge P uv Q uv , or P uv Q uv and the ratio between the lengths |PE| and |EQ| in 3D space, i.e., |P xyz E xyz | / |E xyz Q xyz |=|P uv E uv | / |E uv Q uv Similarly, for a vertex F in the texture plane, uv may be determined.

[0054] Then, in the texture plane, vertex E uv From the above, the vertical line is at edge P uv Q uv It may be drawn to length |V'E uv |=|V xyz E xyz |*|W uv F uv | / |W xyz F xyz A vertex V' may be selected on the vertical line so that |

[0055] Unlike texture coordinate compression in the prior art, in this disclosure, V' is always the same as the line P uv Q uv along the vertex W uv Therefore, in this disclosure, there is no need to flag which side the predicted vertex V′ belongs to.

[0056] Finally, the original coordinates V uv and the predicted coordinates V' may be calculated. The residual may be compressed by using entropy coding, which may be fixed length coding or variable length coding such as Exp-Golomb coding, arithmetic coding, Huffman coding, etc.

[0057] In one embodiment, parallel prediction may be used.

[0058] One embodiment of the present disclosure includes using parallel lines in the texture plane. uv From the line W uv F uv A line may be drawn parallel to the length |V'E uv |=|V xyz E xyz |*|W uv F uv | / |W xyz F xyz A vertex V' may be selected on the parallel line so that |

[0059] Then, the original coordinates Vuv The residual between V′ and the predicted coordinates V′ may be calculated and compressed using entropy coding, which may be fixed-length coding or variable-length coding such as Exp-Golomb coding, arithmetic coding, or Huffman coding.

[0060] In one embodiment, multiple predictors may be used for texture prediction.

[0061] When encoding the texture coordinates of V, in some embodiments, if there are more than one texture coordinate compression prediction candidate available, the average of the predictions may be used. M where M is a positive integer, the average V' is V'=(V1+V2+...+V M ) / M.

[0062] Original coordinate V uv The residual between V and the mean V' may be calculated and compressed using entropy coding, which may be fixed length coding or variable length coding such as Exp-Golomb coding, arithmetic coding, Huffman coding, etc.

[0063] In one embodiment, a constrained predictor may be used for texture prediction.

[0064] In one embodiment, an upper limit L may be set indicating how many texture coordinate compression prediction candidates may be considered, where L may be a positive integer. In this embodiment, if there are more than L texture coordinate compression prediction candidates for a vertex V, the first L prediction candidates and their associated prediction values ​​V1, V2, ..., V L Only V'=(V1+V2+...+V L ) / L.

[0065] Original coordinate V uv The residual between V and the mean V' may be calculated and compressed using entropy coding, which may be fixed length coding or variable length coding such as Exp-Golomb coding, arithmetic coding, Huffman coding, etc.

[0066] FIG. 6 is a flow diagram illustrating a process 600 for predicting texture coordinates for compressing a mesh.

[0067] In operation 605, coordinates of a first vertex and coordinates of predicted candidate vertices in three-dimensional (3D) space may be received for a mesh. The predicted candidate vertices may be common edge vertices of the first vertex, and the first vertex and the common edge vertex may be on opposite sides of two polygonal meshes that share an edge. As an example, as seen in figure 550, V xyz and W xyz 3D coordinates may be received.

[0068] In some embodiments, determining the stretched vertical distance may include determining a first vertical line in 3D space from the first vertex to the shared edge, where the first vertical line divides the shared edge by a first ratio; determining a first intersection point in 2D space that divides the shared edge by the first ratio; determining a second vertical line in 3D space from the predicted candidate vertex to the shared edge, where the second vertical line divides the shared edge by a second ratio; and determining a second intersection point in 2D space that divides the shared edge by the second ratio.

[0069] At operation 610, a stretched vertical distance associated with the first vertex and the predicted candidate vertex may be received. In some embodiments, the stretched vertical distance may be based on a transformation of 3D space into two-dimensional (2D) space. As an example, to determine the stretched vertical distance, V xyz E xyz and W xyz F xyzmay be used.

[0070] In operation 615, the 2D texture coordinates of the first vertex may be predicted based on the stretched vertical distance associated with the first vertex and the predicted candidate vertex. uv is V xyz E xyz and W xyz F xyz It may be determined based on the

[0071] In some embodiments, determining the 2D texture coordinates of the first vertex may include determining, in 2D space, predicted coordinates of the first vertex on a predicted line, the predicted line being a line perpendicular to the shared edge and passing through the first intersection point, and the predicted coordinates of the first vertex on the predicted line being based on the second perpendicular line and the second intersection point.

[0072] In some embodiments, determining the 2D texture coordinates of the first vertex may include determining, in 2D space, predicted coordinates of the first vertex on a parallel line, the parallel line being a line that is parallel to the second perpendicular line and passes through the first intersection point, and the predicted coordinates of the first vertex on the parallel line being based on a distance relationship between the second perpendicular line and the second intersection point.

[0073] In some embodiments, the coordinates of the predicted candidate vertices may include coordinates of two or more predicted candidate vertices, and thus the 2D texture coordinates of the first vertex may be based on a first vertex among the two or more predicted candidate vertices and a respective stretched vertical distance associated with each predicted candidate vertex. In some embodiments, the number of predicted candidate vertices included in the two or more predicted candidate vertices may be less than a first threshold.

[0074] In some embodiments, process 600 may include an operation 620 of encoding predicted coordinates of a first vertex of the mesh.

[0075] In some embodiments, process 600 may also include operations 620 and 625 .

[0076] In operation 620, a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex may be determined. As an example, the original coordinate V uv A residual between the predicted coordinates V′ and the predicted coordinates V′ may be determined.

[0077] In operation 625, the mesh may be compressed based on entropy coding of the residual. As an example, consider the original coordinates V uv The residual between V′ and the predicted coordinates V′ may be compressed using entropy coding, which may be fixed-length coding or variable-length coding such as Exp-Golomb coding, arithmetic coding, or Huffman coding.

[0078] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system 700 suitable for implementing certain embodiments of the present disclosure.

[0079] Computer software can be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similar mechanisms to create code containing instructions that can be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., directly, or via interpretation, microcode execution, etc.

[0080] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0081] 7 for computer system 700 are examples and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be construed as having any dependency or requirement regarding any one or combination of components illustrated in the non-limiting embodiment of computer system 700.

[0082] The computer system 700 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly related to conscious human input, such as audio (such as voice, music, or ambient sounds), images (such as scanned images or photographic images obtained from a still image camera), or video (such as two-dimensional video, three-dimensional video, including stereoscopic video).

[0083] The input human interface devices may include one or more of a keyboard 701, a mouse 702, a trackpad 703, a touchscreen 710, a data glove, a joystick 705, a microphone 706, a scanner 707, and a camera 708 (only one of each is shown).

[0084] The computer system 700 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 710, data gloves, or joystick 705, although haptic feedback devices that do not function as input devices may also be present). For example, such devices may include audio output devices (such as speakers 709, headphones (not shown)), visual output devices (such as screens 710, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capabilities, each with or without haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions, such as through means of stereoscopic output), and printers (not shown).

[0085] The computer system 700 may also include human-accessible storage devices and media associated with the storage devices, such as optical media including CD / DVD ROM / RW 720 with CD / DVD or similar media 721, thumb drives 722, removable hard drives or solid state drives 723, legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0086] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0087] The computer system 700 may also include interfaces to one or more communication networks. The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; wireless LANs; cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; television wired or wireless wide-area digital networks including cable, satellite, and terrestrial broadcast television; vehicular and industrial networks including CANBus; and the like. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port on the computer system 700) or peripheral bus 749; other networks are generally integrated into the core of the computer system 700 by connection to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 700 can communicate with other entities. Such communications may be, for example, one-way receive only (e.g., broadcast television), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, to other computer systems using local or wide area digital networks. Such communications may include communications to a cloud computing environment 755. Specific protocols and protocol stacks may be used with each of these networks and network interfaces, as described above.

[0088] The aforementioned human interface devices, human accessible storage devices, and network interface 754 may be attached to the core 740 of the computer system 700 .

[0089] The core 740 may include one or more central processing units (CPUs) 741, graphics processing units (GPUs) 742, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 743, hardware accelerators 744 for specific tasks, etc. These devices may be connected via a system bus 748, along with read-only memory (ROM) 745, random access memory 746, and internal mass storage device 747, such as an internal non-user-accessible hard drive, SSD, etc. In some computer systems, the system bus 748 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 748 or via a peripheral bus 749. Architectures for peripheral buses include PCI, USB, etc. A graphics adapter 750 may be included with the core 740.

[0090] The CPU 741, GPU 742, FPGA 743, and accelerator 744 may execute specific instructions that, in combination, may constitute the aforementioned computer code. That computer code may be stored in ROM 745 or RAM 746. Transient data may also be stored in RAM 746, while persistent data may be stored, for example, in internal mass storage device 747. Fast storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 741, GPU 742, mass storage device 747, ROM 745, RAM 746, etc.

[0091] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0092] By way of example and not limitation, a computer system having architecture 700, and specifically core 740, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be the user-accessible mass storage devices introduced above, as well as media associated with specific storage of core 740 that is non-transitory in nature, such as core internal mass storage 747 or ROM 745. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by core 740. The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause core 740, and specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to perform particular processes or particular portions of particular processes described herein, including defining data structures stored in RAM 746 and modifying such data structures according to software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 744) that can operate in place of or together with software to perform particular processes or portions of particular processes described herein. Where appropriate, references to software may encompass logic, and vice versa. Where appropriate, references to computer-readable media may encompass circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0093] While this disclosure describes several non-limiting embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope. [Explanation of symbols]

[0094] 100 Communication Systems 110 Terminal 120 terminals 130 terminals 140 terminals 150 Communication Network 200 Streaming System 201 Video Sources 202 Sample Stream 203 Encoder 204 coded video bitstream 205 Streaming Server 206 Streaming Client 209 Video Bitstream 210 Video Decoder 211 output video sample stream 212 Display 213 Capture Subsystem 300 Framework 301 Encoder 305 input meshes 310 meshes with UV atlas 315 Occupancy Map 320 Geometric Map 325 Attribute Map 330 Metadata 335 Decoding Occupancy Map 340 Decoded Geometry Map 345 Decrypt Attribute Map 350 Decrypt Metadata 351 Decoder 360 Restoration Mesh 500 mesh 550 shapes 600 processes 605 operation 610 operation 615 operation 620 operation 625 operation 700 Computer Systems 701 Keyboard 702 Mouse 703 Trackpad 705 Joystick 706 Microphone 707 Scanner 708 Camera 709 Speaker 710 Touchscreen 720 CD / DVD ROM / RW 721 CD / DVD or similar media 722 thumb drive 723 Removable Hard Drive or Solid State Drive 740 cores 741 Central Processing Unit (CPU) 742 Graphics Processing Unit (GPU) 743 Field Programmable Gate Area (FPGA) 744 Hardware Accelerator 745 Read-Only Memory (ROM) 746 Random Access Memory (RAM) 747 Internal Mass Storage 748 System Bus 749 Peripheral Bus 750 graphics adapter 754 network interface 755 Cloud Computing Environment

Claims

1. 1. A method for predicting texture coordinates for compressing a mesh, the method being executed by at least one processor, the method comprising: receiving coordinates of a first vertex and coordinates of predicted candidate vertices in three-dimensional (3D) space for a mesh, the first vertex and the predicted candidate vertex being on opposite sides of two polygonal meshes that share an edge in a texture plane; determining a stretched vertical distance associated with the first vertex and the predicted candidate vertex, the stretched vertical distance being based on a transformation of the 3D space into a two-dimensional (2D) space; predicting texture coordinates of the first vertex based on the stretched normal distance associated with the first vertex and the predicted candidate vertex; encoding the predicted texture coordinates of the first vertex of the mesh; A method comprising:

2. determining the vertical extension distance comprises: determining a first perpendicular line in the 3D space from the first vertex to the shared edge, the first perpendicular line dividing the shared edge by a first ratio; determining a first intersection point in the 2D space that divides the shared edge by the first ratio; determining a second perpendicular line in the 3D space from the predicted candidate vertex to the shared edge, the second perpendicular line dividing the shared edge by a second ratio; determining a second intersection point in the 2D space that divides the shared edge by the second ratio; 2. The method of claim 1, comprising:

3. determining the predicted texture coordinate of the first vertex, determining, in the 2D space, a predicted coordinate of the first vertex on a predicted line, the predicted line being a line perpendicular to the shared edge and passing through the first intersection point, the predicted coordinate of the first vertex on the predicted line being based on the second perpendicular line and the second intersection point; 3. The method of claim 2, comprising:

4. The method comprises: determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex; compressing the mesh based on entropy coding the residual; 4. The method of claim 3, further comprising:

5. determining the predicted texture coordinate of the first vertex, determining, in the 2D space, a predicted coordinate of the first vertex on a parallel line, the parallel line being a line that is parallel to the second perpendicular line and passes through the first intersection point, the predicted coordinate of the first vertex on the parallel line being based on a distance relationship between the second perpendicular line and the second intersection point; 3. The method of claim 2, comprising:

6. The method comprises: determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex; compressing the mesh based on entropy coding the residual; 6. The method of claim 5, further comprising:

7. 2. The method of claim 1 , wherein the coordinates of the predicted candidate vertices include coordinates of two or more predicted candidate vertices, and the predicted texture coordinate of the first vertex is based on the first vertex among the two or more predicted candidate vertices and a respective stretched vertical distance associated with each predicted candidate vertex.

8. The method of claim 7 , wherein the number of predicted candidate vertices included in the two or more predicted candidate vertices is less than a first threshold.

9. 1. An apparatus for predicting texture coordinates for compressing a mesh, said apparatus comprising: at least one memory configured to store program code; at least one processor configured to read said program code and to act as directed by said program code, said program code comprising: receiving code configured to cause the at least one processor to receive, for a mesh, coordinates of a first vertex and coordinates of predicted candidate vertices in three-dimensional (3D) space, the first vertex and the predicted candidate vertex being on opposite sides of two polygonal meshes that share an edge in a texture plane; first decision code configured to cause the at least one processor to determine a stretched vertical distance associated with the first vertex and the predicted candidate vertex, the stretched vertical distance being based on a transformation of the 3D space to a two-dimensional (2D) space; and second decision code configured to cause the at least one processor to predict texture coordinates of the first vertex based on the stretched normal distance associated with the first vertex and the predicted candidate vertex; encoding code configured to cause the at least one processor to encode the predicted texture coordinates of the first vertex of the mesh; and at least one processor including: An apparatus comprising:

10. The first decision code is third decision code configured to cause the at least one processor to determine, in the 3D space, a first perpendicular line from the first vertex to the shared edge, wherein the first perpendicular line divides the shared edge by a first ratio; and fourth decision code configured to cause the at least one processor to determine, in the 2D space, a first intersection point that divides the shared edge by the first ratio; fifth decision code configured to cause the at least one processor to determine, in the 3D space, a second perpendicular line from the predicted candidate vertex to the shared edge, the second perpendicular line dividing the shared edge by a second ratio; and sixth decision code configured to cause the at least one processor to determine, in the 2D space, a second intersection point that divides the shared edge by the second ratio; and 10. The apparatus of claim 9, comprising:

11. The second decision code is seventh decision code configured to cause the at least one processor to determine, in the 2D space, a predicted coordinate of the first vertex on a predicted line, the predicted line being a line perpendicular to the shared edge and passing through the first intersection point, and the predicted coordinate of the first vertex on the predicted line being based on the second perpendicular line and the second intersection point.

11. The apparatus of claim 10, comprising:

12. The program code: an eighth decision code configured to cause the at least one processor to determine a residual between the predicted coordinate of the first vertex and an actual 2D coordinate of the first vertex; and a first compression code configured to cause the at least one processor to compress the mesh based on entropy coding the residual; and The apparatus of claim 11 further comprising:

13. The second decision code is a ninth decision code configured to cause the at least one processor to determine, in the 2D space, a predicted coordinate of the first vertex on a parallel line, the parallel line being a line parallel to the second perpendicular line and passing through the first intersection point, the predicted coordinate of the first vertex on the parallel line being based on a distance relationship between the second perpendicular line and the second intersection point.

11. The apparatus of claim 10, comprising:

14. The program code: tenth decision code configured to cause the at least one processor to determine a residual between the predicted coordinate of the first vertex and an actual 2D coordinate of the first vertex; a second compression code configured to cause the at least one processor to compress the mesh based on entropy coding the residual; and 14. The apparatus of claim 13, further comprising:

15. 1. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device for predicting texture coordinates for compressing a mesh, cause the one or more processors to: receiving, for a mesh, coordinates of a first vertex and coordinates of predicted candidate vertices in three-dimensional (3D) space, the first vertex and the predicted candidate vertex being on opposite sides of two polygonal meshes that share an edge in a texture plane; determining a stretched vertical distance associated with the first vertex and the predicted candidate vertex, the stretched vertical distance being based on a transformation of the 3D space into a two-dimensional (2D) space; predicting texture coordinates of the first vertex based on the stretched vertical distance associated with the first vertex and the predicted candidate vertex; encoding the predicted texture coordinates of the first vertex of the mesh; A non-transitory computer-readable medium comprising one or more instructions for causing a

16. determining the vertical distance of extension determining, in the 3D space, a first perpendicular line from the first vertex to the shared edge, the first perpendicular line dividing the shared edge by a first ratio; determining a first intersection point in the 2D space that divides the shared edge by the first ratio; determining, in the 3D space, a second perpendicular line from the predicted candidate vertex to the shared edge, the second perpendicular line dividing the shared edge by a second ratio; determining a second intersection point in the 2D space that divides the shared edge by the second ratio; and 16. The non-transitory computer-readable medium of claim 15, comprising:

17. Determining the predicted texture coordinate of the first vertex includes: determining, in the 2D space, a predicted coordinate of the first vertex on a predicted line, the predicted line being a line perpendicular to the shared edge and passing through the first intersection point, the predicted coordinate of the first vertex on the predicted line being based on the second perpendicular line and the second intersection point; 17. The non-transitory computer-readable medium of claim 16, comprising:

18. The instructions are sent to the one or more processors. determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex; compressing the mesh based on entropy coding of the residual; 20. The non-transitory computer-readable medium of claim 17, further comprising:

19. Determining the predicted texture coordinate of the first vertex includes: determining, within the 2D space, a predicted coordinate of the first vertex on a parallel line, the parallel line being a line parallel to the second perpendicular line and passing through the first intersection point, the predicted coordinate of the first vertex on the parallel line being based on a distance relationship between the second perpendicular line and the second intersection point; 17. The non-transitory computer-readable medium of claim 16, comprising:

20. The instructions are sent to the one or more processors. determining a residual between the predicted coordinate of the first vertex and the actual 2D coordinate of the first vertex; compressing the mesh based on entropy coding of the residual; 20. The non-transitory computer-readable medium of claim 19, further comprising:

Images