Rate-Distortion Optimization for Time-Varying Texture Mesh Compression

Abstract

An apparatus and method for encoding mesh data is disclosed. The disclosed technique includes receiving a sequence of frames, each including mesh data. For a frame in the sequence, a technique is disclosed for encoding the mesh data of the frame according to a static path and a motion path of a multi-path encoder, calculating static path costs for the encoding according to the static path and motion path costs for the encoding according to the motion path, where these costs are calculated by optimizing a rate-distortion cost function, and selecting a bitstream generated by encoding according to the motion path or the static path based on the calculated motion path costs and static path costs.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of European Patent Application No. 22306231.6, filed August 17, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] Computer-generated objects or camera-captured objects are commonly modeled by dynamic meshes. Large amounts of data are required for high-quality representation and rendering of content that includes dynamic meshes. Furthermore, efficient compression techniques aid in delivering and storing such content to consumers. Generally, a mesh consists of geometric data that represents the topology of a surface and attribute data that represents the physical properties of the surface. The geometric data of a mesh can be encoded directly or relative to a reference mesh. Because the distortion introduced by compressing the geometric data affects the distortion introduced by compressing the attribute data, the choice between direct and relative encoding of the geometric data affects the overall mesh compression efficiency. Therefore, the choice between direct and relative encoding should be made to improve overall encoding performance. Summary of the Invention

[0003] An aspect disclosed in the present disclosure describes a method for encoding mesh data. The method includes receiving a sequence of frames, each including mesh data. For a frame in the sequence, the method further includes: encoding the mesh data of the frame according to a static path and a motion path of a multi-path encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, where the costs are calculated by optimizing a rate-distortion cost function; and then selecting a bitstream generated by encoding according to the motion path or a bitstream generated by encoding according to the static path based on the motion path cost and the static path cost.

[0004] An aspect disclosed in the present disclosure describes an apparatus for encoding mesh data. The apparatus includes at least one processor and a memory that stores instructions. When executed by the at least one processor, the instructions cause the apparatus to receive a sequence of frames, each of which includes mesh data. For a frame in the sequence, the instructions further cause the apparatus to encode the mesh data of the frame according to a static path and a motion path of a multi-path encoder, calculate a static path cost for the encoding according to the static path and a motion path cost for the encoding according to the motion path, where the costs are calculated by optimizing a rate-distortion cost function, and then select a bitstream generated by encoding according to the motion path or the static path based on the motion path cost and the static path cost.

[0005] Aspects disclosed in the present disclosure describe a non-transitory computer-readable medium including instructions executable by at least one processor to perform a method for encoding mesh data. The method includes receiving a sequence of frames, each including mesh data. For a frame in the sequence, the method further includes: encoding the mesh data of the frame according to a static path and a motion path of a multi-path encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, where the costs are calculated by optimizing a rate-distortion cost function; and then selecting a bitstream generated by encoding according to the motion path or the static path based on the motion path cost and the static path cost.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form, which are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Moreover, the claimed subject matter is not limited to limitations that solve any or all of the disadvantages noted in any part of this disclosure. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a functional block diagram of an example system for dynamic mesh coding, according to one aspect of the present disclosure. [Figure 2] 1 is a functional block diagram of an example system for dynamic mesh decoding, according to one aspect of the present disclosure. [Figure 3] FIG. 2 is a functional block diagram of an exemplary base mesh encoder according to one aspect of the present disclosure. [Figure 4] FIG. 1 is a functional block diagram of an exemplary base mesh decoder according to one aspect of the present disclosure. [Figure 5]FIG. 2 is a functional block diagram of an exemplary multi-path encoder according to one aspect of the present disclosure. [Figure 6] 1 is a diagram of an exemplary picture group structure according to one aspect of the present disclosure. [Figure 7] 1 illustrates an exemplary method for determining static frames within a group of pictures, according to one aspect of the present disclosure. [Figure 8] 1 is a flow diagram of an exemplary method for multi-pass coding. DETAILED DESCRIPTION OF THE INVENTION

[0008] Following the MPEG V-Mesh CfP, the solution described by Mammou et al. ("Mammou") was selected to form the basis of the MPEG V-Mesh Test Model. See: K. Mammou, J. Kim, A. Tourapis and D. Podborski, m59281-[V-CG] Apple's Dynamic Mesh Coding CfP Response, Apple Inc, 2022. Mammou's proposed dynamic mesh coding is described herein with reference to FIGS. 1-4. As further described herein, the proposed dynamic mesh coding proposes first encoding the geometric data of a mesh i) directly and ii) relative to a reference mesh, and then selecting the encoding method (direct encoding or relative encoding) that results in the least geometric distortion. However, this approach does not consider how each coding method affects the distortion introduced by encoding non-geometric data (e.g., texture data) associated with the mesh or the bitrate of the coded data bitstream.

[0009] Apparatus and methods are disclosed for encoding a sequence of frames including mesh data. Aspects of a multi-pass encoder are described herein, including encoding the mesh data according to a static pass and a motion pass. The bitstream produced by the encoding pass that results in more efficient compression is selected, with efficiency measured by a rate-distortion cost function. The rate-distortion cost function includes geometric and texture terms, thus considering the overall impact of encoding according to the static pass and the motion pass, allowing for a more efficient selection of the encoding pass. Additionally, disclosed herein is adaptation of the group of pictures (GOP) structure of the sequence of frames based on each selected encoding pass.

[0010] In general, a mesh is a representation of a surface's topology, including vertices associated with three-dimensional (3D) positions on the surface. The vertices are connected by edges to form planes (such as triangles) that approximate the surface. Other information, i.e., vertex attributes (e.g., normal vectors and color values), may be associated with each of the mesh's vertices. In addition to its topology, a surface can be further represented by various attributes, such as texture. Typically, the texture of a surface is described by a two-dimensional (2D) image, i.e., a texture map. To associate the mesh's surface with corresponding texture data, the mesh's 3D surface is mapped into 2D space (e.g., UV parametric space). Similarly, the mesh's surface can be associated with other types of data, provided by other attribute maps, characterizing other physical properties of the surface (e.g., surface reflectance and transparency) that may be required for realistic rendering of the surface. Thus, a surface representation by mesh data includes topology data and attribute data, where the topology of the surface is represented by the mesh M (including geometry and connectivity information, and possibly vertex attributes), and the attributes of the surface are represented by attribute maps A (including attribute maps and respective mapping information). Aspects described herein with respect to texture data (represented by texture maps) are applicable to other types of data (generally represented by attribute maps).

[0011] 1 is a functional block diagram of an example system 100 for dynamic mesh coding according to one aspect of the present disclosure. The system 100 illustrates the encoding of a frame sequence F(i), where data associated with frame i includes a mesh M(i) 105 and a corresponding attribute map A(i) 110. The system 100 includes a mesh decomposer 120 (e.g., part of a preprocessing unit) and an encoder 130. The mesh decomposer 120 is configured to decompose a received mesh M(i) 105 into a base mesh m(i) and a corresponding displacement vector d(i). The generated base mesh m(i) and displacement vector d(i) are then provided to the encoder 130 along with the corresponding attribute map A(i) 110. The encoder 130 encodes the resulting data m(i), d(i), and A(i) to generate respective bitstreams therefrom, including a base mesh bitstream 170, a mesh displacement bitstream 175, and an attribute map bitstream 180. The operation of the mesh decomposer 120 and the operation of the encoder 130 are further described below.

[0012] The decomposer 120 is configured to decompose the mesh M(i) 105 into a base mesh m(i) and a corresponding displacement vector d(i). To generate the base mesh m(i), the decomposer 120 decimates the mesh M(i) by subsampling its vertices. The refined meshes are then generated by subdividing the base mesh m(i), i.e., each surface of the base mesh is subdivided into multiple subsurfaces and additional new vertices are introduced. Any refinement scheme may be applied, optionally iteratively. For example, each triangle of the base mesh surface may be divided into four subtriangles by introducing three new vertices in the middle of the triangle's sides and connecting the three vertices. The decomposer 120 then determines the displacement vector d(i) for each vertex of the refined base mesh, which, when applied to the vertices, generates a deformed mesh that spatially matches the received mesh M(i) 105. Decomposing the received mesh M(i) in this manner improves compression efficiency by allowing the base mesh m(i) and its corresponding displacement vector d(i) to be coded rather than directly coding mesh M(i) 105. This is because the base mesh has fewer vertices relative to mesh M(i) 105 and can therefore be coded using a relatively small number of bits. Furthermore, the displacement vector can be coded efficiently using, for example, a wavelet transform enabled by the subdivision structure. The subdivision structure used can then be determined by the decoder and does not need to be explicitly coded. For example, the decoder can refine the decoded base mesh based on the subdivision scheme type and subdivision iteration count, which can be signaled in the bitstream.

[0013] As shown in Figure 1, the encoder 130 includes a base mesh encoder 135, a base mesh decoder 140, a mesh displacement encoder 145, a mesh displacement decoder 150, a mesh reconstructor 155, and an attribute map encoder 160. The base mesh encoder 135 is configured to encode a base mesh m(i) into a coded base mesh cm(i) and generate a base mesh bitstream 170 therefrom. The base mesh decoder 140 is configured to reconstruct (decode) the base mesh from the coded base mesh cm(i), resulting in a reconstructed quantized base mesh m'(i) and a reconstructed base mesh m''(i). The base mesh encoder 135 and decoder 140 are further described with reference to Figures 3 and 4, respectively. The mesh displacement encoder 145 is configured to receive as input a base mesh m(i) and a reconstructed quantized base mesh m′(i), and based thereon, encode the received displacement vector d(i) into a coded displacement vector cd(i), and generate therefrom a mesh displacement bitstream 175. The mesh displacement decoder 150 is configured to reconstruct (decode) the displacement vector from the coded displacement vector cd(i), resulting in a reconstructed displacement vector d″(i). Based on the reconstructed base mesh m″(i) and the reconstructed displacement vector d″(i), the mesh reconstructor 155 is configured to reconstruct (decode) the mesh into a reconstructed mesh DM(i). Based on the mesh M(i) and the reconstructed mesh DM(i), the attribute map encoder 160 is configured to encode the attribute map A(i) into a coded attribute map, and generate therefrom an attribute map bitstream 180.

[0014] The mesh displacement encoder 145 encodes the displacement vector d(i) as described above and associates it with each vertex of the refined base mesh. To do so, the displacement vector is first updated based on the reconstructed and quantized base mesh m′(i). Next, a wavelet transform is applied to the updated displacement vector d′(i) according to the refinement structure into which the base mesh is refined. The wavelet coefficients are then quantized, packed into a 2D image, and compressed by a video encoder. The mesh displacement decoder 150 generally reverses the operation of the mesh displacement encoder 145. Thus, the mesh displacement decoder 150 uses a video decoder to decode the packed 2D image compressed by the video encoder of the mesh displacement encoder 145 (if the video encoder is lossy). Next, the mesh displacement decoder 150 unpacks the 2D image to obtain the quantized wavelet coefficients, applies inverse quantization, and then applies an inverse wavelet transform to generate the reconstructed displacement vector d″(i).

[0015] It should be noted that the video encoder is applied to the tasks of compressing the packed wavelet coefficients (by the mesh displacement encoder 145) and compressing the attribute map (by the attribute map encoder 160). Any video encoding method (either lossless or lossy) can be adopted for these tasks, according to the requirements of a particular application.

[0016] FIG. 2 is a functional block diagram of an example system 200 for dynamic mesh decoding, according to one aspect of the present disclosure. The system 200 is configured to generally reverse the operation of the system 100 and includes a decoder 230 and a mesh reconstructor 260. The decoder 230 includes a base mesh decoder 235, a mesh displacement decoder 240, and an attribute map decoder 250. The base mesh decoder 235 decodes the reconstructed base mesh m''(i) from the base mesh bitstream 210, 170, as further described with reference to FIG. 4. The mesh displacement decoder 240 decodes the reconstructed displacement vector d''(i) from the mesh displacement bitstream 215, 175, and performs the steps described with reference to the mesh displacement decoder 150 of FIG. 1. The attribute map decoder 250 decodes the attribute map from the attribute map bitstream 220, 180 and reverses the operation of the attribute map encoder 160 to generate a reconstructed attribute map DA(i) 275. The outputs of the decoder 230, the reconstructed base mesh m''(i) and the reconstructed displacement vector d''(i), are used by the mesh reconstructor 260 to reconstruct the decoded mesh DM(i) 270.

[0017] 3 is a functional block diagram of an exemplary base mesh encoder 300 according to one aspect of the present disclosure. The base mesh encoder 300 includes a quantizer 320, a static mesh encoder 340, a motion encoder 350, and a selector 360. As described above with reference to the base mesh encoder 135 of FIG. 1, the base mesh encoder 300 is configured to encode a base mesh m(i) into a base mesh bitstream 380. To this end, two encoders 340, 350 may be used. Thus, following quantization 320, the static mesh encoder 340 encodes the quantized base mesh qm(i) independently of any static mesh encoding method. Furthermore, following quantization 320, the motion encoder 350 encodes the quantized base mesh qm(i) relative to a reference reconstructed quantized base mesh m'(j) (e.g., associated with a previous base mesh m(i-1) in the frame sequence). That is, motion encoder 350 encodes a motion field f(i) that describes the motions that the vertices of m(j) need to perform to reach the respective positions of the corresponding vertices of m(i).

[0018] Therefore, m(i) and m(j) are assumed to share the same number of vertices and the same vertex connections, and only the positions of corresponding vertices in m(i) and m(j) change over time. In one aspect, to ensure that m(i) and m(j) have the same corresponding vertices, the encoder 300 tracks the transformation applied to m(j), obtains m'(j), and applies the same to m(i). Under such conditions, the motion encoder 350 may be configured to first calculate a motion field f(i) and then encode the calculated motion field into the base mesh bitstream 380. The motion field f(i) includes corresponding per-vertex motion vectors in the quantized base mesh qm(i) and the reference reconstructed quantized m'(j), as follows: f(i)=v1(i)-v2(j), (1) where v(i) is a vector q(i) containing the positions of the vertices of the mesh, and v(j) is a vector containing the positions of the corresponding vertices of the mesh m′(j). In one aspect, the motion encoder 350 may further adjust the motion vectors (e.g., based on neighboring motion vectors) and then encode the adjusted motion vectors, for example, using an entropy coder.

[0019] The selection of whether to use the output of the static mesh encoder 340 or the output of the motion encoder 350 can be performed by the selector 360. As mentioned above, Mammou proposes using the D2 feature of the MPEG mesh metric to select the bitstream of the encoder (either the static mesh encoder 340 or the motion encoder 350) that results in the least geometric distortion. That is, if the geometric distortion provided by the motion encoder 350 is smaller (or smaller than a predetermined threshold) than the geometric distortion provided by the static mesh encoder 340, the bitstream generated by the motion encoder 350 is used as the base mesh bitstream 380; otherwise, the bitstream generated by the static mesh encoder 340 is used as the base mesh bitstream 380. However, the D2 feature of the MPEG mesh metric used only reflects the geometric distortion and, for example, does not take into account the global rate.

[0020] 4 is a functional block diagram of an exemplary base mesh decoder 400 according to one aspect of the present disclosure. The base mesh decoder 400 generally reverses the operation of the base mesh encoder 300. The base mesh decoder 400 includes a static mesh decoder 440, a motion decoder 450, and an inverse quantizer 460. As described above with reference to the base mesh decoder 235 of FIG. 2, the base mesh decoder 400 is configured to decode the base mesh m''(i) reconstructed from the base mesh bitstream 420, 380. To do so, the base mesh decoder 400 sends the input base mesh stream 420 (representing the coded base mesh cm(i)) to either the static mesh decoder 440 or the motion decoder 450. Such indication can be based on signaling in the bitstream 420 indicating whether the coded base mesh cm(i) was coded by the static mesh encoder 340 or the motion encoder 350. If the bitstream 420 is sent to the static mesh decoder 440, the latter decodes the base mesh from the bitstream 420, resulting in a reconstructed quantized base mesh m'(i). Otherwise, if the bitstream 420 is sent to the motion decoder 450, the latter decodes the motion field from the bitstream 420 and adds the reconstructed (decoded) motion field f'(i) to the reference reconstructed quantized base mesh m'(j), resulting in a reconstructed quantized base mesh m'(i). The resulting m'(i) is then provided to the inverse quantizer 460, which generates a reconstructed base mesh m''(i) therefrom. As mentioned above, the base mesh decoder 400 is also used in the encoder 130, with the base mesh decoder 140 providing the reconstructed quantized base mesh m'(i) and the reconstructed base mesh m''(i) to the mesh displacement encoder 145 and the mesh reconstructor 155, respectively.

[0021] As mentioned above, the base mesh encoder 300 can choose 360 to directly encode the received base mesh m(i) for frame i (using the static mesh encoder 340) or to encode the received base mesh m(i) relative to a reference base mesh m(j) (using the motion encoder 350). In the latter case, what is encoded is a motion field f(i) that associates corresponding vertices from m(i) and m(j). As mentioned above, using the D2 features of the MPEG mesh metric to determine geometric distortion (based on which the choice of using either the static mesh encoder 340 or the motion encoder 350 is made) may not yield a better choice, as other cost sources introduced by the encoder 130 are not considered in this approach. A preferred approach is to consider the overall rate-distortion cost introduced by the encoder 130 when choosing between the output of the static mesh encoder 340 and the output of the motion path 350. Therefore, according to aspects disclosed herein, a rate-distortion optimization is performed that considers topological and photometric distortions as well as bitrate level. The rate-distortion optimization used can result in the selection of the encoder (340 or 350) that provides the more efficient coding corresponding to the optimal rate-distortion cost, as further explained with reference to FIG. 5.

[0022] FIG. 5 is a functional block diagram of an exemplary multi-path encoder 500 according to one aspect of the present disclosure. The multi-path encoder 500 includes a static path (SP) encoder 520 and a motion path (MP) encoder 525, each configured to encode mesh data of an input frame F(i) 510, including a mesh M(i) and a corresponding attribute map A(i). The SP encoder 520 may include components of the encoder 130 of the system 100 (FIG. 1) and the decoder 230 of the system 200 (FIG. 2), where the static mesh encoder 340 and the static mesh decoder 440 are used (referred to herein as the static path). The MP encoder 525 may include components of the encoder 130 of the system 100 (FIG. 1) and the decoder 230 of the system 200 (FIG. 2), where the motion encoder 350 and the motion decoder 450 are used (referred to herein as the motion path). As shown in FIG. 5, the SP encoder 520 encodes the decoded frame (DF SP Similarly, the MP encoder 525 outputs the decoded frames (DF MP (i)) and the respective MP bitstreams and their bit rates (i.e., MP bit rates).

[0023] The multi-pass encoder 500 generates a frame F(i) 510 and its decoded version DF SP Similarly, the multi-path encoder 500 also includes a frame F(i) 510 and its decoded version DF MPThe multi-path encoder 500 includes an MP distortion metric calculator 535 that calculates various distortions introduced by the MP encoder 525 based on (i). The SP cost calculator 540 of the multi-path encoder 500 is configured to calculate a rate-distortion cost of using the SP encoder 520 based on the SP distortion (provided by the SP distortion metric calculator 530) and based on the SP bitrate of the SP bitstream. Similarly, the MP cost calculator 545 is configured to calculate a rate-distortion cost of using the MP encoder 525 based on the MP distortion (provided by the MP distortion metric calculator 535) and based on the MP bitrate of the MP bitstream. Based on these calculated rate-distortion costs 540, 545, the selector 550 is configured to select either to output the SP bitstream as the output bitstream 560 of the multi-path encoder 500 or to output the MP bitstream as the output bitstream 560 of the multi-path encoder 500, as shown in FIG. 5 . Note that a frame F(i) for which an SP bitstream is selected is referred to herein as a static frame coded in static mode, and a frame F(i) for which an MP bitstream is selected is referred to herein as a motion frame coded in motion mode.

[0024] To choose between the static path 520 and the motion path 525 when encoding an input frame F(i)∈M(i),A(i), the coding cost J associated with each alternative path is calculated, and through an optimization process, the path (either the static path 520 or the motion path 525) that yields the lower optimal cost (or has an optimal cost below a predetermined threshold) is selected. The coding cost used may be a rate-distortion cost function: J = D + λR, (2) where D is the distortion metric, R is the bit rate value, and λ is the Lagrangian multiplier. The Lagrangian multiplier λ can be used to set a trade-off between the quality of the coded data (which is inversely proportional to the distortion D) and the bit rate R. For example, λ can be a function of the quantization parameter (QP) used by the quantizer of the encoder (e.g., λ ∝ e (((QP-3) / 6))) In one aspect, multiple Lagrangian multipliers may be used to balance the distortions introduced by the quantizers of the various encoders (e.g., the base mesh encoder 135, the mesh displacement encoder 145, and / or the attribute map encoder 160) and their respective bit rates.

[0025] The distortion metric D can be calculated as: D=D テックス +αD ジオ , (3) In the formula, D テックス represents the texture distortion, and D ジオ represents the geometric distortion. The parameter α can be set to balance between the texture distortion and the geometric distortion. For example, the texture distortion can be expressed as: D テックス =(β Y D Y +β U D U +β V D V ) / (β Y +β U +β V ), (4) In the formula, D Y , D U and D V denote the respective distortion values given by the coding of the luma Y component and the chroma U and V components of the texture map. The corresponding weighting values β Y , β U and β V can be used to balance different distortion sources. For example, the chroma distortion value D U and D V Luma distortion value D YTo balance the weighting, β Y =6 and β U =β V = 1. The distortion D is typically measured by the mean squared error (MSE) metric. The mean squared error metric is the average of squared error values derived based on a distance metric, and measures the difference between samples from the original data (e.g., texture map) and corresponding samples from the reconstructed data (e.g., reconstructed texture map).

[0026] As disclosed herein, the rate-distortion optimization process is performed by minimizing a cost function, J, over various coding modes (i.e., modes). An optimal cost, J, can be calculated for the static path 540 and for the motion path 545. Based on these optimal costs, it can be determined which bitstream (SP bitstream or MP bitstream) is selected as the output bitstream 560 (see FIG. 5). In one aspect, the cost function includes texture and geometric distortions and various bitrates associated with each bitstream 170, 175, 180 generated by the encoder 130. Optimization of such a cost function can be expressed as follows:

[0027]

number

[0028]

number

[0029]

number

[0030] In one aspect, another distortion metric, D, can be used in optimizing the rate-distortion cost function J. Two MPEG distortion metrics can be used to derive the distortion value. See: MDS21000_WG07_N00231, CfP for Dynamic Mesh Coding, MPEG, 2021-11-08. These metrics can be extended by applying them to data from several adjacent frames. The two metrics are described below.

[0031] In the first metric, the point cloud-based mesh distortion (PCMD) metric, the mesh M(i) and the reconstructed mesh DM(i) are geometrically sampled into color point clouds using their respective texture maps A(i) and DA(i) to compute the point clouds. The color point clouds are then used to calculate the geometric distortion measure MSE. D1 and MSE D2 and texture distortion measure MSE Y , MSE U and MSE V Note that texture distortion is typically affected by geometric distortion, as illustrated by the architecture of encoder 130 in Figure 1. These distortion measures can be combined into a single metric, which is MSE Y is the same as follows: PCMD=(a MSE D1 +b MSE D2 +c MSE Y +d MSE U +e MSE V ) / c (8) Here, the coefficients a, b, c, d, and e can be calculated as follows.

[0032] In the second metric, the Image-Based Sampling Distortion (IBSD) metric, the mesh M(i) and the reconstructed mesh DM(i) are rendered from several different viewpoints using their respective attribute maps. The rendered views are then compared using a geometric distortion measure, MSE, ジオ and texture distortion measure MSE Y , MSE U and MSE V As mentioned above, texture distortion is usually influenced by geometric distortion. These distortion measures can be combined into a single metric, MSE Y is the same as follows: IBSD=(a'MSE ジオ +b'MSE Y +c'MSE U +d'MSEV ) / b', (9) In the formula, the coefficients a', b', c', and d' can be determined as follows.

[0033] To determine the coefficients a, b, c, d, and e in Equation (8) and the coefficients a′, b′, c′, and d′ in Equation (9), a learning process can be utilized, for example, by using perceptual mean opinion scores (MOS) collected from a group of people. For example, each person in the group of people can be asked to rate a total of N videos, where M animated models were rendered using several distortion types and several distortion levels for each distortion type. The coefficients a, b, c, d, and e can then be estimated based on the collected ratings and their respective calculated PCMD metrics. Similarly, the coefficients a′, b′, c′, and d′ can be estimated based on the collected ratings and their respective IBSD metrics. In one aspect, the estimation of the coefficients is performed using leave-one-out cross-validation and learning. It should be noted that any other learning method can be used to estimate the values of the coefficients.

[0034] Additional distortion measures can be added to the PCMD and IBSD metrics, such as distortion measures that detect specific defects of interest (e.g., cracks in the reconstructed mesh surface). Furthermore, distortion measures can be combined linearly or nonlinearly by any function to generate the PCMD or IBSM metric. The measures in both the PCMD and IBSM metrics are scaled MSE. Y Note that the .times. ...

[0035] As disclosed herein, the optimization process according to equation (5) can be simplified by first optimizing the cost associated with encoding the geometric data and then proceeding to optimize the total cost, so that the rate-distortion cost J can be expressed as:

[0036]

number

[0037]

number

[0038]

number

[0039]

number

[0040] Therefore, in the first stage, a geo-mode that optimizes the first term of the static path cost (in Equation 10) is determined, resulting in the optimal geo-mode for the static path (i.e., sp-opt-geo-mode). Similarly, a geo-mode that optimizes the first term of the motion path cost (in Equation 10) is determined, resulting in the optimal geo-mode for the motion path (i.e., mp-opt-geo-mode). Next, the corresponding bitrate values for the optimal geo-mode for the static path and the optimal geo-mode for the motion path are calculated.

[0041]

number

[0042]

number

[0043]

number

[0044] The coding modes for which the rate-distortion cost J is optimized may each include any combination of parameters controlling the operation of the SP encoder 520 and the MP encoder 525. For example, the rate-distortion cost J may be optimized across coding modes defined by parameters such as QPs that are set to control quantizers in the base mesh encoder 135, the mesh displacement encoder 145, and / or the attribute map encoder 160, as described herein. Additionally, the rate-distortion cost J may be optimized across coding modes defined by parameters related to local QP adaptation, target resolution adaptation, and / or slice type adaptation.

[0045] In one aspect, the slice type associated with encoding 145, 160 of a frame's image data may be tied to the selected encoding path for the frame. Thus, because image data for a frame selected to be encoded by the static path is likely to have a different layout compared to image data for a previous frame, the slice type may be adjusted depending on the selected encoding path. For example, if the output of the static path is selected for the frame, the slice type is set to intra. Otherwise, if the output of the motion path is selected for the frame, the slice type is set to inter. Such heuristics can be used to reduce the search space of a rate-distortion optimization algorithm.

[0046] In another aspect, the GOP structure of a GOP sequence can be adapted based on the selected coding path. For example, a sequence of frames F(i) for which a motion path is selected is likely to be temporally stable, so the GOP structure of such a sequence can be dynamically adapted. Techniques for GOP structure adaptation are further described below.

[0047] FIG. 6 illustrates an exemplary GOP structure 600 according to one aspect of the present disclosure. A method for balancing rate-distortion costs across a GOP is to use a hierarchical GOP structure. In FIG. 6, the hierarchy of the GOP structure 600 is indicated by the temporal depth of the frames and the arrows indicating inter-coded dependencies between the frames and other reference frames. As shown, the first frame 610 of the GOP is an intra-coded frame and therefore does not depend on other reference frames for its encoding. Subsequent frames are inter-coded frames that depend on other reference frames for their encoding. Thus, as indicated by the arrows, frame 615 depends on reference frames 610 and 620; frame 620 depends on reference frames 610 and 630; frame 625 depends on reference frames 620 and 630; and so on. The first cycle of the GOP structure (including frames 615, 620, 625, and 630) is repeated by a second cycle of the same GOP structure (including frames 635, 640, 645, and 650), and so on through additional cycles until a new intraframe is coded at the start of a new GOP cycle.

[0048] Typically, each frame of a GOP is assigned a QP whose value relates to the frame's importance within the GOP. A frame's importance is related to its temporal depth and / or the number of times the frame is directly or indirectly referenced by other frames. Table 1 shows an intra-frame 610 and frames of the first cycle of a GOP structure 615, 620, 625, 630, along with their picture order counts (POC), slice types, QP offsets, temporal depths (represented by temporal identification numbers—Tid), and associated reference frames.

[0049] [Table 1]

[0050] The QP offset is an offset added to the target QP typically assigned to the entire GOP sequence. Thus, if the target QP is 32 and the QP offset for a frame is 2, a QP of 32 + 2 = 34 is used to encode that frame. Note that the offset can be adapted depending on the content. Tid indicates the temporal depth of the frame; the higher the Tid, the less impact the frame has on other frames. Reference frames are indicated by POC deltas, where parentheses indicate that the indicated reference frame is used only if available. For example, frame 610 has a POC of 0, a slice type of intra, a QP of 32, and a temporal depth of 0, and is an intra frame, so it does not depend on any reference frame. Frame 630 has a POC of 4, a slice type of inter, a QP of 33, and a temporal depth of 1, and it depends on reference frame 610 (i.e., its POC minus 4). In the second cycle, corresponding frame 650 depends on frame 630 (i.e., its POC minus 4) and frame 610 (i.e., its POC minus 8). The GOP structure is repeated as shown in Table 2, where several cycles of the GOP structure are shown. When a new intra-frame is coded, a new GOP sequence begins with a new cycle of the GOP structure. As described herein, the GOP sequence and GOP structure can be of any size with different numbers of temporal levels and different numbers of applied QP offsets.

[0051] [Table 2]

[0052] In one aspect, the GOP structure may be dynamically adapted to the selection 550 of static path coding or motion path coding. For example, in low-delay coding, the following GOP structure adaptation policy may be applied: The first frame of the GOP sequence is set as an intraframe and encoded using the static path encoder 520. Then, starting from the first cycle of the GOP structure, subsequent frames of the GOP sequence are processed. Thus, for each of the following frames, the following may be performed: 1) encoding according to the static path 520 is performed using a QP offset of 0 and an intra-slice type; 2) encoding according to the motion path 525 is performed using the frame's assigned QP according to its position in the GOP structure and the inter-slice type; 3) selecting 550 the bitstream generated by the more efficient coding path (520 or 525) to form the output bitstream 560; 4) if the selected bitstream is generated by the static path 520, restarting the cycle of the GOP structure; and 5) proceeding to step 1) to process the next frame. The above GOP structure adaptation policy is further demonstrated with reference to Tables 3 and 4.

[0053] [Table 3]

[0054] [Table 4]

[0055] Table 3 shows the GOP structure, indicating each frame's POC, slice type, QP offset, Tid, and associated reference frame. Using this GOP structure, Table 4 shows the process of adapting the GOP structure of a GOP sequence based on the GOP structure adaptation policy described above. Thus, the first frame (POC=0) is set as an intra frame and coded using static path 520. Next, for frames with POC=1, motion path coding (using QP=QP+3 and inter slice type) and static path coding (using QP=QP+0 and intra slice type) are performed, and the bitstreams generated by the motion paths are selected based on their respective RD costs. Next, for frames with POC=2, motion path coding (using QP=QP+2 and inter slice type) and static path coding (using QP=QP+0 and intra slice type) are performed, and the bitstreams generated by the motion paths are selected based on their respective RD costs. Next, for the frame with POC=3, motion path coding (using QP=QP+3 and inter-slice type) and static path coding (using QP=QP+0 and intra-slice type) are performed, and the bitstream generated by the static path is selected based on their respective RD costs. The selection of the bitstream generated by the static path for POC=3 prompts the restart of a new cycle of the GOP structure when processing the next frame with POC=4, as shown in Table 4.

[0056] In random access mode, the selection of whether to encode each frame of a GOP sequence using the static path 520 or the motion path 525 can be made in two stages. In the first stage, the frames in the GOP sequence for which the bitstream generated by the static path is selected are determined. These frames are referred to herein as static frames. In the second stage, the remaining frames are coded using the motion path. A technique for determining the static frames in a sequence is described with reference to FIG. 7.

[0057] 7 illustrates an example method 700 for determining a static frame in a GOP, according to one aspect of the present disclosure. Specifically, the method 700 determines the next static frame relative to a static frame S. To that end, in a first iteration 710, starting with static frame S=0, the method 700 iterates through frames S+k. * G is evaluated up to M frames, where M is the maximum (allowed) number of consecutive motion frames and G is the GOP size (e.g., frames 0+k for k=1, 2...M / 4). * 4 is evaluated). Therefore, for each of these frames, it is tested 550 whether to select a bitstream generated by the static path 520 or the motion path 525. As shown in FIG. 7, based on the testing of frames 4, 8, and 12, frame 12 is the first frame selected for encoding in static mode. In a second iteration 720, the same process is repeated for frames between 8 and 12, evaluating frames every G / 2 frames. In this iteration 720, frame 10 is the first frame selected for encoding in static mode. Then, in a third iteration 730, the same process is repeated for frames between 8 and 10, evaluating frames every G / 4 frames. In this iteration 730, encoding in motion mode is selected for frame 9, and therefore frame 10 is determined to be the next static frame following static frame 0. Therefore, the frames between frames 0 and 10 are coded using the motion path (these frames do not need to test a better coding path). The method 700 repeats starting from frame 10 (ie, S=10).

[0058] In one aspect, when determining the first static frame, it is not necessary to perform a full test (whether to select 550 the bitstream generated by the static path 520 or the bitstream generated by the motion path 525). Instead, an approximation heuristic can be used. Typical heuristics may be based on the energy of the frame differences or the amplitude of the motion vectors.

[0059] FIG. 8 is a flow diagram of an example method 800 for multi-path coding. The method 800 begins, at step 810, by receiving a sequence of frames including mesh data. Steps 820 through 840 may then be performed for each frame in the sequence. At step 820, the mesh data for the frames of the sequence is coded according to a static path and a motion path of a multi-path encoder. Next, at step 830, a static path cost for coding according to the static path and a motion path cost for coding according to the motion path are calculated. The costs may be calculated by optimizing a rate-distortion cost function. Next, at step 840, a selection is made between a bitstream generated by coding according to the motion path and a bitstream generated by coding according to the static path based on the motion path cost and the static path cost. In one aspect, if the motion path cost is lower than the static path cost, the bitstream generated by coding according to the motion path may be selected. In another aspect, if the motion path cost is lower than a predetermined threshold, the bitstream generated by coding according to the motion path may be selected.

[0060] As described above, the rate-distortion cost function includes a geometric term and a texture term (see Equation 5). In one aspect, optimization of the rate-distortion cost function may be performed in two stages (see Equation 10). In the first stage, the geometric term of the rate-distortion cost function is optimized to obtain an optimal mode (among the coding modes) and the respective bitrate. Then, in the second stage, the texture term of the rate-distortion cost function is optimized, the texture term including the respective bitrates provided by the first stage. The optimization of the rate-distortion cost function is across coding modes, each of which includes parameters that control the encoding of mesh data of the frame. In one aspect, such parameters may be associated with local QP adaptation, target resolution adaptation, slice type adaptation, or a combination thereof.

[0061] Method 800 may adapt the GOP structure of the received sequence of frames 810 based on the selection of the bitstream in step 840, as described with reference to Tables 3 and 4. To that end, for each frame in the sequence, the following may be performed: 1) the frame is coded (820) according to a static path, setting the slice type to intra and the quantization parameter to 0; 2) the frame is coded (820) according to a motion path, setting the slice type to inter and the quantization parameter according to the frame's position within the GOP structure; and 3) if a bitstream produced by coding according to the static path is selected (840), such selection causes method 800 to resume cycling through the GOP structures in the sequence of frames (as shown in Table 4).

[0062] Method 800 can also determine static and motion frames within a GOP, as described with reference to FIG. 7. For example, steps 820, 830, and 840 can be performed by method 800 on a subset of frames in a sequence, the subset following a current frame (e.g., frame 0 in FIG. 7) for which a bitstream generated by encoding according to a static path is selected. Next, a first frame in the subset is determined for which a bitstream generated by encoding according to a static path is selected. Mesh data of frames between the current frame and the first frame can then be encoded according to a motion path of a multi-path encoder. The subset of frames can be a series of frames located at the end of a cycle of a GOP structure of a sequence of frames (e.g., frames 4, 8, and 12 in FIG. 7), where the series of frames ends with the frame (e.g., frame 12) for which a bitstream generated by encoding according to a static path is selected. In one embodiment, the first frame may be recursively determined from among the frames between the frame located at the beginning of the last cycle of the series of frames (e.g., frame 8) and the candidate frame determined as the first frame in the previous iteration (e.g., frame 10 in iteration 720).

[0063] The illustrations of the aspects described herein are intended to provide a general understanding of the structure, functionality, and operation of the various aspects. The figures are not intended to serve as a complete description of all of the elements and features of apparatus and systems that utilize the structures or methods described herein. Many other aspects may become apparent to those skilled in the art upon reviewing this disclosure. Other aspects may be utilized and derived from this disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Accordingly, the disclosure and the figures should be considered illustrative, and not restrictive.

[0064] The description of the embodiments is provided to enable one to make or use the embodiments. Various modifications to these embodiments will be readily apparent, and the general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest possible scope consistent with the principles and novel features defined by the following claims.

Claims

1. 1. A method for encoding mesh data, comprising: receiving a sequence of frames each including mesh data; For frames in the sequence encoding the mesh data of the frame, the encoding following a static path and a motion path of a multi-path encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, wherein the costs are calculated by optimizing a rate-distortion cost function; and selecting a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path based on the motion path cost and the static path cost.

2. The method of claim 1 , wherein the rate-distortion cost function includes a geometric term and a texture term.

3. The method of claim 1 or 2, wherein the optimization of the rate-distortion cost function is over coding modes, each mode comprising parameters that control the encoding of the mesh data of the frame.

4. The method according to any one of claims 1 to 3, wherein the parameters of one of the coding modes are associated with one of the following: local QP adaptation, target resolution adaptation, or slice type adaptation.

5. The optimization of the rate-distortion cost function comprises: optimizing a geometric term of the rate-distortion cost function to determine an optimal one of the coding modes and a respective bit rate; 5. The method of claim 1, further comprising: optimizing a texture term of the rate-distortion cost function, the texture term comprising the respective bit rates.

6. The selection is 6. The method according to claim 1, further comprising selecting the bitstream generated by the encoding according to the motion path if the motion path cost is lower than the static path cost.

7. The selection is 6. The method according to claim 1, further comprising selecting the bitstream generated by the encoding according to the motion path if the motion path cost is lower than a predetermined threshold.

8. 6. The method of claim 1, further comprising adapting a GOP structure of the sequence of frames based on the selection, wherein the selection of the bitstream generated by the encoding according to the static path restarts the cycling of the GOP structure within the sequence.

9. The method of any one of claims 1 to 8, wherein the encoding according to the static path comprises setting a slice type to intra and a quantization parameter to 0.

10. The method according to any one of claims 1 to 8, wherein the encoding according to the motion path comprises setting a slice type to inter and setting a quantization parameter according to the frame position within the GOP structure.

11. performing the encoding, the calculation, and the selection for a subset of frames in the sequence, the subset following a current frame from which the bitstream generated by the encoding according to the static path is selected; determining a first frame in the subset from which the bitstream generated by the encoding according to the static path is selected; The method of any one of claims 1 to 7, further comprising: encoding the mesh data of frames between the current frame and the first frame according to the motion paths of the multi-path encoder.

12. The method of any one of claims 1 to 7 or 11, wherein the number of frames between the current frame and the first frame is less than a maximum number of consecutive motion frames.

13. said subset of frames being:

12. The method of claim 1, further comprising a series of frames located at the end of a cycle of a GOP structure of the sequence of frames, the series of frames ending with a frame for which the bitstream generated by the encoding according to the static path is selected.

14. The determination of the first frame includes:

14. The method according to claim 1, further comprising recursively determining the first frame from among frames between the frame located at the beginning of the last cycle of the cycle and a candidate frame determined as the first frame in a previous iteration.

15. 1. An apparatus for encoding mesh data, comprising: at least one processor; When executed by the at least one processor, the apparatus receiving a sequence of frames each including mesh data; For frames in the sequence encoding the mesh data of the frame, the encoding following a static path and a motion path of a multi-path encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, wherein the costs are calculated by optimizing a rate-distortion cost function; and selecting, based on the motion path cost and the static path cost, a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path.

16. The apparatus of claim 15 , wherein the rate-distortion cost function includes a geometric term and a texture term.

17. 17. The apparatus of claim 15 or 16, wherein the optimization of the rate-distortion cost function is across coding modes, each mode comprising parameters that control the encoding of the mesh data of the frame.

18. The apparatus of any one of claims 15 to 17, wherein the parameters of one of the coding modes are associated with one of local QP adaptation, target resolution adaptation, or slice type adaptation.

19. The optimization of the rate-distortion cost function comprises: optimizing a geometric term of the rate-distortion cost function to determine an optimal one of the coding modes and a respective bit rate; and optimizing a texture term of the rate-distortion cost function, the texture term comprising the respective bit rates.

20. The selection is 20. The apparatus of claim 15, further comprising selecting the bitstream generated by the encoding according to the motion path if the motion path cost is lower than the static path cost.

21. The selection is 20. The apparatus according to claim 15, further comprising selecting the bitstream generated by the encoding according to the motion path if the motion path cost is lower than a predetermined threshold.

22. 1. A non-transitory computer-readable medium comprising instructions executable by at least one processor to perform a method for encoding mesh data, the method comprising: receiving a sequence of frames each including mesh data; For frames in the sequence encoding the mesh data of the frame, the encoding following a static path and a motion path of a multi-path encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, wherein the costs are calculated by optimizing a rate-distortion cost function; and selecting, based on the motion path cost and the static path cost, a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path.

Images