Rate distortion optimization for time-varying texture mesh compression

Abstract

An apparatus and method for encoding grid data are disclosed. The disclosed techniques include receiving a sequence of frames, each frame in the sequence of frames including grid data. For a frame in a sequence, techniques are disclosed for encoding grid data of the frame according to a static path of a multipath encoder and according to a motion path of the multipath encoder; calculating a static path cost of the encoding according to the static path and a motion path cost of the encoding according to the motion path, wherein a cost is calculated by optimizing a rate distortion cost function; and then selecting a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path based on the calculated motion path cost and static path cost.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of European Application No. EP22306231.6 filed on August 17, 2022, which is incorporated herein by reference. Background Art

[0003] Computer generated or camera captured objects are often modeled by dynamic meshes. High quality representation and rendering of content containing dynamic meshes requires large amounts of data. In addition, effective compression techniques help deliver such content to consumers and store the content. Typically, a mesh consists of geometric data representing the surface topology and attribute data representing the physical properties of the surface. The geometric data of a mesh can be encoded directly or relative to a reference mesh. Since the distortion introduced by compressing the geometric data affects the distortion introduced by compressing the attribute data, the choice between direct or relative encoding of the geometric data affects the overall mesh compression efficiency. Therefore, the choice between direct or relative encoding should be performed in a way that improves the overall encoding performance. Summary of the invention

[0004] Aspects disclosed in the present disclosure describe a method for encoding mesh data. The method includes receiving a sequence of frames, each frame including mesh data. For a frame in the sequence, the method also includes: encoding the mesh data of the frame according to a static path of a multipath encoder and according to a motion path of the multipath encoder; calculating a static path cost for encoding according to the static path and a motion path cost for encoding according to the motion path, wherein the cost is calculated by optimizing a rate-distortion cost function; and then selecting a bitstream generated by encoding according to the motion path or a bitstream generated by encoding according to the static path based on the motion path cost and the static path cost.

[0005] Aspects disclosed in the present disclosure describe a device for encoding mesh data. The device includes at least one processor and a memory storing instructions. The instructions, when executed by the at least one processor, cause the device to receive a sequence of frames, each frame including mesh data. For a frame in the sequence, the instructions also cause the device to encode the mesh data of the frame according to a static path of a multipath encoder and a motion path according to the multipath encoder; calculate a static path cost for encoding according to the static path and a motion path cost for encoding according to the motion path, wherein the cost is calculated by optimizing a rate-distortion cost function; and then select a bitstream generated by encoding according to the motion path or a bitstream generated by encoding according to the static path based on the motion path cost and the static path cost.

[0006] Aspects disclosed in the present disclosure describe a non-transitory computer-readable medium including instructions executable by at least one processor to perform a method for encoding mesh data. The method includes receiving a sequence of frames, each frame including mesh data. For a frame in the sequence, the method also includes: encoding the mesh data of the frame according to a static path of a multipath encoder and according to a motion path of the multipath encoder; calculating a static path cost for encoding according to the static path and a motion path cost for encoding according to the motion path, wherein the cost is calculated by optimizing a rate-distortion cost function; and then selecting a bitstream generated by encoding according to the motion path or a bitstream generated by encoding according to the static path based on the motion path cost and the static path cost.

[0007] This summary is provided to introduce a selection of concepts in a simplified form, which are further described in the following detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to limitations that address any or all disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a functional block diagram of an example system for dynamic grid coding according to an aspect of the present disclosure.

[0009] Figure 2 is a functional block diagram of an example system for dynamic grid decoding according to an aspect of the present disclosure.

[0010] Figure 3 is a functional block diagram of an example base grid encoder according to one aspect of the present disclosure.

[0011] Figure 4 is a functional block diagram of an example base grid decoder according to one aspect of the present disclosure.

[0012] Figure 5 is a functional block diagram of an exemplary multipath encoder according to an aspect of the present disclosure.

[0013] Figure 6 is a diagram of an example group of pictures structure according to an aspect of the present disclosure.

[0014] Figure 7 is a diagram of an example method for determining a static frame in a group of pictures according to an aspect of the present disclosure.

[0015] Figure 8 is a flow chart of an example method for multipath encoding. DETAILED DESCRIPTION

[0016] According to MPEG V-Mesh CfP, the solution described by Mammou et al. ("Mammou") was selected as the basis for the MPEG V-Mesh test model. See K. Mammou, J. Kim, A. Tourapis and D. Podborski, m59281-[V-CG] Apple's Dynamic Mesh Coding CfP Response, Apple Inc, 2022. Figures 1 to 4 The proposed dynamic mesh coding by Mammou is described in detail below. As further described herein, the proposed dynamic mesh coding suggests first encoding the geometric data of the mesh: i) directly and ii) relative to a reference mesh, and then selecting the encoding method (direct encoding or relative encoding) that results in the least geometric distortion. However, the method does not consider how each encoding method affects the distortion introduced by the encoding of non-geometric data associated with the mesh (e.g., texture data) or the bit rate of the bitstream of the encoded data.

[0017] Apparatus and methods for encoding a sequence of frames containing mesh data are disclosed. Various aspects of a multipath encoder are described herein, including encoding mesh data according to a static path and according to a motion path. A bitstream generated by a coding path that generates a more efficient compression is selected, wherein the efficiency is measured by a rate-distortion cost function. Since the rate-distortion cost function includes a geometric term and a texture term, the overall impact of encoding according to a static path and encoding according to a motion path is taken into account, enabling a more efficient selection of the coding path. Additionally, disclosed herein is an adaptation of a group of pictures (GOP) structure of a sequence of frames based on the corresponding selected coding path.

[0018] Typically, a mesh is a representation of a surface topology, including vertices associated with three-dimensional (3D) positions on the surface; the vertices are connected by edges to form a planar surface (such as a triangle) that approximates the surface. Other information can be associated with each vertex of the mesh, i.e., vertex attributes (e.g., normal vectors and color values). In addition to its topology, the surface can also be represented by various attributes, such as texture. Typically, the texture of a surface is described by a two-dimensional (2D) image, i.e., a texture map. In order to associate the surface of a mesh with corresponding texture data, the 3D surface of the mesh is mapped into a 2D space (e.g., UV parameter space). Similarly, the surface of a mesh can be associated with other data types provided by other attribute maps, which characterize other physical properties of the surface (e.g., surface reflectivity and transparency), which may be required for a realistic reproduction of the surface. Therefore, the surface representation of mesh data includes topological data and attribute data - the topology of the surface is represented by the mesh M (including geometry and connectivity information, and possible vertex attributes), and the attributes of the surface are represented by the attribute map A (including the attribute map and corresponding graph information). Aspects described herein with respect to texture data (represented by texture maps) may be applied to other types of data (typically represented by property maps).

[0019] Figure 1 1 is a functional block diagram of an example system 100 for dynamic mesh encoding according to an aspect of the present disclosure. The system 100 illustrates encoding of a sequence of frames F(i), wherein data associated with frame i includes a mesh M(i) 105 and a corresponding attribute map A(i) 110. The system 100 includes a mesh decomposer 120 (e.g., part of a pre-processing unit) and an encoder 130. The mesh decomposer 120 is configured to decompose a received mesh M(i) 105 into a base mesh m(i) and a corresponding displacement vector d(i). The generated base mesh m(i) and displacement vector d(i) are fed into the encoder 130 together with the corresponding attribute map A(i) 110. The encoder 130 encodes the obtained data m(i), d(i), and A(i), and thereby generates corresponding bitstreams, including a base mesh bitstream 170, a mesh displacement bitstream 175, and an attribute map bitstream 180. The operation of the mesh decomposer 120 and the operation of the encoder 130 are further described below.

[0020] The decomposer 120 is configured to decompose the mesh M(i) 105 into a base mesh m(i) and corresponding displacement vectors d(i). To generate the base mesh m(i), the decomposer 120 decimates the mesh by subsampling the vertices of the mesh M(i). A subdivided mesh is then generated by subdividing the base mesh m(i), i.e., each surface of the base mesh is subdivided into a plurality of sub-surfaces, introducing additional new vertices. Any subdivision scheme may be optionally applied iteratively. For example, each triangle of the base mesh surface may be divided into four sub-triangles by introducing three new vertices in the middle of the triangle edge and by connecting these three vertices. Next, the decomposer 120 determines the displacement vectors d(i) of the corresponding vertices of the subdivided base mesh, so that when applied to those vertices, a deformed mesh that is spatially suitable for the received mesh M(i) 105 is generated. Decomposing the received mesh M(i) in this manner allows encoding the base mesh m(i) and its corresponding displacement vectors d(i) instead of directly encoding the mesh M(i) 105, which improves compression efficiency. This is because the base mesh has fewer vertices relative to the mesh M(i) 105 and can therefore be encoded by a relatively small number of bits. Furthermore, the displacement vectors can be efficiently encoded using, for example, a wavelet transform enabled by the subdivision structure. In turn, the subdivision structure used does not need to be explicitly encoded as it can be determined by the decoder. For example, the decoder can subdivide the decoded base mesh based on the subdivision scheme type and a subdivision iteration count that can be signaled in the bitstream.

[0021] like Figure 1 As shown, the encoder 130 includes a base grid encoder 135, a base grid decoder 140, a grid displacement encoder 145, a grid displacement decoder 150, a grid reconstructor 155 and an attribute graph encoder 160. The base grid encoder 135 is configured to encode the base grid m(i) into an encoded base grid cm(i), and thereby generate a base grid bit stream 170. The base grid decoder 140 is configured to reconstruct (decode) the base grid from the encoded base grid cm(i), thereby generating a reconstructed quantized base grid m'(i) and a reconstructed base grid m"(i). Refer to Figure 3 and Figure 4The base grid encoder 135 and the decoder 140 are further described. The grid displacement encoder 145 receives as input a base grid m(i) and a reconstructed quantized base grid m'(i), and based on the base grid and the reconstructed quantized base grid, the grid displacement encoder is configured to encode the received displacement vector d(i) into an encoded displacement vector cd(i), and thereby generate a grid displacement bitstream 175. The grid displacement decoder 150 is configured to reconstruct (decode) the displacement vector cd(i) from the encoded displacement vector, thereby generating a reconstructed displacement vector d″(i). Based on the reconstructed base grid m”(i) and the reconstructed displacement vector d″(i), the grid reconstructor 155 is configured to reconstruct (decode) the grid into a reconstructed grid DM(i). Based on the grid M(i) and the reconstructed grid DM(i), the attribute map encoder 160 is configured to encode the attribute map into an encoded attribute map A(i), and thereby generate an attribute map bitstream 180.

[0022] The grid displacement encoder 145 encodes the displacement vectors d(i) associated with the individual vertices of the subdivided base mesh as described above. To this end, the displacement vectors are first updated based on the reconstructed quantized base mesh m′(i). Then, a wavelet transform is applied to the updated displacement vectors d′(i) according to the subdivision structure into which the base mesh has been subdivided. The wavelet coefficients are then quantized, packed into a 2D image, and compressed by a video encoder. The grid displacement decoder 150 typically reverses the operation of the grid displacement encoder 145. Therefore, the grid displacement decoder 150 employs a video decoder to decode the packed 2D image compressed by the video encoder of the grid displacement encoder 145 (if the video encoder is lossy). The grid displacement decoder 150 then unpacks the 2D image to obtain the quantized wavelet coefficients, and applies inverse quantization followed by an inverse wavelet transform to generate a reconstructed displacement vector d″(i).

[0023] Note that the video encoder is responsible for the tasks of compressing the packed wavelet coefficients (via the trellis displacement encoder 145) and compressing the attribute map (via the attribute map encoder 160). Any video coding method (lossless or lossy) may be employed for these tasks, depending on the requirements of the particular application.

[0024] Figure 2 is a functional block diagram of an example system 200 for dynamic grid decoding according to an aspect of the present disclosure. System 200 is configured to generally reverse the operation of system 100, including a decoder 230 and a grid reconstructor 260. Decoder 230 includes a base grid decoder 235, a grid displacement decoder 240, and a property map decoder 250. As shown in FIG. Figure 4As further described, the base grid decoder 235 decodes the reconstructed base grid m" (i) from the base grid bit stream 210, 170. The grid displacement decoder 240 decodes the reconstructed displacement vector d" (i) from the grid displacement bit stream 215, 175, and performs reference Figure 1 The steps described by the grid displacement decoder 150 are as follows. The attribute map decoder 250 decodes the attribute map from the attribute map bitstream 220, 180, and reverses the operation of the attribute map encoder 160 to generate a reconstructed attribute map DA(i) 275. The output of the decoder 230 of the reconstructed base grid m″(i) and the reconstructed displacement vector d”(i) is used by the grid reconstructor 260 to reconstruct the decoded grid DM(i) 270.

[0025] Figure 3 3 is a functional block diagram of an example base grid encoder 300 according to an aspect of the present disclosure. The base grid encoder 300 includes a quantizer 320, a static grid encoder 340, a motion encoder 350, and a selector 360. Figure 1 As described above with respect to the base grid encoder 135, the base grid encoder 300 is configured to encode the base grid m(i) into a base grid bitstream 380. To this end, two encoders 340, 350 may be employed. Thus, after quantization 320, the static grid encoder 340 independently encodes the quantized base grid qm(i) according to any static grid encoding method. Additionally, after quantization 320, the motion encoder 350 encodes the quantized base grid qm(i) relative to a reference reconstructed quantized base grid m'(j) (e.g., associated with a previous base grid m(i-1) of a frame sequence). That is, the motion encoder 350 encodes a motion field f(i) that describes the motion that the vertices of m(j) must undergo in order to reach the corresponding positions of the corresponding vertices of m(i).

[0026] Therefore, it is assumed that m(i) and m(j) share the same number of vertices and the same vertex connectivity, and only the positions of the corresponding vertices in m(i) and m(j) change over time. On the one hand, to ensure that m(i) and m(j) have the same corresponding vertices, the encoder 300 can track the transformation applied to m(j) to obtain m'(j) and apply it to m(i). In this case, the motion encoder 350 can be configured to first calculate the motion field f(i) and then encode the calculated motion field into the base grid bitstream 380. The motion field f(i) contains the corresponding motion vectors of the corresponding vertices in the quantized base grid qm(i) and the reference reconstructed quantized m'(j), as follows:

[0027] f(i)=v1(i)-v2(j), (1)

[0028] Where v1(i) is a vector containing the vertex positions of mesh qm(i), and v(j) is a vector containing the corresponding vertex positions of mesh m′(j). In one aspect, the motion encoder 350 can further adjust the motion vector (e.g., based on neighboring motion vectors) and then encode the adjusted motion vector, for example, using an entropy encoder.

[0029] Whether to use the output of the static grid encoder 340 or the output of the motion encoder 350 can be selected by the selector 360. As described above, Mammou proposed using the D2 feature of the MPEG grid metric to select the bitstream of the encoder (static grid encoder 340 or motion encoder 350) that causes the smallest geometric distortion. That is, if the geometric distortion contributed by the motion encoder 350 is lower than the geometric distortion contributed by the static grid encoder 340 (or lower than a predetermined threshold), the bitstream generated by the motion encoder 350 will be used as the base grid bitstream 380, otherwise the bitstream generated by the static grid encoder 340 will be used as the base grid bitstream 380. However, the D2 feature of the MPEG grid metric used only reflects the geometric distortion and, for example, does not take into account the global rate.

[0030] Figure 4 4 is a functional block diagram of an example base grid decoder 400 according to an aspect of the present disclosure. The base grid decoder 400 generally reverses the operation of the base grid encoder 300. The base grid decoder 400 includes a static grid decoder 440, a motion decoder 450, and an inverse quantizer 460. As described above with reference to Figure 2As described in the base grid decoder 235 of FIG. 4 , the base grid decoder 400 is configured to decode the reconstructed base grid m″(i) from the base grid bitstream 420, 380. To this end, the base grid decoder 400 directs the input base grid stream 420 (representing the encoded base grid cm(i)) to the static grid decoder 440 or the motion decoder 450. This direction can be generated based on signaling in the bitstream 420 indicating whether the encoded base grid cm(i) is encoded by the static grid encoder 340 or by the motion encoder 350. If the bitstream 420 is directed to the static grid Decoder 440, the latter decodes the base grid from the bitstream 420 to obtain a reconstructed quantized base grid m'(i). Otherwise, if the bitstream 420 is directed to a motion decoder 450, the motion decoder decodes the motion field f'(i) from the bitstream 420 and adds the reconstructed (decoded) motion field to the reference reconstructed quantized base grid m'(j), thereby generating a reconstructed quantized base grid m'(i). The obtained m'(i) is then provided to the inverse quantizer 460, which thereby generates a reconstructed base grid m"(i). As described above, the base grid decoder 400 is also used in the encoder 130, wherein the base grid decoder 140 provides the reconstructed quantized base grid m'(i) and the reconstructed base grid m"(i) to the grid displacement encoder 145 and the grid reconstructor 155, respectively.

[0031] As described above, the base mesh encoder 300 may choose 360 ​​to directly encode the base mesh m(i) of the received frame i (using the static mesh encoder 340), or may choose to encode the received base mesh m(i) relative to a reference base mesh m(j) (using the motion encoder 350). In the latter, what is encoded is the motion field f(i) associated with the corresponding vertices from m(i) and m(j). As described above, the D2 feature of the MPEG mesh metric is used to determine the geometric distortion, based on which the choice of using the static mesh encoder 340 or the motion encoder 350 may not result in a better choice because other sources of cost introduced in the encoder 130 are not considered in this approach. A preferred approach is to consider the total rate-distortion cost introduced by the encoder 130 when choosing between the output of the static mesh encoder 340 and the output of the motion path 350. Therefore, in accordance with the aspects disclosed herein, a rate-distortion optimization that takes into account topological and photometric distortions as well as bitrate levels is performed. As described in reference Figure 5 As further described, the rate-distortion optimization employed may result in the selection of an encoder (340 or 350) that will provide a more efficient encoding corresponding to an optimal rate-distortion cost.

[0032] Figure 55 is a functional block diagram of an example multipath encoder 500 according to an aspect of the present disclosure. The multipath encoder 500 includes a static path (SP) encoder 520 and a motion path (MP) encoder 525, each of which is configured to encode mesh data of an input frame F(i) 510, including a mesh M(i) and a corresponding attribute map A(i). The SP encoder 520 may include the system 100 ( Figure 1 ) of the encoder 130 and the system 200 ( Figure 2 ) components of decoder 230 of system 100, wherein static grid encoder 340 and static grid decoder 440 (referred to herein as static path) are used. MP encoder 525 may include system 100 ( Figure 1 ) of the encoder 130 and the system 200 ( Figure 2 ) components of the decoder 230, wherein a motion encoder 350 and a motion decoder 450 (referred to herein as a motion path) are used. Figure 5 As shown, the SP encoder 520 outputs the DF SP (i) and the corresponding SP bitstream and its bit rate (i.e., SP bitrate). Similarly, the MP encoder 525 outputs the decoded frame represented by DF MP (i) represents the decoded frame, and the corresponding MP bit stream and its bit rate (i.e., MP bit rate).

[0033] The multipath encoder 500 further comprises an SP distortion metric calculator 530 which is based on the frame F(i) 510 and its decoded DF SP (i) version thereof to calculate the various distortions introduced by the SP encoder 520. Likewise, the multipath encoder 500 includes an MP distortion metric calculator 535, which is based on the frame F(i) 510 and its decoded version DF MP (i) Calculate various distortions introduced by the MP encoder 525. The SP cost calculator 540 of the multipath encoder 500 is configured to calculate the rate-distortion cost of employing the SP encoder 520 based on the SP distortion (provided by the SP distortion metric calculator 530) and based on the SP bit rate of the SP bitstream. Similarly, the MP cost calculator 545 is configured to calculate the rate-distortion cost of employing the MP encoder 525 based on the MP distortion (provided by the MP distortion metric calculator 535) and based on the MP bit rate of the MP bitstream. Based on those calculated rate-distortion costs 540, 545, the selector 550 is configured to select to output the SP bitstream as the output bitstream 560 of the multipath encoder 500, or to output the MP bitstream as the output bitstream 560 of the multipath encoder 500, as described above. Figure 5 Note that the frame F(i) for which the SP bitstream is selected is referred to herein as a static frame encoded in a static mode, while the frame F(i) for which the MP bitstream is selected is referred to herein as a motion frame encoded in a motion mode.

[0034] In order to select between the static path 520 and the motion path 525 when encoding an input frame F(i)∈M(i), A(i), the encoding cost J associated with each alternative path is calculated, and through an optimization process, the path (static path 520 or motion path 525) that results in a lower optimal cost (or has an optimal cost below a predetermined threshold) is selected. The encoding cost used can be the following rate-distortion cost function:

[0035] J=D+λR, ((2)

[0036] where D is the distortion measure, R is the bit rate value, and λ is the Lagrange multiplier. The Lagrange multiplier λ allows to set a trade-off between the quality of the encoded data (inversely proportional to the distortion D) and the bit rate R. For example, λ can be the encoder (e.g., λ∝e (((QP-3) / 6))) ) is a function of the quantization parameter (QP) used by the quantizer of the quantizer. In one aspect, multiple Lagrange multipliers can be used to balance the distortion introduced by the quantizers of various encoders (e.g., the base grid encoder 135, the grid displacement encoder 145, and / or the property map encoder 160) and the corresponding bit rate.

[0037] The distortion metric D can be calculated as:

[0038] D=D tex +αD geo , (3)

[0039] Where D tex Indicates that the texture is distorted and D geo represents geometric distortion. The parameter α can be set to balance between texture distortion and geometric distortion. For example, texture distortion can be represented as:

[0040] D tex =(β Y D Y +β U D U +β V D V ) / (β Y +β U +β V ), (4)

[0041] Where D Y , D U and D V Represents the corresponding distortion value contributed by the encoding of the luminance Y and chrominance components U and V of the texture image. The corresponding weighting value β can be used Y , β U and β VTo balance different sources of distortion. For example, to balance the brightness distortion value D Y and chromaticity distortion value D U and D V , the weight value can be set to β Y =6 and β U =β V = 1. The distortion D is usually measured by a mean square error (MSE) metric (i.e., the average of mean square error values ​​derived based on a distance metric) that measures the difference between samples from the original data (e.g., a texture map) and corresponding samples from the reconstructed data (e.g., a reconstructed texture map).

[0042] As disclosed herein, the rate-distortion optimization process is implemented by minimizing a cost function J over various encoding modes (i.e., modes). The optimal cost J can be calculated with respect to the static path 540 and the motion path 545. Based on these optimal costs, it can be determined which bitstream (SP bitstream or MP bitstream) to select as the output bitstream 560 (see Figure 5 ). In one aspect, the cost function includes texture and geometric distortions and various bit rates associated with the respective bitstreams 170, 175, 180 generated by the encoder 130. The optimization of such a cost function can be expressed as follows:

[0043]

[0044] where λ is the Lagrange multiplier. The term D tex is the texture distortion (eg, associated with the reconstructed attribute map DA(i)), and the term D geo is the geometric distortion (eg, associated with the reconstructed mesh DM(i)). The term R tex is the bit rate of the bitstream representing the texture data (e.g., 180), and the term R geo is the bit rate of the bitstream representing the geometry data (e.g., 170, 175). The encoding mode over which the cost function J is optimized may include any set of parameters that may control the operation of the encoder 130 and its components 135, 145, 160. Note that when the cost function is optimized with respect to a static path, the term R geo include:

[0045]

[0046] When optimizing the cost function with respect to the motion path, the term R geo include:

[0047]

[0048] Where R 网格 and R 运动are the bit rates of the base grid bitstream 170 when it is generated by a static path and when it is generated by a moving path, respectively. And, where R 位移 is the bit rate of the grid displacement bitstream 175 .

[0049] In one aspect, other distortion metrics D can be used to optimize the rate-distortion cost function J. Two MPEG distortion metrics can be used to obtain distortion values. Dynamic Grid Coding (MPEG, 2021-11-08) See MDS21000_WG07_N00231, CfP. These metrics can be extended by applying them to data from several adjacent frames. The two metrics are described below.

[0050] In the first metric for computing point clouds, i.e., the point cloud based mesh distortion (PCMD) metric, the mesh M(i) and reconstructed mesh DM(i) corresponding texture maps A(i) and DA(i) are used to sample the mesh and reconstructed mesh geometry into colored point clouds. Then, the colored point cloud geometry distortion measure is calculated: MSE D1 and MSE D2 , and the texture distortion measure MSE Y , MSE U and MSE V . Note that if Figure 1 As demonstrated by the architecture of encoder 130 of FIG. 1 , texture distortion is often influenced by geometric distortion. These distortion measures can be combined into a single metric, MSE Y The same as follows:

[0051] PCMD=(aMSE D1 +b MSE D2 +c MSE Y +d MSE U +e MSE V ) / c (8)

[0052] The coefficients a, b, c, d and e can be determined as follows.

[0053] In the second metric, the Image Based Sampling Distortion (IBSD) metric, the mesh M(i) and the reconstructed mesh DM(i) are rendered from several different viewpoints using their corresponding property maps. The rendered views are then used to compute the geometric distortion measure MSE geo and texture distortion measure MSE Y , MSE U and MSE V As mentioned above, texture distortion is often influenced by geometric distortion. These distortion measures can be combined into a single metric, similar to MSE Y The same as follows:

[0054] IBSD=(a′MSE GEO +b′MSE Y +C′MSE U +d′MSE V ) / b′, (9)

[0055] The coefficients a', b', c' and d' can be determined as follows.

[0056] To determine the coefficients a, b, c, d and e of equation (8) and the coefficients a', b', c' and d' of equation (9), a learning process can be utilized, for example, by using a perceptual mean opinion score (MOS) collected from a group of people. For example, each person in a group of people can be asked to evaluate a total of N videos, in which M animation models are rendered using several distortion types and multiple distortion levels of each distortion type. Then, the coefficients a, b, c, d and e can be estimated based on the collected evaluations and the corresponding calculated PCMD metrics. Similarly, the coefficients a', b', c' and d' can be estimated based on the collected evaluations and the corresponding IBSD metrics. On the one hand, the estimation of the coefficients is performed using leave-one-out cross-validation and learning. Note that any other learning method can be used to estimate the values ​​of the coefficients.

[0057] Additional distortion measures can be added to the PCMD and IBSD metrics, such as distortion measures that detect specific defects of interest (e.g., cracks in the reconstructed mesh surface). In addition, distortion measures can be combined linearly or nonlinearly by any function to generate a PCMD metric or an IBSM metric. Note that both the PCMD metric and the IBSM metric are scaled by MSE. Y Zoom.

[0058] As disclosed herein, the optimization process according to equation (5) can be simplified by first optimizing the cost associated with encoding geometric data and then continuing to optimize the total cost. To this end, the rate-distortion cost J can be expressed as follows:

[0059]

[0060] where λ and λ′ are Lagrange multipliers. The term D geo is the geometric mesh distortion (eg, associated with the reconstructed mesh DM(i)), and the term D tex is the mesh texture distortion (eg, associated with the reconstructed attribute map) DA(i). The term R tex is the texture bitrate of the attribute map bitstream 180. As mentioned above, the term R geo When optimizing the cost of static paths is equal to the cost of optimizing the motion path. are equal (see equations (6) and (7). In the first stage of the optimization process, the first term in equation (10) is first optimized on the geo mode, each term including a set of parameters applicable to the encoding of geometric data (e.g., belonging to the base grid encoding 135 and the grid displacement encoding 145). In the second stage of the optimization process, the second term is optimized on the tex mode, each tex mode including a set of parameters applicable to the encoding of image data (e.g., belonging to the attribute map encoding 160). Note that in the second term, Represents the optimal geo-mode bitrate determined by the first optimization in the first stage.

[0061] Therefore, in the first stage, the geo mode of the first term of the cost of optimizing the static path (in equation (10)) is determined to generate the optimal geo mode of the static path (i.e., sp-opt-geo-mode). Similarly, the geo mode of the first term of the cost of optimizing the dynamic path (in equation (10)) is determined to generate the optimal geo mode of the moving path (i.e., mp-opt-geo-mode). Next, the corresponding bit rate values ​​at the optimal geo mode of the static path and the optimal geo mode of the moving path are calculated:

[0062]

[0063] as well as

[0064]

[0065] Then, in the second stage, a tex pattern that optimizes J is determined for each of the static path and the moving path. Therefore, the SP optimal cost associated with the static path can be determined based on the second term (in equation (10)), where And, the MP optimal cost associated with the motion path can be determined based on the second term (in equation (10)), where Therefore, the bitstream generated by the encoding path having the lowest optimal cost (SP optimal cost and MP optimal cost) can be selected as the output bitstream 560 of the encoder 500. Alternatively, the bitstream generated by the encoding path having the optimal cost (SP optimal cost or MP optimal cost) below a predetermined threshold can be selected as the output bitstream 560 of the encoder 500.

[0066] The coding modes for which the rate-distortion cost J is optimized may each include any combination of parameters that control the operation of the SP encoder 520 and the MP encoder 525. For example, as described herein, the rate-distortion cost J may be optimized with respect to a coding mode defined by parameters such as QP, where the QP is set to control a quantizer in the base grid encoder 135, the grid displacement encoder 145, and / or the property map encoder 160. Furthermore, the rate-distortion cost J may be optimized over coding modes defined by parameters associated with local QP adaptation, target resolution adaptation, and / or slice type adaptation.

[0067] In one aspect, the slice type associated with the encoding 145, 160 of the image data of a frame can be coupled to the selected encoding path of the frame. Thus, since the image data of a frame selected to be encoded by the static path may have a different layout than the image data of the previous frame, the slice type can be adjusted by the selected encoding path. For example, if the output of the static path is selected for the frame, the slice type is set to intra. Otherwise, if the output of the motion path is selected for the frame, the slice type is set to inter. Using this heuristic step, the search space of the rate-distortion optimization algorithm can be reduced.

[0068] On the other hand, the GOP structure of the GOP sequence can be adapted based on the selected encoding path. For example, since the frame sequence F(i) for which the motion path has been selected may be stable in time, the GOP structure of such a sequence can be dynamically adapted. Techniques for GOP structure adaptation are further described below.

[0069] Figure 6 is a diagram of an example GOP structure 600 according to an aspect of the present disclosure. One approach to balancing rate-distortion costs across a GOP is to use a hierarchical GOP structure. Figure 6 , the hierarchy of the GOP structure 600 is illustrated by the temporal depth of the frames and the arrows indicating inter-coding dependencies between the frames and other reference frames. As shown, the first frame 610 of the GOP is an intra-coded frame and, therefore, its coding does not depend on any other reference frames. The following frames are inter-coded frames that depend on other reference frames for coding. Thus, as indicated by the arrows, frame 615 depends on reference frames 610 and 620, frame 620 depends on reference frames 610 and 630, frame 625 depends on reference frames 620 and 630, and so on. A first cycle of the GOP structure (comprising frames 615, 620, 625, and 630) is repeated by a second cycle of the same GOP structure (comprising frames 635, 640, 645, and 650), and similarly by additional cycles until a new intra-frame is encoded, at which stage a new GOP cycle begins.

[0070] Typically, frames of each GOP are assigned a QP with a value related to the importance of the frame in the GOP. The importance of a frame is associated with its temporal depth and / or the number of times the frame is directly or indirectly referenced by other frames. Table 1 shows the frames in the first cycle of the intra-frame 610 and the GOP structure 615, 620, 625, 630, indicating the picture order count (POC), slice type, QP offset, temporal depth (represented by the temporal identification number Tid) and associated reference frames of these frames.

[0071] Table 1: GOP structure.

[0072] POC Slice Type QP offset Tid Reference Frame 0 Intra 0 0 4 Interframe 1 1 -4, 2 Interframe 2 2 -2,2 1 Interframe 4 3 -1,1 3 Interframe 4 3 -1,1

[0073] The QP offset is an offset added to the target QP normally assigned to the entire GOP sequence. Thus, if the target QP is 32 and the QP offset of a frame is 2, then the frame is encoded using a QP of 32+2=34. Note that the offset can be adjusted depending on the content. Tid indicates the temporal depth of the frame. The higher the Tid, the lower the impact of the frame on other frames. Reference frames are indicated by POC deltas, where when the POC delta is bracketed, the indicated reference frame is used only when available. For example, frame 610 has a POC of 0, a slice type of intra, a QP of 32, a temporal depth of 0, and as an intra frame, it does not depend on a reference frame. Frame 630 has a POC of 4, a slice type of inter, a QP of 33, a temporal depth of 1, and it depends on the reference frame 610 (i.e., its POC minus 4). In the second loop, the corresponding frame 650 depends on frame 630 (i.e., its POC minus 4) and frame 610 (i.e., its POC minus 8). The GOP structure is repeated as shown in Table 2, which demonstrates several cycles of the GOP structure. When a new intra frame is encoded, a new GOP sequence begins with a new cycle of the GOP structure. As described herein, the GOP sequence and GOP structure can have arbitrary sizes, different numbers of temporal levels, and different numbers of applied QP offsets.

[0074] Table 2: Repetition cycle of GOP structure.

[0075]

[0076] In one aspect, the GOP structure can dynamically adapt to the selection of static path or motion path encoding 550. For example, in low-delay encoding, the following GOP structure adaptation strategy can be applied. The first frame of the GOP sequence is set as an intra-frame and encoded using the static path encoder 520. Then, the subsequent frames of the GOP sequence are processed starting from the first loop of the GOP structure. Therefore, for each of the following frames, the following steps can be performed: (1) using a QP offset of 0 and an intra-frame slice type, encoding according to the static path 520 is performed; (2) using the QP assigned to the frame according to the position of the frame in the GOP structure and the inter-slice type, encoding of the motion path 525 is performed; (3) selecting 550 the bitstream generated by the more efficient encoding path (520 or 525) to form an output bitstream 560; (4) if the static path 520 generates the selected bitstream, restart the loop of the GOP structure; and (5) proceed to step 1) to process the next frame. Reference Tables 3 and 4 further demonstrate the above-mentioned GOP structure adaptation strategy.

[0077] Table 3: GOP structure

[0078]

[0079]

[0080] Table 4: Adaptive GOP structure

[0081]

[0082] Table 3 shows a GOP structure indicating the POC, slice type, QP offset, Tid, and associated reference frames of each frame. Using this GOP structure, Table 4 shows the process of adapting the GOP structure of the GOP sequence based on the above-mentioned GOP structure adaptation strategy. Therefore, the first frame (POC=0) is set as an intra-frame and encoded using the static path 520. Then, for frames with POC=1, motion path encoding (using QP=QP+3 and inter-slice type) and static path encoding (using QP=QP+0 and intra-slice type) are performed, where the bitstream generated by the motion path is selected based on the corresponding RD cost. Next, for frames with POC=2, motion path encoding (using QP=QP+2 and inter-slice type) and static path encoding (using QP=QP+0 and intra-slice type) are performed, where the bitstream generated by the motion path is selected based on the corresponding RD cost. Next, for the frame with POC=3, motion path coding (using QP=QP+3 and inter-slice type) and static path coding (using QP=QP+0 and intra-slice type) are performed, where the generated bitstream of the static path is selected based on the corresponding RD cost. As demonstrated in Table 4, when the next frame with POC=4 is processed, the selection of the bitstream generated by the static path with POC=3 causes the restart of a new cycle of the GOP structure.

[0083] In random access mode, the selection of whether to encode each frame of a GOP sequence using a static path 520 or a motion path 525 can be done in two stages. In the first stage, the frames in the GOP sequence for which the bitstream generated by the static path is selected are determined. These frames are referred to herein as static frames. In the second stage, the remaining frames are encoded using the motion path. Figure 7 Techniques for determining static frames in a sequence are described.

[0084] Figure 7 7 is a diagram of an example method 700 for determining a static frame in a GOP according to an aspect of the present disclosure. Specifically, the method 700 determines the next static frame relative to the static frame S. To this end, in a first iteration 710, starting from the static frame S=0, frames S+k*G are estimated up to M frames, where M is the maximum (allowed) number of consecutive motion frames and G is the GOP size (e.g., for k=1, 2...M / 4, frames 0+k*4 are estimated). Therefore, for each of these frames, a test is performed to select 550 whether the bitstream generated by the static path 520 or the motion path 525 is selected. As shown in FIG. Figure 7As shown in , based on the testing of frames 4, 8, and 12, frame 12 is the first frame selected for encoding in static mode. In the second iteration 720, the same process is repeated for frames between 8 and 12, evaluating frames every G / 2 frames. In this iteration 720, frame 10 is the first frame selected for encoding in static mode. Then, in the third iteration 730, the same process is repeated for frames between frames 8 and 10, evaluating frames every G / 4. In the iteration 730 of the present invention, encoding in motion mode is selected for frame 9, and therefore frame 10 is determined to be the next static frame following static frame 0. Therefore, the frames between frame 0 and frame 10 are encoded using the motion path (these frames are not needed to test a better encoding path). The method 700 is repeated starting from frame 10 (i.e., S=10).

[0085] In one aspect, when determining the first static frame, a full test need not be performed (selecting 550 a bitstream generated by the static path 520 or by the motion path 525). Instead, an approximate heuristic step may be used. Typical heuristic steps may be based on the energy of frame differences or on the magnitude of motion vectors.

[0086] Figure 8 8 is a flow chart of an example method 800 for multipath encoding. The method 800 begins in step 810 by receiving a sequence of frames containing mesh data. Then, for the frames in the sequence, steps 820 to 840 may be performed. In step 820, the mesh data of the frames of the sequence are encoded according to the static path of the multipath encoder and according to the motion path of the multipath encoder. Next, in step 830, the static path cost of encoding according to the static path and the motion path cost of encoding according to the motion path are calculated. The cost may be calculated by optimizing a rate-distortion cost function. Then, in step 840, a selection is made between a bitstream generated by encoding according to the motion path and a bitstream generated by encoding according to the static path based on the motion path cost and the static path cost. On the one hand, if the motion path cost is lower than the static path cost, the bitstream generated by encoding according to the motion path may be selected. On the other hand, if the motion path cost is lower than a predetermined threshold, the bitstream generated by encoding according to the motion path may be selected.

[0087] As described above, the rate-distortion cost function includes a geometric term and a texture term (see Equation 5). In one aspect, the optimization of the rate-distortion cost function can be performed in two stages (see Equation 10). In the first stage, the geometric term of the rate-distortion cost function is optimized, resulting in an optimal mode (of the coding mode) and a corresponding bit rate. Then, in the second stage, the texture term of the rate-distortion cost function is optimized, wherein the texture term includes the corresponding bit rate provided by the first stage. The optimization of the rate-distortion cost function is performed on the coding modes, each coding mode including parameters that control the encoding of the grid data of the frame. In one aspect, such parameters can be associated with local QP adaptation, target resolution adaptation, slice type adaptation, or a combination thereof.

[0088] As described with reference to Tables 3 and 4, the method 800 can adapt the GOP structure of the received frame sequence 810 based on the selection of the bitstream in step 840. To this end, for each frame in the sequence, the following steps can be performed: (1) the frame is encoded 820 according to the static path, the slice type is set to intra-frame and the quantization parameter is set to zero, (2) the frame is encoded 820 according to the motion path, the slice type is set to inter-frame and the quantization parameter is set according to the frame position in the GOP structure, and then (3) if the bitstream generated by encoding according to the static path 840 is selected, this selection causes the method 800 to restart the cycle of the GOP structure in the frame sequence (as shown in Table 4).

[0089] The method 800 can also determine the static frames and motion frames in the GOP, such as reference Figure 7 For example, method 800 may perform steps 820, 830, and 840 on a subset of frames in a sequence that follows a current frame for which a bitstream generated by encoding according to a static path is selected (e.g., Figure 7 Next, a first frame in the subset is determined, and a bit stream generated by encoding according to a static path is selected for the first frame. Then, grid data of frames between the current frame and the first frame can be encoded according to the motion path of the multipath encoder. The subset of frames can be a series of frames located at the end of a cycle of a GOP structure of a frame sequence (e.g., Figure 7 4, 8, and 12 in the series), where the series ends with selecting a frame of the bitstream generated by encoding according to the static path (e.g., frame 12). In one aspect, the first frame can be recursively determined among frames between the frame at the beginning of the last cycle of the series (e.g., frame 8) and the candidate frame determined to be the first frame in the previous iteration (e.g., frame 10 at iteration 720).

[0090] The description of the various aspects described herein is intended to provide a general understanding of the structure, function and operation of various aspects. These illustrations are not intended to be used as a complete description of all elements and features of the equipment and system utilizing the structure or method described herein. Many other aspects can be immediately understood by those skilled in the art after reviewing the disclosure. Other aspects can be utilized and derived from the disclosure so that structural and logical replacements and changes can be performed without departing from the scope of the present invention. Therefore, the disclosure and the accompanying drawings should be considered illustrative and not restrictive.

[0091] The description of this aspect is provided to enable the making or use of this aspect. Various modifications to these aspects will be apparent, and the general principles defined herein may be applied to other aspects without departing from the scope of the invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but should conform to the widest possible scope consistent with the principles and novel features defined by the appended claims.

Claims

1. A method for encoding grid data, the method comprising: receiving a sequence of frames, each frame including grid data; as well as For a frame in the sequence: encoding the mesh data of the frame, encoding according to a static path of a multipath encoder and according to a moving path of a multipath encoder, calculating a static path cost of the encoding according to the static path and a moving path cost of the encoding according to the moving path, the costs being calculated by optimizing a rate-distortion cost function, and Based on the motion path cost and the static path cost, a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path is selected. 2 . The method of claim 1 , wherein the rate-distortion cost function comprises a geometry term and a texture term.

3. A method according to claim 1 or 2, wherein the optimization of the rate-distortion cost function is performed over encoding modes, each mode comprising parameters controlling the encoding of the grid data of the frame.

4. The method according to any one of claims 1 to 3, wherein the parameters of a mode in the encoding modes are associated with one of local QP adaptation, target resolution adaptation and slice type adaptation.

5. The method according to any one of claims 1 to 4, wherein the optimization of the rate-distortion cost function comprises: Optimizing the geometric term of the rate-distortion cost function to determine the optimal mode of the coding mode and the corresponding bit rate; as well as A texture term of the rate-distortion cost function is optimized, the texture term comprising the corresponding bit rate.

6. The method according to any one of claims 1 to 5, wherein the selecting comprises: If the motion path cost is lower than the static path cost, the bitstream generated by the encoding according to the motion path is selected.

7. The method according to any one of claims 1 to 5, wherein the selecting comprises: If the motion path cost is below a predetermined threshold, the bitstream generated by the encoding according to the motion path is selected.

8. The method according to any one of claims 1 to 5, further comprising: A GOP structure of the sequence of frames is adapted based on the selection, wherein selection of the bitstream generated by the encoding according to the static path restarts the cycle of the GOP structure in the sequence.

9. The method of any one of claims 1 to 8, wherein the encoding according to the static path comprises setting a slice type to intra and setting a quantization parameter to zero.

10. The method according to any one of claims 1 to 8, wherein the encoding according to the motion path comprises setting a slice type to inter and setting a quantization parameter according to the frame position in the GOP structure.

11. The method according to any one of claims 1 to 7, further comprising: performing said encoding, said calculating and said selecting on a subset of frames in said sequence, said subset following selection of a current frame for which said bitstream generated by said encoding according to said static path is selected; determining a first frame of the bitstream generated by the encoding according to the static path in the subset for selection; as well as The mesh data of frames between the current frame and the first frame are encoded according to the motion path of the multi-path encoder.

12. The method according to any one of claims 1 to 7 or 11, wherein the number of frames between the current frame and the first frame is lower than the maximum number of consecutive motion frames.

13. The method according to any one of claims 1 to 7 or 11, wherein the frame subset comprises: A series of frames at the end of a cycle of the GOP structure of said sequence of frames, said series ending with a frame for which said bitstream generated by said encoding according to said static path was selected.

14. The method of any one of claims 1 to 7, 11 or 13, wherein the determining of the first frame comprises: The first frame is recursively determined among frames between a frame at the beginning of a last cycle of the loop and a candidate frame determined to be the first frame in a previous iteration.

15. A device for encoding grid data, the device comprising: at least one processor; as well as a memory storing instructions that, when executed by the at least one processor, cause the apparatus to: Receive a sequence of frames, each frame including mesh data, and For a frame in the sequence: encoding the mesh data of the frame, encoding according to a static path of a multipath encoder and according to a moving path of a multipath encoder, calculating a static path cost of the encoding according to the static path and a moving path cost of the encoding according to the moving path, the costs being calculated by optimizing a rate-distortion cost function, and Based on the motion path cost and the static path cost, a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path is selected.

16. The apparatus of claim 15, wherein the rate-distortion cost function comprises a geometry term and a texture term.

17. Apparatus according to claim 15 or 16, wherein said optimisation of said rate-distortion cost function is performed over encoding modes, each mode comprising parameters controlling said encoding of said mesh data of said frame.

18. The apparatus according to any one of claims 15 to 17, wherein the parameters of a mode in the encoding modes are associated with one of local QP adaptation, target resolution adaptation, and slice type adaptation.

19. The apparatus according to any one of claims 15 to 18, wherein the optimization of the rate-distortion cost function comprises: Optimizing the geometric terms of the rate-distortion cost function, determining the optimal mode of the coding mode and the corresponding bit rate; as well as A texture term of the rate-distortion cost function is optimized, the texture term comprising the corresponding bit rate.

20. The apparatus of any one of claims 15 to 19, wherein the selecting comprises: If the motion path cost is lower than the static path cost, the bitstream generated by the encoding according to the motion path is selected.

21. The apparatus of any one of claims 15 to 19, wherein the selecting comprises: If the motion path cost is below a predetermined threshold, the bitstream generated by the encoding according to the motion path is selected.

22. A non-transitory computer readable medium comprising instructions executable by at least one processor to perform a method for encoding mesh data, the method comprising: receiving a sequence of frames, each frame including grid data; as well as For a frame in the sequence: encoding the mesh data of the frame, encoding according to a static path of a multipath encoder and according to a moving path of a multipath encoder, calculating a static path cost of the encoding according to the static path and a moving path cost of the encoding according to the moving path, the costs being calculated by optimizing a rate-distortion cost function, and Based on the motion path cost and the static path cost, a bitstream generated by the encoding according to the motion path or a bitstream generated by the encoding according to the static path is selected.

Images