Three-dimensional grid coding and decoding method and device based on self-adaptive wavelet transform

Abstract

The invention discloses a three-dimensional grid coding and decoding method and device based on self-adaptive wavelet transformation, and the coding method comprises the steps: obtaining vertex displacement data, carrying out the wavelet transformation of the vertex displacement data through employing a self-adaptive wavelet transformation method, taking a transformed displacement vector as a wavelet transformation coefficient, and carrying out the wavelet transformation of the vertex displacement data; a quantized wavelet transform coefficient is obtained and coded to obtain a displacement information code stream; the decoding method comprises the following steps: decoding an obtained displacement information code stream and an auxiliary information code stream; carrying out inverse quantization on a decoding result of the displacement information code stream; adaptively determining an update weight in a wavelet transform process based on a decoding result of the auxiliary information code stream; performing wavelet inverse transformation on the inverse quantization result based on the determined update weight to obtain vertex displacement data of the subdivided grid; and obtaining a reconstructed grid by using the vertex displacement data. According to the method, the three-dimensional grid displacement coding and decoding method in the VDMC can be optimized, the optimization method is combined with the V3C standard, and the coding and decoding efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional mesh coding and decoding, and in particular relates to a three-dimensional mesh coding and decoding method and device based on adaptive wavelet transform. Background Art

[0002] In recent years, with the rapid development of multimedia technology, related research results have been rapidly industrialized and have become an indispensable part of people's lives. Three-dimensional models have become a new generation of digital media after audio, images, and videos. Three-dimensional grids are a commonly used way to represent three-dimensional models. Compared with traditional multimedia such as images and videos, three-dimensional grid models have stronger interactivity and realism, making them more and more widely used in various fields such as commerce, manufacturing, construction, education, medicine, entertainment, art, and military.

[0003] Although there are many ways to represent 3D meshes, triangular meshes are still the most common representation method. A 3D mesh can be considered to be composed of three basic elements: vertices, edges, and faces. Vertices are the most basic elements in a mesh, and they define a position in a 3D space. Edges are line segments connecting two vertices in a mesh. Faces can be considered as polygons formed by closed paths of edges. For a triangular mesh, each face is a triangle.

[0004] The information contained in the mesh is usually divided into three categories: geometric information, connection information, and attribute information. Geometric information refers to the position of each vertex of the mesh in three-dimensional space. Connection information describes the association between the elements in the mesh, that is, the connection relationship between vertices. Attribute information is optional, and it can associate attributes to corresponding mesh elements (such as vertex color, normal vector, etc. can be associated with mesh vertices). Mesh parameterization can also be used to map the mesh from three-dimensional space to a two-dimensional plane area. This mapping relationship is usually described by a set of parameter coordinates, called UV coordinates or texture coordinates, which are associated with mesh vertices. This two-dimensional mapping can be used to represent high-resolution attribute information, such as textures, normal vectors, etc.

[0005] In almost all application fields that use 3D meshes (such as computational simulation, entertainment, medical imaging, digital cultural relics, computer design, e-commerce, etc.), as people's demand for higher visual effects of 3D mesh models increases, the models are becoming more and more complex, and the accuracy of the models is also increasing. Therefore, the amount of data required to represent the 3D mesh is also increasing accordingly. The above problems have led to the increasing complexity of the processing, visualization, transmission and storage of 3D meshes. 3D mesh compression can be regarded as a way to solve the above problems. It reduces the size of model data and is conducive to the processing, storage and transmission of 3D meshes. Therefore, it is necessary to propose an efficient and general 3D mesh compression algorithm.

[0006] Recently, MPEG, an international standardization organization in the field of audio and video coding compression, has begun to develop a compression standard VDMC (Video-based dynamic mesh coding) for three-dimensional meshes. This standard is based on the existing V3C (Visual Volumetric Video-based Coding) standard. The V3C standard provides a general method for compressing three-dimensional models, which can be presented in the form of point clouds, meshes, or panoramic videos. Making the compression method of three-dimensional mesh models compatible with this standard is conducive to the promotion and applicability of this method. Therefore, it is of great significance to optimize the three-dimensional mesh displacement encoding and decoding method in VDMC and combine the optimization method with the V3C standard. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a three-dimensional grid encoding and decoding method and device based on adaptive wavelet transform. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0008] In a first aspect, an embodiment of the present invention provides a three-dimensional mesh coding method based on adaptive wavelet transform, comprising:

[0009] Acquire vertex displacement data; wherein the vertex displacement data is obtained according to the displacement information of the vertices in the subdivided mesh after the base mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded;

[0010] Adopting an adaptive wavelet transform method to perform wavelet transform on the vertex displacement data, and using the transformed displacement vector as a wavelet transform coefficient; wherein the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data;

[0011] quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients;

[0012] The quantized wavelet transform coefficients are encoded to obtain a displacement information code stream.

[0013] In one embodiment of the present invention, the adaptive wavelet transform method comprises an adaptive wavelet transform method based on a first-order moment invariance criterion.

[0014] In one embodiment of the present invention, the adaptive wavelet transform method comprises an adaptive wavelet transform method based on a minimum reconstruction error criterion.

[0015] In one embodiment of the present invention, the adaptive wavelet transform method comprises an adaptive wavelet transform method based on mesh connectivity relationship information.

[0016] In a second aspect, an embodiment of the present invention provides a three-dimensional grid coding device based on adaptive wavelet transform, comprising:

[0017] A data acquisition module, used to acquire vertex displacement data; wherein the vertex displacement data is obtained according to the displacement information of the mesh vertices after the basic mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded;

[0018] A wavelet transform module, used to perform wavelet transform on the vertex displacement data by using an adaptive wavelet transform method, and use the transformed displacement vector as a wavelet transform coefficient; wherein the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data;

[0019] A quantization module, used for quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients;

[0020] The encoding module is used to encode the quantized wavelet transform coefficients to obtain a displacement information code stream.

[0021] In a third aspect, an embodiment of the present invention provides a three-dimensional mesh decoding method based on adaptive wavelet transform, comprising:

[0022] Decoding the acquired displacement information code stream and auxiliary information code stream;

[0023] De-quantizing the decoding result of the displacement information code stream to obtain a de-quantization result;

[0024] Based on the decoding result of the auxiliary information code stream, adaptively determine the update weight in the wavelet transform process; based on the determined update weight, perform wavelet inverse transform on the dequantization result to obtain vertex displacement data of the subdivided mesh; wherein the vertex displacement data of the subdivided mesh represents the displacement vector from the vertex of the decoded basic mesh after subdivision to the mesh to be reconstructed;

[0025] The reconstructed mesh is obtained by using the vertex displacement data of the subdivided mesh.

[0026] In one embodiment of the present invention, the adaptive determination of update weights in the wavelet transform process includes:

[0027] The adaptive inverse wavelet transform method based on the first-order moment invariance criterion utilizes the decoding of the auxiliary information code stream. In one embodiment of the present invention, the adaptive determination of the update weights in the wavelet transform process includes:

[0028] The adaptive inverse wavelet transform method based on the minimum reconstruction error criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

[0029] In one embodiment of the present invention, the adaptive determination of update weights in the wavelet transform process includes:

[0030] The connectivity relationship information of the subdivided decoded basic grid is used to determine the update weights in the wavelet transform process.

[0031] In a fourth aspect, an embodiment of the present invention provides a three-dimensional grid decoding device based on adaptive wavelet transform, comprising:

[0032] A decoding module, used for decoding the acquired displacement information code stream and auxiliary information code stream;

[0033] The dequantization module is used to dequantize the decoding result of the displacement information code stream to obtain a dequantization result;

[0034] The inverse wavelet transform module is used to adaptively determine the update weights in the wavelet transform process based on the decoding results of the auxiliary information code stream; based on the determined update weights, the inverse wavelet transform is performed on the inverse quantization results to obtain vertex displacement data of the subdivided mesh; wherein the vertex displacement data of the subdivided mesh represents the displacement vectors from the vertices of the decoded basic mesh after subdivision to the mesh to be reconstructed;

[0035] The mesh reconstruction module is used to obtain a reconstructed mesh using the vertex displacement data of the subdivided mesh.

[0036] In the three-dimensional mesh encoding method based on adaptive wavelet transform provided by the embodiment of the present invention, for the displacement encoding link, firstly, vertex displacement data is obtained; secondly, the vertex displacement data is subjected to wavelet transform by using the adaptive wavelet transform method, and the transformed displacement vector is used as the wavelet transform coefficient; then, the wavelet transform coefficient is quantized to obtain the quantized wavelet transform coefficient; finally, the quantized wavelet transform coefficient is encoded to obtain the displacement information code stream. In the three-dimensional mesh decoding method based on adaptive wavelet transform provided by the embodiment of the present invention, for the displacement decoding link, firstly, the acquired displacement information code stream and auxiliary information code stream are decoded; secondly, the decoding result of the displacement information code stream is inversely quantized to obtain the inverse quantization result; then, based on the decoding result of the auxiliary information code stream, the update weight in the wavelet transform process is adaptively determined; based on the determined update weight, the inverse quantization result is subjected to wavelet inverse transform to obtain the vertex displacement data of the subdivided mesh; finally, the vertex displacement data of the subdivided mesh is used to obtain the reconstructed mesh. The present invention adopts an adaptive wavelet transform method in the process of displacement coding, and adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data, which can ensure that the low-frequency signal has a high degree of approximation with the original signal, thereby improving the accuracy of the prediction in the next lifting iteration process, and further improving the coding efficiency of the displacement data; and in the process of displacement decoding, based on the decoding result of the auxiliary information code stream, the update weight in the wavelet transform process is adaptively determined; the present invention can optimize the three-dimensional grid displacement coding method in VDMC, and combine the optimization method with the V3C standard to improve the coding efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1( a ) is an example diagram of the V3C standard for the encoding end;

[0038] FIG1( b ) is an example diagram of the V3C standard for a decoding end;

[0039] Figure 2(a) is a schematic diagram of the current coding framework of VDMC;

[0040] Figure 2(b) is a schematic diagram of the current decoding framework of VDMC;

[0041] Figure 3 It is a schematic diagram of the Lifting Scheme processing process;

[0042] Figure 4 A schematic flow chart of a three-dimensional grid coding method based on adaptive wavelet transform provided by an embodiment of the present invention;

[0043] Figure 5 A schematic block diagram of the implementation process of displacement coding at the encoding end of the adaptive wavelet transform solution provided by an embodiment of the present invention;

[0044] Figure 6 Schematic diagram of the process of using Lifting Scheme for wavelet transform of dynamic grid displacement data in an embodiment of the present invention;

[0045] Figure 7 It is a block diagram for realizing a one-time adaptive lifting iterative transformation of displacement data in an embodiment of the present invention;

[0046] Figure 8 Simplify the example diagram for the grid;

[0047] Fig. 9 It is a schematic diagram of the geometric displacement vector calculation method;

[0048] Fig.10 Schematic diagram of the basic mesh compression process;

[0049] Fig.11 A block diagram for realizing up-sampling of a low-frequency approximate signal when an adaptive wavelet transform method based on a minimum reconstruction error criterion is adopted in an embodiment of the present invention;

[0050] Fig.12 Schematic diagram of texture map conversion;

[0051] Fig.13 A schematic diagram of the structure of a three-dimensional grid encoding device based on adaptive wavelet transform provided by an embodiment of the present invention;

[0052] Fig.14 A schematic flow chart of a three-dimensional grid decoding method based on adaptive wavelet transform provided by an embodiment of the present invention;

[0053] Fig.15 A schematic block diagram of the implementation process of the adaptive wavelet transform solution provided by the embodiment of the present invention at the decoding end for shift decoding;

[0054] Fig.16 A block diagram for implementing a one-time adaptive lifting inverse iterative transformation of displacement data in an embodiment of the present invention;

[0055] Fig.17 A schematic structural diagram of a three-dimensional grid decoding device based on adaptive wavelet transform provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In order to facilitate understanding of the embodiments of the present invention, the related technologies involved are first introduced briefly.

[0058] 1. V3C Standard

[0059] The V3C standard provides a method for encoding and decoding various three-dimensional media through video or image coding technology. Specifically, it converts the three-dimensional media content from a three-dimensional representation into multiple two-dimensional representations (called V3C components) by projection or other methods before encoding, and then encodes the two-dimensional representation using existing video or image coding technology. V3C components mainly include occupancy components, geometric components, and attribute components. The occupancy component can indicate which areas in the two-dimensional representation are associated with the data of the three-dimensional representation; the geometric component indicates information related to the position of the three-dimensional data in space, and the attribute component can provide attribute information corresponding to the vertex, such as material, texture, etc. In addition, the component also contains information on how to reconstruct a three-dimensional model through these components, which is called atlas information. The example diagrams of the V3C standard for the encoding end and the decoding end are shown in Figure 1(a) and Figure 1(b) respectively. Among them, the atlas information is used to associate all components, and additional information for reconstructing from two-dimensional back to three-dimensional is also included in the atlas component. The atlas consists of multiple basic units, which are called patches. Each patch represents an area in the available two-dimensional components and contains the information required to project the area back to three-dimensional space. Please refer to the relevant technical understanding for the specific meaning of each part in Figure 1, and no detailed explanation will be given here.

[0060] 2. VDMC

[0061] VDMC is a standard developed by MPEG for compressing three-dimensional meshes. Its main idea is to compress three-dimensional meshes by using the existing V3C standard. Since three-dimensional meshes have connection information that needs to be encoded, its specific encoding process is slightly different from V3C. It is necessary to expand the syntax, semantics and decoding operations of the V3C standard decoding end to support the decoding and reconstruction of three-dimensional meshes. Figure 2(a) is the current encoding framework of VDMC, and Figure 2(b) is the current decoding framework of VDMC.

[0062] The overall framework of the encoding end is shown in Figure 2(a). For the input mesh, the mesh is first simplified; then the mesh is parameterized to generate new texture coordinates for the mesh; then the parameterized mesh is subdivided and deformed, that is, new vertices are inserted into the mesh according to a specific subdivision method and the distance from the vertex of the subdivided mesh to the nearest neighbor of the input mesh is calculated, which is represented by a displacement vector, called displacement information (represented by vertex displacement in Figure 2(a)). Subsequently, the mesh before subdivision and deformation is called the base mesh, and the base mesh is compressed (that is, base mesh encoding). Specifically, it can be compressed using an existing mesh encoder and output as a base mesh code stream. Among them, in the inter-frame mode, the base mesh encoding can generate a motion vector for each vertex of the base mesh based on the reference frame, and then only the motion vector is compressed.

[0063] The base mesh is reconstructed after encoding to obtain a reconstructed base mesh, which is then subdivided, and the order of displacement in the displacement information is adjusted according to the vertex order of the subdivided and reconstructed base mesh. Subsequently, the displacement information after the order adjustment is subjected to wavelet transformation, and the transformed displacement vector is used as the wavelet transformation coefficient, and the wavelet transformation coefficient is quantized. Then, the quantized wavelet transformation coefficient can be arranged into a two-dimensional image according to a specific scanning order, and the two-dimensional image is encoded using a video encoder or the like, and a displacement information code stream is output.

[0064] Then, the reconstructed displacement information is applied to the subdivided deformed base grid to reconstruct the deformed grid, and the reconstructed subdivided deformed grid is obtained. The grid and the original input grid and its corresponding texture map are texture-converted to obtain the texture map corresponding to the reconstructed grid (the texture map after conversion in Figure 2 (a)), and the texture map is texture-compressed. The parameters used in the overall encoding process, such as the type of video encoder used, the type of grid encoder, the transform parameters, the quantization parameters, etc., are transmitted to the decoding end through auxiliary information.

[0065] The overall framework of the decoding end is shown in Figure 2(b). For the received code stream, the decoding end first demultiplexes each part of the code stream to obtain the basic grid code stream, displacement information code stream, texture map code stream and auxiliary information code stream. For the basic grid code stream, the grid decoder indicated by the auxiliary information is used to decode the basic grid. The displacement information code stream and the texture map code stream are decoded by the corresponding decoders. For the displacement part, after the displacement information is decoded, the displacement vector needs to be taken out of the image through displacement decoding, and dequantization, inverse transformation and other steps are performed, and then it is applied to the subdivided basic grid to obtain the deformed grid reconstructed by the decoding end. After decoding, the texture map is the texture map corresponding to the reconstructed deformed grid. The subsequent application or rendering module processes the reconstructed deformed grid and the decoded texture map as input.

[0066] The encoding and decoding process of VDMC will be described in detail below in conjunction with the encoding and decoding method of an embodiment of the present invention.

[0067] 3. Lifting Scheme Wavelet Transform Scheme

[0068] Lifting Scheme is one of the most commonly used discrete wavelet transform schemes, such as Figure 3 As shown, the original signal s j After the three steps of splitting (Split), predicting (Predict) and updating (Update), it is decomposed into a low-frequency approximate signal s j-1 and a high-frequency detail signal d j-1 . Then the low-frequency approximation signal s j-1 The above three steps are iterated continuously until the number of iterations is met or the DC component s0 of the signal is obtained. The above three steps are described in detail as follows:

[0069] Split: downsample the original signal into even and odd parts;

[0070] Prediction: Using the local correlation between the even subset and the odd subset, the even subset is used to predict the odd subset, and then the prediction residual d is obtained by subtracting the predicted value from the odd subset. It represents the high-frequency details of the original signal.

[0071] The prediction process is shown in formula (1):

[0072] d j-1 =odd j-1 -P(even j-1 ) (1);

[0073] Among them, d j-1 Represents the prediction residual; odd j-1 Indicates the current odd subset part; even j-1 represents the current even subset part; P represents the prediction weight.

[0074] Update: The even number subset is corrected by the extracted high-frequency signal to obtain the low-frequency approximation of the original signal. The purpose of the correction is to smooth the downsampled signal to ensure that the rough signal after downsampling has the same overall characteristics and trends as the original signal. The update process is shown in formula (2):

[0075] s j-1 =even j-1 +U(d j-1 ) (2);

[0076] Among them, U represents the update weight.

[0077] After multiple liftings, the signal is decomposed into a low-frequency approximate signal and a series of high-frequency detail signals, resulting in a more compact representation of the signal, which helps to improve data compression efficiency.

[0078] In a first aspect, an embodiment of the present invention provides a three-dimensional grid coding method based on adaptive wavelet transform, which is applied to a coding end, such as Figure 4 As shown, the method may include steps Sc1 to Sc4.

[0079] Sc1, obtain vertex displacement data;

[0080] The vertex displacement data is obtained according to the displacement information of the vertices in the subdivided mesh after the base mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded;

[0081] Sc2, using an adaptive wavelet transform method to perform wavelet transform on the vertex displacement data, and using the transformed displacement vector as a wavelet transform coefficient;

[0082] Wherein, the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data;

[0083] Sc3, quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients;

[0084] Sc4, encoding the quantized wavelet transform coefficients to obtain a displacement information code stream.

[0085] The embodiment of the present invention essentially proposes a transformation scheme for dynamic grid displacement data in Sc2, called adaptive wavelet transform, which is applied to the displacement coding link in the three-dimensional grid coding process to form a new three-dimensional grid coding method. The adaptive wavelet transform scheme adaptively adjusts the filter coefficients used by the wavelet transform according to the distribution characteristics of the displacement data, so that the displacement data can be restored by inverse wavelet transform at the decoding end. For the implementation process of displacement coding of the adaptive wavelet transform scheme at the encoding end, please refer to Figure 5 To facilitate understanding of the embodiments of the present invention, first Figure 5 Give a brief introduction.

[0086] Displacement is an important part of the dynamic mesh encoding process, storing the distance information from the vertices of the subdivided mesh obtained after the base mesh is subdivided to the nearest neighbor point of the original input mesh along the normal direction.

[0087] Specifically, Figure 5The displacement data input in is the vertex displacement data obtained by Sc1. Please refer to Figure 2(a) for understanding. The input mesh is mesh simplified and mesh parameterized before subdivision and deformation, and the displacement information of the vertices in the subdivided mesh is output (indicated by vertex displacement in Figure 2(a)). The displacement information is the displacement vector of each vertex in the subdivided mesh to the original input mesh. The mesh before subdivision and deformation is the base mesh. The base mesh is compressed and then reconstructed to obtain a reconstructed base mesh, which is then subdivided. The order of displacement in the displacement information is adjusted according to the vertex order of the subdivided and reconstructed base mesh to obtain the vertex displacement data obtained by Sc1. Figure 5 And the displacement data is simplified in the following text.

[0088] At the encoding end, the embodiment of the present invention first pre-processes the displacement data, and the pre-processing includes all operations on the displacement data before the adaptive wavelet transform, such as coordinate system conversion and other operations. Then, the displacement data after the pre-processing is subjected to an adaptive wavelet transform. In this process, the Lifting Scheme is used to perform a wavelet transform on the displacement data, and the weights are adaptively calculated and updated according to the distribution characteristics of the displacement data, thereby realizing an adaptive wavelet transform. The adaptive wavelet transform outputs the prediction and update weights used in the transformation process as transformation parameters. After the adaptive wavelet transform, the transformed displacement vector is output as a wavelet transform coefficient, and then after quantization, it can be encoded to obtain a displacement information code stream. There can be many specific encoding methods here, such as video encoding, entropy encoding, etc., and the specific encoding method is not limited here.

[0089] Among them, the lifting scheme is used for the specific design steps of wavelet transform of displacement data of dynamic grids as follows Figure 6 shown.

[0090] Figure 6 Each point in the grid corresponds to a displacement data. The Split step divides the grid into a black dot part and a white dot part. The black dot part is similar to the even part subset in the Lifting Scheme, and the white dot part is similar to the odd part subset in the Lifting Scheme. Figure 6 As shown by the red arrow in the middle, the midpoint prediction method is used to calculate the predicted value of the displacement data of the white dot through the displacement data of the two black dots adjacent to the white dot, and then the original displacement data corresponding to the white dot minus the predicted value is subtracted to obtain the prediction residual. The specific prediction formula is shown in formula (3).

[0091]

[0092] in, is the original displacement data of the white dot to be predicted; dj Indicates the displacement data of the white dots The prediction residuals of and for Displacement data of two adjacent black dots; is the prediction weight.

[0093] Update steps are as follows Figure 6 As shown by the blue arrow in the middle, the value of the black point displacement data is adjusted by the residual of all the adjacent white dots of the black point. The update formula is shown in formula (4).

[0094]

[0095] in, Represents the displacement data of the black points to be processed in the update step; express The displacement data value after the update step; U represents the update weight; d j express The predicted residuals of the displacement data of the surrounding adjacent white dots; j = a~f represents The 6 adjacent white dots around it.

[0096] After a lifting, the displacement data on the original input mesh vertices is decomposed into a rough approximate mesh ( Figure 6 The mesh is composed of interconnected black dots) with displacement data on the vertices (low-frequency approximate information) and the residuals on the white dots (high-frequency detail information).

[0097] The embodiment of the present invention adaptively calculates the update weight in each lifting by using the distribution characteristics of the displacement data, thereby realizing adaptive wavelet transform. Different calculation criteria correspond to different adaptive wavelet transform schemes, such as Figure 7 As shown, Figure 7 This is the block diagram for implementing an adaptive lifting iterative transformation of displacement data, where s j Represents the original displacement data of the input, s evenj-1 Represents the displacement data of the even-numbered part (i.e. Figure 6 Displacement data corresponding to the black points in the grid), s oddj-1 Represents the displacement data of the odd part (i.e. Figure 6 The displacement data corresponding to the white dots in the grid), P and U adp Represent the prediction weight and adaptive update weight respectively, s j-1 and d j-1They represent the low-frequency approximate signal and the high-frequency detail signal output by the adaptive wavelet transform. It should be noted that the embodiment of the present invention does not limit the number of lifting iterations performed during the adaptive wavelet transform process.

[0098] In order to understand the solution of the embodiment of the present invention in detail and to understand the entire three-dimensional grid coding process, steps Sc1 to Sc4 are specifically described below in conjunction with the VDMC coding framework.

[0099] (1) For Sc1, please refer to Figure 2(a) for the order of mesh simplification, mesh parameterization, subdivision deformation, basic mesh compression, subdivision and displacement adjustment. Specifically:

[0100] Mesh simplification

[0101] Mesh simplification is to simplify the current input mesh to a basic mesh with relatively few points and faces, and keep the shape of the original mesh as much as possible. The key of mesh simplification is the simplification operation and the corresponding error measurement. A feasible mesh simplification operation is Figure 8 As shown in the figure, the vertices at both ends of the edge are merged into one vertex and the connection between the two vertices is deleted. This process is repeated in the entire mesh according to certain rules to reduce the number of faces and vertices of the mesh to the target value.

[0102] During the simplification process, you can select an error metric to optimize the simplified result. For example, you can select the sum of the coefficients of the equations of all adjacent faces of a vertex as the error metric of the vertex, and the error metric of the corresponding edge is the sum of the error metrics of the two vertices on the edge. In other words, the error generated by merging an edge is the sum of the distances of all adjacent planes from the merged vertex to the original two vertices of the edge.

[0103] After determining the simplification operation and the corresponding error metric, the mesh simplification is iterated. First, the vertex error of the initial mesh is calculated to obtain the error of each edge. Then each edge is arranged from small to large according to the error, and the edge with the smallest error is selected for merging each time. At the same time, the position of the merged vertex is calculated, and the error of all edges related to the merged vertex is updated, that is, the order of edge arrangement is updated to ensure that each iteration is based on the global error metric. Through iteration, the mesh surface is simplified to the number required to meet the lossy coding.

[0104] In the embodiment of the present invention, any existing grid simplification method can be used for implementation, and no specific limitation is made here.

[0105] Mesh parameterization

[0106] Texture coordinates may be regenerated based on the reconstructed base mesh of the current frame. This step requires generating texture coordinates for each attribute map of the input mesh. If multiple attribute maps have similar characteristics, they can share the same texture coordinates. There are many ways to generate texture coordinates, including mesh parameterization. There are many algorithms for parameterizing meshes, such as the Isocharts algorithm, which uses spectral analysis to achieve stretch-driven 3D mesh parameterization, UV-unwrapping, slicing, and packing the 3D mesh into a 2D texture domain.

[0107] In the embodiment of the present invention, the existing grid parameterization method can be used for implementation, and no specific limitation is made here.

[0108] Subdivision Deformation

[0109] Subdivision and deformation are optional steps that can improve the quality of the mesh reconstructed at the decoding end. This process can be omitted when the quality of the basic mesh can meet the application requirements.

[0110] The basic idea of ​​subdivision and deformation is as follows Fig. 9 As shown in , the same concept is applied to the input 3D mesh to generate displacement vector information. Fig. 9 In , the input 2D curve (represented by 2D polylines), called the "original" curve, is first downsampled to generate a base curve / polyline, called the "simplified" curve. The subdivision scheme is then applied to the simplified polyline to generate the "subdivided" curve. The subdivided polyline is then deformed to get a better approximation of the original curve. That is, the geometric displacement vector ( Fig. 9 The shape of the subdivided curve is made as close to the shape of the original curve as possible (as shown by the red arrow in the figure). These geometric displacement vectors are the geometric displacement vector information output by the subdivided deformation, referred to as displacement information, and are represented by vertex displacement in Figure 2(a). The same deformation process is also applied to the attribute information corresponding to the vertex to obtain the corresponding attribute displacement vector.

[0111] Base Mesh Compression

[0112] Basic mesh compression is to encode the basic mesh information output after subdivision deformation. There are two main modes of basic mesh compression, namely intra-frame mode and inter-frame mode. Fig.10 As shown. In intra mode, the input of basic mesh compression is a 3D mesh, which contains geometric coordinates, connection relationships, and attribute information associated with vertices. In inter mode, the input of basic mesh compression is motion vectors and their identifiers, as well as possible intra-frame sub-meshes. After encoding, the encoded mesh needs to be reconstructed so that it can be provided to subsequent links for processing. The following is a brief introduction to the encoding of the two modes:

[0113] ①Intra-frame mode

[0114] The grid coding in intra mode can use the existing grid encoder to encode the input base grid, such as Draco, etc. The grid encoder type is encoded with auxiliary information and passed to the decoding end.

[0115] ②Inter-frame mode

[0116] In inter-frame mode, motion vectors and identifiers need to be encoded. The motion vector identifier is an array consisting of numbers greater than or equal to 0. The array size is the same as the number of vertices of the base grid of the reference frame corresponding to the temporal motion vector identifier. The array can be directly encoded using an existing entropy coding algorithm, such as CABAC.

[0117] Rebuilding the base mesh

[0118] After the base grid is encoded, the base grid needs to be reconstructed. For a grid encoded in intra-frame mode, the encoded grid can be decoded to obtain the reconstructed base grid.

[0119] For the basic mesh information encoded in the inter-frame mode, it is necessary to reconstruct the inter-frame sub-mesh based on the motion vector and the identifier. If there is an intra-frame sub-mesh in the inter-frame mode, it is also necessary to merge the reconstructed inter-frame sub-mesh with the intra-frame sub-mesh to obtain the reconstructed basic mesh. After obtaining the reconstructed basic mesh, it is necessary to subdivide it for subsequent vertex displacement encoding and reconstruction of the deformed mesh.

[0120] Segmentation

[0121] This step is similar to the above-mentioned subdivision steps and will not be described in detail here.

[0122] Adjust displacement order

[0123] This step is the preparation for vertex displacement encoding. To encode vertex displacement, the order of vertex displacement needs to be adjusted according to the reconstructed base mesh after subdivision.

[0124] The main purpose of adjusting the displacement order is to adjust the vertex displacement order according to the reconstructed base mesh after subdivision, so that the displacement order is the same as the vertex order of the subdivided base mesh, and each vertex corresponds to a displacement vector. There is also an optional step in this process, which is to convert the coordinate system of the vertex displacement from the Cartesian coordinate system to the local coordinate system, corresponding to Figure 5 pre-processing stage.

[0125] (2) For Sc2, please refer to Figure 2(a) for the wavelet transform understanding. Specifically:

[0126] Wavelet Transform

[0127] The lifting-based adaptive wavelet transform scheme designed above is applied to this link. The number of lifting iterations in the adaptive wavelet transform is the same as the number of subdivisions of the reconstructed basic grid. For each lifting iteration:

[0128] The split operation uses the displacement data corresponding to the vertices that existed before the reconstruction of the base mesh subdivision as the data points of the even part, and uses the displacement data corresponding to the mesh subdivision interpolation points as the data points of the odd part.

[0129] The prediction operation adopts the midpoint prediction method, and its prediction process is defined as shown in formula (5):

[0130]

[0131] In formula (5), v is the newly inserted midpoint on the edge (v1, v2), Signal(v), Signal(v1) and Signal(v2) are the displacement vectors corresponding to vertices v, v1 and v2, that is, the data points of the odd-numbered part of the displacement and the data points of the two adjacent even-numbered parts. It can be understood that the Signal(v) on the right side and the Signal(v) on the left side in formula (5) respectively represent the displacement vectors corresponding to the vertex v before and after the prediction.

[0132] The update operation adjusts the value of the even-part displacement data point by updating the weight and the odd-part displacement data point adjacent to the even-part displacement data point. The update process is defined as shown in formula (6):

[0133]

[0134] In formula (6), v * is the set of all vertices adjacent to vertex v, Signal(v) is the displacement data of the even part corresponding to vertex v, the Signal(v) on the right and left sides of formula (6) represent the data before and after the update respectively; Signal(w) is the residual value corresponding to the adjacent point, and the transformed displacement vector is called the wavelet transform coefficient, which is the wavelet transform result.

[0135] As described above, the embodiment of the present invention adaptively calculates the update weight in each lifting through the distribution characteristics of the displacement data, thereby realizing adaptive wavelet transform. Different calculation criteria correspond to different adaptive wavelet transform schemes. In an optional implementation, for the update weight U adp According to different calculation criteria, there are three solutions:

[0136] 1) In an optional implementation manner, the adaptive wavelet transform method includes an adaptive wavelet transform method based on a first-order moment invariance criterion.

[0137] The first-order moment is the mean of the signal. The adaptive wavelet transform method based on the first-order moment invariance criterion determines the update weight on the condition that the mean of the updated low-frequency approximate signal is equal to that of the original signal, that is, an update weight U is adaptively calculated for each lifting iteration. adp The definition of this method is shown in formula (7):

[0138]

[0139] In formula (7), x represents the amount of data in the even-numbered shift part (i.e. Figure 6 The number of black dots in the grid); s j-1 Represents the updated data for the even-numbered displacement part, the low-frequency approximate signal output by this adaptive wavelet transform; s j-1,i represents the i-th displacement data in the low-frequency approximate signal; l represents the number of original displacement data (i.e. Figure 6 The number of all vertices in the mesh); s j Represents the original displacement data of this adaptive wavelet transform input; s j,i represents the i-th displacement data in the original displacement data; let y represent the data amount of the odd-numbered part of the displacement (i.e. Figure 6 The number of white dots in the grid). These three variables satisfy equation (8):

[0140] x+y=l (8);

[0141] Substituting the prediction formula expressed by formula (3) and the update formula expressed by formula (4) into formula (7), we can finally derive the formula for adaptively calculating the update weight in this way:

[0142]

[0143] In formula (9), Data representing the even-numbered portion of the original displacement data; represents the i-th displacement data in the even-numbered part of the data; z i Represents the number of adjacent points of the vertex corresponding to the ith displacement data in the even-numbered part of the displacement (i.e. Figure 6 The number of adjacent points of a black point in the grid). Data representing the odd-numbered portion of the original displacement data; represents the i-th displacement data in the odd-numbered part of the data. The encoder can adaptively calculate an update weight in each lifting iteration by substituting the specific displacement data into formula (9).

[0144] Similarly, setting the update weight to a fixed value of 1 / 8 in VDMC also complies with the first-order moment invariance criterion, which is the result obtained by using equation (9) to calculate the displacement data of all test sequences of VDMC. This scheme requires that the adaptive update weights calculated in the entire adaptive wavelet transform process be transmitted as transform parameters for inverse wavelet transform at the decoding end.

[0145] This scheme ensures that the low-frequency signal has a high degree of similarity with the original signal by ensuring that the low-frequency signal outputted in each lifting iteration has a similar concentration trend and center position to the original signal outputted, thereby improving the accuracy of the prediction in the next lifting iteration process and further improving the encoding efficiency of the displacement data.

[0146] 2) In an optional implementation manner, the adaptive wavelet transform method includes an adaptive wavelet transform method based on a minimum reconstruction error criterion.

[0147] The adaptive wavelet transform method based on the minimum reconstruction error criterion is to determine the update weights based on the condition that the upsampled displacement data obtained by upsampling the updated output displacement data is equal to the number of the original displacement data and the data points correspond one to one, and the sum of the squares of the differences between the corresponding points of the upsampled displacement data and the original displacement data is minimized. Specifically, the minimum reconstruction error criterion refers to performing an upsampling operation on the updated output displacement data so that the number of the upsampled displacement data is equal to the number of the original displacement data and the data points correspond one to one, and the sum of the squares of the differences between the corresponding points of the two data is minimized, and the update weights of each lifting iteration are adaptively calculated based on this condition.

[0148] The specific upsampling process is as follows: Fig.11 As shown, let d j-1 =0 for s j-1 Perform inverse transform to obtain the upsampled data s' j , at this time U adp is an unknown quantity and a variable that needs to be solved. Therefore, the definition of the minimum reconstruction error criterion can be expressed as:

[0149]

[0150] In formula (10), f(U adp ) is a function about updating weights; s j Represents the original displacement data of this adaptive wavelet transform input; s j,i represents the i-th displacement data in the original displacement data; s' j Represents the data after upsampling of the low-frequency approximate signal output by this adaptive wavelet transform; s' j,i Represents the i-th displacement data in the upsampled data.

[0151] Substituting the prediction formula represented by formula (3) and the update formula represented by formula (4) into the equation, we can derive:

[0152]

[0153] Among them, (d1+…+d i ) represents the sum of the residuals corresponding to all adjacent points of the data point with an even number of shifts (i.e. Figure 6 The residuals corresponding to the white dots adjacent to the black dots in the grid); d y Represents the residual value corresponding to the data point of the odd number of displacements (i.e. Figure 6 The residual value corresponding to the white dots in the grid);

[0154] d x,x+1 =(d1+…+d i ) x +(d1+…+d i ) x+1 (12);

[0155] Among them, x and x+1 represent two even-numbered data points adjacent to the odd-numbered data points (i.e. Figure 6 Two black dots adjacent to the white dots in the grid). So d x,x+1 represents the sum of the residuals corresponding to all adjacent points of the two even-numbered data points. To solve equation (10), we only need to let Finally, the formula for solving the adaptive update weight in this way is:

[0156]

[0157] The encoder can adaptively calculate an update weight at each lifting iteration by substituting the specific displacement data into formula (13). Similarly, setting the update weight to a fixed value of 1 / 4 in VDMC also meets the minimum reconstruction error criterion. This is the result obtained by using formula (13) to calculate the displacement data of all test sequences of VDMC. This method requires the adaptive update weight calculated during the entire adaptive wavelet transform process to be transmitted as a transform parameter for inverse wavelet transform at the decoder.

[0158] This scheme ensures that the low-frequency signal has a high degree of similarity with the original signal by ensuring that the low-frequency signal outputted by each lifting iteration has a similar degree of fluctuation as the original signal outputted, thereby improving the accuracy of the prediction in the next lifting iteration process and further improving the encoding efficiency of the displacement data.

[0159] 3) In an optional implementation manner, the adaptive wavelet transform method includes an adaptive wavelet transform method based on grid connectivity relationship information.

[0160] The adaptive wavelet transform method based on mesh connectivity relationship information determines the update weight through the connectivity relationship information of the subdivided reconstructed basic mesh.

[0161] In an optional implementation, the update weight may be determined by using the connectivity relationship information of the subdivided reconstructed basic mesh, and the method may be specifically described as follows:

[0162] In the update process of each lifting iteration, for each data point in the even-numbered part of the displacement, the number of adjacent points of the corresponding mesh vertex is obtained. The information of the number of adjacent points is used to adaptively calculate an update weight for each data point in the even-numbered part of the displacement. A simple calculation method is to calculate the update weight by the inverse of the number of adjacent points. The calculation formula is as follows:

[0163]

[0164] In formula (14), U adpx represents the update weight corresponding to a data point in the even part of the displacement, z x Indicates the number of adjacent points of the data point.

[0165] The updated weights calculated in this way do not need to be transmitted to the decoding end. The decoding end only needs to use the connectivity relationship information of the subdivided decoding basic grid and adopt the same calculation method as the encoding end to obtain the same adaptive updated weights.

[0166] The previous two methods designed an update weight for each lifting iteration, while this method uses the connectivity information of the grid to design update weights for each data point in the even part of the displacement. Compared with the previous two methods, this method has higher flexibility and can ultimately obtain a more compact representation of the displacement, thereby improving the encoding efficiency of the displacement data.

[0167] (3) For Sc3, please refer to Figure 2(a) for quantitative understanding. Specifically:

[0168] Quantization: The transformed displacement vector, i.e., the wavelet transform coefficient, can be quantized. There are many ways to quantize. An optional method is shown in equations (15) and (16):

[0169] disp[v]·d[k]=floor(disp[v]·d[k]*scale[k]) (15);

[0170]

[0171] Where disp[v] represents the transformed value of the displacement vector of the vth vertex; d[k] represents the kth value of the displacement vector; floor represents rounding down; qp[k] represents the quantization parameter of the kth wavelet transform coefficient; bitDepthPosition represents the bit depth of the current mesh vertex geometry. As mentioned above, after the displacement vector is converted to the coordinate system, its normal component has a more significant effect on the quality than the tangential component, so a larger quantization parameter can be used for the tangential component.

[0172] At the same time, according to the characteristics of wavelet transform, different quantization parameters can be used for the newly generated vertices and the original vertices. That is, for the subdivided vertices, the quantization parameter update is shown in formula (17):

[0173] scale[k]=scale[k]*lodScale[k] (17);

[0174] Wherein, lodScale[k] represents the coefficient of the quantization parameter of the current subdivision level; scale[k] on the right and left sides of formula (17) represent the original and updated quantization parameters respectively.

[0175] (4) For Sc4, please refer to Figure 2(a) for the two-dimensional arrangement and video compression understanding. Specifically:

[0176] Two-dimensional arrangement

[0177] After quantization, the quantized wavelet transform coefficients can be arranged on a two-dimensional image as follows:

[0178] ① Traverse the wavelet coefficients in order from low frequency to high frequency.

[0179] ② For each coefficient, determine the index of the N×M pixel block where it is located (for example, N=M=16), which should be stored in the raster scan order of the block.

[0180] ③ Calculate the position of the corresponding N×M pixel block on the image according to the Morton order.

[0181] The arrangement is not limited here, and other arrangement schemes may also be used, such as zigzag order, raster order, etc. The encoder may explicitly specify the corresponding arrangement scheme in the bitstream.

[0182] Video Compression

[0183] After arranging the quantized wavelet transform coefficients on a two-dimensional image, they can be directly encoded using a video encoder to obtain a displacement information code stream.

[0184] In summary, in an embodiment of the present invention, when encoding vertex displacements, the order of vertex displacements is first adjusted according to the subdivided reconstructed basic grid; then, the displacements are adaptively transformed and quantized to obtain quantized wavelet transform coefficients; and then, the quantized wavelet transform coefficients of the current frame are arranged in a two-dimensional image, which can ultimately form a YUV video, and the displacement information code stream is obtained by encoding with the help of a video encoder.

[0185] The above are the implementation steps of an adaptive wavelet transform solution provided by an embodiment of the present invention for the displacement coding link in the three-dimensional grid coding process. However, it can be understood that the three-dimensional grid coding also includes the following remaining links.

[0186] Deformed Mesh Reconstruction

[0187] Video compression (ie, displacement vector coding) requires obtaining a reconstructed displacement vector value during encoding, that is, obtaining a displacement vector consistent with the decoding end through inverse quantization and inverse transformation.

[0188] The inverse transformation process is the inverse implementation of the above-mentioned adaptive wavelet transform, and the number of inverse lifting iterations in the inverse transformation process is the same as the number of subdivisions of the reconstructed basic grid. For each inverse lifting iteration, the first step is an update operation. The definition of this process is shown in formula (18):

[0189]

[0190] In formula (18), v * is the set of all vertices adjacent to the vertex v of the reconstructed basic mesh after subdivision; Signal(v) is the displacement data corresponding to the vertex v. The Signal(v) on the right and left sides of formula (18) represent the data before and after the update respectively; Signal(w) is the residual value corresponding to the adjacent points.

[0191] The next step is the prediction operation, which uses the same midpoint prediction method as in the above adaptive wavelet transform process, and its definition is shown in formula (19):

[0192]

[0193] In formula (19), v is the newly inserted midpoint on the edge (v1, v2), Signal(v) is the residual value corresponding to the vertex v of the reconstructed basic mesh after subdivision, and the Signal(v) on the right and left sides of formula (19) represent the data before and after prediction respectively; Signal(v1) and Signal(v2) are the displacement vectors corresponding to the vertices v1 and (v2) of the reconstructed basic mesh after subdivision respectively.

[0194] For updating the weight U adpAccording to the calculation criteria used by the above adaptive wavelet transform, there are three ways to obtain it:

[0195] ① Based on the first-order moment invariance criterion

[0196] The adaptive update weights of this method have been transmitted to the decoding end through the transformation parameters, so for this scheme, the update weights can be directly obtained from the transformation parameters.

[0197] ② Based on the minimum reconstruction error criterion

[0198] The adaptive update weights of this method have been transmitted to the decoding end through the transformation parameters, so for this scheme, the update weights can be directly obtained from the transformation parameters.

[0199] ③ Based on grid connectivity relationship

[0200] The acquisition of the adaptive update weight in this manner requires the use of the connectivity relationship information of the subdivided reconstructed basic grid, and is calculated using the same calculation method as the above-mentioned adaptive wavelet transform.

[0201] After obtaining the reconstructed geometric displacement vector, the reconstructed base mesh after subdivision is obtained according to the corresponding geometric displacement vector to obtain the reconstructed subdivision deformed mesh, which is passed for texture map conversion.

[0202] Property Graph Transformation

[0203] Figure 2(a) shows an example of texture map conversion. Attribute map conversion is to perform attribute map conversion based on the original input mesh, the original input attribute map, and the reconstructed deformed mesh, such as Fig.12 The steps of attribute graph conversion are as follows:

[0204] I, calculate the texture coordinates of each pixel on the attribute map to be generated, such as the texture coordinates corresponding to pixel A(i,j) is P(u,v).

[0205] II. Determine whether the texture coordinate is within a certain triangle face after the parameterization of the subdivided deformed mesh.

[0206] III. If the texture coordinate does not belong to any triangle, the pixel is marked as an empty pixel, which can be filled with a filling algorithm.

[0207] IV, if the texture coordinate belongs to a triangle, then

[0208] i, marks the pixel as filled.

[0209] ii. Calculate the center of gravity coordinates of the texture in the current triangle surface based on the texture coordinates.

[0210] iii. According to the barycentric coordinates and the corresponding triangular face, the two-dimensional texture coordinates are mapped to three-dimensional geometric coordinates, that is, mapped to the points on the subdivided deformed grid corresponding to the texture coordinates, such as Fig.12 It is shown as M(x,y,z) in .

[0211] iv. Find the point on the input original grid that is closest to the three-dimensional coordinates, such as Fig.12 As shown in M'(x,y,z).

[0212] v, calculate the centroid coordinates of the three-dimensional coordinates according to the triangle face on which it is located and map them to two dimensions to calculate its texture coordinates, that is, P'(u',v').

[0213] vi, sample the original attribute map input through the texture coordinates to obtain the value A'(i',j') of the corresponding pixel position. Assign this value to the corresponding pixel A(i,j) on the attribute map to be generated.

[0214] Property Graph Compression

[0215] Figure 2(a) shows an example of texture map compression. After obtaining the converted attribute map, the empty pixels therein can be filled using existing filling algorithms (such as the Push-Pull algorithm). Then, existing video encoders such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used to encode it to obtain the bitstream of the output attribute map (Figure 2(a) shows an example of the texture map bitstream). In addition, operations such as color space conversion and chroma subsampling can be selectively applied to enable video encoding to obtain better rate-distortion performance, such as color space conversion from RGB 444 to YUV420. In the case where there are multiple attribute maps, the texture coordinates corresponding to each attribute map need to be identified through auxiliary information during encoding.

[0216] Auxiliary Information

[0217] As shown in Figure 2(a), in the encoding process, there are some alternative schemes in each link, such as the type of grid encoder, the type of video encoder, the grid subdivision scheme, the spatial displacement vector transformation scheme and the transformation parameters, etc. The proposed framework allows the use of different schemes. Therefore, the selected scheme needs to be passed to the decoding end to guide the correct decoding.

[0218] Auxiliary information includes the type of static grid encoder, the type of video encoder, grid subdivision scheme, number of grid iterations, geometric displacement vector transformation scheme and transformation parameters, wavelet transform coefficient arrangement scheme, color conversion scheme, etc. For the grid part encoded using the inter-frame coding mode, it also includes the corresponding reference frame list, etc.

[0219] After the encoding of all links is completed, the basic grid code stream, displacement information code stream, texture map code stream and auxiliary information code stream are mixed to obtain the encoded code stream finally output by the encoding end.

[0220] The embodiment of the present invention provides a three-dimensional mesh coding method based on adaptive wavelet transform. For the displacement coding link, firstly, vertex displacement data is obtained; secondly, the vertex displacement data is subjected to wavelet transform by using an adaptive wavelet transform method, and the transformed displacement vector is used as a wavelet transform coefficient; then, the wavelet transform coefficient is quantized to obtain a quantized wavelet transform coefficient; finally, the quantized wavelet transform coefficient is encoded to obtain a displacement information code stream. Since the adaptive wavelet transform method used adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data, it can ensure that the low-frequency signal has a high degree of approximation with the original signal, thereby improving the accuracy of the prediction in the next lifting iteration process, and further improving the coding efficiency of the displacement data. It can be seen that the present invention can optimize the three-dimensional mesh displacement coding method in VDMC, and combine the optimization method with the V3C standard to improve the coding efficiency and accuracy.

[0221] In a second aspect, corresponding to the method embodiment described in the first aspect, an embodiment of the present invention further provides a three-dimensional grid encoding device based on adaptive wavelet transform, such as Fig.13 As shown, the device comprises:

[0222] A data acquisition module, used to acquire vertex displacement data; wherein the vertex displacement data is obtained according to the displacement information of the mesh vertices after the basic mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded;

[0223] A wavelet transform module, used to perform wavelet transform on the vertex displacement data by using an adaptive wavelet transform method, and use the transformed displacement vector as a wavelet transform coefficient; wherein the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data;

[0224] A quantization module, used for quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients;

[0225] The encoding module is used to encode the quantized wavelet transform coefficients to obtain a displacement information code stream.

[0226] Optionally, the adaptive wavelet transform method adopted by the wavelet transform module includes an adaptive wavelet transform method based on a first-order moment invariance criterion.

[0227] Optionally, the adaptive wavelet transform method adopted by the wavelet transform module includes an adaptive wavelet transform method based on a minimum reconstruction error criterion.

[0228] Optionally, the adaptive wavelet transform method adopted by the wavelet transform module includes an adaptive wavelet transform method based on grid connectivity relationship information.

[0229] For the processing process of each module, please refer to the relevant description of the three-dimensional grid coding method based on adaptive wavelet transform provided in the first aspect, which will not be repeated here.

[0230] In a third aspect, an embodiment of the present invention provides a three-dimensional grid decoding method based on adaptive wavelet transform, which is applied to a decoding end, such as Fig.14 As shown, the method may include steps Sd1 to Sd4.

[0231] Sd1, decoding the acquired displacement information code stream and auxiliary information code stream;

[0232] Sd2, dequantize the decoding result of the displacement information code stream to obtain a dequantized result;

[0233] Sd3, based on the decoding result of the auxiliary information code stream, adaptively determine the update weight in the wavelet transform process; based on the determined update weight, perform inverse wavelet transform on the dequantization result to obtain vertex displacement data of the subdivided grid;

[0234] The vertex displacement data of the subdivided mesh represents the displacement vectors from the vertices of the subdivided decoded basic mesh to the mesh to be reconstructed.

[0235] Sd4, obtaining a reconstructed mesh using the vertex displacement data of the subdivided mesh.

[0236] The above steps in the embodiment of the present invention mainly propose a displacement decoding solution, which is used in the displacement decoding link of FIG. 2( b ).

[0237] See 15, Fig.15 A schematic block diagram of the implementation process of the adaptive wavelet transform solution provided by the embodiment of the present invention at the decoding end for shift decoding; Fig.15 The decoding end shown is Figure 5Corresponding to the encoding end shown, at the decoding end, step Sd1 decodes the displacement information code stream through a decoding method corresponding to the encoding method, and can also decode the auxiliary information code stream. Step Sd2 performs an inverse quantization operation on the obtained data; step Sd3 performs an inverse wavelet transform on the inverse quantized data. This link can restore the received wavelet transform coefficients to displacement data based on the decoding results of the auxiliary information code stream, such as transformation parameters. The specific recovery step is the inverse operation of the adaptive wavelet transform at the encoding end. Finally, the recovered data is input into the post-processing module to obtain the final result. The post-processing module is the reverse implementation of the pre-processing module at the encoding end, such as operations such as inverse conversion of the coordinate system. The following is a specific description of the three-dimensional grid decoding process of an embodiment of the present invention in conjunction with Figure 2(b).

[0238] Auxiliary information decoding

[0239] The auxiliary information includes the type of static grid encoder, the type of video encoder, grid subdivision scheme, grid iteration number, geometric displacement vector transformation scheme and parameters, coefficient arrangement scheme, color conversion scheme, etc. For the grid part encoded in inter-frame coding mode, it also includes the corresponding reference frame list, etc. It can guide the decoder to perform correct decoding, and the information is represented by the corresponding syntax parameters after decoding.

[0240] Base grid decoding

[0241] The decoding of the base grid can be divided into intra-frame mode and inter-frame mode. This module is only responsible for decoding the information representing the base grid. That is, for the intra-frame mode, the output of the module decoding is a three-dimensional grid, which contains geometric information and connection relationships, as well as possible attribute information and texture coordinate information. For the inter-frame mode, this module is responsible for decoding motion vectors and identifiers and possible intra-frame sub-grids, that is, the output information is motion vectors, identifiers, and intra-frame sub-grids.

[0242] ①Intra-frame mode

[0243] The decoding of the basic grid information in the intra-frame mode is also the decoding of the basic grid. This module uses the corresponding grid decoder to decode the basic grid through the static grid decoder type indicated by the auxiliary information.

[0244] ②Inter-frame mode

[0245] In inter-frame mode, it is necessary to decode motion vectors, identifiers, and possible intra-frame sub-grids. For intra-frame sub-grids, the grid decoder indicated by the auxiliary information can be used for decoding. The motion vector identifier is obtained by entropy decoding, and the motion vector is obtained by deprediction and dequantization after entropy decoding.

[0246] Segmentation

[0247] The subdivision module at the decoding end has the same subdivision operation as the encoding end, and the number of subdivision iterations of different areas of the current basic grid is indicated through auxiliary information.

[0248] Displacement video decoding

[0249] The displacement information bitstream is decoded according to the decoder type indicated in the auxiliary information, and the decoder may specifically be a video decoder. Note that this includes geometric displacement vector videos and possible displacement vector videos with different attributes.

[0250] Bit displacement decoding

[0251] That is, displacement vector decoding. This link is responsible for extracting the displacement vectors in the decoded displacement vector image in vertex order. First, restore them from the two-dimensional image in the corresponding order according to the arrangement scheme. Then perform inverse quantization, inverse transformation and other operations on them to restore the corresponding displacement vector.

[0252] See also Fig.16 As well as the understanding of the processing process of the encoding end in the previous text, Fig.16 It is a block diagram for realizing an adaptive lifting inverse iterative transformation of displacement data in an embodiment of the present invention.

[0253] The inverse transformation process is the inverse implementation of the adaptive wavelet transform at the encoding end, and the number of inverse lifting iterations in the inverse transformation process is the same as the number of subdivisions of the decoding basic grid. For each inverse lifting iteration:

[0254] The first is the update operation, which is defined as shown in formula (20):

[0255]

[0256] In formula (20), v * is the set of all vertices adjacent to the subdivided decoded basic mesh vertex v; Signal(v) is the displacement data corresponding to vertex v. The Signal(v) on the right and left sides of formula (20) are the data before and after the update, respectively; Signal(w) is the residual value corresponding to the adjacent point.

[0257] The next step is the prediction operation. This step uses the same midpoint prediction method as the encoding end, and its definition is shown in formula (21):

[0258]

[0259] In formula (21), v is the newly inserted midpoint on the edge (v1, v2), Signal(v) is the residual value corresponding to the decoded basic mesh vertex v after subdivision, and the Signal(v) on the right and left sides of formula (21) respectively represent the data before and after prediction; Signal(v1) and Signal(v2) are the displacement vectors corresponding to the decoded basic mesh vertices v1 and (v2) after subdivision, respectively.

[0260] For updating the weight U adp , according to the calculation criteria used by the encoding end, there are three ways to obtain it:

[0261] ① Based on the first-order moment invariance criterion

[0262] The adaptive determination of update weights in the wavelet transform process includes:

[0263] The adaptive inverse wavelet transform method based on the first-order moment invariance criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

[0264] Specifically, the adaptive update weights of this method have been transmitted from the encoder to the decoder through the transformation parameters, so the update weights of the decoder can be directly obtained from the received transformation parameters.

[0265] ② Based on the minimum reconstruction error criterion

[0266] The adaptive determination of update weights in the wavelet transform process includes:

[0267] The adaptive inverse wavelet transform method based on the minimum reconstruction error criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

[0268] Specifically, the adaptive update weights of this method have been transmitted from the encoder to the decoder through the transformation parameters. Therefore, for this solution, the update weights of the decoder can be directly obtained from the received transformation parameters.

[0269] ③ Based on grid connectivity relationship

[0270] The adaptive determination of update weights in the wavelet transform process includes:

[0271] The connectivity relationship information of the subdivided decoded basic grid is used to determine the update weights in the wavelet transform process.

[0272] Specifically, the adaptive update weights of this method need to be obtained by using the connectivity relationship information of the subdivided decoding basic grid and using the same calculation method as the encoding end to calculate the weights.

[0273] Reconstructing the deformed mesh

[0274] After the base mesh and displacement vector are decoded and reconstructed, the deformed mesh is reconstructed based on the two parts. The corresponding displacement vector is added to each vertex of the subdivided mesh, as shown in formula (22):

[0275] deformedmesh[i]·v[k]=subdivmesh[i]·v[k]+displacement[k] (22);

[0276] Where subdivmesh[i]·v[k] is the geometric coordinates of the kth vertex after the base mesh is subdivided in the current frame (index is i), displacement[k] is the spatial displacement vector corresponding to the kth vertex, and deformedmesh[i]·v[k] is the geometric coordinates of the kth vertex after subdivision and deformation in the current frame.

[0277] The attribute displacement vector is applied to the corresponding attribute value in the same way. Taking texture coordinates as an example, as shown in formula (23):

[0278] deformedmesh[i]·vt[k]=subdivmesh[i]·vt[k]+attdisplacement[k] (23);

[0279] Among them, subdivmesh[i]·vt[k] represents the k-th texture coordinate after the base mesh is subdivided in the current frame, attdisplacement[k] represents the k-th value of the attribute displacement vector, and deformedmesh[i]·vt[k] represents the value of the k-th texture coordinate after subdivision and deformation in the current frame.

[0280] Property Graph Decoding

[0281] In Figure 2(b), texture map decoding is used as an example. Attribute map decoding is responsible for decoding the attribute map bitstream. The attribute map is decoded using the video decoder indicated in the auxiliary information. The attribute map is converted to an optional color space to obtain an image format consistent with the input attribute map of the encoder, and the attribute map of the final decoded output is obtained. For multiple attribute map bitstreams, after decoding, each attribute map is matched with the corresponding texture coordinates according to the identifier.

[0282] After each link is processed, the deformed mesh and the corresponding attribute map reconstructed by the decoding end are finally obtained. Subsequent applications use the reconstructed deformed mesh and attribute map as input for processing.

[0283] The embodiment of the present invention provides a three-dimensional mesh decoding method based on adaptive wavelet transform. For the displacement decoding link, the acquired displacement information code stream and auxiliary information code stream are first decoded; secondly, the decoding result of the displacement information code stream is inversely quantized to obtain an inverse quantization result; then based on the decoding result of the auxiliary information code stream, the update weight in the wavelet transform process is adaptively determined; based on the determined update weight, the inverse quantization result is inversely transformed by wavelet to obtain the vertex displacement data of the subdivided mesh; finally, the vertex displacement data of the subdivided mesh is used to obtain the reconstructed mesh. The present invention can adaptively determine the update weight in the wavelet transform process based on the decoding result of the auxiliary information code stream, and can ensure that the low-frequency signal has a high degree of approximation with the original signal, thereby improving the accuracy of the prediction in the next lifting iteration process, and further improving the coding efficiency of the displacement data. It can be seen that the present invention can optimize the three-dimensional mesh displacement decoding method in VDMC, and combine the optimization method with the V3C standard to improve decoding efficiency and accuracy.

[0284] In a fourth aspect, corresponding to the method embodiment described in the third aspect, an embodiment of the present invention further provides a three-dimensional grid decoding device based on adaptive wavelet transform, such as Fig.13 As shown, the device comprises:

[0285] A decoding module, used for decoding the acquired displacement information code stream and auxiliary information code stream;

[0286] The dequantization module is used to dequantize the decoding result of the displacement information code stream to obtain a dequantization result;

[0287] The inverse wavelet transform module is used to adaptively determine the update weights in the wavelet transform process based on the decoding results of the auxiliary information code stream; based on the determined update weights, the inverse wavelet transform is performed on the inverse quantization results to obtain vertex displacement data of the subdivided mesh; wherein the vertex displacement data of the subdivided mesh represents the displacement vectors from the vertices of the decoded basic mesh after subdivision to the mesh to be reconstructed;

[0288] The mesh reconstruction module is used to obtain a reconstructed mesh using the vertex displacement data of the subdivided mesh.

[0289] Optionally, when the wavelet inverse transform module adaptively determines the update weight in the wavelet transform process, it is specifically used to:

[0290] The adaptive inverse wavelet transform method based on the first-order moment invariance criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

[0291] Optionally, when the wavelet inverse transform module adaptively determines the update weight in the wavelet transform process, it is specifically used to:

[0292] The adaptive inverse wavelet transform method based on the minimum reconstruction error criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

[0293] Optionally, when the wavelet inverse transform module adaptively determines the update weight in the wavelet transform process, it is specifically used to:

[0294] The connectivity relationship information of the subdivided decoded basic grid is used to determine the update weights in the wavelet transform process.

[0295] For the processing process of the above modules, please refer to the relevant description of the three-dimensional grid decoding method based on adaptive wavelet transform provided in the third aspect, which will not be described in detail here.

[0296] It should be noted that, in the process of processing adaptive wavelet transform or inverse transform at the encoding end and the decoding end of the embodiment of the present invention, the meaning of specific parameters in similar formulas should be understood in combination with the processing link and the interpretation of the formula.

[0297] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A three-dimensional grid coding method based on adaptive wavelet transform, characterized in that: include: Acquire vertex displacement data; wherein the vertex displacement data is obtained according to the displacement information of the vertices in the subdivided mesh after the base mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded; Adopting an adaptive wavelet transform method to perform wavelet transform on the vertex displacement data, and using the transformed displacement vector as a wavelet transform coefficient; wherein the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data; quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients; The quantized wavelet transform coefficients are encoded to obtain a displacement information code stream.

2. The three-dimensional grid coding method based on adaptive wavelet transform according to claim 1 is characterized in that: The adaptive wavelet transform method includes an adaptive wavelet transform method based on a first-order moment invariance criterion.

3. The three-dimensional grid coding method based on adaptive wavelet transform according to claim 1 is characterized in that: The adaptive wavelet transform method includes an adaptive wavelet transform method based on a minimum reconstruction error criterion.

4. The three-dimensional mesh coding method based on adaptive wavelet transform according to claim 1, characterized in that: The adaptive wavelet transform method includes an adaptive wavelet transform method based on mesh connectivity relationship information.

5. A three-dimensional grid coding device based on adaptive wavelet transform, characterized in that: include: A data acquisition module, used to acquire vertex displacement data; wherein the vertex displacement data is obtained according to the displacement information of the mesh vertices after the basic mesh is subdivided; the displacement information is the displacement vector from each vertex in the subdivided mesh to the mesh to be encoded; A wavelet transform module, used to perform wavelet transform on the vertex displacement data by using an adaptive wavelet transform method, and use the transformed displacement vector as a wavelet transform coefficient; wherein the adaptive wavelet transform method adaptively adjusts the update weight of the wavelet transform according to the distribution characteristics of the vertex displacement data; A quantization module, used for quantizing the wavelet transform coefficients to obtain quantized wavelet transform coefficients; The encoding module is used to encode the quantized wavelet transform coefficients to obtain a displacement information code stream.

6. A three-dimensional grid decoding method based on adaptive wavelet transform, characterized in that: include: Decoding the acquired displacement information code stream and auxiliary information code stream; De-quantizing the decoding result of the displacement information code stream to obtain a de-quantization result; Based on the decoding result of the auxiliary information code stream, the update weights in the wavelet transform process are adaptively determined; Based on the determined update weights, the inverse quantization result is subjected to an inverse wavelet transform to obtain vertex displacement data of the subdivided mesh; wherein the vertex displacement data of the subdivided mesh represents the displacement vectors from the vertices of the decoded basic mesh after subdivision to the mesh to be reconstructed; The reconstructed mesh is obtained by using the vertex displacement data of the subdivided mesh.

7. The three-dimensional grid decoding method based on adaptive wavelet transform according to claim 6, characterized in that: The adaptive determination of update weights in the wavelet transform process includes: The adaptive inverse wavelet transform method based on the first-order moment invariance criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

8. The three-dimensional grid decoding method based on adaptive wavelet transform according to claim 6, characterized in that: The adaptive determination of update weights in the wavelet transform process includes: The adaptive inverse wavelet transform method based on the minimum reconstruction error criterion utilizes the transform parameters of the wavelet transform process in the decoding result of the auxiliary information code stream to determine the update weight in the wavelet transform process.

9. The three-dimensional grid decoding method based on adaptive wavelet transform according to claim 6, characterized in that: The adaptive determination of update weights in the wavelet transform process includes: The connectivity relationship information of the subdivided decoded basic grid is used to determine the update weights in the wavelet transform process.

10. A three-dimensional grid decoding device based on adaptive wavelet transform, characterized in that: include: A decoding module, used for decoding the acquired displacement information code stream and auxiliary information code stream; The dequantization module is used to dequantize the decoding result of the displacement information code stream to obtain a dequantization result; The inverse wavelet transform module is used to adaptively determine the update weights in the wavelet transform process based on the decoding result of the auxiliary information code stream; Based on the determined update weights, the inverse quantization result is subjected to an inverse wavelet transform to obtain vertex displacement data of the subdivided mesh; wherein the vertex displacement data of the subdivided mesh represents the displacement vectors from the vertices of the decoded basic mesh after subdivision to the mesh to be reconstructed; The mesh reconstruction module is used to obtain a reconstructed mesh using the vertex displacement data of the subdivided mesh.

Images