Coding method, decoding method, code stream, coder, decoder, and storage medium

Abstract

Disclosed in the embodiments of the present application are a coding method, a decoding method, a code stream, a coder, a decoder, and a storage medium, which can improve decoding accuracy, thereby improving decoding performance. The decoding method comprises: parsing a code stream to determine a first base mesh and texture coordinate information of the first base mesh; on the basis of the first base mesh, performing mesh subdivision and deformation to determine a reconstructed deformed mesh; and on the basis of the texture coordinate information of the first base mesh and the reconstructed deformed mesh, determining texture coordinate information of the reconstructed deformed mesh.

Description

Coding and decoding method, code stream, encoder, decoder and storage medium Technical Field

[0001] The present application relates to the field of video coding and decoding technology, and in particular to a coding and decoding method, a bit stream, an encoder, a decoder, and a storage medium. Background Art

[0002] Video-based Dynamic Mesh Coding (VDMC) is a standard for compressing 3D meshes. It primarily compresses 3D meshes by leveraging the existing Visual Volumetric Video-based Coding (V3C) standard. However, due to the encoding of connectivity information in 3D meshes, the specific encoding process differs slightly from V3C.

[0003] However, during the encoding and decoding process of 3D meshes, the texture coordinates generated by mesh parameterization do not match the reconstructed deformed mesh, which reduces the decoding accuracy and affects the decoding performance.

[0004] Summary of the Invention

[0005] The present application provides a coding and decoding method, a code stream, an encoder, a decoder and a storage medium, which can improve decoding accuracy and decoding performance.

[0006] The technical solution of this application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Parsing the code stream to determine the first basic grid and texture coordinate information of the first basic grid;

[0009] Performing mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

[0010] The texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh.

[0011] In a second aspect, an embodiment of the present application provides an encoding method, applied to an encoder, the method comprising:

[0012] Determine the base grid corresponding to the current image;

[0013] Determining texture coordinate information based on the base mesh;

[0014] The basic grid and the texture coordinate information are encoded, and the obtained encoding bits are written into a bitstream.

[0015] In a third aspect, an embodiment of the present application provides a code stream, which is generated by bit encoding based on information to be encoded; wherein the information to be encoded includes at least one of the following:

[0016] Base mesh and texture coordinate information.

[0017] In a fourth aspect, an embodiment of the present application provides an encoder, comprising

[0018] The grid information determining portion is configured to determine a base grid corresponding to the current image; and determine texture coordinate information based on the base grid;

[0019] The encoding part is configured to encode the basic grid and the texture coordinate information, and write the obtained encoding bits into a bit stream.

[0020] In a fifth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor, wherein:

[0021] a first memory for storing a computer program capable of running on the first processor;

[0022] The first processor is configured to execute the method according to the second aspect when running a computer program.

[0023] In a sixth aspect, an embodiment of the present application provides a decoder, comprising

[0024] A parsing part configured to parse the code stream to determine the first basic grid and texture coordinate information of the first basic grid;

[0025] A subdivision and deformation part is configured to perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

[0026] The coordinate determining part is configured to determine the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh.

[0027] In a seventh aspect, an embodiment of the present application provides a decoder, comprising a second memory and a second processor, wherein:

[0028] a second memory for storing a computer program capable of running on the second processor;

[0029] The second processor is configured to execute the method according to the first aspect when running a computer program.

[0030] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed, implements the method described in the first aspect or the method described in the second aspect.

[0031] The embodiments of the present application provide a coding and decoding method, a bitstream, an encoder, a decoder, and a storage medium. At the decoding end, the bitstream is parsed to determine a first base mesh and its texture coordinate information; mesh subdivision and deformation are performed based on the first base mesh to determine a reconstructed deformed mesh; and texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh. In this way, the texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh obtained through subdivision and deformation. This reduces the distortion of vertex texture coordinates, improves the accuracy of the texture coordinates, and improves the matching degree between the texture coordinates and the reconstructed deformed mesh, thereby improving the quality of the reconstructed deformed mesh and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1A is a schematic diagram of a three-dimensional grid image 1;

[0033] FIG1B is a partially enlarged schematic diagram of a three-dimensional grid image;

[0034] Figure 2 is a schematic diagram of the connection method of the three-dimensional grid;

[0035] FIG3 is a schematic diagram of a 3D mesh coding framework based on subdivision deformation;

[0036] FIG4 is a schematic diagram of an implementation process of a grid simplification operation provided in an embodiment of the present application;

[0037] FIG5 is a schematic diagram of an implementation process of displacement vector calculation provided by an embodiment of the present application;

[0038] FIG6 is a schematic diagram of an implementation process of a subdivision operation provided in an embodiment of the present application;

[0039] FIG7 is a schematic diagram of an implementation process of texture image conversion provided by an embodiment of the present application;

[0040] FIG8 is a schematic diagram of a 3D mesh decoding framework based on subdivision deformation;

[0041] FIG9 is a schematic diagram of the position change of the base mesh vertices provided in an embodiment of the present application;

[0042] FIG10 is a schematic diagram showing a comparison of mesh shapes between a basic subdivision mesh and a reconstructed deformed mesh provided by an embodiment of the present application;

[0043] FIG11 is a schematic diagram of a mesh architecture of a codec provided in an embodiment of the present application;

[0044] FIG12 is a flowchart diagram of a decoding method according to an embodiment of the present application;

[0045] FIG13 is a schematic diagram of a three-dimensional grid decoding framework provided in an embodiment of the present application;

[0046] FIG14 is a second schematic diagram of a three-dimensional grid decoding framework provided in an embodiment of the present application;

[0047] FIG15 is a flowchart diagram of a decoding method provided in an embodiment of the present application;

[0048] FIG16 is a flowchart diagram 1 of a decoding method provided in an embodiment of the present application;

[0049] FIG17 is an optional schematic diagram of at least one candidate location point provided in an embodiment of the present application;

[0050] FIG18 is a schematic diagram of an optional search range provided in an embodiment of the present application;

[0051] FIG19 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0052] FIG20 is a schematic diagram of a three-dimensional grid coding framework provided in an embodiment of the present application;

[0053] FIG21 is a second schematic diagram of a three-dimensional grid coding framework provided in an embodiment of the present application;

[0054] FIG22 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0055] FIG23 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0056] FIG24 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0057] FIG25 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

[0058] FIG26 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0062] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0063] It should be noted that it is possible to decode and synthesize different data format bitstreams within the same video scene. These can include at least image format, point cloud format, and mesh format. In this way, real-time immersive video interaction services can be provided for multiple data formats (e.g., mesh, point cloud, image, etc.) from different sources.

[0064] In embodiments of the present application, the data format-based approach allows for independent processing at the bitstream level of the data format. This means that, similar to tiles or slices in video encoding, different data formats in this scenario can be encoded independently, enabling independent encoding and decoding based on the data format.

[0065] Generally speaking, three-dimensional animation content adopts a keyframe-based representation method, that is, each frame is a static mesh. Static meshes at different times have the same topological structure and different geometric structures. However, the amount of data of the three-dimensional dynamic mesh represented based on keyframes is particularly large, so how to effectively store, transmit and draw it has become a problem faced by the development of three-dimensional dynamic meshes. In addition, the spatial scalability of the mesh needs to be supported for different user terminals (computers, notebooks, portable devices, mobile phones); different mesh bandwidths (broadband, narrowband, wireless) need to support the quality scalability of the mesh. Therefore, three-dimensional dynamic mesh compression is a very critical issue. Among them, "one frame" can be understood as an image. For example, a keyframe can be understood as a key image in a three-dimensional animation.

[0066] A 3D mesh is the surface of a 3D object composed of countless polygons in space. Polygons are composed of vertices and edges. Figure 1A shows a 3D mesh image, and Figure 1B shows a partially enlarged schematic diagram of the 3D mesh image. Figures 1A and 1B show that the mesh surface is composed of closed polygons.

[0067] A two-dimensional image has information expressed at every pixel point and is distributed regularly, so there is no need to record its position information separately. However, the distribution of vertices in the mesh in three-dimensional space is random and irregular, and the way polygons are formed requires additional regulations. Therefore, it is necessary to record the position of each vertex in space and the connection information of each polygon to fully express a mesh image. As shown in Figure 2, the same number of vertices and vertex positions will form completely different surfaces due to different connection methods.

[0068] In addition to the above information, since 3D mesh images are usually encoded using existing 2D image / video encoding methods, the 3D mesh needs to be converted from 3D space to 2D images. The UV coordinates define this conversion process.

[0069] Similar to 2D images, each location in the acquisition process may have corresponding attribute information, usually RGB color values, which reflect the color of the object. For 3D meshes, in addition to color, each vertex's attribute information also commonly includes reflectance values, which reflect the surface material of the object. The attribute information of a 3D mesh is stored in a 2D image, and its mapping from 2D to 3D is specified by UV coordinates. 3D mesh data typically includes 3D geometric coordinate information (x, y, z), geometric connection relationships, UV coordinates, and attribute maps.

[0070] It should also be noted that Video-based Dynamic Mesh Coding (VDMC) is a standard developed by the Moving Pictures Experts Group (MPEG) for compressing three-dimensional meshes. Its main idea is to compress three-dimensional meshes by leveraging the existing Visual Volumetric Video-based Coding (V3C) standard. Since three-dimensional meshes contain connection information that needs to be encoded, its specific encoding process is slightly different from V3C. The syntax, semantics, and decoding operations of the V3C standard decoding end need to be expanded to support the decoding and reconstruction of three-dimensional meshes.

[0071] FIG3 is a schematic diagram of a three-dimensional grid encoding framework, and FIG8 is a schematic diagram of a three-dimensional grid decoding framework.

[0072] Currently, the overall framework of a VDMC encoder is shown in Figure 3. For the input mesh, the base mesh and the corresponding subdivided deformed mesh are first obtained through the base mesh generation module. The input mesh is first simplified by the simplification module. Then, new texture coordinates are generated for the simplified mesh through mesh parameterization. The parameterized mesh is then subdivided and deformed. This involves inserting new vertices on the mesh according to a specific subdivision method and calculating the distance from the subdivided mesh vertex to the nearest neighbor of the input mesh, which is called the displacement. This results in the corresponding subdivided deformed mesh. Subsequently, the parameterized mesh is adjusted based on the displacement information, that is, the vertex positions of the mesh before the subdivision deformation. The adjusted mesh is called the base mesh and sent to the base mesh encoding module for compression using the existing mesh encoder. In inter-frame mode, motion vectors can also be generated for each vertex of the base mesh based on the reference frame. The base mesh module only needs to compress the motion vectors.

[0073] After encoding, the base mesh is reconstructed, and displacements are calculated based on the reconstructed base mesh and the subdivided deformed mesh obtained during the base mesh generation phase. The displacement information is then transformed and quantized, and then encoded using a video encoder or entropy encoder. The reconstructed displacement information is then applied to the subdivided base mesh to produce a reconstructed subdivided deformed mesh. This mesh, along with the original input mesh and its corresponding texture map, is input into a texture map conversion module to produce a texture map corresponding to the reconstructed mesh. This texture map is also encoded using a video encoder.

[0074] The following describes the main modules in the encoder.

[0075] 1. Basic mesh generation module:

[0076] The basic mesh generation module includes the mesh simplification module, the mesh parameterization module and the subdivision deformation module. Each module is introduced below.

[0077] (1) Mesh simplification module:

[0078] Mesh simplification involves simplifying the input mesh into a base mesh with relatively few points and faces, while preserving the original mesh's shape as much as possible. The key to mesh simplification lies in the simplification operation and the corresponding merge error energy function. A possible mesh simplification operation, shown in Figure 4, merges the vertices at both ends of an edge into a single vertex and deletes the connection between them. This process is repeated throughout the mesh according to a specific rule to reduce the number of faces and vertices to the target value.

[0079] (2) Grid parameterization module:

[0080] The mesh parameterization module is used to generate the corresponding texture coordinates for the mesh. Currently, there are many algorithms for parameterizing meshes, such as the Isochart algorithm and the orthogonal projection algorithm. In this encoding framework, both of the above schemes can be used to parameterize the reconstructed base mesh.

[0081] The Isochart algorithm uses spectral analysis to implement stretch-driven 3D mesh parameterization, performing UV unwrapping, slicing, and packing the 3D mesh into a 2D texture domain. A stretch threshold is set, and the algorithm is outlined as follows:

[0082] a) Compute surface spectrum analysis to provide an initial parameterization

[0083] b) Perform stretch optimization iterations

[0084] c) If the stretch of this derived parameterization is less than a threshold, stop

[0085] d) Perform surface spectral clustering to divide the surface into charts

[0086] e) Use graph cut algorithm to optimize chart boundaries

[0087] f) Iteratively split the charts until the stretching criteria are met

[0088] Among them, the orthogonal projection algorithm may include the orthoAtlas algorithm, which is a projection-based mesh parameterization method that generates texture coordinates for the mesh through orthogonal projection. Its main process includes:

[0089] a) Calculate mesh properties, including the neighboring faces of each face and the area and normal vector of each face;

[0090] b) Determine the projection plane of each face based on the normal vector;

[0091] c) Start clustering all faces according to the projection plane to form a connected region, first selecting the starting face of the cluster;

[0092] d) Iterating from the starting face, determining whether adjacent faces of the face added to the connected region can be added to the connected region;

[0093] e) After each connected region is iterated, multiple connected regions are obtained;

[0094] f) determining whether to merge adjacent connected regions based on the error metric;

[0095] g) Check whether there are overlapping areas during projection, remove the overlapping areas and regenerate connected areas;

[0096] h) Arrange all the projected regions into a two-dimensional image.

[0097] (3) Subdivision deformation module:

[0098] The basic idea behind the subdivision deformation module is illustrated in Figure 5. The same concept is applied to the input 3D mesh to generate displacements. In Figure 5, the input 2D curve (represented by a 2D polyline), referred to as the "original" curve, is first downsampled to generate a base curve / polyline, referred to as the "simplified" curve. The subdivision scheme is then applied to the resulting simplified polyline to generate the "subdivided" curve. The subdivided polyline is then deformed to obtain a better approximation of the original curve. This involves calculating a displacement vector for each vertex of the subdivided mesh, such that the shape of the displaced curve approximates the shape of the original curve as closely as possible. These displacement vectors are the displacement information output by the module.

[0099] The subdivision deformation module takes the parameterized mesh as input and first subdivides it. Any subdivision scheme can be chosen. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four subtriangles at each iteration, as shown in Figure 6. New vertices are introduced in the middle of each edge. The subdivision of geometric information and attribute information is performed independently, as the connection between geometric information and attribute information is usually different.

[0100] For the subdivided mesh, we find the nearest neighbor of each point on the original input mesh (including points on the original mesh surface). This can be accelerated using data structures such as kdTree. The displacement of the geometric coordinates of each vertex on the subdivided mesh is obtained by calculating the distance between the geometric coordinates of each vertex on the subdivided mesh and its nearest neighbor on the original input mesh. This module can generate a base mesh and the corresponding subdivided, deformed mesh.

[0101] 2. Basic grid compression module:

[0102] There are three different compression modes for base mesh compression, namely intra mode, inter mode and skip mode. In intra mode, the input of the base mesh compression module is a three-dimensional mesh, which contains geometric coordinates, connection relationships and attribute information associated with vertices, and is encoded through the existing static mesh encoder. In inter mode, the base mesh first uses RDO (rate-distortion trade-off) to decide whether to use skip mode. If skip mode is used, the reference mesh in the mesh buffer is directly selected as the base mesh of the current image. If skip mode is not used, the motion vector of the vertex is calculated based on the reference image in the mesh buffer and the current image, and the corresponding motion vector is encoded. After encoding, the encoded mesh needs to be reconstructed so that it can be provided to subsequent modules for processing.

[0103] 3. Displacement generation module and displacement encoding module:

[0104] The base mesh compression module compresses and reconstructs the base mesh. The displacement generation module calculates the corresponding vertex displacements using the reconstructed base mesh and the subdivided deformed mesh obtained in the subdivision deformation step. Furthermore, due to the existence of subdivision deformation, the displacement encoding module can encode displacements using a LoD (Level of Detail) approach, encoding each level independently to support decoding at the desired level. After obtaining the layered displacement coefficients, each level is independently encoded.

[0105] Displacement encoding can be implemented in a variety of ways. One approach involves transforming the coordinate system of the displacements. Specifically, the coordinate system of each vertex's displacement is converted to a coordinate system constructed using the vertex's normal vector and two components tangent to that normal vector. The displacements are then transformed using techniques such as wavelet transforms. The transformed coefficients are quantized and arranged in the image in scan order, and video encoding is applied to the image. Alternatively, entropy coding can be used to directly encode the generated or processed displacements.

[0106] 4. Deformed Mesh Reconstruction Module:

[0107] Because the quantization process in the displacement encoding stage causes some loss in displacement, the displacement must be reconstructed on the encoder side to ensure consistency with the decoder side. After obtaining the reconstructed displacement, the deformed mesh reconstruction module subdivides the reconstructed base mesh and generates a reconstructed subdivided deformed mesh based on the corresponding reconstructed displacement.

[0108] 5. Texture map conversion module:

[0109] The texture map conversion module first performs texture map conversion based on the input original mesh, the input original texture map, and the reconstructed subdivided deformed mesh, as shown in Figure 7. The steps of texture map conversion are as follows:

[0110] (1) Calculate the texture coordinates of each pixel on the texture map to be generated, for example, the texture coordinates corresponding to pixel A(i,j) are P(u,v).

[0111] (2) Determine whether the texture coordinate is within a certain triangular face after the parameterization of the subdivided deformed mesh.

[0112] (3) If the texture coordinate does not belong to any triangle, the pixel is marked as an empty pixel and can be filled with a filling algorithm.

[0113] (4) If the texture coordinate belongs to a triangle, then

[0114] a) Mark the pixel as filled.

[0115] b) Calculate the center of gravity coordinates of the texture in the current triangle according to the texture coordinates.

[0116] c) Based on the barycentric coordinates and the corresponding triangular face, the two-dimensional texture coordinates are mapped to three-dimensional geometric coordinates, that is, mapped to the points on the subdivided deformed mesh corresponding to the texture coordinates, as shown by M(x, y, z) in the figure.

[0117] d) Find the point on the input original grid that is closest to the three-dimensional coordinate, as shown by M'(x,y,z) in the figure.

[0118] e) Calculate the barycentric coordinates of the three-dimensional coordinates according to the triangle face on which they are located and map them to two dimensions to calculate their texture coordinates, namely P'(u',v').

[0119] f) Sample the input original texture map using the texture coordinates to obtain the value A'(i', j') of the corresponding pixel position.

[0120] g) Assign the value to the corresponding pixel A(i,j) on the texture map to be generated.

[0121] For empty pixels, existing filling algorithms (such as Push-Pull algorithm) can be used to fill these empty pixels.

[0122] 6. Texture compression module:

[0123] The texture image compression module performs optional color space conversion and other processing on the converted texture image, and then encodes it using a video encoder.

[0124] Currently, the overall framework of a VDMC decoder is shown in Figure 8. For the received bitstream, the decoder first demultiplexes the various bitstreams to obtain the basic grid bitstream, the displacement video bitstream, and the texture map video bitstream. The basic grid bitstream is decoded using the grid decoder corresponding to the encoder to obtain the basic grid. The displacement video bitstream and the texture map video bitstream are decoded by the video decoder. For the displacement portion, after video decoding, the displacement needs to be extracted from the image through the displacement decoding module, and dequantized and inverse transformed. This displacement is then applied to the subdivided basic grid to obtain the deformed grid reconstructed by the decoder. After decoding, the texture map is the texture map corresponding to the reconstructed deformed grid. Subsequent application or rendering modules process the reconstructed deformed grid and the decoded texture map as input.

[0125] The following describes the main modules in the decoder.

[0126] 1. Basic grid decoding module:

[0127] The base grid decoding module decodes the input bitstream according to its grid encoding format. If the mode is intra, it uses the corresponding static grid decoder indicated by the auxiliary information. If the mode is inter, it decodes the corresponding motion vectors and reconstructs the corresponding base grid based on the reference image in the grid buffer. If the mode is skip, it directly uses the reference image in the grid buffer as the base grid for the current image.

[0128] 2. Displacement decoding module and displacement reconstruction module.

[0129] The displacement video stream is decoded by the displacement decoding module, and the corresponding level of decoding is performed according to the LoD level required by the decoder. If the encoder compresses the displacement through video encoding, the decoder decodes it through the corresponding video decoder and uses the displacement reconstruction module to restore it from the 2D image in the corresponding order according to the arrangement scheme. Then, inverse transformation and dequantization are performed to restore the displacement to the same level as the encoder. If entropy encoding is used, direct entropy decoding can be performed.

[0130] 3. Deformed Mesh Reconstruction Module:

[0131] After the basic mesh and the displacement decoding and reconstruction of the corresponding level are completed, the subdivided deformed mesh is reconstructed based on these two parts. If the auxiliary information indicates that the displacement encoding process uses a reference image as a reference, that is, the displacement of the vertices in the area matching the reference image in the decoded displacement is the residual of the displacement of the current frame vertex relative to the reference image vertex, then the displacement of the vertices in the current image matching area is first restored through the vertices and residuals of the subdivided deformed mesh reconstructed from the reference image, and then subsequent reconstruction steps are performed. That is, the reconstructed basic mesh is subdivided, consistent with the subdivision method of the encoding end. The corresponding displacement is added to each vertex of the subdivided mesh. The texture coordinates of the reconstructed mesh are obtained by midpoint subdivision interpolation consistent with the encoding end.

[0132] However, in the above encoding and decoding scheme, mesh parameterization is used to generate texture coordinates for the simplified mesh. However, due to the geometric coordinates of the parameterized mesh undergoing subdivision deformation and quantization during the encoding and decoding process, the vertex geometric coordinates will change as shown in Figure 9. Therefore, the generated texture coordinates will not match the reconstructed deformed mesh.

[0133] Furthermore, during the subsequent reconstruction of the deformed mesh, the vertex geometric coordinates need to be displaced after subdivision, resulting in a certain degree of deformation. However, their texture coordinates are obtained only through fixed subdivision interpolation and may not match the deformed geometric coordinates. Figure 10 shows the changes in mesh shape at each level, taking subdivision three times as an example.

[0134] In summary, the texture coordinates generated by the current mesh parameterization do not match the reconstructed deformed mesh, which reduces the decoding accuracy and thus the decoding performance. Therefore, the texture coordinates of the reconstructed deformed mesh need to be adjusted to match the deformed geometric coordinates.

[0135] The present invention provides a coding and decoding method that measures the parametric distortion of the texture coordinates of vertices in a reconstructed deformed mesh and reduces this distortion through a compensation algorithm. The parametric quality compensation algorithm proposed in the present invention can be applied to a video-based dynamic mesh coding framework to address parametric distortion caused by changes in the positions of mesh geometric vertices, thereby improving the quality of the reconstructed deformed mesh and, in turn, decoding accuracy and performance.

[0136] The embodiment of the present application also provides a grid architecture of a codec system including a decoding method and an encoding method. FIG11 is a schematic diagram of a codec grid architecture provided by the embodiment of the present application. As shown in FIG11 , the grid architecture includes one or more electronic devices 13 to 1N and a communication grid 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication grid 01. During implementation, the electronic device can be various types of devices with codec functions. For example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensor device, a server, etc., which are not specifically limited here.

[0137] Here, the decoder or encoder described in the embodiment of the present application can be the above-mentioned electronic device.

[0138] The following describes in detail the various embodiments of the present application with reference to the accompanying drawings.

[0139] Referring to Figure 12, it shows a schematic flow chart of a decoding method provided by an embodiment of the present application. As shown in Figure 12, the method may include:

[0140] S101: Parse a bitstream to determine a first basic mesh and texture coordinate information of the first basic mesh.

[0141] In one case of S101, the bitstream includes a first base mesh bitstream. The first base mesh bitstream is determined by encoding the first base mesh containing texture coordinate information. The decoder can directly parse the bitstream to obtain the first base mesh containing texture coordinate information, thereby determining the first base mesh and the texture coordinate information of the first base mesh. Exemplarily, the decoder demultiplexes the bitstream to determine the bitstream of the base mesh, and then parses the bitstream of the base mesh to determine the first base mesh containing texture coordinate information.

[0142] In some embodiments, the base grid decoding module performs decoding based on the grid coding format of the input code stream. If the mode is intra-frame, the corresponding static grid decoder indicated by the auxiliary information is used for decoding; if the mode is inter-frame, the corresponding motion vector is decoded and the first base grid is reconstructed based on the reference image in the grid buffer; if the mode is skip, the corresponding reference image in the grid buffer is directly used as the first base grid.

[0143] In some embodiments, the displacement code stream is decoded by a displacement decoder, and decoding is performed at the corresponding level of LoD required by the decoder. If the encoder compresses the displacements through video encoding, the decoder decodes them through the corresponding video decoder and restores them from the two-dimensional image in the corresponding order according to the arrangement scheme. Then, inverse transformation and dequantization are performed on them to restore the displacements consistent with the encoder. If entropy encoding is used, entropy decoding can be performed directly.

[0144] For another case of S101, the code stream includes a second basic grid code stream and a texture coordinate code stream. The second basic network code stream is determined by encoding the second basic grid that does not contain texture coordinate information. The decoder demultiplexes the code stream to determine the second basic grid code stream and the texture coordinate code stream. The decoder parses the second basic grid code stream to determine the second basic grid. Here, the process of parsing the second basic grid is consistent with the above-mentioned process of parsing the first basic grid, and will not be repeated here. The decoder parses the texture coordinate code stream to determine the texture coordinate information; based on the texture coordinate information, the second basic grid is reconstructed to determine the first basic grid and the texture coordinate information corresponding to the first basic grid.

[0145] In some embodiments, the second base mesh is reconstructed based on the texture coordinate information to determine the base mesh including the texture coordinate information; the base mesh including the texture coordinate information is cleaned to determine the first base mesh and the texture coordinate information of the first base mesh.

[0146] S102: Perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh.

[0147] S103 : Determine texture coordinate information of a reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the reconstructed deformed mesh.

[0148] In S102, the decoder also parses the bitstream to determine the displacement information corresponding to the first base mesh. The decoder performs mesh subdivision and deformation based on the first base mesh, determining a base subdivided mesh and a reconstructed deformed mesh. The reconstructed deformed mesh includes three-dimensional coordinate information, namely, the geometric coordinates of the vertices in the reconstructed deformed mesh. Initial texture coordinate information for the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the base subdivided mesh. Texture coordinate information for the reconstructed deformed mesh is then determined based on the initial texture coordinate information of the reconstructed deformed mesh.

[0149] The base subdivision mesh is determined by subdividing the first base mesh. The reconstructed deformed mesh is determined by superimposing displacement information on the vertices of the base subdivision mesh using displacement information parsed from the bitstream. The decoder interpolates the texture coordinate information of the first base mesh based on the base subdivision mesh corresponding to the first base mesh, and uses this as the initial texture coordinate information for the reconstructed deformed mesh. The texture coordinate information of the reconstructed deformed mesh includes the texture coordinates of each vertex in the reconstructed deformed mesh.

[0150] In some embodiments, after the first basic mesh and the displacement decoding and reconstruction of the corresponding level are completed, the subdivided deformed mesh is reconstructed based on the two parts, that is, the deformed mesh is reconstructed. If the auxiliary information indicates that the displacement encoding process uses a reference image as a reference, that is, the displacement of the vertices in the area matching the reference image in the decoded displacement is the residual of the displacement of the current image vertex relative to the reference image vertex, then the displacement of the vertices in the current image matching area is first restored through the vertices and residuals of the subdivided deformed mesh reconstructed from the reference image, and then subsequent reconstruction steps are performed. That is, the reconstructed basic mesh is subdivided, consistent with the subdivision method of the encoding end. The corresponding displacement is added to each vertex of the subdivided mesh. The texture coordinates of the reconstructed mesh are obtained by midpoint subdivision interpolation consistent with the encoding end.

[0151] As can be seen, when reconstructing the deformed mesh at the corresponding level, the geometric positions of the vertices in the reconstructed deformed mesh are obtained through midpoint subdivision interpolation and the corresponding displacement, while the corresponding texture coordinates are only interpolated through midpoint subdivision, resulting in a certain deviation from the geometric positions after the displacement. Therefore, the texture coordinates need to be compensated to match the reconstructed deformed mesh after the geometric deformation.

[0152] In some embodiments, based on the decoder framework shown in Figure 8, a decoder framework of an embodiment of the present application can be shown in Figure 13. After the decoding end obtains the code stream, it first demultiplexes the various code streams. The basic grid decoding module decodes the basic grid code stream to determine the basic grid and the texture coordinate information of the basic grid (that is, the texture coordinate information of the first basic grid and the first basic grid). Then, according to the level of detail required by the decoding end, the required displacement level is decoded and reconstructed, and then the reconstructed displacement is applied to the subdivided basic grid to reconstruct the deformed grid to obtain a reconstructed deformed grid with the level of detail required by the decoding end. The reconstructed deformed grids of different detail levels can be described as the reconstructed deformed grid of the L1 level, the reconstructed deformed grid of the L2 level, and the reconstructed deformed grid of the L3 level in Figure 12. Texture coordinate compensation is performed on the texture coordinates of the reconstructed deformed grid of one level, and the texture coordinates of the vertices in the reconstructed deformed grid of the current level are adjusted to reduce its parametric distortion, and finally the deformed grid output by the decoding end is obtained. The texture map corresponding to the grid is obtained by video decoding the texture map code stream.

[0153] In some embodiments, the texture coordinates of the reconstructed deformed mesh can be updated by a variety of compensation algorithms, and the reconstructed deformed mesh after geometric deformation. For example, the texture coordinates of the reconstructed deformed mesh can be updated directly according to the displacement reconstructed by decoding to compensate for the deviation between the geometric coordinates of the reconstructed deformed mesh with the displacement added due to only midpoint subdivision interpolation of the texture coordinates. Alternatively, since the three-dimensional space to the two-dimensional space is not equidistant, the parameterization process will cause distortion. Distortion can be measured in many ways, including the preservation of angles or areas, or how much parameter distance is stretched or shrunk on the surface, etc. According to the distortion measurement method, a corresponding compensation algorithm can be designed to compensate or update the texture coordinates of the reconstructed deformed mesh. The specific selection is made according to the actual situation, and the embodiments of the present application are not limited.

[0154] In some embodiments, based on the decoder framework shown in FIG7 , another decoder framework according to an embodiment of the present application can be shown in FIG14 . After the decoder receives the bitstream, it first demultiplexes the various bitstream components. The base grid decoding module decodes the base grid bitstream. The base grid decoding module decodes the base grid bitstream and determines a second base grid. Then, depending on whether the bitstream contains a texture coordinate bitstream, the module reconstructs the texture coordinates of the second base grid (i.e., texture coordinate reconstruction is optional), reconstructing a second base grid containing texture coordinates. The second base grid containing texture coordinates is then cleaned to obtain a first base grid, where the displacement level of the first base grid corresponds to the L0 level. Based on the level of detail required by the decoder, the corresponding displacement level is decoded and reconstructed. The reconstructed displacement is then applied to the subdivided first base grid to obtain a deformed mesh at the level of detail required by the decoder, i.e., the reconstructed deformed mesh. The texture coordinates of the vertices in the reconstructed deformed mesh are then adjusted to reduce its parametric distortion according to a preset compensation algorithm, ultimately resulting in the deformed mesh output by the decoder. The texture map corresponding to the mesh is obtained by video decoding the texture map bitstream. In this way, the texture coordinates sent by the encoder can be used to further improve the quality of the reconstructed deformed mesh and improve the decoding accuracy and decoding performance.

[0155] It can be understood that in the embodiment of the present application, by updating the texture coordinates of the first basic grid determined by parsing the code stream, the distortion of the texture coordinates of the vertices is reduced, the accuracy of the texture coordinates and the matching degree with the reconstructed deformed grid are improved, thereby improving the quality of the reconstructed deformation and improving the decoding performance.

[0156] In some embodiments, in the above decoding method provided by the embodiments of the present application, the process of determining the texture coordinate information of the reconstructed mesh may be as shown in FIG15 , including:

[0157] S201 : Determine initial texture coordinate information of a reconstructed deformed mesh based on texture coordinate information of a first basic mesh and a basic subdivided mesh.

[0158] S202: Determine a first distortion measure and / or a second distortion measure based on initial texture coordinate information of the reconstructed deformed mesh.

[0159] In some embodiments, based on the initial texture coordinate information of the reconstructed mesh and the three-dimensional coordinate information of the reconstructed mesh, the texture coordinates and three-dimensional coordinates of the triangles corresponding to the vertices in the reconstructed deformed mesh are determined; and based on the texture coordinates and three-dimensional coordinates of the triangles corresponding to the current vertex in the reconstructed deformed mesh, the first distortion measure and / or the second distortion measure are determined.

[0160] This embodiment of the application measures the distortion of a single vertex using two metrics: average stretching of local surface distances and worst-case stretching. The first distortion metric represents the average stretching of local surface distances of a triangle from three-dimensional coordinate space to texture coordinate space; the second distortion metric represents the worst-case stretching of a triangle from three-dimensional coordinate space to texture coordinate space.

[0161] In some embodiments, for a current vertex among the at least one vertex, determining the first distortion measure and / or the second distortion measure based on texture coordinates and three-dimensional coordinates of a triangle corresponding to the current vertex in the reconstructed deformed mesh includes:

[0162] Determine a first length parameter and a second length parameter based on the texture coordinates and the three-dimensional coordinates of the vertices in the triangle patch; determine a first distortion measure based on the first length parameter and the second length parameter; and / or determine a second distortion measure based on the second length parameter.

[0163] The first length parameter represents the minimum length of the unit length vector mapped from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents the maximum length of the unit length vector mapped from the texture coordinate space to the three-dimensional coordinate space;

[0164] In some embodiments, the first length parameter and the second length parameter may be determined by:

[0165] Determining a first partial derivative and a second partial derivative according to an affine mapping between texture coordinates and three-dimensional coordinates of vertices in the triangular patch;

[0166] Determine the partial derivative matrix corresponding to the vertices in the triangle according to the first partial derivative and the second partial derivative;

[0167] Determine a first length parameter according to a first singular value corresponding to the partial derivative matrix;

[0168] A second length parameter is determined according to a second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value.

[0169] For example, the two-dimensional texture coordinates of the three vertices of the triangle T corresponding to the current vertex are p1, p2, and p3 respectively, where p i =(s i ,t i), i = {1, 2, 3}. The three-dimensional geometric coordinates of the triangle T are expressed as q1, q2, q3. The calculation process of the texture coordinate of any point p on the triangle T in the texture coordinate space (i.e., two-dimensional space) to the point affine mapping relationship S(p) = S(s, t) = q on the triangle in three-dimensional space can be shown as formula (1): S(p) = (<p,p2,p3> q1+<p,p3,p1> q2+<p,p1,p2> q3) / <p1,p2,p3> (1)

[0170] In formula (1),<a,b,c> Represents the area of ​​the triangle formed by the three vertices a, b, and c, such as<p,p2,p3> Represents the area of ​​the triangle formed by the three vertices pp, p2, and p3.

[0171] Since the mapping is affine, its partial derivative is a constant at the two-dimensional coordinates (s, t) of point p. According to the above affine mapping, the first partial derivative can be determined by calculating the partial derivative of S(p) at the coordinate s, as shown in formula (2).

[0172] According to the above affine mapping, the second partial derivative can be determined by calculating the partial derivative of S(p) at coordinate t, as shown in formula (3), as follows.

[0173] In formula (2), S s is the first partial derivative, in formula (3), S t is the second partial derivative. In formula (2) and formula (3), A=<p1,p2,p3> =((s2-s1)(t3-t1)-(s3-s1)(t2-t1)) / 2.

[0174] According to the first partial derivative S s With the second partial derivative S t , determine the partial derivative matrix corresponding to the vertices in the triangle patch, such as determining the Jacobian matrix [S s ,S t ], and determine the larger singular value γ of the Jacobian matrix by formula (4) and formula (5) max and smaller singular value γ min , as the first singular value and the second singular value, respectively, as follows:

[0175] In formula (4) and formula (5), a=S s ·S s , b=S s ·S t , c=S t ·St γ max is the first singular value, and γ max As the first length parameter, it represents the maximum length obtained when a unit length vector is mapped from the two-dimensional texture domain to the three-dimensional surface, that is, the maximum local "stretching". min is the second singular value, and γ min As the second length parameter, it represents the minimum length obtained when a unit length vector is mapped from a two-dimensional texture domain to a three-dimensional surface, that is, the minimum local "stretching".

[0176] In some embodiments, the first length parameter γ can be calculated by formula (6). max With the second length parameter γ min ; Determine the first distortion measure L 2 (T), as follows.

[0177] In some embodiments, the second length parameter γ can be calculated by formula (7). min , determine the second distortion measure L ∞ (T), as follows: L ∞ (T) = γ max (7)

[0178] In some embodiments, according to formula (4), formula (5) and formula (6), we can get: That is, according to the first partial derivative S s The corresponding first dot product a=S s ·S s , and the second partial derivative S t The corresponding second dot product c = S t ·S t , determine the first distortion measure.

[0179] S203: Determine texture coordinate information of the reconstructed deformed mesh according to the first distortion measure and / or the second distortion measure.

[0180] In the embodiment of the present application, the first distortion measure L 2 (T) and / or the second distortion measure L ∞ (T) can measure the parametric distortion of the current vertex. For the current vertex, its corresponding texture coordinates need to be adjusted to minimize the parametric distortion of the current vertex.

[0181] In some embodiments, updating the texture coordinates of the current vertex according to the first distortion measure and / or the second distortion measure includes:

[0182] determining a texture coordinate error corresponding to a current vertex in the reconstructed deformed mesh based on the first distortion measure and / or the second distortion measure;

[0183] The texture coordinates of the current vertex are updated by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh.

[0184] In some embodiments, the triangle corresponding to the current vertex includes: at least one triangle adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and / or a second texture coordinate error;

[0185] determining a first texture coordinate error based on a sum of first distortion measures corresponding to at least one triangular facet;

[0186] and / or,

[0187] A second texture coordinate error is determined according to a maximum second distortion measure corresponding to at least one triangle patch.

[0188] Exemplarily, the first texture coordinate error may be determined by formula (8) as follows:

[0189] In formula (8), f represents the triangle adjacent to the current vertex vt, that is, f∈δ vt It can be seen that the first distortion measure L is calculated for each triangle in at least one triangle adjacent to the current vertex vt. 2 (f) The sum of at least one first distortion measure corresponding to at least one triangular facet Determine the first texture coordinate error L 2 (vt).

[0190] Exemplarily, the second texture coordinate error can be determined by formula (9) as follows:

[0191] In formula (9), the maximum second distortion measure corresponding to at least one triangle is Determine the second texture coordinate error L ∞ (vt).

[0192] In some embodiments, the process of updating the texture coordinates of the current vertex by minimizing the texture coordinate error includes:

[0193] Update the texture coordinates of the current vertex by minimizing the first texture coordinate error;

[0194] or,

[0195] Update the texture coordinates of the current vertex by minimizing the second texture coordinate error;

[0196] or,

[0197] Update the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively.

[0198] It should be noted that the process of minimizing the first texture coordinate error and the second texture coordinate error includes: a process of updating the texture coordinates of the current vertex by minimizing the first texture coordinate error, and a process of updating the texture coordinates of the current vertex by minimizing the second texture coordinate error. In other words, the embodiment of the present application optimizes the first texture coordinate error and the second texture coordinate error separately. The embodiment of the present application does not limit the optimization order corresponding to the first texture coordinate error and the second texture coordinate error. Furthermore, when the texture coordinates are updated using the subsequent texture coordinate error, the texture coordinates of the vertex after the previous texture coordinate error are updated are updated.

[0199] It can be understood that in the embodiments of the present application, the distortion of the texture coordinates of the vertices in the reconstructed deformed mesh is measured by using two measures: the average stretching of the surface local distance of the vertex-adjacent triangles from the three-dimensional coordinate space to the texture coordinate space and / or the worst-case stretching of the triangles from the three-dimensional coordinate space to the texture coordinate space. The texture coordinates of the vertices are adjusted according to the distortion, the texture coordinates of the vertices are updated, and the accuracy of the texture coordinates and the matching degree with the reconstructed deformed mesh are improved, thereby improving the quality of the reconstructed deformed mesh and improving the decoding performance.

[0200] Next, based on FIG. 16 , the process of updating the texture coordinates of the current vertex by minimizing the texture coordinate error will be described.

[0201] It should be noted that when updating the texture coordinates of the current vertex by minimizing the first texture coordinate error, the texture coordinate error in the following process includes the first texture coordinate error. When updating the texture coordinates of the current vertex by minimizing the second texture coordinate error, the texture coordinate error in the following process includes the second texture coordinate error. When updating the texture coordinates of the current vertex by minimizing both the first texture coordinate error and the second texture coordinate error, when updating the texture coordinates by minimizing the first texture coordinate error, the texture coordinate error in the following process includes the first texture coordinate error; when updating the texture coordinates by minimizing the second texture coordinate error, the texture coordinate error in the following process includes the second texture coordinate error.

[0202] S301. Determine the initial center point and search range based on the current vertex.

[0203] In S301, the decoder may determine the current vertex as the initial center position point; or determine at least one candidate position point corresponding to the current vertex; determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point. The decoder determines a search range based on the initial center position point.

[0204] In some embodiments, the at least one candidate location point may include: a point at the average position of the current vertex's adjacent points, and / or a midpoint of the current vertex's adjacent edges. For example, the at least one candidate location point corresponding to the current vertex may be as shown in FIG17 , where vt is the current vertex, and vt1′ to vt7′ are the at least one candidate location point corresponding to the current vertex.

[0205] In some embodiments, the above process of determining the initial center position point from at least one candidate position point by determining at least one candidate texture coordinate error corresponding to at least one candidate position point may include: determining at least one candidate texture coordinate error corresponding to at least one candidate position point; and determining the initial center position point based on the minimum value of at least one candidate texture coordinate error and the texture coordinate error corresponding to the current vertex.

[0206] Exemplarily, the texture coordinate error corresponding to each candidate position point in FIG17 is determined by formula (8) and / or formula (9) as the candidate texture coordinate error corresponding to each candidate position point. By comparing at least one candidate texture coordinate error, a minimum candidate texture coordinate error is determined. If the minimum candidate texture coordinate error is smaller than the texture coordinate error corresponding to the current vertex, the candidate position point corresponding to the minimum candidate texture coordinate error is used as the initial center position point. If the minimum candidate texture coordinate error is larger than the texture coordinate error corresponding to the current vertex, the current vertex is used as the initial center position point.

[0207] In some embodiments, the decoder determines a search range with the initial center point as the center and the minimum value of at least one distance between the initial center point and the third edge of at least one adjacent surface as the radius. For example, the search range can be as shown in the dashed circle in FIG18 , where an adjacent circle is constructed with the initial center point as the center as the search range, and the radius of the circle is the minimum value of the distance between the vertex and the third edge of its adjacent surface.

[0208] S302: Perform at least one search based on the initial center position point and the search range to determine at least one initial target position point.

[0209] In step S302, the decoder starts from the initial center position and performs at least one search within the search range to find a position with smaller candidate texture coordinates for updating the texture coordinates. This can be achieved using a variety of search algorithms, and the specific selection depends on the actual situation, which is not limited in the present embodiment.

[0210] For example, based on formula (8) and formula (9), the objective function for minimizing the texture coordinate error can be determined as shown in formula (10), as follows:

[0211] Where N is the search range and vt′ is the location point found each time.

[0212] For example, taking binary search as an example, the process of S202 may include:

[0213] Determine the search direction for each search; based on the initial center position point, perform a binary search within the search range according to the search direction of each search to determine the initial target position point corresponding to each search until at least one search is completed and at least one initial target position point is determined.

[0214] For binary search, in each search of at least one search, the initial center position point is used as the starting point of each search, and the ending point of each search is determined within the search range according to the search direction of each search; when the error of the candidate texture coordinates corresponding to the starting point is less than the error of the candidate texture coordinates corresponding to the ending point, the ending point is updated according to the midpoint between the starting point and the ending point; when the error of the candidate texture coordinates corresponding to the starting point is greater than the error of the corresponding candidate texture coordinates of the ending point, the starting point is updated according to the midpoint between the starting point and the ending point; the above process is used for at least one update until it is determined that the distance between the updated starting point and the updated ending point is less than the preset distance threshold, and the initial target position point corresponding to each search is determined according to the starting point and / or the ending point.

[0215] In some embodiments, when the distance between the updated starting point and the updated ending point is less than the preset distance threshold at least once in each search, it is determined that the distance between the updated starting point and the updated ending point is less than the preset distance threshold.

[0216] In some embodiments, when it is determined that the distance between the updated starting point and the updated ending point is less than a preset distance threshold, the starting point obtained by the current update can be determined as the initial target position point; or the ending point obtained by the current update can be determined as the initial target position point; or the midpoint between the starting point obtained by the current update and the ending point obtained by the current update can be determined as the initial target position point; or the point between the starting point obtained by the current update and the ending point obtained by the current update, in which the candidate texture coordinate error is smaller, can be determined as the target position point. The specific selection is made according to the actual situation and is not limited in the embodiments of the present application.

[0217] Exemplarily, within the search range, a direction is randomly selected as the search direction. The direction is selected by random number generation, and randomly selected directions are more likely to find the optimal solution; the optimal value is gradually approached in the selected direction by bisection. That is, the errors corresponding to the two endpoints are calculated, the center of the circle is recorded as the starting point, and the other end is recorded as the end point. If the error of the starting point is less than the end point, the midpoint of the two end points is used as the end point; if the error of the starting point is greater than the end point, the midpoint of the two end points is used as the starting point, and then the error of the two end points is calculated and compared. Iteration is performed in the above manner until the distance between two consecutive selected vertices is less than the preset threshold, a search is completed, and the initial target position point corresponding to the search is determined. In order to increase the stability of the search, directions are randomly selected multiple times, and the above steps are repeated, that is, at least one search is performed to find the optimal local solution, and the target position point is determined from at least one initial target position point.

[0218] S303: Determine a target position point according to a minimum candidate texture coordinate error corresponding to at least one initial target position point.

[0219] S304: Update the texture coordinates of the current vertex according to the texture coordinates of the target position point.

[0220] In S303-S304, the initial target location point with the smallest texture coordinate error among the at least one initial target location point may be determined as the target location point. It is understood that the target location point is the location point with the smallest distortion within the search range of the current vertex. The decoder updates the texture coordinates of the current vertex based on the texture coordinates of the target location point. For example, the texture coordinates of the current vertex are updated to the texture coordinates of the target location point.

[0221] It can be understood that by searching for the target position point corresponding to the current vertex within the local range (search range), the texture coordinates of the current vertex are updated according to the texture coordinates of the target position point with less distortion, thereby improving the matching degree between the texture coordinates of the current vertex and the current vertex, thereby improving the quality of the reconstructed deformed mesh, and further improving the decoding accuracy and decoding performance.

[0222] In some embodiments, for at least one vertex to be optimized in the reconstructed deformed mesh, at least one texture coordinate error corresponding to the at least one vertex is determined; based on the at least one texture coordinate error, the at least one vertex is organized into a sequence to be optimized; and a vertex corresponding to a predetermined sequence position in the sequence to be optimized is determined as the current vertex; the predetermined sequence position corresponds to the vertex with the largest texture coordinate error among the at least one vertex. Here, the at least one vertex may include some or all of the vertices in the reconstructed deformed mesh.

[0223] In other words, the current vertex is the vertex with the largest texture coordinate error in the sequence to be optimized. Thus, once the target position corresponding to the current vertex is determined, the texture coordinate error corresponding to at least one vertex in the sequence to be optimized can be updated based on the target position, thereby updating the order of the vertices in the sequence to be optimized. Texture coordinates are updated for vertices at the updated preset sequence positions until the preset iteration target is reached, completing the texture coordinate update for the reconstructed deformed mesh.

[0224] For example, when selecting the current vertex for texture coordinate update, a preset data structure, such as a heap data structure, can be used to sort at least one vertex according to its texture coordinate error, resulting in a heap data structure to be optimized. For example, vertices with lower quality are placed at the top of the heap, while vertices with higher quality are placed lower down. In each iteration, the vertex with the largest texture coordinate error is selected from the top of the heap for optimization. The texture coordinates of this vertex are adjusted to minimize its texture coordinate error. Then, the texture coordinate errors of adjacent vertices are updated and placed back into the heap structure to be optimized. As can be seen, the vertex at the top of the heap changes with each optimization iteration, necessitating an update to the heap structure to reflect the latest error ranking. This ensures that the vertex with the lowest quality is still selected for adjustment in subsequent iterations. This strategy allows the entire optimization process to more closely align with the global optimization goal. Vertices with lower quality are adjusted more frequently during iterations, driving the overall parameterization toward a more optimal state. This improves decoding efficiency and accuracy.

[0225] In some embodiments, when the number of updates of the sequence to be optimized reaches a preset update number threshold, and / or the texture coordinate error of the vertex at the preset sequence position is less than a preset error threshold, it is determined that the preset iteration target is reached, and the texture coordinate update of at least one vertex to be optimized in the reconstructed deformed mesh is completed.

[0226] In some embodiments, based on Figures 13 and 14, it can be seen that the texture coordinates of the vertices in each layer of the reconstructed deformed mesh in at least one layer of the reconstructed deformed mesh can be updated. That is, the reconstructed deformed mesh processed each time includes: the i-th layer of the reconstructed deformed mesh corresponding to the base mesh; i is greater than 1. When the decoder completes the update of the texture coordinates of the i-th layer of the reconstructed deformed mesh, it can interpolate based on the texture coordinates of the vertices in the i-th layer of the reconstructed deformed mesh to determine the texture coordinates of the vertices in the i+1-th layer of the reconstructed deformed mesh; i+1 is less than or equal to the preset number of layers; by determining the first distortion measure and / or the second distortion measure of the triangle corresponding to the vertex in the i+1-th layer of the reconstructed deformed mesh, the texture coordinates of the vertices in the i+1-th layer of the reconstructed deformed mesh are updated. The texture coordinates of the vertices in each layer of the reconstructed deformed mesh are adjusted layer by layer using the above process until the texture coordinate update of at least one layer of the reconstructed deformed mesh is completed.

[0227] For example, taking Figure 12 as an example, after decoding to obtain the L1 level and performing texture coordinate compensation, if the decoder needs to further improve the detail level such as obtaining the L2 level, the interpolated position of the midpoint of the L1 level texture coordinate is used as the starting point, and then the texture coordinates of the L2 level are updated, and so on, until the detail level required by the decoding end is reconstructed.

[0228] Referring to Figure 19, it shows a schematic flow chart of an encoding method provided by an embodiment of the present application. As shown in Figure 19, the method may include:

[0229] S401: Determine a basic grid corresponding to the current image.

[0230] S402: Determine texture coordinate information based on the base mesh.

[0231] S403: Encode the basic mesh and texture coordinate information, and write the obtained encoding bits into the bitstream.

[0232] In some embodiments, an encoder framework corresponding to the above-mentioned decoding method can be as described in the current VDMC encoding framework in FIG3 . That is, the input mesh is subjected to a base mesh generation step to obtain a base mesh and a corresponding subdivided deformed mesh. The base mesh is compressed and reconstructed by a base mesh compression module to obtain a reconstructed base mesh. Then, based on the deformed mesh obtained in the base mesh generation stage, the displacements between the corresponding vertices of the subdivided base mesh and the corresponding subdivided deformed mesh are obtained. The displacements are processed by coordinate system conversion, wavelet transform, quantization, etc., and then arranged into an image for encoding using a video encoder or directly entropy encoding. The encoded displacements are then reconstructed to obtain a reconstructed subdivided deformed mesh. This mesh, the original input mesh, and the texture map are sent as input to a texture map conversion module to obtain a texture map corresponding to the currently reconstructed deformed mesh. The texture map is subjected to optional color space conversion and other processing before being encoded using a video encoder.

[0233] The texture coordinate information is determined by mesh parameterization based on the base mesh. The encoder can encode the base mesh containing the texture coordinate information and write the resulting encoded bits into the bitstream. This bitstream corresponds to the first basic bitstream of the decoder.

[0234] In this way, the decoding method in the above embodiment is executed by the decoder to compensate the texture coordinates of the reconstructed deformed mesh, and the matching degree between the texture coordinates and the reconstructed deformed mesh is improved, thereby improving decoding accuracy and decoding performance.

[0235] In some embodiments, unlike the above-mentioned method of merging the texture coordinate information and the base grid into an encoding, since the base grid encoding is a compression encoding, in order to reduce the distortion of the texture coordinate information, the texture coordinate information and the base grid can be encoded separately, and the bitstream corresponding to the base grid that does not contain the texture coordinate information (the second base grid bitstream corresponding to the decoder) and the bitstream corresponding to the texture coordinate information are determined separately.

[0236] In one embodiment, texture coordinate information can be determined during the process of generating a base mesh by a base mesh generation module. The texture coordinate information includes texture coordinate information corresponding to the base mesh. The encoder can perform mesh parameterization based on the base mesh to determine the texture coordinate information corresponding to the base mesh; perform compression encoding on the base mesh to determine the coding bits corresponding to the base mesh; encode the texture coordinate information to determine the coding bits corresponding to the texture coordinate information; and write the coding bits corresponding to the base mesh and the coding bits corresponding to the texture coordinate information into the bitstream, respectively. In this way, the decoder can parse the texture coordinate information in the bitstream to reconstruct the second base mesh obtained from the base bitstream to determine the first base mesh, thereby improving the accuracy of the texture coordinates in the first base mesh, thereby improving encoding and decoding accuracy and performance.

[0237] For example, as shown in FIG20 , the input mesh first passes through the base mesh generation step to obtain a base mesh, which is then compressed and reconstructed by the base mesh compression module to obtain a reconstructed base mesh. The texture coordinates obtained in the base mesh generation stage can be restored based on the texture coordinate information of the current image. Therefore, this information can be passed to the decoding end to avoid directly encoding the texture coordinates. After the texture coordinates are regenerated for the reconstructed base mesh, mesh cleaning is performed to remove duplicate points and degenerate surfaces. Then, through the subdivision deformation step, the displacement between the corresponding vertices of the subdivided base mesh and the corresponding subdivided deformed mesh is obtained. The subsequent processing process is the same as the corresponding encoding process in FIG3 .

[0238] In another embodiment, the texture coordinate information may be determined based on a reconstructed base mesh, and the texture coordinate information includes texture coordinate information corresponding to the reconstructed base mesh. The encoder may reconstruct based on the coded bits of the base mesh to determine the reconstructed base mesh; perform mesh parameterization on the reconstructed base mesh to determine the texture coordinate information; perform compression encoding on the base mesh to determine the coded bits corresponding to the base mesh; encode the texture coordinate information to determine the coded bits corresponding to the texture coordinate information; and write the coded bits corresponding to the base mesh and the coded bits corresponding to the texture coordinate information into the bitstream, respectively.

[0239] For example, as shown in Figure 21, after the input mesh is generated by the base mesh module, it is directly compressed to obtain a base mesh stream and a reconstructed base mesh. The reconstructed base mesh is then parameterized to obtain a reconstructed mesh containing texture coordinates. The texture coordinate information is also encoded as the base mesh stream. The parameterized base mesh is then cleaned to remove duplicate vertices and degenerate faces, and a subdivision deformation operation is performed to obtain the corresponding displacement. The subsequent processing is the same as the corresponding encoding process in Figure 3.

[0240] It can be understood that by separately encoding the texture coordinate information and sending it to the decoder, the distortion of the texture coordinate information is reduced, the encoding and decoding accuracy is further improved, and thus the encoding and decoding performance is improved.

[0241] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Figure 22, which shows a schematic diagram of the composition structure of an encoder provided by an embodiment of the present application. As shown in Figure 22, the encoder 190 may include a grid information determination part 1901 and an encoding part 1902, wherein:

[0242] The mesh generation part 1901 is configured to determine a base mesh corresponding to the current image; and determine texture coordinate information based on the base mesh;

[0243] The encoding part 1902 is configured to encode the basic mesh and the texture coordinate information, and write the obtained encoding bits into a bitstream.

[0244] In some embodiments, the texture coordinate information includes: texture coordinate information corresponding to the basic mesh; the mesh information determination part 1901 is further configured to perform mesh parameterization based on the basic mesh to determine the texture coordinate information.

[0245] In some embodiments, the texture coordinate information includes: texture coordinate information corresponding to the reconstructed basic grid; the grid information determination part 1901 reconstructs based on the coding bits of the basic grid to determine the reconstructed basic grid; and performs grid parameterization on the reconstructed basic grid to determine the texture coordinate information.

[0246] In some embodiments, the encoding portion 1902 is further configured to encode the base grid including the texture coordinate information, and write the obtained encoding bits into a bitstream.

[0247] In some embodiments, the encoding part 1902 is further configured to perform compression encoding on the basic grid to determine the encoding bits corresponding to the basic grid; encode the texture coordinate information to determine the encoding bits corresponding to the texture coordinate information; and write the encoding bits corresponding to the basic grid and the encoding bits corresponding to the texture coordinate information into the bitstream respectively.

[0248] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0249] It is understandable that in the embodiments of the present application, a "part" can be a part of a circuit, a part of a processor, a part of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0250] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or grid device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0251] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 190. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method described in any one of the aforementioned embodiments.

[0252] Based on the composition of the encoder 190 and the computer-readable storage medium, refer to Figure 23, which shows a specific hardware structure diagram of the encoder 190 provided in an embodiment of the present application. As shown in Figure 23, the encoder 190 may include: a first communication interface 2301, a first memory 2302 and a first processor 2303; each component is coupled together through a first bus system 2304. It can be understood that the first bus system 2304 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 2304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 2304 in Figure 20. Among them,

[0253] The first communication interface 2301 is used to receive and send signals when sending and receiving information with other external network elements;

[0254] A first memory 2302 is used to store computer programs that can be run on the first processor 2303;

[0255] The first processor 2303 is configured to execute the encoding method applied to the encoder in the embodiment of the present application when running the computer program.

[0256] It is understood that the first memory 2302 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 2302 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0257] The first processor 2303 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 2303. The above-mentioned first processor 2303 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 2302 , and the first processor 2303 reads the information in the first memory 2302 and completes the steps of the above method in combination with its hardware.

[0258] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0259] Optionally, as another embodiment, the first processor 2303 is further configured to execute the encoding method applied to an encoder described in any one of the aforementioned embodiments when running the computer program.

[0260] Based on the same inventive concept as the above embodiment, referring to FIG24 , a schematic diagram of the structure of a decoder provided by an embodiment of the present application is shown. As shown in FIG24 , the decoder 240 may include the parsing part 2401 and the coordinate determination part 2403, wherein:

[0261] The parsing part 2401 is configured to parse the code stream to determine the first basic mesh and texture coordinate information of the first basic mesh;

[0262] The subdivision and deformation part 2402 is configured to perform mesh subdivision and deformation based on the first basic mesh to determine a reconstructed deformed mesh;

[0263] The coordinate determining part 2403 is configured to determine the texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh.

[0264] In some embodiments, the parsing portion 2401 is further configured to demultiplex the code stream to determine a first basic grid code stream;

[0265] The first basic grid code stream is parsed to determine the first basic grid including the texture coordinate information, thereby determining the first basic grid and the texture coordinate information of the first basic grid.

[0266] In some embodiments, the parsing part 2401 is further configured to demultiplex the code stream to determine a second basic grid code stream and a texture coordinate code stream; parse the second basic grid code stream to determine a second basic grid; parse the texture coordinate code stream to determine texture coordinate information; reconstruct the second basic grid based on the texture coordinate information to determine the first basic grid and the texture coordinate information of the first basic grid.

[0267] In some embodiments, the decoder 240 also includes a reconstruction part, which is configured to reconstruct the second basic grid based on the texture coordinate information to determine the basic grid containing the texture coordinate information; perform grid cleaning on the basic grid containing the texture coordinate information to determine the first basic grid and the texture coordinate information of the first basic grid.

[0268] In some embodiments, the subdivision deformation part 2402 is further configured to perform grid subdivision based on the first basic grid to determine a basic subdivision grid; and determine the reconstructed deformed grid based on the basic subdivision grid and displacement information parsed from the bitstream.

[0269] In some embodiments, the coordinate determination part 2403 is further configured to determine the initial texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first basic mesh and the basic subdivision mesh; determine a first distortion measure and / or a second distortion measure based on the initial texture coordinate information of the reconstructed deformed mesh; the first distortion measure represents the average stretching of the surface local distance of the triangular facets in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; the second distortion measure represents the worst-case stretching of the triangular facets in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; and determine the texture coordinate information of the reconstructed deformed mesh based on the first distortion measure and / or the second distortion measure.

[0270] In some embodiments, the coordinate determination part 2403 is further configured to determine the texture coordinates and three-dimensional coordinates of the triangular facets corresponding to the vertices in the reconstructed deformed mesh based on the initial texture coordinate information of the reconstructed mesh and the three-dimensional coordinate information of the reconstructed mesh; determine a first length parameter and a second length parameter according to the texture coordinates and three-dimensional coordinates of the triangular facets; the first length parameter represents the minimum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents the maximum length corresponding to the mapping of a unit length vector from the texture coordinate space to the three-dimensional coordinate space; determine the first distortion measure according to the first length parameter and the second length parameter; and / or determine the second distortion measure according to the second length parameter.

[0271] In some embodiments, the coordinate determination part 2403 is further configured to determine the first partial derivative and the second partial derivative based on the affine mapping between the texture coordinates of the triangular face and the three-dimensional coordinates; determine the partial derivative matrix corresponding to the vertices in the triangular face based on the first partial derivative and the second partial derivative; determine the first length parameter based on the first singular value corresponding to the partial derivative matrix; determine the second length parameter based on the second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value.

[0272] In some embodiments, the coordinate determining portion 2403 is further configured to determine the first distortion measure based on a first dot product corresponding to the first partial derivative and a second dot product corresponding to the second partial derivative.

[0273] In some embodiments, the coordinate determination part 2403 is further configured to determine the texture coordinate error corresponding to the current vertex in the reconstructed deformed mesh based on the first distortion measure and / or the second distortion measure; and update the texture coordinates of the current vertex by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh.

[0274] In some embodiments, the triangle corresponding to the current vertex includes: at least one triangle adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and / or a second texture coordinate error; the coordinate determining portion 2403 is further configured to determine the first texture coordinate error based on a sum of first distortion measures corresponding to the at least one triangle;

[0275] The second texture coordinate error is determined according to a maximum second distortion measure corresponding to the at least one triangular facet.

[0276] In some embodiments, the coordinate determination part 2403 is further configured to update the texture coordinates of the current vertex by minimizing the first texture coordinate error; or, to update the texture coordinates of the current vertex by minimizing the second texture coordinate error; or, to update the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively.

[0277] In some embodiments, the coordinate determination part 2403 is further configured to determine an initial center position point and a search range based on the current vertex; perform at least one search based on the initial center position point and the search range to determine at least one initial target position point; determine a target position point based on the minimum candidate texture coordinate error corresponding to at least one initial target position point; and update the texture coordinates of the current vertex based on the texture coordinates of the target position point.

[0278] In some embodiments, the coordinate determination part 2403 is further configured to determine at least one candidate position point corresponding to the current vertex; determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point; and determine the search range based on the initial center position point.

[0279] In some embodiments, the at least one candidate position point includes: a point at an average position of adjacent points of the current vertex, and / or a midpoint of an adjacent edge of the current vertex.

[0280] In some embodiments, the coordinate determination part 2403 is further configured to determine at least one candidate texture coordinate error corresponding to the at least one candidate position point; and determine the initial center position point based on the minimum value of the at least one candidate texture coordinate error and the texture coordinate error corresponding to the current vertex.

[0281] In some embodiments, the coordinate determination portion 2403 is further configured to determine the search range with the initial center point as the center and the minimum value of at least one distance from the initial center point to the third side of at least one adjacent face as the radius. In some embodiments, the coordinate determination portion 2403 is further configured to determine the search direction for each search; based on the initial center point, a binary search is performed within the search range according to the search direction of each search to determine the initial target location point corresponding to each search, until at least one search is completed and the at least one initial target location point is determined.

[0282] In some embodiments, the coordinate determination part 2403 is further configured to use the initial center position point as the starting point of each search, and determine the ending point of each search within the search range according to the search direction of each search; when the candidate texture coordinate error corresponding to the starting point is less than the candidate texture coordinate error corresponding to the ending point, update the ending point according to the midpoint between the starting point and the ending point; when the candidate texture coordinate error corresponding to the starting point is greater than the corresponding candidate texture coordinate error of the ending point, update the starting point according to the midpoint between the starting point and the ending point; perform at least one update until it is determined that the distance between the updated starting point and the updated ending point is less than a preset distance threshold, and determine the initial target position point according to the starting point and / or the ending point.

[0283] In some embodiments, the coordinate determination part 2403 is further configured to determine the starting point as the initial target position point; or, determine the ending point as the initial target position point; or, determine the midpoint between the starting point and the ending point as the initial target position point; or, determine the point between the starting point and the ending point with the smallest error in the candidate texture coordinates as the target position point.

[0284] In some embodiments, the coordinate determination part 2403 is further configured to determine at least one texture coordinate error corresponding to the at least one vertex; organize the at least one vertex into a sequence to be optimized based on the at least one texture coordinate error; determine the vertex corresponding to a preset sequence position in the sequence to be optimized as the current vertex; the preset sequence position corresponds to the vertex with the largest texture coordinate error among the at least one vertex.

[0285] In some embodiments, the coordinate determination part 2403 is further configured to update the texture coordinate error corresponding to the current vertex in the at least one vertex according to the target position point, and then update the order of the vertices in the sequence to be optimized; update the texture coordinates of the vertices at the updated preset sequence position until the preset iteration target is reached, thereby completing the reconstruction of the texture coordinate information of the deformed mesh.

[0286] In some embodiments, the preset iteration target includes:

[0287] The update times of the sequence to be optimized reaches a preset update times threshold; and / or the texture coordinate error of the vertex to be optimized at the preset sequence position is less than a preset error threshold.

[0288] In some embodiments, the reconstructed deformed mesh includes: the i-th layer reconstructed deformed mesh corresponding to the first basic mesh; i is greater than 1, and the coordinate determination part 2403 is further configured to interpolate the texture coordinates of the vertices in the i-th layer reconstructed deformed mesh after completing the update of the texture coordinates of the i-th layer reconstructed deformed mesh, and determine the texture coordinates of the vertices in the i+1-th layer reconstructed deformed mesh; i+1 is less than or equal to the preset number of layers; by determining the first distortion measure and / or the second distortion measure of the triangle facets corresponding to the vertices in the i+1-th layer reconstructed deformed mesh, the texture coordinates of the vertices in the i+1-th layer reconstructed deformed mesh are updated.

[0289] It is understood that in this embodiment, a "portion" may be a circuit portion, a processor portion, a program portion, or software portion, and may also be a module or non-modular. Furthermore, the various components in this embodiment may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional modules.

[0290] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer-readable storage medium, which is applied to the decoder 240 and stores a computer program. When the computer program is executed by the second processor, it implements any of the methods in the aforementioned embodiments.

[0291] It should be noted that the description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0292] Based on the composition of the decoder 240 and the computer-readable storage medium, refer to Figure 25, which shows a specific hardware structure diagram of the decoder 240 provided in an embodiment of the present application. As shown in Figure 25, the decoder 240 may include: a second communication interface 2501, a second memory 2502 and a second processor 2503; each component is coupled together through a second bus system 2504. It can be understood that the second bus system 2504 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 2504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 2504 in Figure 25. Among them,

[0293] The second communication interface 2501 is used to receive and send signals during the process of sending and receiving information between other external network elements;

[0294] The second memory 2502 is used to store computer programs that can be run on the second processor 2503;

[0295] The second processor 2503 is configured to execute the decoding method applied to the decoder provided in the embodiment of the present application when running the computer program.

[0296] Optionally, as another embodiment, the second processor 2503 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0297] It can be understood that the hardware functions of the second memory 2502 are similar to those of the first memory 2302, and the hardware functions of the second processor 2503 are similar to those of the first processor 2303; they will not be described in detail here.

[0298] In yet another embodiment of the present application, referring to FIG26 , a schematic diagram of the structure of a coding and decoding system provided by an embodiment of the present application is shown. As shown in FIG26 , the coding and decoding system 260 may include an encoder 2601 and a decoder 2602 .

[0299] In the embodiment of the present application, the encoder 2601 may be the encoder described in any one of the aforementioned embodiments, and the decoder 2602 may be the decoder described in any one of the aforementioned embodiments.

[0300] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0301] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0302] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0303] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0304] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0305] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0306] Embodiments of the present application provide a coding and decoding method, bitstream, encoder, decoder, and storage medium. At the decoding end, the bitstream is parsed to determine a first base mesh and texture coordinate information of the first base mesh; mesh subdivision and deformation are performed based on the first base mesh to determine a reconstructed deformed mesh; and texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh. In this manner, the texture coordinate information of the reconstructed deformed mesh is determined based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh obtained by subdivision and deformation. This reduces the distortion of vertex texture coordinates, improves the accuracy of texture coordinates, and improves the matching degree between the texture coordinates and the reconstructed deformed mesh, thereby improving the quality of the reconstructed deformed mesh and decoding performance. Furthermore, the distortion of vertex texture coordinates in the reconstructed deformed mesh is measured using two metrics: the average stretching of the local surface distance of a triangle from a three-dimensional coordinate space to a texture coordinate space and / or the worst-case stretching of a triangle from a three-dimensional coordinate space to a texture coordinate space. The texture coordinates of the vertices are adjusted based on the distortion, and the texture coordinates of the vertices are updated, improving the accuracy of the texture coordinates and the matching degree between the vertices and the reconstructed deformed mesh, thereby improving the quality of the reconstructed deformed mesh and decoding performance. Furthermore, by separately encoding the texture coordinate information through the encoder and sending it to the decoder, the distortion of the texture coordinate information is reduced, the encoding and decoding accuracy is further improved, and thus the encoding and decoding performance is improved.

Claims

1. A decoding method, applied to a decoder, the method comprising: Parsing a bitstream to determine a first base mesh and texture coordinate information of the first base mesh; Based on the first base mesh, performing mesh subdivision and deformation to determine a reconstructed deformed mesh; Based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh, determining the texture coordinate information of the reconstructed deformed mesh.

2. The method according to claim 1, wherein, The parsing the bitstream to determine the first base mesh and the texture coordinate information of the first base mesh includes: Demultiplexing the bitstream to determine a first base mesh bitstream; Parsing the first base mesh bitstream to determine a first base mesh including texture coordinate information, thereby determining the first base mesh and the texture coordinate information of the first base mesh.

3. The method according to claim 1, wherein, The parsing the bitstream to determine the first base mesh and the texture coordinate information of the first base mesh includes: Demultiplexing the bitstream to determine a second base mesh bitstream and a texture coordinate bitstream; Parsing the second base mesh bitstream to determine a second base mesh; Parsing the texture coordinate bitstream to determine texture coordinate information; According to the texture coordinate information, reconstructing the second base mesh to determine the first base mesh and the texture coordinate information of the first base mesh.

4. The method according to claim 3, wherein The according to the texture coordinate information, reconstructing the second base mesh to determine the first base mesh and the texture coordinate information of the first base mesh includes: According to the texture coordinate information, reconstructing the second base mesh to determine a base mesh including texture coordinate information; Performing mesh cleaning on the base mesh including texture coordinate information to determine the first base mesh and the texture coordinate information of the first base mesh.

5. The method according to claim 1, wherein The based on the first base mesh, performing mesh subdivision and deformation to determine a reconstructed deformed mesh includes: Based on the first base mesh, performing mesh subdivision to determine a base subdivided mesh; Based on the base subdivided mesh and displacement information parsed from the bitstream, determining the reconstructed deformed mesh.

6. The method according to claim 5, wherein, The based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh, determining the texture coordinate information of the reconstructed deformed mesh includes: Based on the texture coordinate information of the first base mesh and the base subdivided mesh, determining initial texture coordinate information of the reconstructed deformed mesh; Based on the initial texture coordinate information of the reconstructed deformed mesh, determining a first distortion measure and / or a second distortion measure; the first distortion measure represents the average stretch of the surface local distance of the triangular patches in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; the second distortion measure represents the worst-case stretch of the triangular patches in the reconstructed deformed mesh from the three-dimensional coordinate space to the texture coordinate space; According to the first distortion measure and / or the second distortion measure, determining the texture coordinate information of the reconstructed deformed mesh.

7. The method according to claim 6, wherein, The based on the initial texture coordinate information of the reconstructed deformed mesh, determining a first distortion measure and / or a second distortion measure includes: Based on the initial texture coordinate information of the reconstructed mesh and the three-dimensional coordinate information of the reconstructed mesh, determine the texture coordinates and three-dimensional coordinates of the triangular patches corresponding to the vertices in the reconstructed deformed mesh; According to the texture coordinates and three-dimensional coordinates of the triangular patches, determine a first length parameter and a second length parameter; the first length parameter represents the minimum length corresponding to mapping a unit length vector from the texture coordinate space to the three-dimensional coordinate space; the second length parameter represents the maximum length corresponding to mapping a unit length vector from the texture coordinate space to the three-dimensional coordinate space; According to the first length parameter and the second length parameter, determine the first distortion measure; and / or, According to the second length parameter, determine the second distortion measure.

8. The method according to claim 7, wherein The determining the first length parameter and the second length parameter according to the texture coordinates and three-dimensional coordinates of the triangular patches includes: Determine a first partial derivative and a second partial derivative according to the affine mapping between the texture coordinates and three-dimensional coordinates of the triangular patches; Determine the partial derivative matrix corresponding to the vertices in the triangular patch according to the first partial derivative and the second partial derivative; Determine the first length parameter according to the first singular value corresponding to the partial derivative matrix; Determine the second length parameter according to the second singular value corresponding to the partial derivative matrix; the first singular value is greater than the second singular value.

9. The method according to claim 8, wherein, The method further includes: Determine the first distortion measure according to the first dot product corresponding to the first partial derivative and the second dot product corresponding to the second partial derivative.

10. The method according to any one of claims 6-9, wherein, The determining the texture coordinate information of the reconstructed deformed mesh according to the first distortion measure and / or the second distortion measure includes: Determine the texture coordinate error corresponding to the current vertex in the reconstructed deformed mesh according to the first distortion measure and / or the second distortion measure; Update the texture coordinates of the current vertex by minimizing the texture coordinate error, thereby determining the texture coordinate information of the reconstructed deformed mesh.

11. The method according to claim 10, wherein, The triangular patches corresponding to the current vertex include: at least one triangular patch adjacent to the current vertex; the texture coordinate error includes: a first texture coordinate error and / or a second texture coordinate error; the determining the texture coordinate error corresponding to the current vertex according to the first distortion measure and / or the second distortion measure includes: Determine the first texture coordinate error according to the sum of the first distortion measures corresponding to the at least one triangular patch; and / or, Determine the second texture coordinate error according to the maximum second distortion measure corresponding to the at least one triangular patch.

12. The method according to claim 11, wherein, The updating the texture coordinates of the current vertex by minimizing the texture coordinate error includes: Update the texture coordinates of the current vertex by minimizing the first texture coordinate error; or, Update the texture coordinates of the current vertex by minimizing the second texture coordinate error; or, Update the texture coordinates of the current vertex by minimizing the first texture coordinate error and the second texture coordinate error respectively.

13. The method according to claim 10, wherein, The updating the texture coordinates of the current vertex by minimizing the texture coordinate error includes: Based on the current vertex, determine the initial center position point and the search range; Based on the initial center position point and the search range, conduct at least one search to determine at least one initial target position point; Determine the target position point according to the minimum candidate texture coordinate error corresponding to at least one initial target position point; Update the texture coordinate of the current vertex according to the texture coordinate of the target position point.

14. The method according to claim 10, wherein The step of "Based on the current vertex, determine the initial center position point and the search range" includes: Determine at least one candidate position point corresponding to the current vertex; Determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point; Based on the initial center position point, determine the search range.

15. The method according to claim 14, wherein The at least one candidate position point includes: The point at the average position of the adjacent points of the current vertex, and / or the midpoint of the adjacent edges of the current vertex.

16. The method according to claim 14, wherein, The step of "Determine the initial center position point from the at least one candidate position point by determining at least one candidate texture coordinate error corresponding to the at least one candidate position point" includes: Determine at least one candidate texture coordinate error corresponding to the at least one candidate position point; Determine the initial center position point according to the minimum value among the at least one candidate texture coordinate error and the texture coordinate error corresponding to the current vertex.

17. The method according to claim 14, wherein, The step of "Based on the initial center position point, determine the search range" includes: Taking the initial center position point as the center and the minimum value among at least one distance from the initial center position point to the third side of at least one of its adjacent faces as the radius, determine the search range.

18. The method according to claim 13, wherein The step of "Based on the initial center position point and the search range, conduct at least one search to determine at least one initial target position point" includes: Determine the search direction for each search; Based on the initial center position point, perform a binary search within the search range in the search direction of each search to determine the initial target position point corresponding to each search until at least one search is completed to determine the at least one initial target position point.

19. The method according to claim 18, wherein, The step of "Based on the initial center position point, perform a binary search within the search range in the search direction of each search to determine the initial target position point corresponding to each search" includes: Taking the initial center position point as the starting point of each search, determine the termination point of each search within the search range in the search direction of each search; In the case where the candidate texture coordinate error corresponding to the starting point is less than the candidate texture coordinate error corresponding to the termination point, update the termination point according to the midpoint between the starting point and the termination point; In the case where the candidate texture coordinate error corresponding to the starting point is greater than the candidate texture coordinate error corresponding to the termination point, update the starting point according to the midpoint between the starting point and the termination point; Perform at least one update until the distance between the updated starting point and the updated ending point is less than a preset distance threshold, and determine the initial target position point according to the starting point and / or the ending point.

20. The method according to claim 19, wherein The determining the initial target position point according to the starting point and / or the ending point includes: Determine the starting point as the initial target position point; Or, Determine the ending point as the initial target position point; Or, determine the midpoint between the starting point and the ending point as the initial target position point; Or, determine the point with a smaller candidate texture coordinate error among the starting point and the ending point as the target position point.

21. The method according to any one of claims 11 - 20, wherein, The method further includes: Determine at least one texture coordinate error corresponding to the at least one vertex; Organize the at least one vertex into an optimization sequence according to the at least one texture coordinate error; Determine the vertex corresponding to the preset sequence position in the optimization sequence as the current vertex; the preset sequence position corresponds to the vertex with the largest texture coordinate error among the at least one vertex.

22. The method according to claim 21, wherein, The determining the texture coordinate information of the reconstructed deformed mesh includes: Update the texture coordinate error corresponding to the current vertex in the at least one vertex according to the target position point, and further update the order of the vertices in the optimization sequence; Perform texture coordinate update on the vertex at the updated preset sequence position until a preset iteration target is reached, and complete the texture coordinate information of the reconstructed deformed mesh.

23. The method according to claim 21, wherein, The preset iteration target includes: The number of updates of the optimization sequence reaches a preset update number threshold; And / or, the texture coordinate error of the vertex at the preset sequence position is less than a preset error threshold.

24. The method according to claim 6 or 22, wherein, The reconstructed deformed mesh includes: the i-th layer reconstructed deformed mesh corresponding to the first base mesh; i>1, the method further includes: In the case of completing the texture coordinate update of the i-th layer reconstructed deformed mesh, perform interpolation based on the texture coordinates of the vertices in the i-th layer reconstructed deformed mesh to determine the texture coordinates of the vertices in the (i + 1)-th layer reconstructed deformed mesh; i + 1 is less than or equal to a preset number of layers; Update the texture coordinates of the vertices in the (i + 1)-th layer reconstructed deformed mesh by determining the first distortion measure and / or the second distortion measure of the triangular patches corresponding to the vertices in the (i + 1)-th layer reconstructed deformed mesh.

25. The method according to claim 24, wherein, The method further includes: Restore the current image by performing texture coordinate update on at least one layer of the reconstructed deformed mesh corresponding to the first base mesh.

26. An encoding method, applied to an encoder, the method includes: Determine the base mesh corresponding to the current image; Determine the texture coordinate information based on the base mesh; Encode the base mesh and the texture coordinate information, and write the obtained encoded bits into the bitstream.

27. The method according to claim 26, wherein, The texture coordinate information includes: the texture coordinate information corresponding to the base mesh; the determining the texture coordinate information based on the base mesh includes: Perform mesh parameterization based on the base mesh to determine the texture coordinate information.

28. The method according to claim 24, wherein The texture coordinate information includes: texture coordinate information corresponding to the reconstructed base mesh; determining the texture coordinate information based on the base mesh includes: Performing reconstruction based on the coded bits of the base mesh to determine the reconstructed base mesh; Performing mesh parameterization on the reconstructed base mesh to determine the texture coordinate information.

29. The method according to claim 27, wherein, Encoding the base mesh and the texture coordinate information and writing the obtained coded bits into the bitstream includes: Encoding the base mesh including the texture coordinate information and writing the obtained coded bits into the bitstream.

30. The method according to claim 27 or 28, wherein, Encoding the base mesh and the texture coordinate information and writing the obtained coded bits into the bitstream includes: Performing compression encoding on the base mesh to determine the coded bits corresponding to the base mesh; Encoding the texture coordinate information to determine the coded bits corresponding to the texture coordinate information; Writing the coded bits corresponding to the base mesh and the coded bits corresponding to the texture coordinate information into the bitstream respectively.

31. A decoder includes: A parsing part configured to parse the bitstream to determine a first base mesh and texture coordinate information of the first base mesh; A subdivision and deformation part configured to perform mesh subdivision and deformation based on the first base mesh to determine a reconstructed deformed mesh; A coordinate determination part configured to determine texture coordinate information of the reconstructed deformed mesh based on the texture coordinate information of the first base mesh and the reconstructed deformed mesh.

32. An encoder includes: A mesh information determination part configured to determine a base mesh corresponding to a current image; Determining texture coordinate information based on the base mesh; An encoding part configured to encode the base mesh and the texture coordinate information and write the obtained coded bits into the bitstream.

33. A decoder includes: A first memory and a first processor; The first memory stores a computer program that can run on the first processor, and when the first processor executes the program, it implements the method according to any one of claims 1-25.

34. An encoder includes: A second memory and a second processor; The second memory stores a computer program that can run on the second processor, and when the second processor executes the program, it implements the method according to any one of claims 26-30.

35. A bitstream, wherein, The bitstream is generated by performing bit encoding on the information to be encoded; wherein, the information to be encoded at least includes: the base mesh and the texture coordinate information.

36. A storage medium, wherein, The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the method according to any one of claims 1 to 25, or implements the method according to any one of claims 26-30.

Images