Slice-based grid decoding method, slice-based grid coding method, and related device

Abstract

The present application belongs to the field of coding and decoding. Disclosed are a slice-based grid decoding method, a slice-based grid coding method, and a related device. The slice-based grid decoding method comprises: a decoding end acquiring at least one inter-frame code stream of a basic grid to be decoded and at least one intra-frame code stream of the basic grid to be decoded; the decoding end decoding the at least one inter-frame code stream in an inter-frame decoding mode, so as to obtain at least one inter-frame slice; the decoding end decoding the at least one intra-frame code stream in an intra-frame decoding mode, so as to obtain at least one intra-frame slice; and the decoding end merging the at least one inter-frame slice and the at least one intra-frame slice, so as to obtain a reconstructed basic grid of the basic grid to be decoded.

Description

Tile-based mesh decoding method, encoding method and related device

[0001] Cross-reference to Related Applications

[0002] This application claims priority to the Chinese patent application No. 202410457668.2 filed on April 16, 2024 in China, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of coding and decoding, and more particularly, to a tile-based mesh decoding method, an encoding method and related devices. BACKGROUND

[0004] For a three-dimensional mesh, an encoding end can first preprocess an original mesh to generate a base mesh and a deformed mesh. Specifically, the encoding end first subdivides the base mesh to obtain a subdivided mesh of the base mesh, and then adjusts the subdivided mesh based on the original mesh to obtain the deformed mesh.

[0005] In related technologies, the encoding end can encode the base mesh by using an intra-frame encoding mode or an inter-frame encoding mode to obtain a base mesh code stream, and the decoding end can decode the base mesh code stream by using an intra-frame decoding mode or an inter-frame decoding mode to obtain a reconstructed base mesh.

[0006] However, if the encoding end only encodes the entire base mesh by using the intra-frame encoding mode or the inter-frame encoding mode, information redundancy or excessively low code rate may exist in the base mesh code stream in the time domain, which may in turn reduce the coding and decoding efficiency of the entire base mesh. SUMMARY

[0007] Embodiments of the present application provide a tile-based mesh decoding method, an encoding method and related devices, which can improve decoding efficiency.

[0008] In a first aspect, a tile-based mesh decoding method is provided, comprising:

[0009] A decoding end obtains at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh.

[0010] The decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame tile.

[0011] The decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame tile.

[0012] The decoding end merges the at least one inter-frame tile and the at least one intra-frame tile to obtain a reconstructed base mesh of the to-be-decoded base mesh.

[0013] In a second aspect, a slice-based mesh coding method is provided, comprising:

[0014] determining, by an encoding end, at least one inter-slice of a raw mesh to be encoded and at least one intra-slice of the raw mesh to be encoded;

[0015] encoding, by the encoding end, the at least one inter-slice using an inter-coding mode to obtain at least one inter-code stream;

[0016] encoding, by the encoding end, the at least one intra-slice using an intra-coding mode to obtain at least one intra-code stream;

[0017] merging, by the encoding end, the at least one inter-code stream and the at least one intra-code stream to obtain a base mesh code stream.

[0018] In a third aspect, a slice-based mesh decoding apparatus is provided, comprising:

[0019] an obtaining unit configured to obtain at least one inter-code stream of a base mesh to be decoded and at least one intra-code stream of the base mesh to be decoded;

[0020] a first decoding unit configured to decode the at least one inter-code stream using an inter-decoding mode to obtain at least one inter-slice;

[0021] a second decoding unit configured to decode the at least one intra-code stream using an intra-decoding mode to obtain at least one intra-slice;

[0022] a merging unit configured to merge the at least one inter-slice and the at least one intra-slice to obtain a reconstructed base mesh of the base mesh to be decoded.

[0023] In a fourth aspect, a slice-based mesh coding apparatus is provided, comprising:

[0024] a determining unit configured to determine at least one inter-slice of a raw mesh to be encoded and at least one intra-slice of the raw mesh to be encoded;

[0025] a first encoding unit configured to encode the at least one inter-slice using an inter-coding mode to obtain at least one inter-code stream;

[0026] a second encoding unit configured to encode the at least one intra-slice using an intra-coding mode to obtain at least one intra-code stream;

[0027] a merging unit configured to merge the at least one inter-code stream and the at least one intra-code stream to obtain a base mesh code stream.

[0028] In a fifth aspect, an electronic device is provided, comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implementing steps of the method according to the first aspect, or implementing steps of the method according to the second aspect.

[0029] In a sixth aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the communication interface is configured to obtain at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid; and the processor is configured to:

[0030] decode the at least one inter-frame code stream using an inter-frame decoding mode to obtain at least one inter-frame slice;

[0031] decode the at least one intra-frame code stream using an intra-frame decoding mode to obtain at least one intra-frame slice;

[0032] merge the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

[0033] In a seventh aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the processor is configured to:

[0034] determine at least one inter-frame slice of a to-be-encoded original grid and at least one intra-frame slice of the to-be-encoded original grid;

[0035] encode the at least one inter-frame slice using an inter-frame encoding mode to obtain at least one inter-frame code stream;

[0036] encode the at least one intra-frame slice using an intra-frame encoding mode to obtain at least one intra-frame code stream;

[0037] merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base grid code stream.

[0038] In an eighth aspect, an electronic device is provided, comprising a memory configured to store data, and a processing circuit configured to implement steps of the method according to the first aspect, or implement steps of the method according to the second aspect.

[0039] In a ninth aspect, a readable storage medium is provided, the readable storage medium storing programs or instructions, the programs or instructions, when executed by a processor, implementing steps of the method according to the first aspect, or implementing steps of the method according to the second aspect.

[0040] In a tenth aspect, a codec system is provided, comprising: an encoding end and a decoding end, the decoding end being configured to perform the steps of the method according to the first aspect, and the encoding end being configured to perform the steps of the method according to the second aspect.

[0041] In an eleventh aspect, a chip is provided, comprising a processor and a communication interface, the communication interface being coupled to the processor, and the processor being configured to run a program or an instruction to implement the steps of the method according to the first aspect or the steps of the method according to the second aspect.

[0042] In a twelfth aspect, a computer program / program product is provided, which is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method according to the first aspect or the steps of the method according to the second aspect.

[0043] In the embodiments of the present application, the decoding end obtains at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh; the decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice; the decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice; and the decoding end combines the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base mesh of the to-be-decoded base mesh. That is, the decoding end avoids directly decoding the entire base mesh by using the intra-frame decoding mode or the inter-frame decoding mode, and the decoding effect can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] FIG. 1 is a schematic diagram of a codec system provided by the embodiments of the present application.

[0046] FIG. 2a is an encoding flowchart of V3C provided by the embodiments of the present application.

[0047] FIG. 2b is a decoding flowchart of V3C provided by the embodiments of the present application.

[0048] FIG. 3 is an encoding flowchart of VDMC provided by the embodiments of the present application.

[0049] FIG. 4 is an example of the principle of mesh simplification provided by the embodiments of the present application.

[0050] FIG. 5 is an example of the basic idea of the subdivision and morphing module provided in an embodiment of the present application.

[0051] FIG. 6 is an example of the principle of the subdivision provided in an embodiment of the present application.

[0052] FIG. 7 is a schematic flowchart of the compression process of the base mesh provided in an embodiment of the present application.

[0053] FIG. 8 is an example of the pattern of the triangular face provided in an embodiment of the present application.

[0054] FIG. 9 is a schematic flowchart of the texture map conversion provided in an embodiment of the present application.

[0055] FIG. 10 is a decoding flowchart of the VDMC provided in an embodiment of the present application.

[0056] FIG. 11 is a schematic flowchart of the slice-based mesh decoding method provided in an embodiment of the present application.

[0057] FIG. 12 is an example of the base mesh bitstream provided in an embodiment of the present application.

[0058] FIG. 13 is another example of the base mesh bitstream provided in an embodiment of the present application.

[0059] FIG. 14 is still another example of the base mesh bitstream provided in an embodiment of the present application.

[0060] FIG. 15 is an example of the adjustment process of the texture coordinates provided in an embodiment of the present application.

[0061] FIG. 16 is a schematic diagram of the slice-based mesh decoding flow provided in an embodiment of the present application.

[0062] FIG. 17 is a schematic diagram of the decoding flow of the displacement bitstream provided in an embodiment of the present application.

[0063] FIG. 18 is a schematic flowchart of the slice-based mesh encoding method provided in an embodiment of the present application.

[0064] FIG. 19 is a schematic diagram of the generation process of at least one inter-frame slice and at least one intra-frame slice provided in an embodiment of the present application.

[0065] FIG. 20 is an example of the registered base mesh, the original mesh to be encoded, the inter-frame slice, the initial intra-frame slice, the intra-frame slice, and the base mesh provided in an embodiment of the present application.

[0066] FIG. 21 is an example of the generation process of the registered base mesh and the configuration subdivision morphing mesh provided in an embodiment of the present application.

[0067] FIG. 22 is an example of the generation process of at least one inter-frame slice and at least one initial intra-frame slice provided in an embodiment of the present application.

[0068] FIG. 23 is an example of a boundary and a boundary point according to an embodiment of the present application.

[0069] FIG. 24 is a schematic diagram of boundary stretching according to an embodiment of the present application.

[0070] FIG. 25 is an example of obtaining at least one intra slice based on at least one initial intra slice according to an embodiment of the present application.

[0071] FIG. 26 is a schematic diagram of a process of generating displacement information according to an embodiment of the present application.

[0072] FIG. 27 is a schematic diagram of inter slice encoding and intra slice encoding according to an embodiment of the present application.

[0073] FIG. 28 is a schematic diagram of reconstructing a deformed mesh according to an embodiment of the present application.

[0074] FIG. 29 is a schematic flowchart of a mesh encoding process based on slices according to an embodiment of the present application.

[0075] FIG. 30 is a schematic diagram of an encoding process of displacement information according to an embodiment of the present application.

[0076] FIG. 31 is a schematic block diagram of a mesh decoding apparatus based on slices according to an embodiment of the present application.

[0077] FIG. 32 is a schematic block diagram of a mesh encoding apparatus based on slices according to an embodiment of the present application.

[0078] FIG. 33 is a schematic block diagram of an electronic device according to an embodiment of the present application.

[0079] FIG. 34 is a schematic structural diagram of a terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0081] The terms "first", "second", and the like in the specification are used to distinguish between similar objects, and are not used to describe a particular sequential or chronological order. It is to be understood that such terms are used interchangeably and that "first", "second", and the like can be interchanged, where appropriate, to distinguish between similar objects in order to describe the embodiments of the present application in the order in which they are presented, and that the objects distinguished by "first", "second", and the like are generally of a class, and are not limited in number, for example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three scenarios, namely, scenario one: including A and not including B; scenario two: including B and not including A; scenario three: including A and including B. The character " / " generally represents that the objects before and after are in an "or" relationship.

[0082] Before introducing the technical solutions provided by the embodiments of the present application, the meanings of some terms therein will be introduced first.

[0083] Three-dimensional mesh: A three-dimensional mesh is a three-dimensional object surface composed of countless polygons in space, and a polygon is composed of vertices and edges.

[0084] In recent years, with the rapid development of multimedia technology, relevant research results have been rapidly industrialized and have become an important part of people's lives. Three-dimensional models have become a new generation of digital media after audio, images, and video. Three-dimensional mesh is a common three-dimensional model representation method. Compared with traditional images, video, and other multimedia, three-dimensional mesh models have stronger interactivity and realism, making them more and more widely used in various fields such as business, manufacturing, construction, education, medicine, entertainment, art, and military.

[0085] Although there are many kinds of three-dimensional mesh representation methods at present, triangular mesh is still the most common representation method. A three-dimensional mesh can be regarded as being composed of three basic elements, namely vertices, edges, and faces. Vertices are the most basic elements in the mesh, which define a position in a three-dimensional space. An edge is a line segment connecting two vertices in the mesh. A face can be regarded as a polygon formed by a closed path of edges. For a triangular mesh, each face is a triangle.

[0086] The information contained in a mesh is generally classified into three categories: geometry information, connectivity information, and attribute information. The geometry information is the position of each vertex of the mesh in the three-dimensional space. The connectivity information describes the relationship between elements in the mesh, i.e., the connection relationship between vertices. The attribute information is optional, which can associate attributes to the corresponding mesh elements (such as vertex color, normal vector, etc. can be associated with the mesh vertex). The mesh parameterization can also be used to map the mesh from the three-dimensional space to a two-dimensional planar region, and the mapping relationship is generally described by a set of parameter coordinates, referred to as UV coordinates or texture coordinates, which are associated with the mesh vertex. The two-dimensional mapping can be used to represent high-resolution attribute information, such as texture, normal vector, etc.

[0087] FIG. 1 is a schematic diagram of a coding system 10 according to an embodiment of the present application. The technical solution of the present application relates to coding (CODEC) of mesh data, including encoding or decoding.

[0088] As shown in FIG. 1, the coding system 10 includes a source device 100 that provides encoded mesh data to be decoded and displayed by a destination device 110. Specifically, the source device 100 provides mesh data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 can include any one or more of a desktop computer, a notebook (i.e., laptop) computer, a tablet computer, a set-top box, a mobile phone, a wearable device (e.g., a smart watch or a wearable camera), a television, a camera, a display device, a vehicle-mounted device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a digital media player, a video game console, a video conferencing device, a video streaming device, a broadcast receiver device, a broadcast transmitter device, a spacecraft, an airplane, a robot, a satellite, etc.

[0089] In the example of FIG. 1, the source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. The destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. The source device 100 represents an example of an encoding device, while the destination device 110 represents an example of a decoding device. In other examples, the source device 100 and the destination device 110 can not include some of the components in FIG. 1, or can include other components not shown in FIG. 1. For example, the source device 100 can acquire mesh data through an external capturing device. Likewise, the destination device 110 can interface with an external display device, rather than include an integrated display device. For another example, the memory 102, the memory 113 can be external memories.

[0090] Although FIG. 1 depicts source device 100 and destination device 110 as separate devices, in some examples, the source and destination devices can be integrated in a single device. In such embodiments, the source device 100 corresponding functionality and the destination device 110 corresponding functionality can be implemented using the same hardware or software, or separate hardware or software, or any combination thereof.

[0091] In some examples, source device 100 and destination device 110 can engage in unidirectional data transmission or bidirectional data transmission. If bidirectional data transmission, source device 100 and destination device 110 can operate in a substantially symmetrical manner. That is, each of source device 100 and destination device 110 can include an encoder and a decoder.

[0092] Data source 101 represents a source of mesh data (i.e., raw, uncoded mesh data) and provides the encoder 200 with mesh data containing the mesh data that the encoder 103 encodes. Source device 100 can include a capture device (e.g., a camera device, a sensor device, or a scanning device), an archive including previously captured mesh data, or a feed interface to receive mesh data from a data content provider. Among others, the camera device can include a normal camera, a stereo camera, and a light field camera, etc., the sensor device can include a laser device, a radar device, etc., and the scanning device can include a three-dimensional laser scanning device, etc. The mesh data can be obtained by capturing a real-world visual scene through the capture device. Alternatively, data source 101 can generate computer graphics based data as source data, or combine real-time data, archived data, and computer generated data. For example, the data source generates mesh data based on a virtual object (e.g., a virtual three-dimensional object and a virtual three-dimensional scene obtained by three-dimensional modeling).

[0093] Memory 102 of source device 100 and memory 113 of destination device 110 represent general purpose memories. In some examples, memory 102 can store raw data from data source 101, and memory 113 can store decoded mesh data from decoder 300. Additionally or alternatively, memory 102, 113 can store software instructions executable by, for example, encoder 200 and decoder 300, respectively. Although memory 102 and memory 113 are shown separately from encoder 200 and decoder 300 in this example, it should be understood that encoder 200 and decoder 300 can also include internal memories for functionally similar or equivalent purposes. If encoder 200 and decoder 300 are deployed on the same hardware device, memory 102 and memory 113 can be the same memory. Furthermore, memory 102, 113 can store encoded mesh data, for example, output from encoder 200 and input to decoder 300. In some examples, portions of memory 102, 113 can be allocated as one or more mesh buffers, for example, for storing raw, decoded, or encoded mesh data.

[0094] In some examples, source device 100 can output encoded data from output interface 104 to memory 113. Similarly, destination device 110 can access encoded data from memory 113 via input interface 111. Memory 113, or memory 102, can comprise any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Versatile Discs (DVD), Compact Disc Read-Only Memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded mesh data.

[0095] Output interface 104 can include any type of medium or device capable of transmitting encoded mesh data from source device 100 to destination device 110. For example, output interface 104 can include a transmitter or a transceiver, such as an antenna, configured to transmit encoded mesh data from source device 100 directly to destination device 110 in real-time. The encoded mesh data can be modulated according to a communication standard of a wireless communication protocol and transmitted to destination device 110.

[0096] Communication medium 120 can include transient media, such as a wireless broadcast or wired network transmission. For example, communication medium 120 can include a Radio Frequency (RF) spectrum or one or more physical transmission lines (e.g., cable). Communication medium 120 can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. Communication medium 120 can also be in a form of a storage medium, such as a hard disk, flash drive, compact disk, digital net grid disk, Blu-ray disk, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded grid data.

[0097] In some embodiments, communication medium 120 can include routers, switches, base stations, or any other devices that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) can receive the encoded grid data from source device 100 and provide to destination device 110, e.g., via network transmission to destination device 110. The server can include a web server (e.g., for a website), a server configured to provide a file transfer protocol service such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol, a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or evolved Multimedia Broadcast Multicast Service (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols such as MPEG Media Transport (MMT) protocol, Dynamic Adaptive Streaming over HTTP (DASH) protocol, HTTP Live Streaming (HLS) protocol, or Real Time Streaming Protocol (RTSP), etc.

[0098] Destination device 110 can access the encoded mesh data from a server, for example, through a wireless channel (e.g., a Wi-Fi connection) or a wired connection (e.g., a Digital Subscriber Line (DSL), a cable modem, etc.) for accessing encoded mesh data stored on the server.

[0099] Output interface 104 and input interface 111 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to the IEEE 802.11 standards or the IEEE 802.15 standards (e.g., ZigBee™), the Bluetooth standard, etc., or other physical components. In examples where output interface 104 and input interface 111 comprise wireless components, output interface 104 and input interface 111 can be configured to transfer data, such as encoded mesh data, according to WIFI, Ethernet, cellular networks (such as 4G, LTE (Long-Term Evolution), LTE-Advanced, 5G, 6G, etc.), or the like.

[0100] Input interface 111 of destination device 110 receives the encoded bitstream from communication medium 120. The encoded bitstream can include high-level syntax elements and encoded data units (e.g., sequences, groups of pictures, pictures, slices, blocks, etc.), where the high-level syntax elements are used to decode the encoded data units to yield decoded mesh data. Display device 114 displays the decoded mesh data to a user. Display device 114 can comprise a Cathode Ray Tube (CRT), a Liquid-Crystal Display (LCD), a plasma display, an Organic Light-Emitting Diode (OLED) display, or other type of display device. In some examples, destination device 110 can not have display device 114, for example, if the decoded mesh data is used to determine a location of a physical object, display device 114 can be replaced with a processor.

[0101] The encoder 200 and the decoder 300 can be implemented as one or more of various processing circuitry, which can include one or more microprocessors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), discrete logic circuitry, hardware, or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure.

[0102] In almost all application fields using three-dimensional mesh (such as computational simulation, entertainment, medical imaging, digital cultural relics, computer design, e-commerce, etc.), with the increasing demand for three-dimensional mesh models in visual effects, the models are becoming more and more complex, and the accuracy of the models is also increasing, so the amount of data required to represent the three-dimensional mesh is also increasing. The above problems lead to the increasing complexity of processing, visualization, transmission and storage of three-dimensional mesh. Three-dimensional mesh compression can be considered as a way to solve the above problems, which reduces the size of the model data, and is beneficial to the processing, storage and transmission of three-dimensional mesh, so it is necessary to propose an efficient and general three-dimensional mesh compression algorithm.

[0103] Recently, the international standardization organization MPEG in the field of audio and video coding compression has begun to develop a compression standard for three-dimensional mesh, namely Video-Based Dynamic Mesh Coding (VDMC), which is based on the existing Visual Volumetric Video-based Coding (V3C) standard. The V3C standard provides a general method for compressing three-dimensional models, which can be presented in the form of point cloud, mesh or panoramic video. The compression method of three-dimensional mesh model is compatible with the standard, which helps to promote the method and its applicability. Therefore, it is of great significance to optimize the three-dimensional mesh coding method in VDMC and combine the optimization method with the V3C standard. One possible optimization method is to optimize the basic mesh coding, which divides a basic mesh or a sub-mesh of a basic mesh into two kinds of slices (intra and inter) for coding.

[0104] V3C standard: The V3C standard provides a method for encoding and decoding various three-dimensional media through video or image encoding technology. Specifically, it converts three-dimensional media content from a three-dimensional representation to multiple two-dimensional representations (referred to as V3C components) through projection or other means before encoding, and then encodes the two-dimensional representations using existing video or image encoding technology. An example diagram of the V3C standard is shown in FIGS. 2a and 2b, and the V3C components mainly include occupancy components, geometry components, and attribute components. The occupancy component can represent which areas in the two-dimensional representation are associated with the data of the three-dimensional representation; the geometry component represents information related to the position of the three-dimensional data in space, and the attribute component can provide attribute information corresponding to the vertices, such as material, texture, etc. In addition, the components also contain information on how to reconstruct the three-dimensional model from these components, which is referred to as atlas information.

[0105] The atlas information is used to associate all the components, and additional information for reconstructing from two dimensions to three dimensions is also included in the atlas component. The atlas is composed of multiple basic units, which are referred to as patches. For each patch, it represents a block of area in the available two-dimensional components and contains the information needed to project the area back to the three-dimensional space.

[0106] VDMC: VDMC is a standard for compressing three-dimensional meshes developed by MPEG, and its main idea is to compress three-dimensional meshes using existing V3C standards. Since three-dimensional meshes have connection information that needs to be encoded, the specific encoding process is slightly different from V3C, and the syntax semantics and decoding operations of the V3C standard decoding end need to be extended to support the decoding and reconstruction of three-dimensional meshes.

[0107] The overall framework of the encoder and decoder provided by the embodiments of the present application is introduced below using VDMC as an example.

[0108] The overall framework of the encoding end is shown in FIG. 3. For an input mesh, first, simplification is performed by a simplification module, then new texture coordinates are generated by mesh parameterization, then the parameterized mesh is subdivided and deformed, i.e. new vertices are inserted into the mesh according to a specific subdivision method and the distance from the subdivided mesh vertices to the nearest neighbor vertices of the input mesh is calculated, which is called displacement information. Then, the position of the vertices of the mesh before subdivision and deformation is adjusted according to the displacement information, and the adjusted mesh is called the base mesh and is input into a base mesh encoding module for compression. When the base mesh is encoded, it is divided into three types of sub-meshes for independent encoding. After the base mesh is encoded, it is reconstructed, and then the order of the displacement is adjusted according to the vertex order of the reconstructed base mesh. Then, the vertex displacement information after the order is adjusted is first wavelet transformed, the transformed coefficients are quantized, then the quantized coefficients are arranged into a two-dimensional image according to a specific scanning order, and a video encoder is used to encode the two-dimensional image. Then, the reconstructed displacement information is applied to the subdivided base mesh to obtain the reconstructed subdivided and deformed mesh, and the mesh, the original input mesh and the corresponding texture map are input into a corresponding texture map conversion module to obtain the texture map corresponding to the reconstructed mesh, and the texture map is also encoded using a video encoder. For the parameters used in the encoding process, such as the type of video encoder, the type of mesh encoder, the transformation parameters, the quantization parameters, etc., are transmitted to the decoding end through auxiliary information.

[0109] The specific process of encoding is described below:

[0110] (1) Mesh simplification.

[0111] Mesh simplification is to simplify the current input mesh to a base mesh with relatively fewer vertices and faces, and to maintain the shape of the original mesh as much as possible. The focus of mesh simplification is the simplification operation and the corresponding error metric. A feasible mesh simplification operation is shown in FIG. 4, in which the vertices at both ends of an edge are combined into one vertex and the connection between the two vertices is deleted. This process is repeated throughout the mesh according to certain rules to reduce the number of faces and vertices of the mesh to the target value.

[0112] During the simplification process, an error metric can be selected to optimize the simplification result. For example, the sum of the equation coefficients of all adjacent faces of a vertex can be selected as the error metric of the vertex, and the error metric of the corresponding edge is the sum of the error metrics of the two vertices on the edge. In other words, the error generated by the combination of an edge is the sum of the distances from the combined vertex to all the adjacent planes of the original two vertices of the edge.

[0113] After the simplification operations and corresponding error metrics are determined, the mesh simplification is performed iteratively. First, the vertex error of the initial mesh is calculated, and then the error of each edge is obtained. Then, each edge is sorted by error from small to large, and the edge with the smallest error is selected for merging. At the same time, the position of the merged vertex is calculated, and the error of all edges related to the merged vertex is updated. That is, the order of edge sorting is updated to ensure that each iteration is based on the global error metric. Through iteration, the mesh is simplified to meet the number required by lossy coding.

[0114] (2) Mesh parameterization.

[0115] Mesh parameterization is used to generate corresponding texture coordinates for the mesh. Mesh parameterization can be performed by algorithms such as Isochart algorithm, orthogonal projection algorithm, etc. In this coding framework, both of the above schemes can be used for parameterization of the reconstructed base mesh. The following briefly introduces the two algorithms.

[0116] Isochart algorithm: This algorithm uses spectral analysis to achieve stretch-driven three-dimensional mesh parameterization, which unfolds, slices, and packs the three-dimensional mesh to a two-dimensional texture domain. A stretch threshold is set, and the algorithm is outlined as follows:

[0117] 1. Calculate the surface spectral analysis to provide an initial parameterization;

[0118] 2. Perform iterations of stretch optimization;

[0119] 3. If the stretch of this derived parameterization is less than the threshold, stop;

[0120] 4. Perform surface spectral clustering to divide the surface into charts;

[0121] 5. Optimize the chart boundaries using a graph cut algorithm;

[0122] 6. Iteratively split the charts until the stretch criterion is met.

[0123] Orthogonal projection algorithm:

[0124] The orthogonal projection algorithm is a projection-based mesh parameterization method that generates texture coordinates for the mesh through orthogonal projection. Its main process includes:

[0125] 1. Calculate the mesh attributes, including the adjacent faces of each face and the area and normal vector of each face;

[0126] 2. Determine the projection plane of each face according to the normal vector;

[0127] 3. Start clustering all faces into a connected region according to the projection plane, first select the starting face for clustering;

[0128] 4. Iterate from the starting face to determine whether the adjacent faces of the face added to the connected region can be added to the connected region;

[0129] 5. After each connected region is iterated, multiple connected regions are obtained;

[0130] 6. Determine whether to merge adjacent connected areas based on the error metric;

[0131] 7. Check whether there are overlapping areas during projection, remove the overlapping surfaces and regenerate connected areas;

[0132] 8. Arrange all the projected areas into a two-dimensional image.

[0133] (3) Subdivision deformation.

[0134] The subdivision and deformation modules are optional modules that can improve the quality of the mesh reconstructed by the decoder. They can be omitted if the base mesh quality already meets the application requirements.

[0135] The basic concept behind the subdivision and deformation modules is illustrated in Figure 5. The same concepts are applied to the input 3D mesh to generate displacement vector information. In Figure 5, the input 2D curve (represented by a 2D polyline), referred to as the "original" curve, is first downsampled to generate a base curve / polyline, referred to as the "simplified" curve. The subdivision scheme is then applied to the resulting simplified polyline to generate the "subdivided" curve. The subdivided polyline is then deformed to obtain a better approximation of the original curve. Specifically, a geometric displacement vector (indicated by the arrows in Figure 5) is calculated for each vertex of the subdivided mesh, so that the shape of the subdivided curve approximates the shape of the original curve as closely as possible. The output of the module, a deformed mesh containing displacement information, is then added to the coordinates of each vertex of the subdivided mesh. The same deformation process is also applied to the attribute information corresponding to the vertex, resulting in the corresponding attribute displacement vector.

[0136] The subdivision and deformation modules subdivide the input mesh. The subdivision scheme can be selected arbitrarily. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four subtriangles in each subdivision iteration, as shown in Figure 6. 0 is the mesh before subdivision, S 1 is the grid of one iteration, S 2 A mesh that is iteratively subdivided twice. A new vertex is introduced in the middle of each edge. The subdivision of geometric information and attribute information is performed independently because the connection relationship between geometric information and attribute information is usually different.

[0137] The scheme calculates the midpoint v of the newly introduced edge (v1, v2) 12 Position Pos(v12 ) is shown as formula (1) in the following way:

[0138] where Pos(v1) and Pos(v2) are the geometric coordinates of vertex v1 and v2 respectively.

[0139] For the subdivided mesh, finding its nearest neighbor of each point on the original input mesh (including the points on the original mesh surface) can be accelerated by data structures such as k-dimensional tree (KD-tree). By calculating the distance between each vertex of the subdivided mesh and its nearest neighbor on the original input mesh, the displacement vector of the geometric coordinates of each vertex of the subdivided mesh is obtained.

[0140] The subdivision and deformation module can also convert the calculated vertex displacement coordinates from the Cartesian coordinate system to the local coordinate system. A feasible method is to convert each vertex displacement coordinate into the coordinate in the coordinate system composed of the corresponding vertex normal vector and the two vectors tangent to the normal vector. The specific conversion process is shown in formula (2):

[0141] where, represents the vertex displacement before coordinate system conversion, represents the vertex displacement after coordinate system conversion, are mutually orthogonal unit vectors, and is the vertex normal vector, is the two vectors tangent to . For the calculation of the vertex normal vector, a feasible method is to equal to the area weighted sum of the adjacent face normal vectors of the vertex.

[0142] (4) Compression of the base mesh.

[0143] The base mesh compression module compresses the base mesh information output by the subdivision and deformation module. There are mainly two different modes of base mesh compression, which are respectively: intra-frame mode and inter-frame mode. As shown in FIG. 7, in the intra-frame mode, the input of the base mesh compression module is a three-dimensional mesh, including geometric coordinates, connection relationship and attribute information associated with the vertex. In the inter-frame mode, the input of the base mesh compression module is the motion vector and its identifier, and possibly existing intra-frame sub-mesh. After encoding, the encoded mesh needs to be reconstructed to provide for the subsequent modules for processing. The encoding of the two modes is briefly introduced as follows:

[0144] Intra mode: The mesh coding in intra mode can use the existing mesh coder to encode the input base mesh, such as Draco, etc. The mesh coder type is coded by auxiliary information and transmitted to the decoding end. The following introduces a usable Draco.

[0145] The main idea of Draco to compress static mesh is mesh compression driven by connection relationship. It traverses all the faces of the mesh according to a specific way, labels each face according to a specific rule, encodes all the labels of the faces obtained by traversal, that is, the connection relationship of the mesh is encoded. Then all the vertex coordinate information is encoded according to the traversal connection relationship.

[0146] The main process of Draco to encode mesh includes: first, for the input mesh, the connection relationship is generated according to its geometric information, that is, the connection relationship of the vertex in three-dimensional space. After the connection relationship of the face is constructed, the initial face is selected to start traversing all the faces of the current mesh, that is, the traversal symbol is generated. Here, the Edgebreaker algorithm is used to generate the traversal symbol, which divides the triangle face into five modes according to the state of the triangle face when the current corner is traversed, for example, L mode, C mode, R mode, S mode and E mode as shown in Figure 8. Among them, in C mode, the triangle (such as X shown in the figure) is completely located in the interior of the mesh and does not intersect with any boundary, and all the vertices are located in the interior of the mesh, so no additional information is needed to identify the boundary. In L mode and R mode, one vertex (such as v shown in the figure) of the triangle is located on the boundary. L mode indicates that the boundary is on the left side of the triangle, and R mode indicates that the boundary is on the right side of the triangle, which needs additional information to identify the direction and position of the boundary. S mode is used to process the branch situation in the triangular mesh, which divides the figure into different parts. In E mode, all the edges of the triangle are located on the boundary of the mesh, which indicates that the triangle is a complete boundary triangle, and the position and direction of all the edges need to be recorded.

[0147] The five modes also define the direction of the next face after traversing the current face (i.e. the arrow shown in the figure). According to the above traversal method, a corresponding symbol is generated for each face defined by the geometric information of the current mesh, and then the symbols are entropy coded to obtain the code stream of the connection relationship defined by the geometric information of the current mesh. At the same time, the traversal of each face also obtains the order of the corresponding vertices, which is transmitted to the geometric information encoder, rearranged according to the traversal order, and quantized according to the predetermined quantization parameter, and then predicted. The prediction uses the parallelogram prediction method.

[0148] Inter-frame mode: In inter-frame mode, the motion vector and the identifier need to be encoded. The motion vector identifier is an array composed of numbers greater than or equal to 0, and the size of the array is the same as the number of vertices of the base mesh of the reference frame corresponding to the time domain motion vector identifier. The array can be directly encoded using existing entropy encoding algorithms, such as CABAC, etc.

[0149] (5) Reconstructing the base mesh.

[0150] After the base mesh is encoded, the base mesh needs to be reconstructed. For the mesh encoded in intra-frame mode, the encoded mesh is decoded to obtain the reconstructed base mesh.

[0151] For the base mesh information encoded in inter-frame mode, the inter-frame sub-mesh needs to be reconstructed according to the motion vector and the identifier. If there is an intra-frame sub-mesh in the inter-frame mode, the reconstructed inter-frame sub-mesh also needs to be merged with the intra-frame sub-mesh to obtain the reconstructed base mesh. After obtaining the reconstructed base mesh, it needs to be subdivided for subsequent vertex displacement encoding and reconstruction of the deformed mesh.

[0152] (6) Encoding of vertex displacement.

[0153] To encode the vertex displacement, first, the order of the vertices of the optimized deformed mesh needs to be adjusted according to the subdivided reconstructed base mesh. Then, the coordinates of the vertices of the optimized deformed mesh are subtracted from the coordinates of the vertices of the subdivided reconstructed base mesh to obtain displacement vectors. Next, wavelet transform and quantization are performed on the displacement to obtain quantized wavelet transform coefficients. Then, the wavelet transform coefficients of the current frame are arranged into a two-dimensional image to form a YUV video, and the displacement code stream is obtained by encoding the video using a video encoder. Next, each module will be introduced in detail:

[0154] 1. Adjusting the order of the vertices of the deformed mesh and calculating the displacement.

[0155] The main function of this module is to adjust the order of the vertices of the optimized deformed mesh according to the order of the vertices of the subdivided reconstructed base mesh, so that the order of the vertices of the optimized deformed mesh is the same as the order of the vertices of the subdivided base mesh. Then, the coordinates of the vertices are subtracted one by one to calculate a displacement vector for each vertex. There is an optional step in this process, which is to convert the coordinate system of the vertex displacement from Cartesian coordinates to local coordinates.

[0156] 2. Wavelet transform.

[0157] A transform can be applied to the displacement vector to reduce the correlation between its data. An optional transform is linear wavelet transform, and its prediction process is defined as shown in equation (3):

[0158] where v is the newly inserted midpoint on edge (v1, v2), Signal(v), Signal(v1) and Signal(v2) are the displacement vectors of vertex v, v1 and v2 respectively. The displacement vector of vertex v is predicted and then updated, the update process is defined as shown in equation (4):

[0159] where v * is the set of all adjacent vertices of vertex v. The transformed displacement vector is called wavelet coefficient.

[0160] 3. Quantization of coefficients.

[0161] The transformed displacement vector, i.e. the wavelet coefficient, can be quantized. There are many ways to quantize, one method is shown in equation (5) and (6):

[0162] where disp[v] represents the transformed value of the displacement vector of the vth vertex, d[k] represents the kth value of the displacement vector, floor represents the floor function. bitDepthPosition represents the bit depth of the current grid vertex geometry position, qp[k] represents the quantization parameter of the kth coefficient. As mentioned before, after transforming the coordinate system of the displacement vector, the normal component has more significant effect on quality than the tangential component, so a larger quantization parameter can be used for the tangential component.

[0163] Meanwhile, according to the characteristics of wavelet transform, different quantization parameters can be used for the newly generated vertices and the original vertices. That is, for the vertices after subdivision, the quantization parameter update is shown in equation (7):

[0164] where lodScale[k] represents the coefficient of the current level of detail quantization parameter.

[0165] 4. Two-dimensional arrangement.

[0166] After quantization, the quantized wavelet transform coefficients can be arranged on a two-dimensional image as follows:

[0167] a. Traverse the wavelet coefficients in order from low frequency to high frequency.

[0168] b. For each coefficient, determine its index in the NxM pixel block (for example, N = M = 16), which should be stored in it according to the raster scan order of the block.

[0169] c. Calculate the position of the corresponding NxM pixel block in the image according to Morton order.

[0170] Here, the arrangement is not limited to the above manner, and other arrangement schemes can also be used, such as zigzag order, raster order, etc. The encoder can explicitly specify the corresponding arrangement scheme in the bitstream.

[0171] 5. Video compression.

[0172] After the wavelet coefficients are arranged on the two-dimensional image, the video encoder can be directly used to encode the wavelet coefficients to obtain the bitstream of the displacement.

[0173] (7) Reconstruction of the deformed mesh

[0174] The displacement vector encoding module needs to obtain the reconstructed displacement vector value during encoding, that is, the displacement vector is obtained (consistent with the decoding end) through inverse quantization and inverse transformation. After the reconstructed geometric displacement vector is obtained, the subdivided sub-mesh is obtained according to the corresponding geometric displacement vector to obtain the reconstructed subdivided deformed mesh, which is transmitted to the texture map conversion module.

[0175] (8) Texture map conversion.

[0176] The texture map conversion module performs texture map conversion according to the input original mesh, the input original texture map, and the reconstructed deformed mesh.

[0177] As shown in FIG. 9, the steps of the texture map conversion are as follows:

[0178] 1. Calculate the texture coordinates of each pixel on the generated texture map, such as the texture coordinates P(u, v) corresponding to the pixel A(i, j).

[0179] 2. Determine whether the texture coordinates are in a triangular face after parameterization of the subdivided deformed mesh.

[0180] 3. If the texture coordinates do not belong to any triangular face, mark the pixel as an empty pixel, and then fill it with a filling algorithm.

[0181] 4. If the texture coordinates belong to a triangular face, then:

[0182] a. Mark the pixel as filled.

[0183] b. Calculate the barycentric coordinates of the pixel in the current triangular face according to the texture coordinates.

[0184] c. Map the two-dimensional texture coordinates to the three-dimensional geometric coordinates according to the barycentric coordinates and the corresponding triangular face, that is, map to the point on the subdivided deformed mesh corresponding to the texture coordinates, as shown by M(x, y, z) in FIG. 9.

[0185] d. Find the nearest point to the three-dimensional coordinate on the input original mesh, as shown as M'(x, y, z) in Fig. 9.

[0186] e. Calculate the barycentric coordinate of the three-dimensional coordinate according to the triangular face where it is located, and map it to two dimensions to calculate its texture coordinate, i.e. P'(u', v').

[0187] f. Sample the input original texture map through the texture coordinate to obtain the value A'(i', j') of the corresponding pixel position.

[0188] Assign the value to the corresponding pixel A(i, j) on the generated texture map.

[0189] (9) Texture map compression.

[0190] After obtaining the converted texture map, for the empty pixels therein, existing filling algorithms (such as the Push-Pull algorithm) can be used to fill the empty pixels. Then, existing video encoders, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used to encode the same to obtain the code stream of the output texture map. In addition, color space conversion and chroma subsampling operations can be selectively applied to make the video encoding obtain better rate-distortion performance, such as color space conversion from RGB 444 to YUV 420. For the case where multiple texture maps exist, the corresponding texture coordinates of each texture map need to be identified through auxiliary information during encoding.

[0191] (10) Auxiliary information encoding.

[0192] In the encoding process, some of the modules have alternative schemes, such as the type of mesh encoder, the type of video encoder, the mesh subdivision scheme, the spatial displacement vector transformation scheme and the transformation parameters, etc. The proposed framework allows the use of different schemes. Therefore, the selected schemes need to be passed to the decoding end to guide the correct decoding. The auxiliary information includes the type of static mesh encoder, the type of video encoder, the mesh subdivision scheme, the number of mesh iterations, the geometric displacement vector transformation scheme and the transformation parameters, the coefficient arrangement scheme, and the color conversion scheme, etc. For the mesh part encoded using the inter-frame encoding mode, the auxiliary information also includes the corresponding reference frame list, etc., which identifies the index list of the used reference frames. The auxiliary information can also include a subdivision identifier, which indicates whether the subdivision deformation operation is needed, i.e. whether it contains displacement information.

[0193] After all the modules are encoded, the base mesh part code stream, the texture coordinate part code stream, the displacement vector video code stream, the texture map video code stream, and the auxiliary information code stream are mixed to obtain the encoding end, and the final output encoding code stream.

[0194] The overall framework of the decoding end is shown in FIG. 10. For the received code stream, the decoding end first demultiplexes each part of the code stream to obtain a base mesh code stream, a displacement video code stream, a texture map video code stream, and an auxiliary information code stream. For the base mesh code stream, a mesh decoder indicated by the auxiliary information is used to decode the base mesh. The displacement video code stream and the texture map video code stream can be decoded by a video decoder. For the displacement part, the displacement needs to be extracted from the image after video decoding by a displacement decoding module, and steps such as inverse quantization and inverse transformation are performed, and then the displacement is applied to the subdivided base mesh to obtain the reconstructed deformed mesh of the decoding end. The texture map after decoding is the texture map corresponding to the reconstructed deformed mesh. A subsequent application or rendering module processes the reconstructed deformed mesh and the decoded texture map as input.

[0195] To better understand the embodiments of the present application, the decoding method and the encoding method provided by the embodiments of the present application are described below with reference to the accompanying drawings.

[0196] The decoding method provided by the embodiments of the present application can be executed by a decoding end, for example, the decoder 300 shown in FIG. 1. The encoding method provided by the embodiments of the present application can be executed by an encoding end, for example, the encoder 200 shown in FIG. 1. The decoding end and the encoding end can be implemented by software, hardware, or a combination thereof. When they are implemented by hardware, the decoding end can be referred to as a decoding end device or a decoding device, and the encoding end can be referred to as an encoding end device or a video encoding device.

[0197] FIG. 11 is a schematic flowchart of a slice-based mesh decoding method 400 according to an embodiment of the present application.

[0198] As shown in FIG. 11, the slice-based mesh decoding method 400 can include at least part of the following contents:

[0199] S410, the decoding end obtains at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh.

[0200] Exemplarily, the inter-frame code stream can also be referred to as a base mesh inter-frame code stream or a base mesh inter-frame code stream segment. The intra-frame code stream can also be referred to as a base mesh intra-frame code stream or a base mesh intra-frame code stream segment. The inter-frame code stream refers to a code stream decoded by using an inter-frame decoding mode, and the intra-frame code stream refers to a code stream decoded by using an intra-frame decoding mode.

[0201] S420, the decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice.

[0202] Exemplarily, the inter-frame slice can also be referred to as a base mesh inter-frame slice. The inter-frame slice can be a mesh region obtained by decoding the inter-frame code stream by using the inter-frame decoding mode.

[0203] Exemplarily, the inter-frame patch can be a mesh region formed by one or more face patches.

[0204] For the encoding end, the inter-frame patch can be a mesh region determined by the encoding end in a registration base mesh of an original mesh to be encoded.

[0205] For example, the inter-frame patch can be a mesh region determined by the encoding end in the registration base mesh based on face patches that do not match between a registration subdivision deformed mesh and the registration base mesh of the original mesh to be encoded. The registration base mesh is a mesh obtained by the encoding end deforming the original mesh to be encoded, and has the same number of vertices as a reference base mesh of the original mesh to be encoded and the same connectivity relationship as the reference base mesh. The registration subdivision deformed mesh is a mesh obtained by the encoding end deforming the original mesh to be encoded, and has the same number of vertices as a reference subdivision deformed mesh of the original mesh to be encoded and the same connectivity relationship as the reference subdivision deformed mesh.

[0206] S430, the decoding end decodes the at least one intra-code stream by using an intra-decoding mode to obtain at least one intra-frame patch.

[0207] Exemplarily, the intra-frame patch can also be referred to as a base mesh intra-frame patch. The intra-frame patch can be a mesh region obtained by decoding the intra-code stream by using the intra-decoding mode.

[0208] Exemplarily, the intra-frame patch can include a mesh region formed by one or more face patches.

[0209] For the encoding end, the intra-frame patch can be a mesh region determined by the encoding end in an original mesh to be encoded.

[0210] For example, the intra-frame patch can be a mesh region determined by the encoding end in the original mesh to be encoded based on face patches that do not match between a registration base mesh of the original mesh to be encoded and a registration subdivision deformed mesh of the original mesh to be encoded. The registration base mesh is a mesh obtained by the encoding end deforming the original mesh to be encoded, and has the same number of vertices as a reference base mesh of the original mesh to be encoded and the same connectivity relationship as the reference base mesh. The registration subdivision deformed mesh is a mesh obtained by the encoding end deforming the original mesh to be encoded, and has the same number of vertices as a reference subdivision deformed mesh of the original mesh to be encoded and the same connectivity relationship as the reference subdivision deformed mesh.

[0211] S440, the decoding end merges the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

[0212] For example, the decoding end can merge the at least one inter-frame slice and the at least one intra-frame slice in the order of inter-frame slice first and then intra-frame slice to obtain the reconstructed base grid of the to-be-decoded base grid. Alternatively, the decoding end can merge the at least one inter-frame slice and the at least one intra-frame slice in the order of intra-frame slice first and then inter-frame slice to obtain the reconstructed base grid of the to-be-decoded base grid.

[0213] In the slice-based grid decoding method provided by the embodiment, the decoding end avoids decoding the entire base grid directly using the intra-frame decoding mode or the inter-frame decoding mode, and thus the decoding effect can be improved.

[0214] In some embodiments, the S410 includes:

[0215] In a case where the decoding mode used by the to-be-decoded base grid is not the intra-frame decoding mode, the decoding end acquires the at least one inter-frame code stream and the at least one intra-frame code stream.

[0216] For example, if the decoding mode used by the to-be-decoded base grid is a P inter-frame mode or a Skip inter-frame mode, it indicates that the decoding mode used by the to-be-decoded base grid is not the intra-frame decoding mode, that is, the decoding end acquires the at least one inter-frame code stream and the at least one intra-frame code stream. The inter-frame decoding mode in which the motion vector is decoded is referred to as the P inter-frame mode, and the inter-frame decoding mode in which the motion vector is not decoded is referred to as the Skip inter-frame mode.

[0217] In some embodiments, the S410 includes:

[0218] The decoding end acquires a base grid code stream of the to-be-decoded base grid, the base grid code stream including header information and a data unit; and the decoding end acquires the at least one inter-frame code stream and the at least one intra-frame code stream from the data unit based on the header information.

[0219] In some embodiments, the header information includes at least one of the following:

[0220] an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0221] an identifier for indicating the data length of the at least one inter-frame code stream;

[0222] an identifier for indicating the number of the at least one inter-frame code stream;

[0223] an identifier for indicating the length of each inter-frame code stream in the at least one inter-frame code stream;

[0224] an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream except the last intra-frame code stream;

[0225] an identifier for indicating the data length of the at least one intra-frame code stream;

[0226] an identifier for indicating the number of the at least one intra-frame code stream;

[0227] an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream;

[0228] an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream except the last intra-frame code stream.

[0229] For example, the identifier in the header information of the base grid code stream is used to distinguish the inter-frame code stream and the intra-frame code stream.

[0230] For example, assuming that the base grid code stream includes one inter-frame code stream and one intra-frame code stream, if the inter-frame code stream precedes the intra-frame code stream, as shown in FIG. 12, the identifier in the header information can indicate the number of bytes occupied by the inter-frame code stream. Based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order. Of course, in other alternative embodiments, if the intra-frame code stream precedes the inter-frame code stream, the identifier in the header information can indicate the number of bytes occupied by the intra-frame code stream; the inter-frame code stream and / or the intra-frame code stream can be byte-aligned, i.e., a multiple of an integer byte.

[0231] For example, assuming that the base grid code stream includes one inter-frame code stream and multiple intra-frame code streams, if there are N intra-frame code streams, as shown in FIG. 13, the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each intra-frame code stream, or the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each of the first N-1 intra-frame code streams. Based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order.

[0232] For example, assuming that the base grid code stream includes multiple inter-frame code streams and multiple intra-frame code streams, as shown in FIG. 14, the identifier in the header information can indicate the number of inter-frame code streams, the number of intra-frame code streams, the number of bytes occupied by each inter-frame code stream, and the number of bytes occupied by each intra-frame code stream; based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order.

[0233] It should be noted that FIG. 12 to FIG. 14 are only examples of the present application and should not be construed as limiting the present application. For example, in other alternative embodiments, in the data unit, the order of the inter-frame code stream and the intra-frame code stream can also be the order of the intra-frame code stream and the inter-frame code stream. Even, the identification in the header information can indicate the order of the inter-frame code stream and the intra-frame code stream, or indicate the order of the intra-frame code stream and the inter-frame code stream. For another example, the various identifications involved above can also be independently coded or carried in the auxiliary information code stream, which is not specifically limited in the present application.

[0234] For example, in order to ensure that the inter-frame code stream and the intra-frame code stream can be distinguished at the decoding end, multiple identifications need to be set in the syntax structure to distinguish the intra-frame code stream and the inter-frame code stream. The syntax structure can be designed based on the V-DMC syntax structure, and the identifications can be placed in the header information, and the inter-frame code stream and the intra-frame code stream can be placed in the data unit. The related syntax structure of the header information is shown in Table 1:

[0235] Table 1

[0236] As shown in Table 1, the meanings of the various syntax elements are as follows:

[0237] smh_inter_segment_count: indicates the number of inter-frame code streams.

[0238] smh_intra_segment_count: indicates the number of intra-frame code streams.

[0239] smh_inter_segment_byte_counts[i]: indicates the number of bytes occupied by the ith inter-frame code stream in the data unit, used to separate the inter-frame code stream and the intra-frame code stream.

[0240] smh_intra_segment_byte_counts[i]: indicates the number of bytes occupied by the ith intra-frame code stream in the data unit, used to separate the inter-frame code stream and the intra-frame code stream.

[0241] For example, the P inter-decoding mode and the Skip inter-decoding mode can be cancelled in the data unit syntax, and the non-intra-decoding mode is uniformly used to represent. When the decoding mode of the to-be-decoded base grid is not the intra-decoding mode, the data unit thereof contains the inter-frame code stream and the intra-frame code stream.

[0242] Table 2

[0243] As shown in Table 2, the meanings of the various syntax elements are as follows:

[0244] sdu_inter_sub_mesh_unit(subMeshID)[i]: indicates data of the i-th inter-frame slice.

[0245] sdu_intra_sub_mesh_unit(subMeshID)[i]: indicates data of the i-th intra-frame slice.

[0246] In some embodiments, the S420 comprises:

[0247] The decoding end decodes a first inter-frame code stream in the at least one inter-frame code stream to obtain first indication information and to-be-decoded information of a first inter-frame slice in the at least one inter-frame slice, the first indication information being used to indicate at least one reference face slice used by the first inter-frame slice in a reference base mesh of the to-be-decoded base mesh; and the decoding end decodes the to-be-decoded information by using the inter-frame decoding mode based on the at least one reference face slice to obtain the first inter-frame slice.

[0248] For example, when the decoding end decodes the to-be-decoded information by using the inter-frame decoding mode based on the at least one reference face slice to obtain the first inter-frame slice, the to-be-decoded information can be decoded by using a corresponding decoding mode of the first inter-frame slice to obtain the first inter-frame slice. The corresponding decoding mode of the first inter-frame slice can be a P inter-frame mode or a Skip inter-frame mode. The inter-frame decoding mode in which a motion vector is decoded is referred to as the P inter-frame mode, and the inter-frame decoding mode in which a motion vector is not decoded is referred to as the Skip inter-frame mode.

[0249] For example, if the corresponding decoding mode of the first inter-frame slice is the P inter-frame mode, the first indication information needs to be decoded first, and then the motion vector information needs to be decoded; if the corresponding decoding mode of the first inter-frame slice is the Skip inter-frame mode, only the first indication information needs to be decoded. Whether the P inter-frame mode or the Skip inter-frame mode, the first indication information needs to be decoded, which is used to indicate which part of the area or which face slice in the reference base mesh is referred to by the first inter-frame slice.

[0250] For the P inter-frame mode, the decoding end further needs to decode the motion vector information. One possible decoding manner is to follow the manner of the V-DMC, and the specific steps are as follows:

[0251] 1. First, decode a Skip mode identifier of a motion vector group of a group of vertices (for example, including each vertex of the first inter-frame slice);

[0252] 2. If the Skip mode identifier indicates that the encoding manner of the group of motion vectors is the Skip mode, the motion vector does not need to be continuously decoded, and the position of the corresponding vertex of the reference base mesh can be directly assigned to the group of vertices.

[0253] 3. If the Skip mode flag indicates that the encoding mode of the set of motion vectors is not Skip mode, continue decoding the prediction mode flag of the set of motion vectors and the motion vector residual of all vertices of the set; according to the prediction mode flag, calculate the motion vector prediction value of the set of vertices using the corresponding prediction mode, and then add the decoded motion vector residual to recover the motion vector information of the set of vertices.

[0254] For the P inter mode, after the decoding end decodes the motion vector information of each vertex, it can decode and reconstruct the inter slice by superimposing the motion vector information on the vertex position corresponding to the reference base mesh of the first inter slice (i.e. the vertex position corresponding to the at least one reference patch).

[0255] In some embodiments, the first indication information is an array with a length equal to the number of patches of the reference base mesh, and the value at any position of the array is used to indicate whether the reference patch used by the first inter slice includes the patch corresponding to the any position in the reference base mesh.

[0256] For example, the decoding mode of the first indication information is to decode the array, i.e. the decoding end can use the same method as the encoding end to reconstruct the inter slice based on the patch index obtained from the decoded array, so as to recover the reconstructed inter slice at the decoding end which is completely consistent with the encoding end.

[0257] In some embodiments, the reconstructed base mesh includes the texture coordinates of the at least one inter slice and the texture coordinates of the at least one intra slice, and the method 400 further comprises:

[0258] The decoding end scales the texture coordinates of the at least one inter slice along a first direction of the texture coordinate axis to obtain the scaled texture coordinates of the at least one inter slice; and scales the texture coordinates of the at least one intra slice along the first direction to obtain the scaled texture coordinates of the at least one intra slice.

[0259] For example, scaling the texture coordinates of the at least one inter slice and the texture coordinates of the at least one intra slice at the decoding end can make the scaled texture coordinates of the at least one inter slice and the scaled texture coordinates of the at least one intra slice re-normalized to the same UV plane, i.e. the distribution ranges of the two parts of texture coordinates do not overlap, so as to generate only one texture map instead of two separate texture maps in the subsequent texture map conversion, thereby improving the decoding performance of the texture map.

[0260] In some embodiments, the decoding end scales the texture coordinates of the at least one inter slice in a first direction along a texture coordinate axis to obtain scaled texture coordinates of the at least one inter slice, including:

[0261] The decoding end multiplies the component of the texture coordinates of the at least one inter slice in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter slice.

[0262] In some embodiments, the decoding end scales the texture coordinates of the at least one intra slice in the first direction to obtain scaled texture coordinates of the at least one intra slice, including:

[0263] The decoding end multiplies the component of the texture coordinates of the at least one intra slice in the first direction by a second coefficient first, and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra slice; wherein the sum of the first coefficient and the second coefficient is 1.

[0264] FIG. 15 is an example of a texture coordinate adjustment process according to an embodiment of the present application.

[0265] As shown in FIG. 15, before the texture coordinates of the intra slice and the inter slice are adjusted, the texture coordinates of the inter slice and the intra slice are both distributed in a normalized two-dimensional UV plane. A feasible adjustment scheme is to scale the texture coordinates of the inter slice and the intra slice separately along a certain direction of UV, and then merge them into a new normalized UV plane. Taking the V direction as an example, a constant coefficient a in the range of (0, 1) can be multiplied to the V component of the texture coordinates of the inter slice; then, a constant coefficient 1-a can be multiplied to the V component of the texture coordinates of the intra slice; finally, a can be added to the V component of the texture coordinates of the intra slice. After the processing, the texture coordinates of the intra slice and the texture coordinates of the inter slice can be obtained, which are re-merged and normalized. Similarly, this strategy can also be used for adjusting the texture coordinates in the U direction. After the adjustment, the texture coordinates of the intra slice and the texture coordinates of the inter slice are re-normalized to the same UV plane, so that only one texture map needs to be generated when performing texture map conversion in the subsequent process, instead of two separate texture maps.

[0266] FIG. 16 is a schematic diagram of a slice-based mesh decoding process according to an embodiment of the present application.

[0267] As shown in FIG. 16, the decoding end first demultiplexes the bitstream to obtain the code streams of each part and sequentially decodes. For the base mesh code stream, first parse into inter-frame code stream and intra-frame code stream, then respectively perform inter-frame slice decoding and intra-frame slice decoding to obtain the corresponding inter-frame slice and intra-frame slice. The inter-frame slice and the intra-frame slice are combined to obtain the reconstructed base mesh. After decoding the displacement code stream, the decoding end combines the reconstructed base mesh to perform the deformed mesh reconstruction, and finally obtains the reconstructed deformed mesh. For the texture map code stream, the decoded texture map is obtained after decoding. Through decoding the auxiliary information code stream, the auxiliary information is obtained to assist in guiding the successful parsing of each module.

[0268] Exemplarily, the mesh decoding process can include the following processes:

[0269] (1) Auxiliary information decoding.

[0270] The decoding end determines the decoding scheme according to the auxiliary information, which mainly includes an intra-frame encoding identifier indicating whether the current frame needs to be constructed according to the reference frame mesh; a reference frame list indicating the index of the reference frame needed by the current frame, which is applied to the subsequent base mesh reconstruction step; a subdivision identifier indicating whether the reconstructed base mesh needs to be subjected to subsequent subdivision deformation operation; a static mesh encoder type guiding the decoding end to use the corresponding static mesh decoder; a video encoder type guiding the decoding end to use the corresponding video decoder; a subdivision scheme, i.e. the scheme of subdividing the base mesh in the reconstructed deformed mesh, the subdivision scheme of the encoding and decoding end should be consistent; and optional displacement transformation scheme, coefficient arrangement scheme, etc.

[0271] It is worth noting that the auxiliary information described here includes independently transmitted auxiliary information and header information that can be included in other code stream parts.

[0272] (2) Base mesh decoding.

[0273] The structure of the base mesh code stream can refer to the related content described above. The decoding end divides the base mesh code stream into inter-frame code stream and intra-frame code stream according to the structure of the base mesh code stream. If multiple inter-frame code streams are included, multiple inter-frame slices are decoded. If multiple intra-frame code streams are included, multiple intra-frame slices are decoded. An intra-frame code stream or an inter-frame code stream can also be referred to as a base mesh slice code stream, which can be independently decoded according to its type (intra-frame or inter-frame).

[0274] Inter-frame slice decoding:

[0275] The inter-frame slice decoding includes decoding at least one of the connection relationship information, vertex geometry information, and texture coordinate information in the inter-frame slice. The decoding mode of the connection relationship information, vertex geometry information, and texture coordinate information corresponds to the encoding mode.

[0276] One way of decoding the connection relationship information is to directly use the connection relationship of the reference base mesh. Whether to directly use the connection relationship of the reference base mesh can be determined according to a syntax element in the header information. One way of decoding the texture coordinate information is to directly use the texture coordinate information of the reference base mesh. The texture coordinates of the vertices in the inter-frame patch directly use the texture coordinates of the corresponding points in the reference base mesh. Moreover, consistent texture coordinate adjustment strategies can be adopted at the encoding and decoding ends, so that the inter-frame patches at the encoding and decoding ends are consistent. The decoding of the vertex geometry information first needs to decode whether the inter-frame decoding mode of the inter-frame patch is a P inter-frame mode or a Skip inter-frame mode.

[0277] Intra-frame base mesh decoding:

[0278] Intra-frame patch decoding includes decoding at least one of the connection relationship information, the vertex geometry information, and the texture coordinate information in the intra-frame patch. The decoding manner of the connection relationship information, the vertex geometry information, and the texture coordinate information corresponds to the encoding manner.

[0279] If the connection relationship information, the vertex geometry information, and the texture coordinate information of the intra-frame patch are directly encoded using the Draco compression static mesh, the connection relationship, the vertex geometry coordinates, and the texture coordinates of the intra-frame patch are decoded using corresponding decoding algorithms. Moreover, consistent texture coordinate adjustment strategies can be adopted at the encoding and decoding ends, so that the inter-frame patches at the encoding and decoding ends are consistent.

[0280] It should be understood that the code stream structure of the base mesh code stream in the present application can refer to the related content described above, for example, can refer to the related content related to FIGS. 12 to 14. To avoid repetition, details are not described here.

[0281] (3) Mesh merging.

[0282] The mesh merging processing module merges the decoded inter-frame patch and the decoded intra-frame patch to obtain a reconstructed base mesh. Specifically, the reconstructed intra-frame patch (i.e., the decoded at least one intra-frame patch) and the reconstructed inter-frame patch (i.e., the decoded at least one inter-frame patch) are merged into a complete reconstructed base mesh. Specifically, the geometry coordinates, triangular facets, and texture coordinates of the reconstructed intra-frame patch and the reconstructed inter-frame patch are sequentially written into the reconstructed base mesh in the order of inter-frame first and intra-frame second. Alternatively, the geometry coordinates, triangular facets, and texture coordinates of the reconstructed intra-frame patch and the reconstructed inter-frame patch are sequentially written into the reconstructed base mesh in the order of intra-frame first and inter-frame second. It should be understood that the mesh merging module is the same as the mesh merging module at the encoding end.

[0283] (4) Displacement decoding.

[0284] Displacement decoding is the inverse process of displacement encoding.

[0285] As shown in FIG. 17, the decoding end first needs to decode the displacement, and the decoding of the displacement needs to be consistent with the encoding section. If the encoding end encodes the displacement using a video encoding mode, the decoding end needs to use a video decoding mode to decode. If the encoding end directly entropy encodes the displacement, the decoding end needs to entropy decode the displacement code stream. After the displacement is decoded, the quantized wavelet transform coefficients are obtained. Then, the quantized wavelet transform coefficients are subjected to inverse quantization and inverse wavelet transform processing.

[0286] (5) Reconstructing the deformed mesh.

[0287] The process of reconstructing the deformed mesh at the decoding end is basically consistent with that at the encoding end. First, the inter-frame patches and the intra-frame patches obtained by decoding are combined to restore the complete reconstructed base mesh. Then, the base mesh is subjected to subdivision processing. After the decoding data of the displacement code stream is subjected to inverse quantization and inverse wavelet transform processing, the displacement is obtained and superimposed on the subdivided base mesh, and finally the reconstructed deformed mesh is obtained.

[0288] (6) Decoding the texture map.

[0289] The texture map decoder is responsible for decoding the texture map code stream, which is decoded using the video decoder indicated in the auxiliary information. After optional color space conversion, an image format consistent with the input texture map at the encoding end is obtained, and the final decoded output texture map is obtained.

[0290] After the processing of each module at the decoding end is completed, the deformed mesh reconstructed at the decoding end and the corresponding texture map are obtained, and subsequent applications can process the reconstructed deformed mesh and the texture map as input. The specific processing of the application is not limited.

[0291] In this embodiment, the decoding end obtains the inter-frame code stream and the intra-frame code stream of the to-be-decoded base mesh. The inter-frame code stream is decoded using the inter-frame decoding mode to obtain the inter-frame patch, and the intra-frame code stream is decoded using the intra-frame decoding mode to obtain the intra-frame patch. The inter-frame patch can be a mesh region determined by the encoding end in the registration base mesh based on the matching between the registration subdivided deformed mesh and the registration base mesh of the to-be-encoded original mesh. The connection relationship, vertex geometric coordinates, texture coordinates, etc. are decoded based on the reference base mesh using the temporal prediction technology, which greatly improves the compression efficiency. The intra-frame patch can be a mesh region determined by the encoding end in the to-be-encoded original mesh based on the registration base mesh of the to-be-encoded original mesh and the registration subdivided deformed mesh of the to-be-encoded original mesh. The connection relationship, vertex geometric coordinates, texture coordinates, etc. are directly decoded. One mesh can contain multiple intra-frame patches and multiple inter-frame patches. The multiple intra-frame patches and the multiple inter-frame patches are connected together and jointly processed for displacement generation, displacement decoding, etc. For subsequent temporal meshes, they can be regarded as a whole reference mesh.

[0292] The decoding method provided in the present application is not only applicable to three-dimensional base mesh decoding, but also applicable to inter-frame three-dimensional base mesh decoding, and also applicable to sub-mesh decoding of three-dimensional base mesh or sub-mesh decoding of three-dimensional inter-frame base mesh. When the mesh to be decoded contains multiple sub-meshes, each sub-mesh can be decoded according to the proposed scheme, that is, the processing unit of the decoding scheme can be a sub-mesh.

[0293] FIG. 18 is a schematic flowchart of a slice-based mesh encoding method 500 according to an embodiment of the present application.

[0294] As shown in FIG. 18, the slice-based mesh encoding method 500 can include at least part of the following contents:

[0295] S510, determining, by an encoding end, at least one inter-frame slice of an original mesh to be encoded and at least one intra-frame slice of the original mesh to be encoded.

[0296] S520, encoding, by the encoding end, the at least one inter-frame slice in an inter-frame encoding mode to obtain at least one inter-frame code stream.

[0297] S530, encoding, by the encoding end, the at least one intra-frame slice in an intra-frame encoding mode to obtain at least one intra-frame code stream.

[0298] S540, merging, by the encoding end, the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream.

[0299] In some embodiments, the S510 includes:

[0300] In the case where the encoding mode used by the original mesh to be encoded is not an intra-frame encoding mode, the encoding end determines at least one inter-frame slice of the original mesh to be encoded and at least one intra-frame slice of the original mesh to be encoded.

[0301] In some embodiments, the S540 includes:

[0302] The encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit; and the encoding end merges the header information of the base mesh code stream and the data unit to obtain the base mesh code stream.

[0303] In some embodiments, the header information includes at least one of the following:

[0304] an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0305] an identifier for indicating the data length of the at least one inter-frame code stream;

[0306] an identifier for indicating the number of the at least one inter-frame code stream;

[0307] an identifier for indicating the length of each of the at least one inter-frame code stream;

[0308] an identifier for indicating the length of each of the at least one intra-frame code stream except the last one;

[0309] an identifier for indicating the data length of the at least one intra-frame code stream;

[0310] an identifier for indicating the number of the at least one intra-frame code stream;

[0311] an identifier for indicating the length of each of the at least one intra-frame code stream;

[0312] an identifier for indicating the length of each of the at least one intra-frame code stream except the last one.

[0313] In some embodiments, the S520 comprises:

[0314] The encoding end encodes a first inter-frame slice in the at least one inter-frame slice based on at least one reference patch in a reference base grid of the original grid to be encoded, using the inter-frame encoding mode, to obtain encoding information of the first inter-frame slice; the encoding end encodes first indication information and the encoding information of the first inter-frame slice to obtain a first inter-frame code stream in the at least one first inter-frame code stream, the first indication information being used to indicate the at least one reference patch.

[0315] Illustratively, the encoding end can encode the first inter-frame slice based on the at least one reference patch, using the corresponding encoding mode of the first inter-frame slice, to obtain the encoding information of the first inter-frame slice. Wherein, the corresponding encoding mode of the first inter-frame slice can be a P inter-frame mode or a Skip inter-frame mode. The inter-frame encoding mode in which the motion vector is encoded is called P inter-frame mode; the inter-frame encoding mode in which the motion vector is not encoded is called Skip inter-frame mode.

[0316] For example, if the corresponding encoding mode of the first inter-frame slice is P inter-frame mode, the first indication information needs to be encoded first, and then the motion vector information needs to be encoded; if the corresponding encoding mode of the first inter-frame slice is Skip inter-frame mode, only the first indication information needs to be encoded. Whether it is P inter-frame mode or Skip inter-frame mode, the first indication information needs to be encoded, which is used to indicate which part of the area or which patches in the reference base grid the first inter-frame slice refers to.

[0317] For P inter mode, the encoding end also needs to further encode the motion vector information. One possible encoding manner is to follow the manner of V-DMC, and the specific steps are as follows:

[0318] 1. Divide each 16 vertices of the inter slice into a group in the order of vertex index;

[0319] 2. Determine whether to encode the motion vector of the group of vertices according to the rate-distortion criterion for Skip mode at the group level of the motion vector;

[0320] 3. If the motion vector of the group of vertices is not encoded, the position of the group of vertices needs to be adjusted to the position of the corresponding vertex in the reference frame;

[0321] 4. If the motion vector of the group of vertices is encoded, the motion vector residual of the three motion vector prediction modes needs to be compared, and the prediction mode with smaller bit overhead is selected. When encoding, the prediction mode identifier of the group of vertices is first encoded, and then the motion vector residual of the group of vertices is encoded.

[0322] In some embodiments, the first indication information is an array with a length equal to the number of patches of the reference base mesh, and the value at any position in the array is used to indicate whether the reference patch used by the first inter slice includes the patch corresponding to the any position in the reference base mesh.

[0323] Exemplarily, one possible encoding manner of the first indication information is as follows:

[0324] 1. Set an array with a length equal to the number of patches in the reference base mesh, and the default value in the array is 0;

[0325] 2. Traverse the reference patch of the inter slice, and change the value of the corresponding position in the array from 0 to 1 according to the index of the reference patch in the reference base mesh;

[0326] 3. Encode the 01 characters in the array in order, and when the value of the corresponding position in the array is 1, it indicates that the triangular patch is a reference patch.

[0327] In some embodiments, the reconstructed base mesh of the original mesh to be encoded includes the texture coordinates of the at least one inter slice and the texture coordinates of the at least one intra slice, and the method 500 further includes

[0328] The encoding end scales the texture coordinates of the at least one inter slice along a first direction of the texture coordinate axis to obtain the scaled texture coordinates of the at least one inter slice; and the encoding end scales the texture coordinates of the at least one intra slice along the first direction to obtain the scaled texture coordinates of the at least one intra slice.

[0329] In some embodiments, the encoding end scales the texture coordinates of the at least one inter-frame patch in a first direction along a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch, including:

[0330] The encoding end multiplies the component of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

[0331] In some embodiments, the encoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch, including:

[0332] The encoding end multiplies the component of the texture coordinates of the at least one intra-frame patch in the first direction by a second coefficient first, and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra-frame patch; wherein the sum of the first coefficient and the second coefficient is 1.

[0333] In some embodiments, the S510 includes:

[0334] The encoding end determines a registration base mesh of the to-be-encoded original mesh and a registration subdivision deformation mesh of the to-be-encoded original mesh based on a reference base mesh of the to-be-encoded original mesh and a reference subdivision deformation mesh of the to-be-encoded original mesh; wherein the registration base mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference base mesh, and having the same connectivity relationship as the reference base mesh; the registration subdivision deformation mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference subdivision deformation mesh, and having the same connectivity relationship as the reference subdivision deformation mesh; the encoding end marks a patch in the registration base mesh as a matched or unmatched patch based on a matching detection result between the registration base mesh and the registration subdivision deformation mesh to obtain a first marking result; and the encoding end determines the at least one inter-frame patch in the registration base mesh based on the first marking result.

[0335] In some embodiments, the S510 includes:

[0336] The encoding end marks a patch in the to-be-encoded original mesh as a matched or unmatched patch based on the marking result to obtain a second marking result; the encoding end determines at least one initial intra-frame patch in the to-be-encoded original mesh based on the second marking result; and the encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch.

[0337] Exemplarily, the encoding end can detect and mark the boundaries of the mismatched area and the matched area according to the first marking result of the registration base mesh, and then delete the mismatched patches to obtain the at least one inter-frame patch; detect and mark the boundary points of the mismatched area and the matched area according to the second marking result of the original mesh to be encoded, and then take out the mismatched patches to obtain the initial intra-frame patch.

[0338] FIG. 19 is a schematic diagram of a generation process of the at least one inter-frame patch and the at least one intra-frame patch provided by the embodiment of the present application.

[0339] As shown in FIG. 19, the encoding end first inputs the original mesh to be encoded and the reference base mesh into a registration base mesh generation module, which uses an inter-frame registration algorithm to deform the reference base mesh according to the original mesh to be encoded, to obtain a registration base mesh whose shape is as close as possible to the original mesh to be encoded, and whose vertex number and connectivity relationship remain consistent with those of the reference base mesh. The registration subdivided deformed mesh is a mesh generated in the registration process, which has the same vertex number and connectivity relationship as the reference frame subdivided deformed mesh, and whose shape is also close to the original mesh to be encoded. Then the encoding end performs mismatched area detection to detect which parts of the registration base mesh are well registered with the registration subdivided deformed mesh and which parts are poorly registered with the registration subdivided deformed mesh. After the mismatched area detection, the inter-frame patch and the initial intra-frame patch are obtained. The inter-frame patch is the well-registered part taken out from the registration base mesh, and can carry the matching information with the reference base mesh, such as the vertex in the reference base mesh corresponding to each vertex, so as to be used in subsequent encoding of the inter-frame patch. The initial intra-frame patch is the poorly-registered area in the original mesh to be encoded, which is further processed in combination with the inter-frame patch to obtain the intra-frame patch. The base mesh generation module can obtain multiple inter-frame patches and / or multiple intra-frame patches. Some vertices in the inter-frame patch and some vertices in the intra-frame patch have the same positions, and the inter-frame patch and the intra-frame patch are connected to form a whole mesh.

[0340] Exemplarily, the encoding end generates the intra-frame patch and the inter-frame patch according to the original mesh to be encoded and the reference base mesh. The reference base mesh is a base mesh of a reconstructed mesh, such as a base mesh of a previous frame reconstructed mesh adjacent in the time domain. The reference base mesh can include base meshes of multiple reconstructed meshes, such as when multiple reference frames are used, the reference base mesh can include base meshes of multiple reconstructed meshes.

[0341] FIG. 20 is an example of the registration base mesh, the original mesh to be encoded, the inter-frame patch, the initial intra-frame patch, the intra-frame patch, and the base mesh provided by the embodiment of the present application.

[0342] As shown in FIG. 20, (a) in FIG. 20 is a registration base mesh, wherein the area enclosed by the solid line in the registration base mesh is an area with high similarity to the registration subdivision deformation mesh, and the area information can be used as an inter-frame slice, for example, as shown in (c) in FIG. 20; the area enclosed by the dashed line in the registration base mesh has low similarity to the registration subdivision deformation mesh. (b) in FIG. 20 is a to-be-encoded original mesh, wherein the area enclosed by the solid line is an area with high similarity between the registration base mesh and the registration subdivision deformation mesh, and the area enclosed by the dashed line in the to-be-encoded original mesh is an area with low similarity between the registration base mesh and the registration subdivision deformation mesh, and the area with low similarity can be used as an initial intra-frame slice, for example, as shown in (d) in FIG. 20. The initial intra-frame slice is processed by a mesh to obtain an intra-frame slice, for example, as shown in (e) in FIG. 20. The intra-frame slice and the inter-frame slice have partially coincident vertices, and are combined to obtain a base mesh, for example, as shown in (f) in FIG. 20.

[0343] It should be noted that, if the boundaries of the inter-frame slice and the boundaries of the intra-frame slice do not overlap, in order to connect the inter-frame slice and the intra-frame slice into a whole mesh, the encoding end needs to adjust (morphological stretching) the vertices at the boundaries of the initial intra-frame slice according to the boundaries of the inter-frame slice; then the initial intra-frame slice is subjected to mesh simplification and mesh parameterization processing, and the boundary vertices of the initial intra-frame slice are kept unchanged in the mesh simplification process, and then the texture coordinates are generated through mesh parameterization, to finally obtain the intra-frame slice; finally, the texture coordinates of the inter-frame slice and the intra-frame slice are uniformly adjusted, so that the texture coordinates of the two parts are re-uniformly normalized. Through the above processes, the base mesh generation module of the encoding end can obtain the to-be-encoded inter-frame slice and intra-frame slice.

[0344] In some embodiments, the encoding end determines the registration base mesh of the to-be-encoded original mesh and the registration subdivision deformation mesh of the to-be-encoded original mesh based on the reference base mesh of the to-be-encoded original mesh and the reference subdivision deformation mesh of the to-be-encoded original mesh, and the determination includes:

[0345] The encoding end deforms the to-be-encoded original mesh with the reference subdivision deformation mesh as a target mesh to obtain an intermediate mesh; the encoding end deforms the reference subdivision deformation mesh with the intermediate mesh as a target mesh to obtain the registration subdivision deformation mesh; and the encoding end deforms the reference base mesh with the registration subdivision deformation mesh as a target mesh to obtain the registration base mesh.

[0346] Exemplarily, the encoding end can deform the reference base mesh by using an inter-frame registration algorithm, so that the registration base mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference base mesh, and having the same connectivity relationship as the reference base mesh; and the registration subdivision deformed mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference subdivision deformed mesh, and having the same connectivity relationship as the reference subdivision deformed mesh.

[0347] FIG. 21 is an example of a generation process of a registration base mesh and a configuration subdivision deformed mesh provided by an embodiment of the present application.

[0348] As shown in FIG. 21, first, the reference subdivision deformed mesh is taken as a target mesh, and a nearest neighbor search algorithm is used to deform the to-be-encoded original mesh to the reference subdivision deformed mesh, to output an intermediate mesh. Then, the intermediate mesh is taken as a target mesh, and a nearest neighbor search algorithm is used to deform the reference subdivision deformed mesh to the intermediate mesh, and a mesh obtained by the deformation is the registration subdivision deformed mesh. Then, the generated registration subdivision deformed mesh is taken as a target mesh, and the reference base mesh is fitted to the target mesh, and a base mesh obtained by the fitting is the reference subdivision deformed mesh.

[0349] FIG. 22 is an example of a generation process of at least one inter-frame patch and at least one initial intra-frame patch provided by an embodiment of the present application.

[0350] As shown in FIG. 22, the encoding end can obtain the registration degree of each region of the registration base mesh and the registration subdivision deformed mesh by using a detection algorithm, divide the registration accurate region in the registration base mesh, and then send the region to a subsequent inter-frame patch encoding module for inter-frame encoding. For the region in the registration base mesh with poor registration, the encoding end can obtain the corresponding part from the to-be-encoded original mesh, process to obtain an intra-frame patch, and then send the intra-frame patch to an intra-frame patch encoding module for intra-frame encoding, thereby increasing the flexibility of encoding. In addition, in order to facilitate subsequent boundary stretching of the initial intra-frame patch, before the division, the encoding end can mark the boundary of the inter-frame patch and the boundary point of the initial intra-frame patch according to the detection result of each patch.

[0351] Of course, in other alternative embodiments, the reference base mesh of the original mesh to be encoded can also be used to replace the registration subdivision deformed mesh, and the present application does not make specific limitations thereto. Specifically, the encoding end obtains the registration degree of each region of the registration base mesh and the registration subdivision deformed mesh through the detection algorithm, divides the registration accurate region in the registration base mesh, and then sends this part to the subsequent inter-frame patch encoding module for inter-frame encoding. For the region with poor registration in the registration base mesh, the encoding end can obtain the corresponding part from the original mesh to be encoded, process the intra-frame patch, and then send it to the intra-frame patch encoding module for intra-frame encoding, thereby increasing the flexibility of encoding.

[0352] In some embodiments, the first marking result includes the marking result of the first patch in the registration base mesh; wherein, the encoding end marks the patch in the registration base mesh as a matched or unmatched patch based on the matching detection result between the registration base mesh and the registration subdivision deformed mesh, to obtain the first marking result, including:

[0353] The encoding end obtains the first bounding box in which the first patch is located; the encoding end obtains the first region in the original mesh to be encoded corresponding to the first bounding box and the second region in the matched subdivision deformed mesh corresponding to the first bounding box based on the first bounding box; the encoding end calculates the first weighted average normal vector of the patch in the first region and the second weighted average normal vector of the patch in the second region; in the case that the included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, the encoding end determines that the marking result of the first patch is that the first patch is marked as an unmatched patch, to obtain the marking result of the first patch, or in the case that the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, the encoding end marks the first patch as a matched patch, to obtain the marking result of the first patch.

[0354] Exemplarily, the encoding end can obtain the marking result (i.e. the first marking result) of each patch in the registration base mesh in the following method:

[0355] 1. Traverse each patch in the registration base mesh to obtain the bounding box of each patch;

[0356] 2. According to the bounding box of each patch in the registration base mesh, the corresponding regions of the original mesh to be encoded and the registration subdivision deformed mesh located in the bounding box are obtained respectively, and the weighted average normal vectors of the patches in these two regions are calculated respectively;

[0357] 3. If the included angle between the two normal vectors is greater than the set threshold, the corresponding registration base mesh patch is determined and marked as an unmatched patch;

[0358] 4. Return to step 1 until all patches in the registration base mesh are traversed.

[0359] In some embodiments, the encoding end determines the at least one interframe patch in the registration base mesh based on the first marking result, including:

[0360] The encoding end obtains at least one third region in which the patches marked as mismatched are located based on the first marking result; the encoding end re-marks the patches in a first region in the at least one third region in which the total area of the patches is less than or equal to a preset first area threshold as matched patches to obtain a third marking result; the encoding end obtains at least one fourth region in which the patches marked as matched are located based on the third marking result; the encoding end re-marks the patches in a second region in the at least one fourth region in which the total area of the patches is less than or equal to the preset first area threshold as mismatched patches to obtain a fourth marking result; and the encoding end determines a region in which the patches marked as matched are located based on the fourth marking result as the at least one interframe patch.

[0361] For example, the encoding end can obtain several regions in the set of mismatched patches based on the first marking result, and perform hole filling processing on the patches in a region in the registration base mesh in which the total area of the patches is less than or equal to a preset first area threshold to obtain a third marking result; then, the encoding end can obtain several regions in the set of matched patches based on the third marking result, and perform small block removal processing on the patches in a region in the registration base mesh in which the total area of the patches is less than or equal to the preset first area threshold to obtain the fourth marking result.

[0362] In some embodiments, the second marking result includes a marking result of a second patch in the original mesh to be encoded; and the encoding end marks the patches in the original mesh to be encoded as matched or mismatched patches based on the marking result to obtain a second marking result, including:

[0363] The encoding end obtains the barycenter of the second patch; the encoding end searches for a nearest neighbor point of the barycenter of the second patch in the registration base mesh; and the encoding end marks the second patch as a matched or mismatched patch based on the marking of the patch in which the nearest neighbor point is located to obtain a marking result of the second patch.

[0364] For example, the encoding end can obtain the marking result of each patch in the original mesh to be encoded in the following manner:

[0365] 1. Traverse the patches in the original mesh to be encoded to obtain the barycenter of each patch;

[0366] 2. For the barycenter of the original mesh patch to be encoded, a nearest neighbor search is performed in the registered base mesh, and according to the mismatch label of the patch where the nearest neighbor point is located, the current patch is labeled with the corresponding mismatch label;

[0367] 3. Return to step 1 until all patches in the original mesh to be encoded are traversed.

[0368] In some embodiments, the encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result, including:

[0369] The encoding end obtains at least one fifth region where the patches labeled as mismatched are located based on the second labeling result; the encoding end re-labels the patches in a third region in the at least one fifth region as matched patches if the total area of the patches in the third region is less than or equal to a preset second area threshold, to obtain a fifth labeling result; the encoding end obtains at least one sixth region where the patches labeled as matched are located based on the fifth labeling result; the encoding end re-labels the patches in a fourth region in the at least one sixth region as mismatched patches if the total area of the patches in the fourth region is less than or equal to a preset second area threshold, to obtain a sixth labeling result; and the encoding end determines a region where the patches labeled as mismatched are located based on the sixth labeling result as the at least one initial intra-frame patch.

[0370] For example, the encoding end can obtain several regions in the set of mismatched patches based on the second labeling result, and perform hole filling processing on the patches in a region in the original mesh to be encoded if the total area of the patches in the region is less than or equal to a preset second area threshold, to obtain a fifth labeling result; then, the encoding end can obtain several regions in the set of matched patches based on the fifth labeling result, and perform small block removal processing on the patches in a region in the original mesh to be encoded if the total area of the patches in the region is less than or equal to a preset second area threshold, to obtain the sixth labeling result.

[0371] In some embodiments, the encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch, to obtain the at least one intra-frame patch, including:

[0372] The encoding end obtains at least one edge of the registration base mesh in which there are adjacent patches marked as matching and non-matching; the encoding end obtains at least one vertex of the original mesh to be encoded in which there are adjacent patches marked as matching and non-matching; for a first vertex corresponding to the at least one edge, the encoding end searches for a nearest neighbor point of the first vertex among the at least one vertex; the encoding end deforms the nearest neighbor point of the first vertex to the first vertex; for a second vertex in the at least one vertex except the nearest neighbor point of the first vertex, the encoding end searches for a nearest neighbor point of the second vertex on the at least one boundary; the encoding end obtains an edge in which the nearest neighbor point of the second vertex is located based on the nearest neighbor point of the second vertex; the encoding end determines a position closest to the second vertex on the edge in which the nearest neighbor point of the second vertex is located; and the encoding end deforms the second vertex to the position closest to the second vertex to obtain the at least one intra-frame patch.

[0373] Exemplarily, the purpose of boundary adjustment (for example, boundary stretching) on the initial intra-frame patch is to make the inter-frame part mesh and the intra-frame part mesh as close as possible at the boundary position, without gaps and overlapping areas. Before stretching, the boundary information of the inter-frame patch and the initial intra-frame patch needs to be marked respectively. A possible boundary marking method is to mark the boundary of the inter-frame patch and the boundary point of the initial intra-frame patch according to the detection result of each patch after the non-matching area detection and before the mesh division.

[0374] FIG. 23 is an example of a boundary and a boundary point provided by an embodiment of the present application.

[0375] As shown in (a) of FIG. 23, the encoding end obtains a current edge in a manner of traversing each edge of the registration base mesh; if there are adjacent patches belonging to the matching area and the non-matching area at the same time, the current edge is considered as a division boundary of the matching area and the non-matching area, that is, a division boundary of the inter-frame patch; the encoding end obtains a next edge and takes it as the current edge to determine whether it is a division boundary, until all edges of the registration base mesh are traversed to obtain the at least one edge. As shown in (b) of FIG. 23, the encoding end obtains all adjacent patches of a current vertex in a manner of traversing each vertex of the original mesh to be encoded; if there are adjacent patches of the matching area and the non-matching area at the same time, the current vertex is considered as a boundary point of the initial intra-frame patch; the encoding end obtains a next vertex and takes it as the current vertex to determine whether it is a boundary point, until all vertices of the initial intra-frame patch are traversed to obtain the at least one vertex.

[0376] FIG. 24 is a schematic diagram of boundary stretching provided by an embodiment of the present application.

[0377] As shown in FIG. 24, the encoding end can perform boundary stretching according to the result of boundary marking. A possible initial intra-slice boundary stretching scheme can be: the encoding end traverses the boundary points of the inter-slice, performs a nearest neighbor search on each point on the initial intra-slice boundary point, and deforms the found nearest neighbor point to the position of the current inter-slice boundary point. Then, the encoding end takes the initial intra-slice boundary points that are not searched as deformation points, searches for the nearest neighbor points on the inter-slice boundary for each deformation point. Then, the encoding end obtains the boundary where the searched nearest neighbor point is located, calculates and compares the shortest distances of the deformation points to the boundary, and takes the position corresponding to the shortest distance as the final deformation position. Finally, the encoding end removes the repeated points and edges from the initial intra-slice after boundary stretching.

[0378] It should be noted that the initial intra-slice can be processed to obtain an intra-slice, which can be generated by mesh simplification and mesh parameterization to be used for subsequent base mesh coding. For example, as shown in FIG. 25, first, the initial intra-slice is subjected to mesh simplification to output an intra-slice without texture coordinates. Then, the intra-slice is subjected to mesh parameterization to generate the texture coordinates of the intra-slice.

[0379] In some embodiments, the method 500 further includes:

[0380] The encoding end obtains a reconstructed base mesh of the original mesh to be coded; the encoding end performs subdivision deformation on the reconstructed base mesh based on the original mesh to be coded, to obtain a subdivision mesh of the original mesh to be coded and a subdivision deformation mesh of the original mesh to be coded; the encoding end determines displacement information of the original mesh to be coded based on the subdivision mesh of the original mesh to be coded and the subdivision deformation mesh of the original mesh to be coded; and the encoding end encodes the displacement information to obtain a displacement code stream.

[0381] FIG. 26 is a schematic diagram of a process of generating displacement information according to an embodiment of the present application.

[0382] As shown in FIG. 26, the encoding end first performs subdivision deformation on the reconstructed base mesh using the original mesh to be coded to obtain a subdivision deformation mesh of the current frame. Then, the encoding end performs subdivision on the reconstructed base mesh to obtain a subdivision mesh having consistent connection relationship and vertex correspondence with the subdivision deformation mesh, so as to calculate the displacement vectors of the vertices.

[0383] FIG. 27 is a schematic diagram of inter-slice coding and intra-slice coding according to an embodiment of the present application.

[0384] Exemplarily, after the encoding end obtains the at least one inter-frame slice and the at least one intra-frame slice, inter-frame slice encoding and intra-frame slice encoding can be performed, and after the two parts are encoded, at least one inter-frame code stream and at least one intra-frame code stream two-part sub-code stream can be obtained, and the at least one inter-frame code stream and the at least one intra-frame code stream are combined to form a basic grid code stream. It is worth noting that additional information can also be added to the basic grid code stream, so that the decoding end can parse the at least one inter-frame code stream and the at least one intra-frame code stream. The additional information can be an identifier in the header information of the basic grid code stream, which can be used to indicate at least one of the number of the at least one inter-frame code stream, the number of the at least one intra-frame code stream, the data length of each inter-frame code stream, and the data length of each intra-frame code stream. The additional information and the combination of the at least one inter-frame code stream and the at least one intra-frame code stream can be completed in the code stream merging module.

[0385] In some embodiments, the method 500 further includes:

[0386] The encoding end deforms the subdivided grid of the original grid to be encoded based on the reconstructed displacement information of the displacement information, to obtain a reconstructed deformed grid of the original grid to be encoded.

[0387] FIG. 28 is a schematic diagram of a reconstructed deformed grid provided by an embodiment of the present application.

[0388] As shown in FIG. 28, the encoding end merges the reconstructed inter-frame slice and the intra-frame slice to recover the complete reconstructed basic grid. Then, the encoding end performs a subdivision process on the basic grid. After the reconstructed displacement is subjected to the inverse quantization and inverse wavelet transform processes, the displacement can be obtained and superimposed on the subdivided basic grid, and finally the reconstructed reconstructed deformed grid can be obtained.

[0389] FIG. 29 is a schematic flow chart of a slice-based grid encoding process provided by an embodiment of the present application.

[0390] As shown in FIG. 29, the encoding end generates two types of encoding slices, i.e., inter-frame slices and intra-frame slices, based on an input grid (i.e., an original grid to be encoded) for encoding. For the inter-frame slices, the time-domain prediction technology is used based on a reference basic grid to encode the inter-frame slices, to obtain an inter-frame code stream. For the intra-frame slices, the intra-frame slices are directly encoded to obtain an intra-frame code stream. The inter-frame code stream and the intra-frame code stream constitute a complete basic grid code stream. The inter-frame code stream includes indication information (i.e., the first indication information referred to above) for indicating a reference surface slice and motion vector information and the like. The intra-frame code stream includes vertex geometry information, connection relationship information, texture coordinate information and the like. The reconstructed inter-frame slice obtained by the inter-frame slice encoding module can be regarded as a reconstructed inter-frame slice after inter-frame slice encoding and decoding. The reconstructed intra-frame slice obtained by the intra-frame slice encoding module can be regarded as a reconstructed grid after intra-frame slice encoding and decoding. The reconstructed inter-frame slice and the reconstructed intra-frame slice are subjected to a grid merging processing module to obtain a reconstructed basic grid.

[0391] The displacement generation module performs subdivision deformation on the reconstructed base mesh according to the original mesh to be encoded, to obtain a subdivision-deformed mesh of the current frame, and then performs subdivision on the reconstructed base mesh to obtain a subdivision mesh having consistent connection relationships and vertex correspondence with the subdivision-deformed mesh, so as to calculate displacement vectors of each vertex, and obtain a displacement bitstream and reconstructed displacement through displacement encoding.

[0392] The deformed mesh reconstruction module generates a reconstructed deformed mesh based on the reconstructed displacement and the reconstructed base mesh. Based on the reconstructed deformed mesh, texture conversion is performed using the original mesh to be encoded and the input texture map to obtain a texture map. The generated texture map is encoded to obtain a texture map bitstream. In addition, the encoding end can also encode auxiliary information for guiding decoding of the decoding end (such as the type of the encoder, the displacement encoding method, the wavelet transform coefficient, and the like). The encoding end mixes the base mesh bitstream, the texture coordinate bitstream, the displacement bitstream, the texture map bitstream, and the auxiliary information bitstream to obtain a final bitstream.

[0393] Exemplarily, the mesh encoding process can include the following processes:

[0394] (1) Base mesh encoding.

[0395] Inter-frame slice encoding:

[0396] The inter-frame slice encoding includes encoding at least one of connection relationship information, vertex geometry information, and texture coordinate information in the inter-frame slice.

[0397] One way of encoding the connection relationship information is to directly use the connection relationship of the reference base mesh, or a syntax element can be added in the header information to indicate whether to directly use the connection relationship of the reference base mesh. One way of encoding the texture coordinate information is to directly use the texture coordinate information of the reference base mesh, and the texture coordinates of the vertices in the inter-frame slice directly use the texture coordinates of the corresponding points in the reference base mesh. The encoding of the vertex geometry information can determine whether to encode the motion vector information of each vertex in the inter-frame slice according to the rate-distortion criterion.

[0398] Intra-frame slice encoding:

[0399] The intra-frame slice encoding includes encoding at least one of connection relationship information, vertex geometry information, and texture coordinate information in the intra-frame slice.

[0400] In one case, the connection information, vertex geometry information, and texture coordinate information of the intra-frame patch are directly encoded. One possible encoding method is to use Draco to compress the static mesh, encode the connection information, vertex geometry coordinates, and texture coordinates of the base mesh, and finally output the intra-frame code stream. The encoding method of the texture coordinates is similar to the encoding of the vertex geometry information, and a prediction method can also be used. When the texture coordinate connection relationship is different from the vertex geometry information connection relationship, the texture coordinate connection relationship needs to be encoded, and the difference between the vertex geometry information connection relationship can also be encoded.

[0401] Base mesh code stream:

[0402] After the inter-frame patch and the intra-frame patch are encoded, the inter-frame code stream and the intra-frame code stream can be obtained, respectively. The combination of the inter-frame code stream and the intra-frame code stream forms the base mesh code stream.

[0403] It is worth noting that additional information can also be added to the base mesh code stream to parse the inter-frame code stream and the intra-frame code stream at the decoding end. The additional information can be an identifier in the header information of the base mesh code stream, which can be used to indicate at least one of the number of inter-frame code streams, the number of intra-frame code streams, the data length of each inter-frame code stream, and the data length of each intra-frame code stream. The additional information and the combination of the inter-frame code stream and the intra-frame code stream can be completed in the code stream merging module.

[0404] It should be understood that the specific generation process of the base mesh code stream in the present application can refer to the related content described above, for example, it can refer to the related content involved in FIGS. 12-14, 19-25, and 27. To avoid repetition, it will not be described here.

[0405] (2) Mesh merging.

[0406] The mesh merging processing module merges the reconstructed intra-frame patch and the reconstructed inter-frame patch into a complete reconstructed base mesh. Specifically, the geometry coordinates, triangular patches, and texture coordinates of the reconstructed intra-frame patch and the reconstructed inter-frame patch are sequentially written into the reconstructed base mesh in the order of inter-frame first and intra-frame second. Alternatively, the geometry coordinates, triangular patches, and texture coordinates of the reconstructed intra-frame patch and the reconstructed inter-frame patch are sequentially written into the reconstructed base mesh in the order of intra-frame first and inter-frame second. It should be understood that the mesh merging module is the same as the mesh merging module at the decoding end.

[0407] (3) Displacement generation.

[0408] The subdivision deformation module takes the parameterized sub-mesh as input, firstly subdivides the original mesh to be encoded, and the subdivision scheme can be selected at will. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four sub-triangles in each iteration of subdivision. A new vertex is introduced in the middle of each edge. The subdivision of geometric information and attribute information is carried out independently because the connection relationship between geometric information and attribute information is usually different.

[0409] It should be understood that the displacement generation process in the present application can refer to the related content described above, for example, it can refer to the related content of FIG. 26, and to avoid repetition, details are not described here.

[0410] (4) Displacement encoding.

[0411] The displacement data generated is mainly encoded, and the implementation block diagram is shown in FIG. 30. The displacement encoding module performs wavelet transform, quantization and other processes on the displacement to obtain quantized wavelet transform coefficients. Finally, the quantized wavelet transform coefficients are encoded to obtain the displacement code stream. There are many possible specific ways to encode the quantized wavelet transform coefficients. One possible way is to reuse the existing V-DMC mode: arrange the quantized wavelet transform coefficients into a video frame and input them into a video encoder for encoding, or directly perform entropy encoding. Of course, another possible way is to directly entropy encode the displacement information without the foregoing wavelet transform, quantization and other links.

[0412] (5) Deformed mesh reconstruction.

[0413] The deformed mesh reconstruction module combines the reconstructed inter-frame patches and intra-frame patches to restore the complete reconstructed base mesh. Then, the base mesh is subdivided at the encoding end. After the reconstructed displacement is subjected to inverse quantization and inverse wavelet transform, the displacement can be obtained and superimposed on the subdivided base mesh, and finally the reconstructed deformed mesh can be obtained.

[0414] It should be understood that the deformed mesh reconstruction process in the present application can refer to the related content described above, for example, it can refer to the related content of FIG. 28, and to avoid repetition, details are not described here.

[0415] (6) Texture map conversion.

[0416] The texture map conversion module converts the texture map according to the input original mesh, the input original texture map and the reconstructed deformed mesh. The texture map conversion can refer to the related description above, for example, the related content of FIG. 9, and to avoid repetition, details are not described here.

[0417] (7) Texture map compression.

[0418] After the converted texture map is obtained, the existing filling algorithm (such as the Push-Pull algorithm) can be used to fill the empty pixels in the texture map. Then the existing video encoder, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc. can be used to encode the texture map to obtain the code stream of the output texture map. In addition, color space conversion and chroma subsampling operations can be selectively applied to make the video encoding obtain better rate-distortion performance, such as color space conversion from RGB 444 to YUV 420.

[0419] (8) Auxiliary information encoding.

[0420] The encoding end determines and encodes the auxiliary information based on the encoding scheme, which mainly includes an intra-frame encoding identifier indicating whether the current frame needs to be constructed according to the reference frame grid, a reference frame list indicating the index of the reference frame needed by the current frame, which is applied to the subsequent basic grid reconstruction step, a subdivision identifier indicating whether the reconstructed basic grid needs to be subjected to subsequent subdivision deformation operation, a static grid encoder type guiding the decoding end to use the corresponding static grid decoder, a video encoder type guiding the decoding end to use the corresponding video decoder, a subdivision scheme, i.e. the scheme of subdividing the basic grid in the reconstructed deformation grid, which should be consistent between the encoding and decoding ends, and optional displacement transformation scheme, coefficient arrangement scheme, etc.

[0421] It is worth noting that the auxiliary information described herein includes auxiliary information transmitted independently and header information that can be included in other code stream parts.

[0422] After all the modules in the encoding end are encoded, the basic grid code stream, the texture coordinate code stream, the displacement code stream, the texture map code stream, the auxiliary information code stream, etc. are mixed to obtain the final output encoded bit stream. The specific implementation of the mixing of the present application is not limited.

[0423] In this embodiment, the three-dimensional mesh is divided into intra-frame patches and inter-frame patches, the inter-frame patches adopt an inter-frame coding mode, and the intra-frame patches adopt an intra-frame coding mode. The inter-frame patches can be mesh regions determined by the coding end based on the face patches matched between the registration base mesh and the registration subdivided deformed mesh of the original mesh to be coded in the registration base mesh, and the connection relationship, vertex geometric coordinates, texture coordinates, etc. of the inter-frame patches are coded based on the reference base mesh using the temporal prediction technology, which greatly improves the compression efficiency. The intra-frame patches can be mesh regions determined by the coding end based on the registration base mesh of the original mesh to be coded and the registration subdivided deformed mesh of the original mesh to be coded, which are regions obtained by boundary stretching, mesh processing, etc. after the initial intra-frame patches are obtained from the original mesh to be coded, and the connection relationship, vertex geometric coordinates, texture coordinates, etc. of the intra-frame patches are directly coded. One mesh can include multiple intra-frame patches and multiple inter-frame patches. The multiple intra-frame patches and the multiple inter-frame patches are connected together and jointly processed for displacement generation, displacement coding, etc. For subsequent temporal meshes, the multiple intra-frame patches and the multiple inter-frame patches can be regarded as a whole reference mesh.

[0424] The coding method provided in the present application is not only suitable for three-dimensional base mesh coding, but also suitable for inter-frame three-dimensional base mesh coding, and suitable for sub-mesh coding process of three-dimensional base mesh or inter-frame three-dimensional base mesh. When the mesh to be coded includes multiple sub-meshes, each sub-mesh can be coded according to the proposed scheme, that is, the processing unit of the coding scheme can be a sub-mesh.

[0425] It should be understood that the coding method can be understood as the inverse process (or reverse process) of the decoding method, therefore, the specific scheme of the piece-based mesh coding method 500 can refer to the related content of the piece-based mesh decoding method 400, and for the convenience of description, the present application will not be described here.

[0426] The piece-based mesh decoding method provided in the embodiments of the present application can be executed by a piece-based mesh decoding device. In the embodiments of the present application, the piece-based mesh decoding method is executed by a piece-based mesh decoding device as an example to illustrate the piece-based mesh decoding device provided in the embodiments of the present application. The piece-based mesh coding method provided in the embodiments of the present application can be executed by a piece-based mesh coding device. In the embodiments of the present application, the piece-based mesh coding method is executed by a piece-based mesh coding device as an example to illustrate the piece-based mesh coding device provided in the embodiments of the present application.

[0427] FIG. 31 is a schematic block diagram of a piece-based mesh decoding device 600 according to the embodiments of the present application.

[0428] As shown in FIG. 31, the piece-based mesh decoding device 600 includes:

[0429] The acquisition unit 610 is configured to acquire at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid.

[0430] The first decoding unit 620 is configured to decode the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice.

[0431] The second decoding unit 630 is configured to decode the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice.

[0432] The merging unit 640 is configured to merge the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

[0433] In some embodiments, the acquisition unit 610 is specifically configured to:

[0434] In a case where a decoding mode used by the to-be-decoded base grid is not an intra-frame decoding mode, the at least one inter-frame code stream and the at least one intra-frame code stream are acquired.

[0435] In some embodiments, the acquisition unit 610 is specifically configured to:

[0436] Acquire a base grid code stream of the to-be-decoded base grid, the base grid code stream including header information and a data unit;

[0437] Based on the header information, the at least one inter-frame code stream and the at least one intra-frame code stream are acquired from the data unit.

[0438] In some embodiments, the header information includes at least one of the following:

[0439] An identifier used to indicate an order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0440] An identifier used to indicate a data length of the at least one inter-frame code stream;

[0441] An identifier used to indicate a number of the at least one inter-frame code stream;

[0442] An identifier used to indicate a length of each inter-frame code stream in the at least one inter-frame code stream;

[0443] An identifier used to indicate a length of each intra-frame code stream except for a last intra-frame code stream in the at least one inter-frame code stream;

[0444] An identifier used to indicate a data length of the at least one intra-frame code stream;

[0445] An identifier used to indicate a number of the at least one intra-frame code stream;

[0446] an identifier for indicating a length of each intra code stream in the at least one intra code stream;

[0447] an identifier for indicating a length of each intra code stream in the at least one intra code stream except the last intra code stream.

[0448] In some embodiments, the first decoding unit 620 is specifically configured to:

[0449] decode a first inter code stream in the at least one inter code stream to obtain first indication information and to-be-decoded information of a first inter tile in the at least one inter tile, the first indication information being used for indicating at least one reference surface tile used by the first inter tile in a reference base mesh of the to-be-decoded base mesh;

[0450] decode the to-be-decoded information by using the inter decoding mode based on the at least one reference surface tile to obtain the first inter tile.

[0451] In some embodiments, the first indication information is an array with a length equal to a number of surface tiles of the reference base mesh, and a value at any position in the array is used to indicate whether a reference surface tile used by the first inter tile includes a surface tile corresponding to the any position in the reference base mesh.

[0452] In some embodiments, the reconstructed base mesh includes texture coordinates of the at least one inter tile and texture coordinates of the at least one intra tile, and the obtaining unit 610 is further configured to:

[0453] scale the texture coordinates of the at least one inter tile in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter tile;

[0454] scale the texture coordinates of the at least one intra tile in the first direction to obtain scaled texture coordinates of the at least one intra tile.

[0455] In some embodiments, the obtaining unit 610 is specifically configured to:

[0456] multiply a component of the texture coordinates of the at least one inter tile in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter tile.

[0457] In some embodiments, the obtaining unit 610 is specifically configured to:

[0458] multiply a component of the texture coordinates of the at least one intra tile in the first direction by a second coefficient and then add the first coefficient to obtain the scaled texture coordinates of the at least one intra tile.

[0459] wherein the sum of the first coefficient and the second coefficient is 1.

[0460] It should be understood that the slice-based mesh decoding apparatus 600 provided by the embodiments of the present application can correspond to the execution subject in the method embodiments of the present application, and each unit in the slice-based mesh decoding apparatus 600 is respectively used to implement the corresponding process of the slice-based mesh decoding method 400 shown in FIG. 11, and for brevity, details are not described herein again.

[0461] The slice-based mesh decoding apparatus provided by the embodiments of the present application can implement each process of the method embodiments shown in FIG. 11 and achieve the same technical effects, and for brevity, details are not described herein again.

[0462] FIG. 32 is a schematic block diagram of a slice-based mesh encoding apparatus 700 according to an embodiment of the present application.

[0463] As shown in FIG. 32, the slice-based mesh encoding apparatus 700 includes:

[0464] A determining unit 710 is configured to determine at least one inter-frame slice of an original mesh to be encoded and at least one intra-frame slice of the original mesh to be encoded.

[0465] A first encoding unit 720 is configured to encode the at least one inter-frame slice by using an inter-frame encoding mode to obtain at least one inter-frame code stream.

[0466] A second encoding unit 730 is configured to encode the at least one intra-frame slice by using an intra-frame encoding mode to obtain at least one intra-frame code stream.

[0467] A merging unit 740 is configured to merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a basic mesh code stream.

[0468] In some embodiments, the determining the at least one inter-frame slice of the original mesh to be encoded and the at least one intra-frame slice of the original mesh to be encoded includes:

[0469] In a case where an encoding mode used by the original mesh to be encoded is not an intra-frame encoding mode, the at least one inter-frame slice of the original mesh to be encoded and the at least one intra-frame slice of the original mesh to be encoded are determined.

[0470] In some embodiments, the merging unit 740 is specifically configured to:

[0471] merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit;

[0472] merge the header information of the basic mesh code stream and the data unit to obtain the basic mesh code stream.

[0473] In some embodiments, the header information comprises at least one of:

[0474] an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0475] an identifier for indicating the data length of the at least one inter-frame code stream;

[0476] an identifier for indicating the number of the at least one inter-frame code stream;

[0477] an identifier for indicating the length of each inter-frame code stream in the at least one inter-frame code stream;

[0478] an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one inter-frame code stream;

[0479] an identifier for indicating the data length of the at least one intra-frame code stream;

[0480] an identifier for indicating the number of the at least one intra-frame code stream;

[0481] an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream;

[0482] an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one intra-frame code stream.

[0483] In some embodiments, the first encoding unit 620 is specifically configured to:

[0484] encode a first inter-frame slice in the at least one inter-frame slice by using the inter-frame encoding mode based on at least one reference patch in a reference base grid of the original grid to be encoded, to obtain encoding information of the first inter-frame slice;

[0485] determine first indication information based on the at least one reference patch, the first indication information being used for indicating the at least one reference patch;

[0486] encode the first indication information and the encoding information of the first inter-frame slice, to obtain a first inter-frame code stream in the at least one first inter-frame code stream.

[0487] In some embodiments, the first indication information is an array with a length equal to the number of patches in the reference base grid, and a value at any position in the array is used to indicate whether the reference patch used by the first inter-frame slice includes a patch in the reference base grid corresponding to the any position.

[0488] In some embodiments, the reconstruction base mesh of the original mesh to be encoded includes texture coordinates of the at least one inter-frame patch and texture coordinates of the at least one intra-frame patch, and the determination unit 710 is further configured to:

[0489] scaling the texture coordinates of the at least one inter-frame patch in a first direction along a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch;

[0490] scaling the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch.

[0491] In some embodiments, the determination unit 710 is specifically configured to:

[0492] multiplying a component of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

[0493] In some embodiments, the determination unit 710 is specifically configured to:

[0494] multiplying a component of the texture coordinates of the at least one intra-frame patch in the first direction by a second coefficient and then adding the first coefficient to obtain the scaled texture coordinates of the at least one intra-frame patch;

[0495] wherein the sum of the first coefficient and the second coefficient is 1.

[0496] In some embodiments, the determination unit 710 is specifically configured to:

[0497] determining, based on a reference base mesh of the original mesh to be encoded and a reference subdivision deformation mesh of the original mesh to be encoded, a registration base mesh of the original mesh to be encoded and a registration subdivision deformation mesh of the original mesh to be encoded;

[0498] wherein the registration base mesh is a mesh obtained by deforming the original mesh to be encoded, having the same number of vertices as the reference base mesh and the same connectivity relationship as the reference base mesh; and the registration subdivision deformation mesh is a mesh obtained by deforming the original mesh to be encoded, having the same number of vertices as the reference subdivision deformation mesh and the same connectivity relationship as the reference subdivision deformation mesh;

[0499] based on a matching detection result between the registration base mesh and the registration subdivision deformation mesh, marking a patch in the registration base mesh as a matched patch or an unmatched patch to obtain a first marking result;

[0500] determine, based on the first marking result, at least one inter-slice in the registration base mesh.

[0501] In some embodiments, the determining unit 710 is specifically configured to:

[0502] deform the to-be-encoded original mesh with the reference subdivision deformation mesh as a target mesh to obtain an intermediate mesh;

[0503] deform the reference subdivision deformation mesh with the intermediate mesh as a target mesh to obtain the registration subdivision deformation mesh;

[0504] deform the reference base mesh with the registration subdivision deformation mesh as a target mesh to obtain the registration base mesh.

[0505] In some embodiments, the first marking result includes a marking result of a first slice in the registration base mesh.

[0506] In some embodiments, the determining unit 710 is specifically configured to:

[0507] obtain a first bounding box in which the first slice is located;

[0508] obtain a first region in the to-be-encoded original mesh corresponding to the first bounding box and a second region in the matching subdivision deformation mesh corresponding to the first bounding box based on the first bounding box;

[0509] calculate a first weighted average normal vector of slices in the first region and a second weighted average normal vector of slices in the second region;

[0510] in a case where an included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, determine that the marking result of the first slice is that the first slice is marked as a mismatched slice, to obtain the marking result of the first slice, or in a case where the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, mark the first slice as a matched slice, to obtain the marking result of the first slice.

[0511] In some embodiments, the determining unit 710 is specifically configured to:

[0512] obtain at least one third region in which a mismatched slice is located based on the first marking result;

[0513] re-mark slices in a first region in which a total area of slices is less than or equal to a preset first area threshold in the at least one third region as matched slices to obtain a third marking result;

[0514] obtain at least one fourth region where the face patches marked as matched are located based on the third marking result;

[0515] re-mark the face patches in a second region in the at least one fourth region as non-matched face patches, where the total area of the face patches in the second region is less than or equal to a preset first area threshold, to obtain a fourth marking result;

[0516] determine the region where the face patches marked as matched are located as the at least one inter-frame patch based on the fourth marking result.

[0517] In some embodiments, the determination unit 710 is specifically configured to:

[0518] mark the face patches in the original mesh to be encoded as matched or non-matched face patches based on the marking result to obtain a second marking result;

[0519] determine at least one initial intra-frame patch in the original mesh to be encoded based on the second marking result;

[0520] perform boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch.

[0521] In some embodiments, the second marking result includes a marking result of a second face patch in the original mesh to be encoded;

[0522] In some embodiments, the determination unit 710 is specifically configured to:

[0523] obtain a barycenter of the second face patch;

[0524] search for a nearest neighbor point of the barycenter of the second face patch in the registered base mesh;

[0525] mark the second face patch as a matched or non-matched face patch based on the marking of the face patch where the nearest neighbor point is located to obtain the marking result of the second face patch.

[0526] In some embodiments, the determination unit 710 is specifically configured to:

[0527] obtain at least one fifth region where the face patches marked as non-matched are located based on the second marking result;

[0528] re-mark the face patches in a third region in the at least one fifth region as matched face patches, where the total area of the face patches in the third region is less than or equal to a preset second area threshold, to obtain a fifth marking result;

[0529] obtain at least one sixth region where the face patches marked as matched are located based on the fifth marking result;

[0530] relabeling, as mismatched patches, patches in a fourth region in the at least one sixth region and having a total patch area less than or equal to a preset second area threshold, to obtain a sixth labeling result;

[0531] determining, based on the sixth labeling result, a region in which a patch labeled as a mismatched patch is located, as the at least one initial intra patch.

[0532] In some embodiments, the determination unit 710 is specifically configured to:

[0533] obtaining at least one edge in the registration base mesh in which there are adjacent patches labeled as matched and mismatched;

[0534] obtaining at least one vertex in the original mesh to be encoded in which there are adjacent patches labeled as matched and mismatched;

[0535] for a first vertex corresponding to the at least one edge, searching for a nearest neighbor point of the first vertex in the at least one vertex;

[0536] deforming the nearest neighbor point of the first vertex to the first vertex;

[0537] for a second vertex in the at least one vertex other than the nearest neighbor point of the first vertex, searching for a nearest neighbor point of the second vertex on the at least one boundary;

[0538] based on the nearest neighbor point of the second vertex, obtaining an edge in which the nearest neighbor point of the second vertex is located;

[0539] determining a position closest to the second vertex on the edge in which the nearest neighbor point of the second vertex is located;

[0540] deforming the second vertex to the position closest to the second vertex, to obtain the at least one intra patch.

[0541] In some embodiments, the apparatus 700 further includes a third encoding unit, configured to:

[0542] obtaining a reconstructed base mesh of the original mesh to be encoded;

[0543] based on the original mesh to be encoded, performing a subdivision deformation on the reconstructed base mesh to obtain a subdivision mesh of the original mesh to be encoded and a subdivision deformation mesh of the original mesh to be encoded;

[0544] based on the subdivision mesh of the original mesh to be encoded and the subdivision deformation mesh of the original mesh to be encoded, determining displacement information of the original mesh to be encoded;

[0545] encoding the displacement information to obtain a displacement code stream.

[0546] In some embodiments, the apparatus 700 further comprises a reconstruction unit configured to:

[0547] deform the subdivision mesh of the original mesh to be encoded based on the reconstructed displacement information of the displacement information, to obtain a reconstructed deformed mesh of the original mesh to be encoded.

[0548] It should be understood that the slice-based mesh encoding apparatus 700 provided by the embodiments of the present application can correspond to the execution subject in the method embodiments of the present application, and each unit in the slice-based mesh encoding apparatus 700 is respectively used to implement the corresponding process of the slice-based mesh encoding method 500 shown in FIG. 18, and for brevity, will not be described here again.

[0549] The slice-based mesh encoding apparatus provided by the embodiments of the present application can implement each process of the method embodiments shown in FIG. 18 and achieve the same technical effects, and for brevity, will not be described here again.

[0550] The embodiments of the present application further provide an electronic device 800, as shown in FIG. 33, comprising a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions executable on the processor 801.

[0551] For example, when the electronic device 800 is a decoding end, the programs or instructions are executed by the processor 801 to implement each step of the slice-based mesh decoding method embodiments described above and achieve the same technical effects. When the electronic device 800 is an encoding end, the programs or instructions are executed by the processor 801 to implement each step of the slice-based mesh encoding method embodiments described above and achieve the same technical effects, and for brevity, will not be described here again. Optionally, the memory 802 can be the memory 102 or the memory 113 in the embodiment shown in FIG. 1, and the processor 801 can implement the functions of the encoder 200 or the decoder 300 in the embodiment shown in FIG. 1.

[0552] The embodiments of the present application further provide an electronic device, comprising a memory configured to store video data, and a processing circuit configured to implement each step of the slice-based mesh decoding method embodiments or the slice-based mesh encoding method embodiments described above. Optionally, the memory can be the memory 102 or the memory 113 in the embodiment shown in FIG. 1, and the processing circuit can implement the functions of the encoder 200 or the decoder 300 in the embodiment shown in FIG. 1.

[0553] The electronic device can be a terminal, or can be another device other than the terminal, such as a server, a Network Attached Storage (NAS), and the like.

[0554] The electronic device can be a terminal, or can be another device other than the terminal, such as a server, a Network Attached Storage (NAS), and the like.

[0555] The terminal can be a mobile phone, a Tablet Personal Computer (PC), a Laptop PC, a notebook computer, a Personal Digital Assistant (PDA), a palm PC, a netbook, an Ultra-mobile PC (UMPC), a Mobile Internet Device (MID), an Augmented Reality (AR) device, a Virtual Reality (VR) device, a mixed reality (MR) device, a robot, a Wearable Device, a flight vehicle, a Vehicle User Equipment (VUE), a shipboard device, a Pedestrian User Equipment (PUE), a smart home (a home device with a wireless communication function, such as a refrigerator, a television, a washing machine, or furniture), a game console, a Personal Computer (PC), a kiosk, or a self-service machine, and the like. The Wearable Device includes a smart watch, a smart bracelet, a smart earphone, smart glasses, smart jewelry (a smart bracelet, a smart necklace, a smart ring, a smart necklace, a smart anklet, a smart necklace, and the like), a smart wristband, smart clothes, and the like. The vehicle-mounted device can also be referred to as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, or a vehicle-mounted unit, and the like. It should be noted that the specific type of the terminal is not limited in the embodiments of the present application.

[0556] The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system, and can also be a cloud server. The cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), or cloud computing services based on big data and artificial intelligence platforms.

[0557] For example, the electronic device described above can include, but is not limited to, a source device 100 or a destination device 110 as shown in FIG. 1.

[0558] For example, the electronic device described above can include, but is not limited to, a source device 100 or a destination device 110 as shown in FIG. 1.

[0559] As shown in FIG. 34, the terminal 900 includes, but is not limited to, at least part of the components such as a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0560] Those skilled in the art can understand that the terminal 900 can also include a power supply (such as a battery) for supplying power to each component. The power supply can be logically connected to the processor 910 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The terminal structure shown in FIG. 34 does not constitute a limitation on the terminal. The terminal can include more or fewer components than shown, or combine certain components, or different component arrangements, which are not described here.

[0561] It should be understood that in the embodiments of the present application, the input unit 904 can include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes image data of a still picture or a video obtained by an image acquisition device (such as a camera) in a video acquisition mode or an image acquisition mode, or can process obtained point cloud data. The display unit 906 can include a display panel 9061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 can include a touch detection device and a touch controller. The other input devices 9072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which are not described here.

[0562] In the embodiment of the present application, the radio frequency unit 901 can transmit data to the processor 910 for processing after receiving data from the opposite end. In addition, the radio frequency unit 901 can send data to the opposite end. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0563] The memory 909 can be used to store software programs or instructions and various data. The memory 909 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), etc. In addition, the memory 909 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM), a static random access memory (Static RAM, SRAM), a dynamic random access memory (Dynamic RAM, DRAM), a synchronous dynamic random access memory (Synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), an enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), a synch link dynamic random access memory (Synch link DRAM, SLDRAM) and a direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 909 in the embodiment of the present application includes, but is not limited to, these and any other suitable types of memory.

[0564] The processor 910 can include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 910.

[0565] As an implementation, the terminal 900 is a decoding end, the radio frequency unit 901 is configured to: acquire at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid; and the processor 910 is configured to:

[0566] decode the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice;

[0567] decode the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice;

[0568] merge the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

[0569] As another implementation, the terminal 900 is an encoding end, and the processor 910 is configured to:

[0570] determine at least one inter-frame slice of a to-be-encoded original grid and at least one intra-frame slice of the to-be-encoded original grid;

[0571] encode the at least one inter-frame slice by using an inter-frame encoding mode to obtain at least one inter-frame code stream;

[0572] encode the at least one intra-frame slice by using an intra-frame encoding mode to obtain at least one intra-frame code stream;

[0573] merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base grid code stream.

[0574] In the embodiment, the decoding end avoids decoding the entire base grid by directly using the intra-frame decoding mode or the inter-frame decoding mode, and the decoding effect can be improved.

[0575] It can be understood that the implementation processes of the implementations mentioned in the embodiment can refer to the related descriptions of the method embodiments and achieve the same or corresponding technical effects. To avoid repetition, they will not be described here again.

[0576] The embodiment also provides a readable storage medium, and the readable storage medium stores a program or instructions, which are executed by a processor to implement each process of the above-described slice-based grid decoding method embodiment or the above-described slice-based grid encoding method embodiment and achieve the same technical effects. To avoid repetition, they will not be described here again.

[0577] The processor is the processor in the terminal in the above-described embodiments. The readable storage medium includes a computer readable storage medium, such as a ROM, a RAM, a magnetic disk, or an optical disk. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0578] The chip provided by the embodiment of the present application includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used to run programs or instructions, realizes each process of the above-mentioned slice-based grid decoding method embodiment or the above-mentioned slice-based grid encoding method embodiment, and can achieve the same technical effects. To avoid repetition, it will not be repeated here.

[0579] It should be understood that the chip mentioned in the embodiment of the present application can include a system-on-chip (also known as a system chip, a chip system or a system-on-chip), and can also include a separate display chip and the like.

[0580] The embodiment of the present application further provides a computer program / program product stored in a storage medium, which is executed by at least one processor to realize each process of the above-mentioned slice-based grid decoding method embodiment or the above-mentioned slice-based grid encoding method embodiment, and can achieve the same technical effects. To avoid repetition, it will not be repeated here.

[0581] The embodiment of the present application further provides a coding and decoding system, including: an encoding end and a decoding end, the encoding end can be used to execute the steps of the encoding method as described above, and the decoding end can be used to execute the steps of the decoding method as described above.

[0582] It should be noted that in this paper, the term "include", "contain" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the present application is not limited to the order of functions shown or discussed, and can also include the execution of functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method can be executed in an order different from the described order, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0583] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of computer software product and general hardware platform, of course, also can be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and includes a plurality of instructions for making the decoding end execute the decoding method or the encoding end execute the encoding method described in each embodiment of the present application.

[0584] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A slice-based mesh decoding method, comprising: obtaining, by a decoding end, at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh; decoding, by the decoding end, the at least one inter-frame code stream using an inter-frame decoding mode to obtain at least one inter-frame slice; decoding, by the decoding end, the at least one intra-frame code stream using an intra-frame decoding mode to obtain at least one intra-frame slice; and merging, by the decoding end, the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base mesh of the to-be-decoded base mesh.

2. The method of claim 1, wherein, The obtaining, by the decoding end, of the at least one inter-frame code stream of the to-be-decoded base mesh and the at least one intra-frame code stream of the to-be-decoded base mesh comprises: in a case where a decoding mode used by the to-be-decoded base mesh is not an intra-frame decoding mode, obtaining, by the decoding end, the at least one inter-frame code stream and the at least one intra-frame code stream.

3. The method of claim 1 or 2, wherein, The obtaining, by the decoding end, of the at least one inter-frame code stream of the to-be-decoded base mesh and the at least one intra-frame code stream of the to-be-decoded base mesh comprises: obtaining, by the decoding end, a base mesh code stream of the to-be-decoded base mesh, the base mesh code stream comprising header information and a data unit; obtaining, by the decoding end, the at least one inter-frame code stream and the at least one intra-frame code stream from the data unit based on the header information.

4. The method of claim 3, wherein, The header information comprises at least one of: an identifier for indicating an order of the at least one inter-frame code stream and the at least one intra-frame code stream; an identifier for indicating a data length of the at least one inter-frame code stream; an identifier for indicating a number of the at least one inter-frame code stream; an identifier for indicating a length of each inter-frame code stream in the at least one inter-frame code stream; an identifier for indicating a length of each intra-frame code stream in the at least one inter-frame code stream except for a last intra-frame code stream; an identifier for indicating a data length of the at least one intra-frame code stream; an identifier for indicating a number of the at least one intra-frame code stream; an identifier for indicating a length of each intra-frame code stream in the at least one intra-frame code stream; an identifier for indicating a length of each intra-frame code stream in the at least one intra-frame code stream except for a last intra-frame code stream.

5. The method of any one of claims 1 to 4, wherein, The decoding, by the decoding end, of the at least one inter-frame code stream using the inter-frame decoding mode to obtain the at least one inter-frame slice comprises: decoding, by the decoding end, a first inter-frame code stream in the at least one inter-frame code stream to obtain first indication information and to-be-decoded information of a first inter-frame slice in the at least one inter-frame slice, the first indication information being used for indicating at least one reference surface slice used by the first inter-frame slice in a reference base mesh of the to-be-decoded base mesh; decoding, by the decoding end, the to-be-decoded information using the inter-frame decoding mode based on the at least one reference surface slice to obtain the first inter-frame slice.

6. The method of claim 5, wherein, The first indication information is an array with a length equal to a number of surface slices of the reference base mesh, and a value at any position in the array is used for indicating whether a reference surface slice used by the first inter-frame slice includes a surface slice in the reference base mesh corresponding to the any position.

7. The method of any one of claims 1 to 6, wherein, The reconstructed base mesh includes texture coordinates of the at least one inter-frame patch and texture coordinates of the at least one intra-frame patch, and the method further includes: The decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch; The decoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch.

8. The method of claim 7, wherein, The decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch, including: The decoding end multiplies components of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

9. The method of claim 8, wherein, The decoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch, including: The decoding end multiplies components of the texture coordinates of the at least one intra-frame patch in the first direction by a second coefficient first, and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra-frame patch. The sum of the first coefficient and the second coefficient is 1.

10. A patch-based mesh encoding method, including: An encoding end determines at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded; The encoding end encodes the at least one inter-frame patch in an inter-frame encoding mode to obtain at least one inter-frame code stream; The encoding end encodes the at least one intra-frame patch in an intra-frame encoding mode to obtain at least one intra-frame code stream; The encoding end combines the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream.

11. The method of claim 10, wherein, The encoding end determines at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded, including: In a case where an encoding mode used by the original mesh to be encoded is not an intra-frame encoding mode, the encoding end determines at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded.

12. The method of claim 10 or 11, wherein, The encoding end combines the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream, including: The encoding end combines the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit; The encoding end combines header information of the base mesh code stream and the data unit to obtain the base mesh code stream.

13. The method of claim 12, wherein, The header information includes at least one of: an identifier for indicating an order of the at least one inter-frame code stream and the at least one intra-frame code stream; an identifier for indicating a data length of the at least one inter-frame code stream; an identifier for indicating a number of the at least one inter-frame code stream; an identifier for indicating lengths of respective inter-frame code streams in the at least one inter-frame code stream; an identifier for indicating lengths of respective intra-frame code streams except a last intra-frame code stream in the at least one inter-frame code stream; an identifier for indicating a data length of the at least one intra code stream; an identifier for indicating a number of the at least one intra code stream; an identifier for indicating a length of each of the at least one intra code stream; an identifier for indicating a length of each of the at least one intra code stream except the last one.

14. The method of any one of claims 10 to 13, wherein, the encoding end encodes the at least one inter patch in the inter encoding mode to obtain at least one inter code stream, including: the encoding end encodes a first inter patch in the at least one inter patch in the inter encoding mode based on at least one reference patch in a reference base mesh of the to-be-encoded original mesh to obtain encoding information of the first inter patch; the encoding end encodes first indication information and the encoding information of the first inter patch to obtain a first inter code stream in the at least one first inter code stream, and the first indication information is used to indicate the at least one reference patch.

15. The method of claim 14, wherein, the first indication information is an array with a length equal to a number of patches in the reference base mesh, and a value at any position in the array is used to indicate whether a reference patch used by the first inter patch includes a patch in the reference base mesh corresponding to the any position.

16. The method of any one of claims 10 to 15, wherein, the reconstructed base mesh of the to-be-encoded original mesh includes texture coordinates of the at least one inter patch and texture coordinates of the at least one intra patch, and the method further includes: the encoding end scales the texture coordinates of the at least one inter patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter patch; the encoding end scales the texture coordinates of the at least one intra patch in the first direction to obtain scaled texture coordinates of the at least one intra patch.

17. The method of claim 16, wherein, the encoding end scales the texture coordinates of the at least one inter patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter patch, including: the encoding end multiplies a component of the texture coordinates of the at least one inter patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter patch.

18. The method of claim 17, wherein, the encoding end scales the texture coordinates of the at least one intra patch in the first direction to obtain scaled texture coordinates of the at least one intra patch, including: the encoding end multiplies a component of the texture coordinates of the at least one intra patch in the first direction by a second coefficient and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra patch; wherein a sum of the first coefficient and the second coefficient is 1.

19. The method of any one of claims 10 to 18, wherein, the encoding end determines the at least one inter patch, including: the encoding end determines a registration base mesh of the to-be-encoded original mesh and a registration subdivision deformation mesh of the to-be-encoded original mesh based on a reference base mesh of the to-be-encoded original mesh and a reference subdivision deformation mesh of the to-be-encoded original mesh; The registration base mesh is a mesh obtained by deforming the original mesh to be encoded, has the same number of vertices as the reference base mesh, and has the same connectivity relationship as the reference base mesh; and the registration subdivided deformed mesh is a mesh obtained by deforming the original mesh to be encoded, has the same number of vertices as the reference subdivided deformed mesh, and has the same connectivity relationship as the reference subdivided deformed mesh. The encoding end marks the face patches in the registration base mesh as matched or unmatched face patches based on the matching detection result between the registration base mesh and the registration subdivided deformed mesh, to obtain a first marking result. The encoding end determines the at least one inter-frame patch in the registration base mesh based on the first marking result.

20. The method of claim 19, wherein, The encoding end determines the registration base mesh of the original mesh to be encoded and the registration subdivided deformed mesh of the original mesh to be encoded based on a reference base mesh of the original mesh to be encoded and a reference subdivided deformed mesh of the original mesh to be encoded, and includes the following steps. The encoding end deforms the original mesh to be encoded by taking the reference subdivided deformed mesh as a target mesh, to obtain an intermediate mesh. The encoding end deforms the reference subdivided deformed mesh by taking the intermediate mesh as a target mesh, to obtain the registration subdivided deformed mesh. The encoding end deforms the reference base mesh by taking the registration subdivided deformed mesh as a target mesh, to obtain the registration base mesh.

21. The method of claim 19 or 20, wherein, The first marking result includes a marking result of a first face patch in the registration base mesh. The encoding end marks the face patches in the registration base mesh as matched or unmatched face patches based on the matching detection result between the registration base mesh and the registration subdivided deformed mesh, to obtain a first marking result, and includes the following steps. The encoding end obtains a first bounding box in which the first face patch is located. The encoding end obtains a first region in the original mesh to be encoded corresponding to the first bounding box and a second region in the matched subdivided deformed mesh corresponding to the first bounding box based on the first bounding box. The encoding end calculates a first weighted average normal vector of the face patches in the first region and a second weighted average normal vector of the face patches in the second region. In a case where an included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, the encoding end determines that the marking result of the first face patch is that the first face patch is marked as an unmatched face patch, to obtain the marking result of the first face patch, or in a case where the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, the encoding end marks the first face patch as a matched face patch, to obtain the marking result of the first face patch.

22. The method of any one of claims 19-21, wherein, The encoding end determines the at least one inter-frame patch in the registration base mesh based on the first marking result, and includes the following steps. The encoding end obtains at least one third region in which the unmatched face patch is located based on the first marking result. The encoding end re-labels the patches in a first region in the at least one third region, in which the total area of the patches is less than or equal to a preset first area threshold, as matched patches to obtain a third labeling result; The encoding end obtains at least one fourth region in which the patches labeled as matched patches are located based on the third labeling result; The encoding end re-labels the patches in a second region in the at least one fourth region, in which the total area of the patches is less than or equal to the preset first area threshold, as unmatched patches to obtain a fourth labeling result; The encoding end determines the region in which the patches labeled as matched patches are located as the at least one inter-frame patch based on the fourth labeling result.

23. The method of any one of claims 19-22, wherein, The encoding end determines the at least one intra-frame patch, comprising: The encoding end labels the patches in the original mesh to be encoded as matched or unmatched patches based on the first labeling result to obtain a second labeling result; The encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result; The encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch.

24. The method of claim 23, wherein, The second labeling result comprises a labeling result of a second patch in the original mesh to be encoded; The encoding end labels the patches in the original mesh to be encoded as matched or unmatched patches based on the labeling result to obtain a second labeling result, comprising: The encoding end obtains the barycenter of the second patch; The encoding end searches for a nearest neighbor point of the barycenter of the second patch in the registration base mesh; The encoding end labels the second patch as a matched or unmatched patch based on the label of the patch in which the nearest neighbor point is located to obtain a labeling result of the second patch.

25. The method of claim 23 or 24, wherein, The encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result, comprising: The encoding end obtains at least one fifth region in which the patches labeled as unmatched patches are located based on the second labeling result; The encoding end re-labels the patches in a third region in the at least one fifth region, in which the total area of the patches is less than or equal to a preset second area threshold, as matched patches to obtain a fifth labeling result; The encoding end obtains at least one sixth region in which the patches labeled as matched patches are located based on the fifth labeling result; The encoding end re-labels the patches in a fourth region in the at least one sixth region, in which the total area of the patches is less than or equal to the preset second area threshold, as unmatched patches to obtain a sixth labeling result; The encoding end determines the region in which the patches labeled as unmatched patches are located as the at least one initial intra-frame patch based on the sixth labeling result.

26. The method of any one of claims 23-25, wherein, The encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch, comprising: The encoding end obtains at least one edge in the registration base mesh in which there are adjacent patches labeled as matched and unmatched; The encoding end obtains at least one vertex in the original mesh to be encoded in which there are adjacent patches labeled as matched and unmatched; The encoding end searches for a nearest neighbor of the first vertex among the at least one vertex corresponding to the at least one edge; The encoding end deforms the nearest neighbor of the first vertex to the first vertex; The encoding end searches for a nearest neighbor of a second vertex among the at least one vertex, where the second vertex is different from the nearest neighbor of the first vertex; The encoding end obtains an edge on which the nearest neighbor of the second vertex is located based on the nearest neighbor of the second vertex; The encoding end determines a position on the edge on which the nearest neighbor of the second vertex is located, which is closest to the second vertex; The encoding end deforms the second vertex to the position closest to the second vertex to obtain the at least one intra-frame patch.

27. The method of any one of claims 10 to 26, wherein, The method further comprises: The encoding end obtains a reconstructed base mesh of the original mesh to be encoded; The encoding end deforms the reconstructed base mesh based on the original mesh to be encoded to obtain a subdivision mesh of the original mesh to be encoded and a subdivision deformed mesh of the original mesh to be encoded; The encoding end determines displacement information of the original mesh to be encoded based on the subdivision mesh of the original mesh to be encoded and the subdivision deformed mesh of the original mesh to be encoded; The encoding end encodes the displacement information to obtain a displacement code stream.

28. The method of claim 27, wherein, The method further comprises: The encoding end deforms the subdivision mesh of the original mesh to be encoded based on reconstructed displacement information of the displacement information to obtain a reconstructed deformed mesh of the original mesh to be encoded.

29. A patch-based mesh decoding apparatus, comprising: an obtaining unit, configured to obtain at least one inter-frame code stream of a base mesh to be decoded and at least one intra-frame code stream of the base mesh to be decoded; a first decoding unit, configured to decode the at least one inter-frame code stream in an inter-frame decoding mode to obtain at least one inter-frame patch; a second decoding unit, configured to decode the at least one intra-frame code stream in an intra-frame decoding mode to obtain at least one intra-frame patch; a merging unit, configured to merge the at least one inter-frame patch and the at least one intra-frame patch to obtain a reconstructed base mesh of the base mesh to be decoded.

30. The apparatus of claim 29, wherein, The obtaining unit is specifically configured to: in a case where a decoding mode used by the base mesh to be decoded is not an intra-frame decoding mode, obtain the at least one inter-frame code stream and the at least one intra-frame code stream.

31. A slice-based trellis encoding apparatus, characterized by comprising: comprising: a determining unit, configured to determine at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded; a first encoding unit, configured to encode the at least one inter-frame patch in an inter-frame encoding mode to obtain at least one inter-frame code stream; a second encoding unit, configured to encode the at least one intra-frame patch in an intra-frame encoding mode to obtain at least one intra-frame code stream; a merging unit, configured to merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream.

32. The apparatus of claim 31, wherein, The determining unit is specifically configured to: In a case where the coding mode used by the original grid to be encoded is not an intra coding mode, determining at least one inter slice of the original grid to be encoded and at least one intra slice of the original grid to be encoded.

33. An electronic device, comprising a processor and a memory, the memory storing programs or instructions executable on the processor, the programs or instructions, when executed by the processor, implement steps of the slice-based grid decoding method according to any one of claims 1 to 9, or implement steps of the slice-based grid encoding method according to any one of claims 10 to 28.

34. A readable storage medium, the readable storage medium storing programs or instructions, the programs or instructions, when executed by a processor, implement steps of the slice-based grid decoding method according to any one of claims 1 to 9, or implement steps of the slice-based grid encoding method according to any one of claims 10 to 28.

35. A chip, the chip comprising a processor and a communication interface, the communication interface and the processor being coupled, the processor being configured to execute programs or instructions, implement steps of the slice-based grid decoding method according to any one of claims 1 to 9, or implement steps of the slice-based grid encoding method according to any one of claims 10 to 28.

Images