Grid decoding method and coding method based on slices and related equipment

Abstract

The invention discloses a slice-based grid decoding method, a slice-based grid coding method and related equipment, and belongs to the field of coding and decoding, and the method comprises the following steps: a decoding end obtains at least one inter-frame code stream of a to-be-decoded basic grid and at least one intra-frame code stream of the to-be-decoded basic grid; the decoding end decodes the at least one inter-frame code stream in an inter-frame decoding mode to obtain at least one inter-frame slice; the decoding end decodes the at least one intra-frame code stream in an intra-frame decoding mode to obtain at least one intra-frame slice; and the decoding end combines the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed basic grid of the basic grid to be decoded. According to the slice-based grid decoding method provided by the embodiment of the invention, the decoding end does not need to directly use an intra-frame decoding mode or an inter-frame decoding mode to decode the whole basic grid, so that the decoding effect can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coding technology, and more particularly, to a slice-based mesh decoding method, a slice-based mesh encoding method and related devices. BACKGROUND

[0002] For a three-dimensional mesh, an encoding end can first preprocess an original mesh to generate a base mesh and a deformed mesh. Specifically, the encoding end first subdivides the base mesh to obtain a subdivided mesh of the base mesh, and then adjusts the subdivided mesh based on the original mesh to obtain the deformed mesh.

[0003] In related technologies, the encoding end can encode the base mesh by using an intra-frame encoding mode or an inter-frame encoding mode to obtain a base mesh code stream, and the decoding end can decode the base mesh code stream by using an intra-frame decoding mode or an inter-frame decoding mode to obtain a reconstructed base mesh.

[0004] However, if the encoding end only encodes the entire base mesh by using the intra-frame encoding mode or the inter-frame encoding mode, information redundancy or excessively low code rate may exist in the base mesh code stream in the time domain, which may in turn reduce the encoding and decoding efficiency of the entire base mesh. SUMMARY

[0005] Embodiments of the present application provide a slice-based mesh decoding method, a slice-based mesh encoding method and related devices, which can improve decoding efficiency.

[0006] In a first aspect, a slice-based mesh decoding method is provided, comprising:

[0007] A decoding end obtains at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh.

[0008] The decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice.

[0009] The decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice.

[0010] The decoding end combines the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base mesh of the to-be-decoded base mesh.

[0011] In a second aspect, a slice-based mesh encoding method is provided, comprising:

[0012] An encoding end determines at least one inter-frame slice of a to-be-encoded original mesh and at least one intra-frame slice of the to-be-encoded original mesh.

[0013] The encoding end encodes the at least one inter-frame slice in an inter-frame coding mode to obtain at least one inter-frame code stream;

[0014] The encoding end encodes the at least one intra-frame slice in an intra-frame coding mode to obtain at least one intra-frame code stream;

[0015] The encoding end combines the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a basic grid code stream.

[0016] In a third aspect, a slice-based grid decoding apparatus is provided, comprising:

[0017] An acquisition unit is configured to acquire at least one inter-frame code stream of a basic grid to be decoded and at least one intra-frame code stream of the basic grid to be decoded;

[0018] A first decoding unit is configured to decode the at least one inter-frame code stream in an inter-frame decoding mode to obtain at least one inter-frame slice;

[0019] A second decoding unit is configured to decode the at least one intra-frame code stream in an intra-frame decoding mode to obtain at least one intra-frame slice;

[0020] A combination unit is configured to combine the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed basic grid of the basic grid to be decoded.

[0021] In a fourth aspect, a slice-based grid encoding apparatus is provided, comprising:

[0022] A determination unit is configured to determine at least one inter-frame slice of an original grid to be encoded and at least one intra-frame slice of the original grid to be encoded;

[0023] A first encoding unit is configured to encode the at least one inter-frame slice in an inter-frame coding mode to obtain at least one inter-frame code stream;

[0024] A second encoding unit is configured to encode the at least one intra-frame slice in an intra-frame coding mode to obtain at least one intra-frame code stream;

[0025] A combination unit is configured to combine the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a basic grid code stream.

[0026] In a fifth aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect or the steps of the method according to the second aspect.

[0027] In a sixth aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the communication interface is configured to acquire at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid; and the processor is configured to:

[0028] decode the at least one inter-frame code stream using an inter-frame decoding mode to obtain at least one inter-frame slice;

[0029] decode the at least one intra-frame code stream using an intra-frame decoding mode to obtain at least one intra-frame slice;

[0030] merge the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

[0031] In a seventh aspect, an electronic device is provided, comprising a processor and a communication interface, wherein the processor is configured to:

[0032] determine at least one inter-frame slice of a to-be-encoded original grid and at least one intra-frame slice of the to-be-encoded original grid;

[0033] encode the at least one inter-frame slice using an inter-frame encoding mode to obtain at least one inter-frame code stream;

[0034] encode the at least one intra-frame slice using an intra-frame encoding mode to obtain at least one intra-frame code stream;

[0035] merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base grid code stream.

[0036] In an eighth aspect, an electronic device is provided, comprising a memory configured to store data, and a processing circuit configured to implement steps of the method according to the first aspect or steps of the method according to the second aspect.

[0037] In a ninth aspect, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement steps of the method according to the first aspect or steps of the method according to the second aspect.

[0038] In a tenth aspect, a coding system is provided, comprising an encoding end and a decoding end, wherein the decoding end is configured to execute steps of the method according to the first aspect, and the encoding end is configured to execute steps of the method according to the second aspect.

[0039] In an eleventh aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is configured to run a program or instructions to implement steps of the method according to the first aspect or steps of the method according to the second aspect.

[0040] In a twelfth aspect, a computer program / program product is provided, which is stored in a storage medium, and the program / program product is executed by at least one processor to implement the steps of the method according to the first aspect, or to implement the steps of the method according to the second aspect.

[0041] In the embodiments of the present application, the decoding end obtains at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh; the decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice; the decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice; and the decoding end combines the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base mesh of the to-be-decoded base mesh. That is, the decoding end avoids directly decoding the entire base mesh by using the intra-frame decoding mode or the inter-frame decoding mode, and thus the decoding effect can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 is a schematic diagram of a coding system provided by the embodiments of the present application.

[0044] Figure 2a is an encoding flowchart of V3C provided by the embodiments of the present application.

[0045] Figure 2b is a decoding flowchart of V3C provided by the embodiments of the present application.

[0046] Figure 3 is an encoding flowchart of VDMC provided by the embodiments of the present application.

[0047] Figure 4 is an example of the principle of mesh simplification provided by the embodiments of the present application.

[0048] Figure 5 is an example of the basic idea of the subdivision and deformation module provided by the embodiments of the present application.

[0049] Figure 6 is an example of the subdivision principle provided by the embodiments of the present application.

[0050] Figure 7is a schematic flowchart of a compression process of a base mesh provided by embodiments of the present application.

[0051] Figure 8 is an example of a mode of a triangular face provided by embodiments of the present application.

[0052] Figure 9 is a schematic flowchart of a texture map conversion provided by embodiments of the present application.

[0053] Figure 10 is a decoding flowchart of a VDMC provided by embodiments of the present application.

[0054] Figure 11 is a schematic flowchart of a tile-based mesh decoding method provided by embodiments of the present application.

[0055] Figure 12 is an example of a base mesh bitstream provided by embodiments of the present application.

[0056] Figure 13 is another example of a base mesh bitstream provided by embodiments of the present application.

[0057] Figure 14 is yet another example of a base mesh bitstream provided by embodiments of the present application.

[0058] Figure 15 is an example of an adjustment process of texture coordinates provided by embodiments of the present application.

[0059] Figure 16 is a schematic diagram of a tile-based mesh decoding flow provided by embodiments of the present application.

[0060] Figure 17 is a schematic diagram of a decoding flow of a displacement bitstream provided by embodiments of the present application.

[0061] Figure 18 is a schematic flowchart of a tile-based mesh encoding method provided by embodiments of the present application.

[0062] Figure 19 is a schematic diagram of a generation process of at least one inter-frame tile and at least one intra-frame tile provided by embodiments of the present application.

[0063] Figure 20 is an example of a registered base mesh, an original mesh to be encoded, an inter-frame tile, an initial intra-frame tile, an intra-frame tile, and a base mesh provided by embodiments of the present application.

[0064] Figure 21 is an example of a generation process of a registered base mesh and a configuration of a subdivision deformation mesh provided by embodiments of the present application.

[0065] Figure 22is an example of a generation process of at least one inter slice and at least one initial intra slice provided by an embodiment of the present application.

[0066] Figure 23 is an example of a boundary and a boundary point provided by an embodiment of the present application.

[0067] Figure 24 is a schematic diagram of boundary stretching provided by an embodiment of the present application.

[0068] Figure 25 is an example of obtaining at least one intra slice based on at least one initial intra slice provided by an embodiment of the present application.

[0069] Figure 26 is a schematic diagram of a generation process of displacement information provided by an embodiment of the present application.

[0070] Figure 27 is a schematic diagram of inter slice encoding and intra slice encoding provided by an embodiment of the present application.

[0071] Figure 28 is a schematic diagram of a reconstructed deformed mesh provided by an embodiment of the present application.

[0072] Figure 29 is a schematic flow chart of a slice-based mesh encoding process provided by an embodiment of the present application.

[0073] Figure 30 is a schematic diagram of an encoding process of displacement information provided by an embodiment of the present application.

[0074] Figure 31 is a schematic block diagram of a slice-based mesh decoding apparatus provided by an embodiment of the present application.

[0075] Figure 32 is a schematic block diagram of a slice-based mesh encoding apparatus provided by an embodiment of the present application.

[0076] Figure 33 is a schematic block diagram of an electronic device provided by an embodiment of the present application.

[0077] Figure 34 is a schematic structural diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0078] The technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0079] The terms "first", "second", and the like in the specification are used to distinguish between similar objects, and are not used to describe a particular sequential or chronological order. It is to be understood that such terms are used interchangeably and that "first", "second", and the like can be interchanged, where appropriate, to distinguish between similar objects in order to describe the embodiments of the present application in other than the specific order illustrated or described. It is also to be understood that the terminology "first", "second", and the like used in the description refers to the objects in a generic sense, and is not to be interpreted in a specific sense unless explicitly so defined in the specification. For example, the first object can be one or more. In addition, "or" in the present application means at least one of the connected objects. For example, "A or B" covers three cases, namely, case one: including A and not including B; case two: including B and not including A; and case three: including both A and B. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0080] Before introducing the technical solutions provided by the embodiments of the present application, the meanings of some terms therein will be introduced first.

[0081] Three-dimensional mesh: A three-dimensional mesh is a three-dimensional object surface composed of numerous polygons in space, and a polygon is composed of vertices and edges.

[0082] In recent years, with the rapid development of multimedia technology, relevant research results have been rapidly industrialized and have become an important part of people's lives. Three-dimensional models have become a new generation of digital media after audio, images, and video. Three-dimensional mesh is a common three-dimensional model representation method. Compared with traditional images, video, and other multimedia, three-dimensional mesh models have stronger interactivity and realism, making them more and more widely used in various fields such as business, manufacturing, construction, education, medicine, entertainment, art, and military.

[0083] Although there are many kinds of three-dimensional mesh representation methods at present, triangular mesh is still the most common representation method. A three-dimensional mesh can be regarded as being composed of three basic elements, namely vertices, edges, and faces. Vertices are the most basic elements in the mesh, which define a position in a three-dimensional space. An edge is a line segment connecting two vertices in the mesh. A face can be regarded as a polygon formed by the closed path of an edge. For a triangular mesh, each face is a triangle.

[0084] The information contained in a mesh is generally classified into three categories: geometry information, connectivity information, and attribute information. The geometry information is the position of each vertex in the mesh in the three-dimensional space. The connectivity information describes the relationship between elements in the mesh, i.e., the connection relationship between vertices. The attribute information is optional, which can associate attributes to the corresponding mesh elements (such as vertex color, normal vector, etc. can be associated with the mesh vertices). The mesh parameterization can also be used to map the mesh from the three-dimensional space to a two-dimensional planar region, and the mapping relationship is generally described by a set of parameter coordinates, referred to as UV coordinates or texture coordinates, which are associated with the mesh vertices. The two-dimensional mapping can be used to represent high-resolution attribute information, such as texture, normal vector, etc.

[0085] Figure 1 is a schematic diagram of a coding system 10 provided by an embodiment of the present application. The technical solution of the embodiment of the present application relates to coding (CODEC) of mesh data (including encoding or decoding).

[0086] As shown in Figure 1 , the coding system 10 includes a source device 100 that provides encoded mesh data that is decoded and displayed by a destination device 110. Specifically, the source device 100 provides mesh data to the destination device 110 via a communication medium 120. The source device 100 and the destination device 110 can include any one or more of a desktop computer, a notebook (i.e., laptop) computer, a tablet computer, a set-top box, a mobile phone, a wearable device (e.g., a smart watch or a wearable camera), a television, a camera, a display device, a vehicle-mounted device, a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a digital media player, a video gaming console, a video conferencing device, a video streaming device, a broadcast receiver device, a broadcast transmitter device, a spacecraft, an airplane, a robot, a satellite, etc.

[0087] In Figure 1 the example, the source device 100 includes a data source 101, a memory 102, an encoder 200, and an output interface 104. The destination device 110 includes an input interface 111, a decoder 300, a memory 113, and a display device 114. The source device 100 represents an example of an encoding device, and the destination device 110 represents an example of a decoding device. In other examples, the source device 100 and the destination device 110 can not include some of the components in Figure 1 , or can include other components not shown in Figure 1The source device 100 and the destination device 110 are illustrated as separate devices, but in some examples, they can be integrated in one device. In such embodiments, the corresponding functionalities of the source device 100 and the destination device 110 can be implemented using the same hardware or software, or using separate hardware or software, or any combination thereof.

[0088] Although Figure 1 The source device 100 and the destination device 110 are illustrated as separate devices, but in some examples, they can be integrated in one device. In such embodiments, the corresponding functionalities of the source device 100 and the destination device 110 can be implemented using the same hardware or software, or using separate hardware or software, or any combination thereof.

[0089] In some examples, the source device 100 and the destination device 110 can perform one-way data transmission or two-way data transmission. If it is two-way data transmission, the source device 100 and the destination device 110 can operate in a substantially symmetrical manner, i.e., each of the source device 100 and the destination device 110 includes an encoder and a decoder.

[0090] The data source 101 represents a source of mesh data (i.e., original, uncoded mesh data) and provides the encoder 200 with mesh data containing mesh data, which is encoded by the encoder 103. The source device 100 can include a capture device (e.g., a camera device, a sensor device, or a scanning device), an archive including previously captured mesh data, or a feed interface for receiving mesh data from a data content provider. Among them, the camera device can include a normal camera, a stereo camera, and a light field camera, etc., the sensor device can include a laser device, a radar device, etc., and the scanning device can include a three-dimensional laser scanning device, etc. The mesh data can be obtained by capturing a real-world visual scene through the capture device. Alternatively, the data source 101 can generate computer graphics-based data as source data, or combine real-time data, archived data, and computer-generated data. For example, the data source generates mesh data based on a virtual object (e.g., a virtual three-dimensional object and a virtual three-dimensional scene obtained by three-dimensional modeling).

[0091] Memory 102 of source device 100 and memory 113 of destination device 110 represent general purpose memories. In some examples, memory 102 can store raw data from data source 101, and memory 113 can store decoded mesh data from decoder 300. Additionally or alternatively, memory 102, 113 can store software instructions executable by, for example, encoder 200 and decoder 300, respectively. Although memory 102 and memory 113 are shown separately from encoder 200 and decoder 300 in this example, it should be understood that encoder 200 and decoder 300 can also include internal memories for functionally similar or equivalent purposes. If encoder 200 and decoder 300 are deployed on the same hardware device, memory 102 and memory 113 can be the same memory. Furthermore, memory 102, 113 can store encoded mesh data, for example, output from encoder 200 and input to decoder 300. In some examples, portions of memory 102, 113 can be allocated as one or more mesh buffers, for example, for storing raw, decoded, or encoded mesh data.

[0092] In some examples, source device 100 can output encoded data from output interface 104 to memory 113. Similarly, destination device 110 can access encoded data from memory 113 via input interface 111. Memory 113 or memory 102 can comprise any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, Digital Versatile Discs (DVD), Compact Disc Read-Only Memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded mesh data.

[0093] Output interface 104 can include any type of medium or device capable of transmitting encoded mesh data from source device 100 to destination device 110. For example, output interface 104 can include a transmitter or a transceiver, such as an antenna, configured to transmit encoded mesh data from source device 100 directly to destination device 110 in real-time. The encoded mesh data can be modulated according to a communication standard of a wireless communication protocol and transmitted to destination device 110.

[0094] Communication medium 120 can include transient media, such as a wireless broadcast or wired network transmission. For example, communication medium 120 can include radio frequency (RF) spectrum or one or more physical transmission lines (e.g., cable). Communication medium 120 can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. Communication medium 120 can also be in a form of storage media, such as a hard disk, flash drive, compact disk, digital tape, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded grid data.

[0095] In some embodiments, communication medium 120 can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from source device 100 to destination device 110. For example, a server (not shown) can receive the encoded grid data from source device 100 and provide to destination device 110, e.g., via network transmission to destination device 110. The server can include a web server (e.g., for a website), a server configured to provide a file transfer protocol service such as File Transfer Protocol (FTP) or File Delivery Over Unidirectional Transport (FLUTE) protocol, a content delivery network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Services (MBMS) or evolved Multimedia Broadcast Multicast Service (eMBMS) server, or a Network-attached storage (NAS) device, etc. The server can implement one or more HTTP streaming protocols such as MPEG Media Transport (MMT) protocol, Dynamic Adaptive Streaming over HTTP (DASH) protocol, HTTP Live Streaming (HLS) protocol, or Real Time Streaming Protocol (RTSP), etc.

[0096] Destination device 110 can access the encoded mesh data from a server, e.g., through a wireless channel (e.g., a Wi-Fi connection) or a wired connection (e.g., a Digital subscriber line (DSL), a cable modem, etc.) for accessing encoded mesh data stored on the server.

[0097] Output interface 104 and input interface 111 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to the IEEE 802.11 standards or the IEEE 802.15 standards (e.g., ZigBee™), the Bluetooth standard, etc., or other physical components. In examples where output interface 104 and input interface 111 comprise wireless components, output interface 104 and input interface 111 can be configured to transfer data, such as encoded mesh data, according to WIFI, Ethernet, cellular networks (such as 4G, LTE (Long-Term Evolution), LTE-Advanced, 5G, 6G, etc.), etc.

[0098] Input interface 111 of destination device 110 receives the encoded bitstream from communication medium 120. The encoded bitstream can include high-level syntax elements and encoded data units (e.g., sequences, groups of pictures, pictures, slices, blocks, etc.), where the high-level syntax elements are used to decode the encoded data units to yield decoded mesh data. Display device 114 displays the decoded mesh data to a user. Display device 114 can comprise a Cathode ray tube (CRT), a liquid-crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other type of display device. In some examples, destination device 110 can not have display device 114, e.g., if the decoded mesh data is used to determine a location of a physical object, display device 114 can be replaced with a processor.

[0099] The encoder 200 and the decoder 300 can be implemented as one or more of various processing circuitry, which can include one or more microprocessors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), discrete logic circuitry, hardware, or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure.

[0100] In almost all application fields using three-dimensional mesh (such as computational simulation, entertainment, medical imaging, digital cultural relics, computer design, e-commerce, etc.), with the increasing demand for three-dimensional mesh models in visual effects, the models are becoming more and more complex, and the accuracy of the models is also increasing, so the amount of data required to represent the three-dimensional mesh is also increasing. The above problems lead to the increasing complexity of processing, visualization, transmission and storage of three-dimensional mesh. Three-dimensional mesh compression can be considered as a way to solve the above problems, which reduces the size of the model data, and is beneficial to the processing, storage and transmission of three-dimensional mesh, so it is necessary to propose an efficient and general three-dimensional mesh compression algorithm.

[0101] Recently, the international standardization organization MPEG in the field of audio and video coding compression has begun to develop a compression standard for three-dimensional mesh, namely Video-based dynamic mesh coding (VDMC), which is based on the existing Visual Volumetric Video-based Coding (V3C) standard. The V3C standard provides a general method for compressing three-dimensional models, which can be presented in the form of point cloud, mesh or panoramic video. The compression method of three-dimensional mesh model is compatible with the standard, which helps to promote the method and its applicability. Therefore, it is of great significance to optimize the three-dimensional mesh coding method in VDMC and combine the optimization method with the V3C standard. One possible optimization method is to optimize the basic mesh coding, which divides a basic mesh or a sub-mesh of a basic mesh into two kinds of slices (intra and inter) for coding.

[0102] V3C standard: The V3C standard provides a method for encoding and decoding various three-dimensional media through video or image encoding technology. Specifically, it converts three-dimensional media content from a three-dimensional representation to multiple two-dimensional representations (referred to as V3C components) through projection or the like before encoding, and then encodes the two-dimensional representations using existing video or image encoding technology. An example diagram of the V3C standard is shown in FIG. 1, which shows that the V3C components mainly include occupancy components, geometry components, and attribute components. The occupancy component can represent which regions in the two-dimensional representation are associated with the data of the three-dimensional representation; the geometry component represents information related to the position of the three-dimensional data in space, and the attribute component can provide attribute information corresponding to the vertices, such as material, texture, etc. In addition, the components also contain information on how to reconstruct the three-dimensional model from these components, which is referred to as atlas information. Figure 2a and 2b

[0103] The atlas information is used to associate all the components, and additional information for reconstructing from two dimensions to three dimensions is also contained in the atlas component. The atlas is composed of multiple basic units, which are referred to as patches. For each patch, it represents a block of regions in the available two-dimensional components and contains the information needed to project the region back to the three-dimensional space.

[0104] VDMC: VDMC is a standard for compressing three-dimensional meshes developed by MPEG, and its main idea is to compress three-dimensional meshes by using the existing V3C standard. Since three-dimensional meshes have connection information that needs to be encoded, the specific encoding process is slightly different from that of V3C, and the syntax semantics and decoding operations of the decoding end of the V3C standard need to be extended to support the decoding and reconstruction of three-dimensional meshes.

[0105] The overall framework of the encoder and decoder provided by the embodiments of the present application is introduced below taking VDMC as an example.

[0106] The overall framework of the encoding end is shown in FIG. 2. Figure 3 ​As shown, for the input mesh, first simplification is performed by the simplification module, then new texture coordinates are generated by mesh parameterization, and then the parameterized mesh is subdivided and deformed, i.e. new vertices are inserted on the mesh according to a specific subdivision method and the distance from the subdivided mesh vertices to the nearest neighbor vertices of the input mesh is calculated, which is called displacement information. Subsequently, the parameterized mesh, i.e. the vertex position of the mesh before subdivision and deformation, is adjusted according to the displacement information, and the adjusted mesh is called the base mesh and is sent to the base mesh encoding module for compression. When encoding the base mesh, it is divided into three types of sub-meshes for independent encoding. After encoding the base mesh, it is reconstructed, and then the order of the displacement is adjusted according to the vertex order of the reconstructed base mesh. Subsequently, the vertex displacement information after adjusting the order is first wavelet transformed, the transformed coefficients are quantized, then the quantized coefficients are arranged into a two-dimensional image according to a specific scanning order, and the two-dimensional image is encoded using a video encoder. Then, the reconstructed displacement information is applied to the subdivided base mesh to obtain the reconstructed subdivided and deformed mesh, and the mesh, the original input mesh and the corresponding texture map are input into the corresponding texture map conversion module to obtain the texture map corresponding to the reconstructed mesh. Similarly, the texture map is encoded using a video encoder. For the parameters used in the encoding process, such as the type of video encoder used, the type of mesh encoder, the transformation parameters, the quantization parameters, etc., are transmitted to the decoding end through auxiliary information.

[0107] The specific process of encoding is described below:

[0108] (1) Mesh simplification.

[0109] Mesh simplification is to simplify the current input mesh to a base mesh with relatively fewer points and faces, and to maintain the shape of the original mesh as much as possible. The focus of mesh simplification is the simplification operation and the corresponding error metric. A feasible mesh simplification operation is shown in Figure 4 The vertices at both ends of an edge are combined into one vertex and the connection between the two vertices is deleted. This process is repeated in the entire mesh according to certain rules to reduce the number of faces and vertices of the mesh to the target value.

[0110] During the simplification process, an error metric can be selected to optimize the simplification result. For example, the sum of the equation coefficients of all adjacent faces of a vertex can be selected as the error metric of the vertex, and the error metric of the corresponding edge is the sum of the error metrics of the two vertices on the edge. In other words, the error generated by the combination of an edge is the sum of the distances from the combined vertex to all the adjacent planes of the original two vertices of the edge.

[0111] After the simplification operations and corresponding error metrics are determined, the mesh simplification is performed iteratively. First, the vertex error of the initial mesh is calculated, and then the error of each edge is obtained. Then, each edge is sorted by error from small to large, and the edge with the smallest error is selected for merging. At the same time, the position of the merged vertex is calculated, and the error of all edges related to the merged vertex is updated. That is, the order of edge sorting is updated to ensure that each iteration is based on the global error metric. Through iteration, the mesh is simplified to meet the number required for lossy encoding.

[0112] (2) Mesh parameterization.

[0113] Mesh parameterization is used to generate corresponding texture coordinates for the mesh. Mesh parameterization can be performed by algorithms such as Isochart algorithm, orthogonal projection algorithm, etc. In this encoding framework, both of the above schemes can be used for parameterization of the reconstructed base mesh. The following briefly introduces the two algorithms.

[0114] Isochart algorithm: This algorithm uses spectral analysis to achieve stretch-driven three-dimensional mesh parameterization, which unfolds, slices and packs the three-dimensional mesh to a two-dimensional texture domain. A stretch threshold is set, and the algorithm is outlined as follows:

[0115] 1. Calculate the surface spectral analysis to provide an initial parameterization;

[0116] 2. Perform iterations of stretch optimization;

[0117] 3. If the stretch of this derived parameterization is less than the threshold, stop;

[0118] 4. Perform surface spectral clustering to divide the surface into charts;

[0119] 5. Optimize the chart boundaries using a graph cut algorithm;

[0120] 6. Iteratively split the charts until the stretch criterion is met.

[0121] Orthogonal projection algorithm:

[0122] The orthogonal projection algorithm is a projection-based mesh parameterization method that generates texture coordinates for the mesh through orthogonal projection. Its main process includes:

[0123] 1. Calculate the mesh attributes, including the adjacent faces of each face and the area and normal vector of each face;

[0124] 2. Determine the projection plane of each face according to the normal vector;

[0125] 3. Start clustering all faces into a connected region according to the projection plane, first select the starting face for clustering;

[0126] 4. From the start face, iterate to determine whether the adjoining faces of the face added to the connected region can be added to the connected region;

[0127] 5. After each connected region iteration is completed, a plurality of connected regions is obtained;

[0128] 6. According to the error metric, determine whether to merge the adjoining connected regions;

[0129] 7. Detect whether there is an overlapping region during projection, and remove the overlapping face and regenerate the connected region;

[0130] 8. Arrange all the regions after projection into a two-dimensional image.

[0131] (3) Subdivision and deformation.

[0132] The subdivision and deformation module is an optional module, which can improve the quality of the reconstructed mesh at the decoding end. When the basic mesh quality can meet the application requirements, this module can not be used.

[0133] The basic idea of the subdivision and deformation module is shown in Figure 5 , the same concept is applied to the input 3D mesh to generate displacement vector information. In Figure 5 , the input 2D curve (represented by a 2D polyline), referred to as the "original" curve, is first down-sampled to generate a basic curve / polyline, referred to as the "simplified" curve. Then a subdivision scheme is applied to the simplified polyline to generate a "subdivided" curve. Subsequently, the subdivided polyline is deformed to obtain a better approximation of the original curve. That is, a geometric displacement vector (indicated by the arrow in Figure 5 ) is calculated for each vertex of the subdivided mesh, so that the shape of the subdivided curve is as close as possible to the shape of the original curve. Adding the calculated geometric displacement vector to each vertex coordinate of the subdivided mesh is the output of this module, which is a deformed mesh containing displacement information. The same deformation process is also applied to the attribute information corresponding to the vertices, thereby obtaining the corresponding attribute displacement vector.

[0134] The subdivision and deformation module subdivides the input mesh, and the subdivision scheme can be arbitrarily selected. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four sub-triangles in each subdivision iteration, as shown in Figure 6 , S 0 is the mesh before subdivision, S 1 is the mesh after one iteration, and S 2 is the mesh after two iterations of subdivision. A new vertex is introduced in the middle of each edge, and the subdivision of geometric information and attribute information is performed independently, because the connection relationship between geometric information and attribute information is usually different.

[0135] The scheme calculates the position Pos(v) of the midpoint v of the newly introduced edge (v1, v2) in the manner shown in equation (1): 12 12

[0136]

[0137] wherein Pos(v1) and Pos(v2) are the geometric coordinates of the vertices v1 and v2, respectively.

[0138] For the subdivided mesh, finding the nearest neighbor of each point on the original input mesh (including the points on the original mesh faces) can be accelerated by using a data structure such as a k-dimensional tree (KD-tree). The displacement vector of the geometric coordinates of each vertex of the subdivided mesh can be obtained by calculating the distance between each vertex of the subdivided mesh and its nearest neighbor on the original input mesh.

[0139] The subdivision and deformation module can also convert the calculated vertex displacement coordinates from the Cartesian coordinate system to the local coordinate system. One feasible method is to convert each vertex displacement coordinate into the coordinate in the coordinate system composed of the normal vector of the corresponding vertex and the two vectors tangent to the normal vector. The specific conversion process is shown in equation (2):

[0140]

[0141] wherein, represents the vertex displacement before coordinate system conversion, represents the vertex displacement after coordinate system conversion, are mutually orthogonal unit vectors, and is the vertex normal vector, is the two vectors tangent to . For the calculation of the vertex normal vector, one feasible method is to equal the area-weighted sum of the normal vectors of the adjacent faces of the vertex.

[0142] (4) Compression of the base mesh.

[0143] The base mesh compression module compresses the base mesh information output by the subdivision and deformation module. The base mesh compression mainly has two different modes, specifically: intra-frame mode and inter-frame mode. For example, Figure 7 ​​As shown, in the intra mode, the input of the base mesh compression module is a 3D mesh, containing geometry coordinates, connectivity and attribute information associated with vertices. In the inter mode, the input of the base mesh compression module is the motion vector and its identifier, and possibly the intra sub-mesh. After encoding, the encoded mesh needs to be reconstructed to provide for the subsequent modules. The following briefly introduces the encoding of the two modes:

[0144] Intra mode: The mesh encoding in the intra mode can use the existing mesh encoder to encode the input base mesh, such as Draco, etc. The type of mesh encoder is encoded and transmitted to the decoding end through auxiliary information. The following introduces a usable Draco.

[0145] The main idea of Draco to compress static mesh is to drive the mesh compression by connectivity. It traverses all the faces of the mesh according to a specific way, labels each face according to a specific rule, and encodes all the labels of the faces obtained by traversal, that is, the connectivity of the mesh is encoded. Then all the vertex coordinate information is encoded according to the order of traversing the connectivity.

[0146] The main process of Draco to encode the mesh includes: first, for the input mesh, the connectivity is generated according to the geometry information, that is, the connectivity of the vertices in the three-dimensional space. After the connectivity of the face is constructed, an initial face is selected to start traversing all the faces of the current mesh, that is, to generate a symbol by traversal. Here, the Edgebreaker algorithm is used to generate a symbol by traversal. According to the state of the triangular face when the current corner is traversed, it is divided into five modes, for example, L mode, C mode, R mode, S mode and E mode as shown in the figure. Figure 8 Among them, in the C mode, the triangle (such as X shown in the figure) is completely located in the interior of the mesh and does not intersect with any boundary, and all the vertices are located in the interior of the mesh, so no additional information is needed to identify the boundary. In the L mode and the R mode, one vertex (such as v shown in the figure) of the triangle is located on the boundary. The L mode indicates that the boundary is on the left side of the triangle, and the R mode indicates that the boundary is on the right side of the triangle, which needs additional information to identify the direction and position of the boundary. The S mode is used to process the branch situation in the triangular mesh, by dividing the figure into different parts. In the E mode, all the edges of the triangle are located on the boundary of the mesh, which indicates that the triangle is a complete boundary triangle, and the position and direction of all the edges need to be recorded.

[0147] The five modes also define the direction of traversal to the next face after the current face (i.e. the arrow as shown in the figure). According to the above traversal method, a corresponding symbol is generated for each face defined by the current mesh geometry information, and then the symbols are entropy encoded to obtain the code stream of the connection relationship defined by the current mesh geometry information. At the same time, the traversal of each face also obtains the order of traversal of the corresponding vertices, which is passed to the geometry information encoder, rearranged in the order of traversal, quantized according to the predetermined quantization parameter, and then predicted. The prediction adopts a parallelogram prediction method.

[0148] Inter-frame mode: In the inter-frame mode, the motion vector and the identifier need to be encoded. The motion vector identifier is an array composed of numbers greater than or equal to 0, and the size of the array is the same as the number of vertices of the base mesh of the reference frame corresponding to the time domain motion vector identifier. The array can be directly encoded using existing entropy encoding algorithms, such as CABAC.

[0149] (5) Reconstruct the base mesh.

[0150] After the base mesh is encoded, the base mesh needs to be reconstructed. For the mesh encoded in the intra-frame mode, the encoded mesh is decoded to obtain the reconstructed base mesh.

[0151] For the base mesh information encoded in the inter-frame mode, the inter-frame sub-mesh needs to be reconstructed according to the motion vector and the identifier. If there is an intra-frame sub-mesh in the inter-frame mode, the reconstructed inter-frame sub-mesh also needs to be combined with the intra-frame sub-mesh to obtain the reconstructed base mesh. After obtaining the reconstructed base mesh, it needs to be subdivided for subsequent vertex displacement encoding and reconstruction of the deformed mesh.

[0152] (6) Encoding of vertex displacement.

[0153] To encode the vertex displacement, first, the order of the optimized deformed mesh vertices needs to be adjusted according to the subdivided reconstructed base mesh. Then, the coordinates of the optimized deformed mesh vertices are subtracted from the coordinates of the subdivided reconstructed base mesh vertices to obtain displacement vectors. Next, wavelet transform and quantization are performed on the displacement to obtain quantized wavelet transform coefficients. Finally, the wavelet transform coefficients of the current frame are arranged into a two-dimensional image to form a YUV video, and the displacement code stream is obtained by encoding with a video encoder. Next, each module will be introduced in detail:

[0154] 1. Adjust the order of the deformed mesh vertices and calculate the displacement.

[0155] The main role of this module is to adjust the optimized deformed mesh vertex order according to the subdivided base mesh vertex order, so that the optimized deformed mesh vertex order is the same as the subdivided base mesh vertex order, and then subtract the vertex coordinates of the two meshes one by one to calculate a displacement vector for each vertex. There is an optional step in this process, which is to convert the coordinate system of the vertex displacement from the Cartesian coordinate system to the local coordinate system.

[0156] 2. Wavelet transform.

[0157] A transform can be applied to the displacement vector to reduce the correlation between its data. An optional transform is the linear wavelet transform, whose prediction process is defined as shown in equation (3):

[0158]

[0159] where v is the newly inserted midpoint on edge (v1, v2), and Signal(v), Signal(v1) and Signal(v2) are the displacement vectors corresponding to vertices v, v1 and v2, respectively. The displacement vector of vertex v is predicted and then updated, and the update process is defined as shown in equation (4):

[0160]

[0161] where v * is the set of all adjacent vertices of vertex v. The transformed displacement vector is called a wavelet coefficient.

[0162] 3. Coefficient quantization.

[0163] The transformed displacement vector, i.e. the wavelet coefficient, can be quantized. There are many ways to quantize, one method is shown in equations (5) and (6):

[0164] disp[v].d[k] = floor(disp[v].d[k] * scale[k]) (5)

[0165]

[0166] where disp[v] represents the transformed value of the displacement vector of the vth vertex, d[k] represents the kth value of the displacement vector, floor represents rounding down. bitDepthPosition represents the bit depth of the current mesh vertex geometry position, and qp[k] represents the quantization parameter of the kth coefficient. As mentioned earlier, after converting the coordinate system of the displacement vector, the normal component has a more significant impact on the quality than the tangential component, so a larger quantization parameter can be used for the tangential component.

[0167] At the same time, according to the characteristics of wavelet transform, different quantization parameters can be used for the newly generated vertices and the original vertices. That is, for the subdivided vertices, the quantization parameter update is shown in formula (7):

[0168] scale[k]=scale[k]*lodScale[k] (7)

[0169] Among them, lodScale[k] represents the coefficient of the quantization parameter of the current subdivision level.

[0170] 4. Two-dimensional arrangement.

[0171] After quantization, the quantized wavelet transform coefficients can be arranged on a two-dimensional image as follows:

[0172] a. Traverse the wavelet coefficients in order from low frequency to high frequency.

[0173] b. For each coefficient, determine the index of the NxM pixel block (e.g., N=M=16) where it is located, where it should be stored in the raster scan order of the block.

[0174] c. Calculate the position of the corresponding NxM pixel block on the image according to the Morton order.

[0175] The arrangement is not limited here, and other arrangement schemes may also be used, such as zigzag order, raster order, etc. The encoder may explicitly specify the corresponding arrangement scheme in the bitstream.

[0176] 5. Video compression.

[0177] After arranging the wavelet coefficients on a two-dimensional image, they can be directly encoded using a video encoder to obtain a shifted code stream.

[0178] (7) Reconstruction of deformed mesh

[0179] The displacement vector encoding module needs to obtain the reconstructed displacement vector value during encoding. This is done by inverse quantization and inverse transformation (consistent with the decoding end). After obtaining the reconstructed geometric displacement vector, the subdivided sub-grid is reconstructed according to the corresponding geometric displacement vector to obtain the subdivided and deformed mesh, which is then passed to the texture map conversion module.

[0180] (8) Texture map conversion.

[0181] The texture map conversion module performs texture map conversion based on the input original mesh, the input original texture map and the reconstructed deformed mesh.

[0182] like Figure 9 As shown, the steps of texture map conversion are as follows:

[0183] 1. Calculate the texture coordinate of each pixel in the generated texture map, such as the texture coordinate P(u, v) corresponding to pixel A(i, j).

[0184] 2. Determine whether the texture coordinate is in a certain triangular face after parameterization of the subdivision deformation mesh.

[0185] 3. If the texture coordinate does not belong to any triangular face, mark the pixel as empty, and then fill it with a filling algorithm.

[0186] 4. If the texture coordinate belongs to a triangular face, then:

[0187] a. Mark the pixel as filled.

[0188] b. Calculate the barycentric coordinate of the pixel in the current triangular face according to the texture coordinate.

[0189] c. Map the two-dimensional texture coordinate to the three-dimensional geometric coordinate according to the barycentric coordinate and the corresponding triangular face, i.e., map it to the point on the subdivision deformation mesh corresponding to the texture coordinate, as shown by M(x, y, z) in the figure.

[0190] d. Find the nearest point to the three-dimensional coordinate on the input original mesh, as shown by M'(x, y, z) in the figure.

[0191] e. Calculate the barycentric coordinate of the three-dimensional coordinate according to the triangular face it belongs to, and map it to two dimensions to calculate its texture coordinate, i.e., P'(u', v').

[0192] f. Sample the input original texture map through the texture coordinate to obtain the value A'(i', j') at the corresponding pixel position.

[0193] Assign the value to the corresponding pixel A(i, j) in the generated texture map.

[0194] (9) Texture map compression.

[0195] After obtaining the converted texture map, the empty pixels can be filled using existing filling algorithms (such as the Push-Pull algorithm). Then, existing video encoders, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc., can be used to encode the filled pixels to obtain the code stream of the output texture map. In addition, color space conversion and chroma subsampling operations can be selectively applied to improve the rate-distortion performance of video encoding, such as converting from RGB 444 to YUV420. For cases where there are multiple texture maps, auxiliary information is needed to identify the corresponding texture coordinates of each texture map during encoding.

[0196] (10) Auxiliary information encoding.

[0197] In the encoding process, there are some alternative schemes in each module, such as the type of grid encoder, the type of video encoder, the grid subdivision scheme, the spatial displacement vector transform scheme and the transform parameters, etc. The proposed framework allows the use of different schemes. Therefore, the selected schemes need to be passed to the decoding end to guide the correct decoding. The auxiliary information includes the type of static grid encoder, the type of video encoder, the grid subdivision scheme, the number of grid iterations, the geometric displacement vector transform scheme and the transform parameters, the coefficient arrangement scheme, and the color conversion scheme, etc. For the grid part encoded using the inter-frame coding mode, the auxiliary information also includes the corresponding reference frame list, etc., which identifies the index list of the used reference frames. The auxiliary information can also include the subdivision identifier, which indicates whether the subdivision deformation operation is needed, i.e., whether it contains displacement information.

[0198] After the encoding of all modules is completed, the base grid part code stream, the texture coordinate part code stream, the displacement vector video code stream, the texture map video code stream, and the auxiliary information code stream are mixed to obtain the encoding end, and the final output of the encoding code stream.

[0199] The overall framework of the decoding end is shown in Figure 10 For the received code stream, the decoding end first demultiplexes each part of the code stream to obtain the base grid code stream, the displacement video code stream, the texture map video code stream, and the auxiliary information code stream. For the base grid code stream, the base grid is decoded using the grid decoder indicated by the auxiliary information. The displacement video code stream and the texture map video code stream can be decoded by the video decoder. For the displacement part, the displacement needs to be taken out from the image and subjected to the steps of inverse quantization and inverse transform, etc. through the displacement decoding module after video decoding, and then it is applied to the subdivided base grid to obtain the reconstructed deformed grid at the decoding end. The decoded texture map is the texture map corresponding to the reconstructed deformed grid. The subsequent application or rendering module processes the reconstructed deformed grid and the decoded texture map as input.

[0200] To better understand the embodiments of the present application, the decoding method and the encoding method provided by the embodiments of the present application are introduced below with reference to the accompanying drawings.

[0201] The decoding method provided by the embodiments of the present application can be executed by a decoding end, for example Figure 1 The decoder 300. The encoding method provided by the embodiments of the present application can be executed by an encoding end, for example Figure 1 The encoder 200. The decoding end and the encoding end can be realized by software, hardware or a combination thereof, and when they are realized by hardware, the decoding end can be referred to as a decoding end device or a decoding device, and the encoding end can be referred to as an encoding end device or a video encoding device.

[0202] Figure 11 is a schematic flowchart of a slice-based mesh decoding method 400 according to an embodiment of the present application.

[0203] As shown in Figure 11 the slice-based mesh decoding method 400 can include at least part of the following contents:

[0204] S410, a decoding end acquires at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh.

[0205] Exemplarily, the inter-frame code stream can also be referred to as a base mesh inter-frame code stream or a base mesh inter-frame code stream segment. The intra-frame code stream can also be referred to as a base mesh intra-frame code stream or a base mesh intra-frame code stream segment. The inter-frame code stream refers to a code stream decoded by using an inter-frame decoding mode, and the intra-frame code stream refers to a code stream decoded by using an intra-frame decoding mode.

[0206] S420, the decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice.

[0207] Exemplarily, the inter-frame slice can also be referred to as a base mesh inter-frame slice. The inter-frame slice can be a mesh region obtained by decoding the inter-frame code stream by using the inter-frame decoding mode.

[0208] Exemplarily, the inter-frame slice can be a mesh region formed by one or more face slices.

[0209] For the encoding end, the inter-frame slice can be a mesh region determined by the encoding end in a registration base mesh of a to-be-encoded original mesh.

[0210] For example, the inter-frame slice can be a mesh region determined by the encoding end in the registration base mesh based on matching of face slices between the registration base mesh and a registration subdivision deformed mesh of the to-be-encoded original mesh. The registration base mesh is a mesh obtained by deforming the to-be-encoded original mesh by the encoding end, has the same number of vertices as a reference base mesh of the to-be-encoded original mesh, and has the same connectivity relationship as the reference base mesh. The registration subdivision deformed mesh is a mesh obtained by deforming the to-be-encoded original mesh by the encoding end, has the same number of vertices as a reference subdivision deformed mesh of the to-be-encoded original mesh, and has the same connectivity relationship as the reference subdivision deformed mesh of the to-be-encoded original mesh.

[0211] S430, the decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice.

[0212] Exemplarily, the intra slice can also be referred to as a base mesh intra slice, and the intra slice can be a mesh region obtained after an intra code stream is decoded by an intra decoding mode.

[0213] Exemplarily, the intra slice can include a mesh region formed by one or more face slices.

[0214] For the encoding end, the intra slice can be a mesh region determined by the encoding end in a to-be-encoded original mesh.

[0215] For example, the intra slice can be a mesh region determined by the encoding end in a to-be-encoded original mesh based on face slices that do not match between a registration base mesh of the to-be-encoded original mesh and a registration subdivision deformed mesh of the to-be-encoded original mesh. The registration base mesh is a mesh obtained after the to-be-encoded original mesh is deformed by the encoding end, has the same number of vertices as a reference base mesh of the to-be-encoded original mesh, and has the same connectivity relationship as the reference base mesh. The registration subdivision deformed mesh is a mesh obtained after the to-be-encoded original mesh is deformed by the encoding end, has the same number of vertices as a reference subdivision deformed mesh of the to-be-encoded original mesh, and has the same connectivity relationship as the reference subdivision deformed mesh of the to-be-encoded original mesh.

[0216] S440, the decoding end merges the at least one inter slice and the at least one intra slice to obtain a reconstructed base mesh of the to-be-decoded base mesh.

[0217] Exemplarily, the decoding end can merge the at least one inter slice and the at least one intra slice in the order of the inter slice first and then the intra slice to obtain the reconstructed base mesh of the to-be-decoded base mesh. Alternatively, the decoding end can merge the at least one inter slice and the at least one intra slice in the order of the intra slice first and then the inter slice to obtain the reconstructed base mesh of the to-be-decoded base mesh.

[0218] In the slice-based mesh decoding method provided by the embodiment, the decoding end avoids directly decoding the entire base mesh by using the intra decoding mode or the inter decoding mode, and the decoding effect can be improved.

[0219] In some embodiments, the S410 includes:

[0220] In a case where the decoding mode used by the to-be-decoded base mesh is not the intra decoding mode, the decoding end acquires the at least one inter code stream and the at least one intra code stream.

[0221] For example, if the decoding mode used by the to-be-decoded base grid is a P inter mode or a Skip inter mode, it indicates that the decoding mode used by the to-be-decoded base grid is not an intra decoding mode, i.e., the decoding end obtains the at least one inter code stream and the at least one intra code stream. The inter decoding mode in which the motion vector is decoded is referred to as a P inter mode; the inter decoding mode in which the motion vector is not decoded is referred to as a Skip inter mode.

[0222] In some embodiments, the S410 comprises:

[0223] The decoding end obtains a base grid code stream of the to-be-decoded base grid, the base grid code stream comprising header information and a data unit; and the decoding end obtains the at least one inter code stream and the at least one intra code stream from the data unit based on the header information.

[0224] In some embodiments, the header information comprises at least one of:

[0225] an identifier for indicating the order of the at least one inter code stream and the at least one intra code stream;

[0226] an identifier for indicating the data length of the at least one inter code stream;

[0227] an identifier for indicating the number of the at least one inter code stream;

[0228] an identifier for indicating the length of each inter code stream in the at least one inter code stream;

[0229] an identifier for indicating the length of each intra code stream except the last intra code stream in the at least one inter code stream;

[0230] an identifier for indicating the data length of the at least one intra code stream;

[0231] an identifier for indicating the number of the at least one intra code stream;

[0232] an identifier for indicating the length of each intra code stream in the at least one intra code stream;

[0233] an identifier for indicating the length of each intra code stream except the last intra code stream in the at least one intra code stream.

[0234] For example, the identifier in the header information of the base grid code stream is used to distinguish the inter code stream from the intra code stream.

[0235] For example, assuming that the base grid code stream comprises one inter code stream and one intra code stream, if the inter code stream precedes the intra code stream, the identifier in the header information of the base grid code stream is set to 0; if the intra code stream precedes the inter code stream, the identifier in the header information of the base grid code stream is set to 1. Figure 12As shown, the identifier in the header information can indicate the number of bytes occupied by the inter-frame code stream. Based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order. Of course, in other alternative embodiments, if the intra-frame code stream precedes the inter-frame code stream, the identifier in the header information can indicate the number of bytes occupied by the intra-frame code stream; the inter-frame code stream and / or the intra-frame code stream can be byte-aligned, i.e., a multiple of an integer byte.

[0236] For another example, assuming that the base grid code stream includes one inter-frame code stream and multiple intra-frame code streams, if containing N intra-frame code streams, as shown in FIG. 2, the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each intra-frame code stream, or the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each of the first N-1 intra-frame code streams. Figure 13 As shown, the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each intra-frame code stream, or the identifier in the header information can indicate the number N of intra-frame code streams and the number of bytes occupied by each of the first N-1 intra-frame code streams. Based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order.

[0237] For another example, assuming that the base grid code stream includes multiple inter-frame code streams and multiple intra-frame code streams, the identifier in the header information can indicate the number of inter-frame code streams, the number of intra-frame code streams, the number of bytes occupied by each inter-frame code stream, and the number of bytes occupied by each intra-frame code stream; based on this, the decoding end can decode each inter-frame code stream and each intra-frame code stream in the data unit in the corresponding order.

[0238] It should be noted that, Figure 12 to Figure 14 The above examples are merely examples of the present application and should not be construed as limiting the present application. For example, in other alternative embodiments, in the data unit, the order of the inter-frame code stream and the intra-frame code stream can also be the order of the intra-frame code stream and the inter-frame code stream. Even, the identifier in the header information can indicate the order of the inter-frame code stream and the intra-frame code stream, or indicate the order of the intra-frame code stream and the inter-frame code stream. For another example, each identifier involved above can also be independently coded or carried in the auxiliary information code stream, which is not specifically limited by the present application.

[0239] For example, in order to ensure that the decoding end can distinguish between the inter-frame code stream and the intra-frame code stream, multiple identifiers need to be set in the syntax structure to distinguish between the intra-frame code stream and the inter-frame code stream. The present syntax structure can be designed based on the V-DMC syntax structure, the identifiers can be placed in the header information, and the inter-frame code stream and the intra-frame code stream can be placed in the data unit. The related syntax structure of the header information is shown in Table 1:

[0240] Table 1

[0241]

[0242] As shown in Table 1, the meanings of each syntax element are as follows:

[0243] smh_inter_segment_count: indicates the number of inter-frame streams.

[0244] smh_intra_segment_count: indicates the number of intra-frame streams.

[0245] smh_inter_segment_byte_counts[i]: indicates the number of bytes occupied by the i-th inter-frame stream in the data unit, used to separate the inter-frame stream and the intra-frame stream.

[0246] smh_intra_segment_byte_counts[i]: indicates the number of bytes occupied by the i-th intra-frame stream in the data unit, used to separate the inter-frame stream and the intra-frame stream.

[0247] Exemplarily, the P inter-decoding mode and the Skip inter-decoding mode can be cancelled in the data unit syntax, and the non-intra-decoding mode is used uniformly. When the decoding mode of the to-be-decoded base mesh is not the intra-decoding mode, the data unit contains the inter-frame stream and the intra-frame stream.

[0248] Table 2

[0249]

[0250] As shown in Table 2, the meanings of the respective syntax elements are as follows:

[0251] sdu_inter_sub_mesh_unit(subMeshID)[i]: indicates the data of the i-th inter-frame slice.

[0252] sdu_intra_sub_mesh_unit(subMeshID)[i]: indicates the data of the i-th intra-frame slice.

[0253] In some embodiments, the S420 comprises:

[0254] The decoding end decodes a first inter-frame stream in the at least one inter-frame stream to obtain first indication information and to-be-decoded information of a first inter-frame slice in the at least one inter-frame slice, the first indication information being used to indicate at least one reference surface slice used by the first inter-frame slice in a reference base mesh of the to-be-decoded base mesh; and the decoding end decodes the to-be-decoded information by using the inter-decoding mode based on the at least one reference surface slice to obtain the first inter-frame slice.

[0255] Exemplarily, when the decoding end decodes the to-be-decoded information based on the at least one reference patch and using the inter-frame decoding mode to obtain the first inter-frame patch, the to-be-decoded information can be decoded using a corresponding decoding mode of the first inter-frame patch to obtain the first inter-frame patch. The corresponding decoding mode of the first inter-frame patch can be a P inter-frame mode or a skip inter-frame mode. The inter-frame decoding mode in which a motion vector is decoded is referred to as a P inter-frame mode, and the inter-frame decoding mode in which a motion vector is not decoded is referred to as a skip inter-frame mode.

[0256] For example, if the corresponding decoding mode of the first inter-frame patch is a P inter-frame mode, the first indication information needs to be decoded first, and then the motion vector information needs to be decoded. If the corresponding decoding mode of the first inter-frame patch is a skip inter-frame mode, only the first indication information needs to be decoded. Whether it is a P inter-frame mode or a skip inter-frame mode, the first indication information needs to be decoded, which is used to indicate which part of the reference base mesh or which patch the first inter-frame patch refers to.

[0257] For the P inter-frame mode, the decoding end further needs to decode the motion vector information. One possible decoding manner is to follow the manner of V-DMC, and the specific steps are as follows:

[0258] 1. First, decode the skip mode identifier of a group of motion vectors of a group of vertices (for example, including the vertices of the first inter-frame patch);

[0259] 2. If the skip mode identifier indicates that the encoding manner of the group of motion vectors is a skip mode, the motion vectors do not need to be continuously decoded, and the position of the corresponding vertex of the reference base mesh can be directly assigned to the group of vertices;

[0260] 3. If the skip mode identifier indicates that the encoding manner of the group of motion vectors is not a skip mode, the prediction mode identifier of the group of motion vectors and the motion vector residual of all vertices of the group are continuously decoded; according to the prediction mode identifier, the prediction value of the motion vector of the group of vertices is calculated using a corresponding prediction manner, and then the prediction value is added to the decoded motion vector residual to recover the motion vector information of the group of vertices.

[0261] For the P inter-frame mode, after the decoding end decodes the motion vector information of each vertex, the motion vector information is superimposed on the position of the corresponding vertex of the reference base mesh of the first inter-frame patch (that is, the position of the corresponding vertex of the at least one reference patch) to decode and reconstruct the inter-frame patch.

[0262] In some embodiments, the first indication information is an array with a length equal to a number of patches of the reference base mesh, and a value at any position of the array is used to indicate whether a reference patch used by the first inter-frame patch includes a patch in the reference base mesh corresponding to the any position.

[0263] Exemplarily, the decoding manner of the first indication information is decoding an array, i.e., the decoding end can obtain patch indexes from the decoded array, and reconstruct the inter-frame patch using a method consistent with the encoding end, so as to recover the reconstructed inter-frame patch at the decoding end which is completely consistent with the encoding end.

[0264] In some embodiments, the reconstructed base mesh includes texture coordinates of the at least one inter-frame patch and texture coordinates of the at least one intra-frame patch, and the method 400 further includes:

[0265] The decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch, and scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch.

[0266] Exemplarily, scaling the texture coordinates of the at least one inter-frame patch and the texture coordinates of the at least one intra-frame patch at the decoding end can make the scaled texture coordinates of the at least one inter-frame patch and the scaled texture coordinates of the at least one intra-frame patch re-normalized to the same UV plane, i.e., the distribution ranges of the two parts of texture coordinates do not overlap, so that only one texture map is generated instead of two separate texture maps in subsequent texture map conversion, thereby improving the decoding performance of the texture map.

[0267] In some embodiments, the decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch, including:

[0268] The decoding end multiplies a component of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

[0269] In some embodiments, the decoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch, including:

[0270] The decoding end multiplies the component of the texture coordinate of the at least one intra slice in the first direction by a second coefficient, and adds the first coefficient to obtain the scaled texture coordinate of the at least one intra slice; wherein the sum of the first coefficient and the second coefficient is 1.

[0271] Figure 15 is an example of the adjustment process of the texture coordinate provided by the embodiments of the present application.

[0272] As shown in Figure 15 , before the adjustment of the texture coordinates of the intra slice and the inter slice, the texture coordinates of the inter slice and the intra slice are distributed in the normalized two-dimensional UV plane. A feasible adjustment scheme is to separately scale the texture coordinates of the inter slice and the intra slice in a certain direction of UV, and then merge them into a new normalized UV plane. Taking the V direction as an example, a constant coefficient a in the range of (0, 1) can be uniformly multiplied to the V component of the texture coordinate of the inter slice; then, a constant coefficient 1-a can be uniformly multiplied to the V component of the texture coordinate of the intra slice; finally, a can be uniformly added to the V component of the texture coordinate of the intra slice. After the processing, the texture coordinates of the intra slice and the texture coordinates of the inter slice that are re-merged and normalized can be obtained. Similarly, this strategy can also be used for texture coordinate adjustment in the U direction. After the adjustment, the texture coordinates of the intra slice and the texture coordinates of the inter slice are re-normalized to the same UV plane, so that only one texture map needs to be generated instead of two separate texture maps when performing texture map conversion subsequently.

[0273] Figure 16 is a schematic diagram of the mesh decoding process based on slices provided by the embodiments of the present application.

[0274] As shown in Figure 16 , the decoding end first demultiplexes the bit stream to obtain the code streams of each part and decodes them in turn. For the base mesh code stream, first, it is parsed into an inter code stream and an intra code stream, and then inter slice decoding and intra slice decoding are performed respectively to obtain the corresponding inter slice and intra slice. The inter slice and the intra slice are merged to obtain the reconstructed base mesh. After the decoding end decodes the displacement code stream to obtain the displacement, the reconstructed base mesh is combined to perform the deformed mesh reconstruction, and finally the reconstructed deformed mesh is obtained. For the texture map code stream, the decoded texture map is obtained after decoding. Through the decoding of the auxiliary information code stream, the auxiliary information is obtained to assist in guiding the successful parsing of the code stream by each module.

[0275] Exemplarily, the mesh decoding process can include the following processes:

[0276] (1) Auxiliary information decoding.

[0277] The decoding end determines the decoding scheme according to the auxiliary information, which mainly includes an intra-frame coding identifier indicating whether the current frame needs to be constructed according to the reference frame grid; a reference frame list indicating the index of the reference frame needed by the current frame, which is applied to the subsequent basic grid reconstruction step; a subdivision identifier indicating whether the subsequent subdivision deformation operation needs to be performed on the reconstructed basic grid; a static grid encoder type guiding the decoding end to use the corresponding static grid decoder; a video encoder type guiding the decoding end to use the corresponding video decoder; a subdivision scheme, that is, the scheme of subdividing the basic grid in the reconstructed deformation grid, which should be consistent between the encoding and decoding ends; and optional displacement transformation schemes, coefficient arrangement schemes and the like.

[0278] It is worth noting that the auxiliary information described herein includes auxiliary information transmitted independently and header information that can be included in other code stream parts.

[0279] (2) Basic grid decoding

[0280] The structure of the basic grid code stream can refer to the related content described above. The decoding end divides the basic grid code stream into inter-frame code streams and intra-frame code streams according to the structure of the basic grid code stream. If multiple inter-frame code streams are included, multiple inter-frame slices are decoded. If multiple intra-frame code streams are included, multiple intra-frame slices are decoded. An intra-frame code stream or an inter-frame code stream can also be referred to as a basic grid slice code stream, which can be independently decoded according to its type (intra-frame or inter-frame).

[0281] Inter-frame slice decoding

[0282] The inter-frame slice decoding includes decoding at least one of the connection relationship information, the vertex geometry information and the texture coordinate information in the inter-frame slice. The decoding mode of the connection relationship information, the vertex geometry information and the texture coordinate information corresponds to the encoding mode.

[0283] One way of decoding the connection relationship information is to directly use the connection relationship of the reference basic grid, whether to directly use the connection relationship of the reference basic grid can be determined according to the syntax element in the header information. One way of decoding the texture coordinate information is to directly use the texture coordinate information of the reference basic grid, the texture coordinates of the vertices in the inter-frame slice directly use the texture coordinates of the corresponding points in the reference basic grid. Moreover, consistent texture coordinate adjustment strategies can be adopted at the encoding and decoding ends, thereby ensuring consistency of the inter-frame slices at the encoding and decoding ends. The decoding of the vertex geometry information first needs to decode whether the inter-frame decoding mode of the inter-frame slice is a P inter-frame mode or a Skip inter-frame mode.

[0284] Intra-frame basic grid decoding

[0285] Intra-slice decoding includes decoding at least one of connection relationship information, vertex geometry information, and texture coordinate information in the intra-slice. The decoding method of the connection relationship information, vertex geometry information, and texture coordinate information corresponds to the encoding method.

[0286] If a static mesh compression algorithm such as Draco is used to directly encode the intra-frame slice connectivity information, vertex geometry information, and texture coordinate information, then the corresponding decoding algorithm can be used to decode the intra-frame slice connectivity, vertex geometry, and texture coordinates. Furthermore, a consistent texture coordinate adjustment strategy can be adopted on the codec side, thereby ensuring consistency between inter-frame slices on the codec side.

[0287] It should be understood that the code stream structure of the basic grid code stream in this application can refer to the relevant content described above, for example, Figure 12 to Figure 14 To avoid repetition, the relevant content will not be repeated here.

[0288] (3) Grid merging.

[0289] The grid merging processing module merges the decoded inter-frame slices and the decoded intra-frame slices to obtain a reconstructed basic grid. Specifically, the reconstructed intra-frame slices (i.e., the at least one intra-frame slice obtained by decoding) and the reconstructed inter-frame slices (i.e., the at least one inter-frame slice obtained by decoding) are merged into a complete reconstructed basic grid. Specifically, the geometric coordinates, triangular facets and texture coordinates of the reconstructed intra-frame slices and the reconstructed inter-frame slices are written into the reconstructed basic grid in the order of first inter-frame and then intra-frame. Alternatively, the geometric coordinates, triangular facets and texture coordinates of the reconstructed intra-frame slices and the reconstructed inter-frame slices are written into the reconstructed basic grid in the order of first intra-frame and then inter-frame. It should be understood that the grid merging module is the same as the grid merging module at the encoding end.

[0290] (4) Displacement decoding.

[0291] Shift decoding is the inverse process of shift encoding.

[0292] like Figure 17 As shown, the decoder first needs to decode the displacement, and the displacement decoding must be consistent with the encoding segment. If the encoder uses video encoding to encode the displacement, the decoder must use video decoding to decode it. If the encoder directly entropy encodes the displacement, the decoder must perform entropy decoding on the displacement bitstream. After the displacement decoding is complete, the quantized wavelet transform coefficients are obtained. These quantized wavelet transform coefficients are then dequantized and inverse wavelet transformed.

[0293] (5) Reconstruct the deformed mesh.

[0294] The process of reconstructing the deformed mesh at the decoding end is basically consistent with that at the encoding end. First, the inter-frame patches and intra-frame patches obtained by decoding are combined to restore the complete reconstructed base mesh. Then the base mesh is subdivided. After the decoding data of the displacement code stream is subjected to inverse quantization and inverse wavelet transform, the displacement can be obtained and superimposed on the subdivided base mesh, and finally the reconstructed deformed mesh can be obtained.

[0295] (6) Texture decoding.

[0296] The texture decoder is responsible for decoding the texture code stream, which is decoded by the video decoder indicated in the auxiliary information. After optional color space conversion, the decoded output texture is obtained in the image format input to the encoding end. Figure 1

[0297] After the processing of each module at the decoding end is completed, the deformed mesh reconstructed at the decoding end and the corresponding texture can be obtained, and subsequent applications can process the reconstructed deformed mesh and texture as input. The specific processing of the application is not limited.

[0298] In this embodiment, the decoding end obtains the inter-frame code stream and intra-frame code stream of the to-be-decoded base mesh. The inter-frame code stream is decoded using the inter-frame decoding mode to obtain the inter-frame patch, and the intra-frame code stream is decoded using the intra-frame decoding mode to obtain the intra-frame patch. The inter-frame patch can be a mesh region determined by the encoding end in the registration base mesh based on the matching of the registration subdivided deformed mesh between the registration base mesh and the to-be-encoded original mesh. The connection relationship, vertex geometric coordinates, texture coordinates, etc. are decoded based on the reference base mesh using the temporal prediction technology, greatly improving the compression efficiency. The intra-frame patch can be a mesh region determined by the encoding end in the to-be-encoded original mesh based on the registration base mesh of the to-be-encoded original mesh and the registration subdivided deformed mesh of the to-be-encoded original mesh. The connection relationship, vertex geometric coordinates, texture coordinates, etc. are directly decoded. One mesh can contain multiple intra-frame patches and multiple inter-frame patches. The multiple intra-frame patches and multiple inter-frame patches are connected together and jointly processed for displacement generation, displacement decoding, etc. For subsequent temporal meshes, they can be regarded as a whole reference mesh.

[0299] The decoding method provided by the present application is not only suitable for three-dimensional base mesh decoding, but also suitable for inter-frame three-dimensional base mesh decoding, and also suitable for three-dimensional base mesh sub-mesh or three-dimensional inter-frame base mesh sub-mesh decoding process. When the to-be-decoded mesh contains multiple sub-meshes, each sub-mesh can be decoded according to the proposed scheme, that is, the processing unit of the decoding scheme can be a sub-mesh.

[0300] Figure 18 is a schematic flowchart of the patch-based mesh encoding method 500 according to an embodiment of the present application.​

[0301] As shown in Figure 18 the slice-based mesh coding method 500 can include at least part of the following contents:

[0302] S510, the encoding end determines at least one inter-frame slice of a to-be-encoded original mesh and at least one intra-frame slice of the to-be-encoded original mesh.

[0303] S520, the encoding end encodes the at least one inter-frame slice in an inter-frame coding mode to obtain at least one inter-frame code stream.

[0304] S530, the encoding end encodes the at least one intra-frame slice in an intra-frame coding mode to obtain at least one intra-frame code stream.

[0305] S540, the encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a basic mesh code stream.

[0306] In some embodiments, the S510 includes:

[0307] In the case where the coding mode used by the to-be-encoded original mesh is not an intra-frame coding mode, the encoding end determines at least one inter-frame slice of a to-be-encoded original mesh and at least one intra-frame slice of the to-be-encoded original mesh.

[0308] In some embodiments, the S540 includes:

[0309] The encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit; and the encoding end merges the header information of the basic mesh code stream and the data unit to obtain the basic mesh code stream.

[0310] In some embodiments, the header information includes at least one of the following:

[0311] an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0312] an identifier for indicating the data length of the at least one inter-frame code stream;

[0313] an identifier for indicating the number of the at least one inter-frame code stream;

[0314] an identifier for indicating the length of each inter-frame code stream in the at least one inter-frame code stream;

[0315] an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one inter-frame code stream;

[0316] an identifier for indicating a data length of the at least one intra code stream;

[0317] an identifier for indicating a number of the at least one intra code stream;

[0318] an identifier for indicating a length of each of the at least one intra code stream;

[0319] an identifier for indicating a length of each of the at least one intra code stream except the last one.

[0320] In some embodiments, the S520 comprises:

[0321] The encoding end encodes a first inter slice in the at least one inter slice based on at least one reference patch in a reference base grid of the original grid to be encoded, to obtain encoding information of the first inter slice; the encoding end encodes first indication information and the encoding information of the first inter slice, to obtain a first inter code stream in the at least one first inter code stream, the first indication information being used for indicating the at least one reference patch.

[0322] Illustratively, the encoding end can encode the first inter slice based on the at least one reference patch, to obtain the encoding information of the first inter slice, using a corresponding encoding mode of the first inter slice. The corresponding encoding mode of the first inter slice can be a P inter mode or a Skip inter mode. The inter encoding mode in which the motion vector is encoded is called P inter mode; the inter encoding mode in which the motion vector is not encoded is called Skip inter mode.

[0323] For example, if the corresponding encoding mode of the first inter slice is P inter mode, the first indication information needs to be encoded first, and then the motion vector information; if the corresponding encoding mode of the first inter slice is Skip inter mode, only the first indication information needs to be encoded. Whether it is P inter mode or Skip inter mode, the first indication information needs to be encoded, which is used for indicating which part of the area or which patch in the reference base grid the first inter slice refers to.

[0324] For P inter mode, the encoding end further needs to encode the motion vector information. One possible encoding mode is to follow the V-DMC mode, and the specific steps are as follows:

[0325] 1. Divide each 16 vertices of the inter slice into a group according to the vertex index order;

[0326] 2. Determine whether to encode the motion vector of this group of vertices according to the rate-distortion criterion for Skip mode judgment at the motion vector group level;

[0327] 3、If the motion vector of the set of vertices is not encoded, the position of the set of vertices needs to be adjusted to the position of the corresponding vertices in the reference frame;

[0328] 4、If the motion vector of the set of vertices is encoded, the encoding bit overhead of the motion vector residual corresponding to the three motion vector prediction modes needs to be compared and selected from the prediction mode with smaller bit overhead; the prediction mode identifier of the set of vertices is encoded first, and then the motion vector residual of the set of vertices is encoded.

[0329] In some embodiments, the first indication information is an array with a length equal to the number of patches of the reference base mesh, and the value at any position of the array is used to indicate whether the reference patch used by the first inter-frame patch includes the patch corresponding to the any position in the reference base mesh.

[0330] Exemplarily, a possible encoding mode of the first indication information is as follows:

[0331] 1、Set an array with a length equal to the number of patches in the reference base mesh, and the default value in the array is 0;

[0332] 2、Iterate through the reference patches of the inter-frame patch, and change the value of the corresponding position in the array from 0 to 1 according to the index of the reference patch in the reference base mesh;

[0333] 3、Encode the 01 characters in the array in order, and when the value of the corresponding position in the array is 1, it indicates that the triangular patch is a reference patch.

[0334] In some embodiments, the reconstructed base mesh of the original mesh to be encoded includes the texture coordinates of the at least one inter-frame patch and the texture coordinates of the at least one intra-frame patch, and the method 500 further includes

[0335] The encoding end scales the texture coordinates of the at least one inter-frame patch along a first direction of the texture coordinate axis to obtain the scaled texture coordinates of the at least one inter-frame patch; and scales the texture coordinates of the at least one intra-frame patch along the first direction to obtain the scaled texture coordinates of the at least one intra-frame patch.

[0336] In some embodiments, the encoding end scales the texture coordinates of the at least one inter-frame patch along a first direction of the texture coordinate axis to obtain the scaled texture coordinates of the at least one inter-frame patch, including:

[0337] The encoding end multiplies the component of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

[0338] In some embodiments, the encoding end scales the texture coordinates of the at least one intra slice in the first direction to obtain scaled texture coordinates of the at least one intra slice, including:

[0339] The encoding end multiplies the component of the texture coordinates of the at least one intra slice in the first direction by a second coefficient and adds the first coefficient to obtain the scaled texture coordinates of the at least one intra slice; wherein the sum of the first coefficient and the second coefficient is 1.

[0340] In some embodiments, the S510 includes:

[0341] The encoding end determines a registration base mesh of the to-be-encoded original mesh and a registration subdivision deformation mesh of the to-be-encoded original mesh based on a reference base mesh of the to-be-encoded original mesh and a reference subdivision deformation mesh of the to-be-encoded original mesh; wherein the registration base mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference base mesh, and having the same connectivity relationship as the reference base mesh; the registration subdivision deformation mesh is a mesh obtained by deforming the to-be-encoded original mesh, having the same number of vertices as the reference subdivision deformation mesh, and having the same connectivity relationship as the reference subdivision deformation mesh; the encoding end marks a face patch in the registration base mesh as a matched or unmatched face patch based on a matching detection result between the registration base mesh and the registration subdivision deformation mesh to obtain a first marking result; and the encoding end determines the at least one inter slice in the registration base mesh based on the first marking result.

[0342] In some embodiments, the S510 includes:

[0343] The encoding end marks a face patch in the to-be-encoded original mesh as a matched or unmatched face patch based on the marking result to obtain a second marking result; the encoding end determines at least one initial intra slice in the to-be-encoded original mesh based on the second marking result; and the encoding end performs boundary adjustment on the at least one initial intra slice based on the at least one inter slice to obtain the at least one intra slice.

[0344] For example, the encoding end can detect and mark the boundaries of the unmatched area and the matched area according to the first marking result of the registration base mesh, then delete the unmatched face patch to obtain the at least one inter slice; detect and mark the boundary points of the unmatched area and the matched area according to the second marking result of the to-be-encoded original mesh, then take out the unmatched face patch to obtain the initial intra slice.

[0345] Figure 19Fig. 1 is a schematic diagram of a generation process of at least one inter-frame patch and at least one intra-frame patch provided by an embodiment of the present application.

[0346] As shown in Figure 19 , the encoding end first inputs the original mesh to be encoded and the reference base mesh into a registration base mesh generation module, which uses an inter-frame registration algorithm to deform the reference base mesh according to the original mesh to be encoded, so as to obtain a registration base mesh whose shape is as close as possible to the original mesh to be encoded, and the number of vertices and the connectivity relationship remain consistent with the reference base mesh. The registration subdivision deformed mesh is a mesh generated in the registration process, which has the same number of vertices and connectivity relationship as the reference frame subdivision deformed mesh, and its shape is also close to the original mesh to be encoded. Then the encoding end performs mismatch region detection to detect which part of the registration base mesh is well registered with the registration subdivision deformed mesh and which part of the registration base mesh is poorly registered with the registration subdivision deformed mesh. After the mismatch region detection, the inter-frame patch and the initial intra-frame patch are obtained. The inter-frame patch is the well-registered part taken from the registration base mesh, and the inter-frame patch can carry the matching information of the reference base mesh, such as the vertex in the reference base mesh corresponding to each vertex, so as to be used in subsequent encoding of the inter-frame patch. The initial intra-frame patch is the poorly-registered region in the original mesh to be encoded, which is combined with the inter-frame patch to further process the intra-frame patch. The base mesh generation module can obtain multiple inter-frame patches and / or multiple intra-frame patches. Some vertices in the inter-frame patch and some vertices in the intra-frame patch have the same position, and the inter-frame patch and the intra-frame patch are connected to form a whole mesh.

[0347] Exemplarily, the encoding end generates the inter-frame patch and the intra-frame patch according to the original mesh to be encoded and the reference base mesh. The reference base mesh is the base mesh of the reconstructed mesh, such as the base mesh of the reconstructed mesh of the previous frame adjacent in the time domain. The reference base mesh can include multiple base meshes of reconstructed meshes, such as when multiple reference frames are used, the reference base mesh can include multiple base meshes of reconstructed meshes.

[0348] Figure 20 Fig. 2 is an example of the registration base mesh, the original mesh to be encoded, the inter-frame patch, the initial intra-frame patch, the intra-frame patch, and the base mesh provided by an embodiment of the present application.

[0349] As shown in Figure 20 , in (a) of Figure 20 , the registration base mesh, wherein the area enclosed by the solid line in the registration base mesh is a highly similar area to the registration subdivision deformed mesh, and this area information can be used as the inter-frame patch, for example, as shown in the inter-frame patch in (c) of Figure 20 ; the area enclosed by the dashed line in the registration base mesh has low similarity to the registration subdivision deformed mesh. Figure 20(b) in FIG. 1 is a to-be-encoded original mesh, wherein the area enclosed by the solid line is an area with a high similarity between the registration base mesh and the registration subdivided deformed mesh, and the area enclosed by the dashed line is an area with a low similarity between the registration base mesh and the registration subdivided deformed mesh, and the area with a low similarity can be used as an initial intra-frame patch, for example, as shown in (d) in FIG. 1. The initial intra-frame patch is processed by a mesh to obtain an intra-frame patch, for example, as shown in (e) in FIG. 1. The intra-frame patch and the inter-frame patch are combined to obtain a base mesh, for example, as shown in (f) in FIG. 1. Figure 20 Figure 20 Figure 20

[0350] It should be noted that if the boundary of the inter-frame patch and the boundary of the intra-frame patch do not overlap, in order to connect the inter-frame patch and the intra-frame patch into a whole mesh, the encoding end needs to adjust (morphological stretching) the vertices at the boundary of the initial intra-frame patch according to the boundary of the inter-frame patch, then perform mesh simplification and mesh parameterization processing on the initial intra-frame patch, and keep the boundary vertices of the initial intra-frame patch unchanged in the mesh simplification process, and then generate texture coordinates through mesh parameterization, to finally obtain an intra-frame patch. Finally, the texture coordinates of the inter-frame patch and the intra-frame patch are uniformly adjusted to re-uniformly normalize the texture coordinates of the two parts. Through the above processes, the base mesh generation module of the encoding end can obtain the to-be-encoded inter-frame patch and intra-frame patch.

[0351] In some embodiments, the encoding end determines the registration base mesh of the to-be-encoded original mesh and the registration subdivided deformed mesh of the to-be-encoded original mesh based on a reference base mesh of the to-be-encoded original mesh and a reference subdivided deformed mesh of the to-be-encoded original mesh, and the determination includes:

[0352] The encoding end deforms the to-be-encoded original mesh to obtain an intermediate mesh, taking the reference subdivided deformed mesh as a target mesh; the encoding end deforms the reference subdivided deformed mesh to obtain the registration subdivided deformed mesh, taking the intermediate mesh as a target mesh; and the encoding end deforms the reference base mesh to obtain the registration base mesh, taking the registration subdivided deformed mesh as a target mesh.

[0353] ​​​Exemplarily, the encoding end can deform the reference base mesh by using an inter-frame registration algorithm, so that the registration base mesh is a mesh obtained by deforming the to-be-encoded original mesh, the number of vertices of the mesh being same as that of the reference base mesh, and the connectivity relationship of the mesh being same as that of the reference base mesh; and the registration subdivision deformed mesh is a mesh obtained by deforming the to-be-encoded original mesh, the number of vertices of the mesh being same as that of the reference subdivision deformed mesh, and the connectivity relationship of the mesh being same as that of the reference subdivision deformed mesh.

[0354] Figure 21 is an example of a generation process of a registration base mesh and a configuration subdivision deformed mesh provided by an embodiment of the present application.

[0355] As shown in Figure 21 , first, the reference subdivision deformed mesh is taken as a target mesh, a nearest neighbor search algorithm is used to deform the to-be-encoded original mesh to the reference subdivision deformed mesh, and an intermediate mesh is output. Then, the intermediate mesh is taken as a target mesh, a nearest neighbor search algorithm is used to deform the reference subdivision deformed mesh to the intermediate mesh, and a mesh obtained by the deformation is the registration subdivision deformed mesh. Then, the generated registration subdivision deformed mesh is taken as a target mesh, and the reference base mesh is fitted and deformed to the target mesh, and a base mesh obtained by the fitting and deformation is the reference subdivision deformed mesh.

[0356] Figure 22 is an example of a generation process of at least one inter-frame patch and at least one initial intra-frame patch provided by an embodiment of the present application.

[0357] As shown in Figure 22 , the encoding end can obtain the registration degree of each region of the registration base mesh and the registration subdivision deformed mesh by using a detection algorithm, divide the registration accurate region in the registration base mesh, and then send the region to a subsequent inter-frame patch encoding module for inter-frame encoding. For the region in the registration base mesh with poor registration, the encoding end can obtain the corresponding part from the to-be-encoded original mesh, process the intra-frame patch, and send the intra-frame patch to an intra-frame patch encoding module for intra-frame encoding, so as to increase the flexibility of encoding. In addition, in order to facilitate subsequent boundary stretching of the initial intra-frame patch, before the division, the encoding end can mark the boundary of the inter-frame patch and the boundary point of the initial intra-frame patch according to the detection result of each patch.

[0358] Of course, in other alternative embodiments, the reference base mesh of the original mesh to be encoded can also be used to replace the registration subdivision deformed mesh, and the present application does not make specific limitations thereto. Specifically, the encoding end obtains the registration degree of each region of the registration base mesh and the registration subdivision deformed mesh through the detection algorithm, divides the registration accurate region in the registration base mesh, and then sends this part to the subsequent inter-frame patch encoding module for inter-frame encoding. For the region with poor registration of the registration base mesh, the encoding end can obtain the corresponding part from the original mesh to be encoded, process the intra-frame patch, and then send it to the intra-frame patch encoding module for intra-frame encoding, thereby increasing the flexibility of encoding.

[0359] In some embodiments, the first marking result includes the marking result of the first patch in the registration base mesh; wherein, the encoding end marks the patch in the registration base mesh as a matched or unmatched patch based on the matching detection result between the registration base mesh and the registration subdivision deformed mesh, to obtain the first marking result, including:

[0360] The encoding end obtains the first bounding box in which the first patch is located; the encoding end obtains the first region in the original mesh to be encoded corresponding to the first bounding box and the second region in the matched subdivision deformed mesh corresponding to the first bounding box based on the first bounding box; the encoding end calculates the first weighted average normal vector of the patch in the first region and the second weighted average normal vector of the patch in the second region; in the case that the included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, the encoding end determines that the marking result of the first patch is that the first patch is marked as an unmatched patch, to obtain the marking result of the first patch, or in the case that the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, the encoding end marks the first patch as a matched patch, to obtain the marking result of the first patch.

[0361] Exemplarily, the encoding end can obtain the marking result (i.e. the first marking result) of each patch in the registration base mesh in the following method:

[0362] 1. Traverse each patch in the registration base mesh to obtain the bounding box of each patch;

[0363] 2. According to the bounding box of each patch in the registration base mesh, the corresponding regions of the original mesh to be encoded and the registration subdivision deformed mesh located in the bounding box are obtained respectively, and the weighted average normal vectors of the patches in these two regions are calculated respectively;

[0364] 3. If the included angle between the two normal vectors is greater than the set threshold, the corresponding registration base mesh patch is determined and marked as an unmatched patch;

[0365] 4. Return to step 1 until all patches in the registration base mesh are traversed.

[0366] In some embodiments, the encoding end determines the at least one interframe patch in the registration base mesh based on the first marking result, including:

[0367] The encoding end obtains at least one third region in which the patches marked as mismatched are located based on the first marking result; the encoding end re-marks the patches in a first region in the at least one third region in which the total area of the patches is less than or equal to a preset first area threshold as matched patches to obtain a third marking result; the encoding end obtains at least one fourth region in which the patches marked as matched are located based on the third marking result; the encoding end re-marks the patches in a second region in the at least one fourth region in which the total area of the patches is less than or equal to the preset first area threshold as mismatched patches to obtain a fourth marking result; and the encoding end determines a region in which the patches marked as matched are located based on the fourth marking result as the at least one interframe patch.

[0368] Exemplarily, the encoding end can obtain several regions in the set of mismatched patches based on the first marking result, and perform hole filling processing on the patches in a region in the registration base mesh in which the total area of the patches is less than or equal to a preset first area threshold to obtain a third marking result; then, the encoding end can obtain several regions in the set of matched patches based on the third marking result, and perform small block removal processing on the patches in a region in the registration base mesh in which the total area of the patches is less than or equal to the preset first area threshold to obtain the fourth marking result.

[0369] In some embodiments, the second marking result includes a marking result of a second patch in the original mesh to be encoded; and wherein the encoding end marks the patches in the original mesh to be encoded as matched or mismatched patches based on the marking result to obtain a second marking result, including:

[0370] The encoding end obtains the barycenter of the second patch; the encoding end searches for a nearest neighbor point of the barycenter of the second patch in the registration base mesh; and the encoding end marks the second patch as a matched or mismatched patch based on the marking of the patch in which the nearest neighbor point is located to obtain a marking result of the second patch.

[0371] Exemplarily, the encoding end can obtain the marking result of each patch in the original mesh to be encoded according to the following method:

[0372] 1. Traverse the patches in the original mesh to be encoded to obtain the barycenter of each patch;

[0373] 2. For the barycenter of the original mesh patch to be encoded, a nearest neighbor search is performed in the registered base mesh, and according to the mismatch label of the patch where the nearest neighbor point is located, the current patch is labeled with the corresponding mismatch label;

[0374] 3. Return to step 1 until all patches in the original mesh to be encoded are traversed.

[0375] In some embodiments, the encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result, including:

[0376] The encoding end obtains at least one fifth region where the patches labeled as mismatched are located based on the second labeling result; the encoding end re-labels the patches in a third region in the at least one fifth region as matched patches if the total area of the patches in the third region is less than or equal to a preset second area threshold, to obtain a fifth labeling result; the encoding end obtains at least one sixth region where the patches labeled as matched are located based on the fifth labeling result; the encoding end re-labels the patches in a fourth region in the at least one sixth region as mismatched patches if the total area of the patches in the fourth region is less than or equal to a preset second area threshold, to obtain a sixth labeling result; and the encoding end determines a region where the patches labeled as mismatched are located based on the sixth labeling result as the at least one initial intra-frame patch.

[0377] For example, the encoding end can obtain several regions in the set of mismatched patches based on the second labeling result, and perform hole filling processing on the patches in a region in the original mesh to be encoded if the total area of the patches in the region is less than or equal to a preset second area threshold, to obtain a fifth labeling result; then, the encoding end can obtain several regions in the set of matched patches based on the fifth labeling result, and perform small block removal processing on the patches in a region in the original mesh to be encoded if the total area of the patches in the region is less than or equal to a preset second area threshold, to obtain the sixth labeling result.

[0378] In some embodiments, the encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch, to obtain the at least one intra-frame patch, including:

[0379] The encoding end obtains at least one edge of the registration base mesh in which there are adjacent patches marked as matching and non-matching; the encoding end obtains at least one vertex of the original mesh to be encoded in which there are adjacent patches marked as matching and non-matching; for a first vertex corresponding to the at least one edge, the encoding end searches for a nearest neighbor point of the first vertex among the at least one vertex; the encoding end deforms the nearest neighbor point of the first vertex to the first vertex; for a second vertex in the at least one vertex except the nearest neighbor point of the first vertex, the encoding end searches for a nearest neighbor point of the second vertex on the at least one boundary; the encoding end obtains an edge in which the nearest neighbor point of the second vertex is located based on the nearest neighbor point of the second vertex; the encoding end determines a position closest to the second vertex on the edge in which the nearest neighbor point of the second vertex is located; and the encoding end deforms the second vertex to the position closest to the second vertex, to obtain the at least one intra-frame patch.

[0380] Exemplarily, the purpose of boundary adjustment (for example, boundary stretching) on the initial intra-frame patch is to make the inter-frame part mesh and the intra-frame part mesh as close as possible at the boundary position, without gaps and overlapping areas. Before stretching, the boundaries of the inter-frame patch and the initial intra-frame patch need to be marked respectively. A possible boundary marking method is to mark the boundaries of the inter-frame patch and the boundary points of the initial intra-frame patch according to the detection results of each patch after non-matching area detection and before mesh division.

[0381] Figure 23 is an example of a boundary and a boundary point provided by an embodiment of the present application.

[0382] As shown in (a) in Figure 23 , the encoding end obtains a current edge in a manner of traversing each edge of the registration base mesh; if there are adjacent patches belonging to a matching area and a non-matching area at the same time, the current edge is considered as a division boundary of the matching area and the non-matching area, that is, a division boundary of the inter-frame patch; the encoding end obtains a next edge and takes it as the current edge to determine whether it is a division boundary, until all edges of the registration base mesh are traversed, to obtain the at least one edge. As shown in (b) in Figure 23 , the encoding end obtains all adjacent patches of a current vertex in a manner of traversing each vertex of the original mesh to be encoded; if there are adjacent patches of a matching area and a non-matching area at the same time, the current vertex is considered as a boundary point of the initial intra-frame patch; the encoding end obtains a next vertex and takes it as the current vertex to determine whether it is a boundary point, until all vertices of the initial intra-frame patch are traversed, to obtain the at least one vertex.

[0383] Figure 24 is a schematic diagram of boundary stretching provided by an embodiment of the present application.

[0384] AsFigure 24 As shown, the encoding end can perform boundary stretching according to the result of the boundary marking. A possible initial intra-slice boundary stretching scheme can be: the encoding end traverses the boundary points of the inter-slice, performs a nearest neighbor search on each point on the initial intra-slice boundary point, and morphs the found nearest neighbor point to the position of the current inter-slice boundary point. Then, the encoding end takes the initial intra-slice boundary points that are not searched as the points to be morphed, searches for the nearest neighbor points on the inter-slice boundary for each point to be morphed. Then, the encoding end obtains the boundary where the searched nearest neighbor point is located according to the searched nearest neighbor point, calculates and compares the shortest distances of the points to be morphed to the boundary, and takes the position corresponding to the shortest distance as the final morphed position. Finally, the encoding end removes the duplicate points and edges from the initial intra-slice after boundary stretching.

[0385] It should be noted that the initial intra-slice can be processed to obtain an intra-slice, which can be generated by mesh simplification and mesh parameterization to be used for subsequent base mesh coding. For example, as shown in Figure 25 First, mesh simplification is performed on the initial intra-slice to output an intra-slice without texture coordinates. Then, mesh parameterization is performed on the intra-slice to generate the texture coordinates of the intra-slice.

[0386] In some embodiments, the method 500 further includes:

[0387] The encoding end obtains a reconstructed base mesh of the original mesh to be coded; the encoding end performs subdivision morphing on the reconstructed base mesh based on the original mesh to be coded, to obtain a subdivision mesh of the original mesh to be coded and a subdivision morphed mesh of the original mesh to be coded; the encoding end determines displacement information of the original mesh to be coded based on the subdivision mesh of the original mesh to be coded and the subdivision morphed mesh of the original mesh to be coded; and the encoding end encodes the displacement information to obtain a displacement code stream.

[0388] Figure 26 FIG. 6 is a schematic diagram of a process for generating displacement information according to an embodiment of the present application.

[0389] As shown in Figure 26 First, the encoding end performs subdivision morphing on the reconstructed base mesh using the original mesh to be coded to obtain a subdivision morphed mesh of the current frame. Then, the encoding end performs subdivision on the reconstructed base mesh to obtain a subdivision mesh having consistent connection relationships and vertex correspondence relationships with the subdivision morphed mesh, so as to calculate the displacement vectors of the vertices.

[0390] Figure 27 FIG. 7 is a schematic diagram of inter-slice coding and intra-slice coding according to an embodiment of the present application.

[0391] Exemplarily, after the encoding end obtains the at least one inter-frame slice and the at least one intra-frame slice, inter-frame slice encoding and intra-frame slice encoding can be performed, and after the two parts are encoded, at least one inter-frame code stream and at least one intra-frame code stream two-part sub-code stream can be obtained, and the at least one inter-frame code stream and the at least one intra-frame code stream are combined to form a basic grid code stream. It is worth noting that additional information can also be added in the basic grid code stream, so that the decoding end can parse the at least one inter-frame code stream and the at least one intra-frame code stream. The additional information can be an identifier in the header information of the basic grid code stream, which can be used to indicate at least one of the number of the at least one inter-frame code stream, the number of the at least one intra-frame code stream, the data length of each inter-frame code stream, and the data length of each intra-frame code stream. The additional information and the combination of the at least one inter-frame code stream and the at least one intra-frame code stream can be completed in the code stream merging module.

[0392] In some embodiments, the method 500 further includes:

[0393] The encoding end deforms the subdivided grid of the original grid to be encoded based on the reconstructed displacement information of the displacement information, to obtain a reconstructed deformed grid of the original grid to be encoded.

[0394] Figure 28 FIG. 6 is a schematic diagram of a reconstructed deformed grid provided by an embodiment of the present application.

[0395] As Figure 28 shown, the encoding end merges the reconstructed inter-frame slice and the intra-frame slice to recover the complete reconstructed basic grid. Then, the encoding end performs a subdivision process on the basic grid. After the reconstructed displacement is subjected to the inverse quantization and inverse wavelet transform processes, the displacement can be obtained and superimposed on the subdivided basic grid, and finally the reconstructed reconstructed deformed grid can be obtained.

[0396] Figure 29 FIG. 7 is a schematic flow chart of a slice-based grid encoding process provided by an embodiment of the present application.

[0397] As Figure 29As shown, the encoding end generates two types of encoding patches, i.e. inter-patch and intra-patch, based on the input mesh (i.e. the original mesh to be encoded) for encoding. For the inter-patch, it is encoded based on the reference base mesh using the time-domain prediction technology to obtain the inter-patch code stream. For the intra-patch, it is directly encoded to obtain the intra-patch code stream. The inter-patch code stream and the intra-patch code stream constitute the complete base mesh code stream. The inter-patch code stream includes the indication information (i.e. the first indication information referred to above) for indicating the reference patch and the motion vector information and the like. The intra-patch code stream includes the vertex geometry information, the connection relationship information, the texture coordinate information and the like. The reconstructed inter-patch obtained by the inter-patch encoding module can be regarded as the reconstructed inter-patch after the inter-patch is encoded and decoded. The reconstructed intra-patch obtained by the intra-patch encoding module can be regarded as the reconstructed mesh after the intra-patch is encoded and decoded. The reconstructed inter-patch and the reconstructed intra-patch are subjected to mesh merging processing by the mesh merging processing module to obtain the reconstructed base mesh.

[0398] The displacement generation module performs subdivision deformation on the reconstructed base mesh according to the original mesh to be encoded to obtain the subdivision deformation mesh of the current frame, and then performs subdivision on the reconstructed base mesh to obtain the subdivision mesh having the same connection relationship and vertex correspondence as the subdivision deformation mesh, so as to calculate the displacement vector of each vertex and obtain the displacement code stream and the reconstructed displacement through displacement encoding.

[0399] The deformed mesh reconstruction module generates the reconstructed deformed mesh based on the reconstructed displacement and the reconstructed base mesh. Based on the reconstructed deformed mesh, the texture map conversion is performed using the original mesh to be encoded and the input texture map to obtain the texture map. The generated texture map is encoded to obtain the texture map code stream. In addition, the encoding end can also encode the auxiliary information for guiding the decoding of the decoding end (such as the encoder type, the displacement encoding mode, the wavelet transform coefficient and the like). The encoding end mixes the base mesh code stream, the texture coordinate code stream, the displacement code stream, the texture map code stream, the auxiliary information code stream and the like to obtain the final bit stream.

[0400] Exemplarily, the mesh encoding process can include the following processes:

[0401] (1) Base mesh encoding.

[0402] Inter-patch encoding:

[0403] The inter-patch encoding includes encoding at least one of the connection relationship information, the vertex geometry information and the texture coordinate information in the inter-patch.

[0404] The connection relationship information can be encoded in the following ways: directly using the connection relationship of the reference base mesh, or adding a syntax element in the header information to indicate whether to directly use the connection relationship of the reference base mesh. The texture coordinate information can be encoded in the following ways: directly using the texture coordinate information of the reference base mesh, and the texture coordinates of the vertices in the inter-frame patch directly using the texture coordinates of the corresponding points in the reference base mesh. The vertex geometry information can be encoded according to the rate-distortion criterion to determine whether to encode the motion vector information of each vertex in the inter-frame patch.

[0405] Intra-frame patch encoding:

[0406] The intra-frame patch encoding includes encoding at least one of the connection relationship information, the vertex geometry information, and the texture coordinate information in the intra-frame patch.

[0407] In one case, the connection relationship information, the vertex geometry information, and the texture coordinate information of the intra-frame patch are directly encoded. One possible encoding method is to use Draco compression for static meshes to encode the connection relationship, the vertex geometry coordinates, and the texture coordinates of the base mesh, and finally output the intra-frame code stream. The encoding method of the texture coordinates is similar to the encoding of the vertex geometry information, and a prediction method can also be used. When the texture coordinate connection relationship is different from the vertex geometry information connection relationship, the texture coordinate connection relationship needs to be encoded, and the difference from the vertex geometry information connection relationship can also be encoded.

[0408] Base mesh code stream:

[0409] After the inter-frame patch and the intra-frame patch are encoded, the inter-frame code stream and the intra-frame code stream can be obtained, and the inter-frame code stream and the intra-frame code stream are combined to form the base mesh code stream.

[0410] It should be noted that additional information can also be added to the base mesh code stream to parse the inter-frame code stream and the intra-frame code stream at the decoding end. The additional information can be an identifier in the header information of the base mesh code stream, which can be used to indicate at least one of the number of inter-frame code streams, the number of intra-frame code streams, the data length of each inter-frame code stream, and the data length of each intra-frame code stream. The additional information and the combination of the inter-frame code stream and the intra-frame code stream can be completed in the code stream merging module.

[0411] It should be understood that the specific generation process of the base mesh code stream in the present application can refer to the related content described above, for example, it can refer to the related content of Figure 12 to Figure 14 , Figure 19 to Figure 25 and Figure 27 . To avoid repetition, details are not described here.

[0412] (2) Mesh merging.

[0413] The grid merging processing module merges the reconstructed intra-frame patches and the reconstructed inter-frame patches into a complete reconstructed base mesh. Specifically, the geometry coordinates, triangular patches and texture coordinates of the reconstructed intra-frame patches and the reconstructed inter-frame patches are sequentially written into the reconstructed base mesh in the order of inter-frame first and intra-frame second. Alternatively, the geometry coordinates, triangular patches and texture coordinates of the reconstructed intra-frame patches and the reconstructed inter-frame patches are sequentially written into the reconstructed base mesh in the order of intra-frame first and inter-frame second. It should be understood that the grid merging module is the same as the grid merging module at the decoding end.

[0414] (3) Displacement generation.

[0415] The subdivision deformation module takes the parameterized sub-mesh as input, and first subdivides the original mesh to be encoded. The subdivision scheme can be arbitrarily selected. One possible scheme is the midpoint subdivision scheme, which subdivides each triangle into four sub-triangles in each iteration of subdivision. A new vertex is introduced in the middle of each edge. The subdivision of geometry information and attribute information is performed independently, because the connection relationship between geometry information and attribute information is usually different.

[0416] It should be understood that the displacement generation process in the present application can refer to the related content described above, for example, it can refer to the related content of Figure 26 . To avoid repetition, details are not described here.

[0417] (4) Displacement encoding.

[0418] The displacement data generated is mainly encoded, and the implementation block diagram is shown in Figure 30 . The displacement encoding module performs wavelet transform, quantization and other processing on the displacement to obtain quantized wavelet transform coefficients. Finally, the quantized wavelet transform coefficients are encoded to obtain the displacement code stream. There are many possible specific ways to encode the quantized wavelet transform coefficients. One possible way is to reuse the existing V-DMC mode: arrange the quantized wavelet transform coefficients into a video frame and input them into a video encoder for encoding, or directly perform entropy encoding. Of course, another possible way is to directly entropy encode the displacement information without the aforementioned wavelet transform, quantization and other links.

[0419] (5) Deformed mesh reconstruction.

[0420] The deformed mesh reconstruction module merges the reconstructed inter-frame patches and intra-frame patches to recover a complete reconstructed base mesh. Then, the base mesh is subdivided at the encoding end. After the reconstructed displacement is dequantized and inverse wavelet transformed, the displacement can be obtained and superimposed on the subdivided base mesh, and finally the reconstructed deformed mesh can be obtained.

[0421] It should be understood that the deformed mesh reconstruction process in the present application can refer to the related content described above, for example, it can refer to the related content of Figure 28The related content of the above-mentioned is not repeated here.

[0422] (6) Texture conversion.

[0423] The texture conversion module performs texture conversion according to the input original mesh, the input original texture, and the reconstructed deformed mesh. The texture conversion can refer to the related description mentioned above, for example Figure 9 The related content of the above-mentioned is not repeated here.

[0424] (7) Texture compression.

[0425] After obtaining the converted texture, for the empty pixels in the texture, an existing filling algorithm (such as the Push-Pull algorithm) can be used to fill the empty pixels. Then, an existing video encoder, such as H.264 / AVC, H.265 / HEVC, H.266 / VVC, etc., can be used to encode the texture to obtain the code stream of the output texture. In addition, color space conversion and chroma subsampling operations can be selectively applied to make the video encoding obtain better rate-distortion performance, such as color space conversion from RGB 444 to YUV420.

[0426] (8) Auxiliary information encoding.

[0427] The encoding end determines the auxiliary information and encodes the auxiliary information based on the encoding scheme, which mainly includes an intra-frame encoding identifier indicating whether the current frame needs to be constructed according to the reference frame mesh, a reference frame list indicating the index of the reference frame needed by the current frame, the reference frame list being applied to the subsequent base mesh reconstruction step, a subdivision identifier indicating whether the reconstructed base mesh needs to be subjected to a subsequent subdivision deformation operation, a static mesh encoder type guiding the decoding end to use a corresponding static mesh decoder, a video encoder type guiding the decoding end to use a corresponding video decoder, a subdivision scheme, i.e., a scheme for subdividing the base mesh in the reconstructed deformed mesh, the encoding and decoding ends should keep consistent, and optional displacement transformation schemes, coefficient arrangement schemes, etc.

[0428] It is worth noting that the auxiliary information described here includes auxiliary information transmitted independently and header information that can be included in other code stream parts.

[0429] After all the modules in the encoding end are encoded, the base mesh code stream, the texture coordinate code stream, the displacement code stream, the texture code stream, the auxiliary information code stream, etc. are mixed to obtain the final output encoded bit stream. The specific implementation of the mixing of the present application is not limited.

[0430] In this embodiment, the three-dimensional mesh is divided into intra-frame slices and inter-frame slices. Inter-frame slices use an inter-frame coding mode, while intra-frame slices use an intra-frame coding mode. An inter-frame slice can be a mesh region determined by the encoder based on matching patches between the registered base mesh and the registered subdivided deformed mesh of the original mesh to be encoded. Its connectivity, vertex geometric coordinates, texture coordinates, etc. are encoded based on the reference base mesh using time-domain prediction technology, greatly improving compression efficiency. An intra-frame slice can be a mesh region determined by the encoder based on mismatching patches between the registered base mesh of the original mesh to be encoded and the registered subdivided deformed mesh of the original mesh to be encoded. It is a region that aligns with the boundary of an inter-frame slice, obtained by obtaining an initial intra-frame slice from the original mesh to be encoded through boundary stretching, mesh processing, and other steps. Its connectivity, vertex geometric coordinates, texture coordinates, etc. are directly encoded. A mesh can contain multiple intra-frame slices and multiple inter-frame slices. Multiple intra-frame slices and multiple inter-frame slices are connected together for joint processing such as displacement generation and displacement coding. For subsequent time domain grids, they can be regarded as a whole reference grid.

[0431] The encoding method provided in this application is applicable not only to encoding of a 3D base grid, but also to encoding of an inter-frame 3D base grid, and also to encoding of a subgrid of a 3D base grid or a subgrid of an inter-frame 3D base grid. When the grid to be encoded contains multiple subgrids, each subgrid can be encoded according to the proposed scheme, i.e., the proposed encoding scheme processing unit can be a subgrid.

[0432] It should be understood that the encoding method can be understood as the inverse process of the decoding method (or called the reverse process). Therefore, the specific scheme of the slice-based grid encoding method 500 can refer to the relevant content of the slice-based grid decoding method 400. For the sake of ease of description, this application will not go into details.

[0433] The slice-based trellis decoding method provided in the embodiments of the present application can be performed by a slice-based trellis decoding device. In the embodiments of the present application, a slice-based trellis decoding device performing the slice-based trellis decoding method is used as an example to illustrate the slice-based trellis decoding device provided in the embodiments of the present application. The slice-based trellis encoding method provided in the embodiments of the present application can be performed by a slice-based trellis encoding device. In the embodiments of the present application, a slice-based trellis encoding device performing the slice-based trellis encoding method is used as an example to illustrate the slice-based trellis encoding device provided in the embodiments of the present application.

[0434] Figure 31 It is a schematic block diagram of a slice-based trellis decoding device 600 provided according to an embodiment of the present application.

[0435] like Figure 31 As shown, the slice-based trellis decoding apparatus 600 includes:

[0436] The acquisition unit 610 is configured to acquire at least one inter-frame code stream of a to-be-decoded base mesh and at least one intra-frame code stream of the to-be-decoded base mesh.

[0437] The first decoding unit 620 is configured to decode the at least one inter-frame code stream in an inter-frame decoding mode to obtain at least one inter-frame slice.

[0438] The second decoding unit 630 is configured to decode the at least one intra-frame code stream in an intra-frame decoding mode to obtain at least one intra-frame slice.

[0439] The merging unit 640 is configured to merge the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base mesh of the to-be-decoded base mesh.

[0440] In some embodiments, the acquisition unit 610 is specifically configured to:

[0441] In a case where a decoding mode used by the to-be-decoded base mesh is not an intra-frame decoding mode, the at least one inter-frame code stream and the at least one intra-frame code stream are acquired.

[0442] In some embodiments, the acquisition unit 610 is specifically configured to:

[0443] The base mesh code stream of the to-be-decoded base mesh is acquired, and the base mesh code stream includes header information and a data unit.

[0444] Based on the header information, the at least one inter-frame code stream and the at least one intra-frame code stream are acquired from the data unit.

[0445] In some embodiments, the header information includes at least one of the following:

[0446] An identifier used to indicate an order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0447] An identifier used to indicate a data length of the at least one inter-frame code stream;

[0448] An identifier used to indicate a number of the at least one inter-frame code stream;

[0449] An identifier used to indicate a length of each inter-frame code stream in the at least one inter-frame code stream;

[0450] An identifier used to indicate a length of each intra-frame code stream except for a last intra-frame code stream in the at least one inter-frame code stream;

[0451] An identifier used to indicate a data length of the at least one intra-frame code stream;

[0452] an identifier for indicating a number of the at least one intra code stream;

[0453] an identifier for indicating a length of each intra code stream in the at least one intra code stream;

[0454] an identifier for indicating a length of each intra code stream except the last intra code stream in the at least one intra code stream.

[0455] In some embodiments, the first decoding unit 620 is specifically configured to:

[0456] decode a first inter code stream in the at least one inter code stream to obtain first indication information and to-be-decoded information of a first inter tile in the at least one inter tile, the first indication information being used for indicating at least one reference face tile used by the first inter tile in a reference base mesh of the to-be-decoded base mesh;

[0457] decode the to-be-decoded information by using the inter decoding mode based on the at least one reference face tile to obtain the first inter tile.

[0458] In some embodiments, the first indication information is an array with a length equal to a number of face tiles of the reference base mesh, and a value at any position in the array is used for indicating whether a reference face tile used by the first inter tile includes a face tile corresponding to the any position in the reference base mesh.

[0459] In some embodiments, the reconstructed base mesh includes texture coordinates of the at least one inter tile and texture coordinates of the at least one intra tile, and the obtaining unit 610 is further configured to:

[0460] scale the texture coordinates of the at least one inter tile in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter tile;

[0461] scale the texture coordinates of the at least one intra tile in the first direction to obtain scaled texture coordinates of the at least one intra tile.

[0462] In some embodiments, the obtaining unit 610 is specifically configured to:

[0463] multiply a component of the texture coordinates of the at least one inter tile in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter tile.

[0464] In some embodiments, the obtaining unit 610 is specifically configured to:

[0465] a component of a texture coordinate of the at least one intra slice in the first direction is multiplied by a second coefficient and then added to the first coefficient to obtain a scaled texture coordinate of the at least one intra slice;

[0466] wherein a sum of the first coefficient and the second coefficient is 1.

[0467] It should be understood that the slice-based mesh decoding apparatus 600 provided by the embodiments of the present application can correspond to the execution subject in the method embodiments of the present application, and each unit in the slice-based mesh decoding apparatus 600 is respectively used to implement the corresponding process of the slice-based mesh decoding method 400 shown in FIG. 4, and for brevity, will not be described here again. Figure 11

[0468] The slice-based mesh decoding apparatus provided by the embodiments of the present application can implement the various processes of the method embodiments and achieve the same technical effects, and for the sake of brevity, will not be described here again. Figure 11

[0469] Figure 32 FIG. 7 is a schematic block diagram of a slice-based mesh encoding apparatus 700 according to an embodiment of the present application.

[0470] As shown in FIG. 7, the slice-based mesh encoding apparatus 700 includes: Figure 32 a determining unit 710, configured to determine at least one inter slice of a to-be-encoded original mesh and at least one intra slice of the to-be-encoded original mesh;

[0471] a first encoding unit 720, configured to encode the at least one inter slice by using an inter encoding mode to obtain at least one inter code stream;

[0472] a second encoding unit 730, configured to encode the at least one intra slice by using an intra encoding mode to obtain at least one intra code stream;

[0473] a merging unit 740, configured to merge the at least one inter code stream and the at least one intra code stream to obtain a basic mesh code stream.

[0474] In some embodiments, the determining of the at least one inter slice of the to-be-encoded original mesh and the at least one intra slice of the to-be-encoded original mesh includes:

[0475] In a case where an encoding mode used by the to-be-encoded original mesh is not an intra encoding mode, the at least one inter slice of the to-be-encoded original mesh and the at least one intra slice of the to-be-encoded original mesh are determined.

[0476] In some embodiments, the merging unit 740 is specifically configured to:

[0477] In some embodiments, the merging unit 740 is specifically configured to:​​

[0478] merge the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit;

[0479] merge the header information of the base mesh code stream and the data unit to obtain the base mesh code stream.

[0480] In some embodiments, the header information comprises at least one of:

[0481] an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream;

[0482] an identifier for indicating the data length of the at least one inter-frame code stream;

[0483] an identifier for indicating the number of the at least one inter-frame code stream;

[0484] an identifier for indicating the length of each inter-frame code stream in the at least one inter-frame code stream;

[0485] an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one inter-frame code stream;

[0486] an identifier for indicating the data length of the at least one intra-frame code stream;

[0487] an identifier for indicating the number of the at least one intra-frame code stream;

[0488] an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream;

[0489] an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one intra-frame code stream.

[0490] In some embodiments, the first encoding unit 620 is specifically configured to:

[0491] encode a first inter-frame slice in the at least one inter-frame slice based on at least one reference patch in a reference base mesh of the original mesh to be encoded, to obtain encoding information of the first inter-frame slice, using the inter-frame encoding mode;

[0492] determine first indication information based on the at least one reference patch, the first indication information being used to indicate the at least one reference patch;

[0493] encode the first indication information and the encoding information of the first inter-frame slice to obtain a first inter-frame code stream in the at least one first inter-frame code stream.

[0494] In some embodiments, the first indication information is an array with a length equal to a number of patches of the reference base mesh, and a value at any position of the array is used to indicate whether a reference patch used by the first inter-frame patch includes a patch in the reference base mesh corresponding to the any position.

[0495] In some embodiments, the reconstructed base mesh of the original mesh to be encoded includes texture coordinates of the at least one inter-frame patch and texture coordinates of the at least one intra-frame patch, and the determining unit 710 is further configured to:

[0496] scaling the texture coordinates of the at least one inter-frame patch along a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch;

[0497] scaling the texture coordinates of the at least one intra-frame patch along the first direction to obtain scaled texture coordinates of the at least one intra-frame patch.

[0498] In some embodiments, the determining unit 710 is specifically configured to:

[0499] multiplying a component of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

[0500] In some embodiments, the determining unit 710 is specifically configured to:

[0501] multiplying a component of the texture coordinates of the at least one intra-frame patch in the first direction by a second coefficient and then adding the first coefficient to obtain the scaled texture coordinates of the at least one intra-frame patch;

[0502] wherein a sum of the first coefficient and the second coefficient is 1.

[0503] In some embodiments, the determining unit 710 is specifically configured to:

[0504] determining a registration base mesh of the original mesh to be encoded and a registration subdivision deformation mesh of the original mesh to be encoded based on a reference base mesh of the original mesh to be encoded and a reference subdivision deformation mesh of the original mesh to be encoded;

[0505] wherein the registration base mesh is a mesh obtained by deforming the original mesh to be encoded, has a same number of vertices as the reference base mesh, and has a same connectivity relationship as the reference base mesh; and the registration subdivision deformation mesh is a mesh obtained by deforming the original mesh to be encoded, has a same number of vertices as the reference subdivision deformation mesh, and has a same connectivity relationship as the reference subdivision deformation mesh.

[0506] based on the matching detection result between the registration base mesh and the registration subdivision deformation mesh, mark a patch in the registration base mesh as a matched patch or an unmatched patch to obtain a first marking result;

[0507] based on the first marking result, determine the at least one inter-frame patch in the registration base mesh.

[0508] In some embodiments, the determination unit 710 is specifically configured to:

[0509] deform the to-be-encoded original mesh with the reference subdivision deformation mesh as a target mesh to obtain an intermediate mesh;

[0510] deform the reference subdivision deformation mesh with the intermediate mesh as a target mesh to obtain the registration subdivision deformation mesh;

[0511] deform the reference base mesh with the registration subdivision deformation mesh as a target mesh to obtain the registration base mesh.

[0512] In some embodiments, the first marking result includes a marking result of a first patch in the registration base mesh.

[0513] In some embodiments, the determination unit 710 is specifically configured to:

[0514] obtain a first bounding box in which the first patch is located;

[0515] based on the first bounding box, obtain a first region in the to-be-encoded original mesh corresponding to the first bounding box and a second region in the matched subdivision deformation mesh corresponding to the first bounding box;

[0516] calculate a first weighted average normal vector of patches in the first region and a second weighted average normal vector of patches in the second region;

[0517] in a case where an included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, determine that the marking result of the first patch is that the first patch is marked as an unmatched patch, or in a case where the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, mark the first patch as a matched patch to obtain the marking result of the first patch.

[0518] In some embodiments, the determination unit 710 is specifically configured to:

[0519] obtain at least one third region where the patches marked as mismatched patches are located based on the first marking result;

[0520] re-mark the patches in a first region in the at least one third region where the total area of the patches is less than or equal to a preset first area threshold as matched patches, to obtain a third marking result;

[0521] obtain at least one fourth region where the patches marked as matched patches are located based on the third marking result;

[0522] re-mark the patches in a second region in the at least one fourth region where the total area of the patches is less than or equal to a preset first area threshold as mismatched patches, to obtain a fourth marking result;

[0523] determine, based on the fourth marking result, a region where the patches marked as matched patches are located as the at least one inter-frame patch.

[0524] In some embodiments, the determination unit 710 is specifically configured to:

[0525] mark the patches in the original mesh to be encoded as matched or mismatched patches based on the marking result, to obtain a second marking result;

[0526] determine at least one initial intra-frame patch in the original mesh to be encoded based on the second marking result;

[0527] perform boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch, to obtain the at least one intra-frame patch.

[0528] In some embodiments, the second marking result includes a marking result of a second patch in the original mesh to be encoded;

[0529] In some embodiments, the determination unit 710 is specifically configured to:

[0530] obtain a barycenter of the second patch;

[0531] search for a nearest neighbor point of the barycenter of the second patch in the registered base mesh;

[0532] mark the second patch as a matched or mismatched patch based on a marking of a patch where the nearest neighbor point is located, to obtain a marking result of the second patch.

[0533] In some embodiments, the determination unit 710 is specifically configured to:

[0534] obtain at least one fifth region where the patches marked as mismatched patches are located based on the second marking result;

[0535] relabeling, as mismatched patches, patches in a third region in the at least one fifth region and having a total patch area less than or equal to the preset second area threshold, to obtain a fifth labeling result;

[0536] obtaining, based on the fifth labeling result, at least one sixth region in which the patches labeled as matched patches are located;

[0537] relabeling, as mismatched patches, patches in a fourth region in the at least one sixth region and having a total patch area less than or equal to the preset second area threshold, to obtain a sixth labeling result;

[0538] determining, as the at least one initial intra-patch, a region in which the patches labeled as mismatched patches are located based on the sixth labeling result.

[0539] In some embodiments, the determining unit 710 is specifically configured to:

[0540] obtaining at least one edge in the registration base mesh in which there are adjacent patches labeled as matched and mismatched;

[0541] obtaining at least one vertex in the original mesh to be encoded in which there are adjacent patches labeled as matched and mismatched;

[0542] for a first vertex corresponding to the at least one edge, searching for a nearest neighbor point of the first vertex in the at least one vertex;

[0543] deforming the nearest neighbor point of the first vertex to the first vertex;

[0544] for a second vertex in the at least one vertex except the nearest neighbor point of the first vertex, searching for a nearest neighbor point of the second vertex on the at least one boundary;

[0545] obtaining an edge in which the nearest neighbor point of the second vertex is located based on the nearest neighbor point of the second vertex;

[0546] determining a position closest to the second vertex on the edge in which the nearest neighbor point of the second vertex is located;

[0547] deforming the second vertex to the position closest to the second vertex to obtain the at least one intra-patch.

[0548] In some embodiments, the apparatus 700 further includes a third encoding unit configured to:

[0549] obtaining a reconstructed base mesh of the original mesh to be encoded;

[0550] Based on the original mesh to be encoded, subdividing and deforming the reconstructed basic mesh to obtain a subdivided mesh of the original mesh to be encoded and a subdivided deformed mesh of the original mesh to be encoded;

[0551] Determining displacement information of the original mesh to be encoded based on the subdivided mesh of the original mesh to be encoded and the subdivided deformed mesh of the original mesh to be encoded;

[0552] The displacement information is encoded to obtain a displacement code stream.

[0553] In some embodiments, the apparatus 700 further includes a reconstruction unit, configured to:

[0554] Based on the reconstructed displacement information of the displacement information, the subdivided grid of the original grid to be encoded is deformed to obtain a reconstructed deformed grid of the original grid to be encoded.

[0555] It should be understood that the slice-based trellis coding device 700 provided in the embodiment of the present application may correspond to the execution subject in the method embodiment of the present application, and the various units in the slice-based trellis coding device 700 are respectively for implementing Figure 18 For the sake of brevity, the corresponding process of the slice-based grid coding method 500 is not repeated here.

[0556] The slice-based grid coding device provided in the embodiment of the present application can achieve Figure 18 The various processes implemented by the method embodiment achieve the same technical effect and are not described here again to avoid repetition.

[0557] The embodiment of the present application further provides an electronic device 800, such as Figure 33 As shown, it includes a processor 801 and a memory 802 , and the memory 802 stores programs or instructions that can be run on the processor 801 .

[0558] For example, when the electronic device 800 is a decoding end, the program or instruction is executed by the processor 801 to implement the various steps of the above-mentioned embodiment of the grid decoding method based on the slice, and can achieve the same technical effect. When the electronic device 800 is an encoding end, the program or instruction is executed by the processor 801 to implement the various steps of the above-mentioned embodiment of the grid encoding method based on the slice, and can achieve the same technical effect. To avoid repetition, it is not repeated here. Optionally, the memory 802 can be Figure 1 In the memory 102 or memory 113 in the embodiment shown, the processor 801 may implement Figure 1 Functionality of the encoder 200 or decoder 300 in the illustrated embodiment.

[0559] The embodiments of the present application also provide an electronic device, comprising: a memory configured to store video data; and a processing circuit configured to implement each step of the above-mentioned slice-based trellis decoding method embodiments or the above-mentioned slice-based trellis encoding method embodiments. Optionally, the memory can be Figure 1 The processing circuit can implement the functions of the encoder 200 or the decoder 300 in the embodiments shown in the above-mentioned slice-based trellis decoding method embodiments or the above-mentioned slice-based trellis encoding method embodiments. Figure 1 The processing circuit can implement the functions of the encoder 200 or the decoder 300 in the embodiments shown in the above-mentioned slice-based trellis decoding method embodiments or the above-mentioned slice-based trellis encoding method embodiments.

[0560] The embodiments of the present application also provide an electronic device, comprising a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is configured to run programs or instructions to implement the steps in the above-mentioned slice-based trellis decoding method embodiments or the above-mentioned slice-based trellis encoding method embodiments. The device embodiments correspond to the above-mentioned method embodiments, and each implementation process and implementation manner of the above-mentioned method embodiments can be applicable to the terminal embodiments, and the same technical effects can be achieved.

[0561] The electronic device can be a terminal, or other devices except the terminal, such as a server, a network attached storage (NAS), etc.

[0562] The terminal can be a mobile phone, a tablet personal computer (Tablet Personal Computer, Tablet PC), a laptop computer (Laptop Computer, Laptop), a notebook computer, a personal digital assistant (Personal Digital Assistant, PDA), a palm computer, a netbook, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR) device, a virtual reality (Virtual Reality, VR) device, a mixed reality (mixed reality, MR) device, a robot, a wearable device, a flight vehicle, a vehicle user equipment (Vehicle User Equipment, VUE), a shipboard device, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (a home device with wireless communication function, such as a refrigerator, a television, a washing machine, or furniture, etc.), a game console, a personal computer (Personal Computer, PC), a teller machine, or a self-service machine, and the like. The wearable device includes a smart watch, a smart bracelet, a smart earphone, smart glasses, smart jewelry (smart bracelet, smart necklace, smart ring, smart necklace, smart anklet, smart necklace, etc.), a smart wristband, smart clothing, and the like. The vehicle-mounted device can also be referred to as a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip, or a vehicle-mounted unit, and the like. It should be noted that the specific type of the terminal is not limited in the embodiments of the present application.

[0563] The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server. The cloud server can provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), or cloud computing services based on big data and artificial intelligence platforms.

[0564] For example, the electronic device described above can include but is not limited to Figure 1 The type of the source device 100 or the destination device 110 shown.

[0565] Taking the electronic device as an example, Figure 34 A hardware structure schematic diagram of a terminal 900 according to an embodiment of the present application is shown.

[0566] As shown in Figure 34 The terminal 900 includes, but is not limited to, at least part of the components such as a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.

[0567] Those skilled in the art can understand that the terminal 900 can also include a power supply (such as a battery) for supplying power to each component, and the power supply can be logically connected to the processor 910 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Figure 34 The terminal structure shown in the figure does not constitute a limitation on the terminal, and the terminal can include more or fewer components than the figure, or combine certain components, or different component arrangements, which are not described here.

[0568] It should be understood that in the embodiments of the present application, the input unit 904 can include a graphics processing unit (GPU) 9041 and a microphone 9042. The graphics processor 9041 processes image data of a still picture or a video obtained by an image acquisition device (such as a camera) in a video capture mode or an image capture mode, or can process obtained point cloud data. The display unit 906 can include a display panel 9061, which can be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. The touch panel 9071 is also called a touch screen. The touch panel 9071 can include a touch detection device and a touch controller. The other input devices 9072 can include, but are not limited to, a physical keyboard, function keys (such as volume control keys, on-off keys, etc.), trackballs, mice, joysticks, etc., which are not described here.

[0569] In the embodiments of the present application, the radio frequency unit 901 can transmit data to the processor 910 for processing after receiving data from the opposite end. In addition, the radio frequency unit 901 can send data to the opposite end. Generally, the radio frequency unit 901 includes, but is not limited to, an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

[0570] The memory 909 can be used to store software programs or instructions and various data. The memory 909 can mainly include a first storage area storing programs or instructions and a second storage area storing data, wherein the first storage area can store an operating system, application programs or instructions required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. In addition, the memory 909 can include a volatile memory or a non-volatile memory. The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Synchronous DRAM (SDRAM), a Double Data Rate SDRAM (DDR SDRAM), an Enhanced SDRAM (ESDRAM), a Synch link DRAM (SLDRAM), and a Direct Rambus RAM (DRRAM). The memory 909 in the embodiments of the present application includes but is not limited to these and any other suitable types of memory.

[0571] The processor 910 can include one or more processing units; optionally, the processor 910 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and an application program, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 910.

[0572] As an implementation manner, the terminal 900 is a decoding end, the radio frequency unit 901 is configured to: acquire at least one inter-frame code stream of a to-be-decoded basic grid and at least one intra-frame code stream of the to-be-decoded basic grid; and the processor 910 is configured to:

[0573] decode the at least one inter-frame code stream in an inter-frame decoding mode to obtain at least one inter-frame slice;

[0574] decode the at least one intra stream by using the intra decoding mode to obtain at least one intra slice;

[0575] merge the at least one inter slice and the at least one intra slice to obtain a reconstructed base mesh of the to-be-decoded base mesh.

[0576] As another implementation manner, the terminal 900 is an encoding end, and the processor 910 is configured to:

[0577] determine at least one inter slice of a to-be-encoded original mesh and at least one intra slice of the to-be-encoded original mesh;

[0578] encode the at least one inter slice by using the inter encoding mode to obtain at least one inter stream;

[0579] encode the at least one intra slice by using the intra encoding mode to obtain at least one intra stream;

[0580] merge the at least one inter stream and the at least one intra stream to obtain a base mesh stream.

[0581] In the mesh decoding method based on slices provided in the embodiment, the decoding end avoids decoding the entire base mesh by directly using the intra decoding mode or the inter decoding mode, and the decoding effect can be improved.

[0582] It can be understood that the implementation processes of the implementation manners mentioned in the embodiment can refer to the related descriptions of the method embodiments and achieve the same or corresponding technical effects. To avoid repetition, they will not be described herein again.

[0583] The embodiment of the present application further provides a readable storage medium, and the readable storage medium stores a program or instructions. When the program or instructions are executed by a processor, each process of the above-mentioned mesh decoding method based on slices or the above-mentioned mesh encoding method based on slices is implemented, and the same technical effects can be achieved. To avoid repetition, they will not be described herein again.

[0584] The processor is the processor in the terminal in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a ROM, a RAM, a magnetic disc, or an optical disc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0585] The embodiment of the present application further provides a chip, and the chip includes a processor and a communication interface. The communication interface is coupled with the processor. The processor is configured to run a program or instructions to implement each process of the above-mentioned mesh decoding method based on slices or the above-mentioned mesh encoding method based on slices, and the same technical effects can be achieved. To avoid repetition, they will not be described herein again.

[0586] It should be understood that the chip mentioned in the embodiments of the present application can include a system-level chip (also referred to as a system chip, a chip system or a system-on-chip chip), and can also include a standalone display chip and the like.

[0587] The embodiments of the present application further provide a computer program / program product stored in a storage medium, which is executed by at least one processor to implement the processes of the above-mentioned slice-based grid decoding method embodiments or the above-mentioned slice-based grid encoding method embodiments, and achieve the same technical effects. To avoid repetition, details are not described herein.

[0588] The embodiments of the present application further provide a coding and decoding system, which includes an encoding end and a decoding end. The encoding end can be used to execute the steps of the encoding method described above, and the decoding end can be used to execute the steps of the decoding method described above.

[0589] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to the order of performing functions as shown or discussed, and can also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method can be performed in an order different from that described, and various steps can also be added, omitted or combined. In addition, the features described with reference to certain examples can be combined in other examples.

[0590] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of computer software products and general hardware platforms, of course, they can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and includes a plurality of instructions for making the decoding end execute the decoding method described in each embodiment of the present application or the encoding end execute the encoding method described in each embodiment of the present application.

[0591] The embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, but not restrictive, and a person of ordinary skill in the art can make many forms of embodiments under the inspiration of the present application without departing from the purpose of the present application and the scope protected by the claims, and these embodiments all belong to the protection of the present application.

Claims

1. A slice-based trellis decoding method, characterized by, The method comprises the following steps: The decoding end obtains at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid; The decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice; The decoding end decodes the at least one intra-frame code stream by using an intra-frame decoding mode to obtain at least one intra-frame slice; The decoding end combines the at least one inter-frame slice and the at least one intra-frame slice to obtain a reconstructed base grid of the to-be-decoded base grid.

2. The method of claim 1, wherein, The decoding end obtains at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid, comprising: In the case that the decoding mode used by the to-be-decoded base grid is not an intra-frame decoding mode, the decoding end obtains the at least one inter-frame code stream and the at least one intra-frame code stream.

3. The method according to claim 1 or 2, characterized in that, The decoding end obtains at least one inter-frame code stream of a to-be-decoded base grid and at least one intra-frame code stream of the to-be-decoded base grid, comprising: The decoding end obtains a base grid code stream of the to-be-decoded base grid, wherein the base grid code stream comprises header information and a data unit; The decoding end obtains the at least one inter-frame code stream and the at least one intra-frame code stream from the data unit based on the header information.

4. The method of claim 3, wherein, The header information comprises at least one of the following: an identifier for indicating the order of the at least one inter-frame code stream and the at least one intra-frame code stream; an identifier for indicating the data length of the at least one inter-frame code stream; an identifier for indicating the number of the at least one inter-frame code stream; an identifier for indicating the length of each inter-frame code stream in the at least one inter-frame code stream; an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one inter-frame code stream; an identifier for indicating the data length of the at least one intra-frame code stream; an identifier for indicating the number of the at least one intra-frame code stream; an identifier for indicating the length of each intra-frame code stream in the at least one intra-frame code stream; an identifier for indicating the length of each intra-frame code stream except the last intra-frame code stream in the at least one intra-frame code stream.

5. The method according to any one of claims 1 to 4, characterized in that, The decoding end decodes the at least one inter-frame code stream by using an inter-frame decoding mode to obtain at least one inter-frame slice, comprising: The decoding end decodes a first inter-frame code stream in the at least one inter-frame code stream to obtain first indication information and to-be-decoded information of a first inter-frame slice in the at least one inter-frame slice, wherein the first indication information is used for indicating at least one reference surface slice used by the first inter-frame slice in a reference base grid of the to-be-decoded base grid; The decoding end decodes the to-be-decoded information by using the inter-frame decoding mode based on the at least one reference surface slice to obtain the first inter-frame slice.

6. The method of claim 5, wherein, The first indication information is an array with a length equal to the number of surface slices of the reference base grid, and the value of any position in the array is used for indicating whether the reference surface slice used by the first inter-frame slice comprises a surface slice corresponding to the any position in the reference base grid.

7. The method according to any one of claims 1 to 6, characterized in that, The reconstructed base mesh includes texture coordinates of the at least one inter-frame patch and texture coordinates of the at least one intra-frame patch, and the method further includes: The decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch; The decoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch.

8. The method of claim 7, wherein, The decoding end scales the texture coordinates of the at least one inter-frame patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter-frame patch, including: The decoding end multiplies components of the texture coordinates of the at least one inter-frame patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter-frame patch.

9. The method of claim 8, wherein, The decoding end scales the texture coordinates of the at least one intra-frame patch in the first direction to obtain scaled texture coordinates of the at least one intra-frame patch, including: The decoding end multiplies components of the texture coordinates of the at least one intra-frame patch in the first direction by a second coefficient first, and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra-frame patch. Wherein, the sum of the first coefficient and the second coefficient is 1.

10. A slice-based trellis encoding method, characterized by, Including: The encoding end determines at least one inter-frame patch of a to-be-encoded original mesh and at least one intra-frame patch of the to-be-encoded original mesh; The encoding end encodes the at least one inter-frame patch in an inter-frame coding mode to obtain at least one inter-frame code stream; The encoding end encodes the at least one intra-frame patch in an intra-frame coding mode to obtain at least one intra-frame code stream; The encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream.

11. The method of claim 10, wherein, The encoding end determines at least one inter-frame patch of a to-be-encoded original mesh and at least one intra-frame patch of the to-be-encoded original mesh, including: In a case where an encoding mode used by the to-be-encoded original mesh is not an intra-frame encoding mode, the encoding end determines at least one inter-frame patch of a to-be-encoded original mesh and at least one intra-frame patch of the to-be-encoded original mesh.

12. The method according to claim 10 or 11, characterized in that, The encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a base mesh code stream, including: The encoding end merges the at least one inter-frame code stream and the at least one intra-frame code stream to obtain a data unit; The encoding end merges the header information of the base mesh code stream and the data unit to obtain the base mesh code stream.

13. The method of claim 12, wherein, The header information includes at least one of: An identifier for indicating an order of the at least one inter-frame code stream and the at least one intra-frame code stream; An identifier for indicating a data length of the at least one inter-frame code stream; An identifier for indicating a number of the at least one inter-frame code stream; An identifier for indicating lengths of respective inter-frame code streams in the at least one inter-frame code stream; An identifier for indicating lengths of respective intra-frame code streams except for a last intra-frame code stream in the at least one inter-frame code stream; an identifier for indicating a data length of the at least one intra code stream; an identifier for indicating a number of the at least one intra code stream; an identifier for indicating a length of each of the at least one intra code stream; an identifier for indicating a length of each of the at least one intra code stream except the last one.

14. The method according to any one of claims 10 to 13, characterized in that, the encoding end encodes the at least one inter patch in the inter encoding mode to obtain at least one inter code stream, including: the encoding end encodes a first inter patch in the at least one inter patch in the inter encoding mode based on at least one reference patch in a reference base mesh of the to-be-encoded original mesh to obtain encoding information of the first inter patch; the encoding end encodes first indication information and the encoding information of the first inter patch to obtain a first inter code stream in the at least one first inter code stream, and the first indication information is used to indicate the at least one reference patch.

15. The method of claim 14, wherein, the first indication information is an array with a length equal to a number of patches in the reference base mesh, and a value at any position in the array is used to indicate whether a reference patch used by the first inter patch includes a patch in the reference base mesh corresponding to the any position.

16. The method according to any one of claims 10 to 15, characterized in that, the reconstructed base mesh of the to-be-encoded original mesh includes texture coordinates of the at least one inter patch and texture coordinates of the at least one intra patch, and the method further includes: the encoding end scales the texture coordinates of the at least one inter patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter patch; the encoding end scales the texture coordinates of the at least one intra patch in the first direction to obtain scaled texture coordinates of the at least one intra patch.

17. The method of claim 16, wherein, the encoding end scales the texture coordinates of the at least one inter patch in a first direction of a texture coordinate axis to obtain scaled texture coordinates of the at least one inter patch, including: the encoding end multiplies a component of the texture coordinates of the at least one inter patch in the first direction by a first coefficient to obtain the scaled texture coordinates of the at least one inter patch.

18. The method of claim 17, wherein, the encoding end scales the texture coordinates of the at least one intra patch in the first direction to obtain scaled texture coordinates of the at least one intra patch, including: the encoding end multiplies a component of the texture coordinates of the at least one intra patch in the first direction by a second coefficient and then adds the first coefficient to obtain the scaled texture coordinates of the at least one intra patch; wherein a sum of the first coefficient and the second coefficient is 1.

19. The method according to any one of claims 10 to 18, characterized in that, the encoding end determines the at least one inter patch, including: the encoding end determines a registration base mesh of the to-be-encoded original mesh and a registration subdivision deformation mesh of the to-be-encoded original mesh based on a reference base mesh of the to-be-encoded original mesh and a reference subdivision deformation mesh of the to-be-encoded original mesh; The registration base mesh is a mesh obtained by deforming the original mesh to be encoded, has the same number of vertices as the reference base mesh, and has the same connectivity relationship as the reference base mesh; and the registration subdivided deformed mesh is a mesh obtained by deforming the original mesh to be encoded, has the same number of vertices as the reference subdivided deformed mesh, and has the same connectivity relationship as the reference subdivided deformed mesh. The encoding end marks the face patches in the registration base mesh as matched or unmatched face patches based on the matching detection result between the registration base mesh and the registration subdivided deformed mesh, to obtain a first marking result. The encoding end determines the at least one inter-frame patch in the registration base mesh based on the first marking result.

20. The method of claim 19, wherein, The encoding end determines the registration base mesh of the original mesh to be encoded and the registration subdivided deformed mesh of the original mesh to be encoded based on a reference base mesh of the original mesh to be encoded and a reference subdivided deformed mesh of the original mesh to be encoded, and includes the following steps. The encoding end deforms the original mesh to be encoded by taking the reference subdivided deformed mesh as a target mesh, to obtain an intermediate mesh. The encoding end deforms the reference subdivided deformed mesh by taking the intermediate mesh as a target mesh, to obtain the registration subdivided deformed mesh. The encoding end deforms the reference base mesh by taking the registration subdivided deformed mesh as a target mesh, to obtain the registration base mesh.

21. The method of claim 19 or 20, wherein, The first marking result includes a marking result of a first face patch in the registration base mesh. The encoding end marks the face patches in the registration base mesh as matched or unmatched face patches based on the matching detection result between the registration base mesh and the registration subdivided deformed mesh, to obtain a first marking result, and includes the following steps. The encoding end obtains a first bounding box in which the first face patch is located. The encoding end obtains a first region in the original mesh to be encoded corresponding to the first bounding box and a second region in the matched subdivided deformed mesh corresponding to the first bounding box based on the first bounding box. The encoding end calculates a first weighted average normal vector of the face patches in the first region and a second weighted average normal vector of the face patches in the second region. In a case where an included angle between the first weighted average normal vector and the second weighted average normal vector is greater than a preset included angle threshold, the encoding end determines that the marking result of the first face patch is that the first face patch is marked as an unmatched face patch, to obtain the marking result of the first face patch, or in a case where the included angle between the first weighted average normal vector and the second weighted average normal vector is less than or equal to the preset included angle threshold, the encoding end marks the first face patch as a matched face patch, to obtain the marking result of the first face patch.

22. The method of any one of claims 19-21, wherein, The encoding end determines the at least one inter-frame patch in the registration base mesh based on the first marking result, and includes the following steps. The encoding end obtains at least one third region in which the unmatched face patch is located based on the first marking result. The encoding end re-labels the patches in a first region in the at least one third region, in which the total area of the patches is less than or equal to a preset first area threshold, as matched patches to obtain a third labeling result; The encoding end obtains at least one fourth region in which the patches labeled as matched patches are located based on the third labeling result; The encoding end re-labels the patches in a second region in the at least one fourth region, in which the total area of the patches is less than or equal to the preset first area threshold, as unmatched patches to obtain a fourth labeling result; The encoding end determines the region in which the patches labeled as matched patches are located as the at least one inter-frame patch based on the fourth labeling result.

23. The method of any one of claims 19-22, wherein, The encoding end determines the at least one intra-frame patch, comprising: The encoding end labels the patches in the original mesh to be encoded as matched or unmatched patches based on the first labeling result to obtain a second labeling result; The encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result; The encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch.

24. The method of claim 23, wherein, The second labeling result comprises a labeling result of a second patch in the original mesh to be encoded; The encoding end labels the patches in the original mesh to be encoded as matched or unmatched patches based on the labeling result to obtain a second labeling result, comprising: The encoding end obtains the barycenter of the second patch; The encoding end searches for a nearest neighbor point of the barycenter of the second patch in the registration base mesh; The encoding end labels the second patch as a matched or unmatched patch based on the label of the patch in which the nearest neighbor point is located to obtain a labeling result of the second patch.

25. The method of claim 23 or 24, wherein, The encoding end determines at least one initial intra-frame patch in the original mesh to be encoded based on the second labeling result, comprising: The encoding end obtains at least one fifth region in which the patches labeled as unmatched patches are located based on the second labeling result; The encoding end re-labels the patches in a third region in the at least one fifth region, in which the total area of the patches is less than or equal to a preset second area threshold, as matched patches to obtain a fifth labeling result; The encoding end obtains at least one sixth region in which the patches labeled as matched patches are located based on the fifth labeling result; The encoding end re-labels the patches in a fourth region in the at least one sixth region, in which the total area of the patches is less than or equal to the preset second area threshold, as unmatched patches to obtain a sixth labeling result; The encoding end determines the region in which the patches labeled as unmatched patches are located as the at least one initial intra-frame patch based on the sixth labeling result.

26. The method of any one of claims 23-25, wherein, The encoding end performs boundary adjustment on the at least one initial intra-frame patch based on the at least one inter-frame patch to obtain the at least one intra-frame patch, comprising: The encoding end obtains at least one edge in the registration base mesh in which there are adjacent patches labeled as matched and unmatched; The encoding end obtains at least one vertex in the original mesh to be encoded in which there are adjacent patches labeled as matched and unmatched; The encoding end searches for a nearest neighbor of the first vertex among the at least one vertex corresponding to the at least one edge corresponding to the first vertex; The encoding end morphs the nearest neighbor of the first vertex to the first vertex; The encoding end searches for a nearest neighbor of a second vertex among the at least one boundary, the second vertex being other than the nearest neighbor of the first vertex among the at least one vertex; The encoding end obtains an edge on which the nearest neighbor of the second vertex is located, based on the nearest neighbor of the second vertex; The encoding end determines a position on the edge on which the nearest neighbor of the second vertex is located, the position being closest to the second vertex; The encoding end morphs the second vertex to the position closest to the second vertex, to obtain the at least one intra-frame patch.

27. The method of any one of claims 10 to 26, wherein, The method further comprises: The encoding end obtains a reconstructed base mesh of the original mesh to be encoded; The encoding end performs a subdivision morphing on the reconstructed base mesh based on the original mesh to be encoded, to obtain a subdivision mesh of the original mesh to be encoded and a subdivision morphed mesh of the original mesh to be encoded; The encoding end determines displacement information of the original mesh to be encoded based on the subdivision mesh of the original mesh to be encoded and the subdivision morphed mesh of the original mesh to be encoded; The encoding end encodes the displacement information to obtain a displacement code stream.

28. The method of claim 27, wherein, The method further comprises: The encoding end performs a morphing on the subdivision mesh of the original mesh to be encoded based on reconstructed displacement information of the displacement information, to obtain a reconstructed morphed mesh of the original mesh to be encoded.

29. A slice-based trellis decoding apparatus, characterized by comprising: Comprise: An acquisition unit, configured to acquire at least one inter-frame code stream of a base mesh to be decoded and at least one intra-frame code stream of the base mesh to be decoded; A first decoding unit, configured to decode the at least one inter-frame code stream in an inter-frame decoding mode, to obtain at least one inter-frame patch; A second decoding unit, configured to decode the at least one intra-frame code stream in an intra-frame decoding mode, to obtain at least one intra-frame patch; A merging unit, configured to merge the at least one inter-frame patch and the at least one intra-frame patch, to obtain a reconstructed base mesh of the base mesh to be decoded.

30. The apparatus of claim 29, wherein, The acquisition unit is specifically configured to: In a case where a decoding mode used by the base mesh to be decoded is not an intra-frame decoding mode, acquire the at least one inter-frame code stream and the at least one intra-frame code stream.

31. A slice-based trellis encoding apparatus, characterized by comprising: Comprise: A determination unit, configured to determine at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded; A first encoding unit, configured to encode the at least one inter-frame patch in an inter-frame encoding mode, to obtain at least one inter-frame code stream; A second encoding unit, configured to encode the at least one intra-frame patch in an intra-frame encoding mode, to obtain at least one intra-frame code stream; A merging unit, configured to merge the at least one inter-frame code stream and the at least one intra-frame code stream, to obtain a base mesh code stream.

32. The apparatus of claim 31, wherein, The determination unit is specifically configured to: In a case where an encoding mode used by the original mesh to be encoded is not an intra-frame encoding mode, determine at least one inter-frame patch of an original mesh to be encoded and at least one intra-frame patch of the original mesh to be encoded.

33. An electronic device, comprising: A chip comprising a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the tile-based mesh decoding method according to any one of claims 1-9 or the steps of the tile-based mesh encoding method according to any one of claims 10-28.

34. A readable storage medium, characterized by A readable storage medium storing a program or instructions, the program or instructions, when executed by a processor, implementing the steps of the tile-based mesh decoding method according to any one of claims 1-9 or the steps of the tile-based mesh encoding method according to any one of claims 10-28.

35. A chip, comprising: A chip comprising a processor and a communication interface, the communication interface coupled to the processor, the processor configured to execute a program or instructions, implementing the steps of the tile-based mesh decoding method according to any one of claims 1-9 or the steps of the tile-based mesh encoding method according to any one of claims 10-28.

Images