A coding method, device, encoder, decoder and storage medium

Abstract

The application provides a coding method and device, an encoder, a decoder and a storage medium. For the application scenario of visual media content including one or more expression formats, isomorphic blocks of different expression formats are spliced into a heterogeneous mixed splicing graph, isomorphic blocks of the same expression format are spliced into an isomorphic splicing graph, the obtained splicing graph and splicing graph information are written into a code stream, and for the heterogeneous mixed splicing graph, isomorphic blocks of different expression formats are allowed to correspond to different information. In this way, mixed coding of data of different expression formats can reduce the number of called encoders and decoders, reduce the implementation cost, and improve the ease of use. Moreover, for the heterogeneous mixed splicing graph, some high-level parameters of different expression format blocks can not be equal, so that the heterogeneous data provides more suitable high-level parameters, which can effectively improve the coding efficiency, that is, reduce the code rate or improve the quality of reconstructed multi-view video or point cloud video.

Description

[0001] Cross-references to related applications

[0002] This application is based on and claims priority to an earlier PCT international application with application number PCT / CN2022 / 125525, application date October 14, 2022, entitled "An Encoding / Decoding Method, Apparatus, Encoder, Decoder and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of image processing technology, and in particular to an encoding / decoding method, apparatus, encoder, decoder and storage medium. Background Technology

[0004] In 3D application scenarios, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), visual media objects with different representation formats may appear in the same scene. For example, in the same 3D scene, the background and some characters and objects may be represented by video, while other characters may be represented by 3D point clouds or 3D meshes.

[0005] Using multi-view video coding, point cloud coding, and mesh coding respectively during compression encoding will preserve the effective information of the original expression format better than projecting all data into multi-view video coding, improve the quality of the rendered viewing window, and improve the overall efficiency of bitrate-quality.

[0006] However, current encoding and decoding technologies require the use of a large number of codecs for multi-view video, point cloud encoding, and grid-based encoding and decoding, resulting in high encoding and decoding costs. Summary of the Invention

[0007] This application provides an encoding / decoding method, apparatus, encoder, decoder, and storage medium.

[0008] In a first aspect, this application provides a decoding method, comprising: decoding a bitstream to obtain a spliced ​​image and spliced ​​image information; when the spliced ​​image is a heterogeneous hybrid spliced ​​image, obtaining at least two isomorphic blocks and isomorphic block information based on the spliced ​​image and the spliced ​​image information; wherein, different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information; when the spliced ​​image is a isomorphic spliced ​​image, obtaining one isomorphic block and isomorphic block information based on the spliced ​​image and the spliced ​​image information; and obtaining visual media content in at least two expression formats based on the isomorphic block and the isomorphic block information.

[0009] Secondly, this application provides an encoding method, comprising: processing visual media content in at least two expression formats to obtain at least two isomorphic blocks; splicing the at least two isomorphic blocks to obtain a spliced ​​image and spliced ​​image information, wherein when the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks, and different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information; encoding the spliced ​​image and spliced ​​image information to obtain a bitstream.

[0010] Thirdly, this application provides a decoding device, comprising:

[0011] The decoding unit is configured to decode the bitstream to obtain the spliced ​​image and spliced ​​image information;

[0012] The splitting unit, when the stitched image is a heterogeneous hybrid stitched image, obtains at least two types of isomorphic blocks and isomorphic block information based on the stitched image and the stitched image information; wherein, different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information;

[0013] The splitting unit, when the stitched image is a homogeneous stitched image, obtains a homogeneous block and homogeneous block information based on the stitched image and the stitched image information;

[0014] The processing unit is configured to obtain visual media content in at least two expression formats based on the isomorphic blocks and the isomorphic block information.

[0015] Fourthly, this application provides an encoding device for use in an encoder, comprising:

[0016] The processing unit is configured to process visual media content in at least two expression formats to obtain at least two isomorphic blocks;

[0017] The splicing unit is configured to splice the at least two isomorphic blocks to obtain a spliced ​​image and spliced ​​image information. When the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks, and different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information.

[0018] The encoding unit is configured to encode the spliced ​​image and spliced ​​image information to obtain a bitstream.

[0019] Fifthly, a decoder is provided, including a first memory and a first processor; the first memory stores a computer program executable on the first processor to perform the methods described in the first aspect or its various implementations.

[0020] In a sixth aspect, an encoder is provided, including a second memory and a second processor; the second memory stores a computer program executable on the second processor to perform the methods of the second aspect or its implementations described above.

[0021] In a seventh aspect, an encoding / decoding system is provided, including an encoder and a decoder. The encoder is used to execute the methods in the second aspect or its implementations described above, and the decoder is used to execute the methods in the first aspect or its implementations described above.

[0022] Eighthly, a chip is provided for implementing the methods of any one of the first to second aspects or their respective implementations. Specifically, the chip includes a processor for calling and running a computer program from a memory, causing a device on which the chip is mounted to perform the methods of any one of the first to second aspects or their respective implementations.

[0023] Ninthly, a computer-readable storage medium is provided for storing a computer program that causes a computer to perform the methods of any one of the first to second aspects or their respective implementations.

[0024] In a tenth aspect, a computer program product is provided, including computer program instructions that cause a computer to perform the methods of any one of the first to second aspects or their respective implementations.

[0025] Eleventhly, a computer program is provided that, when run on a computer, causes the computer to perform the methods of any one of the first to second aspects or their respective implementations.

[0026] In the twelfth aspect, a bitstream is provided, which is generated based on the encoding method described in the second aspect above.

[0027] Based on the above technical solutions, for application scenarios involving visual media content with one or more expression formats, homogeneous blocks of different expression formats are stitched together into a heterogeneous hybrid stitched image. This mixed encoding of data from different expression formats reduces the number of encoders and decoders required, lowers implementation costs, and improves usability. Furthermore, in the heterogeneous hybrid stitched image, certain high-level parameters of blocks from different expression formats can be unequal, allowing the heterogeneous data to provide more suitable high-level parameters, effectively improving encoding efficiency—that is, reducing bitrate or improving the quality of reconstructed multi-view video or point cloud video. Attached Figure Description

[0028] Figure 1 This is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application;

[0029] Figure 2A This is a schematic block diagram of a video encoder involved in an embodiment of this application;

[0030] Figure 2B This is a schematic block diagram of a video decoder involved in an embodiment of this application;

[0031] Figure 3A This is a framework diagram for organizing and representing multi-view video data;

[0032] Figure 3B This is a schematic diagram illustrating the generation of stitched images from multi-view video data;

[0033] Figure 3C This is a diagram illustrating the organization and representation framework of point cloud data;

[0034] Figures 3D to 3F Schematic diagrams for different types of point cloud data;

[0035] Figure 4 This is a schematic diagram of multi-view video encoding.

[0036] Figure 5 This is a schematic diagram of multi-view video decoding;

[0037] Figure 6 This is a schematic diagram of an encoding method provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of a heterogeneous hybrid splicing diagram provided in an embodiment of this application;

[0039] Figure 8 This is a schematic diagram of an isomorphic splicing diagram provided in an embodiment of this application;

[0040] Figure 9 A schematic flowchart illustrating a decoding method provided in an embodiment of this application;

[0041] Figure 10 A schematic diagram of the V3C bitstream structure provided in the embodiments of this application;

[0042] Figure 11 A schematic block diagram of an encoding device provided in an embodiment of this application;

[0043] Figure 12 A schematic block diagram of a decoding apparatus provided in an embodiment of this application;

[0044] Figure 13 A schematic block diagram of an encoder provided in an embodiment of this application;

[0045] Figure 14 A schematic block diagram of a decoder provided in an embodiment of this application;

[0046] Figure 15 This is a schematic diagram of the composition structure of an encoding / decoding system provided in an embodiment of this application. Detailed Implementation

[0047] This application can be applied to the fields of image encoding and decoding, video encoding and decoding, hardware video encoding and decoding, dedicated circuit video encoding and decoding, and real-time video encoding and decoding. For example, the solution of this application can be combined with audio video coding standards (AVS), such as H.264 / Audio Video Coding (AVC) standard, H.265 / High Efficiency Video Coding (HEVC) standard, and H.266 / Versatile Video Coding (VVC) standard. Alternatively, the solution of this application can be combined with other proprietary or industry standards, including ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), which include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. It should be understood that the technology in this application is not limited to any particular codec standard or technology.

[0048] The high-degree-of-freedom immersive coding system can be roughly divided into the following stages according to the task line: data acquisition, data organization and expression, data encoding and compression, data decoding and reconstruction, data synthesis and rendering, and finally presenting the target data to the user.

[0049] The encoding involved in the embodiments of this application is mainly video encoding and decoding. For ease of understanding, it will first be combined with... Figure 1 The video encoding and decoding system involved in the embodiments of this application will be described.

[0050] Figure 1 This is a schematic block diagram of a video encoding and decoding system according to an embodiment of this application. It should be noted that... Figure 1 This is merely an example; the video encoding and decoding system in this application includes, but is not limited to, [other systems]. Figure 1 As shown. Figure 1 As shown, the video encoding / decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device encodes (can be understood as compressing) video data to generate a bitstream and transmits the bitstream to the decoding device. The decoding device decodes the bitstream generated by the encoding device to obtain the decoded video data.

[0051] The encoding device 110 in this application embodiment can be understood as a device with video encoding function, and the decoding device 120 can be understood as a device with video decoding function. That is, the encoding device 110 and the decoding device 120 in this application embodiment include a wider range of devices, such as smartphones, desktop computers, mobile computing devices, laptops (e.g., laptop computers), tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, etc.

[0052] In some embodiments, encoding device 110 may transmit encoded video data (such as a bitstream) to decoding device 120 via channel 130. Channel 130 may include one or more media and / or means capable of transmitting encoded video data from encoding device 110 to decoding device 120.

[0053] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. In this example, encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum; optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0054] In another example, channel 130 includes a storage medium that can store video data encoded by encoding device 110. The storage medium includes various local access data storage media, such as optical discs, DVDs, flash memory, etc. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.

[0055] In another example, channel 130 may include a storage server that can store the video data encoded by encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Optionally, the storage server can store and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0056] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0057] In some embodiments, the encoding device 110 may include a video source 111 in addition to the video encoder 112 and the input interface 113.

[0058] The video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0059] Video encoder 112 encodes video data from video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the encoding information of the pictures or picture sequences in the form of a bitstream. The encoding information may include encoded image data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order within the bitstream.

[0060] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data can also be stored on a storage medium or a storage server for subsequent retrieval by the decoding device 120.

[0061] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122. In some embodiments, in addition to the input interface 121 and the video decoder 122, the decoding device 120 may also include a display device 123.

[0062] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive encoded video data through channel 130.

[0063] The video decoder 122 is used to decode the encoded video data to obtain the decoded video data, and transmit the decoded video data to the display device 123.

[0064] Display device 123 displays the decoded video data. Display device 123 may be integrated with decoding device 120 or external to decoding device 120. Display device 123 may include various display devices, such as liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices.

[0065] also, Figure 1 This is merely an example; the technical solutions in the embodiments of this application are not limited to... Figure 1 For example, the technology of this application can also be applied to one-sided video encoding or one-sided video decoding.

[0066] The video coding framework involved in the embodiments of this application is described below.

[0067] Figure 2A This is a schematic block diagram of a video encoder according to an embodiment of this application. It should be understood that the video encoder 200 can be used for lossy compression of images or lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0068] This video encoder 200 can be applied to image data in luminance / chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents luminance (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) used to describe color and saturation. For example, in color format, 4:2:0 means that there are 4 luminance components and 2 chrominance components (YYYYCbCr) per 4 pixels; 4:2:2 means that there are 4 luminance components and 4 chrominance components (YYYYCbCrCbCr) per 4 pixels; and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0069] For example, the video encoder 200 reads video data and divides each frame into several coding tree units (CTUs). In some examples, CTUs may be called "tree blocks," "largest coding unit" (LCU), or "coding treeblock" (CTB). Each CTU can be associated with a pixel block of equal size within the image. Each pixel can correspond to one luminance (luma) sample and two chrominance (chroma) samples. Therefore, each CTU can be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into several coding units (CUs) for encoding. CUs can be rectangular or square blocks. The CU can be further divided into prediction units (PUs) and transform units (TUs), thus separating encoding, prediction, and transformation for more flexible processing. In one example, the CTU is divided into CUs using a quadtree structure, and the CUs are further divided into TUs and PUs using a quadtree structure.

[0070] The video encoder and decoder support various PU sizes. Assuming a specific CU size of 2N×2N, the video encoder and decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and also support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0071] In some embodiments, such as Figure 2A As shown, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may contain more, fewer, or different functional components.

[0072] Optionally, in this application, the current block can be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block can also be referred to as the predicted image block or the image prediction block, and the reconstructed image block can also be referred to as the reconstruction block or the image reconstruction block.

[0073] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame estimation unit 212. Because there is a strong correlation between adjacent pixels in a frame of a video, intra-frame prediction is used in video encoding and decoding techniques to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video encoding and decoding techniques to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0074] Inter-frame prediction unit 211 can be used for inter-frame prediction, which can include motion estimation and motion compensation. It can reference image information from different frames. Inter-frame prediction uses motion information to find a reference block from the reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames, where P-frames refer to forward prediction frames and B-frames refer to bidirectional prediction frames. The motion information includes a list of reference frames, the reference frame index, and motion vectors. Motion vectors can be integer-pixel or fractional-pixel. If the motion vector is fractional-pixel, interpolation filtering needs to be used in the reference frame to create the required fractional-pixel blocks. Here, the integer-pixel or fractional-pixel blocks in the reference frame found based on the motion vectors are called reference blocks. Some techniques directly use the reference block as the prediction block, while others process the reference block further to generate the prediction block. Processing the reference block further to generate the prediction block can also be understood as using the reference block as the prediction block and then processing it to generate a new prediction block.

[0075] Intra-frame estimation unit 212 refers only to information from the same frame image to predict pixel information within the current code image block, thereby eliminating spatial redundancy. The frame used for intra-frame prediction can be an I-frame.

[0076] Intra-frame prediction has multiple prediction modes. Taking the international digital video coding standards H-series as an example, the H.264 / AVC standard has 8 angular prediction modes and 1 non-angular prediction mode, while H.265 / HEVC extends this to 33 angular prediction modes and 2 non-angular prediction modes. HEVC uses Planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. VVC uses Planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0077] It should be noted that with the increase in angle modes, intra-frame prediction will be more accurate and better meet the needs of the development of high-definition and ultra-high-definition digital video.

[0078] The residual unit 220 can generate a residual block of the CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, the residual unit 220 can generate a residual block of the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel block of the CU and the corresponding sample in the prediction block of the PU of the CU.

[0079] Transform / quantization unit 230 can quantize transform coefficients. Transform / quantization unit 230 can quantize transform coefficients associated with the TU of the CU based on the quantization parameter (QP) value associated with the CU. Video encoder 200 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0080] The inverse transform / quantization unit 240 can apply inverse quantization and inverse transform to the quantized transform coefficients to reconstruct the residual block from the quantized transform coefficients.

[0081] The reconstruction unit 250 can add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by the prediction unit 210 to produce a reconstructed image block associated with the TU. By reconstructing the sampled blocks of each TU of the CU in this way, the video encoder 200 can reconstruct the pixel blocks of the CU.

[0082] The loop filtering unit 260 is used to process the pixels after inverse transformation and inverse quantization to compensate for the distortion information and provide a better reference for subsequent encoded pixels. For example, it can perform deblocking filtering to reduce the block effect of pixel blocks associated with the CU.

[0083] In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit and a sample adaptive compensation / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove block effects and the SAO / ALF unit is used to remove ringing effects.

[0084] The decoded image buffer 270 can store reconstructed pixel blocks. The inter-frame prediction unit 211 can use a reference image containing the reconstructed pixel blocks to perform inter-frame prediction on PUs of other images. In addition, the intra-frame estimation unit 212 can use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra-frame prediction on other PUs in the same image as the CU.

[0085] Entropy coding unit 280 can receive quantized transform coefficients from transform / quantization unit 230. Entropy coding unit 280 can perform one or more entropy coding operations on the quantized transform coefficients to produce entropy-coded data.

[0086] Figure 2B This is a schematic block diagram of a video decoder involved in an embodiment of this application.

[0087] like Figure 2B As shown, the video decoder 300 includes: an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may contain more, fewer, or different functional components.

[0088] The video decoder 300 can receive a bitstream. The entropy decoding unit 310 can parse the bitstream to extract syntax elements from it. As part of parsing the bitstream, the entropy decoding unit 310 can parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the dequantization / transform unit 330, the reconstruction unit 340, and the loop filtering unit 350 can decode the video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0089] In some embodiments, the prediction unit 320 includes an inter-frame prediction unit 321 and an intra-frame estimation unit 322.

[0090] Intra-frame estimation unit 322 can perform intra-frame prediction to generate prediction blocks for the PU. Intra-frame estimation unit 322 can use an intra-frame prediction mode to generate prediction blocks for the PU based on pixel blocks of spatially adjacent PUs. Intra-frame estimation unit 322 can also determine the intra-frame prediction mode of the PU based on one or more syntax elements parsed from the bitstream.

[0091] Inter-frame prediction unit 321 can construct a first reference image list (list 0) and a second reference image list (list 1) based on the syntax elements parsed from the bitstream. Furthermore, if the PU uses inter-frame prediction coding, the entropy decoding unit 310 can parse the motion information of the PU. Inter-frame prediction unit 321 can determine one or more reference blocks of the PU based on the motion information of the PU. Inter-frame prediction unit 321 can generate prediction blocks for the PU based on one or more reference blocks of the PU.

[0092] The dequantization / transformation unit 330 reversibly quantizes (i.e., dequantizes) the transform coefficients associated with the TU. The dequantization / transformation unit 330 can use the QP value associated with the CU of the TU to determine the degree of quantization.

[0093] After the inverse quantization transform coefficients, the inverse quantization / transformation unit 330 can apply one or more inverse transforms to the inverse quantization transform coefficients to generate a residual block associated with the TU.

[0094] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add the sample of the residual block to the corresponding sample of the prediction block to reconstruct the pixel block of the CU, thereby obtaining the reconstructed image block.

[0095] The loop filter unit 350 can perform deblocking filtering operations to reduce the block effect of pixel blocks associated with the CU.

[0096] The video decoder 300 can store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 can use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0097] The basic process of video encoding and decoding is as follows: At the encoding end, a frame image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 can calculate a residual block based on the prediction block and the original block of the current block, that is, the difference between the prediction block and the original block of the current block. This residual block can also be called residual information. This residual block is transformed and quantized by the transform / quantization unit 230, which can remove information that is not sensitive to the human eye to eliminate visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 can be called a temporal residual block, and the temporal residual block after transformation and quantization by the transform / quantization unit 230 can be called a frequency residual block or a frequency domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230, and can perform entropy coding on the quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and the probability information of the binary bitstream.

[0098] At the decoding end, the entropy decoding unit 310 can parse the bitstream to obtain the prediction information and quantization coefficient matrix of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction to generate the prediction block of the current block based on the prediction information. The dequantization / transform unit 330 uses the quantization coefficient matrix obtained from the bitstream to perform dequantization and inverse transform on the quantization coefficient matrix to obtain the residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain the reconstructed block. The reconstructed blocks form the reconstructed image. The loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or based on the blocks to obtain the decoded image. The encoding end also needs similar operations to the decoding end to obtain the decoded image. This decoded image can also be called the reconstructed image, which can be used as a reference frame for inter-frame prediction in subsequent frames.

[0099] It should be noted that the block partitioning information determined at the encoding end, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, are carried in the bitstream when necessary. The decoding end determines the same block partitioning information, prediction, transform, quantization, entropy coding, and loop filtering mode information or parameter information as the encoding end by parsing the bitstream and analyzing existing information, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end.

[0100] The above describes the basic flow of a video codec under a block-based hybrid coding framework. With the development of technology, some modules or steps of this framework or flow may be optimized. This application is applicable to the basic flow of a video codec under this block-based hybrid coding framework, but is not limited to this framework and flow.

[0101] In some application scenarios, multiple heterogeneous contents may appear simultaneously in the same 3D scene, such as multi-view video and point clouds. For this situation, current encoding and decoding methods include at least the following two:

[0102] Method 1: For multi-view video, MPEG (Moving Picture Experts Group) immersive video (MIV) technology is used for encoding and decoding; for point clouds, video based point cloud compression (VPCC) technology is used for encoding and decoding.

[0103] The following section introduces MIV and VPCC technologies.

[0104] MIV technology: To reduce the transmission pixel rate while preserving as much scene information as possible, so as to ensure that there is enough information for rendering the target view, MPEG-I adopts the following scheme: Figure 3A As shown, a limited number of viewpoints are selected as the base viewpoints to represent the visible range of the scene as much as possible. The base viewpoints are used for complete image transmission. Redundant pixels between the remaining non-base viewpoints and the base viewpoints are removed, that is, only the effective information that is not repeatedly expressed is retained. The effective information is then extracted into sub-block images and reorganized with the base viewpoint images to form a larger rectangular image. This rectangular image is called the stitched image. Figure 3A and Figure 3B The process of generating the stitched image is illustrated. The stitched image is fed into the encoder and decoder for compression and reconstruction, and auxiliary data related to the stitching information of the sub-block images are also fed into the encoder to form a bitstream.

[0105] The VPCC encoding method projects point clouds into two-dimensional images or videos, converting three-dimensional information into two-dimensional information for encoding. Figure 3C This is a block diagram of VPCC encoding. The bitstream is roughly divided into four parts: the geometric bitstream, generated by geometric depth map encoding, which represents the geometric information of the point cloud; the attribute bitstream, generated by texture map encoding, which represents the attribute information of the point cloud; and the occupancy bitstream, generated by occupancy map encoding, which indicates the valid regions in the depth and texture maps. All three types of video are encoded and decoded using a video encoder, such as... Figures 3D to 3FAs shown. The auxiliary information bitstream is the bitstream generated by encoding the auxiliary information of the sub-block image, that is, the part related to the patch data unit in the V3C standard, which indicates information such as the position and size of each sub-block image.

[0106] Method 2: Both multi-view video and point cloud are encoded and decoded using frame packing technology in Visual Volumetric Video-based Coding (V3C).

[0107] The following section introduces frame packing technology.

[0108] Taking multi-view video as an example, for instance, such as Figure 4 As shown, the encoding end includes the following steps:

[0109] Step 1: When encoding the acquired multi-view video, some preprocessing is performed to generate multi-view video patches. Then, the multi-view video patches are organized to generate a multi-view video stitching image.

[0110] For example, Figure 4 As shown, multi-view video is input into TIMV for packaging, and a multi-view video stitched image is output. TIMV is a reference software for MIV. Packaging in this embodiment can be understood as stitching.

[0111] Among them, the multi-view video mosaic includes multi-view video texture mosaic and multi-view video geometric mosaic, which only contains multi-view video sub-blocks.

[0112] Step 2: Input the multi-view video stitching image into the frame packer and output the multi-view video composite stitching image.

[0113] Among them, multi-view video hybrid stitching includes multi-view video texture hybrid stitching, multi-view video geometry hybrid stitching, and multi-view video texture and geometry hybrid stitching.

[0114] Specifically, such as Figure 4 As shown, the multi-view video mosaic is frame-packed to generate a multi-view video composite mosaic, with each multi-view video mosaic occupying one region. Accordingly, a flag `pin_region_type_id_minus2` is transmitted for each region in the bitstream. This flag records whether the current region belongs to a multi-view video texture mosaic or a multi-view video geometry mosaic, and this information is needed at the decoding end.

[0115] Step 3: Use a video encoder to encode the multi-view video composite image to obtain the bitstream.

[0116] For example, such as Figure 5 As shown, the decoding process includes the following steps:

[0117] Step 1: During multi-view video decoding, the acquired bitstream is input into the video decoder for decoding to obtain the reconstructed multi-view video composite image.

[0118] Step 2: Input the reconstructed multi-view video stitching image into the frame unpacker and output the reconstructed multi-view video stitching image.

[0119] Specifically, first, the flag pin_region_type_id_minus2 is obtained from the bitstream. If it is determined that pin_region_type_id_minus2 is V3C_AVD, it means that the current region is a multi-view video texture stitching map. Then, the current region is split and output as a reconstructed multi-view video texture stitching map.

[0120] If pin_region_type_id_minus2 is determined to be V3C_GVD, it indicates that the current region is a multi-view video geometry mosaic. The current region is then split and output as a reconstructed multi-view video geometry mosaic.

[0121] Step 3: Decode the reconstructed multi-view video stitch image to obtain the reconstructed multi-view video.

[0122] Specifically, the multi-view video texture mosaic and multi-view video geometric mosaic are decoded to obtain the reconstructed multi-view video.

[0123] The above uses multi-view video as an example to analyze and introduce frame packing technology. The framepacking encoding and decoding method for point clouds is basically the same as that for multi-view video, and can be referred to accordingly. For example, TMC (a reference software for VPCC) is used to pack the point cloud to obtain a point cloud mosaic. The point cloud mosaic is then input into a frame packer for frame packing to obtain a point cloud composite mosaic. The point cloud composite mosaic is then stitched together to obtain the point cloud bitstream. This will not be elaborated further here.

[0124] Currently, when multiple visual media content in different formats appear simultaneously in the same 3D scene, each format is encoded and decoded separately. For example, when point clouds and multi-view video appear simultaneously in the same 3D scene, current packaging technology compresses the point cloud to form a compressed point cloud bitstream (one type of V3C bitstream), compresses the multi-view video information to obtain a compressed multi-view video bitstream (another type of V3C bitstream), and then the system layer multiplexes the compressed bitstreams to obtain a fused 3D scene multiplexed bitstream. During decoding, the compressed point cloud bitstream and the compressed multi-view video bitstream are decoded separately. However, existing technologies use multiple codecs and incur high encoding and decoding costs when encoding and decoding multiple visual media content in different formats.

[0125] To address the aforementioned technical challenges, co-encoding data in different representation formats can reduce the number of encoders and decoders required, lower implementation costs, and improve usability. Furthermore, in heterogeneous hybrid mosaic images, certain high-level parameters of blocks from different representation formats can be unequal, allowing heterogeneous data to provide more suitable high-level parameters. This effectively improves encoding efficiency, reducing bitrate or enhancing the quality of reconstructed multi-view video or point cloud video.

[0126] The following is combined Figure 6 Taking the encoding end as an example, the video encoding method provided in the embodiments of this application will be introduced.

[0127] Figure 6 A flowchart illustrating the encoding method provided in the embodiments of this application is shown below. Figure 6 As shown, the encoding method includes:

[0128] Step 601: Process visual media content in at least two expression formats to obtain at least two isomorphic blocks;

[0129] In 3D application scenarios, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), visual media objects with different expression formats may appear in the same scene. For example, in the same 3D scene, the background and some people and objects may be expressed by video, while another part of the people may be expressed by 3D point cloud or 3D mesh.

[0130] In some embodiments, the visual media content includes visual media content in at least two expression formats, such as multi-view video, point cloud, and mesh. Multi-view video may include multiple viewpoint videos and / or a single viewpoint video. One isomorphic block corresponds to one expression format. Different isomorphic blocks correspond to different expression formats. For example, the expression formats corresponding to at least two isomorphic blocks include at least the following: multi-view video, point cloud, and mesh.

[0131] It should be noted that, in the embodiments of this application, each isomorphic block may include at least one isomorphic block having the same expression format. For example, a point cloud format isomorphic block includes one or more point cloud blocks, a multi-view video format isomorphic region includes one or more multi-view video blocks, and a grid format isomorphic block includes one or more grid blocks.

[0132] Specifically, the visual media content in the first expression format is processed to obtain isomorphic blocks in the first expression format; the visual media content in the second expression format is processed to obtain isomorphic blocks in the second expression format. The first expression format is one of multi-view video, point cloud, and mesh; the second expression format is one of multi-view video, point cloud, and mesh; and the first and second expression formats are different expression formats.

[0133] It should be noted that a block can be a mosaic with a specific shape, such as a mosaic of rectangular areas with a specific length and / or height. For example, a block includes at least one sub-tile, which is then stitched together in an ordered manner, such as according to the area of ​​the sub-tiles from largest to smallest, or according to the length and / or height of the sub-tiles from largest to smallest, to obtain the block corresponding to the visual media content. Optionally, a block can be precisely mapped to a tile in a mosaic (atlas). In practical applications, a block can also be called a tile; that is, a point cloud block can also be called a point cloud tile, a multi-view video block can also be called a multi-view video tile, and a grid block can also be called a grid tile.

[0134] In some embodiments, each sub-tile within a block may have a patch ID to distinguish different sub-tiles within the same block. For example, the same block may include sub-tile 1 (patch1), sub-tile 2 (patch2), and sub-tile 3 (patch3).

[0135] Furthermore, a isomorphic block refers to a block in which each sub-tile has the same representation format. For example, all sub-tiles in a isomorphic block may be multi-view video sub-tiles or point cloud sub-tiles, etc., with the same representation format. The representation format corresponding to each sub-tile in a isomorphic block is the representation format corresponding to the isomorphic block itself.

[0136] In some embodiments, isomorphic blocks may have a tile ID to distinguish different blocks with the same representation format. For example, a point cloud block may include point cloud block 1 or point cloud block 2. For example, multiple visual media contents include point clouds and multi-view video. The point cloud is processed to obtain a point cloud block, where point cloud block 1 includes point cloud sub-tiles 1 to 3; the multi-view video is processed to obtain a multi-view video block, where multi-view video sub-tiles 1 to 4 are included.

[0137] When processing visual media content in one expression format, a homogeneous block of that expression format is obtained. When processing at least two visual media contents, at least two homogeneous blocks of expression formats are obtained. To improve compression efficiency, embodiments of this application process these at least two visual media contents, such as by packaging (also known as stitching), to obtain a block corresponding to each of the at least two visual media contents. For example, blocks can be obtained by stitching together sub-patches corresponding to at least two visual media contents. It should be noted that embodiments of this application process at least two visual media contents separately, and the method of obtaining blocks is not limited.

[0138] In one possible implementation, the visual media content includes visual media content in two expression formats: multi-view video and point cloud. The processing of the visual media content in at least two expression formats to obtain at least two isomorphic blocks includes: projecting and deduplicating the acquired multi-view video, connecting non-repeating pixels to form multi-view video sub-blocks, and stitching the multi-view video sub-blocks into a multi-view video block; and performing parallel projection on the acquired point cloud, forming point cloud sub-blocks from connected points in the projection plane, and stitching the point cloud sub-blocks into a point cloud block.

[0139] Specifically, for multi-view video, taking MPEG-I as an example, a limited number of viewpoints are selected as the basic viewpoints and the visible range of the scene is expressed as much as possible. The basic viewpoints are transmitted as complete images. Redundant pixels between the remaining non-basic viewpoints and the basic viewpoints are removed, that is, only the effective information that is not repeated is retained. The effective information is then extracted into sub-block images and reorganized with the basic viewpoint images to form a larger strip-shaped image. This strip-shaped image is called a multi-view video block.

[0140] In some embodiments, the aforementioned visual media content is media content presented simultaneously in the same three-dimensional space. In some embodiments, the aforementioned visual media content is media content presented at different times in the same three-dimensional space. In some embodiments, the aforementioned visual media content can also be media content in different three-dimensional spaces. That is, in the embodiments of this application, there is no specific limitation on the aforementioned at least two visual media contents.

[0141] Step 602: The at least two isomorphic blocks are spliced ​​together to obtain a spliced ​​image and spliced ​​image information. When the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks. Different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information.

[0142] When the stitched image is a homogeneous stitched image, the stitched image includes a homogeneous block, and a homogeneous block corresponds to a visual media content expression format.

[0143] Specifically, isomorphic splicing is performed on isomorphic blocks of at least two expression formats to generate a heterogeneous hybrid spliced ​​image and spliced ​​image information; isomorphic splicing is performed on isomorphic blocks of the same expression format to generate an isomorphic spliced ​​image and spliced ​​image information. The heterogeneous hybrid spliced ​​image is composed of isomorphic blocks of at least two expression formats, while the isomorphic spliced ​​image is composed of isomorphic blocks of one expression format.

[0144] For example, isomorphic splicing of isomorphic blocks of a first expression format yields a first isomorphic spliced ​​image and spliced ​​image information; isomorphic splicing of isomorphic blocks of a second expression format yields a second isomorphic spliced ​​image and spliced ​​image information. Alternatively, heteromorphic splicing of isomorphic blocks of the first and second expression formats yields a heteromorphic hybrid spliced ​​image and spliced ​​image information. Alternatively, isomorphic splicing of isomorphic blocks of the first expression format yields a first isomorphic spliced ​​image and spliced ​​image information; heteromorphic splicing of isomorphic blocks of the first and second expression formats yields a heteromorphic hybrid spliced ​​image and spliced ​​image information. Alternatively, isomorphic splicing of isomorphic blocks of the second expression format yields a second isomorphic spliced ​​image and spliced ​​image information; heteromorphic splicing of isomorphic blocks of the first and second expression formats yields a heteromorphic hybrid spliced ​​image and spliced ​​image information.

[0145] In other words, a homogeneous mosaic image can include one or more homogeneous blocks of the same expression format, while a heterogeneous hybrid mosaic image includes at least two homogeneous blocks of at least two expression formats. In the embodiments of this application, the first expression format is one of multi-view video, point cloud, and mesh; the second expression format is one of multi-view video, point cloud, and mesh; and the first and second expression formats are different expression formats. Figure 7 As shown, a heterogeneous hybrid stitched image is obtained by stitching together multi-view video block 1, multi-view video block 2, and point cloud block 1.

[0146] For example, the first representation format is multi-view video, and the second representation format is point cloud. A portion of the multi-view video blocks and a portion of the point cloud blocks are stitched together to form a heterogeneous hybrid mosaic; another portion of the multi-view video blocks are stitched together to form a multi-view mosaic; and another portion of the point cloud blocks are stitched together to form a point cloud mosaic.

[0147] The mosaic information is used to reconstruct the mosaic. For example, the mosaic information includes at least mosaic type information, mosaic information of isomorphic blocks, and isomorphic block information. In some embodiments, the mosaic information includes a first syntax element, which indicates whether the mosaic is a heterogeneous hybrid mosaic or a isomorphic mosaic. In some embodiments, the first syntax element is a syntax element of the mosaic sequence parameter set ASPS and / or a syntax element of the mosaic frame parameter set AFPS. The ASPS and / or AFPS are parsed to determine the mosaic type.

[0148] In some embodiments, the first syntax element includes a first sub-syntax element and a second sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: if the value of the first sub-syntax element and the value of the second sub-syntax element are equal, determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

[0149] In some embodiments, the first syntax element includes a first sub-syntax element and a second sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the second sub-syntax element; and determining the spliced ​​graph type when the two determination results are consistent.

[0150] In other words, the bitstream must guarantee absolute consistency between the first and second sub-syntax elements. Only when the two sub-syntax elements are consistent can the splice graph type be determined. For example, the consistency of the two sub-syntax elements can be compared first, and then the splice graph type can be determined based on the value of one of the sub-syntax elements. Alternatively, the splice graph type can be determined based on each sub-syntax element first, and absolute consistency can be guaranteed by comparing whether the splice graph types are the same.

[0151] For example, determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: if the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic; if the value is a second preset value, then the mosaic is determined to be a homogeneous mosaic. In other words, two values ​​or two types of values ​​can be set to identify heterogeneous hybrid mosaics and homogeneous mosaics. For example, the first preset value is 1, and the second preset value is 0.

[0152] For example, in some embodiments, the splicing graph information does not include the first syntax element, and the splicing graph is determined to be an isomorphic splicing graph. In some embodiments, if the splicing graph information does not include the first syntax element, it is inferred that the first syntax element has a second preset value. For example, if the splicing graph information does not include the first sub-syntax element, the splicing graph is determined to be an isomorphic splicing graph, and the first sub-syntax element has a second preset value; if the splicing graph information does not include the second sub-syntax element, the splicing graph is determined to be an isomorphic splicing graph, and the second sub-syntax element has a second preset value.

[0153] For example, determining whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image based on the value includes: if the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including homogeneous blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; if the value is a fourth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the first expression format; if the value is a fifth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the second expression format. That is, multiple values ​​can be set to identify the expression formats of heterogeneous hybrid stitched images and homogeneous stitched images, and even to identify which homogeneous blocks of expression formats are included in the heterogeneous hybrid stitched image. For example, the third preset value is 2, the fourth preset value is 1, and the fifth preset value is 0.

[0154] For example, the first sub-syntax element is a syntax element of the mosaic sequence parameter set (ASPS), and the second sub-syntax element is a syntax element of the mosaic frame parameter set (AFPS). In some embodiments, the first sub-syntax element can be a newly added syntax element in ASPS, or the first sub-syntax element can be a syntax element obtained by logically operating on at least two syntax elements in ASPS. In some embodiments, the second sub-syntax element can be a newly added syntax element in AFPS, or the second sub-syntax element can be a syntax element obtained by logically operating on at least two syntax elements in ASPS. For example, the first sub-syntax element is a syntax element obtained by ANDing two syntax elements in ASPS, and the second sub-syntax element is a syntax element obtained by ANDing two syntax elements in ASPS.

[0155] In this embodiment of the application, asps_heterogeneous_miv_extension_present_flag represents the first sub-syntax element, and afps_heterogeneous_miv_extension_present_flag represents the second sub-syntax element.

[0156] In some embodiments, the method includes: parsing a first sub-syntax element in ASPS; determining whether the concatenated graph is a heterogeneous hybrid concatenated graph or a homogeneous concatenated graph based on the value of the first sub-syntax element; parsing a second sub-syntax element in AFPS; and determining whether the concatenated graph is a heterogeneous hybrid concatenated graph or a homogeneous concatenated graph based on the value of the second sub-syntax element. That is, the first sub-syntax element is directly parsed in ASPS; when the first sub-syntax element exists, its value is analyzed to determine whether the concatenated graph is a heterogeneous hybrid concatenated graph or a homogeneous concatenated graph; when the first sub-syntax element does not exist or is 0, the concatenated graph is determined to be a homogeneous concatenated graph.

[0157] In some embodiments, the method includes: determining that the bitstream includes a bitstream corresponding to visual media content with at least two expression formats; parsing a first sub-syntax element in ASPS; determining whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image based on the value of the first sub-syntax element; parsing a second sub-syntax element in AFPS; and determining whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image based on the value of the second sub-syntax element. That is, when it is determined that the bitstream includes a bitstream corresponding to visual media content with at least two expression formats, the first sub-syntax element is parsed in ASPS. If the first sub-syntax element exists, its value is analyzed to determine whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image; if the first sub-syntax element does not exist or is 0, the stitched image is determined to be a homogeneous stitched image. Currently, when it is determined that the bitstream includes a bitstream corresponding to visual media content with only one expression format, the stitched image is determined to be a homogeneous stitched image.

[0158] For example, in some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information further includes a second syntax element; the expression format of isomorphic blocks in the mosaic is determined according to the second syntax element.

[0159] In some embodiments, the method further includes: parsing a first sub-syntax element in ASPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element; parsing a second sub-syntax element in AFPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the second sub-syntax element; when the spliced ​​graph is a heterogeneous hybrid spliced ​​graph, parsing the second syntax element in AFPS, and determining the expression format of the homogeneous blocks in the spliced ​​graph based on the second syntax element. In some embodiments, after determining that the spliced ​​graph is a heterogeneous hybrid spliced ​​graph based on the first syntax element, the second syntax element of the homogeneous blocks in the heterogeneous hybrid spliced ​​graph is further parsed to determine the type of homogeneous blocks.

[0160] Specifically, the expression format type corresponding to the i-th block in the stitched graph can be indicated by setting different values ​​for the second syntax element. For example, determining the expression format of the isomorphic block in the stitched graph based on the second syntax element includes: when the value of the second syntax element of the i-th block is a sixth preset value, the expression format of the i-th block is determined to be a first expression format; when the value of the second syntax element of the i-th block is a seventh preset value, the expression format of the i-th block is determined to be a second expression format. Taking a point cloud as the first expression format and a multi-view video as the second expression format as an example. Optionally, the sixth preset value is 0, and the seventh preset value is 1.

[0161] Furthermore, the expression format of the i-th block is a first expression format, and the i-th block is encoded using the encoding method corresponding to the first expression format; the expression format of the i-th block is a second expression format, and the i-th block is encoded using the encoding method corresponding to the second expression format.

[0162] In some embodiments, the first sub-syntax element and the second sub-syntax element are extended syntax elements for multiple viewpoints in heterogeneous hybrid stitching graphs, and the second syntax element is a shared syntax element for multiple expression formats.

[0163] Further, in some embodiments, the second syntax element includes a third sub-syntax element and a fourth sub-syntax element. ASPS parses the first sub-syntax element; based on the value of the first sub-syntax element, it determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph; AFPS parses the second sub-syntax element; based on the value of the second sub-syntax element, it determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph; AFPS parses the third sub-syntax element; based on the value of the third sub-syntax element, it determines that the spliced ​​graph is a heterogeneous hybrid spliced ​​graph; AFPS parses the fourth sub-syntax element; based on the root of the fourth sub-syntax element, it determines the expression format of the homogeneous blocks in the spliced ​​graph.

[0164] In some embodiments, ASPS parses the first sub-syntax element; determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element; ASPS parses the second and third sub-syntax elements; determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the second sub-syntax element; when the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph based on the value of the third sub-syntax element, the expression format of the homogeneous blocks in the spliced ​​graph is determined based on the root of the fourth sub-syntax element.

[0165] Specifically, different values ​​are set for the third sub-syntax element to indicate the splicing graph type, and different values ​​are set for the fourth sub-syntax element to indicate the expression format type corresponding to the i-th block in the splicing graph.

[0166] In some embodiments, the concatenation graph is determined to be an isomorphic concatenation graph when the third sub-grammar element does not exist.

[0167] For example, in some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information. The isomorphic block information includes reconstruction information of the isomorphic blocks and other supplementary information, used for decoding and reconstructing the isomorphic blocks.

[0168] For example, the isomorphic block information includes the syntax elements of ASPS and AFPS; different isomorphic block information corresponds to different syntax elements of ASPS and AFPS. In some embodiments, for heterogeneous hybrid mosaic maps, the ASPS and AFPS of isomorphic blocks of different representation formats are at least partially different, that is, the ASPS and AFPS of isomorphic blocks of different representation formats are not completely identical. When encoding heterogeneous hybrid mosaic maps, the high-level information (ASPS and AFPS) of blocks of different representation formats in the heterogeneous hybrid mosaic map can be satisfied as they are not equal. This achieves higher-level parameters that are more suitable for heterogeneous hybrid mosaic maps, effectively improving encoding efficiency, i.e., reducing bitrate or improving the quality of reconstructed multi-view video or point cloud video.

[0169] For example, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first and second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. The first isomorphic block information also includes extended syntax elements of the ASPS parameter set and the AFPS parameter set. Here, extended syntax elements of the ASPS parameter set and the AFPS parameter set are added for multi-view video blocks in heterogeneous hybrid stitched graphs to represent ASPS and AFPS parameters that are not equal to those in point cloud blocks, thereby improving the decoding efficiency of multi-view video blocks. It should be noted that during the decoding and reconstruction of point cloud blocks and multi-view video blocks, these ASPS and AFPS parameters may have the same function but different values.

[0170] For example, the second isomorphic block information includes extended syntax elements of the ASPS parameter set and the AFPS parameter set. That is, extended syntax elements of the ASPS parameter set and the AFPS parameter set can also be added to the point cloud video blocks of the heterogeneous hybrid mosaic image to improve the decoding efficiency of the point cloud blocks. The first isomorphic block information and the extended syntax elements of the second isomorphic block information are also included.

[0171] For example, the extended syntax elements of the ASPS parameter set include: `ashm_geometry_3d_bit_depth_minus1` to represent the bit depth of the geometric coordinates of the reconstructed geometric content; `ashm_geometry_2d_bit_depth_minus1` to represent the bit depth of the geometry when projected onto a 2D image; and `ashm_log2_max_atlas_frame_order_cnt_lsb_minus4` to determine the variable value used for counting the stitched frame order during decoding. The extended syntax elements of the AFPS parameter set include: `afhm_additional_lt_afoc_lsb_len` to determine the value of the variable `MaxLtAtlasFrmOrderCntLsbForMiv` used during decoding of the reference stitched frame list. The ASPS and AFPS parameters represented by these syntax elements for multi-view video blocks are not exactly the same as those for point cloud blocks.

[0172] For example, the first isomorphic block information and the second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set; the first isomorphic block information also includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content. It should be noted that the naming of syntax elements in this embodiment is mainly for ease of understanding and writing. Modifications can be made in actual applications and standard texts, but their semantic content should be consistent or similar. For example, ashm_geometry_3d_bit_depth_minus1 and asps_geometry_3d_bit_depth_minus1_for_miv both represent the first extended syntax element, which can also be understood as a newly added syntax element.

[0173] For example, when the mosaic is a homogeneous mosaic, the mosaic information includes homogeneous block information, which is used to decode and reconstruct the homogeneous blocks in the mosaic.

[0174] In some embodiments, the heterogeneous hybrid mosaic of this application includes at least one of the following: a single-attribute heterogeneous hybrid mosaic and a multi-attribute heterogeneous hybrid mosaic.

[0175] In this context, a single-attribute heterogeneous hybrid mosaic refers to a heterogeneous hybrid mosaic where all homogeneous blocks contain the same attribute information. For example, a single-attribute heterogeneous hybrid mosaic may only include homogeneous blocks with identical attribute information, such as multi-view video texture blocks and point cloud texture blocks. Another example is a single-attribute heterogeneous hybrid mosaic that includes homogeneous blocks with identical geometric information, such as multi-view video geometry blocks and point cloud geometry blocks.

[0176] A multi-attribute heterogeneous hybrid mosaic refers to a heterogeneous hybrid mosaic that includes at least two isomorphic blocks with different attribute information. For example, a multi-attribute heterogeneous hybrid mosaic may include isomorphic blocks with both attribute information and geometric information. As an example, blocks under any one or any two attributes from point clouds, multi-view videos, and meshes can be stitched together to obtain a heterogeneous hybrid mosaic. This application does not limit this.

[0177] In some embodiments, a single-attribute isomorphic block of a first expression format and a single-attribute block of a second expression format are stitched together to obtain a heterogeneous hybrid stitched image. The first and second expression formats are any one of multi-view video, point cloud, and mesh, and the first and second expression formats are different, but their attribute information is the same.

[0178] A single-attribute isomorphic block of multiview video includes at least one of multiview video texture blocks and multiview video geometric blocks.

[0179] A single-attribute isomorphic block of a point cloud includes at least one of a point cloud texture block, a point cloud geometry block, and a point cloud occupancy block.

[0180] A single-attribute isomorphic block of a mesh includes at least one of a mesh texture block and a mesh geometry block.

[0181] For example, stitching together at least two of the following geometric regions—multi-view video, point cloud, and mesh—into a single image yields a heterogeneous hybrid mosaic image. This heterogeneous hybrid mosaic image is called a single-attribute heterogeneous hybrid mosaic image. As another example, stitching together at least two of the following texture regions—multi-view video, point cloud, and mesh—into a single image yields a heterogeneous hybrid mosaic image. This heterogeneous hybrid mosaic image is also called a single-attribute heterogeneous hybrid mosaic image.

[0182] In some embodiments, multi-attribute isomorphic blocks of a first expression format and multi-attribute isomorphic blocks of a second expression format are stitched together to obtain a heterogeneous hybrid stitched image. The first and second expression formats are any one of multi-view video, point cloud, and mesh, and the first and second expression formats are different, with their attribute information not being entirely identical.

[0183] For example, stitching a multi-view video texture block with at least one of a point cloud geometry block and a mesh geometry block into a single image yields a heterogeneous hybrid stitched image. Again, stitching a multi-view video geometry block with at least one of a point cloud texture block and a mesh texture block into a single image yields a heterogeneous hybrid stitched image. Again, stitching a point cloud texture block with at least one of a multi-view video geometry block and a mesh geometry block into a single image yields a heterogeneous hybrid stitched image. Again, stitching a point cloud geometry block with at least one of a multi-view video texture block and a mesh texture block into a single image yields a heterogeneous hybrid stitched image. Again, stitching a point cloud geometry block, a multi-view video texture block, and a multi-view video texture block into a single image yields a heterogeneous hybrid stitched image. For example, point cloud geometric blocks, point cloud texture blocks, and multi-view video texture blocks can be stitched together into a single image to obtain a heterogeneous hybrid stitched image. Here, the resulting heterogeneous hybrid stitched image is called a multi-attribute heterogeneous hybrid stitched image.

[0184] In some embodiments, the isomorphic mosaic graph of this application includes at least one of the following: a single-attribute isomorphic mosaic graph and a multi-attribute isomorphic mosaic graph. In some embodiments, the first attribute isomorphic blocks of the first expression format are spliced ​​together to obtain the isomorphic mosaic graph. Alternatively, the first attribute isomorphic blocks and the second attribute isomorphic blocks of the first expression format are spliced ​​together to obtain the isomorphic mosaic graph.

[0185] In this context, a single-attribute isomorphic mosaic refers to an isomorphic mosaic where all included isomorphic blocks have the same representation format and attribute information. For example, a single-attribute isomorphic mosaic may only include isomorphic blocks with attribute information in a specific representation format, such as a single-attribute isomorphic mosaic that only includes multi-view video texture blocks or only includes point cloud texture blocks. Similarly, a single-attribute isomorphic mosaic may only include isomorphic blocks with geometric information, such as a single-view video geometric block or only includes point cloud geometric blocks.

[0186] A multi-attribute isomorphic mosaic refers to an isomorphic mosaic containing at least two isomorphic blocks with the same representation format but different attribute information. For example, a multi-attribute isomorphic mosaic may include isomorphic blocks containing both attribute information and geometric information. As an example, a multi-attribute isomorphic mosaic might include a multi-view video texture block and a multi-view video set block. Another example is a multi-attribute isomorphic mosaic that includes point cloud geometry blocks and point cloud texture blocks, such as... Figure 8 As shown, a multi-attribute isomorphic mosaic image includes point cloud texture block 1, point cloud geometry block 1, and point cloud geometry block 2.

[0187] In some embodiments, the splicing graph information may further include syntax elements, and the splicing graph is determined to be a single-attribute heterogeneous hybrid splicing graph, a multi-attribute heterogeneous hybrid splicing graph, a single-attribute isomorphic splicing graph, or a multi-attribute isomorphic splicing graph based on the syntax elements.

[0188] Step 603: Encode the stitched image and stitched image information to obtain a bitstream.

[0189] In some embodiments, the parameter set sub-bitstream of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content including at least one expression format in the bitstream is determined based on the third syntax element. For example, the parameter set of the bitstream is V3C_VPS, and the third syntax element can be ptl_profile_toolset_idc in V3C_VPS.

[0190] In some embodiments, a third syntax element is set to different values ​​to indicate a bitstream that includes visual media content in at least one expression format. For example, determining a bitstream that includes visual media content in at least one expression format based on the third syntax element includes: if the third syntax element is a first value, determining a bitstream that simultaneously includes visual media content in a first expression format and visual media content in a second expression format; if the third syntax element is a second value, determining a bitstream that includes visual media content in the first expression format; and if the third syntax element is a third value, determining a bitstream that includes visual media content in the second expression format.

[0191] For example, taking a first expression format of multi-view video and a second expression format of point cloud as an example, when the third syntax element is set to the first value, the first value indicates that the bitstream contains both multi-view video and point cloud bitstreams. As a specific example, when ptl_profile_toolset_idc = X, and X is 128 / 129 / 130 / 131 / 132 / 133, it means that the current bitstream contains both point cloud and multi-view bitstreams. As another example, when the third syntax element is set to the second value, the second value indicates that the bitstream contains only the point cloud bitstream. As a specific example, when ptl_profile_toolset_idc = X, and X is 0 / 1, it means that the current bitstream contains only the point cloud bitstream. As yet another example, when the third syntax element is set to the third value, the third value indicates that the bitstream contains only the multi-view video bitstream. As a specific example, when ptl_profile_toolset_idc = X, and X is 64 / 65 / 66, it indicates that the current bitstream only contains multi-view video bitstreams. It should be understood that the values ​​of the first, second, and third values ​​mentioned above are merely examples, and the embodiments of this application are not limited thereto.

[0192] In some embodiments, the bitstream includes a video compression sub-bitstream and a stitched image information sub-bitstream. Encoding the stitched image and its information to obtain the bitstream includes: encoding the stitched image to obtain a video compression sub-bitstream; encoding the stitched image information to obtain a stitched image information sub-bitstream; and combining the video compression sub-bitstream and the stitched image information sub-bitstream into the final bitstream. This allows for support of heterogeneous source formats such as video, point clouds, and meshes within the same compressed bitstream, enabling the simultaneous presence of multi-view video stitched images, point cloud video stitched images, mesh stitched images, and heterogeneous hybrid stitched images within the compressed bitstream. This reduces the number of 2D video encoders (HEVC, VVC, AVC, AVS, etc.) required, lowers implementation costs, and improves usability.

[0193] In some embodiments, encoding the stitched image and stitched image information to obtain a bitstream includes: if the expression format of the i-th block is a first expression format, determining that the sub-image blocks in the i-th block are encoded using the encoding standard corresponding to the first expression format to obtain a bitstream corresponding to the visual media content of the first expression format; if the expression format of the i-th block is a second expression format, determining that the sub-image blocks in the i-th block are encoded using the encoding standard corresponding to the second expression format to obtain a bitstream corresponding to the visual media content of the second expression format.

[0194] For example, if the second syntax element of the i-th block is known to be 1, it is determined that the current sub-block is encoded using the multi-view video coding standard. If the second syntax element of the i-th block is known to be 0, it is determined that the current sub-block is encoded using the point cloud coding standard.

[0195] In this embodiment, video encoding is performed on the heterogeneous hybrid stitched image and the homogeneous stitched image to obtain the video encoder used for the video compressed sub-stream. This encoder can be the one described above. Figure 2A The video encoder shown is described above. In other words, in this embodiment of the application, a heterogeneous or homogeneous mosaic image is used as a frame image. First, block division is performed. Then, intra-frame or inter-frame prediction is used to obtain the predicted value of the coded block. The predicted value of the coded block is subtracted from the original value to obtain the residual value. After transformation and quantization processing of the residual value, the video compressed sub-bitstream is obtained.

[0196] In this embodiment, while generating at least one mosaick, mosaick information corresponding to each mosaick is also generated. The mosaick information is then encoded to obtain a mosaick information sub-stream. The mosaick information includes a first syntax element indicating the mosaick type and a second syntax element representing the expression format of each isomorphic block in the mosaick. This embodiment does not limit the encoding method of the mosaick information; for example, it can use conventional data compression encoding methods such as fixed-length encoding or variable-length encoding.

[0197] Finally, the video compression sub-stream and the stitched image information sub-stream are written into the same bitstream to obtain the final bitstream. In other words, the embodiments of this application not only support heterogeneous source formats such as video, point cloud, and mesh, but also homogeneous source formats within the same compressed bitstream.

[0198] In some embodiments, the method further includes: encoding the parameter set of the bitstream to obtain a bitstream parameter set sub-bitstream. Specifically, the encoding end synthesizes the video compression sub-bitstream, the stitching information sub-bitstream, and the parameter set sub-bitstream into a bitstream. The parameter set sub-bitstream of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content of at least one expression format is determined based on the third syntax element. That is, the encoding end sends a third syntax element to indicate whether the bitstream simultaneously contains visual media content of at least two expression formats. For example, when the third syntax element indicates that the bitstream includes a bitstream corresponding to visual media content of one expression format, it can be understood that the encoding end processes the visual media content of one expression format to obtain a homogeneous block, and stitches the homogeneous block to obtain a homogeneous stitching image. When the third syntax element indicates that the bitstream includes the bitstream corresponding to visual media content in at least two expression formats, it can be understood that the encoding end obtains at least two isomorphic blocks for the visual media content in at least two expression formats, and splices the at least two isomorphic blocks to obtain an isomorphic spliced ​​image and / or a heteromorphic hybrid spliced ​​image.

[0199] For example, when the third syntax element indicates that the bitstream includes bitstreams corresponding to visual media content in at least two expression formats, the method includes: isomorphically splicing isomorphically blocks of the first expression format to obtain a first isomorphic spliced ​​image, and isomorphically splicing isomorphically blocks of the second expression format to obtain a second isomorphic spliced ​​image; or, heteromorphically splicing isomorphic blocks of the first expression format and isomorphic blocks of the second expression format to obtain a heteromorphic hybrid spliced ​​image; or, isomorphically splicing isomorphic blocks of the first expression format to obtain a first isomorphic spliced ​​image, and heteromorphically splicing isomorphic blocks of the first expression format and isomorphic blocks of the second expression format to obtain a heteromorphic hybrid spliced ​​image; or, isomorphically splicing isomorphic blocks of the second expression format to obtain a second isomorphic spliced ​​image, and heteromorphically splicing isomorphic blocks of the first expression format and isomorphic blocks of the second expression format to obtain a heteromorphic hybrid spliced ​​image.

[0200] In this embodiment, to reduce the number of encoders and lower encoding costs, the visual media content is first processed (i.e., packaged) to obtain multiple isomorphic blocks. Then, at least two isomorphic blocks with different expression formats are stitched together to form a heterogeneous hybrid stitching image, and at least one isomorphic block with identical expression formats is stitched together to form a isomorphic stitching image. The heterogeneous hybrid stitching image and the isomorphic stitching image are encoded to obtain a video compressed sub-bitstream. The stitching image information is encoded to obtain a stitching information sub-bitstream. Finally, the video compressed bitstream and the stitching information bitstream are combined into a compressed bitstream. This encoding method is applicable to various visual media content expression formats, expanding its application scope. Furthermore, by mixing and encoding data with different expression formats, only one video encoder needs to be called during encoding, thereby reducing the number of 2D video encoders such as HEVC, VVC, AVC, and AVS that need to be called, reducing encoding costs, and improving ease of use. Moreover, when encoding heterogeneous hybrid mosaic images, some high-level parameters of different expression format blocks can be unequal, which can retain more effective information of different expression format blocks, improve the image synthesis quality, and improve the overall efficiency of bitrate-quality.

[0201] The above describes the encoding method of this application using the encoding end as an example. The following describes the video decoding method provided in the embodiments of this application using the decoding end as an example.

[0202] Figure 9 This is a schematic flowchart illustrating a decoding method provided in an embodiment of this application. Figure 9 As shown, the decoding method in this application embodiment includes:

[0203] Step 901: Decode the bitstream to obtain the spliced ​​image and spliced ​​image information;

[0204] For example, the bitstream includes a video compression sub-bitstream and a stitched image information sub-bitstream. Decoding the bitstream to obtain the stitched image and stitched image information includes: extracting the stitched image information sub-bitstream and the video compression sub-bitstream respectively; decoding the video compression sub-bitstream to obtain the stitched image; and decoding the stitched image information sub-bitstream to obtain the stitched image information.

[0205] For example, the video compression sub-stream is decoded to obtain a heterogeneous hybrid stitched image, a multi-view stitched image, and a point cloud stitched image; the stitched image information sub-stream is decoded to obtain heterogeneous hybrid stitched image information, multi-view stitched image information, and point cloud stitched image information.

[0206] For example, in some embodiments, the bitstream further includes a parameter set sub-bitstream; the parameter set sub-bitstream includes a third syntax element; the bitstream corresponding to visual media content with at least one expression format is determined based on the third syntax element. That is, during decoding, the bitstream is first judged based on the third syntax element to determine how many expression formats of visual media content it contains. When the third syntax element of the V3C bitstream layer determines that the bitstream contains visual media content with one expression format, it is determined that the stitched image is entirely isomorphic. When the third syntax element determines that the bitstream contains visual media content with two expression formats, it is determined that the stitched image may contain heterogeneous hybrid stitched images, requiring further determination of whether the stitched image is isomorphic or heterogeneous hybrid. Further, the stitched image type is determined based on the first syntax element; when the stitched image is determined to be heterogeneous hybrid, the isomorphic block type is then determined based on the second syntax element.

[0207] In some embodiments, a third syntax element is set to different values ​​to indicate a bitstream that includes visual media content in at least one expression format. For example, determining a bitstream that includes visual media content in at least one expression format based on the third syntax element includes: if the third syntax element is a first value, determining a bitstream that simultaneously includes visual media content in a first expression format and visual media content in a second expression format; if the third syntax element is a second value, determining a bitstream that includes visual media content in the first expression format; and if the third syntax element is a third value, determining a bitstream that includes visual media content in the second expression format.

[0208] For example, taking a first expression format of multi-view video and a second expression format of point cloud as an example, when the third syntax element is set to the first value, the first value indicates that the bitstream contains both multi-view video and point cloud bitstreams. As a specific example, when ptl_profile_toolset_idc = X, and X is 128 / 129 / 130 / 131 / 132 / 133, it means that the current bitstream contains both point cloud and multi-view bitstreams. As another example, when the third syntax element is set to the second value, the second value indicates that the bitstream contains only the point cloud bitstream. As a specific example, when ptl_profile_toolset_idc = X, and X is 0 / 1, it means that the current bitstream contains only the point cloud bitstream. As yet another example, when the third syntax element is set to the third value, the third value indicates that the bitstream contains only the multi-view video bitstream. As a specific example, when ptl_profile_toolset_idc = X, and X is 64 / 65 / 66, it indicates that the current bitstream only contains multi-view video bitstreams. It should be understood that the values ​​of the first, second, and third values ​​mentioned above are merely examples, and the embodiments of this application are not limited thereto.

[0209] Step 902: When the stitched image is a heterogeneous hybrid stitched image, at least two types of isomorphic blocks and isomorphic block information are obtained based on the stitched image and the stitched image information; wherein, different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information;

[0210] The mosaic information is used to reconstruct the mosaic. For example, the mosaic information includes at least mosaic type information, mosaic information of isomorphic blocks, and isomorphic block information. In some embodiments, the mosaic information includes a first syntax element, and the mosaic is determined to be either a heterogeneous hybrid mosaic or a isomorphic mosaic based on the first syntax element. In some embodiments, the first syntax element is a syntax element of the mosaic sequence parameter set ASPS and a syntax element of the mosaic frame parameter set AFPS. The ASPS and AFPS are parsed to determine the mosaic type.

[0211] For example, when the stitched image is a heterogeneous hybrid stitched image, the stitched image is split into at least two types of isomorphic blocks; according to the expression format of the at least two types of isomorphic blocks, the isomorphic block information corresponding to the at least two types of isomorphic blocks is obtained from the stitched image information. For example, the heterogeneous hybrid stitched image is split according to the heterogeneous hybrid stitched image information, and the reconstructed multi-view video isomorphic blocks and isomorphic block information, as well as the reconstructed point cloud isomorphic blocks and isomorphic block information, are output.

[0212] In some embodiments, the first syntax element includes a first sub-syntax element and a second sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: if the value of the first sub-syntax element and the value of the second sub-syntax element are equal, determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

[0213] In other words, the bitstream must guarantee absolute consistency between the first and second sub-syntax elements. Only when the two sub-syntax elements are consistent can the splice graph type be determined. For example, the consistency of the two sub-syntax elements can be compared first, and then the splice graph type can be determined based on the value of one of the sub-syntax elements. Alternatively, the splice graph type can be determined based on each sub-syntax element first, and absolute consistency can be guaranteed by comparing whether the splice graph types are the same.

[0214] For example, determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: if the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic; if the value is a second preset value, then the mosaic is determined to be a homogeneous mosaic. In other words, two values ​​or two types of values ​​can be set to identify heterogeneous hybrid mosaics and homogeneous mosaics. For example, the first preset value is 1, and the second preset value is 0.

[0215] For example, in some embodiments, the splicing graph information does not include the first syntax element, and the splicing graph is determined to be an isomorphic splicing graph. In some embodiments, if the splicing graph information does not include the first syntax element, it is inferred that the first syntax element has a second preset value. For example, if the splicing graph information does not include the first sub-syntax element, the splicing graph is determined to be an isomorphic splicing graph, and the first sub-syntax element has a second preset value; if the splicing graph information does not include the second sub-syntax element, the splicing graph is determined to be an isomorphic splicing graph, and the second sub-syntax element has a second preset value.

[0216] For example, determining whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image based on the value includes: if the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including homogeneous blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; if the value is a fourth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the first expression format; if the value is a fifth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the second expression format. That is, multiple values ​​can be set to identify the expression formats of heterogeneous hybrid stitched images and homogeneous stitched images, and even to identify which homogeneous blocks of expression formats are included in the heterogeneous hybrid stitched image. For example, the third preset value is 2, the fourth preset value is 1, and the fifth preset value is 0.

[0217] For example, the first sub-syntax element is a syntax element of the mosaic sequence parameter set ASPS, and the second sub-syntax element is a syntax element of the mosaic frame parameter set AFPS. In this embodiment, asps_heterogeneous_miv_extension_present_flag represents the first sub-syntax element, and afps_heterogeneous_miv_extension_present_flag represents the second sub-syntax element.

[0218] For example, in some embodiments, when the stitched image is a heterogeneous hybrid stitched image, the stitched image information further includes a second syntax element; the expression format of isomorphic blocks in the stitched image is determined based on the second syntax element. It is understood that after determining that the stitched image is a heterogeneous hybrid stitched image based on the first syntax element, the second syntax element of the isomorphic blocks is further parsed to determine the type of isomorphic blocks. For example, the second syntax element is a syntax element of the stitched image frame parameter set (AFPS).

[0219] Specifically, the expression format type corresponding to the i-th block in the stitched graph can be indicated by setting different values ​​for the second syntax element. For example, determining the expression format of the isomorphic block in the stitched graph based on the second syntax element includes: when the value of the second syntax element of the i-th block is a sixth preset value, the expression format of the i-th block is determined to be a first expression format; when the value of the second syntax element of the i-th block is a seventh preset value, the expression format of the i-th block is determined to be a second expression format. Taking a point cloud as the first expression format and a multi-view video as the second expression format as an example. Optionally, the sixth preset value is 0, and the seventh preset value is 1.

[0220] Furthermore, the expression format of the i-th block is a first expression format, and the i-th block is decoded using the decoding method corresponding to the first expression format; the expression format of the i-th block is a second expression format, and the i-th block is decoded using the decoding method corresponding to the second expression format.

[0221] For example, in some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information. The isomorphic block information includes reconstruction information of the isomorphic blocks and other supplementary information, used for decoding and reconstructing the isomorphic blocks.

[0222] For example, the isomorphic block information includes the syntax elements of ASPS and AFPS; different isomorphic block information corresponds to different syntax elements of ASPS and AFPS. In some embodiments, for heterogeneous hybrid mosaic maps, the ASPS and AFPS of isomorphic blocks of different representation formats are at least partially different, that is, the ASPS and AFPS of isomorphic blocks of different representation formats are not completely identical. When encoding heterogeneous hybrid mosaic maps, the high-level information (ASPS and AFPS) of blocks of different representation formats in the heterogeneous hybrid mosaic map can be satisfied as they are not equal. This achieves higher-level parameters that are more suitable for heterogeneous hybrid mosaic maps, effectively improving encoding efficiency, i.e., reducing bitrate or improving the quality of reconstructed multi-view video or point cloud video.

[0223] For example, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first isomorphic block information and the second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. The first isomorphic block information also includes extended syntax elements of the ASPS parameter set and the AFPS parameter set.

[0224] In some embodiments, the representation format is multi-view video, point cloud, or mesh. One isomorphic block corresponds to one representation format. Different isomorphic blocks correspond to different representation formats. For example, at least two isomorphic blocks correspond to at least two of the following representation formats: multi-view video, point cloud, and mesh. It should be noted that in the embodiments of this application, each isomorphic block may include at least one isomorphic block with the same representation format. For example, a point cloud format isomorphic block includes one or more point cloud blocks, a multi-view video format isomorphic region includes one or more multi-view video blocks, and a mesh format isomorphic block includes one or more mesh blocks.

[0225] Step 903: When the mosaic is a isomorphic mosaic, a type of isomorphic block and isomorphic block information are obtained based on the mosaic and the mosaic information.

[0226] For example, when the stitched image is a isomorphic stitched image, the stitched image is split to obtain isomorphic blocks; isomorphic block information is obtained from the stitched image information. For example, the isomorphic stitched image is split according to the isomorphic stitched image information of the multi-view video, and the reconstructed multi-view video isomorphic blocks and isomorphic block information are output. The isomorphic stitched image is split according to the isomorphic stitched image information of the point cloud, and the reconstructed point cloud isomorphic blocks and isomorphic block information are output.

[0227] For example, when the mosaic is a homogeneous mosaic, the mosaic information includes homogeneous block information, which is used to decode and reconstruct the homogeneous blocks in the mosaic.

[0228] In some embodiments, the heterogeneous hybrid mosaic of this application includes at least one of the following: a single-attribute heterogeneous hybrid mosaic and a multi-attribute heterogeneous hybrid mosaic.

[0229] Step 904: Based on the isomorphic blocks and the isomorphic block information, obtain visual media content in at least two expression formats.

[0230] For example, obtaining visual media content in at least two expression formats based on the isomorphic blocks and the isomorphic block information includes: if the expression format of the i-th block is a first expression format, determining that the sub-blocks in the i-th block are decoded and reconstructed using the decoding method corresponding to the first expression format to obtain visual media content in the first expression format; if the expression format of the i-th block is a second expression format, determining that the sub-blocks in the i-th block are decoded and reconstructed using the decoding method corresponding to the second expression format to obtain visual media content in the second expression format.

[0231] By employing the above technical solution, encoding homogeneous blocks of different expression formats by stitching them together in a heterogeneous hybrid stitched image reduces the number of 2D video codecs such as HEVC, VVC, AVC, and AVS that need to be called, lowering implementation costs and improving ease of use. Furthermore, when decoding the heterogeneous hybrid stitched image, certain high-level parameters of blocks from different expression formats can be unequal, preserving more effective information from these blocks, improving image synthesis quality, and enhancing the overall bitrate-quality efficiency.

[0232] The decoding method provided in the embodiments of this application will be further illustrated below.

[0233] Figure 10This is a schematic diagram of the V3C bitstream structure provided in the embodiments of this application. The V3C parameter set (V3C_parameter_set()) of V3C_VPS may include a third syntax element (ptl_profile_toolset_idc). A ptl_profile_toolset_idc value of 128 to 133 indicates that the current bitstream simultaneously contains a point cloud bitstream (such as VPCCbasic or VPCC extended) and a multi-view video bitstream (such as MIV main, MIV Extended, or MIVGeometry Absent).

[0234] The ASPS parameter set may include the first sub-syntax element (asps_heterogeneous_miv_extension_present_flag). When ptl_profile_toolset_idc is 128 to 133, the current mosaic type is determined based on asps_heterogeneous_miv_extension_present_flag.

[0235] The AFPS parameter set can include a second sub-syntax element (afps_heterogeneous_miv_extension_present_flag). When ptl_profile_toolset_idc is between 128 and 133, afps_heterogeneous_miv_extension_present_flag is used to determine the current mosaic type. The AFPS parameter set also includes a second syntax element (afps_heterogeneous_frame_tile_toolset_miv_present_flag) used to determine the stripe type, thus ensuring that during parsing and decoding, the current stripe should belong to either a multi-viewpoint or point cloud.

[0236] The bitstream must ensure absolute consistency between afps_heterogeneous_miv_extension_present_flag and asps_heterogeneous_miv_extension_present_flag.

[0237] Decoding Case 1

[0238] 1. During decoding, the VPS is obtained from the V3C bitstream, and ptl_profile_toolset_idc (the third syntax element) is obtained from the VPS. If ptl_profile_toolset_idc = 0 or 1, it indicates that only point cloud bitstreams exist in the current bitstream.

[0239] 2. The current bitstream is decoded using the point cloud decoding standard.

[0240] Decoding Case 2

[0241] 1. During decoding, the VPS is obtained from the V3C bitstream, and ptl_profile_toolset_idc is obtained from the VPS. If ptl_profile_toolset_idc = 64 / 65 / 66, it indicates that only a multi-viewpoint bitstream exists in the current bitstream;

[0242] 2. Implement multi-viewpoint decoding standards for the current code stream.

[0243] Decoding Case 3

[0244] 1. During parsing, the VPS is obtained from the V3C bitstream, and ptl_profile_toolset_idc is obtained from the VPS. If ptl_profile_toolset_idc = 128~133, it indicates that the current bitstream contains both point cloud and multi-viewpoint bitstreams.

[0245] 2. Parse high-level syntax such as ASPS and AFPS for each mosaic image:

[0246] Referring to Table 2, before step a), the following steps are also included: In ASPS, asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, and asps_extension_6bits are first parsed, and then the HeterogeneousPresentFlag is obtained by judgment. The following decoding operation will only continue if the HeterogeneousPresentFlag is true.

[0247] a) Parse asps_heterogeneous_miv_extension_present_flag (first sub-syntax element) in ASPS:

[0248] i. If `asps_heterogeneous_miv_extension_present_flag` does not exist (i.e., `asps_extension_6bits = 0`), it indicates that the current mosaic is homogeneous (all stripes are point cloud or multi-view type). ii. If `asps_heterogeneous_miv_extension_present_flag` exists and is equal to 0, it indicates that the current mosaic is homogeneous (all stripes are point cloud or multi-view type). iii. If `asps_heterogeneous_miv_extension_present_flag` exists and is equal to 1, it indicates that the current mosaic is heterogeneous, containing both point cloud stripes and multi-view stripes. Therefore, the ASPS auxiliary high-level information of the current mosaic is split into two sub-sets: one sub-set for decoding multi-view stripes and the other sub-set for decoding point cloud stripes. The auxiliary information required for point cloud stripes can be obtained by parsing Part 8 of Standard 23090-5; the auxiliary information required for multi-viewpoint stripes can be obtained by parsing Part 8 of Standard 23090-5 and the newly added asps_heterogeneous_miv_extension and Part 8 of Standard 23090-12.

[0249] Referring to Table 4, before step b), the process includes: first, parsing afps_miv_extension_present_flag and afps_extension_7bits in AFPS. The following decoding operation can only proceed if HeterogeneousPresentFlag is true.

[0250] b) Parse afps_heterogeneous_miv_extension_present_flag (second sub-syntax element) in AFPS:

[0251] i. If `afps_heterogeneous_miv_extension_present_flag` does not exist (i.e., `afps_extension_7bits = 0`), it indicates that the current mosaic is homogeneous (all stripes are point cloud or multi-view type). ii. If `afps_heterogeneous_miv_extension_present_flag` exists and is equal to 0, it indicates that the current mosaic is homogeneous (all stripes are point cloud or multi-view type). iii. If `afps_heterogeneous_miv_extension_present_flag` exists and is equal to 1, it indicates that the current mosaic is heterogeneous, containing both point cloud stripes and multi-view stripes. Therefore, the AFPS auxiliary high-level information of the current mosaic is split into two sub-sets: one sub-set for decoding multi-view stripes and the other sub-set for decoding point cloud stripes. The auxiliary information required for point cloud stripes can be obtained by parsing Part 8 of Standard 23090-5; the auxiliary information required for multi-view stripes can be obtained by parsing Part 8 of Standard 23090-5 and the newly added afps_heterogeneous_miv_extension and Part 8 of Standard 23090-12.

[0252] c) Parse afps_heterogeneous_frame_tile_toolset_miv_present_flag (the third sub-syntax element in the second syntax element) in AFPS:

[0253] i. If afps_heterogeneous_frame_tile_toolset_miv_present_flag does not exist, it means that all stripes in the current mosaic are of the same type; ii. Traverse all tiles, and determine if afps_heterogeneous_frame_tile_toolset_miv_present_flag[i] = 0 for the i-th strip, indicating that the current strip is a point cloud strip and afps_heterogeneous_frame_tile_toolset_miv_present_flag is the fourth sub-syntax element in the second sub-syntax element; determine if afps_heterogeneous_frame_tile_toolset_miv_present_flag[i] = 1 for the i-th strip, indicating that the current strip is a multi-view strip. In other words, when afps_heterogeneous_frame_tile_toolset_miv_present_flag exists (e.g., its value is 1), all tiles are traversed according to afps_heterogeneous_frame_tile_toolset_miv_present_flag[i] to determine the stripe type of the i-th stripe. In some embodiments, the second syntax element may also only include afps_heterogeneous_frame_tile_toolset_miv_present_flag[i].

[0254] Specifically, the bitstream must ensure absolute consistency between afps_heterogeneous_miv_extension_present_flag and asps_heterogeneous_miv_extension_present_flag.

[0255] 3. Then, analyze the patch_data_unit information of each sub-plot in each strip. Given that the current strip uses multi-view auxiliary information, determine that the current sub-plot uses the multi-view video decoding standard. Given that the current strip uses point cloud auxiliary information, determine that the current sub-plot uses the point cloud video decoding standard.

[0256] The presence of heterogeneous mixed mosaic images is indicated by the `ptl_profile_toolset_idc` number. New `asps_heterogeneous_miv_extension_present_flag` and `afps_heterogeneous_miv_extension_present_flag` flags have been added to determine whether each mosaic image should be classified as point cloud / multi-viewpoint / point cloud + multi-viewpoint. To maintain compatibility with previous standards, the following new syntax and semantics have been implemented, along with constraints on older semantics. Tables 9-1-1 and 9-1-2 respectively show the syntax restrictions for multi-viewpoint toolbox tier components and heterogeneous data toolbox tier components under integrated bitstreams.

[0257] Tables 9-1-1 and 9-1-2 respectively represent the syntax restrictions for toolbox grade components for multi-viewpoint and toolbox grade components for heterogeneous data under integrated bitstream.

[0258] In this embodiment, the type of each stripe in the current mosaic is described by the newly added syntax element afps_heterogeneous_frame_tile_toolset_miv_present_flag in the high-level part (mosaic level and mosaic sequence level), thereby ensuring that the current stripe belongs to the multi-viewpoint or point cloud during parsing and decoding.

[0259] This solution ensures that there is only one usable mosaic level parameter (AFPS) and mosaic sequence level parameter (ASPS) whether it is multi-view parsing or point cloud parsing, and it can achieve that the AFPS and ASPS of multi-view parsing are not completely equal to the AFPS and ASPS of point cloud parsing.

[0260] For detailed explanations of specific grammatical elements, please refer to Tables 1 to 9-1-2 below.

[0261] Table 1 shows an example of available toolset profile components. Table 1 provides a list of toolset profile components defined for V3C and their corresponding identifier syntax element values, such as `ptl_profile_toolset_idc` and `ptc_one_v3c_frame_only_flag`. This definition is for use only in this document. The syntax element `ptl_profile_toolset_idc` provides the main definition of the toolset profile. Additional syntax elements such as `ptc_one_v3c_frame_only_flag` can specify additional features or limitations of the defined profile. `ptc_one_v3c_frame_only_flag` can be used to support only a single V3C frame. It should be noted that values ​​2..63, 67..127, 134..255 in `ptl_profile_toolset_idc` are reserved and currently undefined; the standards organization may specify them in future standards. The profile types defined in Table 1 can include dynamic or static.

[0262] Table 1. Available toolset configuration file components

[0263]

[0264]

[0265] Table 2 shows the RBSP syntax for the General atlas sequence parameter set, which can be used in ISO / IEC 23090-5. The atlas type is indicated by the extended syntax element asps_heterogeneous_miv_extension_present_flag in the atlas sequence parameter set. Specifically, the value of this syntax element determines whether the atlas should be classified as point cloud / multi-viewpoint / point cloud + multi-viewpoint.

[0266] Table 2. RBSP Syntax for General Patchwork Sequence Parameter Sets

[0267]

[0268]

[0269] Table 3 shows the ASPS (Atlas sequence parameter setheterogeneous MIV extension syntax) elements, which can be used in ISO / IEC 23090-5. `ashm_geometry_3d_bit_depth_minus1` represents the bit depth of the geometric coordinates of the reconstructed geometry. `ashm_geometry_2d_bit_depth_minus1` represents the bit depth of the geometry when projected onto a 2D image. `ashm_log2_max_atlas_frame_order_cnt_lsb_minus4` determines the variable value used for counting the stitched frame order during decoding.

[0270] Table 3 ASPS Heterogeneous Multiview Extension Syntax Elements

[0271]

[0272] Table 4 shows the RBSP syntax for the General atlas frame parameter set, which can be used in ISO / IEC 23090-5. The extended syntax element `afps_heterogeneous_miv_extension_present_flag` in the atlas frame parameter set indicates the atlas type. Specifically, the value of this syntax element determines whether the atlas should be classified as point cloud / multi-viewpoint / point cloud + multi-viewpoint.

[0273] Table 4. RBSP Syntax for Frame Stitching

[0274]

[0275] Table 5 lists the AFPS heterogeneous MIV extension syntax elements, which can be used in ISO / IEC 23090-5. `afhm_additional_lt_afoc_lsb_len` is used to determine the value of the variable `MaxLtAtlasFrmOrderCntLsbForMiv` used during decoding of the reference stitched frame list.

[0276] Table 5 lists the extended syntax elements of AFPS heterogeneous MIV.

[0277]

[0278] The semantics of ASPS and AFPS syntax elements are explained below.

[0279] 1. Semantics of ASPS syntax elements:

[0280] An asps_extension_6bits value of 0 indicates that the asps_extension_data_flag does not exist in the ASPS RBSP syntax structure. If it exists, the value of asps_extension_6bits is 0 or 1 in this standard text; values ​​other than 0 and 1 are reserved for future use by ISO / IEC. The decoder should allow values ​​of asps_extension_6bits that are not equal to 0 or 1 and must ignore all asps_extension_data_flag syntax elements in the asps. When it does not exist, the value of asps_extension_6bits is inferred to be equal to 0.

[0281] `asps_heterogeneous_miv_extension_present_flag` equal to 1 indicates that the `asps_heterogeneous_miv_extension()` syntax structure exists within the syntax structure. `asps_heterogeneous_miv_extension_present_flag` equal to 0 indicates that this syntax structure does not exist. When `asps_heterogeneous_miv_extension_present_flag` does not exist, its value is inferred to be 0.

[0282] ASPS extended syntax element semantics: ashm_geometry_3d_bit_depth_minus1 plus 1 indicates the bit depth of the geometric coordinates of the reconstructed volume content. ashm_geometry3d_bitdepth_minus1 should be between 0 and 31, inclusive.

[0283] Increasing 1 to ashm_geometry_2d_bit_depth_minus1 indicates the bit depth of the geometry when projected onto a 2D image. ashm_geometry2d_bit_depth_minus1 should be in the range of 0 to 31, inclusive.

[0284] Adding 4 to ashm_log2_max_atlas_frame_order_cnt_lsb_minus4 specifies the values ​​of the variables Log2MaxAtlasFrmOrderCntLsbForMiv and MaxAtlasFlmOrderCNTLsbForMIv, which are used to count the order of stitching frames during decoding, as shown below:

[0285] Log2MaxAtlasFrmOrderCntLsbForMiv=ashm_log2_max_atlas_frame_order_cnt_lsb_minus4+4

[0286] MaxAtlasFrmOrderCntLsbForMiv=2 Log2MaxAtlasFrmOrderCntLsbForMiv

[0287] Among them, ashm_log2_max_atlas_frame_order_cnt_lsb_minus4 takes values ​​from 0 to 12, including 0 and 31.

[0288] 2. Semantics of AFPS syntax elements:

[0289] An afps_extension_7bits value of 0 indicates that the afps_extension_data_flag syntax element does not exist in the AFPS RBSP syntax structure. If it exists, afps_extension_7bits in the bitstream should be equal to a value of 0 or 1, consistent with this version of the document. Values ​​of afps_extension_7bits other than 0 and 1 are reserved by ISO / IEC for future use. The decoder should allow values ​​of afps_extension_7bits that are not equal to 0 or 1 and should ignore the afps_extension_data_flag syntax element in AFPS. When afps_extension_7bits does not exist, its value is inferred to be 0.

[0290] A value of 1 for `afp_heterogeneous_miv_extension_present_flag` indicates the existence of the `afp_hyterogeneoos_miv_extension()` syntax structure within the AFPS syntax structure. A value of 0 for `afp_heterogeneous_miv_extension_present_flag` indicates that this syntax structure does not exist. When `afp_heterogeneous_miv_extension_present_flag` does not exist, its value is inferred to be 0.

[0291] For bitstreams that conform to this version of this document, afps_heterogeneous_miv_extension_present_flag and asps_heterigeneous_miv_extenson_present_flag should be consistent, meaning they must both exist and have the same value.

[0292] `afps_heterogeneous_frame_tile_toolset_miv_present_flag[i]` equal to 1 indicates that the i-th strip in the heterogeneous composite mosaic is a mosaic strip belonging to the miv (i.e., a multi-view video strip). `afps_heterogeneous_frame_tile_toolset_miv_present_flag[i]` equal to 0 indicates that the i-th tile in the heterogeneous composite mosaic is a mosaic strip belonging to the vpcc (i.e., a point cloud strip). When `afps_heterogeneous_frame_tile_toolset_miv_present_flag[i]` does not exist, its value is inferred to be 0.

[0293] AFPS Extended Syntax Element Semantics:

[0294] afhm_additional_lt_afoc_lsb_len represents the value of the variable MaxLtAtlasFrmOrderCntLsbForMiv used in the decoding of the reference stitched frame list, as shown below:

[0295] MaxLtAtlasFrmOrderCntLsbForMiv=

[0296] 2*(Log2MaxAtlasFrmOrderCntLsbForMiv+afhm_additional_lt_afoc_lsb_len)

[0297] The value of afhm_additional_lt_afoc_lsb_len should be between 0 and 32–Log2MaxAtlasFrmOrderCntLsbForMiv, inclusive.

[0298] When asps_long_term_ref_atlas_frames_flag equals 0, the value of afhm_additional_lt_afoc_lsb_len is inferred to be 0.

[0299] Semantics of data unit headers in spliced ​​image stripes.

[0300] `ath_atlas_frm_order_cnt_lsb` represents the frame order count of the current tiled tile, modulo `MaxAtlasFrmOrderCntLsb`. If `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tiled tile is 0, then the length of the `ath_atlas_frm_order_cnt_lsb` syntax element is equal to `Log2MaxAtlasFrmOrderCntLsb` bits. The value of `ath_atlas_frm_order_cnt_lsb` should be in the range of 0 to `MaxAtlasFrmOrderCntLsb-1`, inclusive. If `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tiled tile is 1, then the length of the `ath_atlas_frm_order_cnt_lsb` syntax element is equal to `Log2MaxAtlasFrmOrderCntLsbForMiv` bits. The value of ath_atlas_frm_order_cnt_lsb should be in the range of 0 to MaxAtlasFrmOrderCntLsbForMiv-1, inclusive.

[0301] `ath_additional_afoc_lsb_val[j]` specifies the value of `FullAtlasFrmOrderCntLsbLt[RlsIdx][j]` for the current mosaic stripe. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current mosaic stripe is equal to 0, then...

[0302] FullAtlasFrmOrderCntLsbLt[RlsIdx][j]=

[0303] ath_additional_afoc_lsb_val[j]*MaxAtlasFrmOrderCntLsb+afoc_lsb_lt[RlsIdx][j]

[0304] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then

[0305] FullAtlasFrmOrderCntLsbLt[RlsIdx][j]=

[0306] ath_additional_afoc_lsb_val[j]*MaxAtlasFrmOrderCntLsbForMiv+afoc_lsb_lt[RlsIdx][j]

[0307] ath_additional_afoc_lsb_val[j] is represented by the bit afps_additional_lt_afoc_lsb_len. When afps_additional_lt_afoc_lsb_len does not exist, the value of ath_additional_afoc_lsb_val[j] is inferred to be equal to 0.

[0308] The increment of 1 in `ath_raw_3d_offset_axis_bit_count_minus1` indicates the fixed bit width of the values ​​of the three syntax elements `rpdu_3d_offset_u[tileID][p]`, `rpdu_3d_offset_v[tileID][p]`, and `rpdu_3e_offset_d[tileID][p]`, where `p` indicates that the sub-tile index is `p`, and `tileID` indicates that the sub-tile is located in a stripe with a stripe ID equal to `tileID`.

[0309] If it exists, and if afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the length of the syntax element used to represent ath_raw_3d_offset_axis_bit_count_minus1 is equal to Floor(Log2(asps_geometry_3d_bit_depth_minus1+1)).

[0310] If it does not exist, the value of the syntax element of ath_raw_3d_offset_axis_bit_count_minus1 is inferred as

[0311] Max(0,asps_geometry_3d_bit_depth_minus1-asps_geometry_2d_bit_depth_minus1)-1.

[0312] The variable RawShift is defined as follows:

[0313] If afps_raw_3d_offset_bit_count_explicit_mode_flag = 1,

[0314] RawShift=asps_geometry_3d_bit_depth_minus1-ath_raw_3d_offset_axis_bit_count_minus1

[0315] otherwise

[0316] RawShift=asps_geometry_2d_bit_depth_minus1+1

[0317] If it exists, and if afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the length of the syntax element used to represent ath_raw_3d_offset_axis_bit_count_minus1 is equal to Floor(Log2(ashm_geometry_3d_bit_depth_minus1+1)).

[0318] If it does not exist, the value of the syntax element of ath_raw_3d_offset_axis_bit_count_minus1 is inferred as

[0319] Max(0,ashm_geometry_3d_bit_depth_minus1-ashm_geometry_2d_bit_depth_minus1)-1.

[0320] The variable RawShift is defined as follows:

[0321] If afps_raw_3d_offset_bit_count_explicit_mode_flag = 1,

[0322] RawShift=ashm_geometry_3d_bit_depth_minus1-ath_raw_3d_offset_axis_bit_count_minus1

[0323] otherwise

[0324] RawShift=ashm_geometry_2d_bit_depth_minus1+1

[0325] Sub-patch data unit semantics

[0326] pdu_3d_offset_u[tileID][p] represents the offset of the reconstructed sub-tile along the tangent axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with stripe index tileID.

[0327] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of pdu_3d_offset_u[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_u[tileID][p] is asps_geometry_3d_bit_depth_minus1+1.

[0328] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of pdu_3d_offset_u[tileID][p] should be between 0 and 2. ashm _geometry_3d_bit_depth_minus1+1The value used to represent the number of bits in pdu_3d_offset_u[tileID][p] is ashm_geometry_3d_bit_depth_minus1+1.

[0329] pdu_3d_offset_v[tileID][p] represents the offset of the reconstructed sub-tile along the dual tangent axes. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0330] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of pdu_3d_offset_v[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_v[tileID][p] is asps_geometry_3d_bit_depth_minus1+1.

[0331] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of pdu_3d_offset_v[tileID][p] should be between 0 and 2. ashm _geometry_3d_bit_depth_minus1+1 The value used to represent the number of bits in pdu_3d_offset_v[tileID][p] is ashm_geometry_3d_bit_depth_minus1+1.

[0332] `pdu_3d_offset_d[tileID][p]` represents the offset of the reconstructed sub-tile along the normal axis, where the current sub-tile belongs to the sub-tile with sub-tile index `p` within the strip with stripe index `tileID`. `Pdu3dOffsetD[tileID][p]` is defined as follows:

[0333] Pdu3dOffsetD[tileID][p]=pdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0334] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of Pdu3dOffsetD[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value used to represent the number of bits in pdu_3d_offset_v[tileID][p] is (asps_geometry_3d_bit_depth_minus1–ath_pos_min_d_quantizer+1).

[0335] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of Pdu3dOffsetD[tileID][p] should be between 0 and 2. ashm _geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value used to represent the number of bits in pdu_3d_offset_v[tileID][p] is (ashm_geometry_3d_bit_depth_minus1–ath_pos_min_d_quantizer+1).

[0336] `pdu_3d_range_d[tileID][p]` (if present) represents the expected maximum nominal offset in the bit-depth sub-tile geometry sample reconstructed along the normal axis after conversion to nominal representation. The current sub-tile belongs to the sub-tile with sub-tile index `p` within the strip with stripe index `tileID`. `Pdu3dRangeD[tileID][p]` is defined as follows:

[0337]

[0338]

[0339] If the current tiled image stripe's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, the variable rangeDBitDepth takes the following values:

[0340] rangeDBitDepth=Min(ashm_geometry_2d_bit_depth_minus1,ashm_geometry_3d_bit_depth_minus1)+1

[0341] If pdu_3d_range_d[tileID][p] does not exist, the value of Pdu3dRangeD[tileID][p] is inferred to be 2. rangeDBitDepth -1. If it exists, the value of Pdu3dRangeD[tileID][p] should be between 0 and 2. rangeDBitDepth The range is within -1 (inclusive).

[0342] This means that the number of bits in pdu_3d_range_d[tileID][p] is equal to (rangeDBitDepth–ath_pos_delta_max_d_quantizer).

[0343] Semantics of Merge Patch data unit

[0344] mpdu_3d_offset_u[tileID][p] indicates that the offset difference between two sub-tiles to be reconstructed along the tangent axis is to be applied. The two sub-tiles are the sub-tile with the strip index tileID and the sub-tile index p in the current mosaic and the sub-tile with the strip index tileID and the sub-tile index RefPatchIdx in the current mosaic.

[0345] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of mpdu_3d_offset_u[tileID][p] should be in the range (-2). asps _geometry_3d_bit_depth_minus1+1 +1) to 2 asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0346] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of mpdu_3d_offset_u[tileID][p] should be in the range (-2). ashm _geometry_3d_bit_depth_minus1+1 +1) to (2) ashm_geometry_3d_bit_depth_minus1+1 Within the range of –1) (inclusive).

[0347] If mpdu_3d_offset_u[tileID][p] does not exist, the value is inferred to be 0.

[0348] mpdu_3d_offset_v[tileID][p] indicates that the offset difference between two sub-tiles should be applied along the dual tangent axes for reconstruction. The two sub-tiles are the sub-tile with tileID and sub-tile index p in the current mosaic and the sub-tile with tileID and sub-tile index RefPatchIdx in the current mosaic.

[0349] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of mpdu_3d_offset_v[tileID][p] should be in the range (-2). asps _geometry_3d_bit_depth_minus1+1 +1) to 2 asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0350] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of mpdu_3d_offset_v[tileID][p] should be in the range (-2). ashm _geometry_3d_bit_depth_minus1+1 +1) to (2) ashm_geometry_3d_bit_depth_minus1+1 Within the range of –1) (inclusive).

[0351] If mpdu_3d_offset_v[tileID][p] does not exist, the value is inferred to be 0.

[0352] `mpdu_3d_offset_d[tileID][p]` represents the offset difference to be applied along the normal axis to the reconstruction between two sub-tiles, where the two sub-tiles are the sub-tile with strip index `tileID` and sub-tile index `p` in the current mosaic, and the sub-tile with strip index `tileID` and sub-tile index `RefPatchIdx` in the current mosaic. `Mpdu3dOffsetD[tileID][p]` is defined as follows:

[0353] Mpdu3dOffsetD[tileID][p]=mpdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0354] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of mpdu_3d_offset_d[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0355] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of mpdu_3d_offset_d[tileID][p] should be in the range (-2). ashm _geometry_3d_bit_depth_minus1+1 +1) to (2) ashm_geometry_3d_bit_depth_minus1+1 Within the range of –1) (inclusive).

[0356] If mpdu_3d_offset_d[tileID][p] does not exist, the value is inferred to be 0.

[0357] Inter-merge patch data unit semantics

[0358] ipdu_3d_offset_v[tileID][p] indicates that the offset difference between two sub-tiles should be applied along the dual tangent axes for reconstruction. The two sub-tiles are the sub-tile with the strip index tileID and sub-tile index p in the current mosaic and the sub-tile with the strip index tileID and sub-tile index RefPatchIdx in the current mosaic.

[0359] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of ipdu_3d_offset_v[tileID][p] should be in the range (-2). asps _geometry_3d_bit_depth_minus1+1 +1) to 2 asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0360] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of ipdu_3d_offset_v[tileID][p] should be in the range (-2). ashm _geometry_3d_bit_depth_minus1+1 +1) to (2) ashm_geometry_3d_bit_depth_minus1+1 Within the range of –1) (inclusive).

[0361] If ipdu_3d_offset_v[tileID][p] does not exist, the value is inferred to be 0.

[0362] `ipdu_3d_offset_d[tileID][p]` represents the offset difference to be applied along the normal axis to the reconstruction between two sub-tiles, where the two sub-tiles are the sub-tile with strip index `tileID` and sub-tile index `p` in the current mosaic, and the sub-tile with strip index `tileID` and sub-tile index `RefPatchIdx` in the current mosaic. `Mpdu3dOffsetD[tileID][p]` is defined as follows:

[0363] Ipdu3dOffsetD[tileID][p]=ipdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0364] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of ipdu_3d_offset_d[tileID][p] should be in the range (-2). asps _geometry_3d_bit_depth_minus1+1 +1) to 2 asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0365] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of ipdu_3d_offset_d[tileID][p] should be in the range (-2). ashm _geometry_3d_bit_depth_minus1+1 +1) to (2) ashm_geometry_3d_bit_depth_minus1+1 Within the range of –1) (inclusive).

[0366] If ipdu_3d_offset_d[tileID][p] does not exist, the value is inferred to be 0.

[0367] The specification in ISO / IEC 23090-5:2022 / Amd 1:- Sub-clause 8.4 applies to the following additional content.

[0368] The bitstream consistency requirement is that asps_geometry_3d_bit_depth_minus1 and asps_geometry_2d_bit-depth_minus1 are equal to gi_geometroy_3d_coordinates_bit_depth_minus1 and gi_geametry_2d_bit_depth_minus1, respectively. However, in special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then gi_geometry_3d_coordinates_bit_depth_minus1 and gi_geametry_2d_bit_depth_minus1 specifically refer to the contents of ISO / IEC 23090-5. ashm_geometroy_3d_bit-depth_minus1 and ashm_geometry_2d_bit_depth_minus1 do not necessarily have to be equal to gi_geominatory_3d_coordinates_bit__depth_nus1 and gi_geometriy_2d_pth_minos1.

[0369] Subplot data cell multi-view extension syntax and semantics

[0370] pdu_depth_occ_threshold[tileID][p] indicates that for sub-tiles with index p in a stripe whose index is equal to tileID, the occupancy value is set to unoccupied when it is below the threshold.

[0371] If the afps_heterogeneous_frame_tile_toolset_miv_present_flag of the current tile strip is equal to 0, then the number of bits in pdu_depth_occ_threshold[tileID][p] is equal to asps_geometry_2d_bit_depth_minus1+1.

[0372] If the afps_heterogeneous_frame_tile_toolset_miv_present_flag of the current tile strip is equal to 1, then the number of bits in pdu_depth_occ_threshold[tileID][p] is equal to ashm_geometry_2d_bit_depth_minus1+1.

[0373] If it does not exist, pdu_depth_occ_threshold[tileID][p] is inferred to be dq_depth_occ_threshold_default[pdu_projection_id[tileID][p]]. Note that pdu_projection_id[tileID][p] corresponds to the view ID of the subtile with index p, in the stripe with index tileID.

[0374] 3. Standard-related decoding design

[0375] (1) Counting the sequence of stitched frames during decoding

[0376] The output of this process is AtlasFrmOrderCntVal, which is the frame order count of the current mosaic stripe. The frame order count is used to identify the output order of the mosaic frames and for decoder consistency checks. Each encoded mosaic frame is associated with a frame order count variable, denoted as AtlasFrmOrderCntVal.

[0377] When the current stitched frame is not an IRAP-encoded stitched frame with NoOutputBeforeRecoveryFlag equal to 1, the variables prevAtlasFrmOrderCntLsb and prevAtrasFrmOrderCntMsb are derived as follows:

[0378] Let prevAtlasFrm be the previous stitched frame in the decoding order, whose TemporalID is equal to 0, and which is not a stitched frame encoded with RASL, RADL, or SLNR.

[0379] The variable prevAtlasFrmOrderCntLsb is set to the LSB value of the stitched image frame order count, which is equal to the value of prevAtrasFrm: atlas_frm_order_cnt_LSB.

[0380] The variable prevAtlasFrmOrderCntMsb is set to be equal to prevAtlasFrmAtlasFrmAtlasFrmaOrderCNTMsb.

[0381] The derivation of the variable AtlasFrmOrderCntMsb for the current stitched frame is as follows:

[0382] If the current mosaic is an IRAP-encoded mosaic and NoOutputBeforeRecoveryFlag is 1, then AtlasFrmaOrderCNTMsb is set to 0.

[0383] Otherwise, the derivation of AtlasFrmOrderCntMsb is as follows:

[0384]

[0385]

[0386] The derivation process of AtlasFrmOrderCntVal is as follows:

[0387] AtlasFrmOrderCntVal=AtlasFrmOrderCntMsb+ath_atlas_frm_order_cnt_lsb

[0388] The value of AtlasFrmOrderCntVal ranges from -2. 31 to 2 31 -1 (inclusive). In a CAS, the AtlasFrmOrderCntVal of any two stitched frames with the same nal_layer_id value are different.

[0389] The AtlasFrmOrderCnt(aFrmX) function is defined as follows:

[0390] AtlasFrmOrderCnt(aFrmX)=AtlasFrmOrderCntVal of the atlas frame aFrmX

[0391] The DiffAtlasFrmOrderCnt(aFrmA, aFrmB) function is defined as follows:

[0392] DiffAtlasFrmOrderCnt(aFrmA,aFrmB)=AtlasFrmOrderCnt(aFrmA)–AtlasFrmOrderCnt(aFrmB)

[0393] The bitstream should not contain values ​​that cause the DiffAtlasFrmOrderCnt(aFrmA, aFrmB) used during decoding to be outside the range of -2. 15 to 2 15 Data within the range of -1 (inclusive).

[0394] Note 1: Assuming X is the current stitched frame, and Y and Z are the other two stitched frames in the same CAS, when DiffAtlasFrmOrderCnt(X,Y) and Diffatlisfrmordercnt(X,Z) are both positive or both are negative, Y and Z are considered to be in the same output order direction as X.

[0395] (2) Reference splicing frame list processing process

[0396] This process is invoked at the start of the decoding process for each stitch strip of a stitched frame.

[0397] Reference mosaic frames are processed using reference indices. A reference index is an index in the Reference Atlas FrmList (RAFL). When decoding an I_TILE mosaic stripe, RAFL is not used to decode the stripe data. When decoding a SKIP_TILE or P_TILE mosaic stripe, a single Reference Atlas FrmList is used to decode the stripe data.

[0398] At the start of the decoding process for each mosaic strip, a RAFL RefAtlasFrmList is exported. RAFL is used for decoding the reference mosaic frame markers or mosaic strip data as specified in sub-clause 9.2.4.4.

[0399] Note 1: For I_TILE stripes of a stitched frame, RefAtlasFrmList can be used for bitstream consistency checks, but its derivation is not required for decoding the current stitched frame or for decoding stitches following the current stitched frame in the decoding order.

[0400] The reference splicing frame list RefAtlasFrmList is constructed as follows:

[0401]

[0402] The first NumRefIdxActive entry in RefAtlasFrmList is called the active entry in RefAtlasVrmList, and the other entries in RefAtlasFrmList are called inactive entries in RefAtrasFrmLists.

[0403] If the current stripe is SKIP_tile, then the array RefAtduTotalNumPatches is set to the array AtduToTotalNumPatches corresponding to the first entry in RefAtlasFrmList, RefAtlasFlmList[0].

[0404] Bitstream consistency requires the application of the following constraints:

[0405] –num_ref_entries[RlsIdx] must not be less than NumRefIdxActive.

[0406] The stitching frame referenced by each active entry in –RefAtlasFrmList should exist in DAB, and its time ID should be less than or equal to the time ID of the current stitching frame.

[0407] The stitched frame referenced by each entry in –RefAtlasFrmList should not be the current stitched frame.

[0408] – Short-term reference mosaic frame entries and long-term reference mosaic frame entries in the mosaic strip RefAtlasFrmList must not reference the same mosaic frame.

[0409] –RefAtlasFrmList should not contain an entry where the difference between the AtlasFrmaOrderCntVal of the current mosaic strip and the AtlasFlmOrderCNTVal of the mosaic frame pointed to by the entry is greater than or equal to 2. 24 Long-term reference mosaic frame entries.

[0410] - Ensure that `setOfRefAtlasFrms` is the only mosaic frame referenced by all entries in `RefAtlasFlmList`, and these entries have the same `nal_layer_id` as the current mosaic frame. The number of mosaic frames in `setOfRefAtlasFrms` should be less than or equal to `asps_max_dec_atlas_frame_buffering_minus1`, and `setOfrefAtlasFrms` should be the same for all mosaic stripes of the mosaic frame.

[0411] Each active entry in –RefAtlasFrmList should reference a mosaic frame with the exact same number of stripes as the current mosaic frame.

[0412] - All stripes in the current mosaic frame should contain the same reference mosaic frame in their RefAtlasFrmList, but there are no restrictions on the order of the reference mosaic frames.

[0413] – If the current mosaic frame (nal_layer_id equals the specific value layerID) is an IRAP-encoded mosaic, then the mosaic referenced by the entry in RefAtlasFrmList should not be located before any previous IRAP-encoded mosaic in either the output or decoding order (in the decoding order, when nal_layer_id equals layerID).

[0414] – When the current mosaic frame is not a RASL-encoded mosaic associated with a CRA-encoded mosaic where NoOutputBeforeRecoveryFlag equals 1, the mosaic referenced by the active entry in RefAtlasFrmList should not exist. This entry is generated during the decoding process to generate an unusable reference mosaic frame for the CRA-encoded mosaic associated with the current mosaic.

[0415] – When the current mosaic frame follows an IRAP-encoded mosaic that has the same nal_layer_id value in both the decoding and output order, the mosaic frame referenced by the active entry in RefAtlasFrmList must not be located before the IRAP-encoded mosaic in either the output or decoding order.

[0416] – When the current mosaic frame follows an IRAP-encoded mosaic with the same nal_layer_id value and all preceding mosaic frames (if any) associated with that IRAP-encoded mosaic in terms of decoding and output order, entries in RefAtlasFrmList should not reference mosaics preceding the IRAP-encoded mosaic in terms of output or decoding order.

[0417] – When the current mosaic frame is a RADL-encoded mosaic, there should be no active entry in RefAtlasFrmList. This entry is a mosaic frame that precedes the decoding order of the related IRAP-encoded mosaic of the RADL-encoded mosaic.

[0418] (3) General decoding process of sub-plot data units

[0419] TilePatch3dOffsetU[tileID][p] represents the offset of the reconstructed sub-tile along the tangent axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0420] If the current tile patch strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of TilePatch3dOffsetU[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is -1 (including the boundary).

[0421] If the current tile patch strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of TilePatch3dOffsetU[tileID][p] should be between 0 and (2).ashm _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0422] TilePatch3dOffsetV[tileID][p] represents the offset of the reconstructed sub-tile along the dual tangent axes. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0423] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of TilePatch3dOffsetV[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is -1 (including the boundary).

[0424] If the current tile patch strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of TilePatch3dOffsetV[tileID][p] should be between 0 and 2. ashm _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0425] TilePatch3dOffsetD[tileID][p] represents the offset of the reconstructed sub-tile along the normal axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0426] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, then the value of TilePatch3dOffsetD[tileID][p] should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0427] If the current tile strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of TilePatch3dOffsetD[tileID][p] should be between 0 and 2. ashm _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0428] TilePatch3dRangeD[tileID][p] (if present) represents the maximum expected nominal offset in the bit-depth sub-tile geometry sample reconstructed along the normal axis after conversion to nominal representation. The current sub-tile belongs to the sub-tile index p within the strip with stripe index tileID.

[0429] If the current tile's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 0, the variable...

[0430] rangeDBitDepth=Min(ashm_geometry_2d_bit_depth_minus1,ashm_geometry_3d_bit_depth_minus1)+1

[0431] If the current tile patch strip's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the variable rangeDBitDepth = Min(ashm_geometry_2d_bit_depth_minus1, ashm_geometry_3d_bit_depth_minus1) + 1. The value of TilePatch3dRangeD[tileID][p] ranges from 0 to 2. rangeDBitDepth The value is between -1 and inclusive.

[0432] 4. Standard related syntax restrictions.

[0433] Table 6 shows the maximum allowed syntax element values ​​for the V-PCC toolset profile components.

[0434] Table 6

[0435]

[0436]

[0437] Table 7. Maximum allowed syntax element values ​​for the heterogeneous toolset profile components.

[0438] Table 7

[0439]

[0440]

[0441] Table 8 shows the maximum allowed values ​​of syntax element values ​​for the MIV toolset profile components.

[0442] Table 8

[0443]

[0444]

[0445]

[0446] Table 9-1-1 Allowable values ​​of syntax element values ​​for the heterogeneous toolset profile components (Extended)

[0447] Table 9-1-1

[0448]

[0449]

[0450]

[0451] Table 9-1-2 Allowable values ​​of syntax element values ​​for the heterogeneous toolset profile components (Extended)

[0452] Table 9-1-2

[0453]

[0454]

[0455]

[0456] New extended syntax element design

[0457] Bitstream consistency: For each bitstream consistency test, all of the following conditions should be met:

[0458] 1. For each coded atlas access unit n (n > 0) associated with a buffered SEI message, let the variable deltaTime90k[n] be specified as follows:

[0459] deltaTime90k[n]=90000*(AuNominalRemovalTime[n]-AuFinalArrivalTime[n-1])

[0460] The value constraints for InitCabRemovalDelay[Htid][SchedSelIdx] are as follows:

[0461] – If hrd_cbr_flag[!NalHrdModeFlag][Htid][SchedSelIdx] equals 0, then the following condition is true:

[0462] InitCabRemovalDelay[Htid][SchedSelIdx]<=Ceil(deltaTime90k[n])

[0463] Otherwise, if hrd_cbr_flag[!NalHrdModeFlag][Htid][SchedSelIdx] equals 1, then the following condition is true:

[0464] Floor(deltaTime90k[n]) <=

[0465] InitCabRemovalDelay[Htid][SchedSelIdx]<=Ceil(deltaTime90k[n])

[0466] Note 1 – The exact number of bits per stitched frame deleted in the CAB depends on which buffer period SEI message is selected to initialize the HRD. The encoder should take this into account to ensure compliance with all specified constraints that the HRD can be initialized in any buffer period SEE message.

[0467] 2. CAB overflow is specified when the total number of bits in a CAB exceeds the size of the CAB. CABs must not overflow.

[0468] 3. When hrd_low_delay_flag[Htid] equals 0, CAB will never underflow. The CAB underflow rules are as follows:

[0469] –CAB underflow is specified as a condition that the nominal CAB removal time AuNominalRemovalTime[n] of the encoded mosaic access unit n is less than the final CAB arrival time AuFinalArrivalTime[n] of the encoded mosaic access unit n, and at least one of them has a value of n.

[0470] 4. The nominal removal time of the stitched frame from the CAB (starting from the second stitched frame in the decoding order) should meet the restrictions on AuNominalRemovalTime[n] and AuCabRemovalTime[n] in Annex A.

[0471] 5. For each current mosaic frame, after calling the procedure to remove the mosaic from the DAB as specified, the number of decoded mosaic frames in the DAB, including all mosaic frames n marked "for reference" or with AtlasFrameOutputFlag equal to 1 and AuCabRemovalTime[n] less than AuCabRemovalTime[currAtlasFrame], where currAtlasFrame is the current mosaic frame and should be less than or equal to asps_max_dec_alas_frame_buffering_minus1.

[0472] 6. When prediction is required, all reference mosaic frames should be present in the DAB. Each mosaic frame with AtlasFrameOutputFlag equal to 1 should appear in the DAB at the time of DAB output, unless removed from the DAB before its output time via one of the procedures specified in the terms.

[0473] 7. For each current mosaic frame that is not IRAP encoded and has NoOutputBeforeRecoveryFlag equal to 1, the value of maxAtlasFrameOrderCnt - minAtlasFrameOrderCnt should be less than MaxAtlasFrmOrderCntLsb / 2. If the current mosaic stripe's afps_heterogeneous_frame_tile_toolset_miv_present_flag is equal to 1, then the value of maxAtlasFrameOrderCnt - minAtlasFrameOrderCnt should be less than MaxAtlasFrmOrderCntLsbForMiv / 2.

[0474] 8. The value of DabOutputInterval[n] is the difference between the output time of a stitched frame and the output time of the first stitched frame after it, and AtlasFrameOutputFlag is equal to 1, which should satisfy the level specified in the bitstream for the configuration file, layer, and specified decoding process.

[0475] 9. For any two stitched frames m and n in the same CAS, when DabOutputTime[m] is greater than DabOutputTime[n], the AtlasFrmOrderCntVal of stitched frame m should be greater than the AtlasFrmOrderCntVal of stitched frame n.

[0476] Recovery point SEI message semantics

[0477] `recovery_afoccnt` specifies the recovery point of the decoded stitched frames in the output order. If a stitched frame `aFrmA` exists in the CAS that is after the current stitched frame in the decoding order (i.e., the stitched frame associated with the current SEI message), and its `AtlasFrmOrderCntVal` is equal to the current stitched frame's `AtlasFrmOrderCntVal` plus the value of `recovery_afoc_cnt`, then atlas frame `aFrmA` is called the recovery point stitched frame. Otherwise, the first stitched frame in the output order whose `AtlasFrmOrderCntVal` is greater than the current stitched frame's `AtlasFrmOrderCntVal` plus the value of `recovery_afoc_cnt` is called the recovery point stitched frame. The recovery point stitched frame should not be decoded before the current stitched frame. Starting from the output order position of the recovery point stitched frame, all decoded stitched frames displayed in output order are correct or approximately correct in content. The value of recovery_afoc_cnt should be within the range of MaxAtlasFrmOrderCntLsb / 2 to MaxAtlasFlmOrderCNTLsb / 2-1 (inclusive). If the afps_heterogeneous_frame_tile_toolset_miv_present_flag of the current mosaic frame strip is equal to 1, then the value of recovery_afoc_cnt should be within the range of MaxAtlasFrmOrderCntLsbForMiv / 2 to MaxAtlasFlmOrderCNTLsbForMIv / 2–1.

[0478] General ASPS-level strings

[0479] The ASPSCommonByteString(stringByte, posByte) function is defined as follows:

[0480]

[0481] VUI parameter semantics

[0482] `vui_display_box_origin[d]` specifies the offset along axis `d` relative to the origin of the coordinate system. If an element of `vui_ddisplay_box_origin[d]` is not present, its value should be assumed to be 0. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tile is 0, it indicates that the bit depth of `vui_display_box_origin[d]` is `asps_geometry_3d_bit_depth_minus1+1`. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tile is 1, it indicates that the bit depth of `vui_display_box_origin[d]` is `ashm_geometriy_3d_pit_depth_minus1+1`. Values ​​of `d` equal to 0, 1, and 2 correspond to the X, Y, and Z axes, respectively.

[0483] `vui_display_box_size[d]` specifies the size of the display box, sampled along axis `d`. Its value is unknown when an element of `vui_display_box_size[d]` is not present. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tiled pattern is 0, it indicates that the bit depth of `vui_display_box_size[d]` is `asps_geometry_3d_bit_depth_minus1+1`. If the `afps_hetrogeneous_frame_tile_toolset_miv_present_flag` of the current tiled pattern is 1, it indicates that the bit depth of `vui_display_box_size[d]` is `ashmgeometry3dbitdepth_minus1+1`.

[0484] The following variables are derived from the display box parameters:

[0485] minOffset[d]=vui_display_box_origin[d](G-1)

[0486] maxOffset[d]=vui_display_box_origin[d]+vui_display_box_size[d]

[0487] The values ​​of d are 0, 1, and 2, corresponding to the X, Y, and Z axes, respectively.

[0488] `vui_anchor_point[d]` represents the position of the anchor point along the d-axis. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tile strip is equal to 0, then the value of `vui_anchor_point[d]` should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The value should be in the range of -1. If `vui_anchor_point[d]` does not exist, it should be assumed to be equal to 0. The number of bits used to represent `vui_anchor_point[d]` is `asps_geometry_3d_bit_depth_minus1+1`. If the `afps_heterogeneous_frame_tile_toolset_miv_present_flag` of the current tile strip is equal to 1, then the value of `vui_anchor_point` should be between 0 and 2. ashm_geometroy_3d_pth_minos1+1 The range is -1. If vui_anchor_point[d] does not exist, it should be inferred to be equal to 0. The number of bits used to represent vui_ancor_point[d] is ashm_geometry_3d_bit_depth_minus1+1. The values ​​of d equal to 0, 1, and 2 correspond to the X, Y, and Z axes, respectively.

[0489] Multi-viewpoint standard, usable with ISO / IEC 23090-12

[0490] Deep expansion process

[0491] This process expands the integer depth values ​​of the stitched image to floating-point depth values ​​(e.g., meters) in scene coordinates.

[0492] Integer depth values ​​can be scaled to the implementation-defined bit depth and range 0…maxSampleD. Otherwise, maxSampleD is set to 2. asps_geometry_2d_bit_depth_minus1+1 -1.

[0493] In special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then maxSampleD is set to 2. ashm_geometriy_2d_pit_deptho_minus1+1 -1

[0494] Reconstructing the MPI process

[0495] This process decodes and reconstructs volumetric frames and MPI frames from a bitstream where ptc_restricted_geometry_flag equals 1.

[0496] Note – The reconstruction process reconstructs the entire MPI frame. This allows the viewport to be formed before projection onto it, without buffering the entire texture and transparency layer set.

[0497] The inputs to this process include:

[0498] - A list of view parameters, containing the internal and external parameters of the (unique) source view with index viewIdx;

[0499] -For each mosaic:

[0500] - The variable atlasID is the mosaic ID;

[0501] - The variables AspsFrameHeight[atlasID] and AspsFrameWidth[atlasID] represent the number of rows and columns of the stitched frames, respectively;

[0502] - Two-dimensional array AtlasBlockToPatchMap;

[0503] -Variable PatchPackingBlockSize;

[0504] The size of the three-dimensional array of -texFrame is 3×AspsFrameHeight[atlasID]×AspsFrameworkWidth[atlasID];

[0505] - The size of the transpFrame 2D data is AspsFrameHeight[atlasID] × AspsFrameWidth[atlasID];

[0506] Note – Transparency level 0 corresponds to a completely transparent sample, while the maximum transparency level is 2. (ai _attribute_2d_bit_depth_minus1[atlasID][attrIdx]+1) –1, where attrIdx is the index of the transparency attribute, corresponding to a completely opaque sample. The encoding rule is linear between the minimum and maximum transparency levels.

[0507] -External and internal parameters of the target view;

[0508] The variable `maxDepthSampleValue` represents the maximum value of the encoded geometry sample, and is set to 2. (asps_geometry_3d_bit_depth_minus1+1) -1; In special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then the variable maxDepthSampleValue, representing the maximum value of the encoded geometry samples, is set to 2. (ashm_geometry_3d_bit_depth_minus1+1) -1.

[0509] - The constant maxNbLayers represents the maximum depth layer of MPI, set to maxDepthSampleValue+1.

[0510] Decoding Case 4

[0511] 1. During parsing, the VPS is obtained from the V3C bitstream, and ptl_profile_toolset_idc is obtained from the VPS. If ptl_profile_toolset_idc = 128~133, it indicates that the current bitstream contains both point cloud and multi-viewpoint bitstreams.

[0512] 2. Parse high-level syntax such as ASPS and AFPS for each mosaic image;

[0513] a) In ASPS, first parse asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, asps_heterogeneous_miv_extension_present_flag (first sub-syntax element), and asps_extension_5bits. asps_vpcc_extension_present_flag is the flag for the point cloud, asps_miv_extension_present_flag is the flag for multiple views, and asps_heterogeneous_miv_extension_present_flag is the flag for both the point cloud and multiple views. In some embodiments, the value of asps_heterogeneous_miv_extension_present_flag is obtained by performing a bitwise AND operation between the values ​​of asps_vpcc_extension_present_flag and asps_miv_extension_present_flag. That is, it can be replaced by: first parsing asps_vpcc_extension_present_flag, asps_miv_extension_present_flag, and asps_extension_6bits in ASPS; then performing a bitwise AND operation between asps_vpcc_extension_present_flag and asps_miv_extension_present_flag to obtain asps_heterogeneous_miv_extension_present_flag.

[0514] b) Analyze asps_heterogeneous_miv_extension_present_flag:

[0515] i. If asps_heterogeneous_miv_extension_present_flag does not exist, it means that the current mosaic is isomorphic content (all stripes are point cloud type or multi-view type);

[0516] ii. If asps_heterogeneous_miv_extension_present_flag exists and is equal to 0, it means that the current mosaic is isomorphic content (all stripes are point cloud type or multi-viewpoint type);

[0517] iii. If asps_heterogeneous_miv_extension_present_flag exists and is equal to 1, it indicates that the current mosaic is heterogeneous content, containing both point cloud stripes and multi-view stripes. Therefore, the ASPS auxiliary high-level information of the current mosaic is split into two sub-sets: one sub-set for decoding multi-view stripes and the other sub-set for decoding point cloud stripes. The auxiliary information required for point cloud stripes can be obtained through parsing Part 8 of Standard 23090-5; the auxiliary information required for multi-view stripes can be obtained through parsing Part 8 of Standard 23090-5 and the newly added syntax asps_geometry_3d_bit_depth_minus1_for_miv and Part 8 of Standard 23090-12.

[0518] c) In AFPS, first parse afps_miv_extension_present_flag, afps_heterogeneous_type_extension_present_flag (the third sub-syntax element in the second syntax element), afps_heterogeneous_miv_extension_present_flag (the second sub-syntax element), and afps_extension_5 bits. In some embodiments, the value of afps_heterogeneous_miv_extension_present_flag is obtained by performing a bitwise AND operation between the values ​​of asps_vpcc_extension_present_flag and asps_miv_extension_present_flag; that is, this can be replaced by: first parse afps_miv_extension_present_flag, afps_heterogeneous_type_extension_present_flag, and afps_extension_6 bits in AFPS. Performing a bitwise AND operation between asps_vpcc_extension_present_flag and asps_miv_extension_present_flag yields afps_heterogeneous_miv_extension_present_flag.

[0519] d) Analyze afps_heterogeneous_miv_extension_present_flag:

[0520] i. If afps_heterogeneous_miv_extension_present_flag does not exist, it means that the current mosaic is isomorphic content (all stripes are point cloud type or multi-view type);

[0521] ii. If afps_heterogeneous_miv_extension_present_flag exists and is equal to 0, it means that the current mosaic is isomorphic content (all stripes are point cloud type or multi-view type);

[0522] iii. If `afps_heterogeneous_miv_extension_present_flag` exists and is equal to 1, it indicates that the current mosaic is heterogeneous content, containing both point cloud stripes and multi-view stripes. Therefore, the AFPS auxiliary high-level information of the current mosaic is split into two sub-sets: one sub-set for decoding multi-view stripes and the other sub-set for decoding point cloud stripes. The auxiliary information required for point cloud stripes can be obtained by parsing Part 8 of Standard 23090-5; the auxiliary information required for multi-view stripes can be obtained by parsing Part 8 of Standard 23090-5 and Part 8 of Standard 23090-12.

[0523] e) Analyze afps_heterogeneous_type_extension_present_flag:

[0524] i. Determine if afps_heterogeneous_type_extension_present_flag does not exist, indicating that all stripes in the current mosaic are of the same type;

[0525] ii. Otherwise, iterate through all tiles and determine if the i-th tile's afps_heterogeneous_tile_type[i] = 0, indicating that the current tile is a point cloud tile; or if the i-th tile's afps_heterogeneous_tile_type[i] = 1, indicating that the current tile is a multi-view tile, where afps_heterogeneous_tile_type[i] is the fourth sub-syntax element in the second syntax element; that is, when afps_heterogeneous_type_extension_present_flag exists (e.g., its value is 1), iterate through all titles based on afps_heterogeneous_tile_type[i] to determine the tile type of the i-th tile. It should be noted that the naming of the grammatical elements in this embodiment is mainly for ease of understanding and writing. They can be modified in actual applications and standard texts, but their semantic content should be consistent or similar. For example, afps_heterogeneous_type_extension_present_flag and afps_heterogeneous_frame_tile_toolset_miv_present_flag both indicate whether the current mosaic is a heterogeneous hybrid mosaic, and afps_heterogeneous_tile_type[i] and afps_heterogeneous_frame_tile_toolset_miv_present_flag[i] both indicate the stripe type of the i-th stripe.

[0526] The bitstream must ensure absolute consistency between afps_heterogeneous_miv_extension_present_flag and asps_heterogeneous_miv_extension_present_flag.

[0527] 3. Then, analyze the patch_data_unit information of each sub-plot in each strip. Given that the current strip uses multi-view auxiliary information, determine that the current sub-plot uses the multi-view video decoding standard. Given that the current strip uses point cloud auxiliary information, determine that the current sub-plot uses the point cloud video decoding standard.

[0528] For detailed explanations of specific grammatical elements, please refer to Tables 1, 2-1, 3, 4-1, VI, 6, 7-1, 8, 9-1-1-1, and 9-1-2-1.

[0529] Table 1 shows an example of available toolset profile components. Table 1 provides a list of toolset profile components defined for V3C and their corresponding identifier syntax element values, such as `ptl_profile_toolset_idc` and `ptc_one_v3c_frame_only_flag`. This definition is for use only in this document. The syntax element `ptl_profile_toolset_idc` provides the main definition of the toolset profile. Additional syntax elements such as `ptc_one_v3c_frame_only_flag` can specify additional features or limitations of the defined profile. `ptc_one_v3c_frame_only_flag` can be used to support only a single V3C frame. It should be noted that values ​​2..63, 67..127, 134..255 in `ptl_profile_toolset_idc` are reserved and currently undefined; the standards organization may specify them in future standards. The profile types defined in Table 1 can include dynamic or static.

[0530] Table 2-1 shows the RBSP syntax for the General atlas sequence parameter set, which can be used in ISO / IEC 23090-5. The extended syntax element `asps_vpcc_extension_present_flag` indicates whether the atlas belongs to a point cloud; `asps_miv_extension_present_flag` indicates whether it belongs to a multi-viewpoint atlas; and `asps_heterogeneous_miv_extension_present_flag` indicates the atlas type. Specifically, the value of this syntax element determines whether the atlas belongs to a point cloud / multi-viewpoint atlas / point cloud + multi-viewpoint atlas. The first extended syntax element `asps_geometry_3d_bit_depth_minus1_for_miv` determines the bit depth of the geometric coordinates for the reconstructed geometric content. Decoding Case 4 only requires adding the first extended syntax element to obtain the auxiliary information needed for the multi-viewpoint stripes.

[0531] Table 2-1

[0532]

[0533]

[0534] Table 4-1 shows the RBSP syntax for the General atlas frame parameter set, which can be used in ISO / IEC 23090-5. `afps_miv_extension_present_flag` indicates whether the mosaic belongs to a multi-viewpoint, and `afps_heterogeneous_miv_extension_present_flag` indicates the mosaic type. Specifically, the value of this syntax element determines whether the mosaic should belong to point cloud / multi-viewpoint / point cloud + multi-viewpoint. `afps_heterogeneous_type_extension_present_flag` is used to determine whether all stripes of the current mosaic are of the same type. If not, each stripe is traversed according to `afps_heterogeneous_tile_type[i]` to determine the stripe type.

[0535] Table 4-1

[0536]

[0537]

[0538] The semantics of ASPS and AFPS syntax elements are explained below.

[0539] 1. Semantics of ASPS syntax elements:

[0540] A value of 1 for `asps_extension_present_flag` indicates that the syntax elements `asps_vpcc_extension_present_flag`, `asps_miv_extension_present_flag`, `asps_hetergeous_miv_extension_present_flag`, and `asps_extension_5` exist in the `atlas_sequence_parameter_set_rbsp()` syntax structure. A value of 0 for `asps_extension_present_flag` indicates that the syntax elements `asps_vpcc_extension_present_flag`, `asps_miv_extension_present_flag`, `asps_hetergeous_miv_extension_present_flag`, and `asps_extension_5` do not exist.

[0541] A value of 1 for `asps_terogeneous_miv_extension_present_flag` indicates that the `asps_geometry_3d_bit_depth_minus1_for_miv` syntax element exists within the `atlas_sequence_parameter_set_rbsp()` syntax structure. A value of 0 for `asps_terogeneous_miv_extension_present_flag` indicates that this syntax does not exist. If it does not exist, it is inferred that the value of `asps_terogeneous_miv_extension_present_flag` is equal to 0.

[0542] A value of 0 for `asps_extension_5bits` indicates that the syntax element `asps_extension_data_flag` does not exist in the asps RBSP syntax structure. If it exists, `asps_extension_5bits` should be equal to 0 in the bitstream conforming to this version of the document. Values ​​of `asps_extension_5bits` not equal to 0 are reserved for future use by ISO / IEC. The decoder should allow values ​​of `asps_extension_5bits` not equal to 0 and should ignore all `asps_extension_data_flag` syntax elements in the ASPS NAL unit. When it does not exist, the value of `asps_extension_5bits` is inferred to be equal to 0.

[0543] `asps_geometry_3d_bit_depth_minus1_for_miv` represents the bit depth of the geometric coordinates for reconstructing the volume content. `asps_geometry_3d_bit_depth_minus1_for_miv` should be in the range of 0 to 31 (inclusive).

[0544] 2. Semantics of AFPS syntax elements:

[0545] A value of 1 for `afs_extension_present_flag` indicates that the syntax elements `afs_miv_extension_present_fag`, `afs_heterogeneous_type_extension_present_flag`, `afs_heterogeneous_miv_expansion_ppresent_fag`, and `afs_expansion_5bits` exist in the `atlas_frame_parame_parame_set_rbsp()` syntax structure. A value of 0 for `afs_extension_present_flag` indicates that the syntax elements `afs_miv_extension_present_fag`, `afs_heterogeneous_type_extension_present_flag`, `afs_heterogeneous_miv_eExtension_ppresent_fag`, and `afs_eextension_5bits` do not exist.

[0546] `afps_heterogeneous_type_extension_present_flag` equal to 1 indicates that stripes referencing this Afps include heterogeneous types. `afps_heterogeneous_type_extension_present_flag` equal to 0 indicates that every stripe referencing this Afps includes the same type. If it does not exist, the value of `afps_heterogeneous_type_extension_present_flag` is inferred to be 0.

[0547] `asps_heterogeneous_miv_extension_present_flag` equal to 1 indicates that the XXXXX syntax element exists in the `atlas_frame_parame_parame_set_rbsp()` syntax structure. `asps_heterogeneous_miv_extension_present_flag` equal to 0 indicates that the XXXXX syntax element does not exist. If it does not exist, then it is inferred that the value of `asps_heterogeneous_miv_extension_present_flag` is equal to 0. For bitstreams conforming to this version of this document, both `asps_heterogeneous_miv_extension_present_flag` and `asps_heterogeneous_miv_extension_present_flag` should appear.

[0548] A value of 0 for `afps_extension_5bits` indicates that the syntax element `afps_extension_data_flag` does not exist in the AFPS RBSP syntax structure. If it exists, `afps_extension_5bits` should be equal to 0 in the bitstream conforming to this version of the document. Values ​​of `afps_extension_5bits` not equal to 0 are reserved for future use by ISO / IEC. The decoder should allow values ​​of `afps_extension_5bits` not equal to 0 and should ignore all `afps_extension_data_flag` syntax elements in the AFPS NAL unit. When it does not exist, the value of `afps_extension_5bits` is inferred to be equal to 0.

[0549] `afps_heterogeneous_tile_type[i]` represents the stripe type for which `tileID` equals `i` as specified in Table VI. This indicates a reserved value intended for future use by ISO / IEC and should not appear in bitstreams conforming to this version of this document. Decoders conforming to this document should ignore such reserved stripe types.

[0550] Table VI. Strip Types

[0551] afps_heterogeneous_tile_type[i] Strip type

[0552] 1 MIV 2…3 reserve

[0553] Semantics of the header of the spliced ​​image strip data cell:

[0554] The increment of 1 in `ath_raw_3d_offset_axis_bit_count_minus1` indicates the fixed bit width of the values ​​of the three syntax elements `rpdu_3d_offset_u[tileID][p]`, `rpdu_3d_offset_v[tileID][p]`, and `rpdu_3e_offset_d[tileID][p]`, where `p` indicates that the sub-tile index is `p`, and `tileID` indicates that the sub-tile is located in a stripe with a stripe ID equal to `tileID`.

[0555] If it exists, and if the current stripe's afps_heterogeneous_tile_type is equal to 0, then the length of the syntax element used to represent ath_raw_3d_offset_axis_bit_count_minus1 is equal to Floor(Log2(asps_geometry_3d_bit_depth_minus1+1)).

[0556] If it does not exist, the value of the syntax element of ath_raw_3d_offset_axis_bit_count_minus1 is inferred as

[0557] Max(0,asps_geometry_3d_bit_depth_minus1-asps_geometry_2d_bit_depth_minus1)-1.

[0558] The variable RawShift is defined as follows:

[0559] If afps_raw_3d_offset_bit_count_explicit_mode_flag = 1,

[0560] RawShift=asps_geometry_3d_bit_depth_minus1-ath_raw_3d_offset_axis_bit_count_minus1

[0561] otherwise

[0562] RawShift=asps_geometry_2d_bit_depth_minus1+1

[0563] If it exists, and if the current stripe's afps_heterogeneous_tile_type is equal to 1, then the length of the syntax element used to represent ath_raw_3d_offset_axis_bit_count_minus1 is equal to Floor(Log2(asps_geometry_3d_bit_depth_minus1_for_miv+1)).

[0564] If it does not exist, the value of the syntax element of ath_raw_3d_offset_axis_bit_count_minus1 is inferred as

[0565] Max(0,asps_geometry_3d_bit_depth_minus1_for_miv-asps_geometry_2d_bit_depth_minus1)- 1.

[0567] The variable RawShift is defined as follows:

[0568] If afps_raw_3d_offset_bit_count_explicit_mode_flag = 1,

[0569] RawShift=asps_geometry_3d_bit_depth_minus1_for_miv-ath_raw_3d_offset_axis_bit_count_minus1

[0570] otherwise

[0571] RawShift=asps_geometry_2d_bit_depth_minus1+1

[0572] Sub-patch data unit semantics

[0573] pdu_3d_offset_u[tileID][p] represents the offset of the reconstructed sub-tile along the tangent axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with stripe index tileID.

[0574] Sub-patch data unit semantics

[0575] pdu_3d_offset_u[tileID][p] represents the offset of the reconstructed sub-tile along the tangent axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with stripe index tileID.

[0576] If the current tile strip's afps_heterogeneous_tile_type is equal to 0, then the value of pdu_3d_offset_u[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_u[tileID][p] is asps_geometry_3d_bit_depth_minus1+1.

[0577] If the current tile strip's afps_heterogeneous_tile_type is equal to 1, then the value of pdu_3d_offset_u[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_u[tileID][p] is asps_geometry_3d_bit_depth_minus1_for_miv+1.

[0578] pdu_3d_offset_v[tileID][p] represents the offset of the reconstructed sub-tile along the dual tangent axes. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0579] If the current tile type afps_heterogeneous_tile_type is equal to 0, then the value of pdu_3d_offset_v[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_v[tileID][p] is asps_geometry_3d_bit_depth_minus1+1.

[0580] If the current tile type afps_heterogeneous_tile_type is equal to 1, then the value of pdu_3d_offset_v[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is -1 (inclusive). The value of the number of bits used to represent pdu_3d_offset_v[tileID][p] is asps_geometry_3d_bit_depth_minus1_for_miv+1.

[0581] `pdu_3d_offset_d[tileID][p]` represents the offset of the reconstructed sub-tile along the normal axis, where the current sub-tile belongs to the sub-tile with sub-tile index `p` within the strip with stripe index `tileID`. `Pdu3dOffsetD[tileID][p]` is defined as follows:

[0582] Pdu3dOffsetD[tileID][p]=pdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0583] If the current tile type afps_heterogeneous_tile_type is equal to 0, then the value of Pdu3dOffsetD[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is -1 (inclusive). The value used to represent the number of bits in pdu_3d_offset_v[tileID][p] is (asps_geometry_3d_bit_depth_minus1–ath_pos_min_d_quantizer+1).

[0584] If the current tile type afps_heterogeneous_tile_type is equal to 1, then the value of Pdu3dOffsetD[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is -1 (inclusive). The value used to represent the number of bits in pdu_3d_offset_v[tileID][p] is (asps_geometry_3d_bit_depth_minus1_for_miv–ath_pos_min_d_quantizer+1).

[0585] `pdu_3d_range_d[tileID][p]` (if present) represents the expected maximum nominal offset in the bit-depth sub-tile geometry sample reconstructed along the normal axis after conversion to nominal representation. The current sub-tile belongs to the sub-tile with sub-tile index `p` within the strip with stripe index `tileID`. `Pdu3dRangeD[tileID][p]` is defined as follows:

[0586]

[0587] If the afps_heterogeneous_tile_type of the current tile pattern strip is equal to 0, the variable rangeDBitDepth takes the following values:

[0588] rangeDBitDepth=Min(asps_geometry_2d_bit_depth_minus1,asps_geometry_3d_bit_depth_minus1_for_miv)+1

[0589] If pdu_3d_range_d[tileID][p] does not exist, the value of Pdu3dRangeD[tileID][p] is inferred to be 2. rangeDBitDepth -1. If it exists, the value of Pdu3dRangeD[tileID][p] should be between 0 and 2. rangeDBitDepth The range is within -1 (inclusive).

[0590] This means that the number of bits in pdu_3d_range_d[tileID][p] is equal to (rangeDBitDepth–ath_pos_delta_max_d_quantizer).

[0591] Semantics of Merge Patch data unit

[0592] mpdu_3d_offset_u[tileID][p] indicates that the offset difference between two sub-tiles to be reconstructed along the tangent axis is to be applied. The two sub-tiles are the sub-tile with the strip index tileID and the sub-tile index p in the current mosaic and the sub-tile with the strip index tileID and the sub-tile index RefPatchIdx in the current mosaic.

[0593] If the current tile strip's afps_heterogeneous_tile_type is equal to 0, then the value of mpdu_3d_offset_u[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0594] If the current tile strip's afps_heterogeneous_tile_type is equal to 1, then the value of mpdu_3d_offset_u[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1_for_miv+1 +1) to (2) asps _geometry_3d_bit_depth_minus1_for_miv+1 Within the range of –1) (inclusive).

[0595] If mpdu_3d_offset_u[tileID][p] does not exist, the value is inferred to be 0.

[0596] mpdu_3d_offset_v[tileID][p] indicates that the offset difference between two sub-tiles should be applied along the dual tangent axes for reconstruction. The two sub-tiles are the sub-tile with tileID and sub-tile index p in the current mosaic and the sub-tile with tileID and sub-tile index RefPatchIdx in the current mosaic.

[0597] If the current tile strip's afps_heterogeneous_tile_type is equal to 0, then the value of mpdu_3d_offset_v[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0598] If the current tile pattern stripe's afps_heterogeneous_tile_type is equal to 1, then the value of mpdu_3d_offset_v[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1_for_miv+1 +1) to (2) asps _geometry_3d_bit_depth_minus1_for_miv+1 Within the range of –1) (inclusive).

[0599] If mpdu_3d_offset_v[tileID][p] does not exist, the value is inferred to be 0.

[0600] `mpdu_3d_offset_d[tileID][p]` represents the offset difference to be applied along the normal axis to the reconstruction between two sub-tiles, where the two sub-tiles are the sub-tile with strip index `tileID` and sub-tile index `p` in the current mosaic, and the sub-tile with strip index `tileID` and sub-tile index `RefPatchIdx` in the current mosaic. `Mpdu3dOffsetD[tileID][p]` is defined as follows:

[0601] Mpdu3dOffsetD[tileID][p]=mpdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0602] If the current tile pattern stripe's afps_heterogeneous_tile_type is equal to 0, then the value of mpdu_3d_offset_d[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0603] If the current tile pattern stripe's afps_heterogeneous_tile_type is equal to 1, then the value of mpdu_3d_offset_d[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1_for_miv+1 +1) to (2) asps _geometry_3d_bit_depth_minus1_for_miv+1 Within the range of –1) (inclusive).

[0604] If mpdu_3d_offset_d[tileID][p] does not exist, the value is inferred to be 0.

[0605] Inter-merge patch data unit semantics

[0606] ipdu_3d_offset_v[tileID][p] indicates that the offset difference between two sub-tiles should be applied along the dual tangent axes for reconstruction. The two sub-tiles are the sub-tile with the strip index tileID and sub-tile index p in the current mosaic and the sub-tile with the strip index tileID and sub-tile index RefPatchIdx in the current mosaic.

[0607] If the current tile strip's afps_heterogeneous_tile_type is equal to 0, then the value of ipdu_3d_offset_v[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0608] If the current tile strip's afps_heterogeneous_tile_type is equal to 1, then the value of ipdu_3d_offset_v[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1_for_miv+1 +1) to (2) asps _geometry_3d_bit_depth_minus1_for_miv+1 Within the range of –1) (inclusive).

[0609] If ipdu_3d_offset_v[tileID][p] does not exist, the value is inferred to be 0.

[0610] `ipdu_3d_offset_d[tileID][p]` represents the offset difference to be applied along the normal axis to the reconstruction between two sub-tiles, where the two sub-tiles are the sub-tile with strip index `tileID` and sub-tile index `p` in the current mosaic, and the sub-tile with strip index `tileID` and sub-tile index `RefPatchIdx` in the current mosaic. `Mpdu3dOffsetD[tileID][p]` is defined as follows:

[0611] Ipdu3dOffsetD[tileID][p]=ipdu_3d_offset_d[tileID][p]< <ath_pos_min_d_quantizer

[0612] If the current tile pattern stripe's afps_heterogeneous_tile_type is equal to 0, then the value of ipdu_3d_offset_d[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1+1 +1) to 2 asps _geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0613] If the current tile pattern stripe's afps_heterogeneous_tile_type is equal to 1, then the value of ipdu_3d_offset_d[tileID][p] should be in the range (-2). asps_geometry_3d_bit_depth_minus1_for_miv+1 +1) to (2) asps _geometry_3d_bit_depth_minus1_for_miv+1 Within the range of –1) (inclusive).

[0614] If ipdu_3d_offset_d[tileID][p] does not exist, the value is inferred to be 0.

[0615] The specification in ISO / IEC 23090-5:2022 / Amd 1:- Sub-clause 8.4 applies to the following additional content.

[0616] The bitstream consistency requirement is that asps_geometry_3d_bit_depth_minus1 and asps_geometry_2d_bit-depth_minus1 are equal to gi_geometroy_3d_coordinates_bit_depth_minus1 and gi_geametry_2d_bit_depth_minus1, respectively. However, in special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then gi_geometry_3d_coordinates_bit_depth_minus1 and gi_geametry_2d_bit_depth_minus1 specifically refer to the contents of ISO / IEC 23090-5. asps_geometry_3d_bit_depth_minus1_for_miv does not necessarily have to be equal to gi_geominatory_3d_coordinates_bit__depth_nus1 and gi_geometriy_2d_pth_minos1.

[0617] (3) General decoding process of sub-plot data units

[0618] TilePatch3dOffsetU[tileID][p] represents the offset of the reconstructed sub-tile along the tangent axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0619] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 0, then the value of TilePatch3dOffsetU[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is -1 (including the boundary).

[0620] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 1, then the value of TilePatch3dOffsetU[tileID][p] should be between 0 and (2). asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is within -1 (inclusive).

[0621] TilePatch3dOffsetV[tileID][p] represents the offset of the reconstructed sub-tile along the dual tangent axes. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0622] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 0, then the value of TilePatch3dOffsetV[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is -1 (including the boundary).

[0623] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 1, then the value of TilePatch3dOffsetV[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is within -1 (inclusive).

[0624] TilePatch3dOffsetD[tileID][p] represents the offset of the reconstructed sub-tile along the normal axis. The current sub-tile belongs to the sub-tile with sub-tile index p in the strip with tile ID.

[0625] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 0, then the value of TilePatch3dOffsetD[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1+1 The range is within -1 (inclusive).

[0626] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 1, then the value of TilePatch3dOffsetD[tileID][p] should be between 0 and 2. asps_geometry_3d_bit_depth_minus1_for_miv+1 The range is within -1 (inclusive).

[0627] TilePatch3dRangeD[tileID][p] (if present) represents the maximum expected nominal offset in the bit-depth sub-tile geometry sample reconstructed along the normal axis after conversion to nominal representation. The current sub-tile belongs to the sub-tile index p within the strip with stripe index tileID.

[0628] If the afps_heterogeneous_tile_type of the current tile pattern strip is equal to 0, the variable...

[0629] rangeDBitDepth=Min(asps_geometry_2d_bit_depth_minus1,asps_geometry_3d_bit_depth_minus1_for_miv)+1

[0630] If the current tile pattern strip's afps_heterogeneous_tile_type is equal to 1, then the variable rangeDBitDepth = Min(asps_geometry_2d_bit_depth_minus1, asps_geometry_3d_bit_depth_minus1_for_miv) + 1. The value of TilePatch3dRangeD[tileID][p] ranges from 0 to 2. rangeDBitDepth -1 to (inclusive).

[0631] 4. Standard related syntax restrictions.

[0632] See Table 6 for the maximum allowed syntax element values ​​for the V-PCC toolset profile components.

[0633] Table 7-1 Maximum allowed syntax element values ​​for the heterogeneous toolset profile components (Extended)

[0634] Table 7-1

[0635]

[0636] See Table 8 for the maximum allowed syntax element values ​​for the MIV toolset profile components.

[0637] Table 9-1-1-1 Allowable values ​​of syntax element values ​​for the heterogeneous toolset profile components (Extended)

[0638] Table 9-1-1

[0639]

[0640]

[0641]

[0642] Table 9-1-2-1 Allowable values ​​of syntax element values ​​for the heterogeneous toolset profile components (Extended)

[0643] Table 9-1-2-1

[0644]

[0645]

[0646]

[0647] The following limitations apply to bitstreams or V3C sub-bitstream sets that conform to the MIV Extended Hybrid VPCC Extended Toolset configuration file components:

[0648] - For each stitched image, all syntax elements in asps except asps_geometry_3d_bit_depth_minus1 should have the same value for both MIV and VPCC.

[0649] - For each stitching graph, if asps_vpcc_extension_present_flag is equal to 1, then asps_terogeneous_miv_extension_present_flag, afps_heterogeneous_miv_extention_present_flag, and afps_hettogeneous_type_extension_present_flag exist and their values ​​should be equal to 1.

[0650] - For each stitched graph, if asps_vpcc_extension_present_flag equals 0, then asps_terogeneous_miv_extension_present_flag, afps_heterogeneous_miv_extention_present_flag, and afps_hettogeneous_type_extension_present_flag do not exist.

[0651] General ASPS-level strings

[0652] The ASPSCommonByteString(stringByte, posByte) function is defined as follows:

[0653]

[0654] VUI parameter semantics

[0655] `vui_display_box_origin[d]` specifies the offset along axis `d` relative to the origin of the coordinate system. If an element of `vui_ddisplay_box_origin[d]` is not present, its value should be assumed to be 0. If the `afps_heterogeneous_tile_type` of the current tile is 0, the bit depth of `vui_display_box_origin[d]` is `asps_geometry_3d_bit_depth_minus1+1`. If the `afps_heterogeneous_tile_type` of the current tile is 1, the bit depth of `vui_display_box_origin[d]` is `asps_geometry_3d_bit_depth_minus1_for_miv+1`. Values ​​of `d` equal to 0, 1, and 2 correspond to the X, Y, and Z axes, respectively.

[0656] `vui_display_box_size[d]` specifies the size of the display box, sampled along axis `d`. Its value is unknown when no element of `vui_display_box_size[d]` exists. If the `afps_heterogeneous_tile_type` of the current tiled strip is 0, it indicates that the bit depth of `vui_display_box_size[d]` is `asps_geometry_3d_bit_depth_minus1+1`. If the `afps_heterogeneous_tile_type` of the current tiled strip is 1, it indicates that the bit depth of `vui_display_box_size[d]` is `asps_geometry_3d_bit_depth_minus1_for_miv+1`.

[0657] The following variables are derived from the display box parameters:

[0658] minOffset[d]=vui_display_box_origin[d](G-1)

[0659] maxOffset[d]=vui_display_box_origin[d]+vui_display_box_size[d]

[0660] The values ​​of d are 0, 1, and 2, corresponding to the X, Y, and Z axes, respectively.

[0661] A value of 1 for `vui_anchor_point_present_flag` indicates that the `vui_anchor_point[d]` syntax element exists within the `vui_parameters()` syntax structure. A value of 0 for `vui_anchor_point_present_flag` indicates that the `vui_anchor_point[d]` syntax element does not exist.

[0662] `vui_anchor_point[d]` represents the position of the anchor point along the d-axis. If the `afps_heterogeneous_tile_type` of the current tile pattern strip is equal to 0, then the value of `vui_anchor_point[d]` should be between 0 and 2. asps _geometry_3d_bit_depth_minus1+1The value of vui_anchor_point[d] should be in the range of -1. If vui_anchor_point[d] does not exist, it should be assumed to be equal to 0. The number of bits used to represent vui_anchor_point[d] is asps_geometry_3d_bit_depth_minus1+1. If the afps_heterogeneous_tile_type of the current mosaic strip is equal to 1, then the value of vui_anchor_point should be in the range of 0 to 2. asps_geometry_3d_bit_depth_minus1_for_miv The range is -1. If vui_anchor_point[d] does not exist, it should be inferred to be equal to 0. The number of bits used to represent vui_ancor_point[d] is asps_geometry_3d_bit_depth_minus1_for_miv+1. The d values ​​of 0, 1, and 2 correspond to the X, Y, and Z axes, respectively.

[0663] Multi-viewpoint standard, usable with ISO / IEC 23090-12

[0664] Deep expansion process

[0665] This process expands the integer depth values ​​of the stitched image to floating-point depth values ​​(e.g., meters) in scene coordinates.

[0666] Integer depth values ​​can be scaled to the implementation-defined bit depth and range 0…maxSampleD. Otherwise, maxSampleD is set to 2. asps_geometry_2d_bit_depth_minus1+1 -1.

[0667] In special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then maxSampleD is set to 2. ashm_geometriy_2d_pit_deptho_minus1+1 -1

[0668] Reconstructing the MPI process

[0669] This process decodes and reconstructs volumetric frames and MPI frames from a bitstream where ptc_restricted_geometry_flag equals 1.

[0670] Note – The reconstruction process reconstructs the entire MPI frame. This allows the viewport to be formed before projection onto it, without buffering the entire texture and transparency layer set.

[0671] The inputs to this process include:

[0672] - A list of view parameters, containing the internal and external parameters of the (unique) source view with index viewIdx;

[0673] -For each mosaic:

[0674] - The variable atlasID is the mosaic ID;

[0675] - The variables AspsFrameHeight[atlasID] and AspsFrameWidth[atlasID] represent the number of rows and columns of the stitched frames, respectively;

[0676] - Two-dimensional array AtlasBlockToPatchMap;

[0677] -Variable PatchPackingBlockSize;

[0678] The size of the three-dimensional array of -texFrame is 3×AspsFrameHeight[atlasID]×AspsFrameworkWidth[atlasID];

[0679] - The size of the transpFrame 2D data is AspsFrameHeight[atlasID] × AspsFrameWidth[atlasID];

[0680] Note – Transparency level 0 corresponds to a completely transparent sample, while the maximum transparency level is 2. (ai _attribute_2d_bit_depth_minus1[atlasID][attrIdx]+1) –1, where attrIdx is the index of the transparency attribute, corresponding to a completely opaque sample. The encoding rule is linear between the minimum and maximum transparency levels.

[0681] -External and internal parameters of the target view;

[0682] The variable `maxDepthSampleValue` represents the maximum value of the encoded geometry sample, and is set to 2. (asps _geometry_3d_bit_depth_minus1+1) -1; In special cases, if asps_heterogeneous_miv_extension_present_flag equals 1, then the variable maxDepthSampleValue, representing the maximum value of the encoded geometry samples, is set to 2. (asps_geometry_3d_bit_depth_minus1_for_miv+1) -1.

[0683] - The constant maxNbLayers represents the maximum depth layer of MPI, set to maxDepthSampleValue+1.

[0684] For bitstreams that integrate multi-viewpoint and point cloud data, it's not possible to have multiple stripes within a single stitched image, each stripe being a collection of sub-block images from either the multi-viewpoint or point cloud. Existing standards can only accommodate one type of stripe within a single stitched image. Therefore, it's necessary to extend relevant standards to distinguish whether a stitched image simultaneously contains both multi-viewpoint and point cloud stripes. This application's embodiment adds two syntax elements, asps_heterogeous_miv_extension_present_flag and afps_heterogeneous_miv_extension_present_flag, for stitched image level parameters (AFPS) and stitched image sequence level parameters (ASPS), thereby determining whether the bitstream simultaneously contains point clouds and multi-viewpoint data. However, unlike decoding case three, decoding case four includes the heterogeneously added ASPS and AFPS syntax elements related to multi-viewpoint data in the ASPS and AFPS parsing, while decoding case three packages these related syntax elements into a new parameter set for parsing.

[0685] In addition, in decoding case four, the task of determining whether each stripe needs to be of type is represented by a new syntax element afps_heterogeneous_type_extension_present_flag.

[0686] This application's embodiments are used to implement an encoding and decoding scheme for multi-view stitched images, point cloud stitched images, and heterogeneous hybrid stitched images simultaneously in the bitstream, and extend relevant standards. It has the following advantages: 1) For application scenarios composed of data in different formats, this method can provide real-time immersive video interaction services for data of different formats (such as 3D meshes, 3D point clouds, multi-view images, etc.), promoting the development of the VR / AR / MR industry; 2) Compared to encoding multi-view video images and point cloud format data separately and then calling their respective decoders to independently decode multiple signals, the number of decoders required is less, fully utilizing the decoder's pixel processing rate and reducing hardware requirements; 3) It retains the rendering advantages of data from different formats (point clouds, etc.), improving the quality of image synthesis; 4) It further improves the reconstruction quality and encoding performance of heterogeneous data.

[0687] This application also provides an encoding device. Figure 11 This is a schematic block diagram of an encoding device 110 provided in an embodiment of this application, which is applied to an encoder. Figure 11 As shown, the encoding device 110 includes:

[0688] The processing unit 1101 is configured to process visual media content in at least two expression formats to obtain at least two isomorphic blocks;

[0689] The splicing unit 1102 is configured to splice the at least two isomorphic blocks to obtain a spliced ​​image and spliced ​​image information. When the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks, and different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information.

[0690] The encoding unit 1103 is configured to encode the spliced ​​image and spliced ​​image information to obtain a bitstream.

[0691] In some embodiments, the splicing graph information includes a first syntax element, and the splicing graph is determined to be a heterogeneous hybrid splicing graph or a homogeneous splicing graph based on the first syntax element.

[0692] In some embodiments, the first syntax element includes a first sub-syntax element and a second sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes:

[0693] If the value of the first sub-syntax element is equal to the value of the second sub-syntax element, the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

[0694] In some embodiments, determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: if the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic; if the value is a second preset value, then the mosaic is determined to be a homogeneous mosaic.

[0695] In some embodiments, determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes:

[0696] If the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including isomorphic blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats;

[0697] If the value is a fourth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the first expression format;

[0698] If the value is the fifth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the second expression format.

[0699] In some embodiments, the first sub-syntax element is a syntax element of the mosaic sequence parameter set ASPS, and the second sub-syntax element is a syntax element of the mosaic frame parameter set AFPS.

[0700] In some embodiments, the method includes: parsing a first sub-syntax element in ASPS; and determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element.

[0701] The second sub-syntax element is parsed in AFPS; the value of the second sub-syntax element determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph.

[0702] In some embodiments, the first syntax element is not included in the splicing graph information, and the splicing graph is determined to be an isomorphic splicing graph.

[0703] In some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information further includes a second syntax element; the expression format of isomorphic blocks in the mosaic is determined based on the second syntax element.

[0704] In some embodiments, determining the expression format of the isomorphic block in the splicing diagram based on the second syntax element includes: when the value of the second syntax element of the i-th block is a sixth preset value, the expression format of the i-th block is determined to be a first expression format; when the value of the second syntax element of the i-th block is a seventh preset value, the expression format of the i-th block is determined to be a second expression format.

[0705] In some embodiments, the second syntax element includes: a third sub-syntax element and a fourth sub-syntax element;

[0706] The method includes: parsing a first sub-syntax element in ASPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element; parsing a second and third sub-syntax elements in AFPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the second sub-syntax element; and when the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph based on the value of the third sub-syntax element, determining the expression format of the homogeneous blocks in the spliced ​​graph based on the fourth sub-syntax element. Here, when the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph based on the value of the third sub-syntax element, the fourth syntax element of each homogeneous block is parsed, and the value of the fourth syntax element determines the expression format of each homogeneous block.

[0707] In some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information.

[0708] In some embodiments, the isomorphic block information includes syntax elements of ASPS and syntax elements of AFPS; different isomorphic block information corresponds to different syntax elements of ASPS and AFPS.

[0709] In some embodiments, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first isomorphic block information and the second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. The first isomorphic block information also includes extended syntax elements of the ASPS parameter set and the AFPS parameter set.

[0710] In some embodiments, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first isomorphic block information and the second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. The first isomorphic block information also includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content.

[0711] In some embodiments, the parameter set sub-bitstream of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content including at least one expression format in the bitstream is determined based on the third syntax element.

[0712] In some embodiments, determining the bitstream corresponding to visual media content in at least one expression format in the bitstream based on the third syntax element includes: the third syntax element being a first value, determining the bitstream corresponding to visual media content in both a first expression format and a second expression format; the third syntax element being a second value, determining the bitstream corresponding to visual media content in the first expression format; and the third syntax element being a third value, determining the bitstream corresponding to visual media content in the second expression format.

[0713] In some embodiments, the encoding unit 1103 is configured to encode the stitched image to obtain a video compressed sub-bitstream; encode the stitched image information to obtain a stitched image information sub-bitstream; and combine the video compressed sub-bitstream and the stitched image information sub-bitstream into the bitstream.

[0714] In some embodiments, the representation format is multi-view video, point cloud, or mesh.

[0715] In some embodiments, the heterogeneous hybrid mosaic includes at least one of the following: a single-attribute heterogeneous hybrid mosaic and a multi-attribute heterogeneous hybrid mosaic; the isomorphic mosaic includes at least one of the following: a single-attribute isomorphic mosaic and a multi-attribute isomorphic mosaic.

[0716] This application also provides a decoding device. Figure 12 This is a schematic block diagram of a decoding apparatus 120 provided in an embodiment of this application, which is applied to a decoder. Figure 12 As shown, the decoding device 120 includes:

[0717] Decoding unit 1201 is configured to decode the bitstream to obtain the spliced ​​image and spliced ​​image information;

[0718] The splitting unit 1202 is configured such that when the spliced ​​image is a heterogeneous hybrid spliced ​​image, it obtains at least two kinds of isomorphic blocks and isomorphic block information based on the spliced ​​image and the spliced ​​image information; wherein, different kinds of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information;

[0719] The splitting unit 1202 is configured such that when the stitched image is a homogeneous stitched image, it obtains a homogeneous block and homogeneous block information based on the stitched image and the stitched image information.

[0720] The processing unit 1203 is configured to obtain visual media content in at least two expression formats based on the isomorphic blocks and the isomorphic block information.

[0721] In some embodiments, the splicing graph information includes a first syntax element, and the splicing graph is determined to be a heterogeneous hybrid splicing graph or a homogeneous splicing graph based on the first syntax element.

[0722] In some embodiments, the first syntax element includes a first sub-syntax element and a second sub-syntax element; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: if the value of the first sub-syntax element and the value of the second sub-syntax element are equal, determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

[0723] In some embodiments, determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: if the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic; if the value is a second preset value, then the mosaic is determined to be a homogeneous mosaic.

[0724] In some embodiments, determining whether the stitched image is a heterogeneous hybrid stitched image or a homogeneous stitched image based on the value includes: if the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including homogeneous blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; if the value is a fourth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the first expression format; if the value is a fifth preset value, then the stitched image is determined to be a homogeneous stitched image including homogeneous blocks of the second expression format.

[0725] In some embodiments, the first sub-syntax element is a syntax element of the mosaic sequence parameter set ASPS, and the second sub-syntax element is a syntax element of the mosaic frame parameter set AFPS.

[0726] In some embodiments, the first syntax element is not included in the splicing graph information, and the splicing graph is determined to be an isomorphic splicing graph.

[0727] In some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information further includes a second syntax element; the expression format of isomorphic blocks in the mosaic is determined based on the second syntax element.

[0728] In some embodiments, determining the expression format of the isomorphic block in the splicing diagram based on the second syntax element includes: when the value of the second syntax element of the i-th block is a sixth preset value, the expression format of the i-th block is determined to be a first expression format; when the value of the second syntax element of the i-th block is a seventh preset value, the expression format of the i-th block is determined to be a second expression format.

[0729] In some embodiments, the second syntax element includes: a third sub-syntax element and a fourth sub-syntax element;

[0730] The method includes: parsing a first sub-syntax element in ASPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the first sub-syntax element; parsing a second and a third sub-syntax element in AFPS; determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value of the second sub-syntax element; and when the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph based on the value of the third sub-syntax element, determining the expression format of the homogeneous blocks in the spliced ​​graph based on the fourth sub-syntax element.

[0731] In some embodiments, when the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information.

[0732] In some embodiments, when the mosaic is a heterogeneous hybrid mosaic, obtaining at least two isomorphic blocks and isomorphic block information based on the mosaic and the mosaic information includes: when the mosaic is a heterogeneous hybrid mosaic, splitting the mosaic to obtain at least two isomorphic blocks; and obtaining isomorphic block information corresponding to the at least two isomorphic blocks from the mosaic information according to the expression format of the at least two isomorphic blocks.

[0733] In some embodiments, the isomorphic block information includes syntax elements of ASPS and syntax elements of AFPS; different isomorphic block information corresponds to different syntax elements of ASPS and AFPS.

[0734] In some embodiments, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first and second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. In some embodiments, the first isomorphic block information further includes extended syntax elements of the ASPS parameter set and the AFPS parameter set. In some embodiments, the first isomorphic block information further includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content.

[0735] In some embodiments, when the isomorphic block is a multi-view video block, it corresponds to first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to second isomorphic block information. The first isomorphic block information and the second isomorphic block information include shared syntax elements of the ASPS parameter set and the AFPS parameter set. The first isomorphic block information also includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content.

[0736] In some embodiments, the parameter set sub-bitstream of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content including at least one expression format in the bitstream is determined based on the third syntax element.

[0737] In some embodiments, determining the bitstream corresponding to visual media content in at least one expression format in the bitstream based on the third syntax element includes: the third syntax element being a first value, determining the bitstream corresponding to visual media content in both a first expression format and a second expression format; the third syntax element being a second value, determining the bitstream corresponding to visual media content in the first expression format; and the third syntax element being a third value, determining the bitstream corresponding to visual media content in the second expression format.

[0738] In some embodiments, decoding the bitstream to obtain a spliced ​​image and spliced ​​image information includes: determining, based on the second syntax element, the bitstream corresponding to visual media content in the bitstream that includes at least two expression formats, and decoding the bitstream to obtain a heterogeneous hybrid spliced ​​image and spliced ​​image information.

[0739] In some embodiments, the decoding unit 1201 is configured to decode the video compression sub-stream to obtain the stitched image; and decode the stitched image information sub-stream to obtain the stitched image information.

[0740] In some embodiments, the representation format is multi-view video, point cloud, or mesh.

[0741] In some embodiments, the heterogeneous hybrid mosaic includes at least one of the following: a single-attribute heterogeneous hybrid mosaic and a multi-attribute heterogeneous hybrid mosaic; the isomorphic mosaic includes at least one of the following: a single-attribute isomorphic mosaic and a multi-attribute isomorphic mosaic.

[0742] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, further details will not be provided here.

[0743] The apparatus and system of the embodiments of this application have been described above from the perspective of functional units in conjunction with the accompanying drawings. It should be understood that these functional units can be implemented in hardware, in software instructions, or in a combination of hardware and software units. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software units in the decoding processor. Optionally, the software unit can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0744] In practical applications, embodiments of this application also provide an encoder. Figure 13 A schematic block diagram of an encoder provided in one embodiment of this application, as shown below. Figure 13 As shown, encoder 1310 includes:

[0745] The second memory 1320 and the second processor 1330; the second memory 1320 stores a computer program that can run on the second processor 1330, and the encoding method on the encoder side when the second processor 1330 executes the program.

[0746] In practical applications, embodiments of this application also provide a decoder. Figure 14 A schematic block diagram of a decoder provided in an embodiment of this application, as shown below. Figure 14 As shown, the decoder 1410 includes:

[0747] A first memory 1420 and a first processor 1430; the first memory 1420 stores a computer program that can run on the first processor 1430, and the first processor 1430 executes the program to define a decoding method on the decoder side.

[0748] In some embodiments of this application, the processor may include, but is not limited to:

[0749] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0750] In some embodiments of this application, the memory includes, but is not limited to:

[0751] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0752] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0753] In another embodiment of this application, see [link to application]. Figure 15 This illustrates a schematic diagram of the composition structure of an encoding / decoding system provided in an embodiment of this application. Figure 15 As shown, the encoding / decoding system 150 may include an encoder 1501 and a decoder 1502. The encoder 1501 may be a device integrating the encoding apparatus described in the foregoing embodiments; the decoder 1502 may be a device integrating the decoding apparatus described in the foregoing embodiments.

[0754] In the embodiments of this application, in the encoding and decoding system 150, both the encoder 1501 and the decoder 1502 can use the color component information of adjacent reference pixels and the pixel to be predicted to calculate the weighting coefficients corresponding to the pixel to be predicted; moreover, different reference pixels can have different weighting coefficients. Applying these weighting coefficients to the chromaticity prediction of the pixel to be predicted in the current block can not only improve the accuracy of chromaticity prediction and save bit rate, but also improve encoding and decoding performance.

[0755] This application also provides a chip for implementing the above-described encoding / decoding method. Specifically, the chip includes a processor for retrieving and running a computer program from a memory, causing an electronic device equipped with the chip to perform the above-described encoding / decoding method.

[0756] This application also provides a computer storage medium storing a computer program that, when executed by a second processor, implements an encoding method of an encoder; or, when executed by a first processor, implements a decoding method of a decoder. Alternatively, this application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods described in the above-described method embodiments.

[0757] This application also provides a bitstream generated according to the above encoding method. Optionally, the bitstream includes the first syntax element, or includes the second syntax element and the third syntax element.

[0758] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0759] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0760] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0761] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0762] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, and second information may appear before, after, or simultaneously with first information.

[0763] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0764] Industrial applicability

[0765] This application provides an encoding / decoding method, apparatus, encoder, decoder, and storage medium. Targeting application scenarios involving visual media content with one or more expression formats, it stitches homogeneous blocks of different expression formats into a heterogeneous hybrid stitched image. Encoding and decoding of homogeneous blocks of different expression formats within this heterogeneous hybrid stitched image reduces the number of encoders and decoders required, lowers implementation costs, and improves usability. Furthermore, in the heterogeneous hybrid stitched image, certain high-level parameters of blocks from different expression formats can be unequal, allowing the heterogeneous data to provide more suitable high-level parameters, effectively improving encoding efficiency—that is, reducing bitrate or improving the quality of reconstructed multi-view video or point cloud video.

Claims

1. A decoding method, wherein, include: Decode the bitstream to obtain the stitched image and stitched image information; When the stitched image is a heterogeneous hybrid stitched image, at least two types of isomorphic blocks and isomorphic block information are obtained based on the stitched image and the stitched image information; wherein, different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information; When the mosaic is a homogeneous mosaic, a homogeneous block and homogeneous block information are obtained based on the mosaic and the mosaic information. Based on the isomorphic blocks and the isomorphic block information, visual media content in at least two expression formats is obtained.

2. The method according to claim 1, wherein, The splicing graph information includes a first syntax element, and the splicing graph is determined to be a heterogeneous hybrid splicing graph or a homogeneous splicing graph based on the first syntax element.

3. The method according to claim 2, wherein, The first syntax element includes a first sub-syntax element and a second sub-syntax element; The step of determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: If the value of the first sub-syntax element is equal to the value of the second sub-syntax element, the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

4. The method according to claim 3, wherein, The step of determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: If the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic. If the value is a second preset value, then the stitched image is determined to be an isomorphic stitched image.

5. The method according to claim 3, wherein, The step of determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: If the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including isomorphic blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; If the value is a fourth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the first expression format; If the value is the fifth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the second expression format.

6. The method according to claim 3, wherein, The first sub-syntax element is a syntax element of the mosaic sequence parameter set ASPS, and the second sub-syntax element is a syntax element of the mosaic frame parameter set AFPS.

7. The method according to claim 6, wherein, The method includes: Parse the first subsyntax element in ASPS; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the first sub-grammar element. Parsing the second subsyntactic element in AFPS; The value of the second sub-grammar element determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph.

8. The method according to claim 7, wherein, The first sub-syntax element is a newly added syntax element in ASPS, or the first sub-syntax element is a syntax element obtained by logical operation of at least two syntax elements in ASPS; The second sub-syntax element is a newly added syntax element in AFPS, or the second sub-syntax element is a syntax element obtained by logical operation of at least two syntax elements in ASPS.

9. The method according to any one of claims 1-8, wherein, When the spliced ​​graph is a heterogeneous hybrid spliced ​​graph, the spliced ​​graph information also includes a second syntax element; the expression format of isomorphic blocks in the spliced ​​graph is determined according to the second syntax element.

10. The method according to claim 9, wherein, Determining the expression format of isomorphic blocks in the splicing graph based on the second syntax element includes: When the value of the second syntax element of the i-th block is the sixth preset value, the expression format of the i-th block is determined to be the first expression format. When the value of the second syntax element of the i-th block is the seventh preset value, the expression format of the i-th block is determined to be the second expression format.

11. The method according to claim 9, wherein, The second grammatical element includes: a third sub-grammatical element and a fourth sub-grammatical element; the method includes: Parse the first subsyntax element in ASPS; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the first sub-grammar element. AFPS parses the second and third sub-syntax elements; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the second sub-grammar element. When the concatenation graph is determined to be a heterogeneous hybrid concatenation graph based on the value of the third sub-syntax element, the expression format of the isomorphic blocks in the concatenation graph is determined based on the fourth sub-syntax element.

12. The method according to claim 1, wherein, When the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information.

13. The method according to claim 12, wherein, When the mosaic is a heterogeneous hybrid mosaic, at least two types of isomorphic blocks and isomorphic block information are obtained based on the mosaic and the mosaic information, including: When the mosaic is a heterogeneous hybrid mosaic, the mosaic is split to obtain at least two isomorphic blocks; Based on the expression formats of the at least two isomorphic blocks, obtain the isomorphic block information corresponding to the at least two isomorphic blocks from the splicing graph information.

14. The method according to claim 12 or 13, wherein, The isomorphic block information includes the syntax elements of ASPS and the syntax elements of AFPS; Different isomorphic block information corresponds to different syntax elements of the ASPS and the AFPS.

15. The method according to any one of claims 12-13, wherein, When the isomorphic block is a multi-view video block, it corresponds to the first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to the second isomorphic block information. The first isomorphic block information and the second isomorphic block information include the shared syntax elements of the ASPS parameter set and the AFPS parameter set; The first isomorphic block information also includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content.

16. The method according to any one of claims 1-8, wherein, The parameter set of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content with at least one expression format is determined based on the third syntax element.

17. The method according to claim 16, wherein, The step of determining the bitstream corresponding to visual media content in the bitstream that includes at least one expression format based on the third syntax element includes: The third syntax element is the first value, which determines that the bitstream simultaneously includes the bitstream corresponding to the visual media content in the first expression format and the bitstream corresponding to the visual media content in the second expression format. The third syntax element is a second numerical value, which determines the bitstream that includes visual media content corresponding to the first expression format. The third syntax element is a third numerical value, which determines the bitstream corresponding to the visual media content in the second expression format.

18. The method according to any one of claims 1-8, wherein, The bitstream includes a video compression sub-bitstream and a stitched image information sub-bitstream. The decoded bitstream yields a stitched image and stitched image information, including: Decode the compressed video sub-stream to obtain the stitched image; Decode the stitched image information sub-stream to obtain the stitched image information.

19. The method according to any one of claims 1-8, wherein, The expression format is multi-view video, point cloud, or grid.

20. An encoding method, wherein, include: Process visual media content in at least two expression formats to obtain at least two isomorphic blocks; The at least two isomorphic blocks are spliced ​​together to obtain a spliced ​​image and spliced ​​image information. When the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks, and different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information. The stitched image and its information are encoded to obtain a bitstream.

21. The method according to claim 20, wherein, The splicing graph information includes a first syntax element, and the splicing graph is determined to be a heterogeneous hybrid splicing graph or a homogeneous splicing graph based on the first syntax element.

22. The method according to claim 21, wherein, The first syntax element includes a first sub-syntax element and a second sub-syntax element; The step of determining whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the first syntax element includes: If the value of the first sub-syntax element is equal to the value of the second sub-syntax element, the spliced ​​graph is determined to be a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph based on the value.

23. The method according to claim 22, wherein, The step of determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: If the value is a first preset value, then the mosaic is determined to be a heterogeneous hybrid mosaic. If the value is a second preset value, then the stitched image is determined to be an isomorphic stitched image.

24. The method according to claim 22, wherein, The step of determining whether the mosaic is a heterogeneous hybrid mosaic or a homogeneous mosaic based on the value includes: If the value is a third preset value, then the stitched image is determined to be a heterogeneous hybrid stitched image including isomorphic blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; If the value is a fourth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the first expression format; If the value is the fifth preset value, then the stitched image is determined to be an isomorphic stitched image that includes isomorphic blocks of the second expression format.

25. The method according to claim 22, wherein, The first sub-syntax element is a syntax element of the mosaic sequence parameter set ASPS, and the second sub-syntax element is a syntax element of the mosaic frame parameter set AFPS.

26. The method according to claim 25, wherein, The method includes: Parse the first subsyntax element in ASPS; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the first sub-grammar element. Parsing the second subsyntactic element in AFPS; The value of the second sub-grammar element determines whether the spliced ​​graph is a heterogeneous hybrid spliced ​​graph or a homogeneous spliced ​​graph.

27. The method according to claim 26, wherein, The first sub-syntax element is a newly added syntax element in ASPS, or the first sub-syntax element is a syntax element obtained by logical operation of at least two syntax elements in ASPS; The second sub-syntax element is a newly added syntax element in AFPS, or the second sub-syntax element is a syntax element obtained by logical operation of at least two syntax elements in ASPS.

28. The method according to any one of claims 20-27, wherein, When the spliced ​​graph is a heterogeneous hybrid spliced ​​graph, the spliced ​​graph information also includes a second syntax element; the expression format of isomorphic blocks in the spliced ​​graph is determined according to the second syntax element.

29. The method according to claim 28, wherein, Determining the expression format of isomorphic blocks in the splicing graph based on the second syntax element includes: When the value of the second syntax element of the i-th block is the sixth preset value, the expression format of the i-th block is determined to be the first expression format. When the value of the second syntax element of the i-th block is the seventh preset value, the expression format of the i-th block is determined to be the second expression format.

30. The method according to claim 29, wherein, The second grammatical element includes: the third sub-grammatical element and the fourth sub-grammatical element; Parse the first subsyntax element in ASPS; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the first sub-grammar element. AFPS parses the second and third sub-syntax elements; The concatenation graph is determined to be either a heterogeneous hybrid concatenation graph or a homogeneous concatenation graph based on the value of the second sub-grammar element. When the concatenation graph is determined to be a heterogeneous hybrid concatenation graph based on the value of the third sub-syntax element, the expression format of the isomorphic blocks in the concatenation graph is determined based on the fourth sub-syntax element.

31. The method according to claim 20, wherein, When the mosaic is a heterogeneous hybrid mosaic, the mosaic information includes at least two types of isomorphic block information, wherein isomorphic blocks with different expression formats correspond to different isomorphic block information.

32. The method according to claim 31, wherein, The isomorphic block information includes the syntax elements of ASPS and the syntax elements of AFPS; Different isomorphic block information corresponds to different syntax elements of the ASPS and the AFPS.

33. The method according to claim 31 or 32, wherein, When the isomorphic block is a multi-view video block, it corresponds to the first isomorphic block information; when the isomorphic block is a point cloud block, it corresponds to the second isomorphic block information. The first isomorphic block information and the second isomorphic block information include the shared syntax elements of the ASPS parameter set and the AFPS parameter set; The first isomorphic block information also includes a first extended syntax element of the ASPS parameter set, used to represent the bit depth of the geometric coordinates of the reconstructed geometric content.

34. The method according to any one of claims 20-27, wherein, The parameter set of the bitstream includes a third syntax element, and the bitstream corresponding to visual media content with at least one expression format is determined based on the third syntax element.

35. The method according to claim 34, wherein, The step of determining the bitstream corresponding to visual media content in the bitstream that includes at least one expression format based on the third syntax element includes: The third syntax element is the first value, which determines that the bitstream simultaneously includes the bitstream corresponding to the visual media content in the first expression format and the bitstream corresponding to the visual media content in the second expression format. The third syntax element is a second numerical value, which determines the bitstream that includes visual media content corresponding to the first expression format. The third syntax element is a third numerical value, which determines the bitstream corresponding to the visual media content in the second expression format.

36. The method according to any one of claims 20-27, wherein, The process of encoding the stitched image and its information to obtain a bitstream includes: The stitched image is encoded to obtain a compressed video sub-stream; The stitched image information is encoded to obtain a stitched image information sub-stream; The video compression sub-stream and the stitching image information sub-stream are combined to form the bitstream.

37. The method according to any one of claims 20-27, wherein, The expression format is multi-view video, point cloud, or grid.

38. A decoding device, wherein, include: The decoding unit is configured to decode the bitstream to obtain the spliced ​​image and spliced ​​image information; The splitting unit, when the stitched image is a heterogeneous hybrid stitched image, obtains at least two types of isomorphic blocks and isomorphic block information based on the stitched image and the stitched image information; wherein, different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information; The splitting unit, when the stitched image is a homogeneous stitched image, obtains a homogeneous block and homogeneous block information based on the stitched image and the stitched image information; The processing unit is configured to obtain visual media content in at least two expression formats based on the isomorphic blocks and the isomorphic block information.

39. An encoding device, wherein, include: The processing unit is configured to process visual media content in at least two expression formats to obtain at least two isomorphic blocks; The splicing unit is configured to splice the at least two isomorphic blocks to obtain a spliced ​​image and spliced ​​image information. When the spliced ​​image is a heterogeneous hybrid spliced ​​image, the spliced ​​image includes at least two isomorphic blocks, and different types of isomorphic blocks correspond to different visual media content expression formats and different isomorphic block information. The encoding unit is configured to encode the spliced ​​image and spliced ​​image information to obtain a bitstream.

40. A decoder, wherein, The decoder includes: First memory and first processor; The first memory stores a computer program that can run on a first processor, which, when executing the program, implements the decoding method according to any one of claims 1 to 19.

41. An encoder, wherein, The encoder includes: Second memory and second processor; The second memory stores a computer program that can run on a second processor, which, when executing the program, implements the encoding method according to any one of claims 20 to 37.

42. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program, which, when executed by a first processor, implements the decoding method according to any one of claims 1 to 19; or, when executed by a second processor, implements the encoding method according to any one of claims 20 to 37.

43. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program and a bitstream, the computer program being executed by a processor to implement the encoding method as described in any one of claims 20 to 37 to generate the bitstream.

Images