A coding method, device, encoder, decoder and storage medium

Abstract

The application provides a coding method and device, an encoder, a decoder and a storage medium. For the application scenario of visual media content including one or more expression formats, isomorphic blocks of different expression formats are spliced into a heterogeneous mixed splicing graph, isomorphic blocks of the same expression format are spliced into an isomorphic splicing graph, and the obtained splicing graph and splicing graph information are written into a code stream. The isomorphic splicing graph (for example, at least one of a multi-view splicing graph, a point cloud splicing graph and a mesh splicing graph) and the heterogeneous mixed splicing graph exist in the code stream at the same time, so that the coding method is applicable to the application scenario of visual media content of multiple expression formats, and the application range is expanded. Moreover, the code stream contains a first syntax element, which can improve the decoding efficiency of the decoding end for the splicing graph. Since the isomorphic blocks of different expression formats are spliced into a heterogeneous mixed splicing graph for coding, the number of called decoders can be reduced, the implementation cost can be reduced, and the ease of use can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a coding and decoding method, a coding and decoding device, an encoder, a decoder and a storage medium. BACKGROUND

[0002] In a three-dimensional application scenario, such as a virtual reality (VR), an augmented reality (AR), a mix reality (MR) or the like, different visual media objects with different expression formats can appear in the same scene. For example, in the same three-dimensional scene, a scene background and part of characters and objects are expressed in a video, and another part of characters is expressed in a three-dimensional point cloud or a three-dimensional mesh.

[0003] When compression encoding is performed, multi-view video encoding, point cloud encoding and mesh encoding are respectively used, which can better maintain effective information of the original expression format than projecting all into multi-view video encoding, improve the quality of a viewing window rendered when viewing, and improve the comprehensive efficiency of code rate-quality.

[0004] However, the current coding and decoding technology respectively codes and decodes multi-view video, point cloud and mesh, and a large number of codecs need to be called in the coding and decoding process, which results in a large coding and decoding cost. SUMMARY

[0005] Embodiments of the present application provide a coding and decoding method, a coding and decoding device, an encoder, a decoder and a storage medium.

[0006] In a first aspect, the present application provides a decoding method applied to a decoder, comprising:

[0007] decoding a code stream to obtain a spliced picture and spliced picture information, wherein the spliced picture information comprises a first syntax element, and the first syntax element is used to determine whether the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture;

[0008] when it is determined according to the first syntax element that the spliced picture is a heterogeneous mixed spliced picture, the spliced picture is split according to the spliced picture information of the spliced picture to obtain at least two homogeneous blocks, wherein the at least two homogeneous blocks correspond to different visual media content expression formats;

[0009] when it is determined according to the first syntax element that the spliced picture is a homogeneous spliced picture, the spliced picture is split according to the spliced picture information of the spliced picture to obtain one homogeneous block, wherein the one homogeneous block corresponds to the same visual media content expression format;

[0010] the homogeneous block is decoded and reconstructed to obtain visual media content in at least one expression format.

[0011] In a second aspect, the present application provides an encoding method applied to an encoder, comprising:

[0012] processing at least one visual media content in at least one expression format to obtain at least one isomorphic block, wherein different isomorphic blocks correspond to different visual media content expression formats;

[0013] stitching the at least one isomorphic block to obtain at least one stitched picture and stitched picture information, wherein the stitched picture information comprises a first syntax element, and according to the first syntax element, it is determined that the stitched picture is a heterogeneous mixed stitched picture or an isomorphic stitched picture, the heterogeneous mixed stitched picture comprises at least two isomorphic blocks, and the isomorphic stitched picture comprises one isomorphic block;

[0014] encoding the at least one stitched picture and the stitched picture information to obtain a bitstream.

[0015] In a third aspect, the present application provides a decoding device applied to a decoder, comprising:

[0016] a decoding unit configured to decode a bitstream to obtain a stitched picture and stitched picture information, wherein the stitched picture information comprises a first syntax element, and according to the first syntax element, it is determined that the stitched picture is a heterogeneous mixed stitched picture or an isomorphic stitched picture;

[0017] a first splitting unit configured to, when it is determined according to the first syntax element that the stitched picture is a heterogeneous mixed stitched picture, split the stitched picture according to the stitched picture information of the stitched picture to obtain at least two isomorphic blocks, wherein the at least two isomorphic blocks correspond to different visual media content expression formats;

[0018] a second splitting unit configured to, when it is determined according to the first syntax element that the stitched picture is an isomorphic stitched picture, split the stitched picture according to the stitched picture information of the stitched picture to obtain one isomorphic block, wherein the one isomorphic block corresponds to the same visual media content expression format;

[0019] a processing unit configured to decode and reconstruct the isomorphic block to obtain at least one visual media content in at least one expression format.

[0020] In a fourth aspect, the present application provides an encoding device applied to an encoder, comprising:

[0021] a processing unit configured to process at least one visual media content in at least one expression format to obtain at least one isomorphic block, wherein different isomorphic blocks correspond to different visual media content expression formats;

[0022] The splicing unit is configured to splice the at least one homogeneous block to obtain at least one spliced picture and spliced picture information. The spliced picture information includes a first syntax element. According to the first syntax element, it is determined that the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture. The heterogeneous mixed spliced picture includes at least two homogeneous blocks. The homogeneous spliced picture includes one homogeneous block.

[0023] The encoding unit is configured to encode the at least one spliced picture and the spliced picture information to obtain a bitstream.

[0024] In a fifth aspect, a decoder is provided, including a first memory and a first processor. The first memory stores a computer program capable of running on the first processor to execute the method in the first aspect or any of the implementation manners thereof.

[0025] In a sixth aspect, an encoder is provided, including a second memory and a second processor. The second memory stores a computer program capable of running on the second processor to execute the method in the second aspect or any of the implementation manners thereof.

[0026] In a seventh aspect, a codec system is provided, including an encoder and a decoder. The encoder is configured to execute the method in the second aspect or any of the implementation manners thereof, and the decoder is configured to execute the method in the first aspect or any of the implementation manners thereof.

[0027] In an eighth aspect, a chip is provided, configured to implement the method in any of the first aspect to the second aspect or any of the implementation manners thereof. Specifically, the chip includes a processor configured to invoke and run a computer program from a memory, so that a device installed with the chip executes the method in any of the first aspect to the second aspect or any of the implementation manners thereof.

[0028] In a ninth aspect, a computer readable storage medium is provided, configured to store a computer program. The computer program causes a computer to execute the method in any of the first aspect to the second aspect or any of the implementation manners thereof.

[0029] In a tenth aspect, a computer program product is provided, including computer program instructions. The computer program instructions cause a computer to execute the method in any of the first aspect to the second aspect or any of the implementation manners thereof.

[0030] In an eleventh aspect, a computer program is provided. When the computer program runs on a computer, the computer program causes the computer to execute the method in any of the first aspect to the second aspect or any of the implementation manners thereof.

[0031] In a twelfth aspect, a bitstream is provided. The bitstream is generated based on the encoding method in the second aspect.

[0032] Based on the above technical solution, for the application scenario of visual media content including one or more expression formats, the isomorphic blocks of different expression formats are spliced into a heterogeneous mixed splicing image, the isomorphic blocks of the same expression format are spliced into an isomorphic splicing image, and the obtained splicing image and splicing image information are written into a code stream. The isomorphic splicing image (for example, at least one of the multi-view splicing image, the point cloud splicing image, and the mesh splicing image) and the heterogeneous mixed splicing image exist in the code stream at the same time, so that the coding and decoding method is applicable to the application scenario of visual media content of multiple expression formats, and the application range of the coding and decoding method is expanded. Moreover, the first syntax element for indicating the splicing image type is included in the splicing image information, which improves the decoding efficiency of the splicing image at the decoding end. Further, since the isomorphic blocks of different expression formats are spliced into a heterogeneous mixed splicing image for coding and decoding, the number of two-dimensional video codecs such as HEVC, VVC, AVC, and AVS that need to be called can be reduced, the implementation cost is reduced, and the ease of use is improved. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a schematic block diagram of a video coding system related to an embodiment of the present application;

[0034] Figure 2A is a schematic block diagram of a video encoder related to an embodiment of the present application;

[0035] Figure 2B is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0036] Figure 3A is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0037] Figure 3B is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0038] Figure 3C is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0039] Figures 3D to 3F is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0040] Figure 4 is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0041] Figure 5 is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0042] Figure 6 is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0043] Figure 7 is a schematic block diagram of a video decoder related to an embodiment of the present application;

[0044] Figure 8 A homomorphism splicing diagram provided for an embodiment of the present application;

[0045] Figure 9 An illustrative diagram of a V3C bitstream structure provided for an embodiment of the present application;

[0046] Figure 10 An illustrative flow chart of a decoding method provided for an embodiment of the present application;

[0047] Figure 11 An illustrative block diagram of an encoding device provided for an embodiment of the present application;

[0048] Figure 12 An illustrative block diagram of a decoding device provided for an embodiment of the present application;

[0049] Figure 13 An illustrative block diagram of an encoder provided for an embodiment of the present application;

[0050] Figure 14 An illustrative block diagram of a decoder provided for an embodiment of the present application;

[0051] Figure 15 An illustrative diagram of a composition structure of a codec system provided for an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application can be applied to the field of image coding, the field of video coding, the field of hardware video coding, the field of special circuit video coding, the field of real-time video coding, etc. For example, the scheme of the present application can be combined with an audio video coding standard (AVS for short), such as the H.264 / audio video coding (AVC for short) standard, the H.265 / high efficiency video coding (HEVC for short) standard, and the H.266 / versatile video coding (VVC for short) standard. Alternatively, the scheme of the present application can be combined with other proprietary or industry standards, including ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific coding standard or technology.

[0053] The high-freedom immersive coding system can be roughly divided into the following links according to the task line: data acquisition, organization and expression of data, data coding compression, data decoding reconstruction, data synthesis rendering, and finally presenting the target data to the user.

[0054] The encoding related to the embodiments of the present application is mainly video coding. In order to facilitate understanding, first, the H.266 / versatile video coding (VVC for short) standard is introduced. Figure 1 The video coding system related to the embodiments of the present application is introduced.

[0055] Figure 1 A schematic block diagram of a video coding system related to the embodiments of the present application is shown. It should be noted that, Figure 1 This is only an example, and the video coding system of the embodiments of the present application includes but is not limited to Figure 1 As shown in the figure, Figure 1 As shown in the figure, the video coding system includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compress) video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0056] The encoding device 110 of the embodiments of the present application can be understood as a device with video encoding function, and the decoding device 120 can be understood as a device with video decoding function, i.e., the embodiments of the present application include more extensive apparatuses, such as smart phones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video gaming consoles, vehicle-mounted computers, etc.

[0057] In some embodiments, the encoding device 110 can transmit the encoded video data (e.g., a bitstream) to the decoding device 120 via a channel 130. The channel 130 can comprise one or more media and / or devices capable of communicating encoded video data from the encoding device 110 to the decoding device 120.

[0058] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit encoded video data directly to the decoding device 120 in realtime. In this example, the encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to the decoding device 120. The communication medium can include wireless communication media, such as radio frequency (RF) spectrum and / or optical, e.g., infrared (IR) spectrum, where applicable. Additionally, or alternatively, the communication medium can include wired communication media such as one or more physical transmission lines.

[0059] In another example, the channel 130 includes a storage medium that can store the encoded video data generated by the encoding device 110. The storage medium can include various local storage media, such as a Blu-ray disc, DVD, flash memory, and / or the like. In this example, the decoding device 120 can access the storage medium and retrieve the encoded video data stored therein.

[0060] In another example, the channel 130 can include a storage server that can store the encoded video data generated by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, and / or the like.

[0061] In some embodiments, the encoding device 110 includes the video encoder 112 and an output interface 113. The output interface 113 can include a modulator / demodulator (modem) and / or a transmitter.

[0062] In some embodiments, the encoding device 110 can include the video source 111 in addition to the video encoder 112 and the output interface 113.

[0063] The video source 111 can include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, a computer graphics system for generating video data, or a video content provider for receiving video data.

[0064] The video encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data can include one or more pictures or sequences of pictures. The bitstream includes encoded information of the pictures or the sequences of pictures in the form of a bitstream. The encoded information can include encoded picture data and associated data. The associated data can include sequence parameter sets (SPSs), picture parameter sets (PPSs), and other syntax structures. An SPS can contain parameters that apply to one or more sequences. A PPS can contain parameters that apply to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0065] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data can also be stored onto a storage medium or a storage server for later reading by the decoding device 120.

[0066] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122. In some embodiments, the decoding device 120 can include a display device 123 in addition to the input interface 121 and the video decoder 122.

[0067] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive the encoded video data through the channel 130.

[0068] The video decoder 122 decodes the encoded video data to obtain decoded video data and transmits the decoded video data to the display device 123.

[0069] The display device 123 displays the decoded video data. The display device 123 can be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 can include various types of display devices such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0070] Further, Figure 1 For example only, the technical solutions of the embodiments of the present disclosure are not limited to Figure 1For example, the techniques of this disclosure can also be applied to single-sided video encoding or single-sided video decoding.

[0071] A video encoding framework to which embodiments of the present disclosure relate is described below.

[0072] Figure 2A is a schematic block diagram of a video encoder to which embodiments of the present disclosure relate. It should be understood that the video encoder 200 can be used for lossy compression of images or lossless compression of images. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0073] The video encoder 200 can be applied to image data in luma chroma (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2 or 4:4:4, Y represents luma, Cb (U) represents blue chroma, Cr (V) represents red chroma, and U and V represent chroma for describing color and saturation. For example, in the color format, 4:2:0 means that there are 4 luma components and 2 chroma components (YYYYCbCr) for every 4 pixels, 4:2:2 means that there are 4 luma components and 4 chroma components (YYYYCbCrCbCr) for every 4 pixels, and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0074] For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTB may be referred to as a "tree block", "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or lumina) sample and two chrominance (chroma or chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chroma sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into a number of coding units (CUs) for encoding. The CU may be a rectangular block or a square block. A CU can be further divided into prediction units (PUs) and transform units (TUs), which allows for separation of coding, prediction, and transform, making processing more flexible. In one example, a CTU is divided into CUs using a quadtree, and a CU is divided into TUs and PUs using a quadtree.

[0075] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0076] In some embodiments, as Figure 2A As shown, the video encoder 200 may include a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components.

[0077] Optionally, in this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0078] In some embodiments, the prediction unit 210 includes an inter prediction unit 211 and an intra estimation unit 212. Due to the strong correlation between adjacent pixels in a frame of a video, the method of using intra prediction in video coding technology eliminates the spatial redundancy between adjacent pixels. Due to the strong similarity between adjacent frames in a video, the method of using inter prediction in video coding technology eliminates the temporal redundancy between adjacent frames, thereby improving the coding efficiency.

[0079] The inter prediction unit 211 can be used for inter prediction, which can include motion estimation and motion compensation, can refer to image information of different frames, and uses motion information to find a reference block in a reference frame, generates a prediction block according to the reference block, and is used to eliminate temporal redundancy; the frames used by inter prediction can be P frames and / or B frames, P frames refer to forward prediction frames, and B frames refer to bidirectional prediction frames. Inter prediction uses motion information to find a reference block in a reference frame, and generates a prediction block according to the reference block. The motion information includes a reference frame list in which the reference frame is located, a reference frame index, and a motion vector. The motion vector can be an integer pixel or a fractional pixel. If the motion vector is a fractional pixel, an interpolation filter needs to be used in the reference frame to obtain the required fractional pixel block. Here, the integer pixel or fractional pixel block found in the reference frame according to the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, and some technologies generate the prediction block by processing the reference block. Generating the prediction block by processing the reference block on the basis of the reference block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block on the basis of the prediction block.

[0080] The intra estimation unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block, which is used to eliminate spatial redundancy. The frame used by intra prediction can be an I frame.

[0081] There are multiple prediction modes for intra prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has 8 angle prediction modes and 1 non-angle prediction mode, and the H.265 / HEVC is extended to 33 angle prediction modes and 2 non-angle prediction modes. The intra prediction modes used by HEVC include Planar, DC and 33 angle modes, a total of 35 prediction modes. The intra prediction modes used by VVC include Planar, DC and 65 angle modes, a total of 67 prediction modes.

[0082] It should be noted that with the increase of angle modes, intra prediction will be more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital video.

[0083] The residual unit 220 may generate a residual block for the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate the residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.

[0084] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0085] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0086] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.

[0087] The loop filter unit 260 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0088] In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit and a sample adaptive offset / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove blocking effects, and the SAO / ALF unit is used to remove ringing effects.

[0089] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use a reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra estimation unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

[0090] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data.

[0091] Figure 2B It is a schematic block diagram of a video decoder involved in an embodiment of the present application.

[0092] likeFigure 2B As shown, video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / unit conversion unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded picture buffer 360. It is noted that video decoder 300 can include more, less, or different functional components.

[0093] Video decoder 300 can receive a bitstream. Entropy decoding unit 310 can parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, entropy decoding unit 310 can entropy-decode syntax elements in the bitstream. Prediction unit 320, inverse quantization / unit conversion unit 330, reconstruction unit 340, and loop filtering unit 350 can decode video data according to the syntax elements extracted from the bitstream, i.e., produce decoded video data.

[0094] In some embodiments, prediction unit 320 includes an inter-prediction unit 321 and an intra-estimation unit 322.

[0095] Intra-estimation unit 322 can perform intra-prediction to produce a prediction block for a PU. Intra-estimation unit 322 can use an intra-prediction mode to produce the prediction block for the PU based on blocks of pixels of spatially neighboring PUs. Intra-estimation unit 322 can also determine the intra-prediction mode for the PU according to one or more syntax elements parsed from the bitstream.

[0096] Inter-prediction unit 321 can construct a first reference picture list (List 0) and a second reference picture list (List 1) according to syntax elements parsed from the bitstream. In addition, if the PU is coded using inter-prediction, entropy decoding unit 310 can parse motion information for the PU. Inter-prediction unit 321 can determine one or more reference blocks for the PU according to the motion information for the PU. Inter-prediction unit 321 can generate the prediction block for the PU according to the one or more reference blocks for the PU.

[0097] Inverse quantization / unit conversion unit 330 can inverse quantize (i.e., de-quantize) transform coefficients associated with a TU. Inverse quantization / unit conversion unit 330 can determine a degree of quantization using a QP value associated with a CU of the TU.

[0098] After inverse quantizing the transform coefficients, inverse quantization / unit conversion unit 330 can apply one or more inverse transforms to the inverse quantized transform coefficients in order to produce a residual block associated with the TU.

[0099] Reconstruction unit 340 uses the residual block associated with a TU of a CU and the prediction block for the PU of the CU to reconstruct a pixel block for the CU. For example, reconstruction unit 340 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block for the CU, resulting in a reconstructed image block.

[0100] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0101] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0102] The basic process of video encoding and decoding is as follows: At the encoder end, a frame of image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block is also called residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 230, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 is called a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 is called a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0103] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction for the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference frame for inter-frame prediction for subsequent frames.

[0104] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0105] The above is the basic process of a video codec under a block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process can be optimized. The present application is applicable to the basic process of a video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0106] In some application scenarios, multiple heterogeneous contents appear simultaneously in the same three-dimensional scene, such as multi-view video and point cloud. For this case, the current coding mode at least includes the following two modes:

[0107] Mode one: for multi-view video, the MPEG Immersive Video (MIV) technology is used for coding, and for point cloud, the Video based Point Cloud Compression (VPCC) technology is used for coding.

[0108] The MIV technology and the VPCC technology are introduced below.

[0109] MIV technology: in order to reduce the transmission pixel rate while retaining as much scene information as possible, so as to ensure that there is enough information for rendering the target view, the scheme adopted by MPEG-I is as shown in Figure 3A , a limited number of viewpoints are selected as base viewpoints and the visible range of the scene is expressed as much as possible. The base viewpoints are transmitted as complete images, and the redundant pixels between the remaining non-base viewpoints and the base viewpoints are removed, i.e., only the effective information that is not redundantly expressed is retained. Then, the effective information is extracted as sub-block images and the base viewpoint images are reorganized to form a larger rectangular image, which is called a stitching image, Figure 3A , and Figure 3B The generation process of the stitching image is shown. The stitching image is sent to the codec for compression and reconstruction, and the auxiliary data related to the stitching information of the sub-block image is also sent to the encoder to form the code stream.

[0110] The encoding method of VPCC is to project the point cloud into a two-dimensional image or video, and to convert the three-dimensional information into two-dimensional information for coding. Figure 3C is the encoding block diagram of VPCC. The code stream is roughly divided into four parts. The geometry code stream is the code stream generated by coding the geometry depth map, which is used to represent the geometric information of the point cloud. The attribute code stream is the code stream generated by coding the texture map, which is used to represent the attribute information of the point cloud. The occupancy code stream is the code stream generated by coding the occupancy map, which is used to indicate the effective area in the depth map and the texture map. The three types of videos are coded by using a video encoder, as shown in Figures 3D to 3FThe auxiliary information code stream is a code stream generated by encoding the auxiliary information of the sub-block image, i.e., the part related to the patch data unit in the V3C standard, indicating the position and size of each sub-block image and the like.

[0111] In the second mode, both the multi-view video and the point cloud are coded using the frame packing technology in the Visual Volumetric Video-based Coding (V3C).

[0112] The frame packing technology will be introduced below.

[0113] For example, as shown in Figure 4 The encoding end includes the following steps:

[0114] Step 1. When encoding the obtained multi-view video, some pre-processing is performed to generate multi-view video sub-blocks, and then the multi-view video sub-blocks are organized to generate a multi-view video mosaic image.

[0115] For example, Figure 4 As shown in

[0116] The multi-view video mosaic image includes a multi-view video texture mosaic image and a multi-view video geometry mosaic image, i.e., only contains multi-view video sub-blocks.

[0117] Step 2. The multi-view video mosaic image is input into the frame packer to output a multi-view video hybrid mosaic image.

[0118] The multi-view video hybrid mosaic image includes a multi-view video texture hybrid mosaic image, a multi-view video geometry hybrid mosaic image, and a multi-view video texture and geometry hybrid mosaic image.

[0119] Specifically, as shown in Figure 4 The multi-view video mosaic image is frame packed to generate a multi-view video hybrid mosaic image, and each multi-view video mosaic image occupies a region of the multi-view video hybrid mosaic image. Accordingly, a flag pin_region_type_id_minus2 needs to be transmitted in the code stream for each region, which records information about whether the current region belongs to the multi-view video texture mosaic image or the multi-view video geometry mosaic image, which needs to be used at the decoding end.

[0120] Step 3, encode the multi-view video hybrid mosaic picture using a video encoder to obtain a code stream.

[0121] As shown in the figure, the decoding end includes the following steps: Figure 5

[0122] Step 1, in the multi-view video decoding, input the obtained code stream into a video decoder to obtain a reconstructed multi-view video hybrid mosaic picture.

[0123] Step 2, input the reconstructed multi-view video hybrid mosaic picture into a frame unpacker to output a reconstructed multi-view video mosaic picture.

[0124] Specifically, first, obtain the flag pin_region_type_id_minus2 from the code stream, and if it is determined that the pin_region_type_id_minus2 is V3C_AVD, it indicates that the current region is a multi-view video texture mosaic picture, then the current region is split and output as a reconstructed multi-view video texture mosaic picture.

[0125] If it is determined that the pin_region_type_id_minus2 is V3C_GVD, it indicates that the current region is a multi-view video geometry mosaic picture, and the current region is split and output as a reconstructed multi-view video geometry mosaic picture.

[0126] Step 3, decode the reconstructed multi-view video mosaic picture to obtain a reconstructed multi-view video.

[0127] Specifically, the multi-view video texture mosaic picture and the multi-view video geometry mosaic picture are decoded to obtain a reconstructed multi-view video.

[0128] The above is an analysis and introduction of the frame packing technology taking multi-view video as an example. The frame packing encoding and decoding mode for point cloud is basically the same as the above multi-view video, and can be referred to, for example, using TMC (a reference software of VPCC) to pack the point cloud to obtain a point cloud mosaic picture, inputting the point cloud mosaic picture into a frame packer to perform frame packing to obtain a point cloud hybrid mosaic picture, and performing splicing on the point cloud hybrid mosaic picture to obtain a point cloud code stream, which will not be repeated here.

[0129] The syntax related to frame packing in the standard is introduced below.

[0130] The V3C unit header syntax is shown in Table 1:

[0131] Table 1

[0132]

[0133] ​The V3C unit header semantics are shown in Table 2.

[0134] Table 2: V3C unit types

[0135]

[0136]

[0137] Currently, if multiple visual media contents in different expression formats appear in the same three-dimensional scene, the multiple visual media contents in different expression formats are respectively coded and decoded. For example, for the case that a point cloud and a multi-view video appear in the same three-dimensional scene, the current packing technology is to compress the point cloud to form a point cloud compressed code stream (i.e., a V3C code stream), compress the multi-view video information to obtain a multi-view video compressed code stream (i.e., another V3C code stream), and then the compressed code streams are multiplexed by a system layer to obtain a fused three-dimensional scene multiplexed code stream. When decoding, the point cloud compressed code stream and the multi-view video compressed code stream are respectively decoded. As can be seen, the existing technology uses multiple codecs when coding and decoding multiple visual media contents in different expression formats, and the coding and decoding cost is high.

[0138] To solve the above technical problems, the embodiments of the present application splice the isomorphic blocks in different expression formats in a heterogeneous mixed splicing graph, splice the isomorphic blocks in the same expression format in a homogeneous splicing graph, code and write the obtained heterogeneous mixed splicing graph and / or homogeneous splicing graph into a code stream, and the code stream can simultaneously contain a homogeneous splicing graph (for example, at least one of a multi-view splicing graph, a point cloud splicing graph, and a mesh splicing graph) and a heterogeneous mixed splicing graph, thereby expanding the application scenarios of the coding and decoding method. Moreover, the splicing graph information includes a first syntax element for indicating the type of the splicing graph, which can improve the decoding efficiency of the splicing graph at the decoding end.

[0139] The embodiments of the present application will be described below in combination with Figure 6 The video coding method provided by the embodiments of the present application will be described below with the coding end as an example.

[0140] Figure 6 The flowchart of the coding method provided by the embodiments of the present application is shown in Figure 6 The coding method includes the following steps.

[0141] Step 601: processing at least one visual media content in an expression format to obtain at least one isomorphic block, wherein different isomorphic blocks correspond to different visual media content expression formats.

[0142] In a three-dimensional application scenario, such as a Virtual Reality (VR), Augmented Reality (AR), Mix Reality (MR) or the like, different visual media objects with different expression formats can appear in the same scene. For example, in the same three-dimensional scene, a scene background and part of characters and objects are expressed in a video, and another part of characters are expressed in a three-dimensional point cloud or a three-dimensional mesh.

[0143] In some embodiments, the visual media content includes visual media content in at least one expression format, such as multi-view video, point cloud, mesh, or the like. A special case of the multi-view video is a single-view video, that is, the multi-view video can include multiple-view videos and / or a single-view video.

[0144] In some embodiments, each isomorphic block corresponds to an expression format. For example, the expression format corresponding to at least one isomorphic block includes at least one of the following: multi-view video, point cloud, mesh. At least two isomorphic blocks correspond to at least two different expression formats, such as at least two isomorphic blocks including at least two different expression formats, such as multi-view video, point cloud, mesh, or the like.

[0145] It should be noted that each isomorphic block in the embodiments of the present application can include at least one isomorphic block with the same expression format. For example, the isomorphic block in the point cloud format includes one or more point cloud blocks, the isomorphic block in the multi-view video format includes one or more multi-view video blocks, and the isomorphic block in the mesh format includes one or more mesh blocks.

[0146] In some embodiments, step 601 can be: processing visual media content in one expression format to obtain one isomorphic block. In some embodiments, step 601 can be: processing visual media content in at least two expression formats to obtain at least two isomorphic blocks, wherein the expression formats of different visual media content are different. Specifically, processing visual media content in a first expression format to obtain an isomorphic block in the first expression format; processing visual media content in a second expression format to obtain an isomorphic block in the second expression format. The first expression format is one of multi-view video, point cloud, and mesh, the second expression format is one of multi-view video, point cloud, and mesh, and the first expression format and the second expression format are different expression formats.

[0147] That is, the visual media content described above includes visual media content in at least one expression format, such as multi-view video, point cloud, mesh, etc. When the visual media content includes one expression format, the visual media content is processed to obtain isomorphic blocks in one expression format. When the visual media content includes multiple expression formats, the visual media content is processed to obtain isomorphic blocks in multiple expression formats.

[0148] It should be noted that the block can also be referred to as a tile, i.e., the point cloud block can also be referred to as a point cloud tile, the multi-view video block can also be referred to as a multi-view video tile, and the mesh block can also be referred to as a mesh tile. The block can be a mosaic with a specific shape, such as a mosaic of a rectangular region with a specific length and / or height. For example, the at least one sub-block can be sequentially mosaicked, such as mosaicked according to the area of the sub-block from large to small, or mosaicked according to the length and / or height of the sub-block from large to small, to obtain the block corresponding to the visual media content. Optionally, one block can be accurately mapped to one atlas block.

[0149] In some embodiments, each sub-block in the block can have a patch ID to distinguish different sub-blocks in the same block. For example, the same block can include sub-block 1 (patch 1), sub-block 2 (patch 2), and sub-block 3 (patch 3).

[0150] Further, the expression format corresponding to each sub-block in the isomorphic block is the same, such as each sub-block in the isomorphic block being a multi-view video sub-block or a point cloud sub-block, etc. The expression format corresponding to each sub-block in the isomorphic block is the expression format corresponding to the isomorphic block.

[0151] In some embodiments, the isomorphic block can have a tile ID to distinguish different blocks of the same expression format. For example, the point cloud block can include point cloud block 1 or point cloud block 2. For example, the multiple visual media contents include point cloud and multi-view video, the point cloud is processed to obtain a point cloud block, the point cloud block 1 includes point cloud sub-blocks 1 to 3, and the multi-view video is processed to obtain a multi-view video block, the multi-view video block includes multi-view video sub-blocks 1 to 4.

[0152] When a visual media content in an expression format needs to be processed, a homogenous block in the expression format is obtained. When at least two visual media contents need to be processed, at least two homogenous blocks in the expression format are obtained. In order to improve compression efficiency, the at least two visual media contents are processed, for example, packed (also referred to as spliced), to obtain a block corresponding to each of the at least two visual media contents. For example, sub-patches corresponding to the at least two visual media contents can be spliced to obtain a block. It should be noted that the manner in which the at least two visual media contents are processed to obtain the blocks is not limited.

[0153] In a possible implementation, the visual media content includes visual media content in two expression formats of multi-view video and point cloud, and the processing of the at least one visual media content in the expression format to obtain at least one homogenous block includes: after projection and de-redundancy processing of the obtained multi-view video, connecting non-repeated pixels into video sub-patches, and splicing the video sub-patches into multi-view video blocks; and after parallel projection of the obtained point cloud, connecting the points in the projection plane into point cloud sub-patches, and splicing the point cloud sub-patches into point cloud blocks.

[0154] Specifically, for multi-view video, taking MPEG-I as an example, a limited number of viewpoints are selected as base viewpoints and express the visible range of the scene as much as possible, the base viewpoints are transmitted as complete images, and the redundant pixels between the remaining non-base viewpoints and the base viewpoints are removed, that is, only the effective information that is not repeatedly expressed is retained, and then the effective information is extracted as sub-block images and the base viewpoint images are reorganized to form a larger strip-shaped image, which is referred to as a multi-view video block.

[0155] In some embodiments, the visual media content is media content presented at the same time in the same three-dimensional space. In some embodiments, the visual media content is media content presented at different times in the same three-dimensional space. In some embodiments, the visual media content can also be media content in different three-dimensional spaces. That is, the at least two visual media contents are not specifically limited in the embodiments of the present application.

[0156] Step 602: Splicing the at least one homogenous block to obtain at least one spliced image and spliced image information, wherein the spliced image information includes a first syntax element, and according to the first syntax element, it is determined that the spliced image is a heterogeneous mixed spliced image or a homogenous spliced image, the heterogeneous mixed spliced image includes at least two homogenous blocks, and the homogenous spliced image includes one homogenous block;

[0157] In some embodiments, the stitching the at least one isochoric block to obtain at least one stitched picture and stitching picture information comprises: stitching isochoric blocks of at least two expression formats to generate a heterogeneous mixed stitched picture and stitching picture information; and stitching isochoric blocks of the same expression format to generate a homogenous stitched picture and stitching picture information.

[0158] For example, the at least one isochoric block comprises a first isochoric block of a first expression format and a second isochoric block of a second expression format. The method specifically comprises: stitching the first isochoric block of the first expression format to obtain a first homogenous stitched picture and stitching picture information, and stitching the second isochoric block of the second expression format to obtain a second homogenous stitched picture and stitching picture information; or stitching the first isochoric block of the first expression format and the second isochoric block of the second expression format to obtain a heterogeneous mixed stitched picture and stitching picture information; or stitching the first isochoric block of the first expression format to obtain a first homogenous stitched picture and stitching picture information, and stitching the first isochoric block of the first expression format and the second isochoric block of the second expression format to obtain a heterogeneous mixed stitched picture and stitching picture information; or stitching the second isochoric block of the second expression format to obtain a second homogenous stitched picture and stitching picture information, and stitching the first isochoric block of the first expression format and the second isochoric block of the second expression format to obtain a heterogeneous mixed stitched picture and stitching picture information.

[0159] That is, the homogenous stitched picture can comprise one isochoric block or multiple isochoric blocks of the same expression format, and the heterogeneous mixed stitched picture comprises at least two isochoric blocks of at least two expression formats. In the embodiments of the present application, the first expression format is one of multi-view video, point cloud and mesh, the second expression format is one of multi-view video, point cloud and mesh, and the first expression format and the second expression format are different expression formats. For example, Figure 7 As shown in FIG. 6, multi-view video block 1, multi-view video block 2 and point cloud block 1 are stitched to obtain a heterogeneous mixed stitched picture.

[0160] For example, the first expression format is multi-view video, and the second expression format is point cloud. The stitching the at least one isochoric block to obtain at least one stitched picture and stitching picture information comprises: stitching a part of multi-view video blocks and a part of point cloud blocks into a heterogeneous mixed stitched picture; stitching another part of multi-view video blocks into a multi-view stitched picture; and stitching another part of point cloud blocks into a point cloud stitched picture.

[0161] The stitching picture information comprises a first syntax element, and the first syntax element is used to indicate whether the stitched picture is a heterogeneous mixed stitched picture or a homogenous stitched picture.

[0162] In some embodiments, the determining the splicing graph as a heterogeneous mixed splicing graph or a homogeneous splicing graph according to the first syntax element comprises: when the first syntax element is a first preset value, determining the splicing graph as a heterogeneous mixed splicing graph comprising homogeneous blocks of a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; when the first syntax element is a second preset value, determining the splicing graph as a homogeneous splicing graph comprising homogeneous blocks of the first expression format; and when the first syntax element is a third preset value, determining the splicing graph as a homogeneous splicing graph comprising homogeneous blocks of the second expression format. That is, different values of the first syntax element are used to indicate the type of the splicing graph. Further, the first syntax element can also be set to other values to indicate that the splicing graph is a homogeneous splicing graph comprising homogeneous blocks of other expression formats, or to indicate that the splicing graph is a heterogeneous mixed splicing graph comprising homogeneous blocks of at least two other expression formats.

[0163] In some embodiments, the first syntax element comprises at least two sub-syntax elements. For example, the first syntax element comprises a first sub-syntax element and a second sub-syntax element, and the determining the splicing graph as a heterogeneous mixed splicing graph or a homogeneous splicing graph according to the first syntax element comprises: when the first sub-syntax element is a fourth preset value, determining that the splicing graph comprises homogeneous blocks of the first expression format; and when the second sub-syntax element is a fifth preset value, determining that the splicing graph comprises homogeneous blocks of the second expression format.

[0164] It can be understood that when the first sub-syntax element is the fourth preset value, it is determined that the splicing graph comprises homogeneous blocks of the first expression format, i.e., the splicing graph is determined as a homogeneous splicing graph comprising homogeneous blocks of the first expression format; when the second sub-syntax element is the fifth preset value, it is determined that the splicing graph comprises homogeneous blocks of the second expression format, i.e., the splicing graph is determined as a homogeneous splicing graph comprising homogeneous blocks of the second expression format; and when the first sub-syntax element is the fourth preset value and the second sub-syntax element is the fifth preset value, it is determined that the splicing graph comprises homogeneous blocks of the first expression format and homogeneous blocks of the second expression format, i.e., the splicing graph is determined as a heterogeneous mixed splicing graph comprising homogeneous blocks of the first expression format and the second expression format.

[0165] In some embodiments, the method further comprises: when the first sub-syntax element is a sixth preset value, determining that the splicing graph does not comprise homogeneous blocks of the first expression format; and when the second sub-syntax element is a seventh preset value, determining that the splicing graph does not comprise homogeneous blocks of the second expression format.

[0166] Specifically, when the first sub-syntactic element is the fourth preset value and the second sub-syntactic element is the fifth preset value, it is determined that the spliced picture is a heterogeneous mixed spliced picture including homogeneous blocks in the first expression format and homogeneous blocks in the second expression format; when the first sub-syntactic element is the fourth preset value and the second sub-syntactic element is the seventh preset value, it is determined that the spliced picture is a homogeneous spliced picture including homogeneous blocks in the first expression format; and when the first sub-syntactic element is the sixth preset value and the second sub-syntactic element is the fifth preset value, it is determined that the spliced picture is a homogeneous spliced picture including homogeneous blocks in the second expression format.

[0167] That is, the expression format of the homogeneous block in the spliced picture can also be determined according to the values of the two sub-syntactic elements. Further, when the spliced picture includes more expression formats, the expression format of the homogeneous block in the spliced picture can also be indicated by multiple syntactic elements. For example, when three expression formats are included, three syntactic elements are set, when four expression formats are included, four syntactic elements are set, or multiple values of one syntactic element can also be set to represent multiple expression formats.

[0168] In some embodiments, the first syntactic element is located in a parameter set of the code stream. For example, the parameter set of the code stream can be V3C_VPS, and the first syntactic element can be ptl_profile_toolset_idc in V3C_VPS.

[0169] In some embodiments, the spliced picture sequence parameter set corresponding to the spliced picture includes the first syntactic element. For example, the spliced picture sequence parameter set corresponding to the spliced picture includes the first sub-syntactic element and the second sub-syntactic element. For example, the first sub-syntactic element is asps_vpcc_extension_present_flag in the spliced picture sequence parameter set, and the second sub-syntactic element is asps_miv_extension_present_flag.

[0170] That is, the first syntactic element can be located in the parameter set of the code stream, so that the decoding end can parse the spliced picture type of each spliced picture earlier. The first syntactic element can also be located in the spliced picture sequence parameter set corresponding to each spliced picture, so that the decoding end can obtain and determine the spliced picture type when parsing each spliced picture.

[0171] In some embodiments, the heterogeneous mixed spliced picture of the embodiments of the present application includes at least one of the following: a single-attribute heterogeneous mixed spliced picture and a multi-attribute heterogeneous mixed spliced picture.

[0172] The single-attribute heterogeneous hybrid mosaic map refers to a heterogeneous hybrid mosaic map including all isomorphic blocks with the same attribute information. For example, a single-attribute heterogeneous hybrid mosaic map includes only isomorphic blocks with attribute information, such as only multi-view video texture blocks and point cloud texture blocks. For another example, a single-attribute heterogeneous hybrid mosaic map includes only isomorphic blocks with geometric information, such as only multi-view video geometry blocks and point cloud geometry blocks.

[0173] The multi-attribute heterogeneous hybrid mosaic map refers to a heterogeneous hybrid mosaic map including at least two isomorphic blocks with different attribute information. For example, a multi-attribute heterogeneous hybrid mosaic map includes both isomorphic blocks with attribute information and isomorphic blocks with geometric information. As an example, any one attribute or any two attributes of at least two of point cloud, multi-view video, and mesh can be spliced into a map to obtain a heterogeneous hybrid mosaic map. The present application does not make any limitation in this regard.

[0174] In some embodiments, the single-attribute isomorphic blocks in the first expression format and the single-attribute blocks in the second expression format are spliced to obtain a heterogeneous hybrid mosaic map. The first expression format and the second expression format are any one of multi-view video, point cloud, and mesh, and the first expression format and the second expression format are different, and the attribute information of the first expression format and the second expression format is the same.

[0175] The single-attribute isomorphic blocks of multi-view video include at least one of multi-view video texture blocks and multi-view video geometry blocks.

[0176] The single-attribute isomorphic blocks of point cloud include at least one of point cloud texture blocks, point cloud geometry blocks, and point cloud occupancy blocks.

[0177] The single-attribute isomorphic blocks of mesh include at least one of mesh texture blocks and mesh geometry blocks.

[0178] For example, at least two of multi-view video geometry blocks, point cloud geometry blocks, and mesh geometry blocks are spliced into a map to obtain a heterogeneous hybrid mosaic map. The heterogeneous hybrid mosaic map is referred to as a single-attribute heterogeneous hybrid mosaic map. For another example, at least two of multi-view video texture blocks, point cloud texture blocks, and mesh texture blocks are spliced into a map to obtain a heterogeneous hybrid mosaic map. The heterogeneous hybrid mosaic map is referred to as a single-attribute heterogeneous hybrid mosaic map.

[0179] In some embodiments, the multi-attribute isomorphic blocks in the first expression format and the multi-attribute isomorphic blocks in the second expression format are spliced to obtain a heterogeneous mixed splicing graph. The first expression format and the second expression format are any one of multi-view video, point cloud and mesh, and the first expression format and the second expression format are different, and the attribute information of the first expression format and the second expression format is not completely the same.

[0180] For example, the multi-view video texture block is spliced with at least one of the point cloud geometry block and the mesh geometry block in a graph to obtain a heterogeneous mixed splicing graph. For another example, the multi-view video geometry block is spliced with at least one of the point cloud texture block and the mesh texture block in a graph to obtain a heterogeneous mixed splicing graph. For another example, the point cloud texture block is spliced with at least one of the multi-view video geometry block and the mesh geometry block in a graph to obtain a heterogeneous mixed splicing graph. For another example, the point cloud geometry block is spliced with at least one of the multi-view video texture block and the mesh texture block in a graph to obtain a heterogeneous mixed splicing graph. For another example, the point cloud geometry block, the multi-view video texture block and the multi-view video texture block are spliced in a graph to obtain a heterogeneous mixed splicing graph. For another example, the point cloud geometry block, the point cloud texture block, the multi-view video texture block and the multi-view video texture block are spliced in a graph to obtain a heterogeneous mixed splicing graph. Here, the obtained heterogeneous mixed splicing graph is called a multi-attribute heterogeneous mixed splicing graph.

[0181] In the following, the first expression format is taken as multi-view video, and the second expression format is taken as point cloud as an example to introduce the splicing method in detail.

[0182] Suppose that the multi-view video block includes multi-view video texture block and multi-view video geometry block, and the point cloud block includes point cloud texture block, point cloud geometry block and point cloud occupancy block. Then, the above-mentioned heterogeneous splicing mode can include but is not limited to the following two modes:

[0183] Mode one: the multi-view video texture block, the multi-view video geometry block, the point cloud texture block, the point cloud geometry block and the point cloud occupancy block are all spliced in a heterogeneous mixed splicing graph.

[0184] Mode two: according to a preset heterogeneous splicing mode, the multi-view video texture block, the multi-view video geometry block, the point cloud texture block, the point cloud geometry block and the point cloud occupancy block are spliced to obtain M heterogeneous mixed splicing graphs, and M is a positive integer greater than or equal to 1.

[0185] In the manner two, at least the following examples can be included: example 1, the multi-view video texture blocks and the point cloud texture blocks are spliced to obtain a heterogeneous hybrid texture splicing map, the multi-view video geometry blocks and the point cloud geometry blocks are spliced to obtain a heterogeneous hybrid geometry splicing map, and the point cloud occupation situation blocks are separately taken as a hybrid splicing map. Example 2, the multi-view video texture blocks and the point cloud texture blocks are spliced to obtain a heterogeneous hybrid texture splicing map, the multi-view video geometry blocks, the point cloud geometry blocks and the point cloud occupation situation blocks are spliced to obtain a heterogeneous hybrid geometry and occupation situation splicing map. Example 3, the multi-view video texture blocks, the point cloud texture blocks and the point cloud occupation situation blocks are spliced to obtain a sub-heterogeneous hybrid splicing map, and the multi-view video geometry blocks and the point cloud geometry blocks are spliced to obtain another sub-heterogeneous hybrid splicing map. Further, after obtaining the M heterogeneous hybrid splicing maps, the M heterogeneous hybrid splicing maps can be respectively video encoded to obtain a video compression sub-code stream.

[0186] In some embodiments, the homogeneous splicing map of the embodiments of the present application includes at least one of the following: a single-attribute homogeneous splicing map and a multi-attribute homogeneous splicing map. In some embodiments, the first attribute homogeneous blocks of the first expression format are spliced to obtain a homogeneous splicing map. Alternatively, the first attribute homogeneous blocks and the second attribute homogeneous blocks of the first expression format are spliced to obtain a homogeneous splicing map.

[0187] The single-attribute homogeneous splicing map refers to a homogeneous splicing map including all homogeneous blocks with the same expression format and the same attribute information. For example, a single-attribute homogeneous splicing map only includes attribute information homogeneous blocks of the first expression format, such as a single-attribute homogeneous splicing map only including multi-view video texture blocks or only including point cloud texture blocks. For another example, a single-attribute homogeneous splicing map only includes geometry information homogeneous blocks, such as only including multi-view video geometry blocks or only including point cloud geometry blocks.

[0188] The multi-attribute homogeneous splicing map refers to a homogeneous splicing map including at least two homogeneous blocks with the same expression format but different attribute information, for example, a multi-attribute homogeneous splicing map including attribute information homogeneous blocks and geometry information homogeneous blocks. As an example, a multi-attribute homogeneous splicing map includes multi-view video texture blocks and multi-view video geometry blocks. For another example, a multi-attribute homogeneous splicing map includes point cloud geometry blocks and point cloud texture blocks, as shown in FIG. 1, a multi-attribute homogeneous splicing map includes point cloud texture block 1, point cloud geometry block 1 and point cloud geometry block 2. Figure 8

[0189] ​In some embodiments, the stitching picture information can further include a syntax element according to which it is determined that the stitching picture is a single-attribute heterogeneous mixed stitching picture, a multi-attribute heterogeneous mixed stitching picture, a single-attribute homogeneous stitching picture, or a multi-attribute homogeneous stitching picture.

[0190] Step 603: encoding the at least one stitching picture and the stitching picture information to obtain a bitstream.

[0191] In some embodiments, the bitstream includes a video compression sub-bitstream and a stitching picture information sub-bitstream. The encoding the at least one stitching picture and the stitching picture information to obtain a bitstream includes: encoding the at least one stitching picture to obtain the video compression sub-bitstream; encoding the stitching picture information of the at least one stitching picture to obtain the stitching picture information sub-bitstream; and combining the video compression sub-bitstream and the stitching picture information sub-bitstream to obtain the bitstream. In this way, it is realized that the same compression bitstream supports video, point cloud, mesh, and other heterogeneous source formats, and that multi-view video stitching pictures, point cloud video stitching pictures, mesh stitching pictures, and heterogeneous mixed stitching pictures exist in the compression bitstream at the same time, which can reduce the number of two-dimensional video encoders such as HEVC, VVC, AVC, and AVS that need to be called, reduce the implementation cost, and improve the ease of use.

[0192] In some embodiments, when it is determined according to the first syntax element that the stitching picture is a heterogeneous mixed stitching picture, the stitching picture information further includes a second syntax element according to which the expression format of an i-th region in the stitching picture is determined. Writing the second syntax element into the bitstream at the encoding end can help improve the decoding accuracy at the decoding end, and at the same time enable the V3C standard to support the same compression bitstream containing multi-view video and point cloud and other different expression formats of visual media content.

[0193] In some embodiments, the determining the expression format of the i-th region in the stitching picture according to the second syntax element includes: when the second syntax element is an eighth preset value, determining that the expression format of the i-th region is a first expression format; and when the second syntax element is a ninth preset value, determining that the expression format of the i-th region is a second expression format.

[0194] Specifically, the expression format type of the i-th region in the stitching picture can be indicated by setting different values for the second syntax element. Taking the first expression format as a point cloud and the second expression format as a multi-view video as an example, if the i-th region is a point cloud region, the second syntax element is set to the eighth preset value; and if the i-th region is a multi-view video region, the second syntax element is set to the ninth preset value. The specific values of the eighth preset value and the ninth preset value are not limited in the embodiments of the present application. Optionally, the eighth preset value is 0. Optionally, the ninth preset value is 1.

[0195] In some embodiments, the encoding the at least one mosaic picture and the mosaic picture information to obtain a bitstream comprises: if the expression format of the ith block is a first expression format, determining that the sub-patch in the ith block is encoded by using a coding standard corresponding to the first expression format to obtain a bitstream corresponding to the visual media content in the first expression format; and if the expression format of the ith block is a second expression format, determining that the sub-patch in the ith block is encoded by using a coding standard corresponding to the second expression format to obtain a bitstream corresponding to the visual media content in the second expression format.

[0196] In some embodiments, the second syntax element is located in a mosaic block data unit header of the ith block of the mosaic picture. In some embodiments, the second syntax element can also be located in a patch data unit (patch_data_unit). For example, it is determined that the current sub-patch is encoded by using a multi-view video coding standard on the premise that the second syntax element (ath_toolset_type) is 1. It is determined that the current sub-patch is encoded by using a point cloud coding standard on the premise that the second syntax element (ath_toolset_type) is 0.

[0197] In some embodiments, the encoding the at least one mosaic picture and the mosaic picture information to obtain a bitstream comprises: calling a video encoder to encode the at least one mosaic picture to obtain a video compression sub-bitstream.

[0198] In the embodiments of the present application, in order to reduce the number of encoders and reduce the coding cost, during encoding, first, at least two visual media contents are processed (i.e., packaged) respectively to obtain a plurality of homogeneous blocks. Then, at least two homogeneous blocks with different expression formats are spliced into a heterogeneous mixed mosaic picture, and at least one homogeneous block with the same expression format is spliced into a homogeneous mosaic picture. The heterogeneous mixed mosaic picture and the homogeneous mosaic picture are encoded to obtain a video compression sub-bitstream. The coding and decoding method is suitable for application scenarios of visual media contents with multiple expression formats, expands the application range, and through splicing homogeneous blocks with different expression formats into a heterogeneous mixed mosaic picture for encoding, only one video encoder needs to be called for encoding during encoding, thereby reducing the number of two-dimensional video encoders such as HEVC, VVC, AVC, and AVS that need to be called, reducing the coding cost, and improving the ease of use.

[0199] In the embodiments of the present application, the video encoder used for video encoding the heterogeneous mixed mosaic picture and the homogeneous mosaic picture to obtain a video compression sub-bitstream can be the above-mentioned Figure 2AThe video encoder shown. That is, the embodiments of the present application take the heterogeneous mixed splicing graph or the homogeneous splicing graph as a frame of image, first perform block division, then use intra-frame or inter-frame prediction to obtain the prediction value of the coding block, subtract the prediction value of the coding block from the original value to obtain the residual value, and after transformation and quantization processing of the residual value, obtain the video compression sub-code stream.

[0200] In the embodiments of the present application, while generating at least one splicing graph, the splicing graph information corresponding to each splicing graph is generated. The splicing graph information is encoded to obtain the splicing graph information sub-code stream. The splicing graph information includes a first syntax element for indicating the type of the splicing graph, and a second syntax element for indicating the expression format of each homogeneous block in the splicing graph. The encoding manner of the splicing graph information in the embodiments of the present application is not limited, for example, using conventional data compression encoding manners such as fixed-length encoding or variable-length encoding for compression.

[0201] Finally, the video compression sub-code stream and the splicing graph information sub-code stream are written in the same code stream to obtain the final code stream. That is, the embodiments of the present application not only realize supporting heterogeneous source formats and homogeneous source formats such as video, point cloud, and mesh in the same compression code stream.

[0202] In some embodiments, the method further includes: encoding the parameter set of the code stream to obtain a code stream parameter set sub-code stream. Specifically, the encoding end synthesizes the video compression sub-code stream, the splicing graph information sub-code stream, and the parameter set sub-code stream into a code stream. The code stream parameter set sub-code stream includes a third syntax element, and according to the third syntax element, the code stream corresponding to the visual media content in at least one expression format is determined. That is, the encoding end sends the third syntax element to indicate whether the code stream contains at least two kinds of visual media content in expression format. For example, when the third syntax element indicates that the code stream includes the code stream corresponding to the visual media content in one expression format, it can be understood that the encoding end processes the visual media content in one expression format to obtain one homogeneous block, and splices the one homogeneous block to obtain the homogeneous splicing graph. When the third syntax element indicates that the code stream includes the code stream corresponding to the visual media content in at least two expression formats, it can be understood that the encoding end processes the visual media content in at least two expression formats to obtain at least two homogeneous blocks, and splices the at least two homogeneous blocks to obtain the homogeneous splicing graph and / or the heterogeneous mixed splicing graph.

[0203] For example, when the third syntax element indicates that the bitstream includes the bitstream corresponding to the visual media content of at least two representation formats, the method includes: performing isomeric splicing on the isomeric blocks of the first representation format to obtain a first isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the second representation format to obtain a second isomeric spliced picture, or performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture, or performing isomeric splicing on the isomeric blocks of the first representation format to obtain a first isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture, or performing isomeric splicing on the isomeric blocks of the second representation format to obtain a second isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture.

[0204] In some embodiments, the third syntax element is set to different values to indicate that the bitstream includes the bitstream corresponding to the visual media content of at least one representation format. That is, certain preset values of the third syntax element can indicate that the bitstream includes the bitstream corresponding to the visual media content of one or more representation formats.

[0205] For example, when the third syntax element indicates that the bitstream includes the bitstream corresponding to the visual media content of at least two representation formats, the method includes: performing isomeric splicing on the isomeric blocks of the first representation format to obtain a first isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the second representation format to obtain a second isomeric spliced picture, or performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture, or performing isomeric splicing on the isomeric blocks of the first representation format to obtain a first isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture, or performing isomeric splicing on the isomeric blocks of the second representation format to obtain a second isomeric spliced picture, performing isomeric splicing on the isomeric blocks of the first representation format and the isomeric blocks of the second representation format to obtain a heterogeneous mixed spliced picture.

[0206] For example, the parameter set of the bitstream can be V3C_VPS, and the third syntax element can be ptl_profile_toolset_idc in V3C_VPS.

[0207] For example, when the third syntax element is set to a first value, the first value is used to indicate that the bitstream contains both multi-view video bitstream and point cloud bitstream. As a specific example, when ptl_profile_toolset_idc=X, X is 128 / 129 / 130 / 132 / 133 / 134, it indicates that the current bitstream contains both point cloud and multi-view bitstream. For another example, when the third syntax element is set to a second value, the second value is used to indicate that the bitstream contains only point cloud bitstream. As a specific example, when ptl_profile_toolset_idc=X, X is 0 / 1, it indicates that the current bitstream contains only point cloud bitstream. For another example, when the third syntax element is set to a third value, the third value is used to indicate that the bitstream contains only multi-view video bitstream. As a specific example, when ptl_profile_toolset_idc=X, X is 64 / 65 / 66, it indicates that the current bitstream contains only multi-view video bitstream. It should be understood that the above first value, second value, and third value are only examples, and embodiments of the present application are not limited thereto.

[0208] In this example, V3C_VPS in the existing V3C standard can be reused, and 0 / 1, 64 / 65 / 66, 128 / 129 / 130 / 132 / 133 / 134, and the like are pre-configured for ptl_profile_toolset_idc to indicate the type of bitstream contained in the current bitstream. When encoding visual media content, embodiments of the present application can help improve the decoding accuracy of the decoding end by adding the value of the third syntax element in the parameter set to indicate the type of bitstream containing the visual media content of the expression format, and can enable the V3C standard to support visual media content in one or more expression formats such as multi-view video, point cloud, mesh, and the like in the same compressed bitstream.

[0209] Table 3 shows one example of available toolset profile components. Table 3 provides a list of toolset profile components defined for V3C and their corresponding identification syntax element values, such as ptl_profile_toolset_idc and ptc_one_v3c_frame_only_flag, which can be used only for this document. Syntax element ptl_profile_toolset_idc provides the main definition of the toolset profile, and additional syntax elements such as ptc_one_v3c_frame_only_flag can specify additional features or restrictions of the defined profile. ptc_one_v3c_frame_only_flag can be used only to support a single V3C frame. It should be noted that 2..63, 67..127, 131, 135..255 in ptl_profile_toolset_idc are reserved and not defined at present, and a standard organization can make further provisions in future standards. The profile types defined in Table 3 can include dynamic or static.

[0210] Available toolset profile components

[0211]

[0212]

[0213] In some embodiments, the parameter set of the bitstream further includes a first syntax element, wherein the first syntax element is used to indicate a type of each spliced picture, specifically to indicate whether the spliced picture is the heterogeneous hybrid spliced picture or the homogeneous spliced picture; and the first syntax element is written into the parameter set of the bitstream. For example, a first syntax element (vps_toolset_type) is added in V3C_VPS to distinguish each spliced picture and the corresponding V3C unit to which the spliced picture belongs, i.e., a point cloud spliced picture, a multi-view spliced picture, or a heterogeneous hybrid spliced picture of point cloud and multi-view. In order to be compatible with the previous standards, the following new syntax and semantics are implemented, and the old semantics are also constrained.

[0214] For example, when the first syntax element is a first preset value, it is determined that the spliced picture is a heterogeneous mixed spliced picture including homogeneous blocks in a first expression format and a second expression format, where the first expression format and the second expression format are different expression formats; when the first syntax element is a second preset value, it is determined that the spliced picture is a homogeneous spliced picture including homogeneous blocks in the first expression format; and when the first syntax element is a third preset value, it is determined that the spliced picture is a homogeneous spliced picture including homogeneous blocks in the second expression format.

[0215] For example, taking a multi-view video as the first expression format and a point cloud as the second expression format, when the first syntax element is a first preset value, the first preset value is used to indicate that the spliced picture is a heterogeneous mixed spliced picture including point cloud blocks and multi-view video blocks; when the first syntax element is a second preset value, the second preset value is used to indicate that the spliced picture is a homogeneous spliced picture (which can be referred to as a multi-view video spliced picture) including multi-view video blocks; and when the first syntax element is a third preset value, the third preset value is used to indicate that the spliced picture is a homogeneous spliced picture (which can be referred to as a point cloud spliced picture) including point cloud blocks.

[0216] For example, after obtaining the third syntax element ptl_profile_toolset_idc = 128 / 129 / 130 / 132 / 133 / 134, in the VPS, the first syntax element (vps_toolset_type) needs to be parsed for each spliced picture to determine whether vps_toolset_type = X, where X is 1, indicating that the spliced picture only includes multi-view video blocks, and the multi-view encoding method requirement needs to be met; X is 2, indicating that the spliced picture only includes point cloud blocks, and the point cloud encoding method requirement needs to be met; and X is 3, indicating that the spliced picture includes both multi-view video blocks and point cloud blocks, and both the multi-view and point cloud encoding method requirements need to be met. It should be understood that the values of the above first syntax element are only examples, and embodiments of the present application are not limited thereto.

[0217] Table 4 shows the syntax of a general V3C parameter set (General V3C parameter set syntax), and the V3C parameter set adds a syntax element vps_toolset_type. Specifically, vps_toolset_type[j] can be used to represent the type of a spliced picture with an index of j. By adding the syntax element vps_toolset_type in the V3C parameter set, the decoding end can obtain vps_toolset_type from the V3C parameter set, quickly determine that each spliced picture and the corresponding V3C unit belong to a point cloud / multi-view / point cloud+multi-view according to vps_toolset_type, and thus determine which encoding method requirement the spliced picture should meet.

[0218] Table 4 General V3C parameter set syntax

[0219]

[0220]

[0221] In some embodiments, the stitching picture corresponding stitching picture sequence parameter set comprises the first syntax element. For example, the stitching picture corresponding stitching picture sequence parameter set comprises the first sub-syntax element and the second sub-syntax element. The first sub-syntax element and the second sub-syntax element are used to indicate the stitching picture type, wherein the stitching picture is the heterogeneous mixed stitching picture or the homogeneous stitching picture.

[0222] For example, when the first sub-syntax element is a fourth preset value and the second sub-syntax element is a fifth preset value, it is determined that the stitching picture is a heterogeneous mixed stitching picture comprising a homogeneous block of a first expression format and a homogeneous block of a second expression format; when the first sub-syntax element is a fourth preset value and the second sub-syntax element is a seventh preset value, it is determined that the stitching picture is a homogeneous stitching picture comprising the homogeneous block of the first expression format; when the first sub-syntax element is a sixth preset value and the second sub-syntax element is a fifth preset value, it is determined that the stitching picture is a homogeneous stitching picture comprising the homogeneous block of the second expression format.

[0223] Optionally, the first sub-syntax element is asps_vpcc_extension_present_flag in the stitching picture sequence parameter set, and the second sub-syntax element is asps_miv_extension_present_flag.

[0224] For example, the V3C_AD bitstream can comprise asps_miv_extension_present_flag and asps_vpcc_extension_present_flag in the NAL-ASPS.

[0225] In the embodiments of the present application, the first sub-syntax element and the second sub-syntax element are set to specific values to indicate that the stitching picture is the heterogeneous mixed stitching picture or the homogeneous stitching picture.

[0226] For example, after obtaining the third syntax element ptl_profile_toolset_idc = 128 / 129 / 130 / 132 / 133 / 134, the asps_vpcc_extension_present_flag = X and asps_miv_extension_present_flag = Y are obtained from the atlas information of the atlas. X is 0 and Y is 1, indicating that the atlas only has a multi-view video block, which should meet the multi-view encoding method requirement; X is 1 and Y is 0, indicating that the atlas only has a point cloud block, which should meet the point cloud encoding method requirement; X is 1 and Y is 1, indicating that the atlas has both a multi-view video block and a point cloud block, which should meet both the multi-view and point cloud encoding method requirements. It should be understood that the values of the above seventh preset value and eighth preset value are only examples, and the embodiments of the present application are not limited thereto.

[0227] Table 5 shows the syntax of the general atlas sequence parameter set (General atlas sequence parameter set RBSP syntax). The atlas sequence parameter set can be understood as the atlas information. The syntax elements asps_vpcc_extension_present_flag and asps_miv_extension_present_flag in the atlas sequence parameter set are used by the encoding end to represent the atlas type. The two syntax elements can be obtained from the atlas parameter set by the encoding end analyzing the code stream, and the atlas should be determined to belong to the point cloud / multi-view / point cloud + multi-view according to the values of the two syntax elements, so as to determine which encoding method requirement the atlas should meet.

[0228] Table 5 Syntax of general atlas sequence parameter set (General atlas sequence parameter set RBSP syntax)

[0229]

[0230]

[0231] After implementing the above syntax, a multi-view and point cloud atlas under a VPS can be realized. Further, it is necessary to realize the case that each isomorphic block in an atlas is a multi-view sub-block atlas or a point cloud sub-block atlas when multiple isomorphic blocks exist in an atlas. Since the prior art can only realize one kind of isomorphic block in an atlas, the second syntax element is added in the embodiments of the present application to determine the expression format of an isomorphic block in an atlas, which is multi-view video, point cloud, or grid, etc.

[0232] In some embodiments, the second syntax element is included in the atlas tile header of the i-th tile, and when the first syntax element indicates that the atlas is a heterogeneous mixed atlas, the atlas information further includes a second syntax element, and the expression format of the i-th tile in the atlas is determined according to the second syntax element.

[0233] In the encoding of the heterogeneous mixed atlas, the second syntax element is set to indicate the expression format of the i-th tile in the heterogeneous mixed atlas, which can help improve the decoding accuracy of the decoding end, and can enable the V3C standard to support different expression formats of visual media content such as multi-view video and point cloud in the same compressed bitstream. For example, the second syntax element can be ath_toolset_type in the atlas tile header (atlas_tile_header).

[0234] For example, when the second syntax element is an eighth preset value, it is determined that the expression format of the i-th tile is a first expression format; and when the second syntax element is a ninth preset value, it is determined that the expression format of the i-th tile is a second expression format.

[0235] For example, when the atlas is a heterogeneous mixed atlas, the atlas_tile_header in the ACL NAL unit type bitstream is parsed in the AD unit, and ath_toolset_type is parsed therefrom. When ath_toolset_type=X, X is 0, it indicates that the current tile is a point cloud tile; and when ath_toolset_type=X, X is 1, it indicates that the current tile is a multi-view video tile.

[0236] Table 6 shows the atlas tile header syntax. The encoding end adds a syntax element ath_toolset_type in the atlas tile header syntax to indicate the tile type. The decoding bitstream can obtain ath_toolset_type from the atlas tile header syntax to determine whether the current tile belongs to multi-view video decoding or point cloud decoding.

[0237] Table 6 Atlas tile header syntax

[0238]

[0239]

[0240] Optionally, the second syntax element can also be located in a subpatch data unit (patch data unit). For example, if the second syntax element (ath_toolset_type) is equal to 1, it is determined that the current subpatch is encoded using the multi-view video coding method. If the second syntax element (ath_toolset_type) is equal to 0, it is determined that the current subpatch is encoded using the point cloud coding method. The patch data unit syntax can be as shown in Table 7:

[0241] Table 7 Patch data unit syntax

[0242]

[0243] In some embodiments, vps_toolset_type[j] value equal to 1 indicates that the value of the syntax elements of the toolset profile component of the atlas with index j shall be in accordance with the values specified in ISO / IEC 23090-12 Table A-1-1 (i.e., Table 8);

[0244] vps_toolset_type[j] value equal to 2 indicates that the value of the syntax elements of the toolset profile component of the atlas with index j shall be in accordance with the values specified in ISO / IEC 23090-5 Table H-3, except for the values of vps_extension_present_flag, vps_packing_information_present_flag, vps_miv_extension_present_flag, vuh_unit_type, vps_atlas_count_minus 1, which shall be in accordance with the values specified in ISO / IEC 23090-12 Table A-1-1;

[0245] vps_toolset_type[j] value equal to 3 indicates that the value of the syntax elements of the toolset profile component of the atlas with index j shall be in accordance with the values specified in the extended ISO / IEC 23090-12 Table A-1-2 (i.e., Table 9-1 and Table 9-2); wherein Table A-1-1 and Table A-1-2 represent the restrictions of the related syntax of the toolset profile component for multi-view under integrated bitstream and the restrictions of the related syntax of the toolset profile component for heterogeneous data, respectively.

[0246] The vps_toolset_type[j] value of 0 or any value from 4 to 7 is reserved for future use by ISO / IEC and shall not appear in bitstreams conforming to this version of this document. Decoders conforming to this version of this document shall ignore such reserved unit types. Allowable values of syntax element values for the MIV toolset profile.

[0247] The ath_toolset_type value for the toolset profile component of the current tile shall be in the range of 0 to 1, as specified in Table A-1 of ISO / IEC 23090-12.

[0248] Table 8 - Allowable values of syntax element values for the MIV toolset profile

[0249]

[0250]

[0251] Table 9-1 - Allowable values of syntax element values for the MIV toolset profile (Extended)

[0252]

[0253]

[0254]

[0255] Table 9-2 - Allowable values of syntax element values for the MIV toolset profile (Extended)

[0256]

[0257]

[0258]

[0259] Figure 9An example of V3C bitstream structure provided by the embodiments of the present application is shown in the figure. The V3C parameter set (V3C_parameter_set()) of V3C_VPS can include ptl_profile_toolset_idc. When ptl_profile_toolset_idc is 128 / 129 / 130 / 132 / 133 / 134, it means that the current bitstream contains both point cloud bitstream (such as VPCC basic or VPCC extended, etc.) and multi-view video bitstream (such as MIV main or MIV Extended or MIV Geometry Absent, etc.).

[0260] The V3C parameter set (V3C_parameter_set()) of V3C_VPS can include a first syntax element (vps_toolset_type). When ptl_profile_toolset_idc is 128 / 129 / 130 / 132 / 133 / 134, vps_toolset_type is 1, which means that the current atlas only contains multi-view video blocks; vps_toolset_type is 2, which means that the current atlas only contains point cloud blocks; and vps_toolset_type is 3, which means that the current atlas contains both multi-view video blocks and point cloud blocks.

[0261] Alternatively, the atlas sub-bitstream (Atlas_sub_bitstream()) of V3C_AD can include asps_vpcc_extension_present_flag and asps_miv_extension_present_flag in the atlas sequence parameter set (Atlas_sequence_parameter_set_rbsp()) in NAL_ASPS. When ptl_profile_toolset_idc is 128 / 129 / 130 / 132 / 133 / 134, asps_vpcc_extension_present_flag=X and asps_miv_extension_present_flag=Y. When X is 0 and Y is 1, it means that the atlas only contains multi-view video blocks; when X is 1 and Y is 0, it means that the atlas only contains point cloud blocks; and when X is 1 and Y is 1, it means that the atlas contains both multi-view video blocks and point cloud blocks.

[0262] The atlas tile information is included in the ACL NAL unit type (ACL_NAL_unit_type) in the V3C_AD's Atlas_sub_bitstream(). For example, the atlas tile data unit (atlas_tiledata_unit()) can include the ath_toolset_type. If the ath_toolset_type is no (i.e., 0), it indicates that the current tile belongs to a point cloud tile, and if the atdu_type_flag is yes (i.e., 1), it indicates that the current tile belongs to a multi-view video tile.

[0263] Further, the patch information data (patch_information_data) includes the patch data unit (pach_data_unit). If the ath_toolset_type is no (i.e., 0), it indicates that the current patch is implemented by using the point cloud video coding method. If the ath_toolset_type is yes (i.e., 1), it indicates that the current patch is implemented by using the multi-view video coding method.

[0264] By obtaining the first syntax element of each atlas, it is determined according to the value of the first syntax element whether the atlas simultaneously includes a point cloud tile and a multi-view video tile. When it is determined that the atlas simultaneously includes a point cloud tile and a multi-view video tile, the ath_toolset_type of each tile in the atlas is obtained to determine the type of the tile.

[0265] The above describes the encoding method of the present application by taking the encoding end as an example. The video decoding method provided by the embodiments of the present application is described below by taking the decoding end as an example.

[0266] Figure 10 A schematic flowchart of a decoding method provided by the embodiments of the present application is shown in FIG. 10. As shown in FIG. 10, the decoding method of the present application includes the following steps. Figure 10

[0267] Step 1001: decoding a code stream to obtain an atlas and atlas information, wherein the atlas information includes a first syntax element, and it is determined according to the first syntax element whether the atlas is a heterogeneous hybrid atlas or a homogeneous atlas.

[0268] ​In some embodiments, the determining, according to the first syntax element, the spliced picture as a heterogeneous mixed spliced picture or a homogeneous spliced picture comprises: when the first syntax element is a first preset value, determining the spliced picture as a heterogeneous mixed spliced picture comprising homogeneous blocks in a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; when the first syntax element is a second preset value, determining the spliced picture as a homogeneous spliced picture comprising homogeneous blocks in the first expression format; and when the first syntax element is a third preset value, determining the spliced picture as a homogeneous spliced picture comprising homogeneous blocks in the second expression format.

[0269] In some embodiments, the first syntax element comprises a first sub-syntax element and a second sub-syntax element, and the determining, according to the first sub-syntax element and the second sub-syntax element, the spliced picture as a heterogeneous mixed spliced picture or a homogeneous spliced picture; accordingly, the determining, according to the first syntax element, the spliced picture as a heterogeneous mixed spliced picture or a homogeneous spliced picture comprises: when the first sub-syntax element is a fourth preset value, determining that the spliced picture comprises homogeneous blocks in the first expression format; and / or when the second sub-syntax element is a fifth preset value, determining that the spliced picture comprises homogeneous blocks in the second expression format; wherein the first expression format and the second expression format are different expression formats.

[0270] In some embodiments, the determining, according to the first syntax element, the spliced picture as a heterogeneous mixed spliced picture or a homogeneous spliced picture further comprises: when the first sub-syntax element is a sixth preset value, determining that the spliced picture does not comprise homogeneous blocks in the first expression format; and when the second sub-syntax element is a seventh preset value, determining that the spliced picture does not comprise homogeneous blocks in the second expression format.

[0271] Specifically, the determining, according to the first syntax element, the spliced picture as a heterogeneous mixed spliced picture or a homogeneous spliced picture comprises: when the first sub-syntax element is the fourth preset value and the second sub-syntax element is the fifth preset value, determining that the spliced picture is a heterogeneous mixed spliced picture comprising homogeneous blocks in the first expression format and homogeneous blocks in the second expression format; when the first sub-syntax element is the fourth preset value and the second sub-syntax element is the seventh preset value, determining that the spliced picture is a homogeneous spliced picture comprising homogeneous blocks in the first expression format; and when the first sub-syntax element is the sixth preset value and the second sub-syntax element is the fifth preset value, determining that the spliced picture is a homogeneous spliced picture comprising homogeneous blocks in the second expression format.

[0272] In some embodiments, the first syntax element is located in a parameter set sub-bitstream of the bitstream.

[0273] In some other embodiments, a spliced picture sequence parameter set corresponding to the spliced picture comprises the first syntax element.

[0274] In some embodiments, the at least one representation format comprises at least one of multi-view video, point cloud and mesh. Specifically, the first representation format is one of multi-view video, point cloud and mesh, the second representation format is one of multi-view video, point cloud and mesh, and the first representation format and the second representation format are different.

[0275] In some embodiments, the bitstream further comprises a parameter set sub-bitstream of the bitstream, the parameter set sub-bitstream of the bitstream comprises the third syntax element, and the bitstream in the bitstream comprises a bitstream corresponding to the visual media content of the at least one representation format according to the third syntax element. The method further comprises: decoding the parameter set sub-bitstream of the bitstream to obtain the parameter set of the bitstream, and obtaining the third syntax element from the parameter set of the bitstream.

[0276] In some embodiments, the method further comprises: when the third syntax element is a first numerical value, determining that the bitstream comprises a bitstream corresponding to the visual media content of the first representation format and a bitstream corresponding to the visual media content of the second representation format; when the third syntax element is a second numerical value, determining that the bitstream comprises a bitstream corresponding to the visual media content of the first representation format; and when the third syntax element is a third numerical value, determining that the bitstream comprises a bitstream corresponding to the visual media content of the second representation format.

[0277] In some embodiments, the decoding of the bitstream to obtain at least one stitched picture comprises: determining, according to the third syntax element, that the bitstream comprises bitstreams corresponding to the visual media content of at least two representation formats, and decoding the bitstream to obtain a heterogeneous mixed stitched picture.

[0278] In some embodiments, the decoding of the bitstream to obtain at least one stitched picture comprises: determining, according to the third syntax element, that the bitstream comprises bitstreams corresponding to the visual media content of at least two representation formats, and decoding the bitstream to obtain a homogeneous stitched picture of the at least two representation formats. That is, when the bitstream comprises bitstreams corresponding to the visual media content of at least two representation formats, each representation format corresponds to one homogeneous stitched picture.

[0279] In some embodiments, the decoding of the bitstream to obtain at least one stitched picture comprises: determining, according to the third syntax element, that the bitstream comprises bitstreams corresponding to the visual media content of at least two representation formats, and decoding the bitstream to obtain a heterogeneous mixed stitched picture and a homogeneous stitched picture of the at least two representation formats. That is, when the bitstream comprises bitstreams corresponding to the visual media content of at least two representation formats, a part of the homogeneous blocks of one representation format construct the heterogeneous mixed stitched picture, and a part of the homogeneous blocks of another representation format construct the homogeneous stitched picture.

[0280] In some embodiments, the heterogeneous mixed spliced picture is at least one of a single-attribute heterogeneous mixed spliced picture and a multi-attribute heterogeneous mixed spliced picture; and the homogeneous spliced picture is at least one of a single-attribute homogeneous spliced picture and a multi-attribute homogeneous spliced picture.

[0281] In some embodiments, the code stream includes a video compression sub-code stream and a spliced picture information sub-code stream, and the decoding the code stream to obtain at least one spliced picture and spliced picture information includes: decoding the video compression sub-code stream to obtain the at least one spliced picture; and decoding the spliced picture information sub-code stream to obtain spliced picture information of the at least one spliced picture. For example, according to the third syntax element, it is determined that the code stream includes code streams corresponding to visual media content in at least two expression formats, the video compression sub-code stream is decoded, and the code stream is decoded to obtain a heterogeneous mixed spliced picture. Alternatively, according to the third syntax element, it is determined that the code stream includes code streams corresponding to visual media content in at least two expression formats, the video compression sub-code stream is decoded, and the code stream is decoded to obtain a heterogeneous mixed spliced picture and a homogeneous spliced picture. Alternatively, according to the third syntax element, it is determined that the code stream includes code streams corresponding to visual media content in at least two expression formats, the video compression sub-code stream is decoded, and at least two expression formats of the homogeneous spliced picture are obtained.

[0282] Step 1002: when it is determined according to the first syntax element that the spliced picture is a heterogeneous mixed spliced picture, the spliced picture is split according to spliced picture information of the spliced picture to obtain at least two homogeneous blocks, wherein the at least two homogeneous blocks correspond to different expression formats of visual media content.

[0283] Step 1003: when it is determined according to the first syntax element that the spliced picture is a homogeneous spliced picture, the spliced picture is split according to spliced picture information of the spliced picture to obtain one homogeneous block, wherein the one homogeneous block corresponds to the same expression format of visual media content.

[0284] Step 1004: the homogeneous block is decoded and reconstructed to obtain visual media content in at least one expression format.

[0285] In some embodiments, when it is determined according to the first syntax element that the spliced picture is a heterogeneous mixed spliced picture, the spliced picture information further includes a second syntax element, and the expression format of the i-th block in the spliced picture is determined according to the second syntax element.

[0286] For example, the determination of the expression format of the i-th block in the spliced picture according to the second syntax element includes: when the second syntax element is an eighth preset value, it is determined that the expression format of the i-th block is a first expression format; and when the second syntax element is a ninth preset value, it is determined that the expression format of the i-th block is a second expression format.

[0287] In some embodiments, the second syntax element is located in a tile group data unit header of the i-th tile of the stitching picture.

[0288] In some embodiments, the decoding and reconstructing the isomorphic tiles to obtain the visual media content in at least one representation format comprises: if the representation format of the i-th tile is a first representation format, determining that the sub-tiles in the i-th tile are decoded and reconstructed by a decoding method corresponding to the first representation format to obtain the visual media content in the first representation format; and if the representation format of the i-th tile is a second representation format, determining that the sub-tiles in the i-th tile are decoded and reconstructed by a decoding method corresponding to the second representation format to obtain the visual media content in the second representation format.

[0289] For example, the decoding code stream obtains a multi-view video stitching picture, a point cloud stitching picture and a heterogeneous hybrid stitching picture. According to the stitching picture information of the heterogeneous hybrid stitching picture, the heterogeneous hybrid stitching picture is split to output reconstructed multi-view video tiles and point cloud tiles; according to the stitching picture information corresponding to the multi-view video stitching picture, the multi-view video stitching picture is split to output reconstructed multi-view video tiles; according to the stitching picture information corresponding to the point cloud stitching picture, the point cloud stitching picture is split to output reconstructed point cloud tiles; all the obtained multi-view video tiles are decoded to generate a reconstructed multi-view video; and all the obtained point cloud tiles are decoded to generate a reconstructed point cloud.

[0290] According to the technical solution, for the application scenario of visual media content including one or more representation formats, isomorphic tiles of different representation formats are stitched into a heterogeneous hybrid stitching picture, isomorphic tiles of the same representation format are stitched into a homogeneous stitching picture, and the obtained stitching picture and stitching picture information are written into a code stream. The code stream simultaneously includes a homogeneous stitching picture (for example, at least one of a multi-view stitching picture, a point cloud stitching picture and a mesh stitching picture) and a heterogeneous hybrid stitching picture, so that the coding and decoding method is applicable to the application scenario of visual media content in multiple representation formats, and the application range of the coding and decoding method is expanded. Moreover, the stitching picture information includes a first syntax element for indicating the type of the stitching picture, which improves the decoding efficiency of the stitching picture at the decoding end. Furthermore, since isomorphic tiles of different representation formats are stitched into a heterogeneous hybrid stitching picture for coding and decoding, the number of two-dimensional video codecs such as HEVC, VVC, AVC and AVS that need to be called is reduced, the implementation cost is reduced, and the ease of use is improved.

[0291] The application also provides an encoding device, Figure 11 An example of the encoding device provided by the application is shown in the schematic block diagram. The encoding device 110 is applied to an encoder. As shown in the figure, the encoding device 110 comprises: Figure 11 a stitching picture information obtaining unit 1101, a stitching picture generating unit 1102, a stitching picture information writing unit 1103 and a stitching picture information decoding unit 1104.

[0292] process at least one visual media content in at least one expression format to obtain at least one isomorphic tile, wherein different isomorphic tiles correspond to different visual media content expression formats;

[0293] stitch at least one isomorphic tile to obtain at least one stitched picture and stitched picture information, wherein the stitched picture information comprises a first syntax element, and the first syntax element is used to determine whether the stitched picture is a heterogeneous mixed stitched picture or an isomorphic stitched picture, the heterogeneous mixed stitched picture comprises at least two isomorphic tiles, and the isomorphic stitched picture comprises one isomorphic tile;

[0294] encode the at least one stitched picture and the stitched picture information to obtain a bitstream.

[0295] In some embodiments, the first syntax element is used to determine whether the stitched picture is the heterogeneous mixed stitched picture or the isomorphic stitched picture, including: when the first syntax element is a first preset value, it is determined that the stitched picture is the heterogeneous mixed stitched picture comprising the isomorphic tile of the first expression format and the second expression format, wherein the first expression format and the second expression format are different expression formats; when the first syntax element is a second preset value, it is determined that the stitched picture is the isomorphic stitched picture comprising the isomorphic tile of the first expression format; and when the first syntax element is a third preset value, it is determined that the stitched picture is the isomorphic stitched picture comprising the isomorphic tile of the second expression format.

[0296] In some embodiments, the first syntax element comprises a first sub-syntax element and a second sub-syntax element, and the first sub-syntax element and the second sub-syntax element are used to determine whether the stitched picture is the heterogeneous mixed stitched picture or the isomorphic stitched picture.

[0297] The first sub-syntax element is used to determine whether the stitched picture is the heterogeneous mixed stitched picture or the isomorphic stitched picture, including: when the first sub-syntax element is a fourth preset value, it is determined that the stitched picture comprises the isomorphic tile of the first expression format; and / or, when the second sub-syntax element is a fifth preset value, it is determined that the stitched picture comprises the isomorphic tile of the second expression format; wherein the first expression format and the second expression format are different expression formats.

[0298] In some embodiments, the first sub-syntax element is used to determine whether the stitched picture is the heterogeneous mixed stitched picture or the isomorphic stitched picture, including: when the first sub-syntax element is a sixth preset value, it is determined that the stitched picture does not comprise the isomorphic tile of the first expression format; and when the second sub-syntax element is a seventh preset value, it is determined that the stitched picture does not comprise the isomorphic tile of the second expression format.

[0299] In some embodiments, the first syntax element is located in a parameter set sub-bitstream of the bitstream.

[0300] In some embodiments, a splicing picture sequence parameter set corresponding to the splicing picture comprises the first syntax element.

[0301] In some embodiments, when the splicing picture is determined to be a heterogeneous mixed splicing picture according to the first syntax element, the splicing picture information further comprises a second syntax element, and an expression format of an i-th region in the splicing picture is determined according to the second syntax element.

[0302] In some embodiments, the determining the expression format of the i-th region in the splicing picture according to the second syntax element comprises: when the second syntax element is an eighth preset value, determining that the expression format of the i-th region is a first expression format; and when the second syntax element is a ninth preset value, determining that the expression format of the i-th region is a second expression format.

[0303] In some embodiments, the second syntax element is located in a splicing picture region data unit header of the i-th region of the splicing picture.

[0304] In some embodiments, the encoding unit 1103 is configured to: if the expression format of the i-th region is the first expression format, determine that sub-patches in the i-th region are encoded by using an encoding method corresponding to the first expression format to obtain a bitstream corresponding to visual media content in the first expression format; and if the expression format of the i-th region is the second expression format, determine that sub-patches in the i-th region are encoded by using an encoding method corresponding to the second expression format to obtain a bitstream corresponding to visual media content in the second expression format.

[0305] In some embodiments, a third syntax element is included in a parameter set sub-bitstream of the bitstream, and at least one expression format of visual media content included in the bitstream is determined according to the third syntax element.

[0306] In some embodiments, the determining the at least one expression format of visual media content included in the bitstream according to the third syntax element comprises: when the third syntax element is a first numerical value, determining that the bitstream simultaneously comprises a bitstream corresponding to visual media content in a first expression format and a bitstream corresponding to visual media content in a second expression format; when the third syntax element is a second numerical value, determining that the bitstream comprises the bitstream corresponding to the visual media content in the first expression format; and when the third syntax element is a third numerical value, determining that the bitstream comprises the bitstream corresponding to the visual media content in the second expression format.

[0307] In some embodiments, when the at least one stitched picture comprises a heterogeneous mixed stitched picture, the third syntax element is used to indicate that the code stream in the code stream comprises visual media content corresponding to at least two representation formats.

[0308] In some embodiments, the encoding unit 1103 is configured to encode the at least one stitched picture to obtain a video compression sub-code stream, encode stitched picture information of the at least one stitched picture to obtain a stitched picture information sub-code stream, and synthesize the video compression sub-code stream and the stitched picture information sub-code stream to obtain the code stream.

[0309] In some embodiments, the at least one representation format comprises at least one of a multi-view video, a point cloud, and a mesh.

[0310] In some embodiments, the heterogeneous mixed stitched picture comprises at least one of a single-attribute heterogeneous mixed stitched picture and a multi-attribute heterogeneous mixed stitched picture, and the homogeneous stitched picture comprises at least one of a single-attribute homogeneous stitched picture and a multi-attribute homogeneous stitched picture.

[0311] Embodiments of the present application also provide a decoding device, Figure 12 A schematic block diagram of a decoding device provided by an embodiment of the present application is shown in FIG. 12. The decoding device 120 is applied to a decoder. As shown in FIG. 12, the decoding device 120 comprises: Figure 12

[0312] A decoding unit 1201 is configured to decode a code stream to obtain a stitched picture and stitched picture information, wherein the stitched picture information comprises a first syntax element, and the first syntax element is used to determine whether the stitched picture is a heterogeneous mixed stitched picture or a homogeneous stitched picture.

[0313] A splitting unit 1202 is configured to, when the first syntax element is used to determine that the stitched picture is a heterogeneous mixed stitched picture, split the stitched picture according to the stitched picture information of the stitched picture to obtain at least two homogeneous blocks, wherein the at least two homogeneous blocks correspond to different representation formats of visual media content.

[0314] The splitting unit 1202 is configured to, when the first syntax element is used to determine that the stitched picture is a homogeneous stitched picture, split the stitched picture according to the stitched picture information of the stitched picture to obtain one homogeneous block, wherein the one homogeneous block corresponds to the same representation format of visual media content.

[0315] A processing unit 1203 is configured to decode and reconstruct the homogeneous blocks to obtain visual media content in at least one representation format.

[0316] ​In some embodiments, the determining, according to the first syntax element, whether the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture comprises: when the first syntax element is a first preset value, determining that the spliced picture is a heterogeneous mixed spliced picture comprising homogeneous blocks in a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; when the first syntax element is a second preset value, determining that the spliced picture is a homogeneous spliced picture comprising homogeneous blocks in the first expression format; and when the first syntax element is a third preset value, determining that the spliced picture is a homogeneous spliced picture comprising homogeneous blocks in the second expression format.

[0317] In some embodiments, the first syntax element comprises a first sub-syntax element and a second sub-syntax element, and the determining, according to the first sub-syntax element and the second sub-syntax element, whether the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture.

[0318] The determining, according to the first syntax element, whether the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture comprises: when the first sub-syntax element is a fourth preset value, determining that the spliced picture comprises homogeneous blocks in a first expression format; and / or when the second sub-syntax element is a fifth preset value, determining that the spliced picture comprises homogeneous blocks in a second expression format; wherein the first expression format and the second expression format are different expression formats.

[0319] In some embodiments, the determining, according to the first syntax element, whether the spliced picture is a heterogeneous mixed spliced picture or a homogeneous spliced picture further comprises: when the first sub-syntax element is a sixth preset value, determining that the spliced picture does not comprise homogeneous blocks in the first expression format; and / or when the second sub-syntax element is a seventh preset value, determining that the spliced picture does not comprise homogeneous blocks in the second expression format.

[0320] In some embodiments, the first syntax element is located in a parameter set sub-code stream of the code stream.

[0321] In some embodiments, a spliced picture sequence parameter set corresponding to the spliced picture comprises the first syntax element.

[0322] In some embodiments, when the spliced picture is determined to be a heterogeneous mixed spliced picture according to the first syntax element, the spliced picture information further comprises a second syntax element, and an expression format of an i-th block in the spliced picture is determined according to the second syntax element.

[0323] In some embodiments, the determining the representation format of the i-th tile in the mosaic according to the second syntax element comprises: when the second syntax element is an eighth preset value, determining the representation format of the i-th tile as a first representation format; and when the second syntax element is a ninth preset value, determining the representation format of the i-th tile as a second representation format.

[0324] In some embodiments, the second syntax element is located in a mosaic tile data unit header of the i-th tile of the mosaic.

[0325] In some embodiments, the processing unit 1203 is configured to, when the representation format of the i-th tile is the first representation format, determine that the sub-tiles in the i-th tile are reconstructed by using a decoding method corresponding to the first representation format to obtain the visual media content in the first representation format; and when the representation format of the i-th tile is the second representation format, determine that the sub-tiles in the i-th tile are reconstructed by using a decoding method corresponding to the second representation format to obtain the visual media content in the second representation format.

[0326] In some embodiments, the parameter set sub-code stream of the code stream comprises a third syntax element, and the code stream comprising the code stream corresponding to at least one representation format of visual media content is determined according to the third syntax element.

[0327] In some embodiments, the determining the code stream comprising the code stream corresponding to at least one representation format of visual media content according to the third syntax element comprises: when the third syntax element is a first numerical value, determining that the code stream comprises a code stream corresponding to visual media content in a first representation format and a code stream corresponding to visual media content in a second representation format; when the third syntax element is a second numerical value, determining that the code stream comprises the code stream corresponding to the visual media content in the first representation format; and when the third syntax element is a third numerical value, determining that the code stream comprises the code stream corresponding to the visual media content in the second representation format.

[0328] In some embodiments, the decoding unit 1201 is configured to, according to the third syntax element, determine that the code stream comprises code streams corresponding to visual media content in at least two representation formats, and decode the code stream to obtain a heterogeneous mosaic.

[0329] In some embodiments, the code stream comprises a video compression sub-code stream and a mosaic information sub-code stream, and the decoding unit 1201 is configured to decode the video compression sub-code stream to obtain the at least one mosaic, and decode the mosaic information sub-code stream to obtain the mosaic information of the at least one mosaic.

[0330] In some embodiments, the at least one representation format comprises at least one of a multi-view video, a point cloud, and a mesh.

[0331] In some embodiments, the heterogeneous hybrid stitching graph is at least one of a single-attribute heterogeneous hybrid stitching graph and a multi-attribute heterogeneous hybrid stitching graph; and the homogeneous stitching graph is at least one of a single-attribute homogeneous stitching graph and a multi-attribute homogeneous stitching graph.

[0332] It should be understood that the apparatus embodiments and the method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, no further description is given here.

[0333] The apparatus and system of the embodiments of the present application are described above from the perspective of functional units in combination with the drawings. It should be understood that the functional units can be implemented in the form of hardware, or in the form of instructions of software, or in the form of a combination of hardware and software units. Specifically, the steps of the method embodiments in the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor and / or the instructions of software form. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processing for execution by the processor, or executed by a combination of hardware and software units in the code processing processor. Alternatively, the software unit can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiments in combination with the hardware thereof.

[0334] In practical applications, the embodiments of the present application also provide an encoder, Figure 13 The schematic block diagram of the encoder provided by an embodiment of the present application is shown in FIG. 13. As shown in FIG. 13, the encoder 1310 includes: Figure 13

[0335] The second memory 1320 stores a computer program executable on the second processor 1330, and the second processor 1330 executes the program to implement the encoding method on the encoder side.

[0336] In practical applications, the embodiments of the present application also provide a decoder, Figure 14 The schematic block diagram of the decoder provided by an embodiment of the present application is shown in FIG. 14. As shown in FIG. 14, the decoder 1410 includes: Figure 14

[0337] The first memory 1420 stores a computer program executable on the first processor 1430, and the first processor 1430 executes the program to implement the decoding method on the decoder side.

[0338] In some embodiments of the present application, the processor can include but is not limited to: ​​

[0339] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0340] In some embodiments of the present application, the memory includes but is not limited to:

[0341] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0342] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0343] In yet another embodiment of the present application, see Figure 15 , which shows a schematic diagram of the composition structure of a coding and decoding system provided by an embodiment of the present application. Figure 15As shown, the codec system 150 can include an encoder 1501 and a decoder 1502. The encoder 1501 can be a device integrated with the aforementioned encoding apparatus; and the decoder 1502 can be a device integrated with the aforementioned decoding apparatus.

[0344] In the embodiments of the present application, in the codec system 150, whether the encoder 1501 or the decoder 1502, the color component information of the neighboring reference pixels and the to-be-predicted pixel can be used to calculate the weighting coefficients corresponding to the to-be-predicted pixel; and different reference pixels can have different weighting coefficients, which are applied to the chroma prediction of the to-be-predicted pixel in the current block, so as to not only improve the accuracy of the chroma prediction and save the code rate, but also improve the coding performance.

[0345] The embodiments of the present application further provide a chip for implementing the above-mentioned coding method. Specifically, the chip includes a processor configured to call and run a computer program from a memory, so that an electronic device installed with the chip performs the coding method as described above.

[0346] The embodiments of the present application further provide a computer storage medium, which stores a computer program. When the computer program is executed by a second processor, the coding method of the encoder is implemented; or when the computer program is executed by a first processor, the decoding method of the decoder is implemented. Alternatively, the embodiments of the present application further provide a computer program product including instructions, which are executed by a computer to make the computer perform the method of the above-mentioned method embodiments.

[0347] The present application further provides a bitstream, which is generated according to the above-mentioned encoding method. Optionally, the bitstream includes the first syntax element, or includes the second syntax element and the third syntax element.

[0348] When implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, or a twisted pair, as examples, then the coaxial cable, fiber optic cable, or twisted pair are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), and Blu-Ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0349] In some embodiments, the functions of the various examples described herein are implemented as part of an operating system or application program. Such implementation is typically done using one or more modular programming techniques. Further, since individual components of the systems have been implemented using a modular design, one of ordinary skill in the art will appreciate that the systems can be implemented in a variety of environments.

[0350] In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus, and method can be implemented in other ways. For example, the apparatus embodiments described above are merely schematic, and the division of the units is merely a logical function division. In actual implementation, another division manner can be used, for example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0351] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may also be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme. For example, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0352] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A decoding method, wherein: include: Decoding the bitstream to obtain a spliced ​​graph and spliced ​​graph information, wherein the spliced ​​graph information includes a first syntax element, and determining whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph according to the first syntax element; When it is determined according to the first syntax element that the mosaic is a heterogeneous mixed mosaic, splitting the mosaic according to the mosaic information of the mosaic to obtain at least two isomorphic blocks, wherein the at least two isomorphic blocks correspond to different visual media content expression formats; When it is determined according to the first syntax element that the spliced ​​graph is a homogeneous spliced ​​graph, splitting the spliced ​​graph according to the spliced ​​graph information of the spliced ​​graph to obtain homogeneous blocks, wherein the homogeneous blocks correspond to the same visual media content expression format; The homogeneous blocks are decoded and reconstructed to obtain visual media content in at least one expression format.

2. The method according to claim 1, wherein The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph includes: If the first syntax element is a first preset value, it is determined that the spliced ​​graph is a heterogeneous mixed spliced ​​graph including homogeneous blocks in a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; If the first syntax element is a second preset value, determining that the spliced ​​graph is an isomorphic spliced ​​graph including isomorphic blocks in the first expression format; If the first syntax element is a third preset value, it is determined that the splicing graph is an isomorphic splicing graph including isomorphic blocks in the second expression format.

3. The method according to claim 1, wherein The first syntax element includes: a first sub-syntax element and a second sub-syntax element, and determining whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph according to the first sub-syntax element and the second sub-syntax element; The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph includes: If the first sub-syntax element is a fourth preset value, it is determined that the spliced ​​graph includes isomorphic blocks in the first expression format; If the second sub-syntax element is a fifth preset value, it is determined that the spliced ​​graph includes isomorphic blocks in a second expression format; The first expression format and the second expression format are different expression formats.

4. The method according to claim 3, wherein: The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph further includes: If the first sub-syntax element is a sixth preset value, it is determined that the splicing graph does not include isomorphic blocks in the first expression format; If the second sub-syntax element is a seventh preset value, it is determined that the splicing graph does not include isomorphic blocks in the second expression format.

5. The method according to any one of claims 1 to 4, wherein: The first syntax element is located in a parameter set sub-codestream of the codestream.

6. The method according to any one of claims 1 to 4, wherein: The mosaic sequence parameter set corresponding to the mosaic includes the first syntax element.

7. The method according to any one of claims 1 to 4, wherein: The method further comprises: When it is determined according to the first syntax element that the spliced ​​graph is a heterogeneous mixed spliced ​​graph, the spliced ​​graph information further includes a second syntax element, and an expression format of the i-th block in the spliced ​​graph is determined according to the second syntax element.

8. The method according to claim 7, wherein: The determining of the expression format of the i-th block in the spliced ​​graph according to the second syntax element includes: The second syntax element is an eighth preset value, determining that the expression format of the i-th block is the first expression format; If the second syntax element is a ninth preset value, it is determined that the expression format of the i-th block is the second expression format.

9. The method according to claim 7, wherein: The second syntax element is located in a mosaic map block data unit header of the i-th block of the mosaic map.

10. The method according to claim 7, wherein: The decoding and reconstruction of the homogeneous blocks to obtain visual media content in at least one expression format includes: If the expression format of the i-th block is a first expression format, determining to decode and reconstruct a sub-block in the i-th block using a decoding method corresponding to the first expression format to obtain visual media content in the first expression format; If the expression format of the i-th block is the second expression format, it is determined that the sub-block in the i-th block is decoded and reconstructed using a decoding method corresponding to the second expression format to obtain visual media content in the second expression format.

11. The method according to claim 1, wherein The parameter set sub-codestream of the codestream includes a third syntax element, and the codestream corresponding to the visual media content including at least one expression format in the codestream is determined according to the third syntax element.

12. The method according to claim 11, wherein The determining, according to the third syntax element, a codestream corresponding to the visual media content including at least one expression format in the codestream includes: The third syntax element is a first value, and determines that the code stream includes both a code stream corresponding to the visual media content in the first expression format and a code stream corresponding to the visual media content in the second expression format; The third syntax element is a second value, which determines that the code stream includes a code stream corresponding to the visual media content in the first expression format; The third syntax element is a third value, which determines the code stream corresponding to the visual media content in the second expression format included in the code stream.

13. The method according to any one of claims 11 to 12, wherein: The decoding bitstream to obtain at least one splicing graph includes: Determine, according to the third syntax element, that the bitstream includes bitstreams corresponding to visual media content in at least two expression formats, and decode the bitstream to obtain a heterogeneous mixed splicing graph.

14. The method according to any one of claims 1 to 4, wherein: The code stream includes a video compression sub-code stream and a splicing graph information sub-code stream, and the decoding code stream obtains at least one splicing graph and splicing graph information, including: Decoding the video compression sub-stream to obtain the at least one spliced ​​graph; The mosaic information sub-code stream is decoded to obtain mosaic information of the at least one mosaic.

15. The method according to any one of claims 1 to 4, wherein: The at least one expression format includes at least one of multi-view video, point cloud, and mesh.

16. The method according to any one of claims 1 to 4, wherein: The heterogeneous mixed mosaic graph is at least one of the following: a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph; The isomorphic mosaic graph includes at least one of the following: a single-attribute isomorphic mosaic graph and a multi-attribute isomorphic mosaic graph.

17. A coding method, wherein: include: Processing visual media content in at least two expression formats to obtain at least one isomorphic block, wherein different isomorphic blocks correspond to different visual media content expression formats; Splicing the at least one isomorphic block to obtain at least one spliced ​​graph and spliced ​​graph information, wherein the spliced ​​graph information includes a first syntax element, and determining, based on the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph, the heterogeneous mixed spliced ​​graph includes at least two isomorphic blocks, and the homogeneous spliced ​​graph includes one isomorphic block; The at least one splicing graph and the splicing graph information are encoded to obtain a code stream.

18. The method according to claim 17, wherein The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph includes: If the first syntax element is a first preset value, it is determined that the spliced ​​graph is a heterogeneous mixed spliced ​​graph including homogeneous blocks in a first expression format and a second expression format, wherein the first expression format and the second expression format are different expression formats; If the first syntax element is a second preset value, determining that the spliced ​​graph is an isomorphic spliced ​​graph including isomorphic blocks in the first expression format; If the first syntax element is a third preset value, it is determined that the splicing graph is an isomorphic splicing graph including isomorphic blocks in the second expression format.

19. The method according to claim 17, wherein The first syntax element includes: a first sub-syntax element and a second sub-syntax element, and determining whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph according to the first sub-syntax element and the second sub-syntax element; The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph includes: If the first sub-syntax element is a fourth preset value, it is determined that the spliced ​​graph includes isomorphic blocks in the first expression format; If the second sub-syntax element is a fifth preset value, it is determined that the spliced ​​graph includes isomorphic blocks in a second expression format; The first expression format and the second expression format are different expression formats.

20. The method according to claim 19, wherein The determining, according to the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph further includes: If the first sub-syntax element is a sixth preset value, it is determined that the splicing graph does not include isomorphic blocks in the first expression format; If the second sub-syntax element is a seventh preset value, it is determined that the splicing graph does not include isomorphic blocks in the second expression format.

21. The method according to any one of claims 17 to 20, wherein: The first syntax element is located in a parameter set sub-codestream of the codestream.

22. The method according to any one of claims 17 to 20, wherein: The mosaic sequence parameter set corresponding to the mosaic includes the first syntax element.

23. The method according to any one of claims 17 to 20, wherein: The method further comprises: When it is determined according to the first syntax element that the spliced ​​graph is a heterogeneous mixed spliced ​​graph, the spliced ​​graph information further includes a second syntax element, and an expression format of the i-th block in the spliced ​​graph is determined according to the second syntax element.

24. The method according to claim 23, wherein The determining of the expression format of the i-th block in the spliced ​​graph according to the second syntax element includes: The second syntax element is an eighth preset value, determining that the expression format of the i-th block is the first expression format; If the second syntax element is a ninth preset value, it is determined that the expression format of the i-th block is the second expression format.

25. The method according to claim 24, wherein The second syntax element is located in a mosaic map block data unit header of the i-th block of the mosaic map.

26. The method according to claim 23, wherein The encoding of the at least one mosaic image and the mosaic image information to obtain a code stream includes: If the expression format of the i-th block is a first expression format, determining that a sub-block in the i-th block is encoded using an encoding method corresponding to the first expression format to obtain a code stream corresponding to the visual media content in the first expression format; If the expression format of the i-th block is the second expression format, it is determined that the sub-block in the i-th block is encoded using the encoding method corresponding to the second expression format to obtain a code stream corresponding to the visual media content in the second expression format.

27. The method according to claim 17, wherein The parameter set sub-codestream of the codestream includes a third syntax element, and the codestream corresponding to the visual media content including at least one expression format in the codestream is determined according to the third syntax element.

28. The method according to claim 27, wherein The determining, according to the third syntax element, a codestream corresponding to the visual media content including at least one expression format in the codestream includes: The third syntax element is a first value, and determines that the code stream includes both a code stream corresponding to the visual media content in the first expression format and a code stream corresponding to the visual media content in the second expression format; The third syntax element is a second value, which determines that the code stream includes a code stream corresponding to the visual media content in the first expression format; The third syntax element is a third value, which determines the code stream corresponding to the visual media content in the second expression format included in the code stream.

29. The method according to any one of claims 27-28, wherein: When the at least one mosaic graph includes a heterogeneous mixed mosaic graph, a code stream corresponding to the visual media content in at least two expression formats in the code stream is determined according to the third syntax element.

30. The method according to any one of claims 17 to 20, wherein: The encoding of the at least one mosaic image and the mosaic image information to obtain a code stream includes: Encoding the at least one spliced ​​image to obtain a video compression sub-stream; Encoding the mosaic information of the at least one mosaic to obtain a mosaic information substream; The video compression sub-stream and the splicing graph information sub-stream are synthesized into the stream.

31. The method according to any one of claims 17 to 20, wherein: The at least one expression format includes at least one of multi-view video, point cloud, and mesh.

32. The method according to any one of claims 17 to 20, wherein: The heterogeneous mixed mosaic graph is at least one of the following: a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph; The isomorphic mosaic graph includes at least one of the following: a single-attribute isomorphic mosaic graph and a multi-attribute isomorphic mosaic graph.

33. A decoding device, wherein: include: a decoding unit, configured to decode a bitstream to obtain a splice graph and splice graph information, wherein the splice graph information includes a first syntax element, and determine whether the splice graph is a heterogeneous mixed splice graph or a homogeneous splice graph according to the first syntax element; a splitting unit, configured to, when determining according to the first syntax element that the mosaic is a heterogeneous mixed mosaic, split the mosaic according to the mosaic information of the mosaic to obtain at least two isomorphic blocks, wherein the at least two isomorphic blocks correspond to different visual media content expression formats; The splitting unit is configured to, when determining that the spliced ​​graph is a homogeneous spliced ​​graph according to the first syntax element, split the spliced ​​graph according to the spliced ​​graph information of the spliced ​​graph to obtain homogeneous blocks, wherein the homogeneous blocks correspond to the same visual media content expression format; The processing unit is configured to decode and reconstruct the homogeneous blocks to obtain visual media content in at least one expression format.

34. An encoding device, wherein: include: a processing unit, configured to process the visual media content in at least two expression formats to obtain at least one isomorphic block, wherein different isomorphic blocks correspond to different visual media content expression formats; a splicing unit, configured to splice the at least one isomorphic block to obtain at least one spliced ​​graph and spliced ​​graph information, wherein the spliced ​​graph information includes a first syntax element, and determining, based on the first syntax element, whether the spliced ​​graph is a heterogeneous mixed spliced ​​graph or a homogeneous spliced ​​graph, the heterogeneous mixed spliced ​​graph including at least two isomorphic blocks, and the homogeneous spliced ​​graph including one isomorphic block; The encoding unit is configured to encode the at least one splicing graph and the splicing graph information to obtain a code stream.

35. A decoder, wherein The decoder comprises: a first memory and a first processor; The first memory stores a computer program that can be run on the first processor, and the first processor implements the decoding method according to any one of claims 1 to 16 when executing the program.

36. An encoder, wherein The encoder comprises: a second memory and a second processor; The second memory stores a computer program that can be run on the second processor, and when the second processor executes the program, the encoding method according to any one of claims 17 to 32 is implemented.

37. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed by a first processor, it implements the decoding method described in any one of claims 1 to 16; or, when the computer program is executed by a second processor, it implements the encoding method described in any one of claims 17 to 32.

38. A computer-readable storage medium, wherein: The computer-readable storage medium stores a code stream generated by a processor executing the method according to any one of claims 17 to 32.

Images