Coding and decoding method and device, equipment and storage medium

CN119013987APending Publication Date: 2024-11-22ZHEJIANG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202280094854.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-04-18
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing encoding and decoding technology requires a large number of codecs to be called during multi-view video, point cloud encoding and mesh encoding, resulting in high encoding and decoding costs.

Method used

By splicing strips corresponding to visual media content in multiple different expression formats into a heterogeneous hybrid splicing image, the number of two-dimensional video codecs such as HEVC, VVC, and AVC that need to be called is reduced, and the splicing encoding method is used to generate code stream.

Benefits of technology

It reduces the number of encoders in the encoding and decoding process, reduces the encoding cost, and improves the efficiency and ease of use of encoding and decoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013987A_ABST
    Figure CN119013987A_ABST
Patent Text Reader

Abstract

The invention provides a coding and decoding method and device, equipment and a storage medium. Isomorphic strips corresponding to various visual media contents with different expression formats are spliced into a heterogeneous mixed spliced graph; visual media contents (namely heterogeneous contents) with different expression formats are orderly grouped and formed in the same heterogeneous mixed spliced graph, for example, a multi-view video strip and a point cloud strip are spliced in the same heterogeneous mixed spliced graph for coding and decoding, so that the number of HEVC, VVC, AVC, AVS and other two-dimensional video codecs needing to be called is reduced as much as possible, and the number of the HEVC, VVC, AVC, AVS and other two-dimensional video codecs needing to be called is reduced as much as possible. Coding and decoding cost is reduced, and usability is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Coding and decoding method, device, equipment and storage medium Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a coding and decoding method, apparatus, device, and storage medium. Background Art

[0002] In 3D application scenarios, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), visual media objects with different expression formats may appear in the same scene. For example, within the same 3D scene, the scene background and some characters and objects may be represented by video, while other characters may be represented by 3D point clouds or 3D meshes.

[0003] Using multi-view video coding, point cloud coding, and grid coding respectively during compression coding can better maintain the effective information of the original expression format than projecting all into multi-view video coding, improve the quality of the rendered viewing window during viewing, and improve the overall efficiency of bit rate-quality.

[0004] However, the current encoding and decoding technology is to encode and decode multi-viewpoint video, point cloud coding and grid grid separately. The number of codecs that need to be called in the encoding and decoding process is large, which makes the encoding and decoding cost high.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a coding and decoding method, apparatus, device, and storage medium to reduce the number of codecs called in the coding and decoding process and reduce the coding and decoding cost.

[0007] In a first aspect, the present application provides an encoding method, comprising:

[0008] Processing at least two visual media contents to obtain at least two isomorphic strips, wherein the visual media contents correspond to at least two different expression formats, and the isomorphic strips correspond to at least two different expression formats;

[0009] Splicing the at least two isomorphic strips to generate a heterogeneous mixed splicing graph;

[0010] The heterogeneous mixed spliced ​​graph is encoded to obtain a code stream.

[0011] In a second aspect, an embodiment of the present application provides a decoding method, including:

[0012] Decode the bitstream to obtain a reconstructed heterogeneous mixed splicing graph;

[0013] Splitting the reconstructed heterogeneous mixed mosaic graph to obtain at least two reconstructed isomorphic strips; the reconstructed isomorphic strips correspond to at least two different expression formats;

[0014] At least two reconstructed visual media contents are obtained according to the at least two reconstructed isomorphic strips, and the reconstructed visual media contents correspond to at least two different expression formats.

[0015] In a third aspect, the present application provides an encoding device for executing the method of the first aspect or its respective implementations. Specifically, the prediction device includes a functional unit for executing the method of the first aspect or its respective implementations.

[0016] In a fourth aspect, the present application provides a decoding device for executing the method in the second aspect or its respective implementations. Specifically, the prediction device includes a functional unit for executing the method in the second aspect or its respective implementations.

[0017] In a fifth aspect, an encoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the method of the first aspect or its respective implementations.

[0018] In a sixth aspect, a decoder is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to execute the method of the second aspect or its respective implementations.

[0019] In a seventh aspect, a coding and decoding system is provided, comprising an encoder and a decoder. The encoder is configured to execute the method of the first aspect or its respective implementations, and the decoder is configured to execute the method of the second aspect or its respective implementations.

[0020] In an eighth aspect, a chip is provided for implementing the method described in any one of the first and second aspects above, or their respective implementations. Specifically, the chip includes a processor configured to load and execute a computer program from a memory, causing a device equipped with the chip to perform the method described in any one of the first and second aspects above, or their respective implementations.

[0021] In a ninth aspect, a computer-readable storage medium is provided for storing a computer program, which enables a computer to execute the method of any one of the first to second aspects or their respective implementations.

[0022] In a tenth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method of any one of the first to second aspects or their respective implementations.

[0023] In an eleventh aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or their respective implementations.

[0024] In a twelfth aspect, a code stream is provided, which is generated based on the method of the first aspect.

[0025] Based on the above technical solution, by splicing the strips corresponding to visual media contents in multiple different expression formats into a heterogeneous mixed splicing graph, the visual media contents in different expression formats (i.e., heterogeneous contents) are orderly grouped and formed in the same heterogeneous mixed splicing graph. For example, multi-view video strips and point cloud strips are spliced ​​into a heterogeneous mixed splicing graph for encoding and decoding. This minimizes the number of two-dimensional video codecs such as HEVC, VVC, AVC, AVS, etc. that need to be called, reduces the encoding and decoding cost, and improves ease of use. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic block diagram of a video encoding and decoding system according to an embodiment of the present application;

[0027] FIG2A is a schematic block diagram of a video encoder according to an embodiment of the present application;

[0028] FIG2B is a schematic block diagram of a video decoder according to an embodiment of the present application;

[0029] FIG3A is a diagram showing the organization and expression framework of multi-view video data;

[0030] FIG3B is a schematic diagram of generating a stitched image of multi-view video data;

[0031] FIG3C is a diagram showing the organization and expression framework of point cloud data;

[0032] 3D to 3F are schematic diagrams of different types of point cloud data;

[0033] FIG4 is a schematic diagram of encoding of a multi-view video;

[0034] FIG5 is a schematic diagram of decoding of multi-view video;

[0035] FIG6 is a schematic diagram of a coding method flow chart provided in an embodiment of the present application;

[0036] FIG7A is a schematic diagram of a point cloud strip provided by an embodiment of the present application;

[0037] FIG7B is a schematic diagram of a multi-view video strip provided by an embodiment of the present application;

[0038] FIG8 is a schematic diagram of an encoding process according to an embodiment of the present application;

[0039] FIG9A is a schematic diagram of a heterogeneous mixed texture mosaic provided by an embodiment of the present application;

[0040] FIG9B is a schematic diagram of a heterogeneous mixed geometric mosaic graph provided by an embodiment of the present application;

[0041] FIG9C is a schematic diagram of a point cloud mixed occupancy mosaic provided by an embodiment of the present application;

[0042] FIG10 is a schematic diagram of a hybrid encoding process provided by an embodiment of the present application;

[0043] FIG11 is a schematic diagram of a V3C bitstream structure provided in an embodiment of the present application;

[0044] FIG12 is another schematic flow chart of an encoding method provided in one embodiment of the present application;

[0045] FIG13 is a schematic flow chart of a decoding method according to an embodiment of the present application;

[0046] FIG14 is a schematic diagram of a hybrid decoding process provided by an embodiment of the present application;

[0047] FIG15 is another schematic flow chart of a decoding method provided by an embodiment of the present application;

[0048] FIG16 is a schematic block diagram of an encoding device provided in an embodiment of the present application;

[0049] FIG17 is a schematic block diagram of a decoding device provided in an embodiment of the present application;

[0050] FIG18 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. For example, the solution of the present application can be combined with an audio and video coding standard (AVS), such as the H.264 / audio and video coding (AVC) standard, the H.265 / high efficiency video coding (HEVC) standard, and the H.266 / versatile video coding (VVC) standard. Alternatively, the solution of the present application can be combined with other proprietary or industry standards and operated, and the standards include ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of this application is not limited to any specific coding standard or technology.

[0052] The high-degree-of-freedom immersive coding system can be roughly divided into the following links according to the task line: data acquisition, data organization and expression, data encoding and compression, data decoding and reconstruction, data synthesis and rendering, and finally presenting the target data to the user.

[0053] The encoding involved in the embodiment of the present application is mainly video encoding and decoding. For ease of understanding, the video encoding and decoding system involved in the embodiment of the present application is first introduced in conjunction with Figure 1.

[0054] FIG1 is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that FIG1 is only an example, and the video encoding and decoding system of the embodiment of the present application includes but is not limited to that shown in FIG1. ​​As shown in FIG1, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compressing) the video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0055] The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, car computers, etc.

[0056] In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via a channel 130. The channel 130 may include one or more media and / or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

[0057] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to the decoding device 120. The communication media includes wireless communication media, such as radio frequency spectrum. Optionally, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0058] In another example, channel 130 includes a storage medium that can store the video data encoded by encoding device 110. The storage medium includes various locally accessible data storage media, such as optical disks, DVDs, and flash memories. In this example, decoding device 120 can retrieve the encoded video data from the storage medium.

[0059] In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0060] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0061] In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

[0062] The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0063] The video encoder 112 encodes the video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream contains the coding information of the picture or picture sequence in the form of a bitstream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. The SPS may contain parameters that apply to one or more sequences. The PPS may contain parameters that apply to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0064] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

[0065] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

[0066] In some embodiments, the decoding device 120 may further include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0067] The input interface 121 includes a receiver and / or a modem and can receive the encoded video data via the channel 130 .

[0068] The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

[0069] The decoded video data is displayed on the display device 123. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include various display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0070] In addition, Figure 1 is only an example, and the technical solution of the embodiment of the present application is not limited to Figure 1. For example, the technology of the present application can also be applied to unilateral video encoding or unilateral video decoding.

[0071] The following is an introduction to the video encoding framework involved in the embodiments of the present application.

[0072] FIG2A is a schematic block diagram of a video encoder according to an embodiment of the present application. It should be understood that the video encoder 200 can be used to perform lossy compression or lossless compression on an image. The lossless compression can be visually lossless or mathematically lossless.

[0073] The video encoder 200 can be applied to image data in a luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in terms of color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0074] For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTB may be referred to as a "tree block", "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chroma sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU can be further divided into a number of coding units (CUs) for encoding. The CU may be a rectangular block or a square block. A CU can be further divided into prediction units (PUs) and transform units (TUs), allowing for separation of coding, prediction, and transform, and greater flexibility in processing. In one example, a CTU is divided into CUs using a quadtree, and a CU is divided into TUs and PUs using a quadtree.

[0075] The video encoder and video decoder can support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0076] In some embodiments, as shown in FIG2A , the video encoder 200 may include a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components.

[0077] Optionally, in this application, the current block may be referred to as the current coding unit (CU) or the current prediction unit (PU), etc. The prediction block may also be referred to as a predicted image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0078] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame estimation unit 212. Because there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding and decoding technologies to eliminate spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames in a video, inter-frame prediction is used in video coding and decoding technologies to eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0079] The inter-frame prediction unit 211 is used for inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. It can refer to image information from different frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction can be P-frames and / or B-frames. P-frames refer to forward-predicted frames, while B-frames refer to bidirectionally predicted frames. Inter-frame prediction uses motion information to find a reference block from a reference frame and generates a prediction block based on the reference block. Motion information includes the reference frame list in which the reference frame is located, the reference frame index, and the motion vector. The motion vector can be integer pixel or fractional pixel. If the motion vector is fractional pixel, interpolation filtering is required to generate the required fractional pixel block in the reference frame. Here, the integer pixel or fractional pixel block in the reference frame found based on the motion vector is called a reference block. Some technologies directly use the reference block as the prediction block, while others further process the reference block to generate a prediction block. Reprocessing the reference block to generate a prediction block can also be understood as using the reference block as the prediction block and then processing the prediction block to generate a new prediction block.

[0080] The intra-frame estimation unit 212 only refers to the information of the same frame image to predict the pixel information in the current code image block to eliminate spatial redundancy. The frame used for intra-frame prediction can be an I frame.

[0081] Intra-frame prediction has multiple prediction modes. For example, the H-series international digital video coding standard H.264 / AVC has eight angular prediction modes and one non-angular prediction mode. H.265 / HEVC expands this to 33 angular prediction modes and two non-angular prediction modes. HEVC uses planar, DC, and 33 angular modes for a total of 35 intra-frame prediction modes. VVC uses planar, DC, and 65 angular modes for a total of 67 intra-frame prediction modes.

[0082] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital videos.

[0083] The residual unit 220 may generate a residual block for the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate the residual block for the CU such that each sample in the residual block has a value equal to the difference between the sample in the pixel blocks of the CU and the corresponding sample in the prediction blocks of the PUs of the CU.

[0084] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.

[0085] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.

[0086] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.

[0087] The loop filter unit 260 is used to process the inverse transformed and inverse quantized pixels to compensate for distortion information and provide a better reference for subsequent coded pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with the CU.

[0088] In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit and a sample adaptive offset / adaptive loop filtering (SAO / ALF) unit, wherein the deblocking filtering unit is used to remove blocking effects, and the SAO / ALF unit is used to remove ringing effects.

[0089] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use a reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra estimation unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

[0090] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data.

[0091] FIG2B is a schematic block diagram of a video decoder according to an embodiment of the present application.

[0092] 2B , the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded picture buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

[0093] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data based on the syntax elements extracted from the bitstream, thereby generating decoded video data.

[0094] In some embodiments, the prediction unit 320 includes an inter-frame prediction unit 321 and an intra-frame estimation unit 322 .

[0095] Intra estimation unit 322 may perform intra prediction to generate a prediction block for a PU. Intra estimation unit 322 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. Intra estimation unit 322 may also determine the intra prediction mode for the PU based on one or more syntax elements parsed from a codestream.

[0096] The inter-frame prediction unit 321 may construct a first reference picture list (List 0) and a second reference picture list (List 1) based on syntax elements parsed from the codestream. In addition, if a PU is encoded using inter-frame prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a prediction block for the PU based on the one or more reference blocks of the PU.

[0097] The inverse quantization / transform unit 330 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0098] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0099] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0100] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with a CU.

[0101] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0102] The basic process of video encoding and decoding is as follows: At the encoder end, a frame of image is divided into blocks. For the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block for the current block. The residual unit 220 calculates a residual block based on the predicted block and the original block of the current block. This residual block is the difference between the predicted block and the original block of the current block. This residual block is also called residual information. This residual block undergoes transformation and quantization by the transform / quantization unit 230, removing information that is insensitive to the human eye and eliminating visual redundancy. Optionally, the residual block before transformation and quantization by the transform / quantization unit 230 is called a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 is called a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized change coefficients output by the change quantization unit 230 and performs entropy coding on these quantized change coefficients to output a bitstream. For example, the entropy coding unit 280 can eliminate character redundancy based on the target context model and probability information of the binary bitstream.

[0103] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction for the current block based on the prediction information to generate a prediction block for the current block. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference frame for inter-frame prediction for subsequent frames.

[0104] It should be noted that the block division information determined by the encoder, as well as mode information or parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and analyzes the existing information to determine the same block division information, prediction, transform, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder, thereby ensuring that the decoded image obtained by the encoder and the decoder are identical.

[0105] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. This application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0106] In some application scenarios, multiple heterogeneous contents appear simultaneously in the same 3D scene, such as multi-view video and point cloud. For this situation, current encoding and decoding methods include at least the following two:

[0107] Method 1: MPEG (Moving Picture Experts Group) immersive video (MIV) technology is used for encoding and decoding multi-view videos, and point cloud video compression (VPCC) technology is used for encoding and decoding point clouds.

[0108] The following is an introduction to MIV technology and VPCC technology.

[0109] MIV technology: To reduce the transmission pixel rate while preserving as much scene information as possible to ensure sufficient information for rendering the target view, MPEG-I uses a solution, as shown in Figure 3A. A limited number of viewpoints are selected as base viewpoints, and the scene's visible range is expressed as much as possible. The base viewpoints are transmitted as complete images, and redundant pixels between the remaining non-base viewpoints and the base viewpoint are removed, retaining only the valid information that is not repeated. This valid information is then extracted into sub-block images, which are reorganized with the base viewpoint images to form a larger rectangular image, called a spliced ​​image. Figures 3A and 3B illustrate the generation process of the spliced ​​image. The spliced ​​image is fed into the codec for compression and reconstruction, and auxiliary data related to the splicing information of the sub-block images is also fed into the encoder to form the bitstream.

[0110] The VPCC encoding method is to project the point cloud into a two-dimensional image or video, converting the three-dimensional information into a two-dimensional information code. Figure 3C is the encoding block diagram of VPCC. The code stream is roughly divided into four parts. The geometry code stream is the code stream generated by the geometric depth map encoding, which is used to represent the geometric information of the point cloud; the attribute code stream is the code stream generated by the texture map encoding, which is used to represent the attribute information of the point cloud; the occupancy code stream is the code stream generated by the occupancy map encoding, which is used to indicate the valid area in the depth map and texture map. All three types of videos are encoded and decoded using a video encoder, as shown in Figures 3D to 3F. The auxiliary information code stream is the code stream generated by the encoding of the auxiliary information of the sub-block image, that is, the part related to the patch data unit in the V3C standard, which indicates information such as the position and size of each sub-block image.

[0111] In the second method, both multi-view videos and point clouds are encoded and decoded using the frame packing technology in Visual Volumetric Video-based Coding (V3C).

[0112] The following is an introduction to frame packing technology.

[0113] Taking multi-view video as an example, as shown in FIG4 , the encoding end includes the following steps:

[0114] In step 1, when encoding the acquired multi-view video, multi-view video sub-blocks (patches) are generated after some pre-processing. Then, the multi-view video sub-blocks are organized to generate a multi-view video mosaic.

[0115] For example, as shown in Figure 4, a multi-view video is input into TIMV for packaging, and a multi-view video mosaic is output. TIMV is a reference software for MIV. The packaging in the embodiment of the present application can be understood as splicing.

[0116] The multi-view video mosaic map includes a multi-view video texture mosaic map and a multi-view video geometry mosaic map, that is, it only contains multi-view video sub-blocks.

[0117] Step 2: Input the multi-view video mosaic image into the frame packer, and output the multi-view video mixed mosaic image.

[0118] Among them, the multi-view video mixed mosaic map includes a multi-view video texture mixed mosaic map, a multi-view video geometry mixed mosaic map, and a multi-view video texture and geometry mixed mosaic map.

[0119] Specifically, as shown in Figure 4, the multi-view video mosaic is frame-packed to generate a multi-view video hybrid mosaic. Each multi-view video mosaic occupies a region of the multi-view video hybrid mosaic. Accordingly, a flag, pin_region_type_id_minus2, is transmitted for each region in the bitstream. This flag records whether the current region belongs to a multi-view video texture mosaic or a multi-view video geometry mosaic. This information is required by the decoder.

[0120] Step 3: Use a video encoder to encode the multi-view video mixed mosaic image to obtain a bit stream.

[0121] Exemplarily, as shown in FIG5 , the decoding end includes the following steps:

[0122] Step 1: During multi-view video decoding, the acquired code stream is input into a video decoder for decoding to obtain a reconstructed multi-view video mixed splicing image.

[0123] Step 2: input the reconstructed multi-view video mixed mosaic image into the frame depacketizer, and output the reconstructed multi-view video mosaic image.

[0124] Specifically, first, obtain the flag pin_region_type_id_minus2 from the bitstream. If it is determined that pin_region_type_id_minus2 is V3C_AVD, it means that the current region is a multi-view video texture mosaic. Then, the current region is split and output as a reconstructed multi-view video texture mosaic.

[0125] If it is determined that the pin_region_type_id_minus2 is V3C_GVD, it means that the current region is a multi-view video geometric mosaic map, and the current region is split and output as a reconstructed multi-view video geometric mosaic map.

[0126] Step 3: decode the reconstructed multi-view video mosaic to obtain the reconstructed multi-view video.

[0127] Specifically, the multi-view video texture mosaic map and the multi-view video geometric mosaic map are decoded to obtain a reconstructed multi-view video.

[0128] The above uses multi-view video as an example to analyze and introduce the frame packing technology. The frame packing encoding and decoding method for point cloud is basically the same as that for the above multi-view video. For example, TMC (a reference software for VPCC) is used to pack the point cloud to obtain a point cloud mosaic map, and the point cloud mosaic map is input into the frame packer for frame packing to obtain a point cloud mixed mosaic map. The point cloud mixed mosaic map is stitched to obtain a point cloud code stream. I will not go into details here.

[0129] The following introduces the syntax related to frame packing in the standard.

[0130] The V3C unit header syntax is shown in Table 1:

[0131] Table 1

[0132]

[0133]

[0134] The semantics of the V3C unit header are shown in Table 2:

[0135] Table 2: V3C unit types

[0136]

[0137] At present, if multiple visual media contents in different expression formats appear simultaneously in the same three-dimensional scene, the visual media contents in the multiple expression formats are encoded and decoded separately. For example, in the case where point cloud and multi-view video appear simultaneously in the same three-dimensional scene, the current packaging technology is to compress the point cloud to form a point cloud compressed code stream (i.e., a V3C code stream), compress the multi-view video information to obtain a multi-view video compressed code stream (i.e., another V3C code stream), and then the system layer multiplexes the compressed code stream to obtain a fused three-dimensional scene multiplexed code stream. During decoding, the point cloud compressed code stream and the multi-view video compressed code stream are decoded separately. It can be seen from this that when the existing technology encodes and decodes visual media contents in multiple expression formats, many codecs are used, and the encoding and decoding cost is high.

[0138] In order to solve the above technical problems, the embodiment of the present application splices the strips corresponding to visual media contents of multiple different expression formats into a heterogeneous mixed splicing graph, so that the visual media contents of different expression formats (i.e., heterogeneous contents) are orderly grouped and formed in the same heterogeneous mixed splicing graph. For example, multi-view video strips and point cloud strips are encoded and decoded in a heterogeneous mixed splicing graph, thereby minimizing the number of two-dimensional video encoders such as HEVC, VVC, AVC, AVS that need to be called, reducing the encoding and decoding cost, and improving ease of use.

[0139] 6 , the video encoding method provided in the embodiment of the present application is introduced by taking the encoding end as an example.

[0140] FIG6 is a flow chart of an encoding method 600 provided in an embodiment of the present application. As shown in FIG6 , the method 600 in the embodiment of the present application includes:

[0141] S601: Process at least two visual media contents to obtain at least two isomorphic strips, wherein the visual media contents correspond to at least two different expression formats. Accordingly, the at least two isomorphic strips correspond to at least two different expression formats.

[0142] The visual media content corresponds to at least two different expression formats, that is, the at least two visual media contents include visual media content in at least two different expression formats. For example, the visual media content in the embodiment of the present application belongs to media content in two different expression formats such as multi-viewpoint video, point cloud, and grid.

[0143] A special case of a multi-view video is a single-view video, that is, the multi-view video may include multiple viewpoint videos and / or a single-viewpoint video.

[0144] The at least two isomorphic strips correspond to at least two different expression formats, that is, the at least two isomorphic strips include at least two isomorphic strips of different expression formats. For example, the isomorphic strips in the embodiment of the present application are strips of two different expression formats, such as multi-view video strips, point cloud strips, and mesh strips. One isomorphic strip corresponds to one expression format.

[0145] In three-dimensional application scenarios, such as virtual reality (VR), augmented reality (AR), and mixed reality (MR), visual media objects with different expression formats may appear in the same scene. For example, in the same three-dimensional scene, the scene background and some characters and objects may be expressed as video, while other characters may be expressed as three-dimensional point clouds or three-dimensional meshes.

[0146] In some embodiments, the at least two visual media contents are media contents presented simultaneously in the same three-dimensional space. In some embodiments, the at least two visual media contents are media contents presented at different times in the same three-dimensional space. In some embodiments, the at least two visual media contents may also be media contents in different three-dimensional spaces. That is, in the embodiments of the present application, there is no specific limitation on the at least two visual media contents.

[0147] In some embodiments, the expression formats of at least two visual media contents in the embodiments of the present application are different. For example, the at least two visual media contents include a point cloud and a multi-view video.

[0148] In some embodiments, the expression formats of some visual media contents in at least two visual media contents of the embodiments of the present application are the same, and the expression formats of some visual media contents are different. For example, at least two visual media contents include two point clouds and one multi-view video.

[0149] To improve compression efficiency, after obtaining at least two visual media contents, embodiments of the present application process the at least two visual media contents, such as by packaging (also known as splicing), to obtain strips corresponding to each of the at least two visual media contents. For example, the strips can be obtained by splicing sub-patches corresponding to the at least two visual media contents. It should be noted that embodiments of the present application do not limit the manner in which the at least two visual media contents are processed separately to obtain the strips.

[0150] It should be understood that a tile can be a mosaic of a specific shape, such as a mosaic of a rectangular area with a specific length and / or height. For example, at least one sub-tile can be sequentially tiled, such as by descending order of sub-tile area, or by descending order of sub-tile length and / or height, to obtain a tile corresponding to the visual media content. Optionally, a tile can be precisely mapped to an atlas tile.

[0151] In some embodiments, each sub-tile in a stripe may have a patch ID (patch ID) to distinguish different sub-tiles in the same stripe. For example, the same stripe may include sub-tile 1 (patch 1), sub-tile 2 (patch 2), and sub-tile 3 (patch 3).

[0152] Furthermore, a homogeneous stripe means that each sub-tile in the stripe has the same representation format. For example, each sub-tile in a homogeneous stripe is a multi-view video sub-tile, or a point cloud sub-tile, using the same representation format. The representation format corresponding to each sub-tile in a homogeneous stripe is the representation format corresponding to the homogeneous stripe.

[0153] In some embodiments, homogeneous strips may have a strip ID (tileID) to distinguish different strips of the same expression format. For example, homogeneous strips of the same expression format, such as point cloud strips, may include point cloud strip 1 or point cloud strip 2.

[0154] For example, multiple visual media contents include point clouds and multi-view videos. The point clouds are processed to obtain point cloud strips, as shown in Figure 7A, where point cloud strip 1 includes point cloud sub-blocks 1 to sub-blocks 3; the multi-view videos are processed to obtain multi-view video strips, as shown in Figure 7B, where the multi-view video strips include multi-view video sub-blocks 1 to sub-blocks 4.

[0155] In one possible implementation, if at least two (for example, two or more) isomorphic strips include multi-view video strips and point cloud strips, the above S601 can be specifically implemented as follows: after projecting and de-redundant processing on the acquired multi-view video, the non-repeated pixels are connected into video sub-blocks, and the video sub-blocks are spliced ​​into multi-view video strips; and the acquired point cloud is parallel projected, the connected points in the projection plane are combined into point cloud sub-blocks, and the point cloud sub-blocks are spliced ​​into point cloud strips.

[0156] Specifically, for multi-viewpoint video, taking MPEG-I as an example, a limited number of viewpoints are selected as basic viewpoints and the visible range of the scene is expressed as much as possible. The basic viewpoints are transmitted as complete images, and the redundant pixels between the remaining non-basic viewpoints and the basic viewpoints are removed, that is, only the valid information expressed in a non-repeated manner is retained, and then the valid information is extracted into sub-block images and reorganized with the basic viewpoint images to form a larger strip-shaped image. This strip-shaped image is called a multi-viewpoint video strip.

[0157] For point clouds, the three-dimensional point cloud is parallel projected to obtain a two-dimensional point cloud. In the projection plane, the connected points in the two-dimensional point cloud are grouped into point cloud sub-blocks, and these point cloud sub-blocks are spliced ​​to obtain point cloud strips.

[0158] According to the above method, after obtaining at least two isomorphic strips, the following S602 and S603 are executed.

[0159] S602: Splice at least two isomorphic strips to generate a heterogeneous mixed splicing graph.

[0160] S603: Encode the heterogeneous mixed spliced ​​graph to obtain a code stream.

[0161] As shown in Figure 8, in an embodiment of the present application, in order to reduce the number of encoders and reduce the encoding cost, during encoding, at least two visual media contents are first processed separately (i.e., packaged) to obtain at least two isomorphic strips. Then, at least two isomorphic strips with different expression formats are spliced ​​into a heterogeneous mixed splicing graph, and the heterogeneous mixed splicing graph is encoded to obtain a code stream. In other words, the embodiment of the present application encodes by splicing isomorphic strips of different expression formats into a heterogeneous mixed splicing graph. During encoding, the video encoder can be called only once, thereby reducing the number of two-dimensional video encoders such as HEVC, VVC, AVC, AVS that need to be called, reducing the encoding cost, and improving ease of use.

[0162] To distinguish from frame packing, in the embodiment of the present application, the process of splicing at least one homogeneous tile into a heterogeneous mixed tiled image is called tile packing.

[0163] In some embodiments, the above S603 includes: encoding the heterogeneous mixed mosaic graph using a video encoder to obtain a video code stream.

[0164] In the embodiment of the present application, when at least two isomorphic strips are spliced ​​into a heterogeneous hybrid spliced ​​image, hybrid splicing information is generated. This hybrid coding information is required during decoding, and therefore, this hybrid coding information needs to be encoded.

[0165] In some embodiments of the present application, a step of encoding the mixed splicing information is further included, that is, the above S603 includes the following steps:

[0166] S603-A, calling a video encoder to perform video encoding on the heterogeneous mixed mosaic image to obtain a video compression sub-stream;

[0167] S603-B, encoding the mixed splicing information of the heterogeneous mixed splicing graph to obtain a mixed splicing information sub-stream;

[0168] S603-C: Write the video compression sub-stream and the mixed splicing information sub-stream into the stream.

[0169] In the embodiment of the present application, the video encoder used to perform video encoding on the heterogeneous mixed mosaic to obtain a video compression substream can be the video encoder shown in Figure 2A above. That is, the embodiment of the present application treats the heterogeneous mixed mosaic as a frame image, first divides the image into blocks, then uses intra-frame or inter-frame prediction to obtain predicted values ​​for the coding blocks, subtracts the predicted values ​​from the original values ​​of the coding blocks to obtain residual values, and transforms and quantizes the residual values ​​to obtain the video compression substream.

[0170] At the same time, the hybrid splicing information of the heterogeneous hybrid splicing graph is encoded to obtain a hybrid splicing information sub-stream. The embodiment of the present application does not limit the encoding method of the hybrid splicing information, for example, conventional data compression encoding methods such as equal length encoding or variable length encoding are used for compression.

[0171] Finally, the video compression sub-stream and the mixed splicing information sub-stream are written into the same stream to obtain the final stream.

[0172] That is to say, the embodiments of the present application not only support heterogeneous source formats such as video, point cloud, and mesh in the same compressed code stream, but also realize that homogeneous strips in different expression formats such as multi-view video strips and point cloud (or mesh) strips exist simultaneously in a heterogeneous mixed splicing graph, thereby minimizing the number of video encoders that need to be called, reducing the implementation cost, and improving ease of use.

[0173] In some embodiments, the heterogeneous mixed mosaic graph of the embodiments of the present application includes a single-attribute heterogeneous mixed mosaic graph.

[0174] A single-attribute heterogeneous mixed mosaic image refers to a mosaic image in which all homogeneous strips have the same attribute information. For example, a single-attribute heterogeneous mixed mosaic image only includes homogeneous strips with attribute information, such as multi-view video texture strips and point cloud texture strips. Another example is a single-attribute heterogeneous mixed mosaic image only includes homogeneous strips with geometric information, such as multi-view video geometry strips and point cloud geometry strips.

[0175] The embodiment of the present application does not limit the expression format of the at least two isomorphic strips. For example, the at least two isomorphic strips include at least two of a multi-view video strip, a point cloud strip, and a grid strip.

[0176] In some embodiments, in the above S602, splicing at least two isomorphic strips to generate a heterogeneous mixed spliced ​​graph includes:

[0177] S602-A: Splice at least a single attribute isomorphic strip in a first expression format and a single attribute strip in a second expression format to obtain a heterogeneous mixed splicing graph.

[0178] The first expression format and the second expression format are any one of multi-view video, point cloud and mesh, and the first expression format and the second expression format are different.

[0179] The single attribute homogeneous strip of the multi-view video includes at least one of a multi-view video texture strip and a multi-view video geometry strip.

[0180] The single attribute isomorphic strip of the point cloud includes at least one of a point cloud texture strip, a point cloud geometry strip, and a point cloud occupancy status strip.

[0181] The single attribute isomorphic strips of the mesh include at least one of a mesh texture strip and a mesh geometry strip.

[0182] For example, at least two of the multi-view video geometric strips, the point cloud geometric strips, and the mesh geometric strips are stitched together into one image to obtain a heterogeneous hybrid stitched image. This heterogeneous hybrid stitched image is called a single-attribute heterogeneous hybrid stitched image.

[0183] For another example, at least two of the multi-view video texture strips, point cloud texture strips, and mesh texture strips are stitched together into one image to obtain a heterogeneous hybrid stitched image, which is called a single-attribute heterogeneous hybrid stitched image.

[0184] In some embodiments, the heterogeneous mixed mosaic of the embodiments of the present application includes a multi-attribute heterogeneous mixed mosaic, that is, a heterogeneous mixed mosaic in which at least two of the homogeneous strips included in the heterogeneous mixed mosaic have different attribute information. For example, a multi-attribute heterogeneous mixed mosaic includes both a homogeneous mosaic of attribute information and a homogeneous mosaic of geometric information. As an example, strips under any one attribute or any two attributes of at least two of the point cloud, multi-view video, and mesh can be spliced ​​into a single image to obtain a heterogeneous mixed mosaic. This application is not limited to this.

[0185] For example, a multi-view video texture strip is spliced ​​with at least one of a point cloud geometry strip and a mesh geometry strip into one image to obtain a heterogeneous hybrid spliced ​​image, which is called a multi-attribute heterogeneous hybrid spliced ​​image.

[0186] For another example, a multi-viewpoint video geometry strip, a point cloud texture strip, or a mesh texture strip are spliced ​​into one image to obtain a heterogeneous hybrid spliced ​​image. This heterogeneous hybrid spliced ​​image is called a multi-attribute heterogeneous hybrid spliced ​​image.

[0187] For another example, a point cloud texture strip is spliced ​​with at least one of a multi-view video geometry strip and a mesh geometry strip into one image to obtain a heterogeneous hybrid spliced ​​image. This heterogeneous hybrid spliced ​​image is called a multi-attribute heterogeneous hybrid spliced ​​image.

[0188] For another example, a point cloud geometry strip is spliced ​​with at least one of a multi-view video texture strip and a mesh texture strip into one image to obtain a heterogeneous hybrid spliced ​​image. This heterogeneous hybrid spliced ​​image is called a multi-attribute heterogeneous hybrid spliced ​​image.

[0189] The following describes S602-A in detail by taking the first expression format as multi-view video and the second expression format as point cloud as an example.

[0190] Assuming that the multi-view video strips include multi-view video texture strips and multi-view video geometry strips, and the point cloud mosaic includes point cloud texture strips, point cloud geometry strips, and point cloud occupancy strips, then the hybrid mosaic method of S602-A can include but is not limited to the following two:

[0191] Method 1: Multi-view video texture strips, multi-view video geometry strips, point cloud texture strips, point cloud geometry strips, and point cloud occupancy strips are all stitched together into a heterogeneous mixed stitching image.

[0192] Method 2: According to a preset hybrid stitching method, multi-view video texture strips, multi-view video geometry strips, point cloud texture strips, point cloud geometry strips and point cloud occupancy strips are stitched together to obtain M heterogeneous hybrid stitching images, where M is a positive integer greater than or equal to 1.

[0193] Method 2 may include at least the following examples:

[0194] In Example 1, multi-view video texture strips and point cloud texture strips are spliced ​​to obtain a heterogeneous mixed texture splicing image, multi-view video geometry strips and point cloud geometry strips are spliced ​​to obtain a heterogeneous mixed geometry splicing image, and the point cloud occupancy strips are used as a separate mixed splicing image.

[0195] For example, assuming at least two visual media contents include a multi-view video, point cloud 1, and point cloud 2, the multi-view video is processed to obtain multi-view video strips, where the multi-view video strips include multi-view video texture strips and multi-view video geometry strips. Point cloud 1 is processed to obtain point cloud texture strip 1, point cloud geometry strip 1, and point cloud 1 occupancy strips. Point cloud 2 is processed to obtain point cloud texture strip 2, point cloud geometry strip 2, and point cloud 2 occupancy strips. Optionally, the point cloud 1 occupancy strips and the point cloud 2 occupancy strips can be merged into a point cloud occupancy mosaic.

[0196] Next, the multi-view video texture strips, point cloud texture strips 1, and point cloud texture strips 2 are mixed and spliced ​​to obtain a heterogeneous mixed texture mosaic, as shown in Figure 9A. Furthermore, the multi-view video geometry strips, point cloud geometry strips 1, and point cloud geometry strips 2 are spliced ​​to obtain a heterogeneous mixed geometry mosaic, as shown in Figure 9B. The point cloud occupancy mosaic is treated as a separate mixed mosaic, as shown in Figure 9C.

[0197] Example 2: Multi-view video texture strips and point cloud texture strips are spliced ​​to obtain a heterogeneous mixed texture splicing map, and multi-view video geometry strips, point cloud geometry strips and point cloud occupancy strips are spliced ​​to obtain a heterogeneous mixed geometry and occupancy splicing map.

[0198] Example 3: Multi-view video texture strips, point cloud texture strips, and point cloud occupancy strips are spliced ​​to obtain a sub-heterogeneous mixed splicing image, and multi-view video geometry strips and point cloud geometry strips are spliced ​​to obtain another sub-heterogeneous mixed splicing image.

[0199] It should be noted that the above examples 1 to 3 are only a part of the mixed splicing methods, and the mixed splicing methods of the embodiments of the present application include but are not limited to the above examples 1 to 3.

[0200] In the second method above, after obtaining M heterogeneous mixed mosaic images, the M heterogeneous mixed mosaic images can be respectively subjected to video encoding to obtain video compression sub-streams.

[0201] For example, a video encoder is used to encode M heterogeneous mixed mosaics separately to obtain a compressed video substream. Optionally, each of the M heterogeneous mixed mosaics can be treated as a frame image and subjected to video encoding to obtain a compressed video substream. For example, a video encoder is used to encode the heterogeneous mixed texture mosaic shown in Figure 9A, the heterogeneous mixed geometric mosaic shown in Figure 9B, and the point cloud mixed occupancy mosaic shown in Figure 9C to obtain a compressed video substream.

[0202] In the embodiment of the present application, while generating M heterogeneous mixed mosaic graphs, mixed mosaic information corresponding to each of the M heterogeneous mixed mosaic graphs is generated. The mixed mosaic information of the M heterogeneous mixed mosaic graphs is encoded to obtain a mixed mosaic information substream of the M heterogeneous mixed mosaic graphs.

[0203] For example, the mixed splicing information corresponding to each heterogeneous mixed splicing graph in the M heterogeneous mixed splicing graphs is combined to form a complete mixed splicing information, and then the complete mixed splicing information is encoded to obtain a mixed splicing information sub-code stream.

[0204] For example, as shown in Figure 10, taking multi-view video and point cloud as examples, the multi-view video is processed, for example, using the TMIV packing technique, to obtain multi-view video texture strips and multi-view video geometry strips. The point cloud is processed, for example, using the TMC2 packing technique, to obtain point cloud texture strips, point cloud geometry strips, and point cloud occupancy strips. Next, a preset hybrid stitching method is used to stitch the multi-view video texture strips, multi-view video geometry strips, point cloud texture strips, point cloud geometry strips, and point cloud occupancy strips together to obtain M heterogeneous hybrid stitching images. For example, using the stripe packing technique, the multi-view video texture strips and point cloud texture strips are stitched together to obtain a heterogeneous hybrid texture stitching image; the multi-view video geometry strips and point cloud geometry strips are stitched together to obtain a heterogeneous hybrid geometry stitching image; and the point cloud occupancy strips are stripe-packed to obtain a point cloud hybrid occupancy stitching image. Then, a video encoder is used to encode the heterogeneous mixed texture mosaic, the heterogeneous mixed geometry mosaic, and the point cloud mixed occupancy mosaic to obtain a video compression substream. The mixed mosaic information is then encoded to obtain a mixed mosaic information substream. Finally, the video compression substream and the mixed mosaic information substream are written into the same compressed stream.

[0205] Since the original V3C standard only supports the combination of homogeneous texture, geometry, and occupancy mosaics into a mixed mosaic, that is, it only supports the packaging of multi-view video mosaics into a multi-view mixed mosaic, or the packaging of point cloud mosaics into a point cloud mixed mosaic, the original V3C standard does not support the packaging of multi-view video strips and point cloud strips into the same heterogeneous mixed mosaic. Therefore, it is necessary to add or modify some syntax elements in the V3C standard.

[0206] In some embodiments, the parameter set of the codestream may include first information, and the first information may be used to indicate whether the codestream simultaneously includes codestreams corresponding to at least two visual media contents in different expression formats.

[0207] The encoder can also write the parameter set of the codestream into the codestream. For example, the encoder writes the video compression sub-codestream, the mixed splicing information sub-codestream, and the parameter set into the same compressed codestream.

[0208] Exemplarily, the parameter set of the code stream may be V3C_VPS, and ptl_profile_toolset_idc in the V3C_VPS may be used to represent the first information.

[0209] In some embodiments, the first information is set to different values ​​to indicate whether the bitstream simultaneously contains two types of bitstreams corresponding to at least two visual media content in different expression formats, or to indicate which two (or which) types of visual media content in different expression formats are simultaneously contained in the bitstream. In other words, certain preset values ​​of the first information can indicate that the bitstream contains at least two types of visual media content in different expression formats, or indicate which two types of visual media content in different expression formats are contained in the bitstream.

[0210] For example, when the first information is set to a first preset value, the first preset value is used to indicate that the codestream contains both a multi-view video codestream and a point cloud codestream. As a specific example, when ptl_profile_toolset_idc = X, and X is 128 / 129 / 130, it means that the current codestream contains both VPCC extend and MIV main codestreams. In this case, the MIV sub-tiles and VPCC sub-tiles can be packed as different tiles in a heterogeneous atlas.

[0211] For another example, when the first information is set to the second preset value, the second preset value is used to indicate that the code stream only includes the code stream of the multi-view video. As a specific example, when ptl_profile_toolset_idc=X, X is 64 / 65 / 66, which means that the current code stream only includes the MIV main code stream.

[0212] For another example, when the first information is set to the third preset value, the third preset value is used to indicate that the code stream only contains the point cloud video code stream. As a specific example, when ptl_profile_toolset_idc=X, X is 0 / 1, which means that the current code stream only contains the VPCC code stream.

[0213] It should be understood that the values ​​of the above first preset value, second preset value, and third preset value are only examples, and the embodiments of the present application are not limited thereto.

[0214] In one example, the profile, tier, and level syntax including the first information may be as shown in Table 3. It should be noted that in this example, V3C_VPS in the existing V3C standard may be reused, and preset values ​​such as 128 / 129 / 130 / 131 / 132 / 133 are preconfigured for ptl_profile_toolset_idc to indicate that the current codestream includes both VPCC (such as VPCC basic or VPCC extended) and MIV (such as MIV main or MIV Extended or MIV Geometry Absent).

[0215] Table 3: Profile, tier, and level syntax

[0216]

[0217] Table 4 shows an example of available toolset profile components. Table 4 provides a list of toolset profile components defined for V3C and their corresponding identification syntax element values, such as ptl_profile_toolset_idc and ptc_one_v3c_frame_only_flag, which can be used only for this document. The syntax element ptl_profile_toolset_idc provides the main definition of the toolset profile, and additional syntax elements such as ptc_one_v3c_frame_only_flag can specify additional features or restrictions of the defined profile. ptc_one_v3c_frame_only_flag can be used to support only a single V3C frame. The reserved value of ptl_profile_toolset_idc can be used for future ISO / IEC use and should not appear in V3C bitstreams conforming to this document version. The profile types defined in Table 4 can include dynamic (Dynamic) or static (Static).

[0218] Table 4

[0219]

[0220]

[0221] In particular, compared with the existing standard, in the embodiment of the present application, the added ptl_profile_toolset_idc=128 is applicable to the standard protocols such as V-PCC Extended and MIV Main, that is, the codestream includes two types of codestreams: V-PCC Extended and MIV Main; the applicable standard protocols when ptl_profile_toolset_idc=129 are V-PCC Extended and MIV Extended, that is, the codestream includes two types of codestreams: V-PCC Extended and MIV Extended; and the applicable standard protocols when ptl_profile_toolset_idc=130 are V-PCC Extended and MIV Geometry Absent, that is, the codestream includes two types of codestreams: V-PCC Extended and MIV Geometry Absent. The added ptl_profile_toolset_idc=131 applies to the standard protocols V-PCC Basic and MIV Main, i.e., the codestream includes two types of codestreams: V-PCC Basic and MIV Main. The applicable standard protocols when ptl_profile_toolset_idc=132 are V-PCC Basic and MIV Extended, i.e., the codestream includes two types of codestreams: V-PCC Basic and MIV Extended. The applicable standard protocols when ptl_profile_toolset_idc=133 are V-PCC Basic and MIV Geometry Absent, i.e., the codestream includes two types of codestreams: V-PCC Basic and MIV Geometry Absent.

[0222] It should be noted that the values ​​in Table 4 above are only examples, and the embodiment of the present application does not limit the specific value of ptl_profile_toolset_idc.

[0223] Therefore, when encoding a heterogeneous mixed mosaic image, the embodiment of the present application indicates that the code stream corresponds to at least two visual media contents in different expression formats simultaneously included in the code stream by adding the value of the first information in the parameter set. This can help improve the decoding accuracy of the decoding end, and at the same time enable the V3C standard to support visual media contents in different expression formats such as multi-viewpoint video, point cloud, and mesh in the same compressed code stream.

[0224] In some embodiments, the hybrid stitching information of the heterogeneous hybrid stitching graph may include a first flag and a second flag, wherein the first flag can be used to indicate whether the code stream contains multi-view video syntax elements, and the second flag can be used to indicate whether the code stream contains point cloud syntax elements.

[0225] Optionally, asps_miv_extension_present_flag may be used to represent the first flag, and asps_vpcc_extension_present_flag may be used to represent the second flag.

[0226] In some embodiments, the first flag and the second flag are located in an atlas sequence parameter set, such as a Network Abstraction Layer-Atlas Sequence Parameter Set (NAL-ASPS).

[0227] For example, the NAL-ASPS of the V3C_AD code stream may include asps_miv_extension_present_flag and asps_vpcc_extension_present_flag.

[0228] In this embodiment of the present application, a first flag is set to a specific value to indicate whether a codestream contains multi-view video syntax elements, and a second flag is set to a specific value to indicate whether a codestream contains point cloud syntax elements. For example, a specific value of asps_miv_extension_present_flag can indicate whether the atlas codestream corresponding to the atlas data contains multi-view video syntax elements, and a specific value of asps_vpcc_extension_present_flag can indicate whether the atlas codestream corresponding to the atlas data contains multi-view video syntax elements.

[0229] Exemplarily, the first flag is set to a first value, which can be used to indicate that the code stream contains multi-view video syntax elements, that is, the multi-view video codec standard can be used for video encoding and decoding; and the second flag is set to a second value, which can be used to indicate that the code stream contains point cloud syntax elements, that is, the point cloud codec standard can be used for video encoding and decoding.

[0230] The embodiments of the present application do not limit the specific values ​​of the first value and the second value.

[0231] Optionally, the first value is 1.

[0232] Optionally, the second value is 1.

[0233] That is, when the first flag is 1, it can indicate that the code stream contains multi-view video syntax elements, otherwise (for example, the first flag is 0) the code stream does not contain multi-view video syntax elements; when the second flag is 1, it can indicate that the code stream contains point cloud syntax elements, otherwise (for example, the first flag is 0) the code stream does not contain point cloud syntax elements.

[0234] In this embodiment of the present application, the number of heterogeneous hybrid mosaic graphs may be one or more. When the number of heterogeneous hybrid mosaic graphs is multiple, the V3C codestream may include multiple atlas codestreams corresponding to multiple atlas data. The NAL_ASPS corresponding to each atlas codestream may include the first flag and / or the second flag.

[0235] Example 1: After determining that ptl_profile_toolset_idc is 128 / 129 / 130 / 131 / 132 / 133, the NAL_ASPS corresponding to the first atlas stream in multiple atlas streams in the V3C_AD stream may include asps_vpcc_extension_present_flag as a first value, and the NAL_ASPS corresponding to the second atlas stream may include asps_miv_extension_present_flag as a second value.

[0236] For example, the NAL_ASPS corresponding to Altas 1 includes asps_vpcc_extension_present_flag = 1, and the NAL_ASPS corresponding to Altas 1 includes asps_miv_extension_present_flag = 1. Optionally, the value of asps_miv_extension_present_flag in the NAL_ASPS corresponding to Altas 1 is not limited, for example, it can be 0 or 1. Optionally, the value of asps_vpcc_extension_present_flag in the NAL_ASPS corresponding to Altas 2 is not limited, for example, it can be 0 or 1.

[0237] In other words, Example 1 does not limit the multi-view video syntax elements and point cloud syntax elements to the NAL_ASPS corresponding to a single atlas bitstream. That is, in the entire V3C bitstream, some of the multiple atlas data may include multi-view video syntax elements but not point cloud syntax elements, while others may include point cloud syntax elements but not multi-view syntax elements.

[0238] Example 2: After determining that ptl_profile_toolset_idc is 128 / 129 / 130 / 131 / 132 / 133, the NAL_ASPS of a certain atlas code stream in the V3C_AD code stream may include asps_vpcc_extension_present_flag=1 and asps_miv_extension_present_flag=1.

[0239] For example, the NAL_ASPS corresponding to Altas 3 includes sps_vpcc_extension_present_flag=1 and asps_miv_extension_present_flag=1.

[0240] Table 5 shows an example of allowable values ​​of syntax element values ​​for the MIV toolset profile components, which may be used by ISO / IEC 23090-12.

[0241] Table 5

[0242]

[0243]

[0244] Among them, the last eight columns in Table 5 are the profiles of eight possible supported heterogeneous mixed splicing graphs compared to the existing standard protocols, as well as the possible values ​​of each syntax element. The ptl_profile_toolset_idc values ​​of the eight newly added profiles are 128 / 129 / 130 / 131 / / 132 / 133. The values ​​of asps_vpcc_extension_present_flag and aaps_vpcc_extension_present_flag in the eight newly added profile tables can be yes or no (i.e. 1 or 0), indicating that multi-view video streams and point cloud streams are allowed to coexist in the stream. The eight newly added profiles are expanded on the basis of the original four profiles. The key is to modify the values ​​of asps_vpcc_extension_present_flag and aaps_vpcc_extension_present_flag to allow them to be yes or no. For example, the MIV Main Mixed V-PCC Basic profile and the MIV Main Mixed V-PCC Extended profile are extensions of the MIV Main profile. Except for ptl_profile_toolset_idc, asps_vpcc_extension_present_flag, and aaps_vpcc_extension_present_flag, the remaining syntax elements in the two newly added tables are the same as those in the MIV Main profile. The other six newly added profiles can also be given by referring to the above example.

[0245] It should be noted that the embodiments of the present application only use Table 5 as an example to illustrate the available values ​​of the syntax element values ​​of the MIV toolset configuration file component. It should be understood that the available values ​​of each syntax element in Table 5 can also be other values ​​in addition to Table 5. For example, the standards organization can modify the standard, such as modifying vps_miv_extension_present_flag and vps_packing_information_present_flag (or other syntax elements) to other values ​​in addition to the values ​​shown in Table 5, but these still fall within the scope of protection of the embodiments of the present application.

[0246] Table 6 shows an example of the maximum allowed syntax element values ​​for the V-PCC toolset profile components (taking the support for mixing VPCC code streams and MIV Main code streams in V3C code streams as an example), which can be used for ISO / IEC23090-5-Annex H.

[0247] Table 6

[0248]

[0249]

[0250] Table 6 takes VPCC mixed MIV main as an example, where the ptl_profile_toolset_idc value corresponding to V-PCC Basic Mixed MIV Main and V-PCC Basic Still Mixed MIV Main is 131; the ptl_profile_toolset_idc value corresponding to V-PCC Extended Mixed MIV Main and V-PCC Extended Still Mixed MIV Main is 128. The four syntax elements vps_miv_extension_present_flag, asps_miv_extension_present_flag, casps_miv_extension_present_flag, and caf_miv_extension_present_flag in the last four columns of Table 6 can have values ​​of yes or no (i.e., 1 or 0). Except for these four syntax elements, the syntax elements represented by each row in the last four columns of Table 6 are the same as those represented by each row in the first four columns. The profile that supports the mixing of VPCC code stream and MIV Extended code stream, or the mixing of VPCC code stream and MIV Extended Restricted Geometry code stream, or the mixing of VPCC code stream and MIV Geometry Absent code stream can also be given according to the example in Table 6.

[0251] Therefore, when encoding a heterogeneous mixed splicing graph, the embodiment of the present application adds a first flag and a second flag to the mixed splicing information to indicate whether the code stream contains multi-view video syntax elements and point cloud syntax elements. This can help improve the decoding accuracy of the decoding end and enable the V3C standard to support visual media content in different expression formats such as multi-view video and point cloud in the same compressed code stream.

[0252] In some embodiments, the hybrid splicing information further includes a third flag, which is used to indicate the expression format type of the i-th strip in the heterogeneous hybrid splicing graph, where i is a positive integer.

[0253] Optionally, atdu_type_flag can be used to represent the third flag. For example, a flag named atdu_type_flag[tileID] can be added to each atlas_tile_data_unit(tileID) to indicate whether the tile with the ID is MIV tile or VPCC tile.

[0254] Specifically, the expression format type corresponding to the i-th strip in the heterogeneous mixed mosaic can be indicated by setting different values ​​for the third flag. Taking the example of at least two homogeneous strips including a multi-view video strip and a point cloud strip, if the i-th strip is a multi-view video strip, the value of the third flag is set to the third value; if the i-th strip is a point cloud strip, the value of the third flag is set to the fourth value.

[0255] The embodiment of the present application does not limit the specific values ​​of the third value and the fourth value.

[0256] Optionally, the third value is 0.

[0257] Optionally, the fourth value is 1.

[0258] Optionally, the third flag may be located in a slice data unit (atlas_tile_data_unit).

[0259] In one example, the hybrid splicing information can be written into the ACL NAL unit type code stream, where the atlas_tile_data_unit contains the atdu_type_flag. If the current slice is a point cloud slice, the atdu_type_flag is 1; if the current slice is a multi-view video slice, the atdu_type_flag is 0. The general atlas tile data unit syntax after adding the atdu_type_flag is shown in Table 7.

[0260] Table 7

[0261]

[0262] Among them, when asps_miv_extension_present_flag and asps_vpcc_extension_present_flag are both yes (i.e. 1), atdu_type_flag can be used to further indicate the expression format type of the current strip. For example, if atdu_type_flag is yes (i.e. 1), it means that the current strip is a point cloud strip, and if atdu_type_flag is no (i.e. 0), it means that the current strip is a multi-view video strip.

[0263] Therefore, when encoding a heterogeneous mixed splicing image, the embodiment of the present application adds a third flag in the mixed splicing information to indicate the expression format type corresponding to the i-th strip in the heterogeneous mixed splicing image, which can help improve the decoding accuracy of the decoding end and enable the V3C standard to support visual media content in different expression formats such as multi-view video and point cloud in the same compressed code stream.

[0264] In some embodiments, the mixed splicing information further includes sub-image block information of the i-th slice, where the sub-image block information is used to indicate encoding and decoding information of the sub-image block in the i-th slice.

[0265] Optionally, the sub-tile information may be located in a sub-tile data unit (patch_data_unit).

[0266] Taking at least two isomorphic strips including a multi-view video strip and a point cloud strip as an example, if the i-th strip is a multi-view video strip, the sub-block information can be used to indicate whether the current sub-block is video encoded using the multi-view video coding standard; if the i-th strip is the point cloud strip, the sub-block information is used to indicate whether the current sub-block is video encoded using the point cloud video coding standard.

[0267] In one example, when atdu_type_flag is 0 and asps_miv_extension_present_flag is 1, the sub-block information of the current sub-block may indicate that the multi-view video coding standard is used for video coding; when atdu_type_flag is 1, the sub-block information of the current sub-block may indicate that the point cloud video coding standard is used for video coding. The sub-block data unit syntax (Patch data unit syntax) may be as shown in Table 8:

[0268] Table 8

[0269]

[0270]

[0271] Among them, when ptl_profile_toolset_idc is 128 / 129 / 130, asps_miv_extension_present_flag is 1, then atdu_type_flag[tileID] is 0, indicating that the current strip is a multi-view video strip. At this time, asps_miv_extension_present_flag&&atdu_type_flag[tileID]==0 holds, indicating that the current sub-tile (tileID, patchIdx) of the current strip is video encoded using the ISO / IEC 23090-12 standard.

[0272] Figure 11 shows a schematic diagram of the V3C bitstream structure provided by an embodiment of the present application, wherein the V3C parameter set (V3C_parameter_set()) of the V3C_VPS may include ptl_profile_toolset_idc, and ptl_profile_toolset_idc being 128 / 129 / 130 indicates that the current bitstream contains both VPCC extend and MIV main bitstreams.

[0273] The Atlas sequence parameter set () (atlas_sequence_parameter_set_rbsp()) in the NAL_ASPS in the atlas sub-bitstream () (atlas_sub_bitstream()) of V3C_AD can include asps_vpcc_extension_present_flag and asps_miv_extension_present_flag. When ptl_profile_toolset_idc is 128 / 129 / 130, asps_vpcc_extension_present_flag is yes (i.e., 1) and asps_miv_extension_present_flag is yes (i.e., 1).

[0274] The ACL NAL unit type (ACL_NAL_unit_type) in the V3C_AD atlas_sub_bitstream() includes hybrid splicing information. For example, the atlas tile data unit (atlas_tile_data_unit()) may include atdu_type_flag. If atdu_type_flag is set to 1, it indicates that the current tile is a point cloud tile. If atdu_type_flag is set to 0, it indicates that the current tile is a multi-view video tile.

[0275] Furthermore, the sub-tile information data (patch_information_data) includes a sub-tile data unit (patch_data_unit). If atdu_type_flag is false and asps_miv_extension_present_flag is true, it indicates that the current sub-tile is implemented using the multi-view video coding standard. If atdu_type_flag is true, it indicates that the current sub-tile is implemented using the point cloud video decoding standard.

[0276] To further illustrate the method of the embodiment of the present application, the following takes visual media content including multi-view video and point cloud as an example, combined with the above-mentioned newly added or modified syntax elements, to introduce the encoding method of the embodiment of the present application. As shown in Figure 12, the encoding method 700 of the embodiment of the present application may include:

[0277] S701, through inter-viewpoint projection, multi-viewpoint video is projected to remove duplication and redundancy, and non-repeated pixels are connected into sub-blocks, which are then spliced ​​into multi-viewpoint video strips; point cloud is projected through parallel projection, and connected pixels in the projection plane are formed into sub-blocks, which are then spliced ​​into point cloud strips.

[0278] S702: Splice the multi-view video strips and the point cloud strips to generate a heterogeneous mixed spliced ​​image.

[0279] At this time, asps_vpcc_extension_present_flag may be set to 1 and asps_miv_extension_present_flag may be set to 1 in the mixed splicing information.

[0280] Furthermore, if the current slice added to the heterogeneous mixed splicing image is a multi-view video slice, atdu_type_flag is set to 0 in the mixed splicing information.

[0281] If the current strip added to the heterogeneous mixed mosaic image is a point cloud strip, atdu_type_flag is set to 1 in the mixed mosaic information.

[0282] S703 , performing video encoding on the heterogeneous mixed mosaic image to obtain a video compression sub-stream.

[0283] Furthermore, under the premise that atdu_type_flag is 0 and asps_miv_extension_present_flag is 1, it is determined that the current sub-block is implemented using the multi-view video coding standard.

[0284] Under the premise that atdu_type_flag is 1 and asps_miv_extension_present_flag is 1, it is determined that the current sub-block adopts the point cloud video coding standard.

[0285] S704 , encoding hybrid splicing information of a heterogeneous hybrid splicing image obtained by splicing the multi-view video strips and the point cloud strips to form a hybrid splicing information sub-stream;

[0286] S705 , the video compression code stream and the mixed splicing information code stream are written into the compression code stream.

[0287] Furthermore, V3C_VPS can be written into the compressed bitstream, with ptl_profile_toolset_idc set to 128 / 129 / 130 in the V3C_VPS parameter set, indicating that the current bitstream contains both the bitstream corresponding to the point cloud and the bitstream corresponding to the multi-view video. The specific value of ptl_profile_toolset_idc may be related to the standard protocol applicable to the multi-view video or point cloud.

[0288] Therefore, the encoder adds atdu_type_flag to the hybrid splicing information to indicate the expression format type of the i-th strip in the heterogeneous hybrid splicing image, increases the value of ptl_profile_toolset_idc to indicate that the bitstream contains both point cloud and multi-view video corresponding to two types of streams, and increases the values ​​of asps_vpcc_extension_present_flag and asps_miv_extension_present_flag to support the bitstream containing both point cloud and multi-view video corresponding to two types of streams. In this way, when decoding, the decoder can determine that the current bitstream includes both point cloud and multi-view video corresponding to two types of streams based on ptl_profile_toolset_idc, determine the expression format type of the current strip based on atdu_type_flag, and further determine the codec standard used by the current sub-block based on atdu_type_flag and asps_miv_extension_present_flag, thereby enabling the decoder to accurately perform decoding.

[0289] The encoding method of the embodiment of the present application processes at least two visual media contents separately to obtain at least two isomorphic strips, wherein at least two of the at least two visual media contents have different expression formats; splices the at least two isomorphic strips to generate a heterogeneous mixed splicing graph; and encodes the heterogeneous mixed splicing graph to obtain a code stream. That is, the present application splices strips corresponding to visual media contents of multiple different expression formats into a heterogeneous mixed splicing graph, so that the visual media contents of different expression formats (i.e., heterogeneous contents) are orderly grouped and formed in the same heterogeneous mixed splicing graph. For example, multi-view video strips and point cloud strips are spliced ​​into a heterogeneous mixed splicing graph for encoding and decoding. This minimizes the number of two-dimensional video encoders such as HEVC, VVC, AVC, AVS, etc. that need to be called, reduces the encoding cost, and improves ease of use.

[0290] The above describes the encoding method of the present application using the encoding end as an example. The following describes the video decoding method provided in the embodiment of the present application using the decoding end as an example.

[0291] FIG13 is a schematic flow chart of a decoding method 800 provided in an embodiment of the present application. As shown in FIG13 , the decoding method in the embodiment of the present application includes:

[0292] S801, decoding the bitstream to obtain a reconstructed heterogeneous hybrid splicing image;

[0293] S802, splitting the reconstructed heterogeneous mixed mosaic graph to obtain at least two reconstructed isomorphic strips; wherein the reconstructed isomorphic strips correspond to at least two different expression formats;

[0294] S803 : Obtain at least two reconstructed visual media contents according to the at least two reconstructed isomorphic strips, wherein the reconstructed visual media contents correspond to at least two different expression formats.

[0295] As can be seen above, during encoding, homogeneous strips with different representation formats are spliced ​​together into a heterogeneous hybrid mosaic for encoding. Therefore, after obtaining the bitstream, the decoder decodes the bitstream to obtain a reconstructed heterogeneous hybrid mosaic. This reconstructed heterogeneous hybrid mosaic is then split into at least two reconstructed homogeneous strips, each corresponding to at least two different representation formats. Finally, the decoder reconstructs the at least two reconstructed homogeneous strips to obtain at least two reconstructed visual media contents.

[0296] In an embodiment of the present application, at least two isomorphic strips of different expression formats are spliced ​​into a heterogeneous mixed splicing graph, so that the visual media content of different expression formats (i.e., heterogeneous content) are orderly grouped and formed in the same heterogeneous mixed splicing graph. In this way, during decoding, the number of two-dimensional video decoders such as HEVC, VVC, AVC, AVS, etc. that need to be called can be minimized, thereby reducing the decoding cost and improving ease of use.

[0297] In some embodiments, the code stream includes a video compression sub-code stream. In this case, S801 includes:

[0298] The video decoder is called to decode the video compression sub-stream to obtain a reconstructed heterogeneous mixed splicing image.

[0299] It should be understood that the bitstream in the embodiments of the present application may include other content in addition to the video compression substream. After obtaining the bitstream, the decoder parses the bitstream to obtain the video compression substream. Furthermore, the decoder can decode the video compression substream to reconstruct the heterogeneous hybrid mosaic. For example, the video compression substream can be input into the video decoder shown in FIG2B for decoding to obtain the reconstructed heterogeneous hybrid mosaic.

[0300] To improve decoding accuracy, the encoder can encode the mixed splicing information to obtain a mixed splicing information substream and write the mixed splicing information substream into the bitstream. That is, the bitstream in this embodiment of the present application may include a mixed splicing information substream in addition to the video compression substream. In this case, the decoding method may further include decoding the mixed splicing information substream to obtain the mixed splicing information.

[0301] Accordingly, the above step 802 may be implemented as follows: splitting the reconstructed heterogeneous mixed splicing image according to the mixed splicing information to obtain at least two reconstructed homogeneous strips.

[0302] Specifically, in this embodiment of the present application, the decoding end parses the bitstream to obtain a video compression substream and a hybrid splicing information substream. Next, the decoding end decodes the video compression substream to obtain a reconstructed heterogeneous hybrid splicing graph, and decodes the hybrid splicing information substream to obtain hybrid splicing information. Finally, the hybrid splicing information is used to split the reconstructed heterogeneous hybrid splicing graph into at least two reconstructed homogeneous strips.

[0303] In some embodiments, the at least two homogeneous strips belong to at least two different types of strips among multi-view video strips, point cloud strips, and mesh strips.

[0304] In some embodiments, the heterogeneous mixed mosaic graph includes at least one of a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph.

[0305] Based on this, the above S802, i.e., splitting the reconstructed heterogeneous mixed mosaic image to obtain at least two reconstructed homogeneous strips, can be implemented as follows:

[0306] The reconstructed heterogeneous mixed mosaic graph is split to obtain at least a single attribute reconstructed isomorphic strip in a first expression format and a single attribute reconstructed isomorphic strip in a second expression format.

[0307] The first expression format and the second expression format are both any one of multi-view video, point cloud and mesh, and the first expression format and the second expression format are different.

[0308] In some embodiments, if the first expression format is a multi-view video and the second expression format is a point cloud, the above S802 may include the following examples:

[0309] Example 1: If the reconstructed heterogeneous mixed mosaic image is a reconstructed heterogeneous mixed texture mosaic image, the reconstructed heterogeneous mixed texture mosaic image is split to obtain multi-view video texture reconstruction strips and point cloud texture reconstruction strips;

[0310] Example 2: If the reconstructed heterogeneous mixed mosaic image is a reconstructed heterogeneous mixed geometric mosaic image, the reconstructed heterogeneous mixed geometric mosaic image is split to obtain multi-view video geometric reconstruction strips and point cloud geometric reconstruction strips.

[0311] Optionally, reconstructing the heterogeneous mixed mosaic map may further include reconstructing a point cloud mixed occupancy situation mosaic map, and the reconstructed point cloud mixed occupancy situation mosaic map may be split to obtain point cloud occupancy situation reconstruction strips.

[0312] In some embodiments, a multi-view video reconstruction strip may be obtained according to the multi-view video texture reconstruction strip and the multi-view video geometry reconstruction strip; and

[0313] According to the point cloud texture reconstruction strips, point cloud geometry reconstruction strips and reconstructed point cloud occupancy strips, a reconstructed point cloud splicing image is obtained.

[0314] For example, as shown in Figure 14, the decoding end inputs the bitstream into the video decoder. The video decoder decodes the compressed video sub-stream to obtain a reconstructed heterogeneous mixed texture mosaic, a reconstructed heterogeneous mixed geometric mosaic, and a reconstructed point cloud mixed occupancy mosaic. The mixed mosaic information sub-stream is decoded to obtain mixed mosaic information. Next, the reconstructed heterogeneous mixed texture mosaic is split based on the mixed mosaic information, for example, using a stripe unpacking technique to split the reconstructed heterogeneous mixed texture mosaic to obtain multi-view video texture reconstruction strips and point cloud texture reconstruction strips. The reconstructed heterogeneous mixed geometric mosaic is split based on the mixed mosaic information, for example, using a stripe unpacking technique to split the reconstructed heterogeneous mixed geometric mosaic to obtain multi-view video geometry reconstruction strips and point cloud geometry reconstruction strips. The reconstructed point cloud mixed occupancy mosaic is split based on the mixed mosaic information, for example, using a stripe unpacking technique to split the reconstructed point cloud mixed occupancy mosaic to obtain point cloud occupancy reconstruction strips. Then, a reconstructed multi-view video is obtained based on the multi-view video texture reconstruction strips and the multi-view video geometry reconstruction strips. For example, the multi-view video texture reconstruction strips and the multi-view video geometry reconstruction strips are processed using the TMIV unpacking technology to obtain the reconstructed multi-view video. A reconstructed point cloud is obtained based on the point cloud texture reconstruction strips, the point cloud geometry reconstruction strips, and the point cloud occupancy reconstruction strips. For example, the point cloud texture reconstruction strips, the point cloud geometry reconstruction strips, and the point cloud occupancy reconstruction strips are processed using the TMC2 unpacking technology to obtain a reconstructed point cloud mosaic.

[0315] Since the expression formats of at least two isomorphic strips in the heterogeneous mixed splicing graph of the embodiment of the present application are different, in order to improve decoding accuracy, the encoding end of the embodiment of the present application includes first information in the parameter set of the bitstream, and the first information is used to indicate that the bitstream simultaneously contains bitstreams corresponding to at least two visual media contents with different expression formats.

[0316] Therefore, after obtaining the bitstream, the decoding end decodes the bitstream to obtain a parameter set of the bitstream, which includes first information for indicating whether the bitstream simultaneously contains bitstreams corresponding to at least two visual media contents in different expression formats.

[0317] In some embodiments, if the first information is a first preset value, it is determined that the code stream includes both the code stream of the multi-view video and the code stream of the point cloud.

[0318] In some embodiments, if the first information is a second preset value, it is determined that the code stream includes a code stream of a multi-view video.

[0319] In some embodiments, if the first information is a third preset value, it is determined that the code stream includes a point cloud code stream.

[0320] Furthermore, to improve decoding accuracy, the hybrid splicing information in the embodiment of the present application includes a first flag and a second flag. After obtaining the hybrid splicing information, the decoder can further parse the hybrid splicing information to obtain the first flag and the second flag. If the first flag is a first value, it is determined that the bitstream contains multi-view video syntax elements; if the second flag is a second value, it is determined that the bitstream contains point cloud syntax elements.

[0321] Optionally, the first value is 1 and the second value is 1.

[0322] Optionally, the first flag and the second flag are located in an atlas sequence parameter set.

[0323] Furthermore, in order to improve decoding accuracy, the hybrid splicing information of the embodiment of the present application further includes a third flag, which is used to indicate the expression format type of the i-th stripe in the reconstructed heterogeneous hybrid splicing graph, where i is a positive integer.

[0324] At this time, the above step 802, that is, splitting the reconstructed heterogeneous mixed mosaic image to obtain at least two reconstructed homogeneous strips, includes the following steps S802-A and S802-B:

[0325] S802-A, for the i-th strip in the reconstructed heterogeneous mixed splicing image, obtain a third flag corresponding to the i-th strip from the mixed splicing information;

[0326] S802-B extracts the i-th stripe into a reconstructed isomorphic stripe of the visual media expression format type corresponding to the i-th stripe according to the third flag corresponding to the i-th stripe.

[0327] Taking the at least two reconstructed isomorphic strips including a reconstructed multi-view video strip and a reconstructed point cloud strip as an example, the above S802-B includes the following steps:

[0328] S802-B1: If the value of the third flag is a third value, determine that the visual media expression format type corresponding to the i-th stripe is a multi-view video;

[0329] S802-B2: If the value of the third flag is the fourth value, the visual media expression format type corresponding to the i-th strip is set to point cloud.

[0330] Optionally, the third value is 0 and the fourth value is 1.

[0331] Optionally, the third flag is located in the stripe data unit.

[0332] Furthermore, in order to improve decoding accuracy, the hybrid splicing information in the embodiment of the present application further includes sub-block information of the i-th slice, where the sub-block information is used to indicate encoding and decoding information of the sub-block in the i-th slice.

[0333] Taking the at least two reconstructed isomorphic strips including a reconstructed multi-view video strip and a reconstructed point cloud strip as an example, the above S803 includes the following steps S803A and S803B:

[0334] S803A: If the i-th slice is the multi-view video slice, perform multi-view video decoding on the current sub-block of the i-th slice to obtain a reconstructed multi-view video;

[0335] S803B: If the i-th strip is the point cloud strip, perform point cloud decoding on the current sub-block of the i-th strip to obtain a reconstructed point cloud.

[0336] Optionally, the sub-tile information is located in a sub-tile data unit.

[0337] The decoding method provided by the embodiment of the present application is further described below through a specific embodiment. As shown in FIG15 , the encoding method 900 of the embodiment of the present application may include:

[0338] S901: Extract the mixed splicing information sub-stream and the video compression sub-stream from the compressed stream.

[0339] In some embodiments, V3C_VPS can be parsed from the compressed bitstream, and ptl_profile_toolset_idc can be parsed from the V3C_VPS parameter set. If ptl_profile_toolset_idc is 128 / 129 / 130 / 131 / 132 / 133, it indicates that the current bitstream contains both bitstreams corresponding to point clouds and bitstreams corresponding to multi-view video. The specific value of ptl_profile_toolset_idc may be related to the standard protocol applicable to multi-view video or point clouds.

[0340] S902: Decode the mixed splicing information sub-stream to obtain mixed splicing information.

[0341] After obtaining the ptl_profile_toolset_idc value of 128 / 129 / 130 / 131 / 132 / 133, you can further parse NAL_ASPS from the V3C_AD code stream to obtain asps_vpcc_extension_present_flag and asps_miv_extension_present_flag.

[0342] S903: Input the compressed video sub-stream into the video decoder, and output the reconstructed heterogeneous hybrid splicing image after decoding.

[0343] S904 , based on the hybrid splicing information, the heterogeneous hybrid splicing image is reconstructed, and reconstructed multi-view video strips and reconstructed point cloud strips are split and output.

[0344] Furthermore, after parsing the hybrid splicing information from the ACL NAL unit type, the atlas_tile_data_unit is parsed, and the ptl_profile_toolset_idc is determined to be 128 / 129 / 130 / 131 / 132 / 133, which yields the atdu_type_flag. If the atdu_type_flag is 1, the current strip is a point cloud strip; if the atdu_type_flag is 0, the current strip is a multi-view video strip.

[0345] S905 , reconstructing the multi-view video strip to generate a reconstructed multi-view video by multi-view video decoding, and reconstructing the point cloud strip to generate a reconstructed point cloud by point cloud decoding.

[0346] Furthermore, by parsing patch_data_unit, if atdu_type_flag is false and asps_miv_extension_present_flag is true, it can be determined that the current sub-block adopts the multi-view video decoding standard. If atdu_type_flag is true and asps_miv_extension_present_flag is true, it can be determined that the current sub-block adopts the point cloud video decoding standard.

[0347] Therefore, the decoding end determines the expression format type of the i-th strip in the heterogeneous mixed splicing image by parsing the atdu_type_flag in the mixed splicing information, as well as the values ​​of asps_vpcc_extension_present_flag and asps_miv_extension_present_flag, and parses the value of ptl_profile_toolset_idc to determine that the code stream contains two types of code streams corresponding to point cloud and multi-view video. This enables the decoding end to determine that the current code stream includes two types of code streams corresponding to point cloud and multi-view video according to ptl_profile_toolset_idc, determine the expression format type of the current strip according to atdu_type_flag, and further determine the video codec standard adopted by the current sub-block according to atdu_type_flag and asps_miv_extension_present_flag, thereby achieving accurate decoding.

[0348] The decoding method of the embodiment of the present application is as follows: the decoding end obtains a reconstructed heterogeneous mixed mosaic by decoding the code stream; the reconstructed heterogeneous mixed mosaic is split to obtain at least two reconstructed isomorphic strips; the at least two reconstructed isomorphic strips are decoded respectively to obtain at least two reconstructed visual media contents, the visual media contents correspond to at least two different expression formats, and the isomorphic strips correspond to at least two different expression formats. That is, the embodiment of the present application splices at least two isomorphic strips of different expression formats into a heterogeneous mixed mosaic, so that the visual media contents (i.e., heterogeneous contents) of different expression formats are orderly grouped and formed in the same heterogeneous mixed mosaic. In this way, when decoding, the rendering advantages of data (point clouds, etc.) from different expression formats are retained, and the synthesis quality of the image is improved. At the same time, the number of two-dimensional video decoders such as HEVC, VVC, AVC, AVS, etc. that need to be called can be minimized, thereby reducing the decoding cost and improving ease of use.

[0349] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0350] It should also be understood that in the various method embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships can exist. Specifically, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the related objects before and after are in an "or" relationship.

[0351] The above describes in detail the method embodiment of the present application in conjunction with Figures 6 to 15, and the following describes in detail the device embodiment of the present application in conjunction with Figures 16 to 18.

[0352] FIG16 is a schematic block diagram of an encoding device 10 provided in an embodiment of the present application, where the encoding device 10 is applied to the above-mentioned video decoding end.

[0353] As shown in FIG16 , the encoding device 10 includes:

[0354] A first splicing unit 11 is configured to process at least two visual media contents respectively to obtain at least two isomorphic strips, wherein the visual media contents correspond to at least two different expression formats, and the isomorphic strips correspond to at least two different expression formats;

[0355] A second splicing unit 12 is configured to splice the at least two isomorphic strips to generate a heterogeneous mixed splicing graph;

[0356] The encoding unit 13 is configured to encode the heterogeneous mixed spliced ​​graph to obtain a bit stream.

[0357] In some embodiments, the encoding unit 13 is specifically configured to:

[0358] Calling a video encoder to perform video encoding on the heterogeneous mixed splicing graph to obtain a video compression sub-stream;

[0359] Encoding the mixed splicing information of the heterogeneous mixed splicing graph to obtain a mixed splicing information sub-stream;

[0360] The video compression sub-stream and the mixed splicing information sub-stream are written into the stream.

[0361] In some embodiments, the at least two homogeneous strips include at least two different types of strips among multi-view video strips, point cloud strips, and mesh strips.

[0362] In some embodiments, the heterogeneous mixed mosaic graph includes at least one of a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph.

[0363] In some embodiments, the encoding unit 13 is further configured to:

[0364] Writing a parameter set of the codestream into the codestream, wherein the parameter set includes first information, and the first information is used to indicate whether the codestream simultaneously includes codestreams corresponding to at least two visual media contents in different expression formats.

[0365] In some embodiments, the encoding unit 13 is further configured to:

[0366] The first information is set to a first preset value, wherein the first preset value is used to indicate that the code stream includes both a code stream of a multi-view video and a code stream of a point cloud.

[0367] In some embodiments, the encoding unit 13 is further configured to:

[0368] Setting the first information to a second preset value, wherein the second preset value is used to indicate that the code stream includes a multi-viewpoint video; or

[0369] The first information is set to a third preset value, wherein the third preset value is used to indicate that the code stream includes a point cloud.

[0370] In some embodiments, the hybrid splicing information of the heterogeneous hybrid splicing graph includes a first flag and a second flag, and the encoding unit 13 is further configured to:

[0371] Setting the first flag to a first value, where the first value is used to indicate that the code stream includes a multi-view video syntax element; and

[0372] The second flag is set to a second value, where the second value is used to indicate that the code stream includes a point cloud syntax element.

[0373] In some embodiments, the first marker and the second marker are in an atlas sequence parameter set.

[0374] In some embodiments, the hybrid splicing information further includes a third flag, and the third flag is used to indicate the expression format type of the i-th strip in the heterogeneous hybrid splicing graph, where i is a positive integer.

[0375] In some embodiments, the encoding unit 13 is further configured to:

[0376] If the i-th slice is the multi-view video slice, setting the value of the third flag to a third value;

[0377] If the i-th strip is the point cloud strip, the value of the third flag is set to a fourth value.

[0378] In some embodiments, the third flag is located in a stripe data unit.

[0379] In some embodiments, the mixed splicing information further includes sub-image block information of the i-th slice, where the sub-image block information is used to indicate encoding and decoding information of the sub-image blocks in the i-th slice.

[0380] In some embodiments, if the i-th slice is the multi-view video slice, the sub-image block information is used to indicate whether the current sub-image block is obtained using a multi-view video coding standard;

[0381] If the i-th strip is the point cloud strip, the sub-block information is used to indicate that the current sub-block is obtained using the point cloud video coding standard.

[0382] In some embodiments, the sub-tile information is located in a sub-tile data unit.

[0383] In some embodiments, the at least two visual media contents are media contents presented simultaneously in the same three-dimensional space.

[0384] It should be understood that the device embodiments and the method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be described here. Specifically, the device 10 shown in Figure 16 can execute the encoding method of the encoding end of the embodiment of the present application, and the aforementioned and other operations and / or functions of the various units in the device 10 are respectively for implementing the corresponding processes in each method such as the encoding method of the encoding end, etc., for the sake of brevity, they will not be described here.

[0385] FIG17 is a schematic block diagram of a decoding device provided in an embodiment of the present application, which is applied to the above-mentioned decoding end.

[0386] As shown in FIG17 , the decoding device 20 may include:

[0387] A decoding unit 21 is used to decode the bitstream to obtain a reconstructed heterogeneous mixed splicing image;

[0388] The splitting unit 22 is used to split the reconstructed heterogeneous mixed mosaic image to obtain at least two reconstructed isomorphic strips; the reconstructed isomorphic strips correspond to at least two different expression formats.

[0389] The processing unit 23 is configured to obtain at least two reconstructed visual media contents according to the at least two reconstructed isomorphic strips, where the reconstructed visual media contents correspond to at least two different expression formats.

[0390] In some embodiments, the code stream includes a video compression sub-code stream; the decoding unit 21 is specifically configured to:

[0391] A video decoder is called to decode the video compression sub-stream to obtain the reconstructed heterogeneous mixed splicing graph.

[0392] In some embodiments, the code stream further includes mixed splicing information sub-code stream decoding, and the decoding unit 21 is further configured to:

[0393] Decoding the mixed splicing information sub-stream to obtain mixed splicing information;

[0394] The splitting unit 22 is specifically used for:

[0395] The reconstructed heterogeneous mixed splicing image is split according to the mixed splicing information to obtain the at least two reconstructed isomorphic strips.

[0396] In some embodiments, the at least two homogeneous strips include at least two different types of strips among multi-view video strips, point cloud strips, and mesh strips.

[0397] In some embodiments, the heterogeneous mixed mosaic graph includes at least one of a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph.

[0398] In some embodiments, the decoding unit 21 is further configured to:

[0399] The code stream is decoded to obtain a parameter set, wherein the parameter set includes first information, and the first information is used to indicate that the code stream simultaneously includes code streams corresponding to at least two visual media contents in different expression formats.

[0400] In some embodiments, the decoding unit 21 is further configured to:

[0401] If the first information is a first preset value, it is determined that the code stream includes both the code stream of the multi-view video and the code stream of the point cloud.

[0402] In some embodiments, the decoding unit 21 is further configured to:

[0403] If the first information is the second preset value, determining that the code stream includes a code stream of a multi-viewpoint video; or

[0404] If the first information is a third preset value, it is determined that the code stream includes a point cloud code stream.

[0405] In some embodiments, the hybrid splicing information of the reconstructed heterogeneous hybrid splicing graph includes a first flag and a second flag, and the decoding unit 21 is further configured to:

[0406] If the first flag is a first value, it is determined that the code stream includes a multi-view video syntax element;

[0407] If the second flag is a second value, it is determined that the code stream includes a point cloud syntax element.

[0408] In some embodiments, the first marker and the second marker are in an atlas sequence parameter set.

[0409] In some embodiments, the hybrid splicing information further includes a third flag, and the third flag is used to indicate the expression format type of the i-th strip in the reconstructed heterogeneous hybrid splicing image, where i is a positive integer.

[0410] In some embodiments, the splitting unit 22 is specifically configured to:

[0411] For an i-th stripe in the reconstructed heterogeneous mixed splicing image, obtaining a third flag corresponding to the i-th stripe from the mixed splicing information;

[0412] According to the third flag corresponding to the i-th stripe, the i-th stripe is extracted as a reconstructed isomorphic stripe of the visual media expression format type corresponding to the i-th stripe.

[0413] In some embodiments, if the at least two reconstructed isomorphic strips include a reconstructed multi-view video strip and a reconstructed point cloud strip, the splitting unit 22 is specifically configured to:

[0414] If the value of the third flag is a third value, it is determined that the expression format of the media content corresponding to the i-th slice is a multi-view video;

[0415] If the value of the third flag is the fourth value, it is determined that the expression format of the media content corresponding to the i-th strip is a point cloud.

[0416] In some embodiments, the third flag is located in a stripe data unit.

[0417] In some embodiments, the mixed splicing information further includes sub-image block information of the i-th slice, where the sub-image block information is used to indicate encoding and decoding information of the sub-image blocks in the i-th slice.

[0418] In some embodiments, the processing unit 23 is specifically configured to:

[0419] If the i-th slice is the multi-view video slice, performing multi-view video decoding on the current sub-image block of the i-th slice to obtain a reconstructed multi-view video;

[0420] If the i-th strip is the point cloud strip, point cloud decoding is performed on the current sub-block of the i-th strip to obtain a reconstructed point cloud.

[0421] In some embodiments, the sub-tile information is located in a sub-tile data unit.

[0422] In some embodiments, the at least two visual media contents are media contents presented simultaneously in the same three-dimensional space.

[0423] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions can refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device 20 shown in Figure 17 may correspond to the corresponding subject in the prediction method of the decoding end of the embodiment of the present application, and the aforementioned and other operations and / or functions of each unit in the device 20 are respectively for implementing the corresponding processes in each method such as the decoding method of the decoding end. For the sake of brevity, no further description is given here.

[0424] The above describes the apparatus and system of the embodiment of the present application from the perspective of functional units in conjunction with the accompanying drawings. It should be understood that the functional unit can be implemented in the form of hardware, can be implemented by instructions in the form of software, or can be implemented by a combination of hardware and software units. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software units in the decoding processor. Optionally, the software unit can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0425] FIG18 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

[0426] As shown in FIG18 , the electronic device 30 may be a video encoder or a video decoder as described in an embodiment of the present application. The electronic device 30 may include:

[0427] The memory 33 and the processor 32 are configured to store a computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the method in the embodiment of the present application.

[0428] For example, the processor 32 may be configured to execute the steps of the method 200 according to the instructions in the computer program 34 .

[0429] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0430] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0431] In some embodiments of the present application, the memory 33 includes but is not limited to:

[0432] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0433] In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to implement the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.

[0434] As shown in FIG18 , the electronic device 30 may further include:

[0435] The transceiver 33 may be connected to the processor 32 or the memory 33 .

[0436] The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

[0437] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0438] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0439] The present application also provides a code stream, which is generated according to the above encoding method. Optionally, the code stream includes the above first flag, or includes the first flag and the second flag.

[0440] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0441] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0442] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the unit is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0443] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0444] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A coding method, characterized in that: include: Processing at least two visual media contents to obtain at least two isomorphic strips, wherein the visual media contents correspond to at least two different expression formats, and the isomorphic strips correspond to at least two different expression formats; Splicing the at least two isomorphic strips to generate a heterogeneous mixed splicing graph; The heterogeneous mixed spliced ​​graph is encoded to obtain a code stream.

2. The method according to claim 1, characterized in that The encoding of the heterogeneous mixed spliced ​​graph to obtain a code stream includes: Calling a video encoder to perform video encoding on the heterogeneous mixed splicing graph to obtain a video compression sub-stream; Encoding the mixed splicing information of the heterogeneous mixed splicing graph to obtain a mixed splicing information sub-stream; The video compression sub-stream and the mixed splicing information sub-stream are written into the stream.

3. The method according to claim 1 or 2, characterized in that The at least two isomorphic strips belong to at least two different types of strips among multi-view video strips, point cloud strips and mesh strips.

4. The method according to any one of claims 1 to 3, characterized in that The heterogeneous mixed mosaic graph includes at least one of a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Writing a parameter set of the codestream into the codestream, wherein the parameter set includes first information, and the first information is used to indicate whether the codestream simultaneously contains codestreams corresponding to at least two visual media contents in different expression formats.

6. The method according to claim 5, characterized in that Also includes: The first information is set to a first preset value, wherein the first preset value is used to indicate that the code stream includes both a code stream of a multi-view video and a code stream of a point cloud.

7. The method according to claim 5, characterized in that Also includes: Setting the first information to a second preset value, wherein the second preset value is used to indicate that the code stream includes a multi-view video; or The first information is set to a third preset value, wherein the third preset value is used to indicate that the code stream includes a point cloud.

8. The method according to claim 6 or 7, characterized in that The hybrid splicing information of the heterogeneous hybrid splicing graph includes a first flag and a second flag, and the method further includes: Setting the first flag to a first value, where the first value is used to indicate that the code stream includes a multi-view video syntax element; and The second flag is set to a second value, where the second value is used to indicate that the code stream includes a point cloud syntax element.

9. The method according to claim 8, characterized in that The first flag and the second flag are located in an atlas sequence parameter set.

10. The method according to any one of claims 7 to 9, characterized in that: The hybrid splicing information further includes a third flag, which is used to indicate the expression format type of the i-th strip in the heterogeneous hybrid splicing graph, where i is a positive integer.

11. The method according to claim 10, characterized in that Also includes: If the i-th slice is the multi-view video slice, setting the value of the third flag to a third value; If the i-th strip is the point cloud strip, the value of the third flag is set to a fourth value.

12. The method according to claim 10 or 11, characterized in that The third flag is located in the stripe data unit.

13. The method according to any one of claims 10 to 12, characterized in that: The mixed splicing information further includes sub-image block information of the i-th slice, where the sub-image block information is used to indicate encoding and decoding information of the sub-image blocks in the i-th slice.

14. The method according to claim 14, characterized in that If the i-th slice is the multi-view video slice, the sub-image block information is used to indicate whether the current sub-image block is obtained by adopting a multi-view video coding standard; If the i-th strip is the point cloud strip, the sub-block information is used to indicate that the current sub-block is obtained using the point cloud video coding standard.

15. The method according to claim 13 or 14, characterized in that The sub-block information is located in a sub-block data unit.

16. The method according to any one of claims 1 to 15, characterized in that The at least two visual media contents are media contents presented simultaneously in the same three-dimensional space.

17. A decoding method, characterized in that: include: Decode the bitstream to obtain a reconstructed heterogeneous mixed splicing graph; Splitting the reconstructed heterogeneous mixed mosaic graph to obtain at least two reconstructed isomorphic strips, wherein the reconstructed isomorphic strips correspond to at least two different expression formats; At least two reconstructed visual media contents are obtained according to the at least two reconstructed isomorphic strips, and the reconstructed visual media contents correspond to at least two different expression formats.

18. The method according to claim 17, characterized in that The code stream includes a video compression sub-code stream; The decoding of the bitstream to obtain a reconstructed heterogeneous hybrid splicing graph includes: A video decoder is called to decode the video compression sub-stream to obtain the reconstructed heterogeneous mixed splicing graph.

19. The method according to claim 17 or 18, characterized in that The code stream also includes mixed splicing information sub-code stream decoding, and the method further includes: Decoding the mixed splicing information sub-stream to obtain mixed splicing information; The step of splitting the reconstructed heterogeneous mixed mosaic image to obtain at least two reconstructed isomorphic strips includes: The reconstructed heterogeneous mixed splicing image is split according to the mixed splicing information to obtain the at least two reconstructed isomorphic strips.

20. The method according to any one of claims 17 to 19, characterized in that: The at least two isomorphic strips belong to at least two different types of strips among multi-view video strips, point cloud strips and mesh strips.

21. The method according to any one of claims 17 to 20, characterized in that: The heterogeneous mixed mosaic graph includes at least one of a single-attribute heterogeneous mixed mosaic graph and a multi-attribute heterogeneous mixed mosaic graph.

22. The method according to any one of claims 17 to 21, characterized in that Also includes: The bitstream is decoded to obtain a parameter set, wherein the parameter set includes first information, and the first information is used to indicate whether the bitstream simultaneously contains bitstreams corresponding to at least two visual media contents in different expression formats.

23. The method according to claim 22, characterized in that Also includes: If the first information is a first preset value, it is determined that the code stream includes both the code stream of the multi-view video and the code stream of the point cloud.

24. The method according to claim 22, characterized in that Also includes: If the first information is a second preset value, determining that the code stream includes a code stream of a multi-view video; or If the first information is a third preset value, it is determined that the code stream includes a point cloud code stream.

25. The method according to claim 23 or 24, characterized in that The hybrid splicing information for reconstructing the heterogeneous hybrid splicing graph includes a first flag and a second flag, and the method further includes: If the first flag is a first value, it is determined that the code stream includes a multi-view video syntax element; If the second flag is a second value, it is determined that the code stream includes a point cloud syntax element.

26. The method according to claim 25, characterized in that The first flag and the second flag are located in an atlas sequence parameter set.

27. The method according to any one of claims 24 to 26, characterized in that The hybrid splicing information further includes a third flag, which is used to indicate an expression format type of the i-th strip in the reconstructed heterogeneous hybrid splicing image, where i is a positive integer.

28. The method according to claim 27, characterized in that The step of splitting the reconstructed heterogeneous mixed mosaic image to obtain at least two reconstructed isomorphic strips includes: For an i-th stripe in the reconstructed heterogeneous mixed splicing image, obtaining a third flag corresponding to the i-th stripe from the mixed splicing information; According to the third flag corresponding to the i-th stripe, the i-th stripe is extracted as a reconstructed isomorphic stripe of the visual media expression format type corresponding to the i-th stripe.

29. The method according to claim 28, characterized in that If the at least two reconstructed isomorphic strips include a reconstructed multi-view video strip and a reconstructed point cloud strip, extracting the i-th strip as a reconstructed isomorphic strip of a visual media expression format type corresponding to the i-th strip according to a third flag corresponding to the i-th strip includes: If the value of the third flag is the third value, it is determined that the visual media expression format type corresponding to the i-th stripe is a multi-view video; If the value of the third flag is the fourth value, it is determined that the visual media expression format type corresponding to the i-th strip is point cloud.

30. The method according to any one of claims 27 to 29, characterized in that The third flag is located in the stripe data unit.

31. The method according to any one of claims 27 to 30, characterized in that The mixed splicing information further includes sub-image block information of the i-th slice, where the sub-image block information is used to indicate encoding and decoding information of the sub-image blocks in the i-th slice.

32. The method according to claim 31, characterized in that Obtaining at least two reconstructed visual media contents according to the at least two reconstructed isomorphic strips includes: If the i-th slice is the multi-view video slice, performing multi-view video decoding on the current sub-image block of the i-th slice to obtain a reconstructed multi-view video; If the i-th strip is the point cloud strip, point cloud decoding is performed on the current sub-block of the i-th strip to obtain a reconstructed point cloud.

33. The method according to claim 31 or 32, characterized in that The sub-block information is located in a sub-block data unit.

34. The method according to any one of claims 17 to 33, characterized in that The multiple visual media contents are media contents presented simultaneously in the same three-dimensional space.

35. An encoding device, characterized in that include: A first splicing unit is configured to process at least two visual media contents respectively to obtain at least two isomorphic strips, wherein the visual media contents correspond to at least two different expression formats, and the isomorphic strips correspond to at least two different expression formats; A second splicing unit is configured to splice the at least two isomorphic strips to generate a heterogeneous mixed splicing graph; The encoding unit is used to encode the heterogeneous mixed splicing graph to obtain a code stream.

36. A decoding device, characterized in that: include: A decoding unit, used for decoding the bitstream to obtain a reconstructed heterogeneous mixed splicing graph; A splitting unit is used to split the reconstructed heterogeneous mixed mosaic to obtain at least two reconstructed isomorphic strips; the reconstructed isomorphic strips correspond to at least two different expression formats The processing unit is configured to obtain at least two reconstructed visual media contents according to the at least two reconstructed isomorphic strips, wherein the reconstructed visual media contents correspond to at least two different expression formats.

37. An electronic device, characterized in that: including processor and memory; The memory shown is used to store computer programs; The processor is configured to call and execute the computer program stored in the memory to implement the method according to any one of claims 1 to 34.

38. A computer-readable storage medium, characterized in that For storing computer programs; The computer program enables a computer to execute the method according to any one of claims 1 to 34.

39. A code stream, characterized in that The code stream is generated based on the method according to any one of claims 1 to 16.