Joint coding of texture and displacement data in dynamic grid coding

Abstract

A mechanism for processing video data is disclosed. The mechanism includes determining that the displacement data is encoded and decoded in a 4: 2: 0 format. A conversion between the visual media data and a bitstream is performed based on the displacement data encoded and decoded in a 4: 2: 0 format.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims the benefit of International Patent Application No. PCT / CN2023 / 089469 filed on April 20, 2023, International Patent Application No. PCT / CN2023 / 071918 filed on January 12, 2023, International Patent Application No. PCT / CN2023 / 106331 filed on July 7, 2023, and International Patent Application No. PCT / CN2023 / 123207 filed on October 7, 2023, which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the generation, storage, and use of digital audio-visual media information in file format. Background Art

[0004] Digital video accounts for the largest share of bandwidth used on the Internet and other digital communications networks. As the number of connected user devices capable of receiving and displaying video increases, the bandwidth demand for digital video usage is likely to continue to grow. Summary of the Invention

[0005] A first aspect relates to a method for processing visual media data including displacement data, comprising: determining that the displacement data is encoded and decoded in a 4:2:0 format; and performing conversion between the visual media data and a bitstream based on the displacement data encoded and decoded in the 4:2:0 format.

[0006] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the displacement data is converted to a 4:2:0 format before encoding and converted to a 4:4:4 format after decoding.

[0007] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that one or more of the sps_chroma_format_idc syntax element and the ChromaFormatIdc variable are set to 1 to encode and decode the displacement data.

[0008] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the main profile or main10 profile is used for encoding and decoding the displacement data.

[0009] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that when the displacement data has only one non-zero component, the displacement data is packed into the luminance component and the chrominance component of the 4:2:0 format.

[0010] Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the sub-block size of the sub-block of the visual media data is 2 S Represented by , where S is a non-negative integer.

[0011] Optionally, in any of the preceding aspects, another implementation of this aspect provides that the bitstream includes a value equal to SK, where K is a non-negative integer.

[0012] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the values ​​in the bitstream are represented as unsigned integer Exp-Golomb codec syntax elements (ue(v)) or as unsigned integers using n bits (u(n)).

[0013] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that K is equal to 0.

[0014] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that K is equal to 6.

[0015] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that K is equal to 7.

[0016] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the value of SK is in the range of 0 to 3 (including endpoints).

[0017] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the visual media data includes displacement data and texture data, and wherein the texture data and the displacement data are included in a single bitstream.

[0018] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the displacement data is converted into a 4:2:0 format, and the converted displacement data is spliced ​​with the texture data in the 4:2:0 format.

[0019] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the displacement data is converted to N bits, wherein the N bits are the bit depth of the texture data, and wherein N is an integer.

[0020] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that N is 10.

[0021] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data and the displacement data are encoded and decoded in different strips.

[0022] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that one or more of the location and size of the texture data are included in the single bitstream.

[0023] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that one or more of the position and size of the displacement data are included in the single bitstream.

[0024] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that one or more of the position and size of the texture data are inferred based on information in the single bitstream.

[0025] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that one or more of the position and size of the displacement data are inferred based on information in the single bit stream.

[0026] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data and the displacement data are included in a single bitstream and use different encoding and decoding methods.

[0027] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the different encoding and decoding methods include a first quantization parameter and a second quantization parameter different from the first quantization parameter, and wherein the texture data uses the first quantization parameter and the displacement data uses the second quantization parameter.

[0028] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the different coding method includes lossless coding, and all video units of the displacement data use the lossless coding.

[0029] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the different coding method includes a transform quantization bypass mode from the High Efficiency Video Coding (HEVC) standard, and wherein all video units of the displacement data use the transform quantization bypass mode.

[0030] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the different coding and decoding methods include a transform skip mode and a quantization parameter, and wherein all video units of the displacement data use the transform skip mode and the quantization parameter.

[0031] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the quantization parameter is equal to 4+6*K, where K is an integer.

[0032] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the value of K is zero.

[0033] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that data other than the texture data and the displacement data is used to fill the picture in the single bitstream.

[0034] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the picture is filled with a fixed value.

[0035] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the picture is padded with intermediate pixel values, and wherein for 8-bit video, the intermediate pixel value is 128, and for 10-bit video, the intermediate pixel value is 512.

[0036] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the image is filled with the value of the closest pixel in the texture data or with the value of the closest pixel in the displacement data.

[0037] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that when filling the image, N rows of luminance samples and N / 2 rows of chrominance samples are inserted between the texture data and the displacement data, where N is an integer.

[0038] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the value of N is 16.

[0039] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the value of N is 0.

[0040] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that all samples in the N rows of luminance samples and the N / 2 rows of chrominance samples have the same value.

[0041] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the same value includes an intermediate pixel value.

[0042] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that a smoothing process is applied to the pictures in the single bitstream.

[0043] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that it is derived at the decoder whether the displacement data has only one non-zero component or three non-zero components.

[0044] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that, at the decoder, it is determined whether the displacement data has only one non-zero component or three non-zero components based on the video resolution of the displacement data and the number of base grid points.

[0045] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that, at the decoder, it is determined whether the displacement data has only one non-zero component or three non-zero components based on the video resolution and the number of vertices of the displacement data.

[0046] Optionally, in any of the foregoing aspects, another implementation of the aspect provides that the texture data is converted to a color space before encoding and the texture data is converted back to the color space after decoding, wherein the color space is not a blue-green-red (BGR) color space or a YUV color space, wherein Y represents a luminance component and U and V represent a blue chrominance component and a red chrominance component.

[0047] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data in the bitstream is encoded and decoded in a green-blue-red (GBR) color space.

[0048] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data in the bitstream is encoded and decoded in a green-red-blue (GRB) color space.

[0049] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data in the bitstream is encoded and decoded in the YCgCo color space, where Y represents the luminance component, Cg represents the green chrominance component, and Co represents the orange chrominance component.

[0050] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the texture data in the bitstream is encoded and decoded in the YCoCg color space, where Y represents the luminance component, Co represents the orange chrominance component, and Cg represents the green chrominance component.

[0051] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the color space is used for lossless encoding and decoding of the texture data.

[0052] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that indices representing primary colors of a green-red-blue (GRB) color space are used for encoding and decoding according to an International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) standard.

[0053] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that a first sub-block size (SB1) used to encode and decode a first color component of the visual media data is different from a second sub-block size (SB2) used to encode and decode a second color component of the visual media data.

[0054] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the second sub-block size is larger than the first sub-block size.

[0055] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the first sub-block size is 100 and the second sub-block size is 200.

[0056] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the first sub-block size is 128 and the second sub-block size is 256.

[0057] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the displacement data includes inter-frame displacement data and intra-frame displacement data, and wherein the inter-frame displacement data uses a different block size from the intra-frame displacement data.

[0058] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that when a packed information scheme is used to support joint encoding and decoding of displacement data and texture data in dynamic mesh encoding and decoding, the pin occupancy present flag (Pinoccupancy Present Flag) is equal to 0.

[0059] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that when the pin occupation existence flag is not equal to 0, the dynamic grid decoder ignores the area of ​​the visual media data corresponding to the pin occupation existence flag.

[0060] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the pin occupancy present flag is designated as pin_occupancy_present_flag[j].

[0061] Optionally, in any of the aforementioned aspects, another implementation of this aspect stipulates that the pin area type syntax element corresponding to the visual media data is not 0.

[0062] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the pin region type syntax element is specified as pin_region_type_id_minus2[j][i].

[0063] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that any region in the packaging information scheme does not overlap with another region in the packaging information scheme.

[0064] Optionally, in any of the aforementioned aspects, another implementation of this aspect provides that the converting includes encoding the media data into a bitstream.

[0065] Optionally, in any of the preceding aspects, another implementation of this aspect provides that the converting includes decoding the media data from a bitstream.

[0066] A second aspect relates to an apparatus for processing media data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform a method as described in any one of the disclosed embodiments.

[0067] A third aspect relates to a non-transitory computer-readable medium, comprising a computer program product for use by a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs a method as described in any one of the disclosed embodiments.

[0068] A fourth aspect relates to a non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing device, wherein the method includes the method as described in any one of the disclosed embodiments.

[0069] A fifth aspect relates to a method for storing a bitstream of a video, including the method as described in any one of the disclosed embodiments.

[0070] A sixth aspect relates to a method, apparatus or system described in the present disclosure.

[0071] For clarity, any of the foregoing embodiments may be combined with one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0072] These and other features will become more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0074] Figure 1 An example decoder design for dynamic trellis coding is shown.

[0075] Figure 2 An example structure of a dynamic grid codec test model is shown.

[0076] Figure 3 An example is shown in which texture data and displacement data are combined into one picture.

[0077] Figure 4 Another example of combining texture data and displacement data into one picture is shown.

[0078] Figure 5 is a block diagram illustrating an example video processing system.

[0079] Figure 6 is a block diagram of an example video processing device.

[0080] Figure 7 is a flow chart of an example method of video processing.

[0081] Figure 8 is a block diagram illustrating an example video encoding and decoding system.

[0082] Figure 9 is a block diagram illustrating an example encoder.

[0083] Figure 10 is a block diagram illustrating an example decoder.

[0084] Figure 11 is a schematic diagram of an example encoder.

[0085] Figure 12 is a flow chart of an example method of video processing. DETAILED DESCRIPTION

[0086] First, it should be understood that although exemplary implementations of one or more embodiments are provided below, the disclosed systems and / or methods may be used with any number of technical facts, whether currently known or yet to be developed. The present disclosure should not be limited in any way to the exemplary implementations, drawings, and techniques shown below, including the exemplary designs and implementations shown and described herein, but may be modified within the full scope of the appended claims and their equivalents.

[0087] The section titles used in this disclosure are for ease of understanding and are not intended to limit the applicability of the techniques and embodiments disclosed in each section to that section. In addition, the techniques described herein are applicable to other video encoder protocols and designs.

[0088] 1. Preliminary Discussion

[0089] This disclosure relates to dynamic mesh encoding and decoding based on Moving Picture Experts Group (MPEG)-I video. Specifically, it relates to how to jointly encode and decode displacement and texture data. This disclosure may also be applicable to other immersive video encoding and decoding standards or codecs.

[0090] 2. Further Discussion

[0091] In computer graphics, three-dimensional (3D) / immersive content is often represented using 3D meshes and texture maps. These mesh and texture data can be machine-generated or converted from images captured by multiple cameras from different angles. Similar to two-dimensional (2D) video, when 3D content changes over time, the mesh and texture data also change, forming a dynamic mesh sequence. The amount of data required for dynamic meshes is often very large and difficult to store and transmit. To meet the needs of applications using dynamic meshes, MPEG has issued a request for proposals. To effectively use 2D codecs, one requirement is to use 2D video codec standards to compress most of the data, while keeping the remaining components simple and low-complexity. This requirement ensures that the representation can leverage 2D video hardware / software systems, eliminating the need for extensive system redesign for dynamic meshes.

[0092] MPEG received several responses to the RFP. One proposal performed better than the others. This established a test model for the planned development of a dynamic trellis codec standard.

[0093] The test model for the dynamic mesh codec can be found at: http: / / mpegx.int-evry.fr / software / MPEG / dmc / mpeg-vmesh-tm / - / tags / v2.0; and the latest working draft document is WD 1.0.

[0094] 2.1 Data Representation in Dynamic Mesh Encoding and Decoding

[0095] Figure 1 An example decoder design for dynamic trellis coding is shown. Figure 1 An example decoder design is shown. As can be seen, the dynamic mesh decoder receives three bit streams and performs decoding to reconstruct the dynamic mesh plus texture signal. The first bit stream represents the base mesh, that is, a simplified version of the original mesh. The second bit stream represents the displacement vector between the reconstructed base mesh and the original mesh. The displacement vectors are arranged as a 2D video and compressed using a codec that complies with the 2D video codec standard. The third bit stream is used to represent the texture (or attribute map). The attribute map is also arranged as a 2D video and compressed using a codec that complies with the 2D video codec standard. The design concept is to make the base mesh part small enough so that the module that processes the basic mesh can be simply implemented. On the other hand, the displacement vectors and attribute maps occupy most of the data volume of the entire dynamic mesh data, which can be processed by a dedicated efficient 2D video codec system. Such a design can reduce the additional workload of implementing the dynamic mesh codec system and ensure high throughput and codec efficiency of the dynamic mesh data.

[0096] 2.2 Dynamic Mesh Encoding and Decoding Test Model

[0097] Figure 2 An example structure of a dynamic grid codec test model is shown. Figure 2 The structure of an example dynamic mesh codec model is shown. In this model, Draco is used to compress the base mesh, while the High Efficiency Video Codec (HEVC) Test Model (HM) is used to compress the displacement vectors and attribute maps. However, it should be noted that dynamic mesh codecs can also use other mesh or video codec systems.

[0098] A base mesh m is generated from the original mesh using a downsampling scheme. Its quantized version m' is then encoded and decoded using Draco. The reconstructed base mesh m" is obtained by dequantizing m'. Displacement vectors d' are generated by taking the difference between the original mesh and the subdivided version of m" obtained using the subdivision scheme.

[0099] 2.3 Displacement Vector Encoding and Decoding

[0100] After obtaining the displacement vector d' (the difference between the original mesh and the subdivided base mesh), a lifting-based wavelet transform is applied to further concentrate the energy. The wavelet transform coefficients are then traversed from low frequency to high frequency using the Morton order to form 2D coefficient blocks. The various 2D coefficient blocks constitute the picture to be processed by the 2D codec.

[0101] 2.4 Motion Field Codec

[0102] In the example test model, the motion field between the base meshes is directly encoded and decoded using arithmetic codec. The example design also studies the encoding and decoding of the motion field of a standard-compliant 2D codec system, and the results show that the loss in codec efficiency is minimal. Therefore, it may be meaningful to further transfer the motion field encoding and decoding process to the 2D video codec.

[0103] 2.5 chroma format

[0104] In H.264 / Advanced Video Codec (AVC), H.265 / HEVC, and H.266 / Versatile Video Codec (VVC), different chroma formats are supported. The format can be signaled via the syntax element sps_chroma_format_idc and represented by the variable ChromaFormatIdc. The following table shows the chroma formats corresponding to different sps_chroma_format_idc:

[0105] sps_chroma_format_idc Chroma format 0 monochrome 1 4:2:0 2 4:2:2 3 4:4:4

[0106] 2.6 Lossless Codec

[0107] In H.264 / AVC and H.266 / VVC, lossless coding can be achieved by setting the QP to 4 and applying reversible spatial transform or transform skipping to the block. In H.265 / HEVC, in addition to the above methods, lossless coding can also be achieved by setting the cu_transquant_bypass_flag of the codec unit to 1.

[0108] 2.7 Example Design

[0109] In the example design, the idea of ​​combining multiple attributes (including texture, displacement data, and occupancy data) into a single video for encoding / decoding is proposed, without requiring multiple encoding / decoding capabilities on the device. This disclosure details how to jointly encode and decode texture data and displacement data for dynamic mesh encoding and decoding.

[0110] 2.8 Arithmetic Coding and Decoding of Displacement Data

[0111] In the example design of dynamic mesh coding [5], displacement data can be encoded and decoded using arithmetic coding. The following table shows the syntax and semantics.

[0112] J.7.1.3.7 Syntax of data unit in displacement frame

[0113]

[0114]

[0115] In the above table, subblock_size is represented by u(16) in the bitstream.

[0116] J.7.3.5 Semantics of data units within displacement frames

[0117] The arithmetic decoding engine is a context-separated binary arithmetic decoder that performs binary renormalization and produces binary output.

[0118] The displacement value is derived through arithmetic decoding.

[0119] diu_last_sig_coeff[k] indicates the index of the last position of the non-zero displacement coefficient level in the k-th component.

[0120] diu_coded_block_flag[k][b] indicates whether the block with index b has any non-zero displacement coefficient levels in the kth component (when 1) or not (when 0).

[0121] diu_coded_subblock_flag[k][b][s] indicates whether the subblock with index s of the block with index b has any non-zero displacement coefficient levels in the kth component (when 1) or not (when 0).

[0122] diu_coeff_abs_level_gt0[k][b][s][v] indicates whether the absolute value of the kth component of the displacement coefficient level associated with the vertex with index v of the sub-block with index s of the block with index b is greater than zero (when 1) or not (when 0).

[0123] diu_coeff_abs_level_gt1[k][b][s][v] indicates whether the absolute value of the kth component of the displacement coefficient level associated with the vertex indexed v of the sub-block indexed s of the block indexed b is greater than one (when 1) or (when 0). If diu_coeff_abs_level_gt1[k][b][s][v] is not present, it should be inferred to be equal to 0.

[0124] diu_coeff_sign[k][b][s][v] indicates whether the absolute value of the kth component of the displacement coefficient level associated with the vertex indexed by v of the sub-block indexed by s of the block indexed by b has a positive sign (when 1) or not (when 0). If diu_coeff_sign[k][b][s][v] is not present, it should be inferred to be equal to 1.

[0125] diu_coeff_abs_level_rem[k][b][s][v] indicates the absolute value of the k-th component of the displacement coefficient level associated with the vertex indexed v of the block indexed b, minus 2. If diu_coeff_abs_level_rem[k][b][s][v] is not present, it shall be inferred to be equal to 0.

[0126] Packaging information in 2.9V3C design

[0127] In the latest V3C design [7], a packetized information scheme is supported to encode and decode multiple types of information in one frame. The V3C unit type is V3C_PVD which uses this scheme. The corresponding syntax and semantics are as follows:

[0128] 8.3.4.7 Packing Information Syntax

[0129]

[0130]

[0131]

[0132] 8.4.4.7 Packaging Information Semantics

[0133] A packed video frame can be divided into one or more rectangular regions. A region should be mapped to one atlas slice. The rectangular regions of a packed video frame are not allowed to overlap.

[0134] pin_codec_id[j] indicates the mapping index of the codec identifier of the video decoder used to decode the packetized video sub-bitstream associated with the atlas with ID j (if present), as described in section 9.6. pin_codec_id[j] shall be in the range 0 to 255 (inclusive).

[0135] pin_occupancy_present_flag[j] equal to 0 indicates that the packed video frame of the atlas with atlas ID j does not contain a region with occupancy data. pin_occupancy_present_flag[j] equal to 1 indicates that the packed video frame of the atlas with atlas ID j contains a region with occupancy data. When pin_occupancy_present_flag[j] is not present, its value is inferred to be 0.

[0136] Bitstream conformance requirement: If pin_occupancy_present_flag[j] is equal to 1 for an atlas with atlas ID j, then vps_occupancy_video_present_flag[j] shall be equal to 0 for all atlases with the same atlas ID j.

[0137] pin_geometry_present_flag[j] equal to 0 indicates that the packed video frame of the atlas with atlas ID j does not contain regions with geometry data. pin_occupancy_present_flag[j] equal to 1 indicates that the packed video frame of the atlas with atlas ID j contains regions with geometry data. When pin_geometry_present_flag[j] is not present, its value is inferred to be 0.

[0138] Bitstream conformance requirement: If pin_geometry_present_flag[j] is equal to 1 for an atlas with atlas ID j, then vps_geometry_video_present_flag[j] shall be equal to 0 for all atlases with the same atlas ID j.

[0139] pin_attribute_present_flag[j] equal to 0 indicates that the packed video frame of the atlas with atlas ID j does not contain a region with attribute data. pin_attribute_present_flag[j] equal to 1 indicates that the packed video frame of the atlas with atlas ID j contains a region with attribute data. When pin_attribute_present_flag[j] is not present, its value is inferred to be 0.

[0140] Bitstream conformance requirement: If pin_attribute_present_flag[j] is equal to 1 for an atlas with atlas ID j, then vps_attribute_video_present_flag[j] shall be equal to 0 for all atlases with the same atlas ID j.

[0141] pin_occupancy_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth that the decoded region containing occupancy data for the atlas with atlas ID j should be converted to. pin_occupancy_msb_align_flag[j] should be in the range 0 to 31 (inclusive).

[0142] pin_occupancy_msb_align_flag[j] indicates how to convert the decoded region containing occupancy samples associated with the atlas with atlas ID j to samples at the nominal occupancy bit-depth, as specified in Annex B.

[0143] pin_lossy_occupancy_compression_threshold[j] indicates the threshold used to derive binary occupancy information from the decoded region containing occupancy data for the atlas with atlas ID j, as specified in Annex B. pin_lossy_occupancy_compression_threshold[j] shall range from 0 to 255 (inclusive).

[0144] pin_geometry_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which the decoded region containing geometry data for the atlas with atlas ID j should be converted. pin_geometry_2d_bit_depth_minus1[j] should range from 0 to 31 (inclusive).

[0145] pin_geometry_msb_align_flag[j] indicates how the decoded region containing the geometry samples associated with the atlas with atlas ID j is converted to samples at the nominal occupied bit depth, as specified in Annex B.

[0146] pin_geometry_3d_coordinates_bit_depth_minus1[j] plus 1 indicates the bit depth of the geometry coordinates of the reconstructed volume content of the atlas with atlas ID j. pin_geometry_3d_coordinates_bit_depth_minus1[j] shall be in the range of 0 to 31 (inclusive).

[0147] pin_attribute_count[j] indicates the number of attributes of a unique attribute type present in the packed video frame for the atlas with atlas ID j. pin_attribute_count[j] shall be in the range of 1 to 127 (inclusive).

[0148] pin_attribute_type_id[j][i] indicates the attribute type of attribute index i for the atlas with atlas ID j. Table 4 describes the list of supported attribute types.

[0149] pin_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth that the region containing the attribute with attribute index i should be converted to for the atlas with atlas ID j. pin_attribute_2d_bit_depth_minus1[j][i] should range from 0 to 31 (inclusive).

[0150] pin_attribute_msb_align_flag[j][i] indicates how the decoded region containing the attribute with attribute index i is to be converted to samples at the nominal attribute bit depth, as specified in Annex B, for the atlas with atlas ID j.

[0151] pin_attribute_map_absolute_coding_persistence_flag[j][i] equal to 1 indicates that the decoded region of the attribute map containing the attribute with index i (corresponding to the atlas with atlas ID j) is coded without using any form of map prediction. pin_attribute_map_absolute_coding_persistence_flag[j][i] equal to 0 indicates that the decoded region of the attribute map containing the attribute with index i (corresponding to the atlas with atlas ID j) shall use the same map prediction method as the geometry component of the atlas with atlas ID j. If pin_attribute_map_absolute_coding_persistence_flag[j][i] is not present, its value shall be inferred to be equal to 1.

[0152] The 3D array AttributeMapAbsoluteCodingEnabledFlag, which indicates whether a particular map of an attribute is to be encoded or decoded with or without prediction, is obtained as follows:

[0153] if(pin_attribute_map_absolute_coding_persistance_flag[j][i]==1){

[0154] for(k=0;k <vps_map_count_minus1[j];k++)

[0155] PackingAttributeMapAbsoluteCodingEnabledFlag[j][i][k]=1

[0156] }

[0157] else{

[0158] for(k=0;k <vps_map_count_minus1[j];k++)

[0159] PackingAttributeMapAbsoluteCodingEnabledFlag[j][i][k]=

[0160] vps_map_absolute_coding_enabled_flag[j][i]

[0161] }

[0162] pin_attribute_dimension_minus1[j][i] plus 1 indicates the total dimension (ie, number of channels) of the region containing the attribute with index i for the atlas with atlas ID j. pin_attribute_dimension_minus1[j][i] should range from 0 to 63 (inclusive).

[0163] pin_attribute_dimension_partitions_minus1[j][i] plus 1 indicates the number of partition groups that the attribute channels of the region containing the attribute with index i should be grouped into for the atlas with atlas ID j. pin_attribute_dimension_partitions_minus1[j][i] should range from 0 to 63 (inclusive).

[0164] pin_attribute_partition_channels_minus1[j][i][k] plus 1 indicates the number of channels allocated to the dimension partition group with index k for the region containing the attribute with index i for the atlas with atlas ID j. For all dimension partition groups, ai_attribute_partition_channels_minus1[j][i][k] should range from 0 to pin_attribute_dimension_minus1[j][i] (inclusive).

[0165] pin_regions_count_minus1[j] plus 1 indicates the number of regions packed into one video frame for the atlas with ID j. The range of pin_regions_count_minus1 shall be 0 to 255 (inclusive).

[0166] pin_region_tile_id[j][i] indicates the atlas tile ID of the atlas with ID j corresponding to the region with index i.

[0167] pin_region_type_id_minus2[j][i] plus 2 indicates the ID of the region with index i for the atlas with ID j. The value of pin_region_type_id_minus2[j][i] should range from 0 to 2 (inclusive).

[0168] pin_region_top_left_x[j][i] specifies the horizontal position (in units of luma samples) of the top left sample of the region with index i in the atlas with ID j in the packed video component frame.

[0169] pin_region_top_left_y[j][i] specifies the vertical position (in units of luma samples) of the top left sample of the region with index i in the atlas with ID j in the packed video component frame.

[0170] pin_region_width_minus1[j][i] plus 1 specifies the width (in luma samples) of the region with index i in the atlas with ID j.

[0171] pin_region_height_minus1[j][i] plus 1 specifies the height (in luma samples) of the region with index i in the atlas with ID j.

[0172] pin_region_unpack_top_left_x[j][i] specifies the horizontal position (in units of luma samples) of the top left sample of the region with index i in the atlas with ID j in the unpacked video component frame.

[0173] pin_region_unpack_top_left_y[j][i] specifies the vertical position (in units of luma samples) of the top left sample of the region with index i in the atlas with ID j in the unpacked video component frame.

[0174] pin_region_rotation_flag[j][i] equal to 0 indicates that no rotation is performed on the region with index i of the atlas with ID j. pin_region_rotation_flag[j][i] equal to 1 indicates that the region with index i of the atlas with ID j is rotated 90 degrees.

[0175] pin_region_map_index[j][i] specifies the map index of the region with index i in the atlas with ID j.

[0176] pin_region_auxiliary_data_flag[j][i] is equal to 1 to indicate that the region with index i of atlas with ID j contains only RAW and / or EOM codec points. pin_region_auxiliary_data_flag is equal to 0 to indicate that the region with index i of atlas with ID j may contain RAW and / or EOM codec points.

[0177] pin_region_attr_index[j][i] indicates the attribute index of the region with index i of the atlas with ID j. The value of pin_region_attr_index[j][i] should range from 0 to pin_attribute_count[j]–1 (inclusive).

[0178] pin_region_attr_partition_index[j][i] indicates the attribute partition index of the region with index i of the atlas with ID j. When not present, the value of pin_region_attr_partition_index[j][i] is inferred to be equal to 0.

[0179] 3. Technical Problems Solved by the Disclosed Technical Solution

[0180] In the example design of dynamic mesh encoding and decoding, the following problems exist:

[0181] First, texture data and displacement data are encoded and decoded into two separate bitstreams, which requires two encoders or decoders to support this implementation. Therefore, a single bitstream is more preferred.

[0182] Secondly, the displacement data is encoded and decoded in 4:4:4 or 4:0:0 format, which requires more encoding / decoding resources in HEVC.

[0183] Third, the encoding and decoding efficiency of texture data can be improved when the data is in the green-red-blue (GRB) / green-blue-red (GBR) / YCgCo-R domain, where Y represents the luminance component, Cg represents the green chrominance component, and Co represents the orange chrominance component.

[0184] Fourth, when using arithmetic coding to encode and decode the shifted data, the sub-block size is signaled as u(16), which has a range that is too wide and cannot be implemented using shifting.

[0185] Fifth, when encoding and decoding displacement data using arithmetic codec, it may be better to use different sub-block sizes for different color components or different types of frames.

[0186] Sixth, when using the packing_information() structure to jointly encode and decode texture and displacement, there should be some modifications / constraints.

[0187] 4. List of solutions and implementation examples

[0188] The detailed list below should be considered as examples to explain the general concept. These examples should not be interpreted narrowly. In addition, these examples can be combined in any way.

[0189] 1. To solve problem 1, texture data and displacement data are combined into one video for encoding / decoding.

[0190] a. In one example, the displacement data can be converted to 4:2:0 format and spliced ​​with the texture in 4:2:0 format.

[0191] b. In one example, the displacement data may be converted to N bits, ie, the N bits are the bit depth of the texture.

[0192] a. In one example, N is 10.

[0193] c. In one example, texture data and displacement data are encoded and decoded in different stripes.

[0194] d. In one example, the location and / or size of the texture region and / or displacement data region may be signaled in the bitstream.

[0195] e. In one example, the location and / or size of texture regions and / or displacement data regions may be inferred from the bitstream.

[0196] 2. To better solve problem 1, the texture part and displacement part encoded and decoded in a video can use different encoding strategies / methods.

[0197] a. In one example, textures may use one quantization parameter (QP), while displacement data may use a different QP.

[0198] b. In one example, all video units of the displacement data use lossless codecs.

[0199] a. In one example, all video units of displacement data apply the transquant_bypass mode in HEVC.

[0200] b. In one example, all video units of displacement data have transform skip mode applied and QP equal to 4+6*K.

[0201] i. In one example, K is 0.

[0202] 3. To better solve problem 1, the encoder can fill data into areas in the combined picture that are not texture or displacement data.

[0203] a. In one example, the area is filled with a fixed value.

[0204] a. In one example, the region is filled with intermediate pixel values, i.e., 1<<(bitdepth-1), e.g., 128 for 8-bit video and 512 for 10-bit video.

[0205] b. In one example, the region is filled with the value of the nearest pixel in the texture region or displacement data region.

[0206] c. In one example, N rows of luma samples and N / 2 rows of chroma samples are inserted between the texture data and the displacement data.

[0207] a. In one example, N is 16.

[0208] b. In one example, N is 0.

[0209] c. In one example, all interpolated samples have the same value, eg, the middle pixel value.

[0210] 4. To better solve problem 1, smoothing can be applied to the combined image.

[0211] a. In one example, only the filled areas may be smoothed.

[0212] 5. To solve problem 2, displacement data can be encoded and decoded in 4:2:0 format.

[0213] a. In one example, displacement data can be converted to 4:2:0 before encoding and converted back to 4:4:4 after decoding.

[0214] b. In one example, sps_chroma_format_idc and / or

[0215] ChromaFormatIdc is set to 1 to perform displacement data encoding and decoding.

[0216] c. In one example, in HEVC, the encoding and decoding of displacement data can use the main profile or

[0217] main10 grade.

[0218] d. In one example, when the displacement data has only one non-zero component, the displacement data may be packed into luma and chroma components in a 4:2:0 format.

[0219] 6. To better solve problem 2, it is possible to deduce at the decoder whether the displacement data has only one non-zero component or three non-zero components.

[0220] a. In one example, based on the displacement video resolution and the number of base grid points, it can be derived at the decoder whether the displacement data has only one non-zero component or three non-zero components.

[0221] b. In one example, based on the displacement video resolution and the number of vertices, it can be inferred at the decoder whether the displacement data has only one non-zero component or three non-zero components.

[0222] 7. To solve problem 3, it is proposed to convert texture data to a color space other than YUV or BGR before encoding, and convert it back after decoding.

[0223] a. In one example, texture data can be encoded and decoded in GBR color space.

[0224] b. In one example, texture data may be encoded and decoded in the GRB color space.

[0225] c. In one example, texture data may be encoded and decoded in the YCgCo color space.

[0226] d. In one example, texture data may be encoded and decoded in the YCoCg color space.

[0227] e. In the above sub-item, the color space can be used for lossless encoding and decoding of texture data.

[0228] 8. In order to better solve problem 3, the International Telecommunication Union - Telecommunication Standardization Sector (ITU-T) Recommendation H.273 proposes using a new primary color index to represent the GRB color space.

[0229] 9. To solve problem 4, it is proposed to represent the sub-block size as 2^S and transmit it through signal (SK) in the bit stream as ue(v) or u(n), where S is a non-negative integer, where ue(v) is an unsigned integer Exp-Golomb codec syntax element (ue(v)), and where u(n) is an unsigned integer using n bits (u(n)), and where K is a non-negative integer.

[0230] a. In one example, K is equal to 0.

[0231] b. In one example, K is equal to 6.

[0232] c. In one example, K is equal to 7.

[0233] d. In one example, SK should range from 0 to 3 (inclusive).

[0234] 10. To solve problem 5, the sub-block size (denoted as SB2) used for encoding and decoding the second and third color components can be different from the sub-block size (denoted as SB1) used for encoding and decoding the first color component.

[0235] a. In one example, SB2 is greater than SB1.

[0236] a. In one example, SB1 is equal to 100 and SB2 is equal to 200.

[0237] b. In one example, SB1 is equal to 128 and SB2 is equal to 256.

[0238] 11. To solve problem 4, alternatively, a different sub-block size may be used for inter-frame displacement data than for intra-frame displacement data.

[0239] 12. To address issue 6, when using a packed information scheme to jointly support texture and displacement codecs in dynamic mesh codecs, one or more of the following constraints or a combination thereof are proposed:

[0240] a. In one example, pin_occupancy_present_flag[j] should be equal to 0.

[0241] i. Alternatively, when pin_occupancy_present_flag[j] is not equal to 0, the dynamic grid decoder shall ignore the region corresponding to this flag.

[0242] b. In one example, pin_region_type_id_minus2[j][i] should not be equal to 0.

[0243] c. In one example, each area described in the packaging information should not overlap with each other.

[0244] 5. Examples

[0245] Example 1

[0246] Figure 3 An example of combining texture data and displacement data into one picture is shown. Assume that the size of the texture is texture width × texture height (WT × HT), and the size of the displacement data is displacement width × bit shift height (WD × HD), in units of luma samples. Figure 3 An exemplary combination of texture data and displacement data is shown. The texture is placed from (0,0) to (WT-1,HT-1) in the combined picture. After conversion to 4:2:0 format, the displacement will be placed from (0,floor(HT / N)*N) to (WD,floor(HT / N)*N+HT-1), where N is the length of the minimum decoding unit. The picture has a minimum size that covers both texture data and displacement data and can be directly processed by a standard-compliant codec system. The new strip starts at the upper left corner of the displacement data area in the combined picture. The rest are padding areas where the encoder fills in data according to some method.

[0247] 5.2. Example 2

[0248] Figure 4Another example of combining texture data and displacement data into a single picture is shown. Assume that the size of the texture is WT×HT, and the size of the displacement data is WD×HD, in units of luma samples. The figure below shows another exemplary combination of texture data and displacement data. The texture is placed from (0,0) to (WT-1, HT-1) in the combined picture. The displacement data is first converted to 4:2:0 format and then stretched into a rectangular area of ​​height H, where H is the length of the basic displacement block (currently H is equal to 16, in units of luma samples). The stretched displacement data will be placed from top to bottom and left to right starting from (0, floor(HT / N)*N) codec units (usually this value is 8). This picture has a minimum size that covers both texture and displacement data and can be directly processed by a standard-compliant codec system. The rest are padding areas where the encoder fills in data according to some method.

[0249] 5.3 Example 3

[0250] The following modifications are based on [7].

[0251] The most relevant sections that have been added or modified are in bold, while deleted sections are in bold italics. There may be some other changes that are editorial in nature and therefore not indicated. There may also be other changes that are not indicated in this article.

[0252] 8.4.2.2 V3C Unit Header Semantics

[0253] Table 2 - V3C unit types

[0254]

[0255] 6. References

[0256] [1]MPEG technical requirements, "CfP for.Dynamic Mesh Coding," ISO / IECJTC 1 / SC 29 / WG 2doc.no.N145, in Oct.2021.

[0257] [2]K.Mammou, J.Kim, A.Tourapis and D.Podborski, "[V-CG]Apple's DynamicMesh Coding CfP Response," ISO / IEC JTC 1 / SC 29 / WG 7doc.no.m59281, in Apr.2022.

[0258] [3]MPEG output document, “WD 1.0of V-DMC,” ISO / IEC JTC 1 / SC 29 / WG7doc.no.N0486, in Nov.2022.

[0259] [4] C. Huang,

[0260] [5] "WD 3.0 of V-DMC," ISO / IEC JTC 1 / SC 29 / WG 7doc.no.N611, in Apr.2023.

[0261] [6]ITU-T recommendation H.273: Coding-independent code points for video signal type identification, can be accessed via https: / / www.itu.int / rec / T-REC-H.273.

[0262] [7] "Text of ISO / IEC FDIS23090-5 2nd Edition Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC)," ISO / IEC JTC1 / SC 29 / WG 7doc.no.N553, in Jan.2023.

[0263] Figure 5 is a block diagram illustrating an example video processing system 4000 in which the various techniques disclosed herein may be implemented. Various implementations may include some or all of the components of system 4000. System 4000 may include an input 4002 for receiving video content. The video content may be received in a raw or uncompressed format (e.g., 8-bit or 10-bit multi-component pixel values), or may be received in a compressed or encoded format. Input 4002 may represent a network interface, a peripheral bus interface, or a storage interface. Examples of network interfaces include wired interfaces such as Ethernet, a passive optical network (PON), etc., and wireless interfaces such as wireless fidelity (Wi-Fi) or a cellular interface.

[0264] System 4000 may include a codec component 4004 that may implement various codecs or coding methods described in this disclosure. Codec component 4004 may reduce the average bit rate of the video from the input 4002 to the output of codec component 4004 to generate a codec representation of the video. Therefore, codec technology is sometimes referred to as video compression or video transcoding technology. The output of codec component 4004 may be stored or sent via a communication connection as represented by component 4006. The bitstream (or codec) representation of the stored or communicated video received at input 4002 may be used by component 4008 to generate pixel values ​​or displayable video, which is sent to display interface 4010. The process of generating a user-viewable video from a bitstream representation is sometimes referred to as video decompression. In addition, although some video processing operations are referred to as "codec" operations or tools, it should be understood that coding tools or operations are used in encoders, and decoders will perform corresponding decoding tools or operations that reverse the coding results.

[0265] Examples of peripheral bus interfaces or display interfaces may include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), or DisplayPort, etc. Examples of storage interfaces include Serial Advanced Technology Attachment (SATA), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, etc. The technology described in this disclosure may be embodied in various electronic devices such as mobile phones, laptop computers, smartphones, or other devices capable of performing digital data processing and / or video display.

[0266] Figure 6 4 is a block diagram of an example video processing device 4100. Device 4100 can be used to implement one or more methods described herein. Device 4100 can be embodied as a smartphone, a tablet, a computer, an Internet of Things (IoT) receiver, etc. Device 4100 may include one or more processors 4102, one or more memories 4104, and video processing circuitry 4106. Processor(s) 4102 can be configured to implement one or more methods described in this disclosure. Memory(s) 4104 can be used to store data and code for implementing the methods and techniques described herein. Video processing circuitry 4106 can be used to implement some of the techniques described in this disclosure in hardware circuitry. In some embodiments, video processing circuitry 4106 can be at least partially included in processor 4102, for example, a graphics co-processor.

[0267] Figure 74 is a flow chart of an example method 4200 for video processing. Method 4200 includes determining, at step 4202, that texture data and displacement data are combined into a set of video media data. At step 4204, converting between the visual media data and a bitstream based on the combined texture data and displacement data is performed. According to an example, the conversion of step 4204 may include encoding at an encoder or decoding at a decoder.

[0268] It should be noted that method 4200 can be implemented in an apparatus for processing video data, the apparatus including a processor and non-transitory memory having instructions thereon, such as video encoder 4400, video decoder 4500, and / or encoder 4600. In this case, the instructions, when executed by the processor, cause the processor to perform method 4200. Furthermore, method 4200 can be performed by a non-transitory computer-readable medium including a computer program product for use with a video codec device. The computer program product includes computer-executable instructions stored on a non-transitory computer-readable medium, such that when executed by the processor, the video codec device performs method 4200.

[0269] Figure 8 4 is a block diagram illustrating an example video codec system 4300 that can utilize the techniques of this disclosure. Video codec system 4300 can include a source device 4310 and a destination device 4320. Source device 4310 generates encoded video data, which can be referred to as a video encoding device. Destination device 4320 can decode the encoded video data generated by source device 4310, which can be referred to as a video decoding device.

[0270] Source device 4310 may include a video source 4312, a video encoder 4314, and an input / output (I / O) interface 4316. Video source 4312 may include a source such as a video capture device, an interface for receiving video data from a video content provider, and / or a computer graphics system for generating video data, or a combination of such sources. The video data may include one or more pictures. Video encoder 4314 encodes the video data from video source 4312 to generate a bitstream. The bitstream may include a series of bits that form a codec representation of the video data. The bitstream may include codec pictures and associated data. A codec picture is a codec representation of a picture. Associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I / O interface 4316 may include a modulator / demodulator (modem) and / or a transmitter. The encoded video data may be sent directly to target device 4320 via network 4330 via I / O interface 4316. The encoded video data may also be stored on storage media / server 4340 for access by target device 4320.

[0271] Target device 4320 may include an I / O interface 4326, a video decoder 4324, and a display device 4322. I / O interface 4326 may include a receiver and / or a modem. I / O interface 4326 may obtain encoded video data from source device 4310 or storage medium / server 4340. Video decoder 4324 may decode the encoded video data. Display device 4322 may display the decoded video data to a user. Display device 4322 may be integrated with target device 4320, or may be external to target device 4320, and the target device may be configured to interface with an external display device.

[0272] The video encoder 4314 and the video decoder 4324 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, the Versatile Video Coding (VVC) standard, and other current and / or additional standards.

[0273] Figure 9 is a block diagram illustrating an example of a video encoder 4400, which may be Figure 8 Video encoder 4314 in system 4300 is shown. Video encoder 4400 can be configured to perform any or all of the techniques of this disclosure. Video encoder 4400 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video encoder 4400. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0274] The functional components of the video encoder 4400 may include a segmentation unit 4401, a prediction unit 4402 (which may include a mode selection unit 4403, a motion estimation unit 4404, a motion compensation unit 4405, an intra-frame prediction unit 4406), a residual generation unit 4407, a transform processing unit 4408, a quantization unit 4409, an inverse quantization unit 4410, an inverse transform unit 4411, a reconstruction unit 4412, a cache 4413 and an entropy coding unit 4414.

[0275] In other examples, the video encoder 4400 may include more, fewer, or different functional components. In an example, the prediction unit 4402 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in IBC mode, where at least one reference picture is a picture in which the current video block is located.

[0276] Furthermore, some components, such as the motion estimation unit 4404 and the motion compensation unit 4405 , may be highly integrated but are represented separately in the example of the video encoder 4400 for explanatory purposes.

[0277] The segmentation unit 4401 may segment a picture into one or more video blocks. The video encoder 4400 and the video decoder 4500 may support various video block sizes.

[0278] The mode selection unit 4403 can select one of a plurality of coding modes based on the error result, for example, and provide the resulting intra-frame or inter-frame coded block to the residual generation unit 4407 to generate residual block data and provide it to the reconstruction unit 4412 to reconstruct the coded block for use as a reference picture. In some examples, the mode selection unit 4403 can select a joint intra-frame and inter-frame prediction (CIIP) mode, where the prediction is based on an inter-frame prediction signal and an intra-frame prediction signal. The mode selection unit 4403 can also select a resolution of motion vectors for the block in the case of inter-frame prediction (e.g., sub-pixel or integer pixel precision).

[0279] To perform inter-frame prediction on the current video block, the motion estimation unit 4404 may generate motion information for the current video block by comparing the current video block with one or more reference frames from the buffer 4413. The motion compensation unit 4405 may determine a predicted video block for the current video block based on the motion information and decoded samples of a picture from the buffer 4413 (other than the picture associated with the current video block).

[0280] The motion estimation unit 4404 and the motion compensation unit 4405 may perform different operations on the current video block, eg, depending on whether the current video block is in an I slice, a P slice, or a B slice.

[0281] In some examples, motion estimation unit 4404 may perform unidirectional prediction on the current video block, and motion estimation unit 4404 may search for a reference video block for the current video block in the reference pictures in list 0 or list 1. Motion estimation unit 4404 may then generate a reference index that indicates a reference picture in list 0 or list 1, the reference picture containing the reference video block and a motion vector that indicates a spatial displacement between the current video block and the reference video block. Motion estimation unit 4404 may output the reference index, the prediction direction indicator, and the motion vector as motion information for the current video block. Motion compensation unit 4405 may generate a predicted video block for the current block based on the reference video block indicated by the motion information for the current video block.

[0282] In other examples, the motion estimation unit 4404 may perform bidirectional prediction on the current video block. The motion estimation unit 4404 may search for a reference video block for the current video block in the reference pictures in list 0 and may also search for another reference video block for the current video block in the reference pictures in list 1. The motion estimation unit 4404 may then generate a reference index indicating a reference picture in list 0 and list 1, each of which contains a reference video block and a motion vector indicating a spatial displacement between the reference video block and the current video block. The motion estimation unit 4404 may output the reference index and motion vector for the current video block as motion information for the current video block. The motion compensation unit 4405 may generate a predicted video block for the current video block based on the reference video block indicated by the motion information for the current video block.

[0283] In some examples, motion estimation unit 4404 may output a complete set of motion information for use in the decoding process of the decoder. In some examples, motion estimation unit 4404 may not output a complete set of motion information for the current video. Rather, motion estimation unit 4404 may reference motion information of another video block to signal the motion information of the current video block. For example, motion estimation unit 4404 may determine that the motion information of the current video block is sufficiently similar to the motion information of the adjacent video block.

[0284] In one example, the motion estimation unit 4404 may indicate a value in a syntax structure associated with the current video block that indicates to the video decoder 4500 that the current video block has the same motion information as another video block.

[0285] In another example, the motion estimation unit 4404 can identify another video block and a motion vector difference (MVD) in a syntax structure associated with the current video block. The motion vector difference indicates the difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 4500 can use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.

[0286] As discussed above, the video encoder 4400 can predictively signal motion vectors.Two examples of predictive signaling techniques that can be implemented by the video encoder 4400 include Advanced Motion Vector Prediction (AMVP) and Merge mode signaling.

[0287] Intra-frame prediction unit 4406 can perform intra-frame prediction on the current video block. When intra-frame prediction unit 4406 performs intra-frame prediction on the current video block, intra-frame prediction unit 4406 can generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block can include the predicted video block and various syntax elements.

[0288] The residual generation unit 4407 can generate residual data for the current video block by subtracting the predicted video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks corresponding to different sample components of the samples in the current video block.

[0289] In other examples, such as in skip mode, for the current video block, there may be no residual data for the current video block, and the residual generation unit 4407 may not perform a subtraction operation.

[0290] Transform processing unit 4408 may generate a transform coefficient video block for a current video block by applying one or more transforms to the residual video block associated with the current video block.

[0291] After the transform processing unit 4408 generates a transform coefficient video block associated with the current video block, the quantization unit 4409 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values ​​associated with the current video block.

[0292] Inverse quantization unit 4410 and inverse transform unit 4411 may apply inverse quantization and inverse transform, respectively, to the transform coefficient video block to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 4412 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by prediction unit 4402 to produce a reconstructed video block associated with the current block for storage in buffer 4413.

[0293] After the reconstruction unit 4412 reconstructs the video block, a loop filtering operation may be performed to reduce video block artifacts in the video block.

[0294] The entropy coding unit 4414 may receive data from other functional components of the video encoder 4400. When the entropy coding unit 4414 receives data, the entropy coding unit 4414 may perform one or more entropy coding operations to generate entropy-coded data and output a bitstream including the entropy-coded data.

[0295] Figure 10 is a block diagram illustrating an example of a video decoder 4500, which may be Figure 8 Video decoder 4324 in system 4300 is shown. Video decoder 4500 can be configured to perform any or all of the techniques of this disclosure. In the example shown, video decoder 4500 includes multiple functional components. The techniques described in this disclosure can be shared between the various components of video decoder 4500. In some examples, a processor can be configured to perform any or all of the techniques described in this disclosure.

[0296] In the example shown, the video decoder 4500 includes an entropy decoding unit 4501, a motion compensation unit 4502, an intra-frame prediction unit 4503, an inverse quantization unit 4504, an inverse transform unit 4505, a reconstruction unit 4506, and a buffer 4507. In some examples, the video decoder 4500 can perform a decoding process (pass) that is generally the inverse of the encoding process described with respect to the video encoder 4400.

[0297] The entropy decoding unit 4501 can retrieve the encoded bitstream. The encoded bitstream may include entropy-encoded video data (e.g., encoded video data blocks). The entropy decoding unit 4501 can decode the entropy-encoded video data, and the motion compensation unit 4502 can determine motion information based on the entropy-encoded video data, including motion vectors, motion vector precision, reference picture list index, and other motion information. The motion compensation unit 4502 can determine such information, for example, by implementing AMVP and Merge modes.

[0298] The motion compensation unit 4502 may generate motion compensated blocks, possibly performing interpolation based on interpolation filters. Identification of the interpolation filters used with sub-pixel precision may be included in a syntax element.

[0299] The motion compensation unit 4502 may calculate interpolated values ​​of sub-integer pixels of a reference block using interpolation filters used by the video encoder 4400 during encoding of the video block. The motion compensation unit 4502 may determine the interpolation filters used by the video encoder 4400 based on received syntax information and use the interpolation filters to generate a prediction block.

[0300] The motion compensation unit 4502 can use some syntax information to determine the size of the blocks used to encode (one or more) frames and / or (one or more) slices of the encoded video sequence, partitioning information describing how each macroblock of the pictures of the encoded video sequence is partitioned, a mode indicating how to encode each partition, one or more reference frames (and reference frame lists) for each inter-frame codec block, and other information used to decode the encoded video sequence.

[0301] The intra-frame prediction unit 4503 can form a prediction block from spatially adjacent blocks using, for example, an intra-frame prediction mode received in the bitstream. The inverse quantization unit 4504 inversely quantizes, i.e., inversely quantizes, the quantized video block coefficients provided in the bitstream and decoded by the entropy decoding unit 4501. The inverse transform unit 4505 applies an inverse transform.

[0302] The reconstruction unit 4506 can add the residual block to the corresponding prediction block generated by the motion compensation unit 4502 or the intra-frame prediction unit 4503 to form a decoded block. If necessary, a deblocking filter can also be applied to the decoded block to remove blocking artifacts. The decoded video block is then stored in a buffer 4507, which provides reference blocks for subsequent motion compensation / intra-frame prediction and also produces decoded video for presentation on a display device.

[0303] Figure 11 is a schematic diagram of an example encoder 4600. The encoder 4600 is suitable for implementing VVC technology. The encoder 4600 includes three loop filters, namely, a deblocking filter (DF) 4602, a sample adaptive offset (SAO) 4604, and an adaptive loop filter (ALF) 4606. Unlike the DF 4602, which uses a predefined filter, the SAO 4604 and the ALF 4606 use the original samples of the current picture to reduce the mean square error between the original samples and the reconstructed samples by adding an offset and applying a finite impulse response (FIR) filter, respectively, and using the codec's side information to signal the offset and filter coefficients. The ALF 4606 is located at the last processing stage for each picture and can be seen as a tool that attempts to capture and repair artifacts caused by previous stages.

[0304] The encoder 4600 also includes an intra-frame prediction component 4608 and a motion estimation / compensation (ME / MC) component 4610 configured to receive input video. The intra-frame prediction component 4608 is configured to perform intra-frame prediction, while the ME / MC component 4610 is configured to perform inter-frame prediction using reference pictures retrieved from a reference picture cache 4612. The residual block from the inter-frame prediction or intra-frame prediction is fed into a transform (T) component 4614 and a quantization (Q) component 4616 to generate quantized residual transform coefficients, which are fed into an entropy codec component 4618. The entropy codec component 4618 performs entropy coding and decoding on the prediction results and the quantized transform coefficients and sends the entropy-coded quantized transform coefficients to a video decoder (not shown). The quantized components output from the quantization component 4616 can be fed into an inverse quantization (IQ) component 4620, an inverse transform component 4622, and a reconstruction (REC) component 4624. The REC component 4624 can output images to the DF 4602 , SAO 4604 , and ALF 4606 for filtering before these pictures are stored in the reference picture cache 4612 .

[0305] Figure 12Flowchart 5200 is an example method of video processing. In one embodiment, the method is performed by an encoder (e.g., an encoding device) or by a decoder (e.g., a decoding device). In block 5202, the method includes determining that the displacement data is encoded or decoded in a 4:2:0 format. In block 5204, the method includes performing conversion between the visual media data and a bitstream based on the displacement data encoded or decoded in a 4:2:0 format. According to an example, the conversion of step 5204 may include encoding at the encoder or decoding at the decoder.

[0306] A list of solutions preferred by some examples is provided next.

[0307] The following solutions illustrate examples of the techniques discussed herein.

[0308] 1. A method for processing video data, comprising: determining that texture data and displacement data are combined into a set of video media data; and performing conversion between the visual media data and the bitstream based on the combined texture data and displacement data.

[0309] 2. The method of solution 1, wherein the displacement data is converted to a 4:2:0 format and concatenated with the texture in the 4:2:0 format.

[0310] 3. The method of any one of solutions 1 to 2, wherein the displacement data is converted to equal N bits, the N bits being the bit depth of the texture.

[0311] 4. The method as described in any one of solutions 1 to 3, wherein the bit depth of N bits is 10 bits.

[0312] 5. The method according to any one of solutions 1 to 4, wherein texture data and displacement data are encoded and decoded in different slices.

[0313] 6. A method as described in any of the solutions 1 to 5, wherein the position of the texture area, the size of the texture area or the displacement data area is transmitted by signal in the bitstream.

[0314] 7. A method as described in any of solutions 1 to 6, wherein the position of the texture area, the size of the texture area or the displacement data area is derived from the bitstream.

[0315] 8. The method according to any one of solutions 1 to 7, wherein the texture data and the displacement data are encoded and decoded using different encoding and decoding mechanisms.

[0316] 9. The method according to any one of solutions 1 to 8, wherein the texture data and the displacement data are encoded and decoded using different quantization parameters (QPs).

[0317] 10. The method according to any one of solutions 1 to 9, wherein the displacement data is encoded and decoded using a lossless codec.

[0318] 11. The method according to any one of solutions 1 to 10, wherein the displacement data is encoded and decoded using a transquant_bypass mode.

[0319] 12. The method of any of solutions 1 to 11, wherein the displacement data is encoded using transform skip mode and a QP equal to 4+6*K, where k is equal to zero.

[0320] 13. The method of any one of solutions 1 to 12, wherein the area of ​​the picture including the texture data and the displacement data is filled.

[0321] 14. A method as described in any of solutions 1 to 13, wherein the area is filled with a fixed value.

[0322] 15. A method as described in any of solutions 1 to 14, wherein the area is filled with intermediate pixel values.

[0323] 16. A method as described in any of solutions 1 to 15, wherein the area is filled with the closest pixel value.

[0324] 17. The method according to any one of solutions 1 to 16, wherein smoothing is applied to the image comprising the texture data and the displacement data.

[0325] 18. The method of any one of solutions 1 to 17, wherein the smoothing process is applied only to a filled area in the picture including the texture data and the displacement data.

[0326] 19. The method according to any one of solutions 1 to 18, wherein the displacement data is encoded and decoded in a 4:2:0 format.

[0327] 20. A method as described in any one of solutions 1 to 19, wherein when the conversion includes encoding, the displacement data is converted from 4:4:4 to 4:2:0, and when the conversion includes decoding, the displacement data is converted from 4:2:0 back to 4:4:4.

[0328] 21. The method according to any one of solutions 1 to 20, wherein for displacement data encoding and decoding, sps_chroma_format_idc or ChromaFormatIdc is set to 1.

[0329] 22. A method as described in any one of solutions 1 to 21, wherein the displacement data is encoded and decoded using main profile or main10 profile.

[0330] 23. An apparatus for processing video data, comprising: a processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of solutions 1 to 22.

[0331] 24. A non-transitory computer-readable medium comprising a computer program product for use by a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, so that when executed by a processor, the video codec device performs a method as described in any one of Solutions 1 to 22.

[0332] 25. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by a video processing apparatus, wherein the method comprises: determining that texture data and displacement data are combined into a set of video media data; and generating the bitstream based on the determination.

[0333] 26. A method for storing a bitstream of a video, comprising: determining that texture data and displacement data are combined into a set of video media data; generating the bitstream based on the determination; and storing the bitstream in a non-transitory computer-readable recording medium.

[0334] 27. A method, apparatus or system as described in this patent document.

[0335] In the solution described herein, an encoder can conform to the format rules by generating a codec representation according to the format rules. In the solution described herein, a decoder can use the format rules to parse syntax elements in the codec representation and, knowing the presence or absence of syntax elements according to the format rules, produce decoded video.

[0336] In this disclosure, the term "video processing" may refer to video encoding, video decoding, video compression, or video decompression. For example, a video compression algorithm may be applied during conversion from a pixel representation of a video to a corresponding bitstream representation (or vice versa). For example, the bitstream representation of a current video block may correspond to bits that are co-located or distributed at different locations in the bitstream, as defined by the syntax. For example, a macroblock may be encoded based on error residual values ​​from a transform and a codec, and also encoded using bits in a header and other fields in the bitstream. In addition, during conversion, a decoder may parse the bitstream based on a determination that certain fields may or may not be present, as described in the above solution. Similarly, an encoder may determine whether to include certain syntax fields and generate the codec representation accordingly by including or excluding the syntax fields in the codec representation.

[0337] The disclosed and other solutions, examples, embodiments, modules, and functional operations described in this disclosure can be implemented in digital electronic circuitry or computer software, firmware, or hardware, or a combination of one or more thereof, including the structures disclosed in this disclosure and their equivalents. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions, encoded on a computer-readable medium for execution by a data processing apparatus or to control the operation of the data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that effects a machine-readable propagated signal, or a combination of one or more thereof. The term "data processing apparatus" encompasses all apparatus, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver device.

[0338] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or any other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files that store portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers, located at one site or distributed across multiple sites and interconnected by a communications network.

[0339] The processes or logic flows described in this disclosure may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and devices may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).

[0340] By way of example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to one or more mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and compact disk read-only memory (CD ROM) and digital versatile disk read-only memory (DVD-ROM) disks. The processor and memory may be supplemented by, or incorporated in, special-purpose logic circuitry.

[0341] Although this disclosure contains many details, these details should not be interpreted as limitations on the scope of any subject matter or what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular technology. Certain features described in the context of separate embodiments in this disclosure may also be implemented in combination in a single embodiment. Conversely, individual features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any appropriate subcombination. Furthermore, although features may function in certain combinations as described above, and may even be initially claimed in this manner, in some cases, one or more features in a claimed combination may be omitted from that combination, and a claimed combination may be directed to a subcombination or variant of a subcombination.

[0342] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed sequentially in the particular order or sequence shown, or that all illustrated operations be performed to achieve desired results. Furthermore, the partitioning of various system components in the embodiments described in this disclosure should not be understood as requiring such partitioning in all embodiments.

[0343] Only a few implementations and examples are described, and other implementations, improvements, and variations can be made based on what is described and illustrated in this disclosure.

[0344] A first component is directly coupled to a second component when there are no intervening components other than a line, trace, or other medium between the first and second components. A first component is indirectly coupled to a second component when there are intervening components other than a line, trace, or other medium between the first and second components. The term "coupled" and its variations encompass both direct and indirect couplings. The use of the term "about" is intended to encompass a range of ±10% of the subsequent figure unless otherwise indicated.

[0345] Although several embodiments are provided in this disclosure, it should be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated into another system, or certain features may be omitted or not implemented.

[0346] In addition, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques, or methods without departing from the scope of the present disclosure. Other items shown or discussed as coupled may be directly connected, or may be indirectly coupled or communicate through some interface, device, or intermediate component, whether electrically, mechanically, or otherwise. Other examples of variations, substitutions, and alterations are ascertainable to those skilled in the art and may be made without departing from the spirit and scope of the present disclosure.

Claims

1. A method for processing visual media data including displacement data, comprising: Determining that the displacement data is encoded and decoded in a 4:2:0 format; as well as Conversion between the visual media data and a bit stream is performed based on the displacement data encoded and decoded in a 4:2:0 format.

2. The method of claim 1, wherein the displacement data is converted to a 4:2:0 format before encoding and converted to a 4:4:4 format after decoding. 3 . The method of claim 1 , wherein one or more of the sps_chroma_format_idc syntax element and the ChromaFormatIdc variable are set to 1 to encode and decode the displacement data.

4. The method according to claim 1, wherein the encoding and decoding of the displacement data uses the main profile or the main10 profile.

5. The method of claim 1, wherein when the displacement data has only one non-zero component, the displacement data is packed into the luma component and the chroma component of the 4:2:0 format.

6. The method of claim 1, wherein the sub-block size of the sub-block of the visual media data is 2 S Represented by , where S is a non-negative integer.

7. The method of claim 1, wherein the bitstream comprises a value equal to SK, where K is a non-negative integer.

8. The method of claim 1, wherein the values ​​in the bitstream are represented as unsigned integer Exp-Golomb codec syntax elements (ue(v)) or as unsigned integers using n bits (u(n)).

9. The method of any one of claims 7 to 8, wherein K is equal to 0.

10. The method of any one of claims 7 to 8, wherein K is equal to 6.

11. The method of any one of claims 7 to 8, wherein K is equal to 7.

12. The method of any one of claims 7 to 8, wherein the value of SK ranges from 0 to 3, inclusive.

13. The method of any one of claims 1 to 12, wherein the visual media data comprises the displacement data and texture data, and wherein the texture data and the displacement data are included in a single bitstream.

14. The method according to any one of claims 1 to 13, further comprising converting the displacement data into a 4:2:0 format, and splicing the converted displacement data with the texture data in the 4:2:0 format.

15. The method of any one of claims 1 to 13, further comprising converting the displacement data to N bits, wherein the N bits are the bit depth of the texture data, and wherein N is an integer. The method of claim 15 , wherein N is 10.

17. The method of any one of claims 1 to 16, wherein the texture data and the displacement data are encoded and decoded in different slices.

18. The method of any one of claims 1 to 17, wherein one or more of the location and size of the texture data is included in the single bitstream.

19. A method as claimed in any one of claims 1 to 17, wherein one or more of the position and magnitude of the displacement data is included in the single bitstream.

20. The method of any one of claims 1 to 17, wherein one or more of the location and size of the texture data is inferred based on information in the single bitstream.

21. A method as claimed in any one of claims 1 to 17, wherein one or more of the position and size of the displacement data is inferred based on information in the single bitstream.

22. The method of any one of claims 1 to 21, wherein the texture data and the displacement data are included in a single bitstream and use different encoding and decoding methods.

23. The method of claim 22, wherein the different coding methods include a first quantization parameter and a second quantization parameter different from the first quantization parameter, and wherein the texture data uses the first quantization parameter and the displacement data uses the second quantization parameter.

24. The method of any one of claims 22 to 23, wherein the different codec methods include a lossless codec, and wherein all video units of the displacement data use the lossless codec.

25. The method of any one of claims 22 to 24, wherein the different codec method comprises a transform quantization bypass mode from the High Efficiency Video Coding (HEVC) standard, and wherein all video units of the displacement data use the transform quantization bypass mode.

26. The method of any one of claims 22 to 25, wherein the different codec methods include a transform skip mode and a quantization parameter, and wherein all video units of the displacement data use the transform skip mode and the quantization parameter.

27. The method of claim 26, wherein the quantization parameter is equal to 4+6*K, where K is an integer.

28. The method of claim 27, wherein the value of K is zero.

29. The method of any one of claims 1 to 28, further comprising padding pictures in the single bitstream with data other than the texture data and the displacement data.

30. The method of claim 29, wherein the picture is padded with a fixed value.

31. The method of any one of claims 29 to 30, wherein the picture is padded with intermediate pixel values, and wherein for 8-bit video, the intermediate pixel values ​​are 128, and for 10-bit video, the intermediate pixel values ​​are 512.

32. The method of claim 29, wherein the picture is filled with the value of the nearest pixel in the texture data or with the value of the nearest pixel in the displacement data.

33. The method of claim 29, further comprising inserting N rows of luma samples and N / 2 rows of chroma samples between the texture data and the displacement data when padding the picture, where N is an integer.

34. The method of claim 33, wherein the value of N is 16.

35. The method of claim 33, wherein the value of N is 0.

36. The method of claim 33, wherein all of the N rows of luma samples and the N / 2 rows of chroma samples have the same value.

37. The method of claim 36, wherein the same value comprises an intermediate pixel value.

38. The method of any one of claims 1 to 37, further comprising applying smoothing to pictures in the single bitstream.

39. The method of claim 1, further comprising deriving at a decoder whether the displacement data has only one non-zero component or three non-zero components.

40. The method of claim 1, further comprising determining, at a decoder, whether the displacement data has only one non-zero component or three non-zero components based on a video resolution of the displacement data and a number of base grid points.

41. The method of claim 1, further comprising determining, at a decoder, whether the displacement data has only one non-zero component or three non-zero components based on a video resolution of the displacement data and a number of vertices.

42. The method of claim 1 , further comprising converting the texture data to a color space before encoding and converting the texture data back to the color space after decoding, wherein the color space is not a blue-green-red (BGR) color space or a YUV color space where Y represents a luma component and U and V represent a blue chroma component and a red chroma component.

43. The method of claim 42, wherein the texture data in the bitstream is encoded in a green-blue-red (GBR) color space.

44. The method of claim 42, wherein the texture data in the bitstream is encoded in a Green-Red-Blue (GRB) color space.

45. The method of claim 42, wherein the texture data in the bitstream is encoded in a YCgCo color space, where Y represents a luma component, Cg represents a green chroma component, and Co represents an orange chroma component.

46. ​​The method of claim 42, wherein the texture data in the bitstream is encoded in a YCoCg color space, where Y represents a luma component, Co represents an orange chroma component, and Cg represents a green chroma component.

47. The method of claim 42, wherein the color space is used for lossless encoding and decoding of the texture data.

48. The method of claim 1, further comprising representing indices of primary colors of a Green-Red-Blue (GRB) color space for encoding and decoding according to an International Telecommunication Union-Telecommunication Standardization Sector (ITU-T) standard.

49. The method of claim 1, wherein a first sub-block size (SB1) used to encode a first color component of the visual media data is different from a second sub-block size (SB2) used to encode a second color component of the visual media data.

50. The method of claim 49, wherein the second sub-block size is larger than the first sub-block size.

51. The method of claim 50, wherein the first sub-block size is 100 and the second sub-block size is 200.

52. The method of claim 50, wherein the first sub-block size is 128 and the second sub-block size is 256.

53. The method of claim 1, wherein the displacement data comprises inter-frame displacement data and intra-frame displacement data, and wherein the inter-frame displacement data uses a different block size than the intra-frame displacement data.

54. The method of claim 1, wherein the pin occupancy presence flag is equal to 0 when a packed information scheme is used to support joint encoding and decoding of displacement data and texture data in dynamic mesh encoding and decoding.

55. The method of claim 54, wherein when the pin occupation existence flag is not equal to 0, the dynamic grid decoder ignores the area of ​​the visual media data corresponding to the pin occupation existence flag.

56. The method of any one of claims 54 to 55, wherein the pin occupancy present flag is designated as pin_occupancy_present_flag[j].

57. The method of claim 1, wherein a pin region type syntax element corresponding to the visual media data is not zero.

58. The method of claim 57, wherein the pin region type syntax element is specified as pin_region_type_id_minus2[j][i].

59. The method of claim 54, wherein no region in the packing information scheme overlaps with another region in the packing information scheme.

60. The method of any one of claims 1 to 59, wherein the converting comprises encoding the media data into a bitstream.

61. The method of any one of claims 1 to 59, wherein the converting comprises decoding the media data from a bitstream.

62. An apparatus for processing media data, comprising: processor; and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method of any one of claims 1 to 61.

63. A non-transitory computer-readable medium comprising a computer program product for use with a video codec device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium, such that when executed by a processor, the video codec device performs the method of any one of claims 1 to 61.

64. A non-transitory computer-readable recording medium storing a bit stream of a video generated by a method performed by a video processing apparatus, wherein the method comprises the method according to any one of claims 1 to 61.

65. A method for storing a bitstream of a video, comprising the method of any one of claims 1 to 61.

66. A method, apparatus, or system as described in this patent document.

Images