3D video decoding device and 3D video encoding device
By introducing a neural network post-filter into the 3D data decoding device and processing it for V3C video streams, the problem of insufficient interoperability of video streams in the prior art is solved, and the quality and consistency of video streams are improved.
Patent Information
- Application Number
- CN202410255220.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-01-31
- Filing Date
- 2024-03-06
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing neural network post-filters lack sufficient program restrictions when applied to V3C occupancy video streams, geometric video streams, and attribute video streams, and cannot guarantee interoperability between encoder devices, decoder devices and bitstreams. , resulting in inconsistent video streaming quality.
By introducing neural network post-filters into 3D data decoding devices, processing of occupancy video streams, geometric video streams and attribute video streams, achieving interoperability, and ensuring consistency of video streams through specific syntax tables and activation mechanisms .
Improve the quality of occupancy video streams, geometric video streams and attribute video streams, and realize interoperability between devices and consistent output of video streams.
Smart Images

Figure CN118945372A_ABST
Abstract
Description
Related Applications
[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 465,833, filed on May 11, 2023, which is hereby incorporated by reference in its entirety. Technical Field
[0002] Embodiments of the present invention relate to a 3D video decoding apparatus and a 3D video encoding apparatus. Background Art
[0003] Video encoding devices that generate encoded data by encoding a video and video decoding devices that generate decoded video by decoding the encoded data are used for efficient transmission or recording of videos. Video encoding schemes (video codecs) include H.264 / AVC, High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC), etc.
[0004] In such a video coding scheme, images (pictures) constituting a video are managed in a hierarchical structure including slices obtained by dividing an image, coding tree units (CTUs) obtained by dividing a slice, coding units (coding units; referred to as CUs) obtained by dividing a coding tree unit, and transform units (TUs) obtained by dividing a coding unit, and encoding / decoding is performed for each CU. In addition, there is a neural network post-filtering technique for filtering 2D video using supplemental enhancement information (SEI).
[0005] In order to effectively transmit or record 3D data, a 3D data encoding device is required to convert the 3D data into 2D images and encode them using a video encoding scheme to generate encoded data; a 3D data decoding device is also required to decode and reconstruct the 2D images in the encoded data and generate 3D data.
[0006] For example, specific 3D data coding schemes include 'ISO / IEC 23090-5 V3C (Visual Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression). V3C is a 3D data coding method that uses a video codec as the basis of a 3D image. V3C is used to encode and decode a point cloud consisting of point positions and attribute information. In addition, ISO / IEC23090-12 (MPEG Immersive Video, MIV) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding, V-DMC) currently being standardized are used to encode and decode multi-view videos and mesh videos. Non-patent document 1 (VSEI) discloses a method for applying a neural network post-filter to an encoded video stream. Non-patent document 2 discloses the specific application of VSEI in V3C applications. Citation List Non-patent literature
[0007] NPL 1: "Improvements under consideration for neural network post filter SEI messages" by Sean McCarthy, Sachin Deshpande, Miska M. Hannuksela, Hendry, Gary J. Sullivan, Ye-Kui Wang, JVET-AC2032, January 2023.
[0008] NPL 2: G. Martin-Cocher, “V3C–on SEI messages and Rec profiles”, ISO / IEC JTC1 SC29 WG7, m63263, April 2023. Summary of the invention Technical issues
[0009] The NPL1 method applies a neural network post filter (NNPF) to a coded video stream, and NPL2 shows the application of NNPF to V3C applications. However, they both lack sufficient procedures or restrictions to ensure interoperability between various encoder devices, decoder devices, and bitstreams when applying NNPF to V3C occupancy video streams, geometry video streams, and attribute video streams. Problem Solution
[0010] The present invention aims to perform neural network post-filtering processing on V3C occupancy video stream, geometry video stream and attribute video stream to achieve interoperability. Advantageous Effects of the Invention
[0011] According to an aspect of the present invention, the quality of an occupancy video stream, a geometry video stream, and an attribute video stream may be improved while achieving consistency. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] Figure 1 : Schematic diagram of the 3D data transmission system. Figure 2 : The hierarchical structure of the data in the encoded stream. Figure 3 : 3D data, occupancy frame, geometry frame and attribute frame. Figure 4 : A 3D data decoding device 31 used in an embodiment of the present invention. Figure 5 : Relationship between V3C layer and video layer Figure 6 : Syntax table of neural network post-filter characteristics SEI Figure 7 : Syntax table of neural network post-filter activation SEI Figure 8 : Syntax table of V3C neural network post-filter information SEI Fig. 9 : Syntax table of V3C neural network post-filter information SEI Fig.10 : A 3D data encoding device 11 used in an embodiment of the present invention. DETAILED DESCRIPTION First embodiment
[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.
[0014] The following is a description of embodiments of the present invention with reference to the accompanying drawings.
[0015] Figure 1 3D data transmission system 1 is a schematic diagram of a 3D data transmission system 1. 3D data transmission system 1 is a system that encodes 3D data into an encoded stream, transmits the encoded stream, decodes the transmitted encoded stream, and displays 3D data. 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31, and a 3D data display device 41. 3D data T is input to the 3D data encoding device 11. Network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. Network 21 is the Internet, a wide area network (WAN: Wide Area Network), a small network (LAN: Local Area Network), or a combination thereof. Network 21 is not necessarily a two-way communication network, but may be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting, satellite broadcasting, etc. Network 21 may also be replaced by a storage medium such as DVD (Digital Versatile Disc: Registered Trademark) and BD (Blu-ray Disc: Registered Trademark) that records the encoded stream Te.
[0016] The 3D data (3D information) in this paper is a collection of position (x, y, z) and attribute (e.g., r, g, b) information in 3D space. For example, 3D data is represented in the form of a point cloud, which is a collection of points of position and attribute information in 3D space, or a collection of meshes (or polygons) composed of triangular vertices and faces, where the vertices include a collection of positions (x, y, z) and the faces have attribute images, which can be represented by UV atlas methods.
[0017] The 3D data decoding device 31 decodes each encoded stream Te transmitted from the network 21 and generates one or more decoded 3D data Td.
[0018] The 3D data display device 41 displays all or part of one or more decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 is equipped with a display device such as a liquid crystal display, an organic EL (electroluminescence) display, etc. The form of the display may be fixed, mobile, HMD, etc. If the 3D data decoding device 31 has high processing power, an image with high image quality is displayed, and if it only has low processing power, an image that does not require high processing power and display ability is displayed. Operator
[0019] The operators used in this article are listed below. >> is a right shift, << is a left shift, & is a bitwise AND, | is a bitwise OR, |= is an or assignment operator, and || represents a logical OR. x?y:z is a ternary operator that takes y if x is true (non-zero) and takes z if x is false (zero) y..z represents the set of integers from y to z. Abs(x) is a function that returns x (when x ≥ 0) or -x (when x < 0). Floor(x) is a function that returns the largest integer less than or equal to x. Clip3(x,y,z) is a function that returns x (when z < x) or y (when z > y) or z (otherwise). Round(x) is a function that returns Sign(x)*Floor(Abs(x)+0.5) The structure of the encoded stream Te
[0020] The data structure of the encoded stream Te generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31 is described below.
[0021] Figure 2 Shows the hierarchical structure of the data in the encoded stream Te. The encoded stream Te has the data structure of either a V3C sample stream or a V3C unit stream. The V3C sample stream includes a sample stream header and V3C units, while the V3C unit stream includes V3C units.
[0022] The V3C unit contains a V3C unit header and a V3C unit payload. The unit type of the V3C unit header can be any one of V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, V3C_OVD, etc. The V3C unit payload with a specific unit type is also called V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, V3C_OVD.
[0023] If the unit type is V3C_VPS (Video Parameter Set), the V3C unit payload contains the V3C parameter set.
[0024] If the unit type is V3C_AD (atlas data), the V3C unit payload includes VPS ID, atlasID, sample stream nal header and multiple NAL units. ID stands for Idenification and is an integer value greater than or equal to 0. atlasID can be used as an element of the applicable SEI.
[0025] A NAL unit contains NALUnitType, layerID, temporalID, and RBSP (Raw Byte Sequence Payload).
[0026] The NAL unit is identified by NALUnitType and includes ASPS (map sequence parameter set), AAPS (map adaptation parameter set), ATL (map tile layer), AFPS (map frame parameter set), SEI (supplemental enhancement information).
[0027] ATL includes an ATL header and an ATL data unit. The ATL data unit contains information such as the location and size of a block, including block information data.
[0028] SEI includes payloadType indicating the type of SEI, payloadSize indicating the size (in bytes) of SEI, and sei_payload of SEI data.
[0029] If UnitType is V3C_AVD (attribute video data, attribute data), VPS ID, atlasID, attrIdx (attribute image ID), partIdx (partition ID), mapIdx (mapping ID), auxFlag, and video sub-bitstream (attribute video stream, attribute sub-bitstream). auxFlag indicates whether the data is auxiliary data. Video sub-bitstream is video data encoded using AVC, HEVC, VVC, etc.
[0030] If UnitType is V3C_GVD (Geometry Video Data, Geometry Data), it includes VPS ID, atlasID, mapIdx, auxFlag, and video sub-bitstream (Geometry Video Stream, Geometry Sub-Bitstream). mapIdx indicates a specific depth to be used for projection.
[0031] If UnitType is V3C_OVD (Occupied Video Data, Occupied Data), it includes VPS ID, atlasID and video sub-bitstream (Occupied Video Stream, Occupied Sub-Bitstream).
[0032] Figure 3 3D data, an occupancy frame (a 2D image representing occupancy information), a geometry frame (a 2D image representing depth or position information), and an attribute frame (a 2D image representing attribute information) are shown. The point cloud and mesh constituting the 3D data are divided into a plurality of parts (regions) by the 3D data encoding device 11, and the point cloud contained in each part is represented in the form of a 3D bounding box. The 3D data encoding device 11 generates a plurality of blocks from the projected point cloud, including information about the 3D bounding box (coordinates, size, etc.), and information about the mapping to the projection plane (the projection plane of each block, including coordinates, size, and rotation) is referred to as atlas information. The occupancy frame is a 2D binary frame / image (for example, 1 for a valid area and 0 for an invalid area) showing the valid area (area where a valid point exists) in each block ( Figure 3 (b)). Values other than 0 and 1, such as 255 and 0, can be used for valid and invalid areas. The geometry frame is a 2D image that shows the depth value (distance) of each block to the projection plane ( Figure 3 The relationship between depth value and pixel value can be linear, or the distance can be derived from the pixel value through a lookup table, a mathematical formula, or a relational expression based on a combination of value-based branches. The attribute frame is a 2D image ( Figure 3 (d-0)(d-1)). Projections in the same direction may have different depths for object surfaces (e.g., far and near surfaces). mapIdx indicates the specific depth used for projection. Geometry and attributes may be projected to multiple different specific depths. Figure 3 In FIG. 2 , two mapping situations of the geometry frame and the attribute frame are shown.
[0033] These occupancy frames, geometry frames, attribute frames and atlas information may contain (packed) partial images (blocks) from different projection planes onto a specific 2D image. Figure 3 In the occupancy frame, the geometry frame, and the attribute frame, three blocks (block 0, block 1, and block 2) are included. The atlas information includes information about the number of blocks and the projection planes corresponding to the blocks. The 3D data decoding device 31 reconstructs the coordinates and attribute information of the point cloud or mesh according to the atlas information, the occupancy frame, the geometry frame, and the attribute frame. In this article, a point is each point in the point cloud or the vertex of the mesh. The mesh information can be divided into a basic mesh and a mesh displacement (a basic mesh is a subset of a mesh and constitutes a basic mesh, and a mesh displacement indicates a displacement from a basic mesh to indicate a mesh other than a basic mesh) and then is encoded, decoded, and sent.
[0034] Figure 5 The relationship between the V3C layer and the video layer is shown. The 3D data decoding device 31 decodes the V3C unit specified in the V3C layer. Figure 2As shown, the V3C unit consists of a V3C unit header and a V3C unit payload. The V3C unit payload may consist of a graph sub-bitstream or a video sub-bitstream. Video_sub_bitstream(numBytes) contains a portion of a video unit stream (e.g., AVC, HEVC, or VVC, etc.) of size numBytes as an ordered stream of bytes or bits, in which the position of the unit boundary can be identified based on a pattern in the data. The codec of this video sub-bitstream is identified by a codec group or by a four-character code (4CC code) specified by the ptl_profile_codec_group_idc syntax element in the VPS or the ccm_codec_4CC syntax element in the component codec map (ccm) SEI message. The 4CC code may be any one of "avc3", "hev1", "hev1", "vvi1" or provided by the ccm SEI. avs3, hev1, vvi1 indicate AVC, HEVC, VVC, respectively. ptl_profile_codec_group_iddc indicates a codec group profile component. The component codec mapping SEI message informs the V3C decoder of the codec mapping for the codec ids of the component sub-bitstreams signaled in the VPS. Figure 4 The sub-bitstream decoding part 309 (occupancy decoding part 303, geometry decoding part 304, attribute decoding part 305) in the video sub-bitstream is decoded. The syntax, semantics and decoding process of the video sub-bitstream are specified in other specifications. The video sub-bitstream can be considered and processed in a layer different from the V3C layer (lower layer) (here referred to as the video layer).
[0035] The following terms are used hereinafter. Atlas: A collection of 2D bounding boxes and their associated information placed on a rectangular frame, corresponding to the volume of rendered volumetric data in 3D space. Attributes: Scalar or vector attributes optionally associated with each point in the volume frame, such as color, reflectivity, surface normal, transparency, material ID, etc. Property frame: A 2D rectangular array created by aggregating blocks containing specific property values. Property frame: A 2D rectangular array created by aggregating blocks containing specific property values. Attribute Map: An attribute frame containing attribute block information projected at a specific depth indicated by the corresponding geometry map. Bit stream: An ordered sequence of bits that forms a coded representation of data Coded map sequence (CAS): a sequence of coded map access units that are IRAP coded map access units with NoOutputBeforeRecoveryFlag equal to 1, in decoding order, followed by zero or more coded map access units that are not IRAP coded map access units with NoOutputBeforeRecoveryFlag equal to 1, including all subsequent coded map access units up to but not including any subsequent coded map access unit that is an IRAP coded map access unit with NoOutputBeforeRecoveryFlag equal to 1. Geometry: A collection of Cartesian coordinates associated with a volume frame. Geometry Frame: A 2D array created by aggregating the geometry information associated with each tile. Geometry Map: A geometry frame containing information about geometry blocks projected at a specific depth. Occupancy: A value indicating whether a sample of the atlas corresponds to an associated sample in 3D space. Occupancy frame: A collection of occupancy values that form a 2D array and represent the entire occupancy information of a single atlas frame. Block: A rectangular area within the atlas that is associated with volume information Configuration of 3D data decoding equipment
[0036] Figure 4 3D is a functional block diagram showing a schematic configuration of the 3D data decoding device 31 .
[0037] The 3D data decoding device 31 is composed of a V3C unit decoding part 301, a spectrum decoding part 302, an occupancy decoding part 303, a geometry decoding part 304, an attribute decoding part 305, a neural network post-filter part 306, a post-decoding conversion part 30B, a pre-reconstruction part 310, a reconstruction part 311, and a post-reconstruction part 312. The part composed of the spectrum decoding part 302, the occupancy decoding part 303, the geometry decoding part 304, the attribute decoding part 305, and the neural network post-filter part 306 can be processed in one part, which is called a sub-bitstream decoding part 309.
[0038] The V3C unit decoding part 301 receives the encoded data (bitstream) in the byte stream format, ISO base media file format (ISOBMFF), etc., and decodes the V3C unit header and V3C VPS. The V3C unit decoding part 301 uses the atlas decoding part 302, the occupancy decoding part 303, the geometry decoding part 304, and the attribute decoding part 305 according to the UnitType of the V3C unit header. When the UnitType is V3C_AD, the atlas decoding part 302 is used. Similarly, when the UnitType is V3C_OVD, V3C_GVD, V3C_AVD, the occupancy decoding part 303, the geometry decoding part 304, and the attribute decoding part 305 are used to decode the occupancy video stream, the geometry video stream, and the attribute video stream, respectively.
[0039] Fig.10 1 is a functional block diagram showing a schematic configuration of the 3D data encoding device 11. The 3D data encoding device 11 is composed of a block generation part 121, a block packing and frame generation part 122, a V3C unit encoding part 101, a spectrum encoding part 102, an occupancy encoding part 103, a geometry encoding part 104, an attribute encoding part 105, a neural network post filter part 306, a post-decoding conversion part 30B, a pre-reconstruction part 310, a reconstruction part 311, and a post-reconstruction part 312. The part composed of the spectrum decoding part 302, the occupancy decoding part 303, the geometry decoding part 304, the attribute decoding part 305, and the neural network post filter part 306 can be processed in one part, which is called a sub-bitstream encoding part 109. The block generation part 121 receives 3D data and projects it to generate a group of blocks. The block packing and frame generation part 122 merges (packs) a group of blocks in time series and generates an occupancy frame, a geometry frame, and an attribute frame. The occupancy coding part 103, the geometry coding part 104, and the attribute coding part 105 use the video codec to encode the generated occupancy frame, geometry frame, and attribute frame into respective video sub-bitstreams. The graph coding part 102 encodes the block information. The V3C unit coding part 101 encodes the encoded video sub-bitstream and the graph information into V3C encoded data.
[0040] The spectrum decoding part 302 receives the spectrum information encoding stream and decodes the spectrum information.
[0041] The occupancy decoding part 303 decodes the occupancy video stream encoded by VVC, HEVC, etc., and outputs the decoded occupancy frame DecOccFrames[frameIdx][compIdx][y][x]. Wherein DecOccFrames, frameIdx, compIdx, y and x are respectively the decoded occupancy video frame, frame index, component index, row index and column index. DecOccBitDepth, DecOccHeight, DecOccWidth and DecOccChromaFormat are expressed as the occupied video bit depth, occupied video height, occupied video width and occupied chroma format.
[0042] The geometry decoding part 304 decodes the geometry video stream encoded by VVC, HEVC, etc., and outputs the decoded geometry frame DecGeoFrames[frameIdx][mapIdx][compIdx][y][x]. Wherein DecGeoFrames, frameIdx, mapIdx, compIdx, y and x are respectively the decoded geoupancy video frame, mapping index, frame index, component index, row index and column index. DecGeoBitDepth, DecGeoHeight, DecGeoWidth, DecGeoChromaFormat are represented as geometry video bit depth, geometry video height, geometry video width and geometry chroma format. The decoded geometry frame may contain multiple sets of geometry mappings (geometry frames with different depth projections), where mapIdx is used to distinguish the mappings.
[0043] The attribute decoding part 304 decodes the attribute video stream encoded by VVC, HEVC, etc., and outputs the decoded attribute frame DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]. Wherein DecAttrFrames, frameIdx, attrIdx, partIdx, mapIdx, compIdx, y and x are respectively the decoded atrupancy video frame, frame index, attribute index, part index, mapping index, component index, row index and column index. DecAttrBitDepth, DecAttrHeight, DecAttrWidth, the decoded attribute frame may contain multiple sets of attribute mappings (attribute frames with different depth projections), where mapIdx is used to distinguish the mapping. DecAttrChromaFormat is represented as attribute video bit depth, attribute video height, attribute video width and attribute chroma format. The decoded attribute video frame consists of multiple attributes, such as color (R, G, B), reflection, α, and normal direction. And multiple attributes can be sent by more than one attribute video stream, where attrIdx is used to distinguish them. For example, {R,G,B} is sent by attribute video stream 0 (attrIdx=0), {reflection} is sent by attribute video stream 1 (attrIdx=1), and {α} is sent by attribute video stream 2 (attrIdx=2). Attributes can be sent by spilltng partionions to multiple video streams, where partIdx is used to distinguish them. mapIdx is as described above.
[0044] The neural network post filter part 306 receives one or more decoded video frames, such as a decoded occupancy video frame, a decoded geometry frame, and a decoded attribute frame, and outputs a modified decoded video frame. The frame image filtering part 307 may decode the neural network characteristic SEI to obtain the neural network model information (neural network topology information and neural network parameter information), and decode the neural network activation SEI to specify which frame is applied by applying which neural network model. If the neural network post filter 306 is applied to the decoded occupancy video frame, the decoded geometry video frame, and the attribute video frame, respectively, the neural network post filter part 306 may be included in the occupancy decoding part 303, the geometry decoding part 304, and the attribute decoding part 305. The neural network post filter 306 is abbreviated as NNPF.
[0045] The post-decoding conversion part 30B receives the decoded atlas information, the decoded occupancy frame DecOccFrame, the decoded geometry frame DecGeoFrame, and the decoded attribute frame DecAttrFrame, and converts them in a nominal format. The output is the nominal format version of the occupancy frame OccFrameNF, the geometry frame GeoFrameNF, and the attribute frame AttrFrameNF.
[0046] Nominal formats are collectively referred to as the nominal bit depth, resolution, chroma format, and composition time index to which the decoded video should be converted.
[0047] Each video sub-bitstream and each region of the packed video sub-bitstream is associated with a nominal bit depth, which is the target bit depth at which all reconstruction operations are expected to be performed. The nominal bit depth OccBitDepthNF of the occupancy component is set equal to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1, or to pin_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1 if pin_occupancy_present_flag[ConvAtlasID] is equal to 1. oi_occupancy_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which the occupancy video should be converted for the atlas with atlas ID j. pin_occupancy_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which the decoded region containing occupancy data for the atlas with atlas ID j will be converted. pin_occupancy_present_flag[j] equal to 0 indicates that the packed video frames of the atlas with atlas ID j do not contain regions with occupancy data. pin_occupancy_present_flag[j] equal to 1 indicates that the packed video frames of the atlas with atlas ID j do contain regions with occupancy data. When pin_occupancy_present_flag[j] is not present, its value is inferred to be equal to 0. The nominal bit depth GeoBitDepthNF of each geometry video component is set equal to gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1, or to pin_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 if pin_geometry_present_flag[ConvAtlasID] is equal to 1. gi_geometry_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which all geometry video should be converted for atlas with atlas ID j. pin_geometry_2d_bit_depth_minus1[j] plus 1 indicates the nominal 2D bit depth to which decoded regions containing geometry data for a map with map ID j will be converted. pin_geometry_present_flag[j] equal to 0 indicates that the packed video frame for the map with map ID j does not contain regions with geometry data.pin_geometry_present_flag[j] equal to 1 indicates that the packed video frame of the atlas with atlas ID j does contain regions with geometry data. When pin_geometry_present_flag[j] is not present, its value is inferred to be equal to 0. Finally, the nominal bit depth AttrBitDepthNF[attrIdx] of each attribute video component with attribute index attrIdx is set equal to ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1, or to pin_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx] if pin_attribute_present_flag[ConvAtlasID] is equal to 1. ai_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which all attribute video components with attribute index i should be converted for atlas with atlas ID j. pin_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which, for the atlas with atlas ID j, regions containing attributes with attribute index i should be converted. pin_attribute_present_flag[j] equal to 0 indicates that the packed video frames for the atlas with atlas ID j do not contain regions with attribute data. pin_attribute_present_flag[j] equal to 1 indicates that the packed video frames for the atlas with atlas ID j do contain regions with attribute data. When pin_attribute_present_flag[j] is not present, its value is inferred to be equal to 0.
[0048] Where ConvAtlasID is set equal to vuh_atlas_id, or determined externally if the V3C unit header is not available. vuh_atlas_id is signaled in the V3C unit header for V3C_AD, V3C_OVD, V3C_GVD, V3C_AVD, etc. vuh_atlas_id specifies the ID of the atlas corresponding to the current V3C unit.
[0049] The nominal frame resolution of the non-auxiliary video component is defined by the nominal width VideoWidthNF, which is set equal to asps_frame_width, and the nominal height VideoHeightNF, which is set equal to asps_frame_height. asps_frame_width indicates the atlas frame width expressed in an integer number of samples, where the samples correspond to the luma samples of the video component. V3C bitstream conformance requires that the value of asps_frame_width shall be equal to the value of vps_frame_width[j], where j is the ID of the current atlas. asps_frame_height indicates the atlas frame height expressed in an integer number of samples, where the samples correspond to the luma samples of the video component. V3C bitstream conformance requires that the value of asps_frame_height shall be equal to the value of vps_frame_height[j], where j is the ID of the current atlas. The nominal frame resolution of the auxiliary video component is defined by the nominal width and height specified by the variables AuxVideoWidthNF and AuxVideoHeightNF, respectively. AuxVideoWidthNF and AuxVideoHeightNF are derived from the auxiliary video sub-bitstream associated with the map.
[0050] The nominal chroma format is defined as 4:4:4.
[0051] The post-decoding conversion section 30B consists of bit depth conversion, resolution conversion, output order conversion, atlas composition alignment, atlas dimension alignment, chroma upsampling, geometry mapping synthesis, and attribute mapping synthesis. The video frames provided by the V3C decoding section 309 may require additional processing steps before being input to the reconstruction process. Such processing steps may include converting the decoded video frames to a nominal format (e.g., nominal resolution, bit depth, chroma format, etc.). Note that the information of the nominal format is signaled in the V3C VPS.
[0052] The pre-reconstruction part 310 receives the decoded atlas information, the decoded occupancy frame and the decoded geometry frame and the decoded attribute frame and can refine / modify them. Specifically, if the occupancy synthesis flag os_method_type[k] is equal to 1, which indicates the block boundary filtering method, occupancy synthesis is called with OccFramesNF[compTimeIdx][0] and GeoFramesNF[0][compTimeIdx][0] as input and with the modified array OccFramesNF[compTimeIdx][0] as output. OccFramesNF indicates a decoded occupancy frame in nominal format, and GeoFramesNF indicates a decoded geometry frame in nominal format.
[0053] The reconstruction part 311 reconstructs 3D data point cloud data or mesh data based on the nominal video frame derived in the pre-reconstruction part 310, OccFramesNF[compTimeFrame][0][y][x], GeoFramesNF[mapIdx][compTimeFrame][0][y][x] and AttrFramesNF[attrIdx][compTimeFrame][0][y][x] as inputs. AttrFramesNF indicates attribute frames decoded in the nominal format. The reconstruction part 311 exports the variable pointCnt as the number of points in the reconstructed point cloud frame, the 1D array pointToPatch[pointCnt] as the patch index corresponding to each reconstructed point, the 2D array pointToPixel[pointCnt][dimIdx] as the atlas coordinates corresponding to each reconstructed point, the 2D array recPcGeo[pointCnt][dimIdx] as the list of coordinates corresponding to each reconstructed point, and the 3D array recPcAttr[pointCnt][attrIdx][compIdx] as the attributes associated with the points in the reconstructed point cloud frame. Among them, pointCnt, dimIdx, attrIdx, compIdx correspond to the index of the reconstructed point, the attribute index and the attribute dimension, respectively.
[0054] Specifically, the reconstruction section 311 derives recPcGeo and recPcAttr as follows. where compTime is the target / composition time index. rawPos1D is the 1D position gFrame, aFrame and oFrame[y][x] are the geometry frame, attribute video frame and occupancy frame respectively in nominal format. TilePatch3dOffsetU is the associated tile block parameter from the tile. ai_attribute_count[j] indicates the number of attributes associated with the atlas with atlas ID j. rawPos1D is derived as follows. Among them, AtlasPatchRawPoints, AtlasPatch2dPosX, AtlasPatch2dPosY, AtlasPatch2dSizeX and AtlasPatch2dSizeY are block information derived from the atlas information in the atlas decoding part 302.
[0055] The arrays gFrame[mapIdx][y][x], aFrame[mapIdx][attrIdx][compIdx][y][x] are exported as follows: Wherein ai_attribute_dimension_minus1 plus indicates the total number of dimensions (ie, the number of channels) of the attribute signaled in the V3C VPS and decoded by the V3C unit decoding part 301 .
[0056] The post-reconstruction portion 312 refines the 3D data point cloud data or mesh data after processing by the reconstruction portion 311. The post-reconstruction portion 312 receives pointCnt (as the number of reconstructed points of the current point cloud frame associated with the current atlas), a 1D array attrBitDepth[] (as the nominal bit depth), oFrame[y][x], recPcGeo, recPcAttr, and outputs recPcGeo (possibly modified by applying geometric smoothing) and recPcAttr (possibly modified by applying attribute smoothing). Neural Network Post-Filter SEI
[0057] The sub-bitstream decoding section 309 (neural network post filter section 306) decodes the neural network post filter characteristics (NNPFC) SEI and the neural network post filter activation (NNPFA) SEI.
[0058] Figure 6 A syntax table of the neural network post-filter characteristics SEI is shown.
[0059] Figure 7 A syntax table of the neural network post-filter activation SEI is shown.
[0060] SEI (Supplemental Enhancement Information) is specified in ITU-T H.274 | ISO / IEC 23002-7 and is signaled in video bitstreams including occupancy video streams, geometry video streams, and attribute video streams.
[0061] The NNPFC SEI specifies the neural network model information, including its purpose (function of the process), neural network parameters, input / output information and complexity, etc. In the NNPFC SEI, the neural network parameters are represented by MPEG neural network coding (ISO / IEC15938-17), and it can be signaled in the form of a uniform resource identifier (URI) or a bitstream payload in the SEI. Differential coding can be used in the NNPFC using the incremental update function of the NNC, where once the basic network parameters are signaled by the NNPFC, the subsequent NNPFC SEI effectively signals the neural network parameters with reference to the basic network parameters. The NNPFC SEI has an identifier nnpfc_id. The NNPFC syntax elements and filtering process are described below in the neural network post-filter characteristics SEI message semantics.
[0062] The NNPFA SEI specifies the duration and which neural network model is applied when the corresponding NNPFC with nnpfc_id equal to nnpfa_target_id is activated. The NNPFC syntax elements and activation process are described below in the neural network post filter activation SEI message semantics. V3C neural network post-filter information SEI
[0063] Figure 8 The syntax table of V3C neural network post filter information SEI is shown. In the case where the sub-bitstream decoding part 309 (neural network post filter part 306) applies the NNPC specified in the VSEI, the sub-bitstream decoding part 309 needs to prepare the neural network post filtering process. The NNPFC SEI and NNPFA SEI bitstreams and the V3C unit are signaled in different layers. Specifically, since the NNPFC SEI and NNPFA SEI are signaled in the video sub-bitstream or in the coded video streams (occupancy video stream, geometry video stream, and attribute video stream) encoded in H.264 / AVC, H.265 / HEVC, and H.266 / VVC, the information needs to be decoded by the corresponding video codec. This means that the 3D data decoding device 31 may not obtain the information directly. In order to solve this problem, the V3C neural network post filter information SEI (V3C NNPFI) is introduced. The V3C NNPFI SEI is signaled in the atlas layer or the atlas sub-bitstream (V3C_AD). The V3CNNPFI SEI specifies which type of neural network post-processing is used, the persistence of the NNPF, and which type of NNPF processing is used. V3C Neural Network Post-Filter Information SEI Message Semantics
[0064] The graph decoding part 302 may decode the following syntax elements, and the graph encoding part 102 may encode the following syntax elements. nnpfi_type indicates which type (specification) of the neural network post filter is used. nnpfi_type equal to 1 indicates the neural network post filter characteristics SEI, and the neural network post filter activation in ISO / IEC 23002-7 (VSEI) is used. nnpfi_cancel_flag equal to 1 indicates that the persistence of the neural network post-processing filter is canceled. The neural network post-processing filter is used for any of the occupancy video stream, geometry video stream or attribute video stream and is specified by ISO / IEC23002-7 (VSEI). nnpfi_cancel_flag equal to 0 indicates that nnpfi_persistence_flag follows. nnpfi_persistence_flag specifies the persistence of the target neural network post-processing filter for the current layer. nnpfi_persistence_flag equal to 0 specifies that the target neural network post-processing filter can be used for post-processing filtering of the current picture only. Let aFrmA be the current atlas frame. nnpfi_persistence_flag equal to 1 specifies that the target neural network post-processing filter can be used to post-process the current picture and all subsequent pictures of the current layer in output order until any of the following conditions is true: - A new CAS starts. - End of bitstream. - Output contains the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag etc. The coded map frame aFrmB in the current layer in the map access unit of the NNPFA SEI message of 1, It follows the current picture in output order. nnpfi_map_count_minus1 plus 1 indicates the number of mappings. nnpfi_attribute_count specifies the number of attributes associated with the graph. nnpfi_occupancy_enabled_flag equal to 1 indicates that the neural network post-filter specified in ISO / IEC 23002-7 (VSEI) can be applied to the occupancy video stream. nnpfi_occupancy_enabled_flag equal to 0 indicates that the neural network post-filter specified in ISO / IEC 23002-7 (VSEI) is not applied to the occupancy video stream. nnpfi_occupancy_process_idc indicates the process / purpose of the neural network post filter applied to the occupancy video stream. This value is specified in the nnpfc_purpose specification in the Neural Network Post Filter Characteristics SEI message semantics in ISO / IEC 23002-7 (VSEI). nnpfi_geometry_enabled_flag[j] equal to 1 indicates that the neural network post filter specified in ISO / IEC 23002-7 (VSEI) can be applied to the j-th map of the geometry video stream. nnpfi_geometry_enabled_flag[j] equal to 0 indicates that the neural network post filter specified in ISO / IEC 23002-7 (VSEI) is not applied to the j-th map of the geometry video stream. nnpfi_geometry_process_idc[j] indicates the process / purpose of the neural network post filter applied to the jth map of the geometry video stream. This value is specified in the nnpfc_purpose specification in the Neural Network Post Filter Characteristics SEI message semantics in ISO / IEC 23002-7 (VSEI). nnpfi_attribute_enabled_flag[i][j] equal to 1 indicates that the neural network post filter specified in ISO / IEC 23002-7 (VSEI) can be applied to the j-th mapping of the i-th attribute video stream. nnpfi_attribute_enabled_flag[i][j] equal to 0 indicates that the neural network post filter specified in ISO / IEC 23002-7 (VSEI) is not applied to the j-th mapping of the i-th attribute video stream. nnpfi_attribute_process_idc[i][j] indicates the process / purpose of the neural network post filter applied to the jth mapping of the i-th attribute video stream. This value is specified in the nnpfc_purpose specification in the Neural Network Post Filter Characteristics SEI message semantics in ISO / IEC 23002-7 (VSEI). Alternative V3C Neural Network Post Filter Information SEI message semantics
[0065] The graph decoding part 302 may decode the following optional syntax elements, and the graph encoding part 102 may encode the following syntax elements. nnpfi_occupancy_process_idc indicates the purpose of the neural network post-filter applied to the occupancy video stream. (nnpfi_occupancy_process_idc & 0x4) != 0 indicates that resolution resampling is used for the occupancy video stream. (nnpfi_occupancy_process_idc & 0x10) != 0 indicates that bit depth upsampling is used for the occupancy video stream. nnpfi_geometry_process_idc[j] indicates the purpose of the neural network post-filter applied to the geometry video stream. (nnpfi_geometry_process_idc&0x4)!=0 indicates that resolution resampling is used for the geometry video stream. (nnpfi_geometry_process_idc[j] 0x10)!=0 indicates that bit depth upsampling is used for the geometry video stream. nnpfi_attribute_process_idc[i][j] indicates the purpose of the neural network post-filter applied to the attribute video stream. (nnpfi_attribute_process_idc[i][j]&0x2)!=0 indicates that chroma upsampling is used for the attribute video stream. (nnpfi_attribute_process_idc[i][j]&0x4)!=0 indicates that resolution resampling is used for the attribute video stream. (nnpfi_attribute_process_idc[i][j] 0x10)!=0 indicates that bit depth upsampling is used for the attribute video stream.
[0066] Fig. 9 The syntax table of the V3C neural network post-filter information SEI is also shown.
[0067] In this case, the following syntax regarding neural network model recognition may be encoded by the graph encoding part 102 and decoded by the graph decoding part 302 . nnpfi_occupancy_model_idc indicates the ID value of the neural network model of the neural network post-filter applied to the occupancy video stream. nnpfi_geometry_model_idc[j] indicates the ID value of the neural network model of the j-th mapped neural network post filter applied to the geometry video stream. nnpfi_attribute_model_idc[i][j] indicates the ID value of the j-th mapped neural network post-filter applied to the i-th attribute video stream. For the same neural network model, the values of nnpfi_occupancy_model_idc, nnpfi_geometry_model_idc[j], and nnpfi_attribute_model_idc[i][j] should be unique in CAS.
[0068] This unique value signaling and decoding for each occupancy and / or geometry and / or attribute enables the 3D data decoding device 31 to know in advance at which timing the neural network model should be reloaded and how many neural networks should be stored in the 3D data decoding device 31.
[0069] In addition, the following syntax regarding the neural network complexity information may be encoded by the graph encoding part 102 and decoded by the graph decoding part 302 . nnpfi_occupancy_complexity_present_flag indicates whether complexity information of the neural network post-filter applied to the occupancy video stream is present. nnpfi_geometry_complexity_present_flag[j] indicates whether complexity information of the j-th mapped neural network post filter applied to the geometry video stream is present. nnpfi_attribute_complexity_present_flag[i][j] indicates whether complexity information of the j-th mapped neural network post-filter applied to the i-th attribute video stream is present. nnpfi_complexity_info(kind, idx, mapIdx) may include the following syntax elements. nnpfc_parameter_type_idc equal to 0 indicates that the neural network uses only integer parameters. nnpfc_parameter_type_flag equal to 1 indicates that the neural network can use floating point or integer parameters. nnpfc_parameter_type_idc equal to 2 indicates that the neural network uses only binary parameters. nnpfc_log2_parameter_bit_length_minus3 equal to 0, 1, 2, and 3 indicates that the neural network does not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. nnpfc_num_parameters_idc indicates the maximum number of neural network parameters for the NNPF in powers of 2048. nnpfc_num_parameters_idc equal to 0 indicates the maximum number of unknown neural network parameters. The value of nnpfc_num_parameters_idc should be in the range of 0 to 52, inclusive. If the value of nnpfc_num_parameters_idc is greater than 0, the variable maxNumParameters is derived as follows: maxNumParameters = (2048 << nnpfc_num_parameters_idc) - 1 (100) nnpfc_num_kmac_operations_idc greater than 0 indicates that the maximum number of multiply-accumulate operations per sample of the NNPF is less than or equal to nnpfc_num_kmac_operations_idc * 1000. nnpfc_num_kmac_operations_idc equal to 0 indicates that the maximum number of multiply-accumulate operations of the network is unknown. nnpfc_total_kilobyte_size greater than 0 indicates the total size in kilobytes required to store the uncompressed parameters of the neural network. The total size in bits is a number equal to or greater than the sum of the bits used to store each parameter. nnpfc_total_kilobyte_size is the total size in bits divided by 8000, rounded up. nnpfc_total_kilobyte_size equal to 0 indicates that the total size required to store the parameters for the neural network is unknown.
[0070] This complexity signaling and decoding for each occupancy and / or geometry and / or attribute enables the 3D data decoding device 31 to know which neural network post filter can be processed according to its capabilities and to decide which NNPF is applied to which occupancy and / or geometry and / or attribute video stream. Applying NNPC in V3C Specification
[0071] NNPFA and NNPFC are specified in the VSEI specification. The following procedures and / or requirements are for processing NNPC in the 3D data decoding device 31 .
[0072] The sub-bitstream decoding part 309 (neural network post filter part 306) and the sub-bitstream encoding part 109 decode and encode the neural network post filter characteristics (NNPFC) SEI and the neural network post filter activation (NNPFA) SEI, respectively. Bit depth conversion in NNPF usage in V3C specification
[0073] In the case where the sub-bitstream decoding part 309 (neural network post-filter part 306) applies the neural network post-filter specified in ISO / IEC 23002-7 (VSEI), the output tensor should be converted to an integer using functions OutY, OutC and variable targetBitDepth. FilteredYPic[i][xY][yY]=OutY(FilteredYPic[i][xY][yY]) FilteredCbPic[i][xY][yY]=OutC(FilteredCbPic[i][xY][yY]) FilteredCrPic[i][xY][yY]=OutC(FilteredCrPic[i][xY][yY]) Where i, xX, yY are the index, x-coordinate, and y-coordinate, and OutY and OutC are defined as follows.
[0074] If the value of the nnpfc_out_sample_idc syntax element in the NNPFC SEI is equal to 0, the functions OutY and OutC are specified as follows: OutY(x)=Clip3(0,(1< <targetBitDepth)-1,Round(x*((1<<BitDepthY)-1)))OutC(x)=Clip3(0,(1<<targetBitDepth)-1,Round(x*((1<<BitDepthC)-1))) Otherwise (if nnpfc_out_sample_idc in NNPFC SEI is equal to 1), it indicates that the sample value output by the post-processing filter is an unsigned integer. The functions OutY and OutC are specified as follows: shift = outTensorBitDepth - targetBitDepth if (outTensorBitDepth> = targetBitDepth) OutY(x)=Clip3(0,(1< <targetBitDepth)-1,(x+(1<<(shift-1)))> >shift)else OutY(x)=x<<(targetBitDepth-outTensorBitDepth) shift=outTensorBitDepth-targetBitDepthif(outTensorBitDepth>=targetBitDepth) OutC(x)=Clip3(0,(1< <targetBitDepth)-1,(x+(1<<(shift-1)))> >shift)else OutC(x)=x<<(targetBitDepth-outTensorBitDepth)
[0075] This bit depth conversion ensures the consistency that the output of the 3D data decoding device 31 becomes the same between different 3D data decoding devices. Hereinafter, conformance means that 3D data decoding devices that conform to the specification can decode the same 3D content and produce the same output to achieve interoperability. Alternative bit depth conversion in NNPF usage in V3C specification
[0076] In the case where the sub-bitstream decoding part 309 (neural network post-filter part 306) applies the neural network post-filter specified in ISO / IEC 23002-7 (VSEI), the value of the output tensor (outputTensor) should be converted into an integer using functions OutY, OutC and the variable targetBitDepth. If(!nnpfc_component_last_flag){ FilteredYPic[i][xY][yY]=OutY(outputTensor[0][i][0][yP][xP]) FilteredCbPic[i][xC][yC]=OutC(outputTensor[0][i][1][yPc][xPc]) FilteredCrPic[i][xC][yC]=OutC(outputTensor[0][i][2][yPc][xPc]) }else{ FilteredYPic[i][xY][yY]=OutY(outputTensor[0][i][yP][xP][0]) FilteredCbPic[i][xC][yC]=OutC(outputTensor[0][i][yPc][xPc][1]) FilteredCrPic[i][xC][yC]=OutC(outputTensor[0][i][yPc][xPc][2]) } Alternative bit depth conversion in NNPF usage in V3C specification
[0077] The sub-bitstream decoding section 309 (neural network post-filter section 306) may also apply the transformation after video decoding as follows. When NNPF is applied to the occupied video stream, DecOccFrames[frameIdx][compIdx][y][x]=OutY(DecOccFrames[frameIdx][compIdx][y][x]). In the case of applying NNPF to a geometric video stream, DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]=OutY(DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]). When NNPF is applied to the attribute video stream, DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]=OutY(DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]). Bit depth conversion configuration in NNPF use in V3C specification
[0078] In one embodiment, the sub-bitstream decoding part 309 (neural network post-filter part 306) sets the targetBitdepth to the codec bit depth, expressed as DecOccBitDepth, DecGeoBitDepth and DecAttrBitDepth. This embodiment achieves the required smaller complexity while ensuring the consistency of the 3D data decoding device 31. Alternative bit depth conversion configuration for NNPF usage in V3C specification
[0079] In another embodiment, the sub-bitstream decoding section 309 (neural network post-filter section 306) sets the bit depth of the nominal format, denoted as nominalBitDepth. The nominalBitDepth is signaled in the V3C VPS for the occupancy video stream, the geometry video stream, and the attribute video stream.
[0080] When NNPF is applied to the occupied video stream, nominalBitdepth=oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1.
[0081] When NNPF is applied to the geometry video stream, nominalBitdepth=gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1.
[0082] When a neural network post-filter is applied to the attribute video stream, nominalBitdepth=ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1.
[0083] Among them, ConvAtlasID is the atlas ID, and attrIdx is the attribute information.
[0084] Alternatively, the following assignments may be used.
[0085] When NNPF is applied to the occupied video stream, nominalBitdepth=asps_occupancy_2d_bit_depth_minus1+1.
[0086] When a neural network post-filter is applied to the geometry video stream, nominalBitdepth=asps_geometry_2d_bit_depth_minus1+1.
[0087] When a neural network post-filter is applied to the attribute video stream, nominalBitdepth=asps_attribute_2d_bit_depth_minus1+1.
[0088] asps_geometry_2d_bit_depth_minus1 plus 1 indicates the bit depth of the geometry when projected onto a 2D image. asps_geometry_2d_bit_depth_minus1 shall be in the range of 0 to 31, inclusive. asps_occupancy_2d_bit_depth_minus1 plus 1 indicates the bit depth of occupancy when projected onto a 2D image. asps_attribute_2d_bit_depth_minus1 plus 1 indicates the bit depth of the attribute when projected onto a 2D image. Occupancy of the NNPF process in video decoding
[0089] The occupied decoding part 303 may apply the NNPF specified in ISO / IEC 23002-7 (VSEI) to the occupied video stream. When the NNPF is applied, DecOccBitDepth, DecOccHeight, DecOccWidth, DecOccChromaFormat, DecOccChromaSamplingPosition, DecOccFullRange, DecOccColourPrimaries, DecOccTransferCharacteristics, DecOccMatrixCoeffs may not be the content of the codec picture, but the content of the output of the neural network post-filter. DecOccChromaSamplingPosition (if present) indicates the video chroma sampling position as specified in ISO / IEC 23091-2. DecOccFullRange (if present) indicates the video full range code points as specified in ISO / IEC 23091-2. DecOccFullRange (if present) indicates the video full range code points as specified in ISO / IEC 23091-2. DecOccColourPrimaries (if present) indicates the chromaticity coordinates of the source primaries as specified in ISO / IEC 23091-2. DecOccTransferCharacteristics (if present) indicates the transfer characteristics as specified in ISO / IEC 23091-2. DecOccMatrixCoeffs (if present) indicates the matrix coefficients as specified in ISO / IEC 23091-2.
[0090] When applying the NNPF specified in ISO / IEC 23002-7 (VSEI), bitstream conformance requires the following constraints to apply: Only at most one neural network post-filter is activated / applied to each frame of the occupied video stream. (nnpfc_purpose & 0x02) should be 0, in which case chroma upsampling should not be used. (nnpfc_purpose & 0x08) should be 0, where frame rate upsampling should not be used. (nnpfc_purpose & 0x20) should be 0, in which case shading upsampling should not be used.
[0091] The following can be applied. nnpfc_padding_type is 0 only when only zero padding is done. nnpfc_auxiliary_inp_idc should be 0. In case only luma post-filter is allowed, nnpfc_inp_order_idc should be 0. nnpfc_out_order_idc shall be 0, indicating that only the luma post-filter is allowed. nnpfc_inp_order_idc indicates a method of ordering a sample array of a cropped decoded output picture which is one of the input pictures of the NNPF.
[0092] In another embodiment, the following may apply. nnpfc_padding_type should be 0,1,2,4 only. nnpfc_inp_order_idc should be 0,2,3 only. nnpfc_out_order_idc should be only 0, 2, 3.
[0093] In another embodiment, the following may apply. nnpfc_constant_patch_size_flag shall be set equal to 1, where the explicit block size shall be specified in the NNPFC SEI. When the slice QP-based filtering strength control value strengthControlVal is not used, nnpfc_auxiliary_inp_idc should be 0.
[0094] It is important in the V3C specification and applications to be codec agnostic, and the above constraints are needed to ensure conformance.
[0095] In another embodiment, the following may apply. When (nnpfc_purpose & 0x04) != 0, the output resolution of the neural network post-filter shall be equal to the nominal resolution of the occupied component, i.e., asps_frame_height × asps_frame_width. When (nnpfc_purpose&0x10)!=0, the output of the neural network post-filter shall be equal to the nominal bit depth of the occupancy component, i.e., oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1 When (nnpfc_purpose & 0x04) != 0, nnpfcOutputPicHeight shall be equal to asps_frame_height. When (nnpfc_purpose & 0x04) != 0, nnpfcOutputPicWidth shall be equal to asps_frame_width. When (nnpfc_purpose&0x10)!=0, nnpfc_output_format_idc shall be equal to 1, and (nnpfc_out_tensor_luma_bitdepth_minus8+8) shall be equal to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1. NNPF process in geometric video decoding
[0096] The geometry decoding portion 304 may apply the NNPF specified in ISO / IEC 23002-7 (VSEI) to the geometry video stream. When the NNPF is applied, DecGeoBitDepth, DecGeoHeight, DecGeoWidth, DecGeoChromaFormat, DecGeoChromaSamplingPosition, DecGeoFullRange, DecGeoColourPrimaries, DecGeoTransferCharacteristics, DecGeoMatrixCoeffs may not be the content of the codec picture, but the content of the output of the neural network post-filter. DecGeoChromaSamplingPosition (if present) indicates the geometry chroma sampling as specified in ISO / IEC 23091-2. DecGeoFullRange (if present) indicates the video full range code points as specified in ISO / IEC 23091-2. DecGeoColourPrimaries (if present) indicates the chromaticity coordinates of the source primaries as specified in ISO / IEC 23091-2. DecGeoTransferCharacteristics (if present) indicates the transfer characteristics as specified in ISO / IEC 23091-2. DecGeoMatrixCoeffs (if present) indicates the matrix coefficients as specified in ISO / IEC 23091-2.
[0097] When applying the NNPF specified in ISO / IEC 23002-7 (VSEI), bitstream conformance requires the following constraints to apply: Only at most one neural network post-filter is activated / applied to each frame of the geometry video stream. (nnpfc_purpose & 0x02) should be 0, in which case chroma upsampling should not be used. (nnpfc_purpose & 0x08) should be 0, where frame rate upsampling should not be used. (nnpfc_purpose & 0x20) should be 0, in which case shading upsampling should not be used.
[0098] The following can be applied. nnpfc_padding_type is 0 only when only zero padding is done. nnpfc_auxiliary_inp_idc should be 0. In case only luma post-filter is allowed, nnpfc_inp_order_idc should be 0. nnpfc_out_order_idc shall be 0, indicating that only the luma post-filter is allowed.
[0099] In another embodiment, the following may apply. nnpfc_padding_type should be 0,1,2,4 only. nnpfc_inp_order_idc should be 0,2,3 only. nnpfc_out_order_idc should be only 0, 2, 3.
[0100] In another embodiment, the following may apply. nnpfc_constant_patch_size_flag shall be set equal to 1. nnpfc_auxiliary_inp_idc should be 0.
[0101] In another embodiment, the following may apply. When (nnpfc_purpose & 0x04)! = 0, the output resolution of the neural network post-filter shall be equal to the nominal resolution of the geometric component, i.e., asps_frame_height × asps_frame_width. Specifically, when (nnpfc_purpose & 0x04)! = 0, nnpfcOutputPicHeight shall be equal to asps_frame_height, and nnpfcOutputPicWidth shall be equal to asps_frame_width. When (nnpfc_purpose&0x10)!=0, the output of the neural network post-filter shall be equal to the nominal bit depth of the occupied component, i.e., gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1. NNPF process in attribute video decoding
[0102] The attribute decoding part 305 may apply the NNPF specified in ISO / IEC 23002-7 (VSEI) to the attribute video stream. When the NNPF is applied, DecAttrBitDepth, DecAttrHeight, DecAttrWidth, DecAttrChromaFormat, DecAttrChromaSamplingPosition, DecAttrFullRange, DecAttrColourPrimaries, DecAttrTransferCharacteristics, DecAttrMatrixCoeffs may not be the content of the codec picture, but the content of the output of the neural network post-filter. DecAttrChromaSamplingPosition (if present) indicates the attribute chroma sampling as specified in ISO / IEC 23091-2. DecAttrFullRange (if present) indicates the video full range code points as specified in ISO / IEC 23091-2. DecAttrColourPrimaries (if present) indicates the chromaticity coordinates of the source primaries as specified in ISO / IEC 23091-2. DecAttrTransferCharacteristics (if present) indicates the transfer characteristics as specified in ISO / IEC 23091-2. DecAttrMatrixCoeffs (if present) indicates the matrix coefficients as specified in ISO / IEC 23091-2.
[0103] When applying the NNPF specified in ISO / IEC 23002-7 (VSEI), bitstream conformance requires the following constraints to apply: Only at most one neural network post-filter is activated / applied to each frame of the geometry video stream. (nnpfc_purpose & 0x08) should be 0, where frame rate upsampling should not be used. (nnpfc_purpose & 0x20) should be 0, in which case shading upsampling should not be used. nnpfc_auxiliary_inp_idc should be 0.
[0104] In another embodiment, the following may apply. nnpfc_padding_type should be 0,1,2,4 only.
[0105] In another embodiment, the following may apply. nnpfc_constant_patch_size_flag shall be set equal to 1. nnpfc_auxiliary_inp_idc should be 0.
[0106] In another embodiment, the following may apply. When (nnpfc_purpose & 0x02) != 0, the output chroma format of the neural network post-filter shall be equal to the nominal chroma format of the attribute components 4:4:4. When (nnpfc_purpose & 0x04)! = 0, the output resolution of the neural network post-filter shall be equal to the nominal resolution of the attribute component, i.e., asps_frame_height × asps_frame_width. Specifically, when (nnpfc_purpose & 0x04)! = 0, nnpfcOutputPicHeight shall be equal to asps_frame_height, and nnpfcOutputPicWidth shall be equal to asps_frame_width. When (nnpfc_purpose&0x10)!=0, the output of the neural network post-filter shall be equal to the nominal bit depth of the occupied component, i.e., ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1. The process of applying NNPF when the codec is HEVC
[0107] In the case where the codec of the corresponding video sub-bitstream is HEVC, the sub-bitstream decoding part 309 (atlas decoding part 302, occupancy decoding part 303, geometry decoding part 304, attribute decoding part 305) can apply the following process to apply the NNPF specified in ITU-T H.274|ISO / IEC 23002-7 to the occupancy and / or geometry and / or attribute video sub-bitstream. The sub-bitstream decoding part 309 can derive the codec to respectively decode the occupancy, geometry, attribute video streams. Decodes oi_occumpancy_codec_id, gi_geometry_codec_id, or ai_attribute_codec_id in the VPS. Let currCodedPic be the coded picture, and the neural network post-processing filter (NNPF) defined by the Neural Network Post-Filter Characteristics (NNPFC) SEI message activated by the Neural Network Post-Filter Activation (NNPFA) SEI message. The variable pictureRateUpsamplingFlag is set equal to (nnpfc_purpose & 0x08)!=0. The variable numInputPics is set equal to nnpfc_num_input_pics_minus1+1. The array inputPicPoc[i] specifies the picture order count values of the input pictures of the NNPF for all values of i in the range of 0 to numInputPics-1 (inclusive), and is derived as follows: –inputPicPoc[0] is set equal to PicOrderCntVal of currCodedPic. – When numInputPics is greater than 1, the following applies for each value of i in the range 1 to numInputPics-1 (inclusive), in increasing order of i: – If currCodedPic is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and a specific value of fp_current_frame_is_frame0_flag, then inputPicPoc[i] is set equal to the PicOrderCntVal of the picture that precedes the picture associated with index i-1 in output order and is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and the same value of fp_current_frame_is_frame0_flag. – Otherwise (currCodedPic is not associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5), inputPicPoc[i] is set equal to the PicOrderCntVal of the picture that precedes the picture associated with index i-1 in output order. For the purpose of interpreting the NNPFC SEI message, the following variables are specified: – If pictureRateUpsamplingFlag is equal to 1 and there is a second NNPF defined by at least one NNPFC SEI message, activated by a NNPFA SEI message for currCodedPic and with resolutionResamplingFlag equal to 1, the following applies: –CroppedWidth is set equal to the nnpfcOutputPicWidth defined for the second NNPF. –CroppedHeight is set equal to the nnpfcOutputPicHeight defined for the second NNPF. - Otherwise, the following applies: – CroppedWidth is set equal to the value of pic_width_in_luma_samples–SubWidthC*(conf_win_left_offset+conf_win_right_offset) of currCodedPic. – CroppedHeight is set equal to the value of pic_height_in_luma_samples–SubHeightC*(conf_win_top_offset+conf_win_bottom_offset) of currCodedPic. - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are derived as follows for each value of i in the range 0 to numInputPics-1 (inclusive): – Let sourcePic be the cropped decoded output picture whose PicOrderCntVal is equal to inputPicPoc[i] in the CLVS containing currCodedPic. – If pictureRateUpsamplingFlag is equal to 0, the following applies: - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are set to the two-dimensional arrays of decoded sample values of the Y, Cb and Cr components of sourcePic, respectively. – Otherwise (pictureRateUpsamplingFlag is equal to 1), the following applies: – The variable sourceWidth is set equal to the value of sourcePic’s pic_width_in_luma_samples–SubWidthC*(conf_win_left_offset+conf_win_right_offset). – The variable sourceHeight is set equal to the value of sourcePic’s pic_height_in_luma_samples–SubHeightC*(conf_win_top_offset+conf_win_bottom_offset). – If sourceWidth is equal to CroppedWidth and sourceHeight is equal to CroppedHeight, then set inputPic to be the same as sourcePic. – Otherwise (sourceWidth is not equal to CroppedWidth or sourceHeight is not equal to CroppedHeight), the following applies: – There shall be a NNPF, hereinafter referred to as resolution-resampling NNPF, defined by at least one NNPFC SEI message, activated by the NNPFA SEI message for sourcePic, and with resolutionResamplingFlag equal to 1, nnpfc_pic_width_in_luma_samples equal to CroppedWidth, and nnpfc_pic_height_in_luma_samples equal to CroppedHeight. –inputPic is set to the output of the neural network inference of the super-resolution NNPF with sourcePic as input. - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are set to the two-dimensional arrays of decoded sample values of the Y, Cb and Cr components of inputPic, respectively. –BitDepthY and BitDepthC are set equal to BitDepthY and BitDepthC. – ChromaFormatIdc is set equal to chroma_format_idc. – StrengthControlVal is set equal to the value of (SliceQpY+QpBdOffsetY)÷(51+QpBdOffsetY) of the first slice of currCodedPic. where BitDepthY and QpBdOffsetY are derived using the values of the sps_bitdepth_minus8 and sps_bitdepth_minus8 syntax elements in the sub-bitstream as follows. BitDepth=8+sps_bitdepth_luma_minus8 QpBdOffsetY=6*sps_bitdepth_luma_minus8
[0108] In this embodiment, when the codec of the sub-bitstream is HEVC, the 3D decoding device 31 decodes the slice QP SliceQPY and the offset QpBdOffsetY, and derives StrengthControlVal, which is set equal to the value of (SliceQpY+QpBdOffsetY)÷(51+QpBdOffsetY) of the first slice of the current encoded picture, and sets StrengthControlVal as the input tensor, and applies a neural network post filter to the sub-bitstream. The process of applying NNPF when the codec is VVC
[0109] When the codec of the corresponding video sub-bitstream is VVC, the sub-bitstream decoding part 309 (atlas decoding part 302, occupancy decoding part 303, geometry decoding part 304, attribute decoding part 305) can apply the following process to apply the NNPF specified in ITU-T H.274|ISO / IEC 23002-7 to the occupancy and / or geometry and / or attribute video sub-bitstream. Let currCodedPic be the coded picture, and the neural network post-processing filter (NNPF) defined by the Neural Network Post-Filter Characteristics (NNPFC) SEI message activated by the Neural Network Post-Filter Activation (NNPFA) SEI message. The variable pictureRateUpsamplingFlag is set equal to (nnpfc_purpose & 0x08)!=0. The variable numInputPics is set equal to nnpfc_num_input_pics_minus1+1. The array inputPicPoc[i] specifies the picture order count values of the input pictures of the NNPF for all values of i in the range of 0 to numInputPics-1 (inclusive), and is derived as follows: –inputPicPoc[0] is set equal to PicOrderCntVal of currCodedPic. – When numInputPics is greater than 1, the following applies for each value of i in the range 1 to numInputPics-1 (inclusive), in increasing order of i: – If currCodedPic is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and a specific value of fp_current_frame_is_frame0_flag, then inputPicPoc[i] is set equal to the PicOrderCntVal of the picture that precedes the picture associated with index i-1 in output order and is associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5 and the same value of fp_current_frame_is_frame0_flag. – Otherwise (currCodedPic is not associated with a frame packing arrangement SEI message with fp_arrangement_type equal to 5), inputPicPoc[i] is set equal to the PicOrderCntVal of the picture that precedes the picture associated with index i-1 in output order. For the purpose of interpreting the NNPFC SEI message, the following variables are specified: – If pictureRateUpsamplingFlag is equal to 1 and there is a second NNPF defined by at least one NNPFC SEI message, activated by a NNPFA SEI message for currCodedPic and with resolutionResamplingFlag equal to 1, the following applies: –CroppedWidth is set equal to the nnpfcOutputPicWidth defined for the second NNPF. – CroppedHeight is set equal to the nnpfcOutputPicHeight defined for the second NNPF. - Otherwise, the following applies: –CroppedWidth is set equal to the value of pps_pic_width_in_luma_samples-SubWidthC*(pps_conf_win_left_offset+pps_conf_win_right_offset) of currCodedPic. –CroppedHeight is set equal to the value of pps_pic_height_in_luma_samples-SubHeightC*(pps_conf_win_top_offset+pps_conf_win_bottom_offset) of currCodedPic. - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are derived as follows for each value of i in the range 0 to numInputPics-1 (inclusive): – Let sourcePic be the cropped decoded output picture whose PicOrderCntVal is equal to inputPicPoc[i] in the CLVS containing currCodedPic. – If pictureRateUpsamplingFlag is equal to 0, the following applies: - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are set to the two-dimensional arrays of decoded sample values of the Y, Cb and Cr components of sourcePic, respectively. – Otherwise (pictureRateUpsamplingFlag is equal to 1), the following applies: – The variable sourceWidth is set equal to the value of sourcePic’s pps_pic_width_in_luma_samples-SubWidthC*(pps_conf_win_left_offset+pps_conf_win_right_offset). – The variable sourceHeight is set equal to the value of sourcePic’s pps_pic_height_in_luma_samples-SubHeightC*(pps_conf_win_top_offset+pps_conf_win_bottom_offset). – If sourceWidth is equal to CroppedWidth and sourceHeight is equal to CroppedHeight, then set inputPic to be the same as sourcePic. – Otherwise (sourceWidth is not equal to CroppedWidth or sourceHeight is not equal to CroppedHeight), the following applies: – There shall be a NNPF, hereinafter referred to as resolution-resampling NNPF, defined by at least one NNPFC SEI message, activated by the NNPFA SEI message for sourcePic, and with resolutionResamplingFlag equal to 1, nnpfc_pic_width_in_luma_samples equal to CroppedWidth, and nnpfc_pic_height_in_luma_samples equal to CroppedHeight. –inputPic is set to the output of the neural network inference of the super-resolution NNPF with sourcePic as input. - The luma sample array CroppedYPic[i] and the chroma sample arrays CroppedCbPic[i] and CroppedCrPic[i], when present, are set to the two-dimensional arrays of decoded sample values of the Y, Cb and Cr components of inputPic, respectively. – Both BitDepthY and BitDepthC are set equal to BitDepth. –ChromaFormatIdc is set equal to sps_chroma_format_idc. – StrengthControlVal is set equal to the value of (SliceQpY+QpBdOffset)÷(63+QpBdOffset) of the first slice of currCodedPic. where BitDepth and QpBdOffset are derived using the values of the sps_bitdepth_minus8 and sps_bitdepth_minus8 syntax elements in the sub-bitstream as follows. BitDepth=8+sps_bitdepth_minus8 QpBdOffset=6*sps_bitdepth_minus8 Neural Network Post-Filter Characteristics SEI Message Semantics nnpfc_purpose indicates the purpose of the NNPF, as shown below, where (nnpfc_urpose and bitMask) not equal to 0 indicates that the NNPF has a purpose associated with the bitMask value. When nnpfc_purpose is greater than 0 and (nnpfc_urpose and bitMask) is equal to 0, the purpose associated with the bitMask value is not applicable to the NNPF. When nnpfc_pupose is equal to 0, the NNPF can be used as determined by the application. The variables chromaUpsamplingFlag, resolutionResamplingFlag, pictureRateUpsamplingFlag, bitDepthUpsamplingFlag, and colourizationFlag specify whether nnpfc_purpose indicates that the purpose of the NNPF includes chroma upsampling, resolution upsampling, picture rate upsampling, bit depth upsampling, and colourization, respectively, and are derived as follows: chromaUpsamplingFlag = ((nnpfc_purpose & 0x02) > 0)? 1:0 resolutionResamplingFlag = ((nnpfc_purpose & 0x04) > 0)? 1:0 pictureRateUpsamplingFlag = ((nnpfc_purpose & 0x08) > 0)? 1:0 bitDepthUpsamplingFlag = ((nnpfc_purpose & 0x10) > 0)? 1:0 colourizationFlag = ((nnpfc_purpose & 0x20) > 0)? 1:0 When the NNPFC SEI message is the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, the following applies: – This SEI message specifies the basic NNPF. – This SEI message is relevant to the current decoded picture and all subsequent decoded pictures of the current layer in output order, Until the current CLVS ends. nnpfc_mode_idc equal to 0 indicates that the SEI message contains an ISO / IEC 15938-17 bitstream that specifies a base NNPF or an update to a base NNPF with the same nnpfc_id value. When the NNPFC SEI message is the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS, nnpfc_mode_idc equal to 1 specifies that the base NNPF associated with the nnpfc_id value is a neural network identified by a URI indicated by nnpfc_uri having a format identified by a tag URI nnpfc_tag_uri. When the NNPFC SEI message is neither the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS nor a repetition of the first NNPFC SEI message in decoding order, nnpfc_mode_idc equal to 1 specifies that updates relative to the base NNPF with the same nnpfc_id value are defined by the URI indicated by nnpfc_uri, which has a format identified by the tag URI nnpfc_tag_uri. When this SEI message is the first NNPF CSEI message in decoding order with a particular nnpfc_id value within the current CLVS, the NNPF PostProcessingFilter() is assigned to be the same as the base NNPF. When this SEI message is neither the first NNPFC SEI message in decoding order with a particular nnpfc_id value within the current CLVS nor a repetition of the first NNPFC SEI message in decoding order, the NNPF PostProcessingFilter() is obtained by applying the updates defined by the SEI message to the base NNPF. Updates are not cumulative, but each update is applied on the base NNPF, which is the NNPF specified by the first NNPFC SEI message in decoding order with a specific nnpfc_id value within the current CLVS. In bitstreams conforming to this version of this document, nnpfc_reserved_zero_bit_a shall be equal to 0. When nnpfc_reserved_zero_bit_a is not equal to 0, the decoder shall ignore the NNPFC SEI. nnpfc_tag_uri contains a tag URI having syntax and semantics as specified in IETF RFC 4151, identifying the format and associated information about a neural network used as a base NNPF or an update relative to a base NNPF having the same nnpfc_id value specified by nnpfc_uri. nnpfc_uri contains a URI having syntax and semantics as specified in IETF Internet Standard 66, identifying a neural network used as a base NNPF or an update relative to a base NNPF having the same nnpfc_id value. nnpfc_property_present_flag equal to 1 specifies that syntax elements related to filter purpose, input format, output format and complexity are present. nnpfc_property_present_flag equal to 0 specifies that no syntax elements related to filter purpose, input format, output format and complexity are present. nnpfc_base_flag equal to 1 specifies that the SEI message specifies a base NNPF. nnpf_base_flag equal to 0 specifies that the SEI message specifies an update relative to the base NNPF. When not present, the value of nnpfc_base_flag is inferred to be equal to 0. nnpfc_num_input_pics_minus1 plus 1 specifies the number of pictures used as input to the NNPF. The value of npfc_num_input_pics_minus1 should be in the range of 0 to 63 (inclusive). The variable numInputPics that specifies the number of pictures used as input to NNPF is derived as numInputPics = nnpfc_num_input_pics_minus1+1 nnpfc_out_sub_c_flag specifies the values of the variables outSubWidthC and outSubHeightC when chromaUpsamplingFlag is equal to 1. nnpfc_out_sub_c_flag equal to 1 specifies that outSubWidthC is equal to 1 and outSubHeightC is equal to 1. nnpfc_out_sub_c_flag equal to 0 specifies that outSubWidthC is equal to 2 and outSubHeightC is equal to 1. When colourizationFlag is equal to 1, nnpfc_out_colour_format_idc specifies the colour format of the NNPF output and therefore specifies the values of the variables outSubWidthC and outSubHeightC. nnpfc_out_colour_format_idc equal to 1 specifies that the colour format of the NNPF output is in 4:2:0 format, and outSubWidthC and outSubHeightC are both equal to 2. nnpfc_out_colour_format_idc equal to 2 specifies that the colour format of the NNPF output is in 4:2:2 format, and outSubWidthC is equal to 2 and outSubHeightC is equal to 1. nnpfc_out_colour_format_idc equal to 3 specifies that the colour format of the NNPF output is in 4:4:4 format, and outSubWidthC and outSubHeightC are both equal to 1. When chromaUpsamplingFlag and colourizationFlag are both equal to 0, outSubWidthC and outSubHeightC are inferred to be equal to SubWidthC and SubHeightC, respectively. nnpfc_pic_width_num_minus1 plus 1 and nnpfc_pic_width_denom_minus1 plus 1 specify the numerator and denominator of the resampling ratio of the NNPF output picture width relative to CroppedWidth, respectively. The value of (nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denom_minus1+1) should be in the range of 1 / 16 to 16, inclusive. When nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 are not present, the values of nnpfc_pic_width_num_minus1 and nnpfc_pic_width_denom_minus1 are inferred to be equal to 0. The variable nnpfcOutputPicWidth represents the width of the luma sample array of the picture resulting from applying the NNPF identified by nnpfc_id to the input picture, and is derived as follows: nnpfcOutputPicWidth = Ceil(CroppedWidth*(nnpfc_pic_width_num_minus1+1)÷(nnpfc_pic_width_denom_minus1+1)) nnpfc_pic_height_num_minus1 plus 1 and nnpfc_pic_height_denom_minus1 plus 1 specify the numerator and denominator of the resampling ratio of the NNPF output picture height relative to the CroppedHeight, respectively. The value of (nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denom_minus1+1) should be in the range of 1 / 16 to 16, inclusive. When nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are not present, the values of nnpfc_pic_height_num_minus1 and nnpfc_pic_height_denom_minus1 are inferred to be equal to 0. The variable nnpfcOutputPicHeight represents the height of the luma sample array of the picture resulting from applying the NNPF identified by nnpfc_id to the input picture, and is derived as follows: nnpfcOutputPicHeight = Ceil(CroppedHeight*(nnpfc_pic_height_num_minus1+1)÷(nnpfc_pic_height_denom_minus1+1)) nnpfc_interpolated_pics[i] specifies the number of interpolated pictures generated by the NNPF between the i-th and (i+1)-th pictures used as input to this NNPF. The variable NumInpPicsInOutputTensor that specifies the number of pictures that have a corresponding input picture and are present in the output tensor of the NNPF, the variable InpIdx[idx] that specifies the input picture index of the idx-th picture that is present in the output tensor of the NNPF and has a corresponding input picture, and the variable numOutputPics that specifies the total number of pictures that are present in the output tensor of the NNPF are derived as follows: for(i=0,numOutputPics=0;i <numInputPics;i++) if(nnpfc_input_pic_output_flag[i]){ InpIdx[numOutputPics] = i numOutputPics++ } NumInpPicsInOutputTensor=numOutputPics if(pictureRateUpsamplingFlag) for(i=0;i<=numInputPics-2;i++) numOutputPics+=nnpfc_interpolated_pics[i] nnpfc_component_last_flag equal to 1 indicates that the last dimension in the input tensor inputTensor of NNPF and the output tensor outputTensor produced by NNPF is used for the current channel. nnpfc_component_last_flag equal to 0 indicates that the third dimension in the input tensor inputTensor of NNPF and the output tensor outputTensor produced by NNPF is used for the current channel. nnpfc_inp_format_idc indicates a method of converting sample values of the cropped decoded output picture into input values input to the NNPF. When nnpfc_inp_format_idc is equal to 0, the input values input to the NNPF are real numbers, and the functions InpY() and InpC() are specified as follows: InpY(x)=x÷((1< <BitDepthY)-1) InpC(x)=x÷((1< <BitDepthC)-1) When nnpfc_inp_format_idc is equal to 1, the input value to the NNPF is an unsigned integer, and the functions InpY() and InpC() are specified as follows: The variable inpTensorBitDepthY is derived from the syntax element nnpfc_inp_tensor_luma_bitdepth_minus8, as specified below. The variable inpTensorBitDepthC is derived from the syntax element nnpfc_inp_tensor_chroma_bitdepth_minus8, as specified below. nnpfc_inp_tensor_luma_bitdepth_minus8 plus 8 specifies the bit depth of the luma sample values in the input integer tensor. The value of inpTensorBitDepthY is derived as follows: inpTensorBitDepthY = nnpfc_inp_tensor_luma_bitdepth_minus8 + 8 nnpfc_inp_tensor_chroma_bitdepth_minus8 plus 8 specifies the bit depth of the chroma sample values in the input integer tensor. The value of inpTensorBitDepthC is derived as follows: inpTensorBitDepthC = nnpfc_inp_tensor_chroma_bitdepth_minus8+8 nnpfc_inp_order_idc value description A block is a rectangular array of samples of a component (eg, luma or chroma components) from a picture. nnpfc_auxiliary_inp_idc greater than 0 indicates that the auxiliary input information is present in the input tensor of the NNPF. nnpfc_auxiliary_inp_idc equal to 0 indicates that the auxiliary input is not present in the input tensor. nnpfc_auxiliary_inp_idc equal to 1 specifies that the auxiliary input data is derived as specified in the formula NNPF. When nnpfc_auxiliary_inp_idc is equal to 1, the variable strengthControlScaledVal is derived as follows: if(nnpfc_inp_format_idc==1) strengthControlScaledVal=Floor(StrengthControlVal* ((1< <inpTensorBitDepthY)-1)) else strengthControlScaledVal=StrengthControlVal The procedure DeriveInputTensors() for deriving an input tensor inputTensor for a given vertical sample coordinate cTop and horizontal sample coordinate cLeft of the top-left sample position of a block of samples included in the specified input tensor is specified as follows: The nnpfc_separate_colour_description_present_flag being equal to 1 indicates that different combinations of the colour primaries, transfer characteristics, and matrix coefficients of the picture generated from the NNPF are specified in the SEI message syntax structure. The nnpfc_separate_colour_description_present_flag being equal to 0 indicates that the combination of the colour primaries, transfer characteristics, and matrix coefficients of the picture generated from the NNPF is the same as that indicated in the VUI parameters of the CLVS. nnpfc_colour_primaries specifies the colour primaries of the picture generated by applying the NNPF specified in the SEI message, rather than the colour primaries for the CLVS. nnpfc_transfer_characteristics specifies the transfer characteristics of the picture generated by applying the NNPF specified in the SEI message, rather than the transfer characteristics for the CLVS. nnpfc_matrix_coeffs specifies the matrix coefficients of the picture generated by applying the NNPF specified in the SEI message, rather than the matrix coefficients for the CLVS. nnpfc_out_format_idc being equal to 0 indicates that the sample values output by the NNPF are real numbers in the range from 0 to 1 (including the end values), linearly mapped to the unsigned integer value range from 0 to (1<<bitDepth)-1 (including the end values) for any required bit depth bitDepth, for subsequent post-processing or display. nnpfc_out_format_idc being equal to 1 indicates that the luminance sample values output by the NNPF are unsigned integers in the range from 0 to (1<<(nnpfc_out_tensor_luma_bitdepth_minus8+8))-1 (including the end values), and the chrominance sample values output by the NNPF are unsigned integers in the range from 0 to (1<<(nnpfc_out_tensor_chroma_bitdepth_minus8+8))-1 (including the end values). nnpfc_out_tensor_luma_bitdepth_minus8 plus 8 specifies the bit depth of the luminance sample values in the output integer tensor. nnpfc_out_tensor_chroma_bitdepth_minus8 plus 8 specifies the bit depth of the chrominance sample values in the output integer tensor. nnpfc_out_order_idc indicates the output order of the samples generated from the NNPF. Description of nnpfc_out_order_idc values The procedure StoreOutputTensors() for deriving sample values in the filtered output sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic from the output tensor outputTensor given vertical sample coordinates cTop and horizontal sample coordinates cLeft of the top left sample position of the sample block included in the specified input tensor is specified as follows: nnpfc_overlap indicates the overlapping horizontal and vertical sample counts of adjacent input tensors to NNPF. nnpfc_constant_patch_size_flag equal to 1 indicates that the NNPF accepts exactly the patch size indicated by nnpfc_patch_width_minus1 and nnpfc_patch_height_minus1 as input. nnpfc_constant_patch_size_flag equal to 0 indicates that NNPF accepts as input any block size of width inpPatchWidth and height inpPatchHeight, so that the width of the extended block (i.e., the block plus the overlapped area) equal to inpPatchWidth+2*nnpfc_overlap is a positive integer multiple of nnpfc_extended_patch_width_cd_delta_minus1+1+2*nnpfc_overlap, and the height of the extended block equal to inpPatchHeight+2*nnpfc_overlap is a positive integer multiple of nnpfc_extended_patch_height_cd_delta_minus1+1+2*nnpfc_overlap. When nnpfc_constant_patch_size_flag is equal to 1, nnpfc_patch_width_minus1 plus 1 indicates the horizontal sample count of the block size required for input to the NNPF. When nnpfc_constant_patch_size_flag is equal to 1, nnpfc_patch_height_minus1 plus 1 indicates the vertical sample count of the block size required for input to the NNPF. When nnpfc_constant_patch_size_flag is equal to 0, nnpfc_extended_patch_width_cd_delta_minus1 plus 1 plus 2*nnpfc_overlap indicates the common divisor of all allowed values of the width of the extended patch required for input to the NNPF. When nnpfc_constant_patch_size_flag is equal to 0, nnpfc_extended_patch_height_cd_delta_minus1 plus 1 plus 2*nnpfc_overlap indicates the common divisor of all allowed values of the height of the extended patch required as input to the NNPF. Let the variables inpPatchWidth and inpPatchHeight be the patch size width and patch size height respectively. If nnpfc_constant_patch_size_flag is equal to 0, the following applies: – The values of inpPatchWidth and inpPatchHeight may be provided by external means not specified in this document, or may be set by the post processor itself. Otherwise (nnpfc_constant_patch_size_flag is equal to 1), the value of inpPatchWidth is set equal to nnpfc_patch_width_minus1+1, and the value of inpPatchHeight is set equal to nnpfc_patch_height_minus1+1. The variables outPatchWidth, outPatchHeight, horCScaling, verCScaling, outPatchCWidth, and outPatchCHeight are derived as follows: outPatchWidth=(nnpfcOutputPicWidth*inpPatchWidth) / CroppedWidth outPatchHeight=(nnpfcOutputPicHeight*inpPatchHeight) / CroppedHeight horCScaling=SubWidthC / outSubWidthC verCScaling=SubHeightC / outSubHeightC outPatchCWidth=outPatchWidth*horCScaling outPatchCHeight=outPatchHeight*verCScaling nnpfc_padding_type indicates the padding process when referencing sample positions outside the boundaries of a cropped decoded output picture. Informative description of nnpfc_padding_type values nnpfc_luma_padding_val indicates the luma value used for padding when nnpfc_padding_type is equal to 4. nnpfc_cb_padding_val indicates the Cb value used for padding when nnpfc_padding_type is equal to 4. nnpfc_cr_padding_val indicates the Cr value used for padding when nnpfc_padding_type is equal to 4. The function InpSampleVal(y,x,picHeight,picWidth,croppedPic) whose input is the vertical sample position y, the horizontal sample position x, the picture height picHeight, the picture width picWidth and the sample array croppedPic returns the value of sampleVal derived as follows: The following exemplary process may be used with NNPF PostProcessingFilter() to generate filtered and / or interpolated pictures in a block-by-block manner, which include Y, Cb, and Cr sample arrays FilteredYPic, FilteredCbPic, and FilteredCrPic, respectively, as indicated by nnpfc_out_order_idc: The order of images in the stored output tensor is the output order, and the output order generated by applying NNPF in output order is interpreted as being in output order (and does not conflict with the output order of the input images). nnpfc_complexity_info_present_flag equal to 1 specifies that one or more syntax elements indicating the complexity of the NNPF associated with nnpfc_id are present. nnpfc_complexity_info_present_flag equal to 0 specifies that there is no syntax element indicating the complexity of the NNPF associated with nnpfc_id. nnpfc_parameter_type_idc equal to 0 indicates that the neural network uses only integer parameters. nnpfc_parameter_type_flag equal to 1 indicates that the neural network can use floating point or integer parameters. nnpfc_parameter_type_idc equal to 2 indicates that the neural network uses only binary parameters. nnpfc_log2_parameter_bit_length_minus3 equal to 0, 1, 2, and 3 indicates that the neural network does not use parameters with bit lengths greater than 8, 16, 32, and 64, respectively. When nnpfc_parameter_type_idc is present and nnpfc_log2_parameter_bit_length_minus3 is not present, the neural network does not use parameters with bit lengths greater than 1. nnpfc_num_parameters_idc indicates the maximum number of neural network parameters for the NNPF in powers of 2048. nnpfc_num_parameters_idc equal to 0 indicates the maximum number of unknown neural network parameters. If the value of nnpfc_num_parameters_idc is greater than 0, the variable maxNumParameters is derived as follows: maxNumParameters=(2048< <nnpfc_num_parameters_idc)-1 nnpfc_num_kmac_operations_idc greater than 0 indicates that the maximum number of multiply-accumulate operations per sample of the NNPF is less than or equal to nnpfc_num_kmac_operations_idc * 1000. nnpfc_num_kmac_operations_idc equal to 0 indicates that the maximum number of multiply-accumulate operations of the network is unknown. nnpfc_total_kilobyte_size greater than 0 indicates the total size in kilobytes required to store the uncompressed parameters of the neural network. The total size in bits is a number equal to or greater than the sum of the bits used to store each parameter. nnpfc_total_kilobyte_size is the total size in bits divided by 8000, rounded up. nnpfc_total_kilobyte_size equal to 0 indicates that the total size required to store the parameters for the neural network is unknown. In bitstreams conforming to this version of this document, nnpfc_reserved_zero_bit_b shall be equal to 0. When nnpfc_reserved_zero_bit_b is not equal to 0, the decoder shall ignore the NNPFC SEI message. nnpfc_payload_byte[i] contains the i-th byte of a bitstream conforming to ISO / IEC 15938-17. The byte sequence nnpfc_payload_byte[i] for all current values of i shall be a complete bitstream conforming to ISO / IEC 15938-17. Neural network post-filter activation SEI message semantics
[0110] The Neural Network Post Filter Activation (NNPFA) SEI message activates or deactivates the possible use of the target Neural Network Post-Processing Filter (NNPF) identified by nnpfa_target_id for post-processing filtering of a set of pictures. For a specific picture for which the NNPF is activated, the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id, which precedes the first VCL NAL unit of the current picture in decoding order and is not a repetition of the NNPFC SEI message containing the base NNPF. nnpfa_target_id indicates the target NNPF, which is specified by one or more NNPFC SEI messages related to the current picture and having nnpfc_id equal to nnfpa_target_id. nnpfa_cancel_flag equal to 1 indicates that the persistence of the target NNPF established by any previous NNPFA SEI message with the same nnpfa_target_id as the current SEI message is cancelled, i.e. the target NNPF is not used again unless it is activated by another NNPFA SEI message with the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag equal to 0. nnpfa_cancel_flag equal to 0 indicates that nnpfa_persistence_flag follows. nnpfa_persistence_flag specifies the persistence of the target NNPF for the current layer. nnpfa_persistence_flag equal to 0 specifies the target NNPF that can be used for post-processing filtering of the current picture. nnpfa_persistence_flag equal to 1 specifies that the target NNPF can be used for post-processing filtering of the current picture and all subsequent pictures of the current layer in output order until one or more of the following conditions are true: – A new CLVS starts for the current layer. – End of bitstream. – The current layer has the same nnpfa_target_id as the current SEI message and nnpfa_cancel_flag is equal to The picture associated with the NNPFA SEI message of 1 is located after the current picture in the output order. Let nnpfcTargetPictures be the picture set to which the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id belongs that precedes the current NNPFA SEI message in decoding order. Let nnpfaTargetPictures be the picture set for which the target NNPF is activated by the current NNPFA SEI message.
Claims
1. A device comprising one or more processors, wherein the one or more processors are configured to: Export variables BitDepthY and BitDepthC equal to BitDepthY and BitDepthC; Decode the slice QP SliceQPY and offset QpBdOffsetY; deriving StrengthControlVal equal to the value of (SliceQpY+QpBdOffsetY)÷(51+QpBdOffsetY) of the first slice of the current coded picture; and Set StrengthControlVal to the input tensor.
2. The apparatus of claim 1, wherein the processor is further configured to: Parsing a first syntax element in a neural network post filter characteristic (NNPFC) supplemental enhancement information (SEI) message, wherein the first syntax element indicates a purpose of a neural network post-processing filter (NNPF); If pictureRateUpsamplingFlag is equal to 1, there is a second NNPF defined by at least one NNPFC SEI message, and nnpfc_purpose indicates that the resolution resampling flag is equal to 1, The variable CroppedWidth is set equal to the nnpfcOutputPicWidth defined for the second NNPF, and Setting the variable CroppedHeight equal to nnpfcOutputPicHeight defined for the second NNPF; and otherwise, Set the variable CroppedWidth equal to the value of currPic's pic_width_in_luma_samples - SubWidthC * (conf_win_left_offset + conf_win_right_offset), and Set the variable CroppedHeight to the value of currPic's pic_height_in_luma_samples - SubHeightC * (conf_win_top_offset + conf_win_bottom_offset).
3. A device comprising one or more processors, wherein the one or more processors are configured to: Encode BitDepthY and BitDepthC; deriving variables BitDepthY and BitDepthC equal to said BitDepthY and said BitDepthC; Encode the slice QP SliceQPY and offset QpBdOffsetY; deriving StrengthControlVal equal to the value of (SliceQpY+QpBdOffsetY)÷(51+QpBdOffsetY) of the first slice of the current coded picture; and Set StrengthControlVal to the input tensor.
4. A 3D video encoding device, the 3D video encoding device being configured as: Encode atlas frames, geometry frames, occupancy maps, and attribute frames; Export atlas and geometry video streams, occupancy video streams and attribute video streams; as well as 3D information is created based on the geometry frame, the occupancy map and the property frame, wherein the 3D encoding apparatus is further configured to: encoding neural network post filter characteristics (NNPFC) supplemental enhancement information (SEI) and neural network post filter activation (NNPFA) SEI; and A neural network post filter is applied to the geometry frame, the occupancy map, or the attribute frame.
5. A 3D video decoding apparatus, comprising a device according to claim 1 and a processor, wherein the processor is configured to at least limit the purpose of a neural network post filter characteristic (NNPFC) for a geometry video stream, an occupancy video stream or an attribute.
6. A 3D video decoding apparatus, comprising a device according to claim 1 and a processor, wherein the processor is configured to convert an output tensor or an output frame into an integer value when a neural network post-filter is applied to a geometry video stream, an occupancy video stream, or an attribute video stream. 7 . The 3D video decoding apparatus according to claim 6 , wherein the integer-valued target bit depth is a bit depth of a corresponding encoded video. 8 . The 3D video decoding device according to claim 6 , wherein the integer-valued target bit depth corresponds to a bit depth of a nominal format.
9. A 3D video decoding device, comprising a device according to claim 1 and a processor, wherein the processor is configured to decode visual volumetric video based coding neural network post filter information (V3C NNPFI) supplemental enhancement information (SEI) signaled in atlas information, wherein the V3C NNPFI at least indicates which type of neural network is used or which processing is used in a neural network post filter characteristic (NNPFC) SEI.
