3D data coding apparatus and 3D data decoding apparatus

Abstract

To solve the problem in which appropriate refinement cannot be applied to three-dimensional data using a plurality of images.SOLUTION: A 3D data decoding apparatus for decoding coded data and decoding 3D data including attribute information includes a header decoder that decodes characteristics information of refinement and activation information of refinement, and a refiner that performs refinement processing of an attribute frame or a geometry frame. Refinement target information indicating which of an occupancy, a geometry, or an attribute is to be used is decoded from coded data of the characteristics information and refinement is performed using an image specified according to the refinement target information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a 3D data encoding device and a 3D data decoding device. [Background technology]

[0002] In order to efficiently transmit or record 3D data, there are 3D data encoding devices that convert 3D data into 2D images and encode them using a video encoding method to generate encoded data, and 3D data decoding devices that decode and reconstruct 2D images from the encoded data to generate 3D data. There is also a technology that performs filtering on 2D images using auxiliary extension information of a deep learning postfilter.

[0003] Specific examples of 3D data encoding methods include MPEG-I's ISO / IEC 23090-5 V3C (Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression). V3C is used for encoding and decoding point clouds consisting of point positions and attribute information. Furthermore, ISO / IEC 23090-12 (MPEG Immersive Video, MIV) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding, V-DMC), which is currently being standardized, are also used for encoding and decoding multi-viewpoint images and mesh images. In addition, in the auxiliary extension information of the neural network postfilter in Non-Patent Document 1, model information of the neural network is transmitted in the characteristic SEI, and is called for a frame specified in the application SEI, thereby enabling adaptive filtering of point cloud data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] K.Takada, Y.Tokumo, T.Chujoh. T.Ikai,“[V-PCC][EE2.8] V3C neural-network post-filter SEI messages,” ISO / IEC JTC 1 / SC 29 / WG7, m61805, Jan. 2023 Summary of the Invention [Problem to be solved by the invention]

[0005] In Non-Patent Document 1, since filtering is performed only on attributes, there is a problem that filtering using the relationship between occupancy, geometry and attributes cannot be performed. In other words, in a method of decoding 3D data using multiple images and videos, refinement using information between images cannot be performed.

[0006] The present invention aims to solve the above-mentioned problems in encoding and decoding 3D data using a video encoding and decoding method, and to further reduce encoding distortion by using auxiliary information for refinement, thereby encoding and decoding 3D data with high quality. [Means for solving the problem]

[0007] In order to solve the above problem, a 3D data decoding device according to one aspect of the present invention is a 3D data decoding device that decodes encoded data and decodes 3D data including geometry and attributes, the 3D data decoding device comprising: a refinement information decoding unit that decodes refinement characteristic information and refinement activation information from the encoded data; a geometry decoding unit that decodes a geometry frame from the encoded data; an attribute decoding unit that decodes an attribute frame from the encoded data; and a refinement unit that performs a refinement process on the attribute frame or the geometry frame, the refinement information decoding unit decodes refinement target information indicating which of occupancy, geometry, or attribute is to be used from the encoded data of the characteristic information, and performs refinement using an image designated according to the refinement target information. do. Effect of the Invention

[0008] According to one aspect of the present invention, distortion caused by encoding of color images can be reduced, and 3D data can be encoded and decoded with high quality. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram showing the configuration of a 3D data transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 3] FIG. 2 is a diagram for explaining 3D data, occupancy, geometry frames, and attribute frames. [Figure 4] FIG. 11 is a diagram showing the relationship between characteristic SEI and applied SEI according to an embodiment of the present invention. [Diagram 5] 1 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. [Figure 6] FIG. 2 is a functional block diagram showing the configuration of a refinement unit 306. [Figure 7] 1 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. [Figure 8] FIG. 2 is a functional block diagram showing the configuration of a refinement unit 306. [Figure 9] 13 is a flowchart showing a decoding process of SEI. [Figure 10] 13 is a flowchart showing the operation of a refinement unit 306. [Figure 11] 1 shows the syntax of the characteristic SEI. [Figure 12] FIG. 13 is a diagram illustrating an example of refinement target information. [Figure 13] 1 shows part of the syntax of the characteristic SEI. [Figure 14] 1 shows part of the syntax of the characteristic SEI. [Figure 15] 1 shows part of the syntax of the characteristic SEI. [Figure 16] An example of the syntax configuration of an applicable SEI is shown below. [Figure 17] 1 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to an embodiment of the present invention. [Figure 18] 3D data encoding device, 3D data decoding device and SEI MESSAGE [Figure 19] FIG. 13 is a diagram showing an example of a syntax configuration of an application SEI. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] FIG. 1 is a schematic diagram showing the configuration of a 3D data transmission system 1 according to this embodiment.

[0012] The 3D data transmission system 1 is a system that transmits an encoded stream obtained by encoding 3D data to be encoded, decodes the transmitted encoded stream, and displays the 3D data. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31, and a 3D data display device 41.

[0013] 3D data T is input to the 3D data encoding device 11.

[0014] The network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination of these. The network 21 is not necessarily limited to a bidirectional communication network, and may be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. The network 21 may also be a communication network that transmits digital content such as DVDs (Digital Versatile Disc: registered trademark) and Blu-ray Discs (BD: registered trademark). The above-mentioned may be replaced by a storage medium on which the encoded stream Te is recorded, such as a digital still camera (DV camera) or the like.

[0015] The 3D data decoding device 31 decodes each of the encoded streams Te transmitted by the network 21, and generates one or more decoded 3D data Td.

[0016] The 3D data display device 41 displays all or part of one or more pieces of decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 includes a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. Display forms include stationary, mobile, HMD, and the like. Furthermore, when the 3D data decoding device 31 has high processing power, it displays high quality images, and when it has only lower processing power, it displays images that do not require high processing power or display power.

[0017] <operator> The operators used in this specification are listed below.

[0018] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR, |= is the OR assignment operator, and || indicates logical OR.

[0019] x?y:z is a ternary operator that takes y if x is true (non-zero) and z if x is false (zero). Let y..z denote the ordered set of integers from y to z.

[0020] Floor(a) is a function that returns the largest integer less than or equal to a.

[0021] <Structure of the coding stream Te> The data structure of the coded stream Te generated by the 3D data coding device 11 and decoded by the 3D data decoding device 31 will be described.

[0022] 2 is a diagram showing a hierarchical structure of data in the coded stream Te. The coded stream Te has a data structure of either a V3C sample stream or a V3C unit stream. A V3C sample stream includes a sample stream header and a V3C unit, while a V3C unit stream includes a V3C unit.

[0023] A V3C unit includes a V3C unit header and a V3C unit payload. The beginning of a V3C unit (=V3C unit header) is a Unit Type, which is an ID that indicates the type of V3C unit, and takes a value indicated by a label such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, or V3C_OVD.

[0024] If the Unit Type is V3C_VPS (Video Parameter Set), the V3C unit payload includes a V3C parameter set.

[0025] If the Unit Type is V3C_AD (Atlas Data), the V3C unit payload is VPS It includes ID, atlasID, sample stream nal header and multiple NAL units. ID is an abbreviation for Idenfication and takes an integer value equal to or greater than 0. This atlasID may be used as an element of the applicable SEI.

[0026] A NAL unit contains a NALUnitType, a layerID, a TemporalID, and a RBSP (Raw byte sequence payload).

[0027] NAL units are identified by NALUnitType, and are classified as ASPS (Atlas Sequence Parameter Set), AAPS (Atlas Adaptation Parameter Set), ATL (Atlas Tile layer), SEI (Supplemental Enhancement Information), etc.

[0028] The ATL includes an ATL header and an ATL data unit, and the ATL data unit includes information such as the position and size of a patch, such as patch information data.

[0029] The SEI includes payloadType, which indicates the type of SEI, payloadSize, which indicates the size (number of bytes) of the SEI, and sei_payload, which is the data of the SEI.

[0030] If UnitType is V3C_AVD (Attribute Video Data), it includes VPS ID, atlasID, attribute frame ID attrIdx (syntax name vuh_attribute_index), partition ID partIdx (vuh_attribute_partition_index), map ID mapIdx (vuh_map_index), flag auxFlag (vuh_auxiliary_video_flag) indicating whether it is auxiliary data, and video stream. Video stream indicates data such as HEVC or VVC.

[0031] If UnitType is V3C_GVD (Geometry Video Data), it includes VPS ID, atlasID, mapIdx, auxFlag, and video stream.

[0032] If UnitType is V3C_OVD (Occupancy Video Data), it contains the VPS ID, atlasID, and video stream.

[0033] (Data structure of 3D information) In this specification, three-dimensional information (3D data) is a collection of position information (x, y, z) and attribute information in three-dimensional space. For example, 3D data is expressed in the form of a point cloud, which is a group of points representing position information and attribute information in three-dimensional space, or a mesh (or polygon) consisting of the vertices and faces of triangles.

[0034] FIG. 3 is a diagram for explaining 3D data, occupancy frames (two-dimensional images showing occupancy information, hereinafter referred to as occupancy frames), geometric frames (two-dimensional images showing depth or position information, hereinafter referred to as geometric frames), and attribute frames (two-dimensional images showing attribute information, hereinafter referred to as attribute frames). The point clouds and meshes constituting the 3D data are divided into a plurality of parts (areas) by the 3D data encoding device 11, and the point clouds included in each part are projected onto any of the planes of a 3D bounding box (FIG. 3(a)) set in the 3D space. The 3D data encoding device 11 generates a plurality of patches from the projected point clouds. Information on the 3D bounding box (coordinates, size, etc.) and information on mapping onto the projection surface (projection surface, coordinates, size, presence or absence of rotation, etc. of each patch) are called atlas information. The occupancy frame is a two-dimensional image that shows the valid area of ​​each patch (area where point clouds and meshes exist) as a 2D binary image (e.g., valid area is 1, invalid area is 0) (Fig. 3(b). Note that the values ​​of the valid area and invalid area may be values ​​other than 0 and 1, such as 255 and 0. The geometry frame is a two-dimensional image that shows the depth value (distance) of each patch with respect to the projection surface (Fig. 3(c-1)(c-2)). The relationship between the depth value and the pixel value may be linear, or the distance may be derived from the pixel value using a lookup table, a formula, or a relational formula that combines branching by value. The attribute frame is a two-dimensional image that shows the attribute information of the points (e.g., RGB color) (Fig. 3(d-0)(d-1)). Even when projecting in the same direction, the depth of the object surface (distant view and near view) may differ, and mapIdx indicates which depth to project.

[0035] These occupancy frames, geometric frames, attribute frames, and atlas information may be images in which partial images (patches) from different projection planes are mapped (packed) onto a 2D image. Atlas information includes information on the number of patches and the projection planes corresponding to the patches. The 3D data decoding device 31 reconstructs the coordinates and attribute information of the point cloud or mesh from the atlas information, occupancy frames, geometric frames, and attribute frames. Here, a point refers to each point of a point cloud or a vertex of a mesh. Note that instead of the occupancy frames and geometric frames, mesh information (position information) indicating the vertices of the mesh is encoded, decoded, and transmitted. The mesh information may be divided into a base mesh constituting a basic mesh, which is a subset of the mesh, and a mesh displacement, and then encoded, decoded, and transmitted. The mesh displacement is a displacement from the base mesh to indicate a mesh other than the base mesh.

[0036] <Terminology> Atlas: A collection of 2D bounding boxes and related information placed in a rectangular frame that corresponds to the volume in 3D space where the volumetric data is being rendered. Attributes: Scalar or vector properties arbitrarily associated with each point in a volumetric frame, such as color, reflectivity, surface normal, transparency, material ID, etc. Attribute Frame: A two-dimensional rectangular array created as a collection of patches containing values ​​for a particular attribute. Attribute Map: An attribute frame containing attribute patch information projected to a particular depth indicated by the corresponding geometry map. Bit Stream: An ordered series of bits that form a coded representation of data. Encoded Data. Coded Atlas Sequence (CAS): A sequence of Coded Atlas AUs (access units) of an IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 1, followed in decoding order by a Coded Atlas AU that is not an IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 0 or greater and 1. All subsequent Coded Atlas AUs up to, but not including, the IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 1 are included. The IRAP coded atlas AU may be an instantaneous decoding refresh (IDR) coded access unit, a broken link access (BLA) coded access unit, or a clean random access (CRA) coded access unit. Geometry: A set of Cartesian coordinates relative to a volumetric frame. Geometry Frame: A 2D array created by aggregating the geometry patch information associated with each patch. Geometry Map: A geometry frame containing geometry patch information projected to a particular depth. Occupancy: A value indicating whether an atlas sample corresponds to its associated sample in 3D space. Occupancy Frame: A collection of occupancy values ​​organized as a 2D array, representing the total occupancy of one atlas frame. Patch: A rectangular region associated with volumetric information in the atlas.

[0037] (Overview of characteristic SEI and application SEI) FIG. 4(a) is a diagram showing the relationship between a refinement characteristic SEI (hereinafter, characteristic SEI, nn_refinement_characteristics SEI, NNRC_SEI, characteristic information) and a refinement application SEI (hereinafter, application SEI, nn_refinement_activation SEI, NNRA_SEI, activation SEI, activation information) according to an embodiment of the present invention. The NNC decoding unit decodes a compressed NN (Neural Network) model transmitted in the characteristic SEI to derive an NN model. MPEG Neural Network Coding (ISO / IEC 15938-19) may be used for encoding and decoding the NN model. The refinement unit 306 includes an NN filter unit 611, which performs a filtering process using the derived NN model. The NN filter unit 611 will be described in detail later. The characteristic SEI used for refinement is specified by a syntax element of the application SEI, for example, nnra_target_id. The applied SEI transmits persistence information using a syntax element and specifies that the attribute frame is to be filtered for a duration specified by the persistence information. The NN filter unit 611 may include an NNC decoder. The applied SEI (activation information) is not limited to the SEI in FIG. 16, and may be data including NAL units and atlas data.

[0038] FIG. 4B is another diagram showing the relationship between the refinement characteristics SEI and the refinement application SEI according to the embodiment of the present invention. The refinement unit 306 includes a linear filter unit (ALF unit) 610. The meter decoder decodes the syntax of the filter parameters transmitted in the characteristic SEI and derives the coefficients of a linear filter. This is not limited to a loop filter, but may be a post filter. Although this is described as linear, it may also include nonlinear elements such as simple clipping and squared components. The ALF unit 610 performs filtering on the attribute frame or geometry frame using the derived linear model.

[0039] (Configuration of 3D data decoding device) 5 and 7 are functional block diagrams showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention.

[0040] The 3D data decoding device 31 is composed of a V3C unit decoding unit 301, an atlas decoding unit 302 (refinement information decoding unit), an occupancy decoding unit 303, a geometry decoding unit 304, an attribute decoding unit 305, a post-decoding conversion unit 308, a pre-reconstruction unit 310, a reconstruction unit 311, and a post-reconstruction unit 312.

[0041] The V3C unit decoder 301 receives coded data (bit stream) in a byte stream format, ISO Base Media File Format (ISOBMFF), or the like, and decodes the V3C unit header and V3C VPS. The V3C unit decoder 301 selects an atlas decoder 302, an occupancy decoder 303, a geometry decoder 304, or an attribute decoder 305 according to the UnitType of the V3C unit header. When the UnitType is V3C_AD, the atlas decoder 302 is used, and when the UnitType is V3C_OVD, V3C_GVD, or V3C_AVD, the occupancy decoder 303, the geometry decoder 304, or the attribute decoder 305 are used to decode the occupancy frame, the geometry frame, and the attribute frame, respectively.

[0042] The atlas decoding unit 302 receives the atlas data and decodes the atlas information.

[0043] The atlas decoding unit 302 (refinement information decoding unit) decodes a characteristic SEI indicating the characteristics of the refinement process from the encoded data. The refinement information decoding unit decodes information on the target to which the refinement is applied (refinement target information). Furthermore, the atlas decoding unit 302 decodes the application SEI from the encoded data.

[0044] The atlas decoding unit 302 decodes an identifier atlasID indicating target atlas information indicating the target of refinement from a V3C unit including the applicable SEI.

[0045] The occupancy decoding unit 303 decodes occupancy data encoded by VVC, HEVC, or the like, and outputs an occupancy frame DecOccFrames[frameIdx][compIdx][y][x]. Here, DecOccFrames, frameIdx, compIdx, y, and x respectively indicate the decoded occupancy frame, frame ID, component ID, row index, and column index. In DecOccFrames, compIdx=0 may be used.

[0046] The geometry decoding unit 304 decodes geometry data encoded by VVC, HEVC, etc., and outputs a geometry frame DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]. Here, DecGeoFrames, frameIdx, mapIdx, compIdx, y, and x indicate the decoded geometry frame, frame ID, map ID, component ID, row index, and column index, respectively. DecGeoBitDepth, DecGeoHeight, DecGeoWidth, and DecGeoChromaFormat are expressed as the bit depth of the geometry frame, the height of the geometry frame, the width of the geometry frame, and the chroma format of the geometry frame. The decoded geometry frame may include multiple geometry maps (geometry frames with different depth projections), and mapIdx is used to distinguish the maps. In DecGeoFrames, compIdx=0 may be used.

[0047] The attribute decoding unit 304 decodes attribute data encoded by VVC, HEVC, etc., and outputs an attribute frame DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]. Here, DecAttrFrames, frameIdx, attrIdx, partIdx, mapIdx, compIdx, y, and x indicate the decoded attribute frame, frame ID, attribute ID, partition ID, map ID, component ID, row index, and column index, respectively. DecAttrBitDepth, DecAttrHeight, DecAttrWidth, and DecAttrChromaFormat are indicated as the attribute frame, the bit depth of the attribute frame, the height of the attribute frame, the width of the attribute frame, and the chroma format of the attribute frame. The decoded attribute frame may include multiple attribute maps (attribute frames of different depth projections), and mapIdx is used to distinguish the maps. The decoded attribute frame is composed of multiple attributes such as color (R, G, B), reflectance, alpha, and normal direction. Multiple attributes can be transmitted in multiple attribute data, and attrIdx is used to distinguish them. For example, {R, G, B} is attribute data 0 (attrIdx=0), {reflection} is attribute data 1 (attrIdx=1), and {alpha} is attribute data 2 (attrIdx=2). Attributes can be divided into multiple bitstreams for transmission, and partIdx is used to distinguish them. mapIdx is as described above.

[0048] The post-decoding conversion unit 308 receives the decoded atlas information, the decoded occupied frames DecOccFrames, the decoded geometric frames DecGeoFrames, and the decoded attribute frames DecAttrFrames, and converts them into a nominal format. The post-decoding conversion unit 308 outputs the nominal format OccFramesNF[frameIdx][CompTimeIdx][y][x], GeoFramesNF[mapIdx][CompTimeIdx][frameIdx][y][x], and AttrFramesNF[attrIdx][mapIdx][CompTimeIdx][compIdx][y][x] of the occupied frames, geometric frames, and attribute frames. Here, frameIdx, CompTimeIdx, y, x, mapIdx, attrIdx, and compIdx respectively indicate the frame ID, the composition time index, the row index, the column index, the map ID, the attribute ID, and the component ID.

[0049] Nominal format collectively refers to the nominal bit depth, resolution, chroma format, and constituent time indexes to which the decoded video should be converted. Each video sub-bitstream and each region of the packed video sub-bitstream is associated with a nominal bit depth. This is the expected target bit depth for all operations for reconstruction. The nominal bit depth for occupancy bit depth OccBitDepthNF is set to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1, or pin_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1. oi_occupancy_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which the occupancy frames of the atlas with atlasID=j should be converted. pin_occupancy_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which the decoded region containing occupancy data of the atlas with atlasID=j should be converted. pin_occupancy_present_flag[j] equal to 0 indicates that the packed video frame of the atlas with atlasID=j does not contain any regions with occupancy data. pin_occupancy_present_flag[j] equal to 1 indicates that the packed video frame of the atlas with atlasID=j has regions with occupancy data. If pin_occupancy_present_flag[j] is not present, its value is inferred to be equal to 0. The nominal bit depth of each geometry frame, GeoBitDepthNF, is set to gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 if pin_geometry_present_flag[ConvAtlasID] is equal to 1. gi_geometry_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which all geometry frames of the atlas with atlasID=j will be transformed. pin_geometry_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which the decoded region containing geometry data of the atlas with atlasID=j will be transformed.pin_geometry_present_flag[j]=0 indicates that the packed video frame of the atlas with atlasID=j does not contain any region with geometry data. pin_geometry_present_flag[j]=1 indicates that the packed video frame of the atlas with atlasID=j has a region with geometry data. If pin_geometry_present_flag[j] is not present, its value is equal to 0. Finally, if pin_attribute_present_flag[ConvAtlasID]=1, the nominal bit depth AttrBitDepthNF[attrIdx] of each attribute frame with attrIdx is set to ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1 or pin_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]. ai_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which all attribute frames with attrIdx=i are converted for the atlas with atlasID=j. pin_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which regions containing the attribute with attrIdx=i are converted for the atlas with atlasID=j. pin_attribute_present_flag[j] = 0 indicates that the packed video frame of the atlas with atlasID=j does not contain a field for attribute data. pin_attribute_present_flag[j] = 1 indicates that the packed video frame of the atlas with atlasID=j does contain a field for attribute data.

[0050] ConvAtlasID is set equal to vuh_atlas_id or determined by external means if the V3C unit header is not available. vuh_atlas_id specifies the ID of the atlas that corresponds to the current V3C unit, as indicated in the V3C unit headers such as V3C_AD, V3C_OVD, V3C_GVD, and V3C_AVD.

[0051] asps_frame_width is the frame width of the atlas in integer samples, which correspond to the luma samples of the video component. It is a V3C bitstream conformance requirement that asps_frame_width be equal to the value of vps_frame_width[j], where j is the current atlasID. asps_frame_height is the frame height of the atlas in integer samples, which correspond to the luma samples of the video component. It is a V3C bitstream conformance requirement that asps_frame_height be equal to the value of vps_frame_height[j], where j is the current atlasID. The nominal frame resolution of the auxiliary video component is defined by its nominal width and height, specified by the variables AuxVideoWidthNF and AuxVideoHeightNF, respectively. AuxVideoWidthNF and AuxVideoHeightNF are obtained from the auxiliary video sub-bitstream associated with the atlas.

[0052] The nominal chroma format is defined as 4:4:4.

[0053] The post-decoding transform unit 308 consists of bit depth conversion, resolution conversion, output order conversion, atlas composition alignment, atlas dimension alignment, chroma upsampling, geometry map synthesis, and attribute map synthesis. The video frames provided by the V3C decoder unit 309 may require additional processing steps before being input into the reconstruction process. Such processing steps may include conversion of the decoded frames to a nominal format (e.g., resolution, bit depth, chroma format, etc.). The nominal format information is signaled in the V3C VPS.

[0054] The pre-reconstruction unit 310 can receive the decoded atlas information, the decoded occupancy frames, the decoded geometry frames, and the decoded attribute frames and refine / modify them. Specifically, when the occupancy synthesis flag os_method_type[k] is equal to 1 indicating patch border filtering, the occupancy synthesis is started with OccFramesNF[compTimeIdx][0] and GeoFramesNF[0][compTimeIdx][0] as inputs and the modified array OccFramesNF[compTimeIdx][0] as output. OccFramesNF indicates the occupancy frames decoded in nominal format, and GeoFramesNF indicates the geometry frames decoded in nominal format. compTimeIdx is the index of synthesis time.

[0055] The refinement unit 306 performs refinement on occupancy, geometry, and attributes on a frame-by-frame basis within the pre-reconstruction unit 310. Refinement is performed by inputting any one of occupancy, geometry, and attributes, and refining the occupancy, geometry, and attributes. The refinement unit 306 may be a filter that outputs one of the two-dimensional occupancy, geometry, and attributes. The refinement unit 306 may perform refinement processing by simultaneously using two-dimensional occupancy, geometry, and attributes.

[0056] Furthermore, the refinement unit 306 may include an NN filter unit 611 or an ALF unit 610, and may perform filtering based on the transmitted neural network parameters or linear parameters.

[0057] The reconstruction unit 311 inputs atlas information, occupancy, and geometry, and reconstructs the positions, attributes, or meshes of vertices or point clouds in 3D space. The reconstruction unit 311 reconstructs point cloud data or meshes of 3D data based on the reconstructed geometry information (e.g., recPcGeo) and attribute information (e.g., recPcAttr). Specifically, the reconstruction unit 311 inputs the nominal video frame derived by the pre-reconstruction unit 310, OccFramesNF[compTimeIdx][0][y][x], GeoFramesNF[mapIdx][compTimeIdx][0][y][x], and AttrFramesNF[attrIdx][mapIdx][compTimeIdx][ch][y][x] to reconstruct point cloud data or meshes of 3D data. AttrFramesNF indicates an attribute frame decoded in the nominal format. The reconstruction unit 311 derives a variable pointCnt as the number of points in the reconstructed point cloud frame, a one-dimensional array pointToPatch[pointCnt] as the patch index corresponding to each reconstructed point, and a two-dimensional array pointToPixel[pointCnt][dimIdx] as the atlas coordinates corresponding to each reconstructed point. There is a 2D array recPcGeo[pointCnt][dimIdx] as a list of coordinates corresponding to each reconstructed point, and a 3D array recPcAttr[pointCnt][attrIdx][compIdx] as attributes related to points in the reconstructed point cloud frame. Here, pointCnt, attrIdx, and compIdx correspond to the size of the reconstructed point, the ID of the attribute frame, and the component ID, respectively. dimIdx represents the (x, y) components of each reconstructed point. Specifically, the reconstruction unit 311 derives recPcGeo and recPcAttr as follows. m = 0 for(k=0; k<3; k++) { for(pointIdx=0; pointIdx<AtlasPatchRawPoints[pIdx]; pointIdx++) { y = AtlasPatch2dPosY[pIdx] + (m / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (m % AtlasPatch2dSizeX[pIdx]) rawPos1D[m] = gFrame[mapIdx][y][x] m++ } } for(pointIdx=0; pointIdx<AtlasPatchRawPoints[pIdx]; pointIdx++) { recPcGeo[pointCnt][0] = rawPos1D[pointIdx] + AtlasPatch3dOffsetU[pIdx] recPcGeo[pointCnt][1] = rawPos1D[pointIdx + AtlasPatchRawPoints[pIdx]] + AtlasPatch3dOffsetV[pIdx] recPcGeo[pointCnt][2] = rawPos1D[pointIdx + 2 * AtlasPatchRawPoints[pIdx]] + AtlasPatch3dOffsetD[pIdx] y = AtlasPatch2dPosY[pIdx] + (pointIdx / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (pointIdx % AtlasPatch2dSizeX[pIdx]) for(attrIdx=0; attrIdx<ai_attribute_count[RecAtlasID]; attrIdx++) { attrDim = ai_attribute_dimension_minus1[RecAtlasID][attrIdx] + 1 for(compIdx=0; compIdx <attrDim; compIdx++) { recPcAttr[pointCnt][attrIdx][compIdx] = aFrame[attrIdx][compIdx][y][x] } } if(ai_attribute_count[RecAtlasID]>0) { attrPresent[pointCnt] = 1 } pointToPixel[pointCnt][0] = -1 pointToPixel[pointCnt][1] = -1 pointToPatch[pointCnt] = pIdx pointCnt++ } Here, pIdx is the patch index, compTimeIdx is the composition time index, rawPos1D is the 1D position, and gFrame and aFrame are the geometry frame and attribute frame, respectively. TilePatch3dOffsetU is the associated tile patch parameter from the patch. ai_attribute_count[j] indicates the number of attributes associated with the atlas with atlasID=j. AtlasPatch3dOffsetU, AtlasPatch3dOffsetV, and AtlasPatch3dOffsetD are parameters indicating the position of the 3D bounding box in Figure 3(a) on the 3D model.

[0058] for(k=0; k<3; k++) { for(pointIdx=0; pointIdx <AtlasPatchRawPoints[pIdx]; pointIdx++) { y = AtlasPatch2dPosY[pIdx] + (m / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (m % AtlasPatch2dSizeX[pIdx]) rawPos1D[m] = gFrame[mapIdx][y][x] m++ } } Here, AtlasPatchRawPoints, AtlasPatch2dPosX, AtlasPatch2dPosY, AtlasPatch2dSizeX, and AtlasPatch2dSizeY are patch information derived from the atlas information by the atlas decoding unit 302. The arrays oFrame[y][x], gFrame[mapIdx][y][x], and aFrame[mapIdx][attrIdx][compTimeIdx][y][x] are derived as follows: for(j=0; j <asps_frame_height; j++) { for(i=0; i <asps_frame_width; i++) { oFrame[j][i] = OccFramesNF[compTimeIdx][0][j][i] for(m=0; m <asps_map_count_minus1+1; m++) { gFrame[m][j][i] = GeoFramesNF[m][compTimeIdx][0][j][i] for(a=0; a <ai_attribute_count[RecAtlasID]; a++) { for(c=0; c <ai_attribute_dimension_minus1[RecAtlasID][a]+1; c++) { aFrame[m][a][c][j][i] = AttrFramesNF[a][m][compTimeIdx][c][j][i] } } } } } Here, ai_attribute_dimension_minus1+1 indicates the total number of dimensions (i.e., the number of channels) of the attributes signaled by the V3C VPS and decoded by the V3C unit decoder 301. asps_map_count_minus1+1 indicates the number of maps used to encode the geometry data and attribute data of the current atlas. RecAtlasID indicates the ID of the decoded atlas.

[0059] The post-reconstruction unit 312 modifies (updates) the 3D data point cloud data or mesh after processing by the reconstruction unit 311. The post-reconstruction unit 312 receives pointCnt as the number of reconstructed points of the current point cloud frame related to the current atlas, one-dimensional arrays attrBitDepth[] as the nominal bit depth, oFrame[][], recPcGeo[][], and recPcAttr[][][], applies geometric smoothing, and outputs modified recPcGeo[][] and recPcAttr[][][].

[0060] (Refinement Part 306) FIG. 6 is a functional block diagram showing the configuration of the refinement unit 306 (306DEC).

[0061] A refinement unit 306 (306DEC) applies refinement to DecOccFrames, DecGeoFrames, and DecAttrFrames that have been decoded by the occupancy decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305 and have not yet been converted by the post-decoding conversion unit 308. The refinement unit 306 (306DEC) then outputs the refined DecOccFrames, DecGeoFrames, and DecAttrFrames to the post-decoding conversion unit 308.

[0062] The refinement unit 306 inputs DecOccFrames[0][frameIdx][y][x], mapIdx, DecGeoFrames[mapIdx][frameIdx][0][y][x] for each frameIdx, and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] for each attrIdx, partIdx, mapIdx, frameIdx, and compIdx according to the refinement target information, and outputs modified DecOccFrames, DecGeoFrames, and DecAttrFrames. The refinement unit 306 stores the output in DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x], and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] according to the refinement target information.

[0063] The refinement unit 306 may perform refinement processing on the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1). In this case, oFrame=DecOccFrames[0][frameIdx], gFrame=DecGeoFrames[mapIdx][frameIdx][0], and aFrame=DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx] may be input. Here, attrIdx and mapIdx may be values ​​nnrc_map_index and nnrc_attribute_index obtained by decoding the NNRC SEI. attrIdx=nnrc_attribute_index, mapIdx=nnra_map_index. The frameIdx may be a compTimeIdx that is an index of the composition time. Also, attrIdx and mapIdx may be the values ​​nnra_map_index and nnra_attribute_index obtained by decoding the NNRA SEI. attrIdx = nnra_attribute_index, mapIdx = nnrc_map_index. The same applies below.

[0064] According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristic SEI, and either occupancy, geometry, or attribute is specified using the refinement target information specified in the characteristic SEI, and refinement processing is applied to multiple inputs simultaneously, thereby improving image quality.

[0065] FIG. 7 shows an example in which the refinement unit 306 applies refinement to an image signal (video signal) in a nominal format.

[0066] 8 is a functional block diagram showing the configuration of the refinement unit 306 (306NF). The refinement unit 306 (306NF) applies refinement to the OccFramesNF, GeoFramesNF, and AttrFramesNF in the nominal format converted by the post-decoding conversion unit 308 according to the refinement target information, and outputs the OccFramesNF, GeoFramesNF, and AttrFramesNF after the refinement to the pre-reconstruction unit 310. The processing of the refinement unit 306 will be explained below, but the processing of the refinement unit 306DEC is performed by replacing the OccFramesNF, GeoFramesNF, and AttrFramesNF with DecOccFrames, DecGeoFrames, and DecAttrFrames in FIG. 6.

[0067] The refinement unit 306 may perform refinement processing on the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1). In this case, oFrame = OccFramesNF[compTimeIdx][0], gFrame = GeoFramesNF[mapIdx][compTimeIdx][frameIdx][0], and aFrame = AttrFramesNF[attrIdx][mapIdx][compTimeIdx] may be input.

[0068] 10(a), when the refinement target information indicates an occupancy (S3062I), the occupancy is added to the input tensor (S3063I). For example, the following (IN-OCC) may be used.

[0069] ch = 0 if (OccupancyUsedFlag) { (expression IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yP][xP]) ch++ } If the refinement target information indicates a geometry (S3064I), the geometry is added to the input tensor (S3065I). For example, the following (formula IN-GEO) may be used.

[0070] if (GeometryUsedFlag) { (expression IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yP][xP]) ch++ } If the refinement target information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, the following (IN-ATTR) may be used.

[0071] if (AttributeUsedFlag) { (formula IN-ATTR) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[0][yP][xP]) ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[1][yP][xP]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[2][yP][xP]); ch++ According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristics SEI. Using the refinement target information specified in the characteristics SEI, one of occupancy, geometry, and attributes is specified, and refinement processing is applied simultaneously to multiple inputs, resulting in improved image quality.

[0072] (NN filter section 611) The NN filter unit 611 performs filtering using a neural network. The neural network is represented by a neural network model and may include a convolution (Conv).

[0073] A neural network model (hereinafter, NN model) refers to the elements and connection relationships (topology) of a neural network, and the parameters (weights, bias) of the neural network. The NN filter unit 611 may fix the topology and switch only the parameters according to the image to be filtered. The neural network may also include a convolution defined by the kernel size, the number of input channels, and the number of output channels.

[0074] The input to the refinement unit 306 is DecFrame. The refinement unit 306 derives an input InputTensor to the NN filter unit 611 from the input image DecFrame, and the NN filter unit 611 performs filtering using a neural network model using the inputTensor to derive an outputTensor. The neural network model used is a model corresponding to nnra_target_id. The input image may be an image for each component, or may be an image having multiple components as channels.

[0075] The NN filter unit 611 may repeatedly apply the following process.

[0076] The NN filter unit 611 performs a convolution operation (conv, convolution) on the inputTensor and kernel k[m][n][yy][xx] for the number of layers, and generates an output image outputTensor by adding bias. Here, m is the number of channels in the inputTensor, n is the number of channels in the outputTensor, yy is the height of kernel k, and xx is the width of kernel k.

[0077] At each layer, an outputTensor is generated from an inputTensor.

[0078] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Here, nn=0..n-1, mm=0..m-1, yy=0..height-1, xx=0..width-1, i=0..yy-1, j=0..xx-1. width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. Σ is the sum of mm=0..m-1, i=0..yy-1, j=0..xx-1, respectively. of is the width or height of the area required around the inputTensor to generate the outputTensor.

[0079] In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, set of=0. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, set of=1. When the value of yy+j-of is less than 0 or greater than or equal to height, or when the value of xx+i-of is less than 0 or greater than or equal to width, the value of inputTensor[mm][yy+j-of][xx+i-of] may be set to 0. Alternatively, the value of inputTensor[mm][yy+j-of][xx+i-of] may be set to inputTensor[mm][yclip][xclip], where yclip is max(0, min(yy+j-of, height-1)) and xclip is (0, min(xx+i-of, width-1)).

[0080] In the next layer, the obtained outputTensor is used as a new inputTensor, and the same process is repeated for the number of layers. An activation layer may be placed between layers. A pooling layer or skip connection may also be used. Finally, a FilteredFrame is derived from the obtained outputTensor.

[0081] In addition, a process called Depth-wise Conv, which is shown in the following formula, may be performed using a kernel k[n][yy][xx], where nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0082] outputTensor[nn][yy][xx]=ΣΣ(k[nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Also, a nonlinear process called Activate, for example, ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used.

[0083] leakyReLU(x) = x >= 0 ? x : a*x Here, a is a predetermined value less than 1, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the above values ​​of k, bias, and a may be set to integers, and a right shift may be performed after conv to generate the outputTensor.

[0084] With ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, with leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. With ReLU, the gradient for values ​​less than 0 disappears, which can make it difficult for learning to progress. With leakyReLU, the gradient for values ​​less than 0 remains, making the above problem less likely to occur. Of the above leakyReLU(x), PReLU, which uses a parameterized value of a, may also be used.

[0085] (Filtering process of NN filter unit 611) The NN filter unit 611 derives input data inputTensor[][][] to the NN filter unit 611 based on one or more of the occupancy frame oFrame, the geometric frame gFrame, and the attribute frame aFrame. The following is one configuration example, and the configuration described later may also be used.

[0086] for(yP = -overlapSize; yP < blockHeight+overlapSize; yP++) for(xP = -overlapSize; xP < blockWidth+overlapSize; xP++) { yT = cTop + yP xT = cLeft + xP ch=0 if (OccupancyUsedFlag) { (Equation IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yT][xT]) ch++ } if (GeometryUsedFlag) { (Equation IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yT][xT]) ch++ } if (AttributeUsedFlag) { (Equation IN-ATTR) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[0][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[1][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[2][yT][xT]); ch++ } } } if(nnrc_auxiliary_inp_idc == 1) { inputTensor[ch][yP+overlapSize][xP+overlapSize] = InpS(nnra_strength_idc) } nnra_strength_idc is a parameter that is entered into the input tensor to indicate the strength of refinement. Note that InpSampleVal(y, x, FrameHeight, FrameWidth, targetPic) { if(y<0 || x<0 || y>=FrameHeight || x>=FrameWidth) { sampleVal = 0 sampleVal = targetPic[Clip3(0, picHeight-1, y)][Clip3(0, picWidth-1, x)]} } else { sampleVal = targetPic[y][x] } It may be defined as follows using the above. Instead of Inp(oFrame[yT][xT]), it may be defined as Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth,oFrame)) which includes screen boundary processing. Instead of Inp(gFrame[yT][xT]), you can also use Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth,gFrame)), which includes screen boundary processing. Instead of Inp(aFrame[k][yT][xT]), you can use Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth, aFrame[k])), which includes screen boundary processing. k=0,1,2,..., etc. And so on. Here, overlapSize is the size of the overlap, and a value decoded from the encoded data of the characteristic SEI may be used.

[0087] The NN filter unit 611 performs NN filtering to derive an outputTensor from an inputTensor. Refinement processing (filtering) indicated by RefineFilter() may be performed in units of blocks (blockWidth x blockHeight) as shown below.

[0088] for(cTop=0;cTop <FrameHeight;cTop+=blockHeight) for(cLeft=0;cLeft <FrameWidth;cLeft+=blockWidth){ DeriveInputTensors() outputTensor=RefineFilter(inputTensor) StoreOutputTensors() } Here, DeriveInputTensors() is a function that sets the input data, and StoreOutputTensors() is a function that stores the output data. FrameWidth and FrameHeight are the size of the input data. BlockWidth and blockHeight are the width and height of the block. An example of the processing of StoreOutputTensors() is shown below.

[0089] for(yP=0;yP <blockHeight; yP++) for(xP=0; xP <blockWidth; xP++) { yT = cTop + yP xT = cLeft + xP if (yT <FrameHeight && xT<FrameWidth) ch=0 if (OccupancyUsedFlag) { oFrame[yT][xT]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } if (GeometryUsedFlag) { gFrame[yT][xT]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } if (AttributeUsedFlag) { aFrame[0][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[1][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[2][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } } (ALF section 610) The refinement specified by the persistence information of the NNRA SEI and the identifier of the characteristic SEI may be performed using a filter (Wiener Filter) using a linear model. The linear filter may perform filtering processing in the ALF unit 610. Specifically, the filter target image DecFrame is divided into small regions (x=xSb..xSb+bSW-1, y=xSb..xSb+bSH-1) of a certain size (for example, 4x4, 1x1). (xSb, ySb) are the upper left coordinates of the small region, and bSW, bSH are the width and height of the small region. Then, filtering is performed on a small region basis. The refined image outFrame is derived from DecFrame using the selected filter coefficient coeff[].

[0090] outframe[cIdx][y][x] = Σ(coeff[i] * DecFrame[cIdx][y+ofy][x+ofx] + offset) >> shift Here, ofx and ofy are the offsets of the reference position determined according to the filter position i. offset=1<<(shift-1). shift corresponds to the precision of the filter coefficients and is a constant such as 6, 7, 8, etc.

[0091] Note that the image may be classified in units of small regions, and the filter coefficient coeff[classId][] may be selected according to the classId (=classId[y][x]) of the derived small region to perform the filtering process.

[0092] outframe[cIdx][y][x] = (Σ(coeff[classId][i] * DecFrame[cIdx][y+ofy][x+ofx] + offset) >> shift Here, DecFrame[ch] = inputTensor[ch][yP+overlapSize][xP+overlapSize], that is, the following is also acceptable.

[0093] ch=0 if (OccupancyUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(oFrame[yP][xP]) ch++ } if (GeometryUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(gFrame[yP][xP]) ch++ } if (AttributeUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[0][yP][xP]); ch++ DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[1][yP][xP]); ch++ DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[2][yP][xP]); ch++ } Note that classId may be derived using the activity or directionality of a block (pixel in the case of 1x1). For example, classId may be derived using activity Act derived from the sum of absolute difference as shown below.

[0094] classId=Act Act may be derived as follows:

[0095] Act = Σ |x-xi| Here, i=0..7 and xi indicates the adjacent pixels of the target pixel x. The directions of the adjacent pixels may be eight directions, including up, down, left, right, and four directions at 45 degrees with respect to i=0..7. Act = Σ|-xl+2*x-xr|+Σ|-ya+2*x-yb| may be used. Note that l, r, a, and b are the opposites of left, right, above, and below, respectively, and indicate the pixels to the left, right, top, and bottom of x. Act=Min(Num-1, Act>>shift) It is also possible to quantize the signal by the shift value as shown above, and then clip the resulting signal to NumA-1. Furthermore, classId may be derived using the directionality D using the following formula.

[0096] classId=Act+D*NumA Here, for example, Act=0..NumA-1, D=0..4 etc.

[0097] The ALF unit 610 outputs the OutFrame as it is as a FilteredFrame.

[0098] (Decryption of applicable SEI and application of filter) 9 is a diagram showing a flowchart of the process of the 3D data decoding device. The 3D data decoding device performs the following process including decoding the applied SEI message.

[0099] S6001: The atlas decoding unit 302 decodes nnra_cancel_flag and nnra_target_id from the applicable SEI. nnra_cancel_flag may be nnrc_cancel_flag.

[0100] S6002: If nnra_cancel_flag is 1, end the processing for the frame that is the target of nnra_cancel_flag. If nnra_cancel_flag is 0, proceed to S6003.

[0101] S6003: The atlas decoding unit 302 decodes the nnra_persistence_flag from the applicable SEI.

[0102] S6005: Identify a characteristic SEI having the same nnrc_id as the nnra_target_id, and derive parameters of the NN model from the characteristic SEI.

[0103] S6006: The NN filter unit 611 and the ALF unit 610 execute refinement (filtering) processing using the parameters of the derived NN model.

[0104] <Syntax configuration example> (Neural network refinement characteristics SEI) Figure 11 shows the syntax of nn_refinement_characteristics(payloadSize) (characteristic SEI). The argument payloadSize represents the number of bytes of this SEI message.

[0105] The persistence scope of a property SEI is the CAS, i.e., the SEI is applied on a CAS basis.

[0106] The syntax element nnrc_id indicates the ID of a characteristic SEI. In the application SEI, the refinement process to be applied is specified by transmitting the value of nnrc_id of the characteristic SEI indicating the refinement characteristic to be applied as the target ID (nnra_target_id). If nnrc_mode_idc is 0, then this SEI message contains an ISO / IEC 15938-17 bit stream and specifies a base NNRE (neural-network refinement) or indicates that it is an update from a base NNRE with the same nnrc_id value. If the NNRC SEI message is the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, then if nnrc_mode_idc is equal to 1, then the base NNRE for that nnrc_id value is identified as the neural network identified by the URI indicated by nnrc_uri, in the format identified by the tag URI nnrc_tag_uri. If the NNRC SEI message is neither the first NNRC SEI message in decoding order with a particular nnrc_id value in the current CAS nor a repetition of the first NNRC SEI message in decoding order, and nnrc_mode_idc is equal to 1, it indicates that updates to a base NNRE with the same nnrc_id value are defined by the URI indicated in nnrc_uri, which is in the format identified by the tag URI nnrc_tag_uri. The value of nnrc_mode_idc shall be in the range 0 to 1 (inclusive) in a bitstream that conforms to this document. If this SEI message is the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, then the RefineFilter() is assigned the same as the base NNRE. If this SEI message is neither the first NNRCSEI message in decoding order with a particular nnrc_id value in the current CAS, nor a repeat of the first NNRCSEI message in decoding order, apply the updates defined in this SEI message to the base NNRE to obtain the RefineFilter(). Updates are not cumulative, and each update applies to the base NNRE, which is the NNRE specified in the first NNRC SEI message (in decoding order) with a particular nnrc_id value within the current CAS.

[0107] nnrc_reserved_zero_bit_a shall be equal to 0 in a bitstream conforming to this document. Decoders shall ignore NNRCSEI messages where nnrc_reserved_zero_bit_a is not 0.

[0108] The nnrc_tag_uri contains a tag URI with syntax and semantics specified in IETF RFC4151 that identifies the format and associated information of a neural network to be used as a base NNRE or an update to a base NNRE with the same nnrc_id value specified in the nnrc_uri. The tag is identified by the nnrc_uri, which is equal to "tag:iso.org,2023:15938-17". This indicates that the neural network data used complies with ISO / IEC 15938-17.

[0109] The nnrc_uri contains a URI with syntax and semantics specified in IETF Internet Standard 66 that identifies the neural network to be used as a base NNRE, or an update to a base NNRE with the same nnrc_id value.

[0110] nnrc_property_present_flag equal to 1 indicates the presence of syntax elements related to input format, output format, and complexity. nnrc_property_present_flag equal to 0 indicates the absence of syntax elements related to input format, output format, and complexity. nnrc_property_present_flag shall be equal to 1 when this SEI message is the first NNRC SEI message, in decoding order, with a particular nnrc_id value within the current CAS. When nnrc_property_present_flag is equal to 0, the values ​​of all syntax elements that can only be present when nnrc_property_present_flag is equal to 1, for which no inferred value is specified, are inferred to be the same as their corresponding syntax elements in the NNRC SEI message that contains the base NNRE that this SEI updates.

[0111] nnrc_base_flag equal to 1 indicates that the SEI message specifies a base NNRE. nnrc_base_flag equal to 0 indicates that the SEI message specifies an update relative to a base NNRE. If not present, the value of nnrc_base_flag is inferred to be equal to 0. The following constraints apply to the value of nnrc_base_flag: - If the NNRC SEI message is the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, the value of nnrc_base_flag shall be equal to 1. - If the NNRC SEI message nnrcB is not the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS and the value nnrc_base_flag is equal to 1, then the NNRC SEI message is a repeat of the first NNRC SEI message nnrcA with the same nnrc_id in decoding order, i.e. the payload content of the next nnrcB shall be the same as that of nnrcA. When the NNRC SEI message is not the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, and is not a repeat of the first NNRC SEI message with that particular nnrc_id, the following applies: -This SEI message defines an update to the preceding base NNRE in decoding order with the same nnrc_id value.

[0112] - This SEI message relates to the current decoded picture and the decoded pictures of all subsequent layers in output order up to the end of the current CAS or up to the pictures that follow the current decoded picture in output order within the current CAS, excluding the decoded picture associated with a subsequent NNRC SEI message in decoding order with the specified nnrc_id value within the current CAS, whichever comes first.

[0113] The syntax element nnrc_input_mode indicates the target of the refinement process (refinement target information). The refinement target information includes at least attributes. It may also include geometry and occupancy.

[0114] If the refinement target information indicates that an attribute is included, the input channel of the attribute may be specified. Furthermore, the attrID may be specified. Furthermore, the mapID of the geometry and attribute may be specified.

[0115] The refinement target information may be information for selecting {occupancy, attribute}, information for selecting {geometry, attribute}, or information for selecting {occupancy, geometry, attribute}. The refinement target information is characterized in that a plurality of refinement targets can be selected in a overlapping manner.

[0116] Fig. 12 is a diagram showing an example of refinement target information. As shown in the figure, nnrc_input_mode may indicate whether or not refinement is performed on occupancy, geometry, and attributes. Y indicates that refinement is performed, and - indicates that refinement is not performed. In nnrc_input_mode, the most significant bit indicates occupancy, the second bit indicates geometry, and the least significant bit indicates attributes.

[0117] 0 No refinement 1 Refine your attributes 2 Refine the geometry 3 Refine the geometry and attributes 4. Refine your occupancy 5 Refine occupancy and attributes 6 Refining occupancy and geometry 7 Refine occupancy, geometry and attributes The bitwise AND's &4, &2, &1 determine whether they apply to occupancy, geometry, or attributes, respectively.

[0118] OccupancyUsedFlag = (nnrc_input_mode&4) GeometryUsedFlag = (nnrc_input_mode&2) AttributeUsedFlag = (nnrc_input_mode&1) The bit positions are not limited to those mentioned above.

[0119] OccupancyUsedFlag = (nnrc_input_mode&1) GeometryUsedFlag = (nnrc_input_mode&2) AttributeUsedFlag = (nnrc_input_mode&4) That's fine too. The following definition may be used, excluding no refinement:

[0120] 0 Refine to Attributes 1 Refine to geometry 2. Refine geometry and attributes 3. Refine to Occupancy 4. Refine occupancy and attributes 5. Refine Occupancy and Geometry 6 Refine occupancy, geometry and attributes in this case, OccupancyUsedFlag = (nnrc_input_mode+1)&4 GeometryUsedFlag = (nnrc_input_mode+1)&2 AttributeUsedFlag = (nnrc_input_mode+1)&1 The syntax element nnrc_inp_out_format_idc (input / output tensor format information) indicates how to convert the pixel values ​​of the decoded image into input / output values ​​for the refinement process. If the value of nnrc_inp_out_format_idc is 0, the input values ​​to the refinement process (especially the input tensor) are real numbers (floating-point value format) specified by IEEE754, and the function Inp is specified as follows: The range of values ​​of the input tensor is 0..1.

[0121] Inp(x) = x÷((1< <BitDepth)-1) InpS(x) = x If the value of nnrc_inp_out_format_idc is 1, the input and output values ​​to the refinement process are unsigned integers, and the function Inp is specified as follows: The range of input tensor values ​​is 0..1<<(inpTensorBit Depth)-1.

[0122] shift=BitDepth-inpTensorBitDepth if(inpTensorBitDepth>=BitDepth) Inp(x)=x<<(inpTensorBitDepth-BitDepth) else Inp(x) = Clip3(0,(1< <inpTensorBitDepth)-1,(x+(1<<(shift-1)))> >shift) InpS(x) = x ÷ ((1< <inpTensorBitDepth) - 1) The value of the syntax element nnrc_inp_tensor_bitdepth_minus8 plus 8 indicates the pixel bit length of the luma pixel values ​​of the input integer tensor. The value of the variable inpTensorBitDepth is derived from the syntax element nnrc_inp_tensor_bitdepth_minus8 as follows:

[0123] inpTensorBitDepth=nnrc_inp_tensor_bitdepth_minus8+8 A block is an image array. The refinement process is performed in fixed block units. A block may also be called a patch.

[0124] The syntax element nnrc_block_size_idc indicates the block size, which may be a multiple of 64, such as 64, 128, 192, etc.

[0125] blockWidth = 64*nnrc_block_size_idc (=nnrc_block_size_idc<<6) blockHeight = 64*nnrc_block_size_idc (=nnrc_block_size_idc<<6) It can also be less than nnrc_block_size_idc excluding 0.

[0126] blockWidth = (nnrc_block_size_idc+1)<<6 blockHeight = (nnrc_block_size_idc+1)<<6 The syntax element nnrc_overlap_size_idc specifies the number of horizontal and vertical pixels to overlap adjacent input tensors. The value of nnrc_overlap_size_idc may be a multiple of 4 to specify the size of the overlap.

[0127] overlapSize = nnrc_overlap_size_idc << 2 The syntax element nnrc_auxiliary_inp_idc indicates whether there is auxiliary data for inputTensor. If it is 0, there is no auxiliary data, and if it is greater than 0, input the auxiliary data for inputTensor. The auxiliary data may be a parameter (e.g. nnra_strength_idc) decoded from the NNRA SEI.

[0128] A function Out that converts the luminance pixel value and the chrominance pixel value output by the post-processing into integer values ​​of the pixel bit length is specified as follows using the pixel bit length BitDepth:

[0129] Out(x) = Clip3(0,(1< <BitDepth)-1,Round(x*((1<<BitDepth)-1))) The function Out is specified as follows:

[0130] shift=outTensorBitDepth-BitDepth if(outTensorBitDepth>=BitDepth) Out(x) = Clip3(0,(1< <BitDepth)-1,(x+(1<<(shift-1)))> >shift) else Out(x)=x<<(BitDepth-outTensorBitDepth) nnrc_out_tensor_bitdepth_minus8+8 specifies the pixel bit depth of pixel values ​​in the output integer tensor. The value of outTensorBitDepth is derived from the syntax element nnrc_out_tensor_bitdepth_minus8 as follows:

[0131] outTensorBitDepth=nnrc_out_tensor_bitdepth_minus8+8 The syntax element nnrc_reserved_zero_bit_b shall be equal to 0.

[0132] The syntax element nnrc_payload_byte[i] contains the i-th byte of an ISO / IEC 15938-17 compliant bitstream. All nnrc_payload_byte[i] must be fully ISO / IEC 15938-17 compliant bitstreams.

[0133] <Specify input tensors and output tensors according to the information to be refined> FIG. 10 is a flowchart showing the operation of the refinement unit 306 .

[0134] The example shown in FIG. 10 will now be described in further detail.

[0135] In FIG. 10A, the refinement unit 306 or the atlas decoding unit 302 decodes the refinement target information (S3061).

[0136] Depending on nnrc_input_mode, the refinement unit 306 may derive inputTensor. Hereinafter, ch indicates the index (position) of the channel to be set. ch++ is an abbreviation of ch=ch+1, and indicates that the index of the channel to be set is incremented by +1.

[0137] ch=0 If the refinement target information indicates an occupancy (S3062I), the occupancy is added to the input tensor (S3063I). For example, the following formula (IN-OCC) may be used.

[0138] if (OccupancyUsedFlag) { (expression IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yT][xT]) ch++ } If the refinement target information indicates a geometry (S3064I), the geometry is added to the input tensor (S3065I). For example, the following (formula IN-GEO) may be used.

[0139] if (GeometryUsedFlag) { (expression IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yT][xT]) ch++ } If the refinement target information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, the following (IN-ATTR) may be used.

[0140] if (AttributeUsedFlag) { (expression IN-ATTR) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[0][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[1][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[2][yT][xT]); ch++ } In FIG. 10(b), depending on nnrc_input_mode, the refinement unit 306 may derive (update) the output frame from the outputTensor.

[0141] ch=0 If the refinement target information indicates occupancy (S3062O), the specific components of the output tensor are For example, the following formula (OUT-OCC) may be used:

[0142] ch=0 if (OccupancyUsedFlag) { oFrame[yP][xP]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } If the refinement target information indicates a geometry (S3064O), a specific component of the output tensor is set to the geometry (S3065O). For example, the following (formula OUT-GEO) may be used.

[0143] if (GeometryUsedFlag) { gFrame[yP][xP]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } If the refinement target information indicates an attribute (S3066O), a specific component of the output tensor is set to the attribute (S3067O). For example, the following (formula OUT-ATTR) may be used.

[0144] if (AttributeUsedFlag) { aFrame[0][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[1][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[2][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } In the above, the atlas decoding unit 302 decodes the refinement target information from the encoded data, and the refinement unit 306 uses the decoded refinement target information to set the occupancy, geometry, and attributes to the input tensor. Furthermore, the refinement unit 306 refines the input tensor to obtain an output tensor. Using the decoded refinement target information, the refinement unit 306 sets the occupancy, geometry, and attributes to the output tensor. According to the above configuration, since the refinement process can be performed by combining multiple occupancies, geometries, and attributes, the efficiency is improved. In addition, by decoding the refinement target information from the encoded data, the input and output of the model can be set with a degree of freedom according to the complexity. In addition, the content of the refinement can be determined and processing can be performed.

[0145] Fig. 13 shows a part of the syntax of nn_refinement_characteristics(payloadSize) (characteristics SEI). The characteristic SEI shown in Fig. 13(a) further includes, as refinement target information, a syntax element nnrc_attribute_num_minus1 indicating the number of attributes.

[0146] Depending on the nnrc_input_mode, the refinement unit 306 may derive the inputTensor.

[0147] ch=0 If the refinement target information indicates occupancy, add the occupancy to the input tensor, e.g. (IN-OCC).

[0148] If the refinement target information indicates geometry, add the geometry to the input tensor, e.g. (expression IN-GEO).

[0149] If the refinement target information indicates an attribute, add the attribute to the input tensor. For example, the following (expression IN-ATTR-NUM):

[0150] if (AttributeUsedFlag) { (expression IN-ATTR-NUM) for (k=0;k<=nnrc_attribute_num_minus1;k++) { inputTensor[ch][yP+overlapSize][xP+overlapSize]=Inp(aFrame[k][yT][xT]); ch++ } } Note that the refinement unit 306 may use ai_attribute_dimension_minus1[RecAtlasID][attrIdx] obtained by decoding the encoded data of V3C VPS without including nnrc_attribute_num_minus1 as a syntax element of the characteristic SEI. Also, ai_attribute_dimension_minus1=2 may be used.

[0151] if (AttributeUsedFlag) { for (k=0;k<=ai_attribute_dimension_minus1[RecAtlasID][attrIdx];k++) { inputTensor[ch][yP+overlapSize][xP+overlapSize]=Inp(aFrame[k][yT][xT]); ch++ } } Depending on the nnrc_input_mode, the refiner 306 may derive (update) the output frame from the outputTensor.

[0152] ch=0 If the refinement target information indicates occupancy, a specific component of the output tensor is set to the occupancy, for example (OUT-OCC).

[0153] If the refinement target information indicates geometry, add the geometry to a specific component of the output tensor, e.g. (expression OUT-GEO).

[0154] If the refinement target information indicates an attribute, the attribute is added to a specific component of the output tensor. For example, the following (expression OUT-ATTR-NUM):

[0155] if (AttributeUsedFlag) { (expression OUT-ATTR-NUM) for (k=0;k<=nnrc_attribute_num_minus1;k++) { aFrame[k][yT][xT]=Out(outputTensor][ch][yP+overlapSize][xP+overlapSize]); ch++ } } In the above, the atlas decoding unit 302 further decodes the number of attributes as refinement target information from the encoded data, and the refinement unit 306 uses the decoded refinement target information to select one of occupancy, geometry, and attributes and set it in the input tensor. Furthermore, the refinement unit 306 refines the input tensor to obtain an output tensor. Using the decoded refinement target information, the refinement unit 306 sets the occupancy, geometry, and attributes in the obtained output tensor. The refinement unit 306 selects one of occupancy, geometry, and attributes and sets the occupancy, geometry, and attributes from the output tensor. According to the above configuration, it is possible to vary the number of attributes to which refinement is applied, and it is possible to set the input and output of the model with a degree of freedom according to the complexity.

[0156] The characteristic SEI shown in FIG. 13(b) further includes, as refinement target information, the syntax element nnrc_attribute_start_pos indicating the initial position of the attribute and the syntax element Contains the element nnrc_attribute_num_minus1.

[0157] Depending on the nnrc_input_mode, the refiner 306 may derive the inputTensor.

[0158] ch=0 If the refinement target information indicates occupancy, add the occupancy to the input tensor, e.g. (IN-OCC).

[0159] If the refinement target information indicates geometry, add the geometry to the input tensor, e.g. (expression IN-GEO).

[0160] If the refinement target information indicates an attribute, add the attribute to the input tensor. For example, the following (expression IN-ATTR-POS):

[0161] if (AttributeUsedFlag) { (expression IN-ATTR-POS) for (k=nnrc_attribute_start_pos; k<=nnrc_attribute_start_pos+nnrc_attribute_num_minus1; k++) { inputTensor[ch][yP+overlapSize][xP+overlapSize]=Inp(aFrame[k][yT][xT]); ch++ } } Alternatively, depending on nnrc_input_mode, the refiner 306 may derive (update) the output frame from the outputTensor.

[0162] ch=0 If the refinement target information indicates occupancy, a specific component of the output tensor is set to the occupancy, for example (OUT-OCC).

[0163] If the refinement target information indicates geometry, add the geometry to a specific component of the output tensor, e.g. (expression OUT-GEO).

[0164] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute. For example, the following (expression OUT-ATTR-POS):

[0165] if (AttributeUsedFlag) { (expression OUT-ATTR-POS) for (k=nnrc_attribute_start_pos; k<=nnrc_attribute_start_pos+nnrc_attribute_num_minus1; k++) { aFrame[k][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } } In the above, the atlas decoding unit 302 decodes the number of attributes as refinement target information from the encoded data, and the refinement unit 306 uses the refinement target information to set the occupancy, geometry, and attributes to the input tensor. Furthermore, the refinement unit 306 refines the input tensor to obtain an output tensor. Using the refinement target information, the output tensor is set to the occupancy, geometry, and attributes. According to the above configuration, the number of attributes to which refinement is applied can be changed, and the input and output of the model can be set with a degree of freedom according to the complexity.

[0166] Figure 14 shows the syntax of nn_refinement_characteristics(payloadSize) (characteristic SEI). A part of the information is shown. The characteristic SEI shown in Fig. 14 includes syntax elements nnrc_occupancy_flag, nnrc_geometry_flag, and nnrc_attribute_num as refinement target information. nnrc_occupancy_flag is a flag indicating whether or not to refine the occupancy, and nnrc_geometry_flag is a flag indicating whether or not to refine the geometry. nnrc_attribute_num is the number of attributes that are refinement target information, and can indicate whether or not to refine the attribute. The refinement unit 306 may be set as follows to perform the process already described. OccupancyUsedFlag= nnrc_occupancy_flag GeometryUsedFlag= nnrc_geometry_flag AttributeUsedFlag= nnrc_attribute_num The characteristic SEI may include nnrc_attribute_flag and nnrc_attribute_num_minus1 as syntax elements instead of nnrc_attribute_num. In this case, nnrc_attribute_flag and nnrc_attribute_num_minus1 may be decoded instead of nnrc_attribute_num. The refinement unit 306 may be set as follows and perform the processes already described. OccupancyUsedFlag = nnrc_occupancy_flag GeometryUsedFlag = nnrc_geometry_flag AttributeUsedFlag = nnrc_attribute_flag According to the information to be refined, the refinement unit 306 may derive the inputTensor.

[0167] ch = 0 When the nnrc_occupancy_flag of the information to be refined is other than 0, that is, when it indicates occupancy, add occupancy to the input tensor. For example (Equation IN - OCC).

[0168] When the nnrc_geometry_flag of the information to be refined is other than 0, that is, when it indicates geometry, add geometry to the input tensor. For example (Equation IN - GEO).

[0169] Add attributes to the input tensor by the number of nnrc_attribute_num of the information to be refined. For example, the following (Equation IN - ATTR - NUM2).

[0170] for (k = 0; k < nnrc_attribute_num; k++) { (Equation IN - ATTR - NUM2) inputTensor[ch][yP + overlapSize][xP + overlapSize]=Inp(aFrame[compTimeIdx][k][yT][xT]); ch++ } When nnrc_attribute_num = 0, the attribute frame is not added to the inputTensor. Therefore, whether to perform refinement processing on the attribute frame (when nnrc_attribute_num > 0) or not (when nnrc_attribute_num == 0) can be controlled by the value of nnrc_attribute_num.

[0171] According to nnrc_input_mode, the refinement unit 306 may derive (update) the output frame from the outputTensor.

[0172] ch = 0 When the nnrc_occupancy_flag of the information to be refined is other than 0, that is, when it indicates occupancy, specific components of the output tensor are set to occupancy. For example (Equation OUT - OCC).

[0173] When the nnrc_geometry_flag of the information to be refined is other than 0, that is, when it indicates geometry, specific components of the output tensor are set to geometry. For example (Equation OUT - GEO).

[0174] Set specific components of the output tensor to attributes by the number of nnrc_attribute_num of the information to be refined. For example, the following (Equation OUT - ATTR - NUM2).

[0175] for (k = 0; k < nnrc_attribute_num; k++) { (Equation OUT - ATTR - NUM2) aFrame[k][yT][xT]=Out(outputTensor[ch][yP + overlapSize][xP + overlapSize]); ch++ } In the above, the atlas decoding unit 302 sets the input tensor using separate syntax elements for occupancy, geometry, and attributes. Furthermore, the refinement unit 306 refines the input tensor to obtain an output tensor. The output tensor is further set to occupancy, geometry, and attributes using refinement target information.

[0176] The refinement target information may further include a syntax element nnrc_attribute_start_pos, which is a syntax element indicating the start position of the attribute to be refined in FIG.

[0177] Add attributes to the input tensor by the number of nnrc_attribute_num_minus1+1 in the refinement target information. For example, the following (expression IN-ATTR-POS2).

[0178] for (k=nnrc_attribute_start_pos; k<=nnrc_attribute_start_pos+nnrc_attribute_num_minus1; k++) { (formula IN-ATTR-POS2) inputTensor[ch][yP+overlapSize][xP+overlapSize]=Inp(aFrame[k][yP][xP]); ch++ } Set specific components of the output tensor to attributes by the number of refinement target information nnrc_attribute_num_minus1+1. For example, the following (expression OUT-ATTR-POS2).

[0179] for (k=nnrc_attribute_num_start_pos; k<=nnrc_attribute_num_start_pos+nnrc_attribute_num_minus1) { aFrame[k][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } Fig. 15 shows a part of the syntax of nn_refinement_characteristics(payloadSize) (characteristics SEI). When the characteristic SEI shown in Fig. 15 includes geometry and attributes as refinement target information, it includes a syntax element nnrc_map_index indicating a map to be refined.

[0180] If the refinement target information indicates occupancy, add the occupancy to the input tensor, e.g. (IN-OCC).

[0181] If the refinement target information indicates a geometry, add the geometry indicated by nnrc_map_index to the input tensor. For example, set mapIdx=nnrc_map_index in (IN-GEO).

[0182] If the refinement target information indicates an attribute, add the attribute indicated by nnrc_map_index to the input tensor. For example, set mapIdx=nnrc_map_index in (IN-ATTR), (IN-ATTR-NUM), (IN-ATTR-POS), (IN-ATTR-NUM2), and (IN-ATTR-POS2).

[0183] If the refinement target information indicates occupancy, a specific component of the output tensor may be set to the occupancy, for example (OUT-OCC).

[0184] If the refinement target information indicates a geometry, a specific component of the output tensor is set to the geometry indicated by nnrc_map_index. For example, in the expression OUT-GEO, mapIdx=nnrc_map_index is set.

[0185] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute indicated by nnrc_map_index. For example, mapIdx=nnrc_map_index may be set in (expression OUT-ATTR), (expression OUT-ATTR-NUM), (expression OUT-ATTR-POS), (expression OUT-ATTR-NUM2), and (expression OUT-ATTR-POS2).

[0186] In the above, the atlas decoding unit 302 decodes the nnrc_map_index of the geometry and attribute to be refined from the characteristic SEI, and derives mapIdx. Since the geometry and attribute specified by mapIdx are selected and refined, it is possible to apply refinement optimized for a specific mapIdx.

[0187] Furthermore, if the characteristic SEI includes an attribute as refinement target information, it may include a syntax element nnrc_attribute_index indicating the attrIdx of the refinement target.

[0188] If the refinement target information indicates an attribute, the attributes indicated by nnrc_attribute_index and nnrc_map_index are added to the input tensor. For example, mapIdx=nnrc_map_index and attrIdx=nnrc_attribute_index may be set in (expression IN-ATTR), (expression IN-ATTR-NUM), (expression IN-ATTR-POS), (expression IN-ATTR-NUM2), and (expression IN-ATTR-POS2).

[0189] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute indicated by nnrc_map_index. For example, mapIdx=nnrc_map_index and attrIdx=nnrc_attribute_index may be set in (expression OUT-ATTR), (expression OUT-ATTR-NUM), (expression OUT-ATTR-POS), (expression OUT-ATTR-NUM2), and (expression OUT-ATTR-POS2).

[0190] The characteristic SEI may further include nnrc_attribute_partition_index and nnrc_auxiliary_video_flag. The nnrc_attribute_partition_index is an index indicating the partition of the attribute to be refined, and the nnrc_auxiliary_video_flag is a flag indicating the auxiliary data to be refined.

[0191] In the above, the atlas decoding unit 302 decodes the geometry and nnrc_attribute_index of the attribute to be refined from the characteristic SEI, and derives attrIdx. Since the attribute specified by attrIdx is selected and refined, it is possible to apply refinement optimized for a specific attrIdx.

[0192] (Neural network refinement applied SEI) FIG. 16(a) shows an example syntax configuration of an application SEI.

[0193] For this applied SEI, the atlas decoding unit 302 (refinement information decoding unit) and the atlas encoding unit 102 (refinement information encoding unit) decode and encode the following syntax elements.

[0194] nnra_target_id indicates the ID of the characteristic SEI to be applied (the identifier of the characteristic information of the refinement, identification information). The refinement process specified by the characteristic SEI having the nnrc_id equal to nnra_target_id is applied to the image.

[0195] nnra_cancel_flag is a cancellation flag. When nnra_cancel_flag is 1, it indicates that the retention of refinement set for the image in the already decoded NNRA SEI is canceled. When nnra_cancel_flag is 0, it indicates that the following syntax element (nnra_persistence_flag) is transmitted, encoded, and decoded.

[0196] nnra_persistence_flag indicates the persistence information of the target refinement. nnra_persistence_flag, when set to 0, indicates that the refinement in question applies only to the picture indicated by atlasID. nnra_persistence_flag, when set to 1, indicates that the refinement in question applies to the current picture and all subsequent pictures until one of the following conditions is met: A new CAS is started. The bitstream ends. NNRA SEI messages with nnra_cancel_flag=1 are output following the current picture in output order, i.e., are cancelled by nnra_cancel_flag=1. The following is also acceptable. NNRA SEI messages that have the same nnra_target_id as the current SEI message and have nnra_cancel_flag=1 are output following the current picture in output order, i.e., are cancelled by nnra_cancel_flag=1.

[0197] 16B shows an example of the syntax configuration of the applied SEI. The atlas decoding unit 302 and the atlas encoding unit 102 decode and encode the syntax element indicating the second refinement target information. The second refinement target information may include nnra_map_index indicating the mapIdx to which the refinement is applied, nnra_attribute_index indicating the attrIdx to which the refinement is applied, nnra_attribute_partition_index indicating the partIdx to which the refinement is applied, and nnra_auxiliary_video_flag indicating whether the refinement target is auxiliary data.

[0198] If the refinement target information indicates occupancy, add the occupancy to the input tensor, e.g. (IN-OCC).

[0199] If the refinement target information indicates a geometry, the geometry indicated by the second refinement target information nnra_map_index is added to the input tensor. For example, mapIdx=nnra_map_index may be set in (expression IN-GEO).

[0200] If the refinement target information indicates an attribute, the attribute indicated by the second refinement target information nnra_map_index is added to the input tensor. For example, mapIdx=nnra_map_index may be set in (expression IN-ATTR), (expression IN-ATTR-NUM), (expression IN-ATTR-POS), (expression IN-ATTR-NUM2), and (expression IN-ATTR-POS2).

[0201] If the refinement target information indicates occupancy, a specific component of the output tensor may be set to the occupancy, for example (OUT-OCC).

[0202] When the refinement target information indicates a geometry, a specific component of the output tensor is set to the geometry indicated by the second refinement target information nnra_map_index. For example, it may be set to mapIdx=nnra_map_index in (expression OUT-GEO).

[0203] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute indicated by the second refinement target information nnra_map_index. For example, mapIdx=nnra_map_index may be set in (expression OUT-ATTR), (expression OUT-ATTR-NUM), (expression OUT-ATTR-POS), (expression OUT-ATTR-NUM2), and (expression OUT-ATTR-POS2).

[0204] In the above, the atlas decoding unit 302 decodes the nnra_map_index of the geometry and attributes to be refined from the applied SEI to derive the mapIdx. Since the geometry and attributes specified by the mapIdx of the applied SEI are selected and refined, it is possible to apply refinement optimized for a specific mapIdx. Furthermore, it is possible to apply refinement specified by the same characteristic SEI to geometries and attributes indicated by the same mapIdx value. At the same time, it is possible to apply refinement specified by different characteristic SEIs to geometries and attributes indicated by different mapIdx. The refinement specified by I can be applied.

[0205] Furthermore, when an attribute is included as the refinement target information, a syntax element nnra_attribute_index indicating the attrId of the refinement target may be included.

[0206] If the refinement target information indicates an attribute, the attributes indicated by the second refinement target information nnra_attribute_index and nnra_map_index are added to the input tensor. For example, mapIdx=nnra_map_index and attrIdx=nnra_attribute_index may be set in (expression IN-ATTR), (expression IN-ATTR-NUM), (expression IN-ATTR-POS), (expression IN-ATTR-NUM2), and (expression IN-ATTR-POS2).

[0207] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute indicated by the second refinement target information nnra_map_index and nnra_attribute_index. For example, mapIdx=nnra_map_index and attrIdx=nnra_attribute_index may be set in (expression OUT-ATTR), (expression OUT-ATTR-NUM), (expression OUT-ATTR-POS), (expression OUT-ATTR-NUM2), and (expression OUT-ATTR-POS2). It may also include nnrc_attribute_partition_index and nnrc_auxiliary_video_flag.

[0208] In the above, the atlas decoding unit 302 decodes the nnra_attribute_index of the geometry and attribute to be refined from the application SEI, and derives attrIdx. The attribute specified by attrIdx is selected and refined. Therefore, it is possible to apply a refinement optimized for a specific attrIdx. In addition, refinements specified by the same characteristic SEI can be applied to attributes indicated by the same value of attrIdx. Refinement specified by different characteristic SEIs can be simultaneously applied to attributes indicated by different attrIdx.

[0209] FIG. 19 shows an example of the syntax configuration of an application SEI. Here, the number of attributes nnra_attribute_count and nnra_map_count_minus1, which is the number of maps minus 1, are included as syntax elements. Also, nnra_target_id[i][j] indicating the ID of the characteristic SEI indicating the NNRE to be applied for each attribute attrIdx=i and map mapIdx=j is included. Also, nnra_enabled_flag[i][j], nnra_cancel_flag[i][j], and nnra_persistent_flag[i][j] indicating whether to apply refinement for each attribute attrIdx=i and map mapIdx=j may be included. Also, nnra_strength_present_flag[i][j] may be included.

[0210] If the refinement target information indicates an attribute, the attributes indicated by i and j are added. For example, mapIdx=i and attrIdx=j may be set in (expression IN-ATTR), (expression IN-ATTR-NUM), (expression IN-ATTR-POS), (expression IN-ATTR-NUM2), and (expression IN-ATTR-POS2).

[0211] If the refinement target information indicates an attribute, it is set to the attribute indicated by i and j. For example, mapIdx=i and attrIdx=j may be set in (expression OUT-ATTR), (expression OUT-ATTR-NUM), (expression OUT-ATTR-POS), (expression OUT-ATTR-NUM2), and (expression OUT-ATTR-POS2).

[0212] (Configuration of 3D data encoding device) FIG. 17 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to an embodiment of the present invention.

[0213] The 3D data encoding device 11 includes a patch generation unit 101, an atlas encoding unit 102, an occupancy generation unit 103, an occupancy encoding unit 104, a geometry generation unit 105, a geometry encoding unit 106, an attribute generation unit 108, an attribute encoding unit 109, a refinement parameter derivation unit 110, and a multiplexing unit 111. The 3D data encoding device 11 inputs a point cloud or a mesh as 3D data, and outputs encoded data.

[0214] The patch generation unit 101 inputs 3D data, generates a set of patches (rectangular images in this case), and outputs it. Specifically, the 3D data is divided into multiple regions, and each region is projected onto one of the planes of a 3D bounding box set in the 3D space to generate multiple patches. The patch generation unit 101 outputs information about the 3D bounding box (coordinates, size, etc.) and information about mapping onto the projection surface (projection surface, coordinates, size, whether or not each patch is rotated, etc.) as atlas information.

[0215] The atlas encoding unit 102 encodes the atlas information output from the patch generating unit 101 and outputs atlas data.

[0216] The occupancy generation unit 103 inputs a set of patches output from the patch generation unit 101, and generates an occupancy that shows the valid area (area where 3D data exists) of each patch as a 2D binary image (for example, valid area is 1 and invalid area is 0). Note that other values ​​such as 255 and 0 may be used as the values ​​of the valid area and invalid area.

[0217] The occupancy encoding unit 104 inputs the occupancy output from the occupancy generation unit 103, and outputs the occupancy and occupancy data. As the encoding method, VVC, HEVC, or the like is used.

[0218] The geometry generation unit 105 generates a geometric frame that stores a depth value for the projection surface of each patch based on the 3D data, occupancy, occupancy data, and atlas information. The geometric frame generation unit 105 derives a point with a minimum depth for the projection surface among the points projected onto pixel g(x,y) as p_min(x,y,z). In addition, the geometric frame generation unit 105 derives a point with a maximum depth among the points projected onto pixel g(x,y) and at a predetermined distance d from p_min(x,y,z) as p_max(x,y,z). The geometric frame in which p_min(x,y,z) is projected onto all pixels on the projection surface is set as a geometric frame of the Near layer. The geometric frame in which p_max(x,y,z) is projected onto all pixels on the projection surface is set as a geometric frame of the Far layer.

[0219] The geometry encoding unit 106 receives a geometry frame and outputs the geometry frame and geometry data. The encoding method used is VVC, HEVC, or the like.

[0220] The attribute generation unit 108 generates an attribute frame that stores color information (e.g., YUV values, RGB values, etc.) for the projection surface of each patch based on the 3D data, occupancy, geometric frame, and atlas information. The attribute generation unit 108 obtains the value of the attribute corresponding to the point p_min(x, y, z) with the minimum depth calculated by the geometry generation unit 105, and sets the attribute frame into which the value is projected as the attribute frame of the Near layer. Similarly, the attribute frame obtained for p_max(x, y, z) is set as the attribute frame of the Far layer.

[0221] The attribute encoding unit 109 inputs an attribute frame and outputs attributes and attribute data. The encoding method used is VVC, HEVC, or the like.

[0222] The refinement parameter derivation unit 110 inputs an attribute frame and an original attribute frame, or a geometry frame and an original geometry frame, and selects or derives optimal filter parameters for NN filtering and outputs them. The refinement parameter derivation unit 110 sets values ​​such as nnra_target_id, nnra_cancel_flag, and nnra_persistance_flag in the above SEI.

[0223] The multiplexing unit 111 receives the filter parameters output from the refinement parameter derivation unit 110 and outputs them in a predetermined format. The predetermined format is, for example, SEI, which is supplementary extension information of video data, ASPS, AFPS, and META, which are information specifying a data structure in the V3C standard. The multiplexing unit 111 also multiplexes the atlas data, occupancy data, geometry data, attribute data, and the above filter parameters, and outputs the multiplexed data as coded data. The multiplexing method used is the byte stream format, ISOBMFF, or the like.

[0224] FIG. 18 is a block diagram showing the relationship between the 3D data encoding device 11, the 3D data decoding device 31, and the SEI message according to the embodiment of the present invention.

[0225] The 3D data encoding device 11 is composed of a video encoding unit and an SEI encoding unit. In the configuration of Fig. 17, the video encoding unit corresponds to the occupancy encoding unit 104, the geometry encoding unit 106, and the attribute encoding unit 109. In the configuration of Fig. 17, the SEI encoding unit corresponds to the atlas encoding unit 102.

[0226] The 3D data decoding device 31 is composed of a video decoding unit, an SEI decoding unit, a switch, and a refinement unit. In the configuration of Fig. 5, the video decoding unit corresponds to the occupancy decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305. In the configuration of Fig. 5, the SEI decoding unit corresponds to the atlas decoding unit 302. In the configuration of Fig. 5, the refinement unit corresponds to the refinement unit 306. The refinement unit 306 may include either an NN filter unit 611 or an ALF unit 610, or both.

[0227] The occupancy, geometry frames, and attribute frames generated from the 3D data are coded in the video coding unit. The coded data is decoded in the video decoding unit, and a decoded image is reconstructed. The SEI coding unit generates a characteristic SEI and an application SEI from the 3D data. The SEI decoding unit decodes these SEI messages. The application SEI is input to a switch to specify the image to be refined, and only the image to be refined is input to the refinement unit. The characteristic SEI is input to the refinement unit and specifies the refinement to be applied to the decoded image. The refined image, or the decoded image, is displayed on the 3D data display unit 41 (Figure 1).

[0228] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0229] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0230] The embodiments of the present invention can be suitably applied to a 3D data decoding device that decodes encoded data in which 3D data is encoded, and a 3D data encoding device that generates encoded data in which 3D data is encoded. Also, the embodiments of the present invention can be suitably applied to the data structure of encoded data generated by the 3D data encoding device and referenced by the 3D data decoding device. [Explanation of symbols]

[0231] 11 3D data encoding device 101 Patch Generation Unit 102 Atlas Encoding Unit 103 Occupancy Generation Unit 104 Occupancy coding unit 105 Geometry Generation Unit 106 Geometry Encoding Unit 108 Attribute Generation Unit 109 Attribute Encoding Unit 110 Refine Parameter Derivation Part 111 Multiplexer 21 Network 31 3D data decoding device 301 Header Decoding Unit 302 Atlas Information Decoding Unit 303 Occupancy Decoding Unit 304 Geometry Decoding Unit 306 Geometry Reconstruction Unit 307 Attribute Decoding Unit 308 Refining Department 309 V3C Decoding Unit 310 Pre-reconstruction unit 311 Most constructed part 41 3D data display device

Claims

1. A 3D data decoding device that decodes encoded data and decodes 3D data including geometry and attributes, a refinement information decoding unit for decoding refinement characteristic information and refinement activation information from the encoded data; a geometry decoding unit for decoding a geometric frame from the encoded data; an attribute decoding unit for decoding an attribute frame from the encoded data; a refining unit for refining the attribute frame or the geometry frame, The refinement information decoding unit decodes refinement target information indicating whether to use occupancy, geometry, or attributes from the encoded data of the characteristic information, and performs refinement using an image specified according to the refinement target information.

2. The 3D data decoding device described in claim 1, characterized in that when the refinement target information indicates the use of attributes, the refinement information decoding unit further decodes the number of attributes to be applied as the refinement target information, and the refinement unit performs refinement using the specified number of attributes.

3. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit individually decodes syntax elements indicating whether occupancy, geometry, or attributes are to be used as the refinement target information, and when occupancy, geometry, or attributes are specified, the refinement unit performs refinement using them.

4. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit further decodes information indicating the map ID to be applied from the characteristic information, and the refinement unit performs refinement on the specified map ID.

5. The 3D data decoding device according to claim 4, characterized in that when the refinement information decoding unit indicates that geometry or attributes are to be used as the refinement target information, it further decodes information indicating a map ID to be applied.

6. The 3D data decoding device according to claim 4, characterized in that when the refinement information decoding unit indicates that an attribute is to be used as the refinement target information, the refinement information decoding unit further decodes information indicating an attribute index, a partition index, and an auxiliary flag to be applied.

7. The 3D data decoding device according to claim 1, characterized in that the refinement information decoding unit decodes, as the activation information, an index indicating characteristic information, a cancel flag, and a persistence flag.

8. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit decodes information indicating the map ID to be applied as the activation information, and the refinement unit performs refinement using geometry or attributes specified by the map ID.

9. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit decodes information indicating the map ID to be applied as the activation information, and the refinement unit performs refinement using geometry or attributes specified by the map ID.

10. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit decodes information indicating the attributes to be applied as the activation information, and the refinement unit performs refinement using the attributes specified in the information indicating the attributes.

11. A 3D data encoding device that encodes encoded data and encodes 3D data including geometry and attributes, a refinement information encoding unit for encoding refinement characteristic information and neural network refinement activation information; a geometry encoding unit for encoding a geometric frame; an attribute encoding unit for encoding an attribute frame; a refining unit for refining the attribute frame or the geometry frame, The refinement information encoding unit encodes refinement target information indicating whether to use occupancy, geometry, or attributes as encoded data of the characteristic information, and performs refinement using an image specified according to the refinement target information.

Images