3D data encoding device and 3D data decoding device

Abstract

To solve a problem where it is not possible to apply suitable refinement to 3D data using a plurality of images.SOLUTION: A 3D data decoding device including a geometry decoding unit for decoding a geometric frame from encoded data and an attribute decoding unit for decoding an attribute frame further includes a refinement information decoding unit for decoding refinement characteristic information and refinement activation information from the encoded data, and a refinement processing unit for decoding a syntax element indicating the number of refinements from the activation information, decoding an index indicating the characteristic information of the number of the decoded refinements, and performing refinement processing on the attribute frame or the geometric frame according to the characteristic information.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to a 3D data encoding device and a 3D data decoding device. [Background technology]

[0002] In order to transmit or record 3D data efficiently, the 3D data is converted into 2D images and video A 3D data encoding device that encodes the 3D data using an encoding method to generate encoded data, and a 3D data decoding device that decodes and reconstructs a 2D image from the encoded data to generate 3D data. In addition, the 2D image is filtered using the auxiliary augmentation information of the deep learning post-filter. There is technology to do this.

[0003] Specific examples of 3D data encoding methods include MPEG-I's ISO / IEC 23090-5 V3C (Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression). V3C is a point cloud that consists of point positions and attribute information. In addition, ISO / IEC 23090-12 (MPEG Immersive Video, MIV) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding, V-DMC) are currently being standardized. It is also used for encoding and decoding multi-view images and mesh images. In addition, in the auxiliary extension information of the neural network postfilter in Non-Patent Document 1, model information of the neural network is transmitted in the characteristic SEI, and is called for a frame specified in the application SEI, thereby enabling adaptive filtering of point cloud data. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] K.Takada, Y.Tokumo, T.Chujoh. T.Ikai,“[V-PCC][EE2.8] V3C neural-network post-filter SEI messages,” ISO / IEC JTC 1 / SC 29 / WG7, m61805, Jan. 2023 Summary of the Invention [Problem to be solved by the invention]

[0005] In Non-Patent Document 1, since filtering is performed only on attributes, there is a problem that filtering using the relationship between occupancy, geometry and attributes cannot be performed. In other words, in a method of decoding 3D data using multiple images and videos, refinement using information between images cannot be performed.

[0006] The present invention aims to solve the above-mentioned problems in encoding and decoding 3D data using a video encoding and decoding method, and to further reduce encoding distortion by using auxiliary information for refinement, thereby encoding and decoding 3D data with high quality. [Means for solving the problem]

[0007] In order to solve the above problem, a 3D data decoding device according to one embodiment of the present invention is a 3D data decoding device comprising a geometry decoding unit that decodes a geometric frame from encoded data and an attribute decoding unit that decodes an attribute frame, characterized in that the 3D data decoding device further comprises a refinement information decoding unit that decodes refinement characteristic information and refinement activation information from the encoded data, and a refinement processing unit that decodes a syntax element indicating the number of refinements from the activation information, decodes an index indicating the characteristic information of the decoded number of refinements, and performs a refinement process on the attribute frame or the geometric frame according to the characteristic information.

[0008] The 3D data encoding device has a geometry encoding unit that encodes a geometric frame and an attribute encoding unit that encodes an attribute frame, and is characterized in that it further comprises a refinement information encoding unit that encodes refinement characteristic information and refinement activation information from the encoded data, and further comprises a refinement processing unit that encodes a syntax element indicating the number of refinements from the activation information, encodes an index indicating the characteristic information of the encoded number of refinements, and performs a refinement process on the attribute frame or the geometric frame according to the characteristic information. Effect of the Invention

[0009] According to one aspect of the present invention, distortion caused by encoding of color images can be reduced, and 3D data can be encoded and decoded with high quality. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing the configuration of a 3D data transmission system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing a hierarchical structure of data in an encoded stream. [Diagram 3] FIG. 2 is a diagram for explaining 3D data, occupancy, geometry frames, and attribute frames. [Figure 4] FIG. 11 is a diagram showing the relationship between characteristic SEI and applied SEI according to an embodiment of the present invention. [Diagram 5] 1 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. [Figure 6] FIG. 2 is a functional block diagram showing the configuration of a refinement unit 306. [Figure 7] 1 is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. [Figure 8] FIG. 2 is a functional block diagram showing the configuration of a refinement unit 306. [Figure 9] 13 is a flowchart showing a decoding process of SEI. [Figure 10] 13 is a flowchart showing the operation of a refinement unit 306. [Figure 11] 1 shows part (first half) of the syntax of the characteristic SEI. [Figure 12] This shows part (second half) of the syntax of the characteristic SEI. [Figure 13] An example of the syntax configuration of an applicable SEI is shown below. [Figure 14] 1 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to an embodiment of the present invention. [Figure 15] 3A to 3D data encoding device, a 3D data decoding device, and a relationship with an SEI message according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] FIG. 1 is a schematic diagram showing the configuration of a 3D data transmission system 1 according to this embodiment.

[0013] The 3D data transmission system 1 is a system that transmits an encoded stream obtained by encoding 3D data to be encoded, decodes the transmitted encoded stream, and displays the 3D data. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31, and a 3D data display device 41.

[0014] 3D data T is input to the 3D data encoding device 11.

[0015] The network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 may be the Internet, a wide area network (WAN), a local area network (LAN), or It is a combination of these. Network 21 is not necessarily limited to a two-way communication network, but also includes terrestrial The network 21 may be a one-way communication network that transmits broadcast waves such as digital broadcasts and satellite broadcasts. The network 21 may also be a one-way communication network that transmits digital broadcast waves such as digital broadcast waves and satellite broadcasts. The above-mentioned may be replaced by a storage medium on which the encoded stream Te is recorded, such as a digital still camera (DV camera) or the like.

[0016] The 3D data decoding device 31 decodes each of the encoded streams Te transmitted by the network 21, and generates one or more decoded 3D data Td.

[0017] The 3D data display device 41 displays all or part of one or more pieces of decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 is equipped with a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. The display may be in the form of a stationary display, a mobile display, an HMD, or the like. When the device has a high processing capability, it displays a high quality image, and when the device has a lower processing capability, it displays an image that does not require high processing and display capabilities.

[0018] <operator> The operators used in this specification are listed below.

[0019] >> is a right bit shift, << is a left bit shift, & is a bitwise AND, | is a bitwise OR , |= is the OR assignment operator, and || indicates logical OR.

[0020] x?y:z is a ternary operator that takes y if x is true (non-zero) and z if x is false (zero). Let y..z denote the ordered set of integers from y to z.

[0021] Clip3(a, b, c) is a function that clips c to a value between a and b (inclusive). If c < a, it returns a; if c > b, it returns b; otherwise, it returns c (assuming a <= b).

[0022] abs(a) is a function that returns the absolute value of a.

[0023] Floor(a) is a function that returns the largest integer less than or equal to a.

[0024] Round(a) is a function that returns the integer closest to a by rounding.

[0025] a / d represents the division of a by d, with the decimal part truncated.

[0026] <Structure of the Encoded Stream Te>[[]]END]] The data structure of the encoded stream Te, which is generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31, will be described.

[0027] Figure 2 is a diagram showing the hierarchical structure of the data in the encoded stream Te. The encoded stream Te has either the data structure of the V3C sample stream or the V3C unit stream. The V3C sample stream includes a sample stream header and V3C units, while the V3C unit stream includes V3C units. The V3C unit includes a V3C unit header and a V3C unit payload. The beginning of the V3C unit (= V3C unit header) is the Unit Type, which is an ID indicating the type of the V3C unit and takes values indicated by labels such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, V3C_OVD, etc.

[0028]

[0029] ​If the Unit Type is V3C_VPS (Video Parameter Set), the V3C unit payload includes a V3C parameter set.

[0030] If the Unit Type is V3C_AD (Atlas Data), the V3C unit payload is VPS It includes ID, atlasID, sample stream nal header and multiple NAL units. ID is an abbreviation for Idenfication and takes an integer value equal to or greater than 0. This atlasID may be used as an element of the applicable SEI.

[0031] A NAL unit contains a NALUnitType, a layerID, a TemporalID, and a RBSP (Raw byte sequence payload).

[0032] NAL units are identified by NALUnitType, and are classified as ASPS (Atlas Sequence Parameter Set), AAPS (Atlas Adaptation Parameter Set), ATL (Atlas Tile layer), SEI (Supplemental Enhancement Information), etc.

[0033] The ATL includes an ATL header and an ATL data unit, and the ATL data unit includes information such as the position and size of a patch, such as patch information data.

[0034] The SEI includes payloadType, which indicates the type of SEI, payloadSize, which indicates the size (number of bytes) of the SEI, and sei_payload, which is the data of the SEI.

[0035] If UnitType is V3C_AVD (Attribute Video Data), the VPS ID, atlasID, attrIdx (syntax name vuh_attribute_index) of the attribute frame ID, The data includes a partition ID partIdx (vuh_attribute_partition_index), a map ID mapIdx (vuh_map_index), a flag auxFlag (vuh_auxiliary_video_flag) indicating whether the data is auxiliary data, and a video stream. The video stream indicates data such as HEVC or VVC.

[0036] If UnitType is V3C_GVD (Geometry Video Data), it includes VPS ID, atlasID, mapIdx, auxFlag, and video stream.

[0037] If UnitType is V3C_OVD (Occupancy Video Data), it contains the VPS ID, atlasID, and video stream.

[0038] Attribute data and geometry data have multiple maps distinguished by mapIdx, and attribute data has multiple attributes distinguished by attrIdx.

[0039] (Data structure of 3D information) In this specification, three-dimensional information (3D data) refers to position information (x, y, z) in three-dimensional space and attributes. For example, 3D data is expressed in the form of a point cloud, which is a group of points containing position information and attribute information in three-dimensional space, or a mesh (or polygon) consisting of the vertices and faces of triangles.

[0040] Fig. 3 shows the 3D data, the occupancy frame (a two-dimensional image that represents occupancy information, hereafter referred to as the occupancy frame), the geometric frame (a two-dimensional image that represents depth or position information, hereafter referred to as the geometric frame), and the To explain tribute frames (two-dimensional images that represent attribute information, hereafter referred to as attribute frames) 3(a) is a diagram of the 3D data encoding device 11. The point cloud and mesh that constitute the 3D data are divided into multiple parts (areas) by the 3D data encoding device 11, and the point cloud contained in each part is projected onto one of the planes of a 3D bounding box (Fig. 3(a)) set in the 3D space. The 3D data encoding device 11 generates multiple patches from the projected point cloud. Information about the 3D bounding box (coordinates, size, etc.) and information about mapping onto the projection plane (projection plane, coordinates, size, rotation presence or absence of each patch, etc.) are called atlas information. The occupied frame is a two-dimensional image that shows the valid area of ​​each patch (area where the point cloud and mesh exist) as a 2D binary image (e.g., valid area is 1 and invalid area is 0) (Fig. 3(b). The values ​​of the valid area and invalid area are You can use values ​​other than 0 and 1, such as 255 and 0. The geometric frame is the projection plane of each patch. The attribute frame is a two-dimensional image showing the depth value (distance) of a point (Fig. 3(c-1)(c-2)). The relationship between the depth value and the pixel value may be linear, or the distance may be derived from the pixel value using a lookup table, a mathematical formula, or a relational formula based on a combination of branching by values. The attribute frame is a two-dimensional image showing the attribute information of the point (e.g. RGB color). (Fig. 3(d-0)(d-1) ). Even when projecting in the same direction, the depth of the object surface (foreground and background) may differ, and mapIdx indicates the depth at which projection is to be performed.

[0041] The occupancy frame, geometric frame, attribute frame, and atlas information may be images in which partial images (patches) from different projection planes are mapped (packed) onto a certain two-dimensional image. The atlas information includes information on the number of patches and the projection planes corresponding to the patches. The 3D data decoding device 31 reconstructs coordinates and attribute information of the point cloud or mesh from the atlas information, occupancy frame, geometric frame, and attribute frame. Here, the point is each point of the point cloud or the vertex of the mesh. Note that, instead of the occupancy frame and geometric frame, mesh information (position information) indicating the vertices of the mesh may be coded, decoded, and transmitted. In addition, the mesh information may be coded, decoded, and transmitted by dividing it into a base mesh constituting a basic mesh, which is a subset of the mesh, and a mesh displacement. The mesh displacement is a displacement from the base mesh to indicate a mesh other than the base mesh.

[0042] <Terminology> Atlas: A collection of 2D bounding boxes and related information placed in a rectangular frame that corresponds to the volume in 3D space where the volumetric data is being rendered. Attributes: Scalar or vector properties arbitrarily associated with each point in a volumetric frame, such as color, reflectivity, surface normal, transparency, material ID, etc. Attribute frame: a two-dimensional rectangular array created as a collection of patches that contain values ​​for a particular attribute. . Attribute Map: An attribute frame that contains attribute patch information projected to a particular depth indicated by the corresponding geometric map. Bit Stream: An ordered series of bits that form a coded representation of data. Encoded Data. Coded Atlas Sequence (CAS): A sequence of Coded Atlas AUs (access units) of an IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 1 followed in decoding order by a Coded Atlas AU that is not an IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 0 or greater and 1. All subsequent Coded Atlas AUs up to, but not including, an IRAP Coded Atlas AU with NoOutputBeforeRecoveryFlag equal to 1 are included. IRAP coded atlas AU is an IDR (instantaneous decoding refresh) coded access unit. , BLA (broken link access) coding access unit, CRA (clean random access) coding It may be any of the access units. Geometry: A set of Cartesian coordinates relative to a volumetric frame. Geometry Frame: A 2D array created by aggregating the geometry patch information associated with each patch. Geometry Map: A geometry frame that contains the geometry patch information projected to a particular depth. occupancy: A value indicating whether the atlas sample corresponds to the associated sample in 3D space. Occupancy frame: A collection of occupancy values ​​forming a 2D array, which represents one atlas frame. Represents the entire occupancy. Patch: A rectangular region associated with volumetric information in the atlas.

[0043] (Overview of characteristic SEI and application SEI) FIG. 4(a) shows the refinement characteristic SEI (hereinafter referred to as characteristic SEI, nn_refinement_characteristics SEI, NNRC_SEI, characteristic information) and the refinement application SEI (hereinafter referred to as application SEI, nn_refinement 1 is a diagram showing the relationship between the characteristic SEI, the activation SEI, the NNRA_SEI, the activation SEI, and the activation information. The NNC decoding unit decodes a compressed NN (Neural Network) model transmitted in the characteristic SEI, and derives the NN model. MPEG Neural Network Coding (ISO / IEC 15938-19) may be used for encoding and decoding the NN model. The refinement unit 306 includes an NN filter unit 611. Then, the NN filter unit 611 performs a filtering process using the derived NN model. The details of the portion 611 will be described later. The characteristic SEI used for refinement is a syntax element of the application SEI, For example, it is specified by nnra_target_id. The applicable SEI is held using the syntax element The NN filter unit 611 transmits persistence information and specifies that filtering is performed on the attribute frame for a duration specified by the persistence information. The NN filter unit 611 may include an NNC decoder. The applied SEI (activation information) is not limited to the SEI in FIG. 13, and may be data including NAL units and atlas data.

[0044] FIG. 4B is another diagram showing the relationship between the refinement characteristics SEI and the refinement application SEI according to the embodiment of the present invention. The refinement unit 306 includes a linear filter unit (ALF unit) 610. The meter decoder decodes the syntax of the filter parameters transmitted in the characteristic SEI and converts them into line The ALF unit 610 derives coefficients for a shape filter. This is not limited to a loop filter, but may be a post filter. Although this is described as linear, it may include nonlinear elements such as simple clipping and squared components. The ALF unit 610 performs filtering on the attribute frame or geometry frame using the derived linear model.

[0045] (Configuration of 3D data decoding device) 5 and 7 are functional block diagrams showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention.

[0046] The 3D data decoding device 31 includes a V3C unit decoding unit 301 and an atlas decoding unit 302 (refining information decoding unit), an occupancy decoding unit 303, a geometry decoding unit 304, an attribute decoding unit 305, It is composed of a post-decoding conversion unit 308 , a pre-reconstruction unit 310 , a reconstruction unit 311 , and a post-reconstruction unit 312 .

[0047] The V3C unit decoder 301 receives coded data (bit stream) in a byte stream format, ISO Base Media File Format (ISOBMFF), or the like, and decodes the V3C unit header and V3C VPS The V3C unit decoder 301 decodes the attribute according to the UnitType in the V3C unit header. The atlas decoder 302, the occupancy decoder 303, the geometry decoder 304, and the attribute decoder 305 are selected. If the UnitType is V3C_AD, the atlas decoder 302 is used. If V3C_OVD, V3C_GVD, or V3C_AVD, the occupancy decoding unit 303, the geometry decoding unit 304, The attribute decoder 305 decodes the occupied frame, the geometric frame, and the attribute frame. Decrypt the

[0048] The atlas decoding unit 302 receives the atlas data and decodes the atlas information.

[0049] The atlas decoding unit 302 (refinement information decoding unit) extracts refinement information from the encoded data. The refinement information decoding unit decodes the characteristic SEI that indicates the characteristic of the refinement information. Furthermore, the atlas decoding unit 302 decodes the information of the target of refinement. The applicable SEI is decrypted from the data.

[0050] The atlas decoding unit 302 decodes an identifier atlasID indicating target atlas information indicating the target of refinement from a V3C unit including the applicable SEI.

[0051] The occupancy decoding unit 303 decodes the occupancy data encoded by VVC, HEVC, or the like, and outputs the occupancy frames DecOccFrames[frameIdx][compIdx][y][x]. mes, frameIdx, compIdx, y, and x respectively indicate the decoded occupied frame, frame ID, component ID, row index, and column index. In DecOccFrames, compIdx=0. It's fine.

[0052] The geometry decoding unit 304 decodes geometry data encoded in AVC, VVC, HEVC, or the like. and output the geometric frame DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]. where DecGeoFrames, frameIdx, mapIdx, compIdx, y, and x are the decoded Indicates the geometric frame, frame ID, map ID, component ID, row index, and column index. DecGeoBitDepth, DecGeoHeight, DecGeoWidth, and DecGeoChromaFormat indicate the geometric frame, frame ID, map ID, component ID, row index, and column index. It is expressed as the bit depth of the frame, the height of the geometry frame, the width of the geometry frame, and the chroma format of the geometry frame. A decoded geometry frame may contain multiple geometry maps (geometry frames with different depth projections), and mapIdx is used to distinguish between the maps. It is allowed that compIdx=0 for DecGeoFrames.

[0053] The attribute decoding unit 304 decodes attribute data encoded by VVC, HEVC, etc., and outputs an attribute frame DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]. Here, DecAttrFrames, frameIdx, attrIdx, partIdx, mapIdx, compIdx, y, and x respectively indicate the decoded attribute frame, frame ID, attribute ID, partition ID, map ID, component ID, row index, and column index. DecAttrBitDepth, DecAttrHeight, DecAttrWidth, and DecAttrChromaFormat indicate the attribute data. The decoded attribute frame is indicated as the attribute frame's attribute frame's bit depth, attribute frame's height, attribute frame's width, and attribute frame's chroma format. The decoded attribute frame may contain multiple attribute maps (attribute frames with different depth projections), and the mapIdx is used to distinguish between the maps. The decoded attribute frame is indicated as the attribute frame's color (R,G,B), reflectance, alpha, normal It consists of multiple attributes such as direction. Multiple attributes can be transmitted in multiple attribute data, and attrIdx is used to distinguish them. For example, {R {, G, B} is attribute data 0 (attrIdx = 0), {reflection} is attribute data 1 (attrIdx = 1), and {alpha} is attribute data 2 (attrIdx = 2). The partIdx is used to distinguish between the two streams. The mapIdx is as described above.

[0054] The post-decoding conversion unit 308 receives the decoded atlas information, the decoded occupation frames DecOccFrames, the decoded geometry frames DecGeoFrames, and the decoded attribute frames DecAttrFrames, and converts them into a nominal format. The output is OccFramesNF[frameIdx][CompTimeIdx][y][x], GeoFramesNF[mapIdx][CompTimeIdx][frameIdx][y][x], AttrFramesNF[attrIdx][mapIdx][CompTimeIdx][compIdx][y][x] of the nominal format of the attribute frame, where frameIdx, CompTimeIdx, y, x, mapIdx, attrIdx, and compIdx are the frame ID, composite time index, row index, column index, map ID, attribute ID, and component ID, respectively.

[0055] Nominal format collectively refers to the nominal bit depth, resolution, chroma format, and constituent time indexes to which the decoded video should be converted. Each video sub-bitstream and each region of the packed video sub-bitstream is associated with a nominal bit depth. This is the expected target bit depth for all operations for reconstruction. The nominal occupancy bit depth OccBitDepthNF is set to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1. oi_occupancy_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which the occupancy frames of the atlas with atlasID=j should be converted. pin_occupancy_2d_bit_depth_minus1[j]+1 contains the occupancy data of the atlas with atlasID=j. Indicates the nominal 2D bit depth to which the decoded region is converted. If atlasID=j is equal to , the packed video frame of the atlas with atlasID=j is the occupancy data. If pin_occupancy_present_flag[j] is equal to 1, the packed video frame of atlas with atlasID=j contains an area with occupancy data. If pin_occupancy_present_flag[j] is not present, its value is inferred to be equal to 0. The nominal bit depth GeoBitDepthNF of each geometry frame is set to gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 if pin_geometry_present_flag[ConvAtlasID] is equal to 1. gi_geometry_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which all geometry frames of the atlas with atlasID=j are transformed. pin_geometry_2d_bit_depth_minus1[j]+1 indicates the nominal 2D bit depth to which the decoded region containing geometry data of the atlas with atlasID=j is transformed. pin_geometry_present_flag[j]=0 represents the packed image frame of the atlas with atlasID=j. pin_geometry_present_flag[j]=1 indicates that the packed image frame of the atlas with atlasID=j contains no region with geometry data. If pin_geometry_present_flag[j] is not present, its value is equal to 0. Finally, if pin_attribute_present_flag[ConvAtlasID]=1, the nominal bit depth AttrBitDepthNF[attrIdx] of each attribute frame with attrIdx is set to ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1 or pin_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]. ai_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which all attribute frames with attrIdx=i are converted for the atlas with atlasID=j. pin_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth to which all attribute frames with attrIdx=i are converted for the atlas with atlasID=j. indicates the nominal 2D bit depth to which the atlas is converted. pin_attribute_present_flag[j]=0 indicates that the packed video frame of atlas with atlasID=j does not contain any attribute data. pin_attribute_present_flag[j] = 1 indicates that the packed video frame of the atlas with atlasID=j contains a field of attribute data.

[0056] ConvAtlasID is set equal to vuh_atlas_id or determined by external means if the V3C unit header is not available. vuh_atlas_id is one of the following: V3C_AD, V3C_OVD, V3C_GVD, V3C_AVD Specifies the ID of the atlas which is indicated in the V3C unit header and which corresponds to the current V3C unit.

[0057] asps_frame_width is the frame width of the atlas in integer samples, corresponding to the luma samples of the video component. For V3C bitstream compliance, asps_frame_width must be equal to the value of vps_frame_width[j], where j is the current atlasID. asps_frame_height is the frame height of the atlas in integer samples. The pulls correspond to the luma samples of the video component. It is a V3C bitstream conformance requirement that the value of asps_frame_height be equal to the value of vps_frame_height[j]. where j denotes the current atlasID. The nominal frame resolution of the auxiliary video component is the nominal width and height specified by the variables AuxVideoWidthNF and AuxVideoHeightNF, respectively. AuxVideoWidthNF and AuxVideoHeightNF are the auxiliary video subframes associated with the atlas. Obtained from the bitstream.

[0058] The nominal chroma format is defined as 4:4:4.

[0059] The post-decoding conversion unit 308 performs bit depth conversion, resolution conversion, output order conversion, atlas conversion, and the like. The video frames provided by the V3C decoder 309 may require additional processing steps before being input to the reconstruction process. Such processing steps may include conversion of the decoded frames to a nominal format (e.g., resolution, bit depth, chroma format, etc.). This information is signaled in the V3C VPS.

[0060] The pre-reconstruction unit 310 receives the decoded atlas information, the decoded occupied frames, and the decoded It receives the geometry frame and the decoded attribute frame and can refine / modify them. Specifically, if the occupancy synthesis flag os_method_type[k] is set to patchbo If it is equal to 1, which indicates patch border filtering, then occupancy compositing is invoked with OccFramesNF[compTimeIdx][0] and GeoFramesNF[0][compTimeIdx][0] as inputs and the modified array OccFramesNF[compTimeIdx][0] as output. OccFramesNF indicates the occupancy frames decoded in nominal format, and GeoFramesNF indicates the occupancy frames decoded in nominal format. compTimeIdx indicates the geometry frame decoded in this bit. compTimeIdx is the index of the composition time.

[0061] The refinement unit 306 refines the occupancy, geometry, and attributes on a frame-by-frame basis within the pre-reconstruction unit 310. The refinement may be a filter that inputs any one of the occupancy, geometry, and attributes and outputs any one of the occupancy, geometry, and attributes. The refinement unit 306 may simultaneously use two-dimensional occupancy, geometry, and attributes to perform the refinement process.

[0062] Furthermore, the refinement unit 306 may include an NN filter unit 611 or an ALF unit 610, and may perform filtering based on the transmitted neural network parameters or linear parameters.

[0063] The reconstruction unit 311 receives atlas information, occupancy, and geometry, and reconstructs the 3D space The reconstruction unit 311 reconstructs the positions, attributes, or meshes of the vertices or points. Geometry information (e.g. recPcGeo) and attribute information (e.g. recPcAttr) Based on the above, the reconstruction unit 311 reconstructs point cloud data or mesh of the 3D data. Specifically, the reconstruction unit 311 inputs the nominal video frame, OccFramesNF[compTimeIdx][0][y][x], GeoFramesNF[mapIdx][compTimeIdx][0][y][x], and AttrFramesNF[attrIdx][mapIdx][compTimeIdx][ch][y][x] derived by the pre-reconstruction unit 310, and reconstructs point cloud data or mesh of the 3D data. AttrFramesNF indicates an attribute frame decoded in the nominal format. The reconstruction unit 311 inputs the variable pointCnt as the number of points in the reconstructed point cloud frame, The patch index corresponding to each point is the one-dimensional array pointToPatch[pointCnt]. We derive a 2D array pointToPixel[pointCnt][dimIdx] as the atlas coordinates corresponding to each reconstructed point. We have a 2D array recPcGeo[pointCnt][dimIdx] as the list of coordinates corresponding to each reconstructed point, and a 3D array recPcAttr[pointCnt][attrIdx][compIdx] as the attributes associated with the points in the reconstructed point cloud frame, where pointCnt, attrIdx, and compIdx correspond to the size of the reconstructed point, the ID of the attribute frame, and the component ID, respectively. dimIdx represents the (x,y) component of each reconstructed point. Specifically, the reconstruction unit 311 derives recPcGeo and recPcAttr as follows. m = 0 for(k=0; k<3; k++) { for(pointIdx=0; pointIdx <AtlasPatchRawPoints[pIdx]; pointIdx++) { y = AtlasPatch2dPosY[pIdx] + (m / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (m % AtlasPatch2dSizeX[pIdx]) rawPos1D[m] = gFrame[mapIdx][y][x] m++ } } for(pointIdx=0; pointIdx<AtlasPatchRawPoints[pIdx]; pointIdx++) { recPcGeo[pointCnt][0] = rawPos1D[pointIdx] + AtlasPatch3dOffsetU[pIdx] recPcGeo[pointCnt][1] = rawPos1D[pointIdx + AtlasPatchRawPoints[pIdx]] + AtlasPatch3dOffsetV[pIdx] recPcGeo[pointCnt][2] = rawPos1D[pointIdx + 2 * AtlasPatchRawPoints[pIdx]] + AtlasPatch3dOffsetD[pIdx] y = AtlasPatch2dPosY[pIdx] + (pointIdx / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (pointIdx % AtlasPatch2dSizeX[pIdx]) for(attrIdx=0; attrIdx<ai_attribute_count[RecAtlasID]; attrIdx++) { attrDim = ai_attribute_dimension_minus1[RecAtlasID][attrIdx] + 1 for(compIdx=0; compIdx<attrDim; compIdx++) { recPcAttr[pointCnt][attrIdx][compIdx] = aFrame[attrIdx][compIdx][y][x]} } if(ai_attribute_count[RecAtlasID]>0) { attrPresent[pointCnt] = 1 } pointToPixel[pointCnt][0] = -1 pointToPixel[pointCnt][1] = -1 pointToPatch[pointCnt] = pIdx pointCnt++ } where pIdx is the patch index, compTimeIdx is the composition time index, rawPos1D is the 1D position, gFrame and aFrame are the geometry and attribute frames, respectively, TilePatch3dOffsetU is the associated tile patch parameter from the patch, and ai_attribute_count[j] is the number of attributes associated with the atlas with atlasID=j (the number of attributes encoded and decoded in the VPS). AtlasPatch3dOffsetU, AtlasPatch3dOffsetV, and AtlasPatch3dOffsetD are parameters that indicate the position of the 3D bounding box in Figure 3(a) on the 3D model.

[0064] for(k=0; k<3; k++) { for(pointIdx=0; pointIdx <AtlasPatchRawPoints[pIdx]; pointIdx++) { y = AtlasPatch2dPosY[pIdx] + (m / AtlasPatch2dSizeX[pIdx]) x = AtlasPatch2dPosX[pIdx] + (m % AtlasPatch2dSizeX[pIdx]) rawPos1D[m] = gFrame[mapIdx][y][x] m++ } } Here, AtlasPatchRawPoints, AtlasPatch2dPosX, AtlasPatch2dPosY, AtlasPatch2dSizeX, and AtlasPatch2dSizeY are patch information derived from the atlas information by the atlas decoding unit 302. The arrays oFrame[y][x], gFrame[mapIdx][y][x], and aFrame[mapIdx][attrIdx][compTimeIdx][y][x] are derived as follows: for(j=0; j <asps_frame_height; j++) { for(i=0; i <asps_frame_width; i++) { oFrame[j][i] = OccFramesNF[compTimeIdx][0][j][i] for(m=0; m <asps_map_count_minus1+1; m++) { gFrame[m][j][i] = GeoFramesNF[m][compTimeIdx][0][j][i] for(a=0; a <ai_attribute_count[RecAtlasID]; a++) { for(c=0; c <ai_attribute_dimension_minus1[RecAtlasID][a]+1; c++) { aFrame[m][a][c][j][i] = AttrFramesNF[a][m][compTimeIdx][c][j][i] } } } } } Here, ai_attribute_dimension_minus1 is encoded and decoded by the encoded data of the V3C VPS, and ai_attribute_dimension_minus1+1 is the total dimension of the attribute decoded by the V3C unit decoder 301. The number of elements (i.e., the number of channels, components). asps_map_count_minus1 is the code for V3C VPS. sps_map_count_minus1+1 indicates the number of maps of geometry data and attribute data of the current atlas. RecAtlasID indicates the ID of the decoded atlas.

[0065] The post-reconstruction unit 312 modifies (updates) the 3D data point cloud data or meshes after processing by the reconstruction unit 311. The post-reconstruction unit 312 modifies (updates) the current point cloud frame associated with the current atlas. It takes pointCnt as the number of reconstructed points, the one-dimensional array attrBitDepth[] as the nominal bit depth, oFrame[][], recPcGeo[][], and recPcAttr[][][], applies geometric smoothing, and outputs the modified recPcGeo[][], recPcAttr[][][].

[0066] (Refinement Part 306) FIG. 6 is a functional block diagram showing the configuration of the refinement unit 306 (306DEC).

[0067] The refinement unit 306 (306DEC) includes an occupancy decoding unit 303, a geometry decoding unit 304, and an attribute decoding unit 306DEC. The refinement is applied to the DecOccFrames, DecGeoFrames, and DecAttrFrames that have been decoded by the view decoding unit 305 and before being converted by the post-decoding conversion unit 308. The refined DecOccFrames, DecGeoFrames, and DecAttrFrames are then output to the post-decoding conversion unit 308.

[0068] The refining unit 306, in accordance with the refinement target information, refines DecOccFrames[0][frameIdx][y][x], mapIdx, DecGeoFrames[mapIdx][frameIdx][0][y][x] for each frameIdx, attrIdx, and partIdx. The refinement unit 306 inputs DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] for each mapIdx, frameIdx, and compIdx, and outputs modified DecOccFrames, DecGeoFrames, and DecAttrFrames. The refinement unit 306 stores the output in DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x], and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] according to the refinement target information.

[0069] The refinement unit 306 refines the two-dimensional array oFrame of asps_frame_height × asps_frame_width, the two-dimensional array gFrame of asps_frame_height × asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1) × asps_frame_height × asps_frame_width. Refinement processing may be performed. In this case, oFrame = DecOccFrames[0][frameIdx], gFrame = DecGeoFrames[mapIdx][frameIdx][0], aFrame = DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx] may be input. Here, attrIdx and mapIdx are used to decode the NNRC SEI. attrIdx = nnrc_attribute_index, mapIdx = nnra_target_map_index. frameIdx may be compTimeIdx, which is the index of the composition time. Also, attrIdx and mapIdx may be the values ​​nnra_target_map_index and nnra_target_attribute_index obtained by decoding the NNRA SEI. attrIdx = nnra_target_attribute_index, mapIdx = nnrc_map_index. The same applies below.

[0070] According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristics SEI. Then, using the refinement target information specified in the characteristic SEI, occupancy, geometry, By specifying one of the attributes and applying refinement processing to multiple inputs simultaneously, the image quality is improved.

[0071] FIG. 7 shows a case where the refining unit 306 refines an image signal (video signal) in a nominal format. Here is an example of how to apply this ment.

[0072] 8 is a functional block diagram showing the configuration of the refinement unit 306 (306NF). The refinement unit 306 (306NF) applies refinement to the OccFramesNF, GeoFramesNF, and AttrFramesNF in the nominal format converted by the post-decoding conversion unit 308 according to the refinement target information, and outputs the GeoFramesNF and AttrFramesNF after the refinement to the pre-reconstruction unit 310. The processing of the refinement unit 306 will be explained below, but the processing of the refinement unit 306DEC is performed by replacing OccFramesNF, GeoFramesNF, and AttrFramesNF with DecOccFrames, DecGeoFrames, and DecAttrFrames in FIG. 6.

[0073] The refinement unit 306 may perform refinement processing on the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1). In this case, oFrame = OccFramesNF[compTimeIdx][0], gFrame = GeoFramesNF[mapIdx][compTimeIdx][frameIdx], and aFrame = AttrFramesNF[attrIdx][mapIdx][compTimeIdx] may be input.

[0074] In FIG. 10(a), when the refinement target information indicates an occupancy (S3062I), the input Add occupancy to the console (S3063I). For example, the following (Eq. IN-OCC) may be used.

[0075] ch = 0 if (OccupancyUsedFlag) { (Eq. IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yP][xP]) ch++ } If the refinement target information indicates a geometry (S3064I), add the geometry to the input tensor (S3065I). For example, the following (Eq. IN-GEO) may be used.

[0076] if (GeometryUsedFlag) { (Eq. IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yP][xP]) ch++ } If the refinement target information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, the following (Eq. IN-ATTR) may be used.

[0077] if (AttributeUsedFlag) { (Eq. IN-ATTR) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[0][yP][xP]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[1][yP][xP]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[2][yP][xP]); ch++ According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristics SEI. Using the refinement target information specified in the characteristic SEI, occupancy, geometry, By specifying one of the attributes and applying refinement processing to multiple inputs simultaneously, the image quality is improved.

[0078] (NN filter section 611) The NN filter unit 611 performs filtering using a neural network. The neural network is represented by a neural network model, and convolution (Co nv).

[0079] The neural network model (hereinafter, NN model) means the elements and connection relationship (topology) of a neural network, and the parameters (weights, biases) of the neural network. Note that the NN filter unit 611 fixes the topology and filters the image to be filtered. Alternatively, only the parameters may be switched depending on the image. The neural network may include a convolution defined by a kernel size, the number of input channels, and the number of output channels.

[0080] The input to the refinement unit 306 is DecFrame. The refinement unit 306 derives an input InputTensor to the NN filter unit 611 from the input image DecFrame, and the NN filter unit 611 performs filtering using the inputTensor with a neural network model to derive an outputTensor. The neural network model used is a model corresponding to nnra_target_id. The input image may be an image for each component, or may be an image having multiple components as channels.

[0081] The NN filter unit 611 may repeatedly apply the following process.

[0082] The NN filter unit 611 performs convolutional operations on the inputTensor and the kernel k[m][n][yy][xx] for the number of layers. The output image outputTensor is generated by performing convolution and adding bias. Here, m is the number of channels in the inputTensor, n is the number of channels in the outputTensor, yy is the height of kernel k, and xx is the width of kernel k.

[0083] At each layer, an outputTensor is generated from an inputTensor.

[0084] outputTensor[nn][yy][xx]=ΣΣΣ(k[mm][nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Here, nn=0..n-1, mm=0..m-1, yy=0..height-1, xx=0..width-1, i=0..yy-1, j=0..xx-1. width is the width of inputTensor and outputTensor, and height is the height of inputTensor and outputTensor. Σ is the sum of mm=0..m-1, i=0..yy-1, j=0..xx-1, respectively. of is the width or height of the area required around the inputTensor to generate the outputTensor.

[0085] In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, of=0 is set. In the case of 3x3 Conv, Σ represents the sum of mm=0..m-1, i=0..2, j=0..2. , set of=1. When the value of yy+j-of is less than 0 or greater than or equal to height, or when the value of xx+i-of is less than 0 or greater than or equal to width, the values ​​of inputTensor[mm][yy+j-of][xx+i-of] may be set to 0. Alternatively, the values ​​of inputTensor[mm][yy+j-of][xx+i-of] may be set to inputTensor[mm][yclip][xclip]. where yclip is max(0, min(yy+j-of, height-1)), and xclip is (0, min(xx+i-of, width-1)).

[0086] In the next layer, the obtained outputTensor is used as a new inputTensor, and similar processing is repeated for each layer. Repeat for each layer. An activation layer may be placed between layers. Also, a pooling layer or skip layer may be placed. Finally, we derive the FilteredFrame from the outputTensor.

[0087] In addition, using the kernel k[n][yy][xx], the process shown in the following equation, called Depth-wise Conv, is performed. where nn=0..n-1, xx=0..width-1, and yy=0..height-1.

[0088] outputTensor[nn][yy][xx]=ΣΣ(k[nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn]) Also, a nonlinear process called Activate, for example, ReLU, may be used. ReLU(x) = x >= 0 ? x : 0 Alternatively, leakyReLU shown in the following formula may be used.

[0089] leakyReLU(x) = x >= 0 ? x : a*x Here, a is a predetermined value less than 1, for example, 0.1 or 0.125. In order to perform integer arithmetic, all the above values ​​of k, bias, and a may be set to integers, and a right shift may be performed after conv to generate the outputTensor.

[0090] In ReLU, 0 is always output for values ​​less than 0, and the input value is output as is for values ​​greater than or equal to 0. On the other hand, in leakyReLU, linear processing is performed with the gradient set by a for values ​​less than 0. In ReLU, the gradient for values ​​less than 0 disappears, so learning may not progress smoothly. In leakyReLU, the gradient for values ​​less than 0 is left, making the above problem less likely to occur. In addition, among the above leakyReLU(x), PReLU, which uses a parameterized value of a, may be used.

[0091] (Filtering process of NN filter unit 611) Set attrIndex to nnrc_tareget_attribute_index and mapIndex to nnra_tareget_map_index[ i ] and apply the following. Invoke the following process using OccFramesNF[ compTimeIdx ]

[0000] , GeoFramesNF[ mapIdx ] [ compTimeIdx ]

[0000] , AttrFramesNF[ attrIdx ] [ mapIdx ] [ compTimeIdx ], and nnra_strength[ i ] as input, and the modified GeoFramesNF[ mapIdx ] [ compTimeIdx ]

[0000] , and AttrFramesNF[ attrIdx ] [ mapIdx ] [ compTimeIdx ] as output. The inputs for this process are: The variable nnrc_occupancy_flag, specified in the characteristic SEI, indicates whether to use occupied frames. The variable nnrc_geometry_flag, specified in the feature SEI, indicates whether to use a geometry frame. The variable nnrc_attribute_flag, specified in the characteristic SEI, indicates whether to use an attribute frame. The variable nnrc_strength_num is specified in the characteristic SEI and indicates the number of strength information to be used. oFrame is a 2D array of size asps_frame_height × asps_frame_width. gFrame is a 2D array of size asps_frame_height × asps_frame_width. 3D array aFrame, size is ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ] × asps_frame_height × asps_frame_width. 1D array nnraStrength, size nnrc_strength_num. nnraStrength is the activation information. The encoded data (nnra_Strength[]) is decoded and input to the input tensor. This is a parameter that indicates the strength of the ment. The output of this process is the modified arrays gFrame and aFrame.

[0092] Here, ai_attribute_dimension_minus1 is the number of elements of the attribute being encoded and decoded in VPS minus 1. For YUV and RGB, this is usually 3.

[0093] Additionally, in configurations that support simultaneous refinement of multiple attributes, mapIndex may be set to nnra_tareget_map_index[ i ] and the following may be applied: The following program is executed with OccFramesNF[ compTimeIdx ]

[0000] , GeoFramesNF[ mapIdx ][ compTimeIdx ]

[0000] , AttrFramesNF[ attrIdx ][ mapIdx ][ compTimeIdx ], nnra_strength[ i ] as input and the modified GeoFramesNF[ mapIdx ][ compTimeIdx ]

[0000] , AttrFramesNF as output. Call the process. The inputs for this process are: The variable OccupancyUsedFlag (nnrc_occupancy_flag), which is specified in the characteristic SEI, is used to indicate whether an occupied frame is used. This indicates whether The variable GeometryUsedFlag (nnrc_geometry_flag), which is specified in the characteristic SEI and uses the geometry frame. Indicates whether The variable AttributeUsedFlag (nnrc_attribute_flag), which is specified in the characteristic SEI and uses the attribute frame This indicates whether The variable nnrc_strength_num is specified in the characteristic SEI and indicates the number of strength information to be used. oFrame is a 2D array of size asps_frame_height × asps_frame_width. gFrame is a 2D array of size asps_frame_height × asps_frame_width. 5D array aFrame, size ai_attribute_count[ RecAtlasID ] × ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ] × asps_frame_height × asps_frame_width. 1D array nnraStrength, size nnrc_strength_num, which indicates the strength information specified in the characteristic SEI that controls the strength of the refinement process. The output of this process is the modified arrays gFrame and aFrame.

[0094] The NN filter unit 611 performs NN filtering to derive an outputTensor from an inputTensor. Refinement processing (filtering) indicated by RefineFilter() may be performed in units of blocks (blockWidth x blockHeight) as shown below.

[0095] for(cTop=0;cTop <FrameHeight;cTop+=blockHeight) for(cLeft=0;cLeft <FrameWidth;cLeft+=blockWidth){ DeriveInputTensors() outputTensor=RefineFilter(inputTensor) StoreOutputTensors() } Here, DeriveInputTensors() is a function that sets input data, and StoreOutputTensors() is a function that stores output data. FrameWidth and FrameHeight are the size of the input data and may be asps_frame_width and asps_frame_height. BlockWidth and blockHeight are the width of the block. It is the height.

[0096] The NN filter unit 611 performs the following in accordance with the refinement target information in DeriveInputTensors(): Based on one or more of the occupancy frame oFrame, the geometric frame gFrame, and the attribute frame aFrame, the input data inputTensor[][][] to the NN filter unit 611 is derived. The following is one configuration example, and the configuration described later may also be used.

[0097] for(yP = -overlapSize; yP < blockHeight+overlapSize; yP++) for(xP = -overlapSize; xP < blockWidth+overlapSize; xP++) { yT = cTop + yP xT = cLeft + xP ch=0 if (OccupancyUsedFlag) { (Eq. IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yT][xT]) ch++ } if (GeometryUsedFlag) { (Eq. IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yT][xT]) ch++ } if (AttributeUsedFlag) { (Eq. IN-ATTR) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[0][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[1][yT][xT]); ch++ inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(aFrame[2][yT][xT ]); ch++ } } } for (i=0; i <nnrc_strength_num; i++) { inputTensor[ch][yP+overlapSize][xP+overlapSize] = InpS( nnraStrength[i]); ch++; } Here, OccupancyUsedFlag, GeometryUsedFlag, and AttributeUsedFlag are variables that indicate the information to be refined, and are expressed as the syntax (described later) obtained by decoding the characteristic SEI as follows: The element may be used.

[0098] OccupancyUsedFlag = nnrc_occupancy_flag GeometryUsedFlag = nnrc_geometry_flag AttributeUsedFlag = nnrc_attribute_flag <Specify input tensors and output tensors according to the information to be refined> Another configuration of the above will now be described.

[0099] FIG. 10 is a flowchart showing the operation of the refinement unit 306 .

[0100] The example shown in FIG. 10 will now be described in further detail.

[0101] In FIG. 10(a), the refinement unit 306 or the atlas decoding unit 302 receives refinement target information. The information is decrypted (S3061).

[0102] The refinement unit 306 may derive the inputTensor according to the refinement target information. Hereinafter, ch indicates the index (position) of the channel to be set. ch++ is an abbreviation for ch=ch+1. Indicates that the index of the channel to be set is to be incremented by +1.

[0103] ch=0 If the refinement target information indicates an occupancy (S3062I), the occupancy is added to the input tensor (S3063I). For example, the following formula (IN-OCC) may be used.

[0104] if (OccupancyUsedFlag) { (Eq. IN-OCC) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(oFrame[yT][xT]) ch++ } If the refinement target information indicates a geometry (S3064I), the geometry is added to the input tensor (S3065I). For example, the following (formula IN-GEO) may be used.

[0105] if (GeometryUsedFlag) { (Eq. IN-GEO) inputTensor[ch][yP+overlapSize][xP+overlapSize] = Inp(gFrame[yT][xT]) ch++ } If the refinement target information indicates an attribute (S3066I), add the attribute to the input tensor (S3067I). For example, as shown in the following (expression IN-ATTR-DIM), input all the channels of the attribute specified by the index of attrIdx to inputTensor.

[0106] if (AttributeUsedFlag) { (Eq. IN-ATTR-DIM) for (k=0;k< ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ];k++) { inputTensor[ch][yP+overlapSize][xP+overlapSize]= Inp(InpSampleVal(yT, xT, asps_frame_height, asps_frame_width, aFrame[k]); ch++ } Note that InpSampleVal(y, x, FrameHeight, FrameWidth, targetPic) { if(y<0 || x<0 || y>=FrameHeight || x>=FrameWidth) { sampleVal = 0 sampleVal = targetPic[Clip3(0, FrameHeight -1, y)][Clip3(0, FrameWidth -1, x)]} } else { sampleVal = targetPic[y][x] } Alternatively, if the refinement target information indicates an attribute, it is specified by the indices of attrIdx starting from nnrc_attribute_index and going up to nnrc_attribute_num_minus1+1. You can also input all the channels of the attributes to the inputTensor.

[0107] if (AttributeUsedFlag) { (Eq. IN-ATTR-POS) for (attrIdx = nnrc_attribute_index; attrIdx < nnrc_attribute_index + nnrc_attribute_num_minus1+1; attrIdx++) { for (k=0;k<= ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ];k++) { inputTensor[ch][yP+overlapSize][xP+overlapSize] = inputTensor[0][ch][yP+overlapSize][xP+overlapSize] = Inp(InpSampleVal(yT, xT, asps_frame_height, asps_frame_width), aFrame[attrIdx][mapIdx][ compTimeIdx ][k]); ch++ } } } In Fig. 10(b), DeriveInputTensors() selects the input tensors according to the refinement target information. The fine unit 306 may derive (update) the output frame from the outputTensor.

[0108] If the refinement target information indicates a geometry (S3064O), a specific component of the output tensor is set to the geometry (S3065O). For example, the following (formula OUT-GEO) may be used.

[0109] ch=0 if (GeometryUsedFlag) { gFrame[yP][xP]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } If the refinement target information indicates an attribute (S3066O), a specific component of the output tensor is set to the attribute (S3067O). For example, as shown in the following (expression OUT-ATTR-DIM), the value of outputTensor is set to the number of channels of the attribute specified by the index of attrIdx. Good too.

[0110] if (AttributeUsedFlag) {(expression OUT-ATTR-DIM) for (k=0;k< ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ];k++) { aFrame[k][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } Alternatively, if the refinement target information indicates an attribute, You may start by inputting all channels of attributes specified by the indices of attrIdx up to the number nnrc_attribute_num_minus1+1 into inputTensor.

[0111] if (AttributeUsedFlag) { for (attrIdx = nnrc_attribute_index; attrIdx < nnrc_attribute_index + nnrc_attribute_num_minus1+1; attrIdx++) { for (k=0;k<= ai_attribute_dimension_minus1[ RecAtlasID ][ attrIdx ];k++) { aFrame[attrIdx][mapIdx][0][k][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } } The refinement information decoding unit further decodes the number of pieces of strength information strengthNum from the characteristic information. The refinement processing unit is characterized in that it inputs at least one of an occupancy frame, a geometry frame, or an attribute frame to the input tensor of the neural network, and further inputs each of the strengthNum values ​​derived from the decoded strength information to each of the strengthNum channels of the input tensor of the neural network. Instead of Inp(oFrame[yT][xT]), you can also use Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth,oFrame)), which includes screen boundary processing. Instead of Inp(gFrame[yT][xT]), you can also use Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth,gFrame)), which includes screen boundary processing. Instead of Inp(aFrame[k][yT][xT]), you can use Inp(InSampleVal(yT,xT,FrameHeight,FrameWidth, aFrame[k])), which includes screen boundary processing. k=0,1,2,..., etc. And so on. Here, overlapSize is the size of the overlap, and a value decoded from the encoded data of the characteristic SEI may be used.

[0112] The NN filter unit 611 performs the following in StoreOutputTensors() according to the refinement target information. Then, set the contents of outputTensor to the modified geometry frame oFrame and attribute frame aFrame. for(yP=0; yP <blockHeight; yP++) for(xP=0; xP <blockWidth; xP++) { yT = cTop + yP xT = cLeft + xP if (yT <FrameHeight && xT<FrameWidth) ch=0 if (OccupancyUsedFlag) { oFrame[yT][xT]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } if (GeometryUsedFlag) { gFrame[yT][xT]=outputTensor[ch][yP+overlapSize][xP+overlapSize] ch++ } if (AttributeUsedFlag) { aFrame[0][yT][xT]=Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[1][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ aFrame[2][yT][xT]= Out(outputTensor[ch][yP+overlapSize][xP+overlapSize]); ch++ } } (ALF section 610) Note that the refinement specified by the persistence information of the NNRA SEI and the identifier of the characteristic SEI is linear. This can be done using a model-based filter (Wiener Filter). Linear filters are ALF Specifically, the filtering target image DecFrame may be filtered by the filtering unit 610. The image is divided into small regions (x=xSb..xSb+bSW-1, y=xSb..xSb+bSH-1) of a given size (e.g. 4x4, 1x1). (xSb, ySb) are the top left coordinates of the small region, and bSW, bSH are the width and height of the small region. Then, filter processing is performed on a small region basis. The refinement method is extracted from DecFrame using the selected filter coefficient coeff[]. The post-processing image outFrame is derived.

[0113] outframe[cIdx][y][x] = Σ(coeff[i] * DecFrame[cIdx][y+ofy][x+ofx] + offset) >> shift Here, ofx and ofy are the offsets of the reference position determined according to the filter position i. offset=1<<(shift-1). shift corresponds to the precision of the filter coefficients and is a constant such as 6, 7, 8, etc.

[0114] In addition, the image is classified into small regions, and the classId (=classId[y][x]) of the derived small region is used. The filter coefficient coeff[classId][] may be selected according to the need to perform filtering.

[0115] outframe[cIdx][y][x] = (Σ(coeff[classId][i] * DecFrame[cIdx][y+ofy][x+ofx] + offset) >> shift Here, DecFrame[ch] = inputTensor[ch][yP+overlapSize][xP+overlapSize], that is, the following is also acceptable.

[0116] ch=0 if (OccupancyUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(oFrame[yP][xP]) ch++ } if (GeometryUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(gFrame[yP][xP]) ch++ } if (AttributeUsedFlag) { DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[0][yP][xP]); ch++ DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[1][yP][xP]); ch++ DecFrame[ch][yP+overlapSize][xP+overlapSize] = Out(aFrame[2][yP][xP]); ch++ } Note that classId may be derived using the activity or directionality of a block (pixel in the case of 1x1). For example, classId may be derived using activity Act derived from the sum of absolute difference as shown below.

[0117] classId=Act Act may be derived as follows:

[0118] Act = Σ|x-xi| Here, i=0..7 and xi indicates the adjacent pixels of the target pixel x. The direction of the adjacent pixels is up or down with respect to i=0..7. It is also possible to add four directions at 45 degrees to the left and right, for a total of eight directions. Act = Σ|-xl+2*x-xr|+Σ|-ya+2*x-yb| may be used. Note that l, r, a, and b are the opposites of left, right, above, and below, respectively, and indicate the pixels to the left, right, top, and bottom of x. Act=Min(Num-1, Act>>shift) It is also possible to quantize the signal by the shift value as shown above, and then clip the resulting signal to NumA-1. Furthermore, classId may be derived using the directionality D using the following formula.

[0119] classId=Act+D*NumA Here, for example, Act=0..NumA-1, D=0..4 etc.

[0120] The ALF unit 610 outputs the OutFrame as it is as a FilteredFrame.

[0121] (Decryption of applicable SEI and application of filter) 9 is a diagram showing a flowchart of the process of the 3D data decoding device. performs the following processing, including decoding the applied SEI message.

[0122] S6001: The atlas decoding unit 302 decodes nnra_cancel_flag and nnra_target_id from the applicable SEI. nnra_cancel_flag may be nnrc_cancel_flag.

[0123] S6002: If nnra_cancel_flag is 1, end the processing for the frame that is the target of nnra_cancel_flag. If nnra_cancel_flag is 0, proceed to S6003.

[0124] S6003: The atlas decoding unit 302 decodes the nnra_persistence_flag from the applicable SEI.

[0125] S6005: Identify a characteristic SEI having the same nnrc_id as the nnra_target_id, and derive parameters of the NN model from the characteristic SEI.

[0126] S6006: The NN filter unit 611 and the ALF unit 610 execute refinement (filtering) processing using the parameters of the derived NN model.

[0127] <Syntax configuration example> (Neural network refinement characteristics SEI) Figure 11 and Figure 12 show the syntax of nn_refinement_characteristics(payloadSize) (characteristic SEI). The argument payloadSize indicates the number of bytes in this SEI message.

[0128] The persistence scope of a property SEI is the CAS, i.e., the SEI is applied on a CAS basis.

[0129] The syntax element nnrc_id indicates the ID of the characteristic SEI. The value of nnrc_id of the characteristic SEI indicating the network characteristic is transmitted as the target ID (nnra_target_id). and specifies the refinement operation to be applied. If nnrc_mode_idc is 0, then this SEI message contains an ISO / IEC 15938-17 bit stream and specifies a base NNRE (neural-network refinement) or indicates that it is an update from a base NNRE with the same nnrc_id value. If the NNRC SEI message is the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, then if nnrc_mode_idc is equal to 1, then the base NNRE for that nnrc_id value shall be read by the URI indicated by nnrc_uri, in the format identified by nnrc_tag_uri, where nnrc_tag_uri is the tag URI. The neural network identified by the If the NNRC SEI message is neither the first NNRC SEI message in decoding order with a particular nnrc_id value in the current CAS, nor a repetition of the first NNRC SEI message in decoding order, and nnrc_mode_idc is equal to 1, this indicates that updates to a base NNRE with the same nnrc_id value are defined by the URI indicated by nnrc_uri, which is identified by the tag URI nnrc_tag_uri. It is a format. The value of nnrc_mode_idc shall be in the range of 0 to 1 (inclusive) in a bitstream conforming to this document. The objective must be This SEI message is the first NNRC SEI in decoding order with a particular nnrc_id value within the current CAS. If it is a message, the RefineFilter() is assigned the same as the base NNRE. If this SEI message is neither the first NNRCSEI message in decoding order with a particular nnrc_id value in the current CAS, nor a repeat of the first NNRCSEI message in decoding order, apply the updates defined in this SEI message to the base NNRE to obtain the RefineFilter(). Updates are not cumulative, each update applies to the base NNRE. The base NNRE is The NNRE specified in the first NNRC SEI message (in decoding order) with a particular nnrc_id value.

[0130] nnrc_reserved_zero_bit_a shall be equal to 0 in a bitstream conforming to this document. Decoders MUST ignore NNRCSEI messages where nnrc_reserved_zero_bit_a is not 0.

[0131] nnrc_tag_uri specifies the format and associated information for a neural network to be used as a base NNRE or an update to a base NNRE with the same nnrc_id value specified in nnrc_uri. Contains a tag URI, with syntax and semantics specified in IETF RFC4151, that identifies the The tag is identified by the nnrc_uri, which is equal to "tag:iso.org,2023:15938-17". This indicates that the neural network data used complies with ISO / IEC 15938-17.

[0132] The nnrc_uri is a string defined in IETF Internet Standard 66 that identifies a neural network to be used as a base NNRE or an update to a base NNRE with the same nnrc_id value. Contains a URI, which has syntax and semantics.

[0133] If nnrc_property_present_flag is equal to 1, the input format, output format If nnrc_property_present_flag is 0, the syntax elements related to input format, output format, and complexity are not present. Indicates that the statement element is not present. This SEI message is the first NNRC SEI in decoding order with a particular nnrc_id value within the current CAS. When a message, nnrc_property_present_flag shall be equal to 1. When nnrc_property_present_flag is equal to 0, all syntax element values ​​that may only be present when nnrc_property_present_flag is equal to 1, for which no inferred value is specified, shall be set to the inferred value for each of the NNRC SEI messages that contain the base NNRE that this SEI updates. It is inferred that each is the same as the corresponding syntax element.

[0134] nnrc_base_flag equal to 1 indicates that the SEI message specifies a base NNRE. The equal nnrc_base_flag indicates that the SEI message specifies an update relative to the base NNRE. If not present, the value of nnrc_base_flag is inferred to be equal to 0. The following constraints apply to the value of nnrc_base_flag: - If the NNRC SEI message is the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS, the value of nnrc_base_flag shall be equal to 1. - If the NNRC SEI message nnrcB is not the first NNRC SEI message in decoding order with a particular nnrc_id value within the current CAS and the value nnrc_base_flag is equal to 1, then the NNRC SEI message , in decoding order, is a repetition of the first NNRC SEI message nnrcA with the same nnrc_id, i.e. the payload content of the next nnrcB must be the same as that of nnrcA. The NNRC SEI message is a repeat of the first NNRC SEI message with a particular nnrc_id, rather than the first NNRC SEI message in decoding order with that particular nnrc_id value within the current CAS. , the following applies: - This SEI message is an update to the preceding base NNRE in decoding order with the same nnrc_id value. Define new.

[0135] - This SEI message continues until the end of the current CAS or until the end of the next NNRC SEI message in decoding order that follows the current decoded picture in output order within the current CAS and has the specified nnrc_id value within the current CAS. The decoded picture relates to the current decoded picture and all subsequent decoded pictures in output order up to, but not including, the decoded picture associated with that layer.

[0136] The syntax elements nnrc_occupancy_flag, nnrc_geometry_flag, and nnrc_attribute_flag indicate the target of the refinement process (refinement target information). They indicate whether or not the input or output of the refinement includes occupancy, geometry, or attributes, respectively.

[0137] The syntax element nnrc_attribute_index is information to be refined, and indicates the index (attrIdx) of the attribute to be refined. When nnrc_attribute_flag !=0, nnrc_attribute_index is encoded and decoded. Although not shown, the syntax element nnra_all_attribute_flag may be decoded and encoded. Here, nnra_all_attribute_flag=1 indicates that all This indicates that all attributes are to be refined. Indicates that all attrIdx of all 0.. ai_attribute_count[RecAtlasID]-1 are included. When nnra_all_attribute_flag = 1, nnrc_attribute_index and nnrc_attribute_num_minus1 are not encoded or decoded, and inferred as nnrc_attribute_index = 0, nnrc_attribute_num_minus1 = ai_attribute_count[RecAtlasID]-1. Also, the list of nnrc_attribute_index may be inferred as 0.. ai_attribute_count[RecAtlasID]-1. When nnra_all_attribute_flag = 0, nnrc_attribute_index and nnrc_attribute_num_minus1 are encoded and decoded.

[0138] The syntax element nnrc_attribute_num_minus1 is the refinement target information. Indicates the number of attributes to be imported. If nnrc_attribute_flag != 0, nnrc_attribute_num_minus1 is encoded and decoded.

[0139] The syntax element nnrc_strength_num is additional information for the refinement input. Indicates the number of strength information nnra_strength input to processing (inputTensor), strengthNum.

[0140] The syntax element nnrc_inp_out_format_idc (input / output tensor format information) specifies the format of the decoded image This shows how to convert pixel values ​​to input and output values ​​for the refinement process. If the value of nnrc_inp_out_format_idc is 0, the input values ​​to the refinement process (especially the input tensor) are real numbers (floating-point value format) specified by IEEE754, and the function Inp is specified as follows. The range of values ​​of the input tensor is 0..1.

[0141] Inp(x) = x÷((1< <BitDepth)-1) InpS(x) = x ÷ 63 If the value of nnrc_inp_out_format_idc is 1, the input and output values ​​to the refinement process are unsigned integers, and the function Inp is specified as follows: The range of input tensor values ​​is 0..1<<(inpTensorBitDepth)-1.

[0142] shift=BitDepth-inpTensorBitDepth if(inpTensorBitDepth>=BitDepth) Inp(x)=x<<(inpTensorBitDepth-BitDepth) else Inp(x) = Clip3(0,(1< <inpTensorBitDepth)-1,(x+(1<<(shift-1)))> >shift) InpS(x) = x The value of the syntax element nnrc_inp_tensor_bitdepth_minus8 plus 8 indicates the pixel bit depth of the luminance pixel values ​​of the input integer tensor. The tensor depth is derived from the tensor element nnrc_inp_tensor_bitdepth_minus8 as follows:

[0143] inpTensorBitDepth=nnrc_inp_tensor_bitdepth_minus8+8 A block is an image array. The refinement process is performed in fixed block units. A block may also be called a patch.

[0144] The syntax element nnrc_block_size_idc indicates the block size. The block size is , 64, 128, 192, or any other multiple of 64.

[0145] blockWidth = 64*nnrc_block_size_idc (=nnrc_block_size_idc<<6) blockHeight = 64*nnrc_block_size_idc (=nnrc_block_size_idc<<6) It can also be less than nnrc_block_size_idc excluding 0.

[0146] blockWidth = (nnrc_block_size_idc+1)<<6 blockHeight = (nnrc_block_size_idc+1)<<6 The syntax element nnrc_overlap_size_idc specifies the size of the overlap between adjacent input tensors. Specifies the number of horizontal and vertical pixels to overlap. The value of nnrc_overlap_size_idc The size of the loop may be a multiple of four.

[0147] overlapSize = nnrc_overlap_size_idc << 2 A function Out that converts the luminance pixel value and the chrominance pixel value output by the post-processing into integer values ​​of the pixel bit length is specified as follows using the pixel bit length BitDepth:

[0148] Out(x) = Clip3(0,(1< <BitDepth)-1,Round(x*((1<<BitDepth)-1))) The function Out is specified as follows:

[0149] shift=outTensorBitDepth-BitDepth if(outTensorBitDepth>=BitDepth) Out(x) = Clip3(0,(1< <BitDepth)-1,(x+(1<<(shift-1)))> >shift) else Out(x)=x<<(BitDepth-outTensorBitDepth) nnrc_out_tensor_bitdepth_minus8+8 specifies the pixel bit depth of the pixel values ​​in the output integer tensor. The value of outTensorBitDepth is derived from the syntax element nnrc_out_tensor_bitdepth_minus8 as follows:

[0150] outTensorBitDepth=nnrc_out_tensor_bitdepth_minus8+8 The syntax element nnrc_reserved_zero_bit_b shall be equal to 0.

[0151] The syntax element nnrc_payload_byte[i] contains a bitstream that conforms to ISO / IEC15938-17. Contains the i-th byte of the stream. nnrc_payload_byte[i] must all be fully ISO / IEC15938-17 compliant bitstreams.

[0152] (Neural network refinement applied SEI) FIG. 13 shows an example syntax configuration of an application SEI.

[0153] For this applied SEI, the atlas decoding unit 302 (refinement information decoding unit) and the atlas encoding unit 102 (refinement information encoding unit) decode and encode the following syntax elements.

[0154] nnra_target_id is the ID of the characteristic SEI to be applied (the identifier of the refined characteristic information, identification information) Apply the refinement specified by the feature SEI with nnrc_id equal to nnra_target_id to the image.

[0155] nnra_cancel_flag is the cancel flag. nnra_cancel_flag is set to 1 if the Cancel the maintenance of refinements set for an image in the decoded NNRA SEI. When nnra_cancel_flag is 0, it indicates that the following syntax element (nnra_persistence_flag) is transmitted, encoded, and decoded.

[0156] nnra_persistence_flag indicates persistence information of the target refinement. When nnra_persistence_flag is 0, it indicates that the target refinement applies only to the picture indicated by atlasID. When nnra_persistence_flag is 1, it indicates that the target refinement indicated by nnra_target_id applies to the current picture and all subsequent pictures until one of the following conditions is met: A new CAS is started. The bitstream ends. NNRA SEI messages with nnra_cancel_flag=1 are output following the current picture in output order, i.e., are cancelled by nnra_cancel_flag=1. The following is also acceptable. -Has the same nnra_target_id as the current SEI message and has nnra_cancel_flag=1 The NNRA SEI message that corresponds to the current picture is output following the current picture in output order, i.e., is canceled by nnra_cancel_flag=1.

[0157] As shown in Fig. 13, the atlas decoding unit 302 and the atlas encoding unit 102 decode and encode a syntax element indicating the second refinement target information. The second refinement target information includes nnra_target_map_index indicating the mapIdx to which the refinement is applied. However, the second refinement target information may further include nnra_target_attribute_index indicating the index attrIdx of the attribute to which refinement is applied, nnra_attribute_num indicating the number of attributes to which refinement is applied, nnra_attribute_partition_index indicating the partIdx to which refinement is applied, and nnra_auxiliary_video_flag indicating whether the target to which refinement is applied is auxiliary data. Note that nnra_target_attribute_index and nnra_attribute_num have already been set. This is the same as nnrc_target_attribute_index and nnrc_attribute_num described above.

[0158] In the syntax structure of FIG. 13, multiple refinement processes are specified and transmitted in one application SEI. Here, for the number of refinements N (>=1), the value of N (actually the value of N-1) is transmitted, and N refinements are transmitted. Specifies the refinement process. nnra_count_minus1 indicates the number of instances of the refinement process. The syntax elements include: nnra_cancel_flag[i], nnra_persistent_flag[i], and nnra_target_id[i], which indicate the ID of the characteristic SEI indicating NNRE, for each i from i=0 to i=nnra_count_minus1+1. nnra_strength_num[i] may also be included. nnra_strength_num[i] is a parameter indicating the number of strength information to be used. nnra_streangth_num[i] is the combination of nnrc_strength_num specified in the NNRC SEI message identified by nnra_target_id[i] and Must be equal.

[0159] In addition, for each i, nnra_target_map_index[i] is set as the second refinement target information. include.

[0160] Furthermore, for each i, nnra_all_map_flag[i] is indicated as the second refinement target information. Syntax elements may be decoded and encoded. Here, nnra_all_map_flag[i]=1 indicates that refinement is to be applied to all maps. In other words, all mapIdx to be refined is Indicates that all mapIdx is included. If nnra_all_map_flag[i]=1, nnra_target_map_index[i] is not encoded or decoded, and inferred as nnra_target_map_index[i]=1, i=0..asps_map_count_minus1. If nnra_all_map_flag[i]=0, nnra_target_map_index[i] is encoded and decoded. Note that nnra_all_map_flag[i] and nnra_target_map_index[i] may be coded and decoded in the characteristic SEI instead of the activation SEI. In that case, the characteristic SEI As tax elements, nnrc_all_map_flag and nnrc_target_map_index are encoded and decoded.

[0161] Furthermore, although not shown, for each i, nnra_all_partition_flag[i] may be included as second refinement target information. Here, nnra_all_partition_flag[i]=1 means that all attributes are refined. In other words, it indicates that all attrIdx are included in the attrIdx to which refinement is applied. When nnra_all_partiotion_flag[i]=1, nnra_target_partiotion_index[i] is not encoded or decoded, and inferred as nnra_target_partiotion_index[i]=1, i=0..asps_map_count_minus1. When nnra_all_partiotion_flag[i]=0, nnra_target_partiotion_index[i] is encoded and decoded.

[0162] If nnra_cancel_flag[i] indicates no cancellation, for example, if nnra_cancel_flag[i]==0 (!nnra_cancel_flag[i]), the atlas decoding unit 302 decodes the persistence information nnra_persistent_flag[i] and further decodes the number of strength information to be applied, nnra_strength_num[i]. Then, decode nnra_strength[i][j], which indicates the strength information to be applied.

[0163] The refinement unit 306 sets the input data as mapIdx = nnra_target_map_index[i] and attrIdx = nnra_target_attribute_index for each i from i = 0.. nnra_count_minus1+1. , oFrame = OccFramesNF[compTimeIdx][0], gFrame = GeoFramesNF[mapIdx][compTimeIdx][frameIdx][0], aFrame = AttrFramesNF[attrIdx][mapIdx][compTimeIdx]. Then, using the neural network model of the characteristic SEI (NNRC SEI having the same value as nnrc_id[i] as nnra_target_id[i]) specified and activated by the activation information, The following process is performed.

[0164] If the refinement target information indicates an attribute, add the attribute frame of the map indicated by the second refinement target information nnra_target_map_index[i] to the input tensor. For example, set mapIdx=nnra_target_map_index[i] in (Eq. IN-ATTR-DIM) and (Eq. IN-ATTR-POS). You may do so.

[0165] If the refinement target information indicates occupancy, a specific component of the output tensor may be set to the occupancy, for example (Eq. OUT-OCC).

[0166] If the refinement target information indicates a geometry, a specific component of the output tensor is set to the geometry frame having the mapIdx indicated by the second refinement target information nnra_target_map_index[i]. For example, mapIdx=nnra_target_map_index[i] may be set in (Eq. OUT-GEO).

[0167] If the refinement target information indicates an attribute, a specific component of the output tensor is set to the attribute indicated by the second refinement target information nnra_target_map_index. For example, (Eq. You may set mapIdx = nnra_target_map_index[i] in (Eq. OUT-ATTR), (Eq. OUT-ATTR-DIM), and (Eq. OUT-ATTR-POS).

[0168] When an attribute is included as the refinement target information, the refinement target information may include a syntax element nnrc_target_attribute_index[i] indicating the attrId of the refinement target.

[0169] If the refinement target information indicates an attribute, the attribute indicated by the second refinement target information nnrc_target_attribute_index and nnra_target_map_index is added to the input tensor. For example, in (Eq. IN-ATTR-POS) and (Eq. IN-ATTR-POS2), mapIdx = nnra_target_map_index[i] (or nnrc_target_map_index), attrIdx = nnrc_target_attribute_index (also It may be set to nnra_target_attribute_index[i] or nnra_target_attribute_index[i].

[0170] If the refinement target information indicates an attribute, the specific component of the output tensor is assigned to the attribute indicated by the second refinement target information nnra_target_map_index[i] and nnra_target_attribute_index[i]. Set to the tribute. For example, you may set mapIdx = nnra_target_map_index[i] (or nnrc_target_map_index, attrIdx = nnrc_target_attribute_index (or nnra_target_attribute_index[i]) in (Eq. OUT-ATTR), (Eq. OUT-ATTR-POS), (Eq. OUT-ATTR-POS2). It may also include nnra_attribute_partition_index and nnra_auxiliary_video_flag.

[0171] In the above, the atlas decoding unit 302 decodes the syntax element nnra_target_map_index from the applicable SEI to derive the mapIdx to be refined, decodes the syntax element nnra_target_attribute_index[i] from the applicable SEI to derive the attrIdx of the attribute to be refined, Derive the attribute specified by attrIdx and refine it. Therefore, , the advantage is that it is possible to apply optimized refinements to specific mapIdx and attrIdx. In addition, refinements specified in the same characteristic SEI can be applied to attributes indicated by the same value of attrIdx, and refinements specified in different characteristic SEIs can be applied simultaneously to attributes indicated by different attrIdx.

[0172] To summarize the above, the 3D data decoding device has a geometry decoding unit that decodes a geometric frame from encoded data and an attribute decoding unit that decodes an attribute frame, and is characterized in that it is equipped with a refinement information decoding unit that decodes refinement characteristic information and refinement activation information from the encoded data, and is equipped with a refinement processing unit that decodes a syntax element indicating the number of refinements from the activation information, decodes an index indicating the characteristic information of the decoded number of refinements, and performs a refinement process on the attribute frame or geometric frame according to the characteristic information.

[0173] The refinement information decoding unit further restores the number strengthNum of strength information from the characteristic information. The refining unit decodes the strengthNum pieces of strength information, and uses the neural network Input at least one of the occupancy frame, geometry frame, or attribute frame to the input tensor of the work, and then assign each of the strengthNum values ​​to the input tensor of the neural network. The feature of this method is that the input is input to each of the strengthNum channels of the sensor.

[0174] The refinement information decoding unit is characterized in that it decodes a flag indicating whether or not refinement processing is to be performed on all maps, and when the flag indicates that refinement processing is not to be performed on all maps, it decodes an index indicating which map or maps are to be subjected to refinement processing.

[0175] The refinement information decoding unit decodes the index of the attribute frame to be refined and the number of attribute frames from the encoded data of the characteristic information or the activation information, and performs refinement processing by inputting attribute frames, equal to the number of decoded attribute frames, into the input tensor of the neural network, starting from the index of the decoded attribute frame.

[0176] The refinement information decoding unit decodes refinement characteristic information and information on the map to be refined from the encoded data of the activation information, decodes a cancel flag indicating whether or not to cancel the refinement process related to the refinement characteristic information and the map to be refined, and if the cancel flag indicates not to cancel, further decodes persistence information.

[0177] A 3D data encoding device having a geometry encoding unit for encoding a geometric frame and an attribute encoding unit for encoding an attribute frame, further comprising a refinement information encoding unit for encoding refinement characteristic information and refinement activation information from the encoded data, encoding a syntax element indicating the number of refinements from the activation information, and encoding an index indicating the characteristic information of the number of refinements encoded. and further comprising a refinement processing unit which performs a refinement process on the attribute frame or the geometry frame in accordance with the characteristic information.

[0178] The refinement information encoding unit further obtains the number of pieces of strength information, strengthNum, from the characteristic information. The refinement processing unit encodes the strength information of the strengthNum, and the refinement processing unit At least one of the occupancy frame, geometry frame, or attribute frame is input to the input tensor of the network, and further, strengthNum number of frames derived from the encoded intensity information are input. Each value is input to each of the strengthNum channels of the neural network's input tensor. It is characterized by the above.

[0179] The refinement information encoding unit is characterized in that it encodes a flag indicating whether or not refinement processing is to be performed on all maps, and when the flag indicates that refinement processing is not to be performed on all maps, it encodes an index indicating which map or maps are to be subjected to refinement processing.

[0180] The refinement information encoding unit is characterized in that it encodes the index of the attribute frame to be refined and the number of attribute frames from the encoded data of the characteristic information or the activation information, and performs refinement processing by inputting attribute frames to the input tensor of the neural network as many times as the number of encoded attribute frames, starting from the index of the encoded attribute frame.

[0181] The refinement information encoding unit encodes refinement characteristic information and information on the map to be refined from the encoded data of the activation information, encodes a cancel flag indicating whether or not to cancel the refinement process related to the refinement characteristic information and the map to be refined, and further encodes persistence information if the cancel flag indicates not to cancel.

[0182] (Configuration of 3D data encoding device) FIG. 14 is a functional block diagram showing a schematic configuration of a 3D data encoding device 11 according to an embodiment of the present invention.

[0183] The 3D data encoding device 11 includes a patch generating unit 101, an atlas encoding unit 102, an occupancy generating unit 103, an occupancy encoding unit 104, a geometry generating unit 105, a geometry encoding unit 106, an attribute generating unit 108, an attribute encoding unit 109, a refinement parameter derivation unit 110, The 3D data encoding device 11 is composed of a point cloud as 3D data. It inputs a code or mesh and outputs encoded data.

[0184] The patch generation unit 101 receives 3D data and generates a set of patches (rectangular images in this case). Specifically, the 3D data is divided into a plurality of regions, and each region is projected onto one of the planes of a 3D bounding box set in the 3D space to generate a plurality of patches. The patch generation unit 101 receives information about the 3D bounding box (coordinates, size, etc.) Information regarding mapping onto the projection surface (such as the projection surface, coordinates, size, and whether or not each patch is rotated) is output as atlas information.

[0185] The atlas encoding unit 102 encodes the atlas information output from the patch generating unit 101 and outputs atlas data.

[0186] The occupancy generation unit 103 receives a set of patches output from the patch generation unit 101, and converts the valid area of ​​each patch (the area where 3D data exists) into a 2D binary image (for example, the valid area is 1, Generate an occupancy with invalid areas set to 0. Note that the valid and invalid areas have a value of 255. and 0 may be used.

[0187] The occupancy encoding unit 104 receives the occupancy output from the occupancy generating unit 103 and outputs occupancy and occupancy data. etc. are used.

[0188] The geometry generation unit 105 generates 3D data, occupancy, occupancy data, and atlas. Based on the depth information, a geometric frame is generated that stores the depth value of each patch with respect to the projection plane. The geometric frame generation unit 105 selects the depth value of the point projected onto the pixel g(x, y) with respect to the projection plane. The point with the smallest depth is derived as p_min(x,y,z). The point with the maximum depth among the points projected onto pixel g(x,y) and located at a predetermined distance d from p_min(x,y,z) is derived as p_max(x,y,z). The geometric frame obtained by projecting p_min(x,y,z) onto all pixels on the projection plane is set as the geometric frame of the Near layer. The geometric frame obtained by projecting p_max(x,y,z) onto all pixels on the projection plane is set as the geometric frame of the Far layer.

[0189] The geometry encoding unit 106 receives a geometry frame, and encodes the geometry frame and the geometry data. The encoding method used is VVC, HEVC, etc.

[0190] The attribute generation unit 108 generates 3D data, occupancy, geometric frame and atlas information. Based on the information, an attribute frame is generated that stores color information (e.g., YUV values, RGB values, etc.) for the projection surface of each patch. The attribute generation unit 108 calculates the value of the attribute corresponding to the point p_min(x, y, z) with the minimum depth calculated by the geometry generation unit 105, and sets the attribute frame into which the value is projected as the attribute frame of the Near layer. Similarly, an attribute frame calculated for p_max(x, y, z) is set as the attribute frame of the Far layer.

[0191] The attribute encoding unit 109 receives the attribute frame and encodes the attribute and It outputs view data using VVC, HEVC, etc. as the encoding method.

[0192] The refinement parameter derivation unit 110 extracts an attribute frame and an original attribute frame. Alternatively, the refinement parameter derivation unit 110 inputs the geometric frame and the original geometric frame, selects or derives optimal filter parameters in the NN filter processing, and outputs the selected filter parameters. Set it up.

[0193] The multiplexing unit 111 inputs the filter parameters output from the refinement parameter derivation unit 110 and outputs them in a predetermined format. The predetermined format is, for example, SEI, which is auxiliary extension information of video data, ASPS and AFPS, which are designation information of data structure in the V3C standard, and ISOBMFF, which is a media file format standard. The multiplexing unit 111 also multiplexes the atlas data, occupancy data, geometry data, attribute data, and the above filter parameters, and outputs the multiplexed data as coded data. As a multiplexing method, a byte stream format, ISOBMFF, etc. are used.

[0194] FIG. 15 shows a 3D data encoding device 11, a 3D data decoding device 31, and an SEI message according to an embodiment of the present invention. FIG. 13 is a block diagram showing the relationship of messages.

[0195] The 3D data encoding device 11 is composed of a video encoding unit and an SEI encoding unit. In the configuration of FIG. 14, the SEI coding unit corresponds to the occupancy coding unit 104, the geometry coding unit 106, and the attribute coding unit 109. In the configuration of FIG. 14, the SEI coding unit corresponds to the atlas coding unit 102. do.

[0196] The 3D data decoding device 31 is composed of a video decoding unit, an SEI decoding unit, a switch and a refinement unit. In the configuration of FIG. 5, the video decoding unit includes an occupancy decoding unit 303, a geometry decoding unit 304, and an The SEI decoding unit corresponds to the atlas decoding unit 302 in the configuration of Fig. 5. The refinement unit corresponds to the refinement unit 306 in the configuration of Fig. 5. The refinement unit 306 is a NN It may include either a filter unit 611 or an ALF unit 610, or both.

[0197] The occupancy, geometry, and attribute frames generated from the 3D data are encoded by the video encoding unit. The encoded data is decoded by the video decoding unit, and the decoded image is reconstructed. The encoding unit generates a characteristic SEI and an application SEI from the 3D data. The SEI decoding unit decodes these SEI messages. The application SEI is input to the switch to specify the image to be refined. Only the image to be refined is input to the refinement section. The refinement to be applied to the input decoded image is specified. The refined image or the decoded image is displayed on the 3D data display unit 41 (FIG. 1).

[0198] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0199] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. In other words, the technical scope of the present invention also includes embodiments obtained by combining technical means that are appropriately modified within the scope of the claims. [Industrial Applicability]

[0200] The embodiments of the present invention can be suitably applied to a 3D data decoding device that decodes encoded data in which 3D data is encoded, and a 3D data encoding device that generates encoded data in which 3D data is encoded. Also, the embodiments of the present invention can be suitably applied to the data structure of encoded data generated by the 3D data encoding device and referenced by the 3D data decoding device. [Explanation of symbols]

[0201] 11 3D data encoding device 101 Patch Generation Unit 102 Atlas Encoding Unit 103 Occupancy Generation Unit 104 Occupancy coding unit 105 Geometry Generation Unit 106 Geometry Encoding Unit 108 Attribute Generation Unit 109 Attribute Encoding Unit 110 Refine Parameter Derivation Part 111 Multiplexer 21 Network 31 3D data decoding device 301 Header Decoding Unit 302 Atlas Information Decoding Unit 303 Occupancy Decoding Unit 304 Geometry Decoding Unit 306 Geometry Reconstruction Unit 307 Attribute Decoding Unit 308 Refining Department 309 V3C Decoding Unit 310 Pre-reconstruction unit 311 Most constructed part 41 3D data display device

Claims

1. A 3D data decoding device including a geometry decoding unit that decodes a geometry frame from encoded data and an attribute decoding unit that decodes an attribute frame, a refinement information decoding unit for decoding refinement characteristic information and refinement activation information from the encoded data, Decoding a syntax element indicating a number of refinements from the activation information, and decoding an index indicating the characteristic information of the decoded number of refinements; A 3D data decoding device comprising a refinement processing unit that performs refinement processing of the attribute frame or the geometry frame in accordance with the characteristic information.

2. The refinement information decoding unit further restores the number strengthNum of strength information from the characteristic information. The refining unit decodes the strengthNum pieces of strength information, and uses the neural network Input at least one of the occupancy frame, geometry frame, or attribute frame to the input tensor of the work, and then assign each of the strengthNum values ​​to the input tensor of the neural network.

2. The 3D display according to claim 1, wherein the strengthNum channels of the sensor are input. Data decoding device.

3. The 3D data decoding device according to claim 1, characterized in that the refinement information decoding unit decodes a flag indicating whether or not to perform refinement processing on all maps, and when the flag indicates that refinement processing is not to be performed on all maps, decodes an index indicating which map or maps are to be subjected to refinement processing.

4. The 3D data decoding device according to claim 1, characterized in that the refinement information decoding unit decodes an index of an attribute frame to be refined and a number of attribute frames from the encoded data of the characteristic information or the activation information, and performs refinement processing by inputting attribute frames to an input tensor of the neural network by the number of decoded attribute frames, starting from the index of the decoded attribute frame.

5. The 3D data decoding device described in claim 1, characterized in that the refinement information decoding unit decodes refinement characteristic information and information on the map to be refined from the encoded data of the activation information, decodes a cancel flag indicating whether to cancel the refinement process related to the refinement characteristic information and the map to be refined, and further decodes persistence information if the cancel flag indicates not to cancel.

6. A 3D data encoding device that includes a geometry encoding unit that encodes a geometric frame and an attribute encoding unit that encodes an attribute frame and generates encoded data, a refinement information encoding unit for encoding refinement characteristic information and refinement activation information to generate the encoded data; from said activation information, encoding a syntax element indicating a number of refinements, and encoding an index indicating said characteristic information of the encoded number of refinements; A 3D data encoding device comprising a refinement processing unit that performs refinement processing of the attribute frame or the geometry frame in accordance with the characteristic information.

7. The refinement information encoding unit further obtains the number of pieces of strength information, strengthNum, from the characteristic information. The strengthNum pieces of strength information are encoded, and the refinement processing unit performs a neural At least one of the occupancy frame, geometry frame, or attribute frame is input to the input tensor of the network, and furthermore, strengthNum number of frames derived from the encoded intensity information are input. Each value is input to each of the strengthNum channels of the neural network's input tensor.

7. The 3D data encoding device according to claim 6,

8. The 3D data encoding device according to claim 6, characterized in that the refinement information encoding unit encodes a flag indicating whether or not to perform refinement processing on all maps, and when the flag indicates that refinement processing is not to be performed on all maps, encodes an index indicating which map or maps are to be subjected to refinement processing.

9. The 3D data encoding device according to claim 6, characterized in that the refinement information encoding unit encodes an index of an attribute frame to be refined and the number of attribute frames from the encoded data of the characteristic information or the activation information, and performs refinement processing by inputting attribute frames to an input tensor of the neural network by the number of encoded attribute frames, starting from the index of the encoded attribute frame.

10. The 3D data encoding device described in claim 6, characterized in that the refinement information encoding unit encodes refinement characteristic information and information on the map to be refined from the encoded data of the activation information, encodes a cancellation flag indicating whether to cancel the refinement process related to the refinement characteristic information and the map to be refined, and further encodes persistence information if the cancellation flag indicates not to cancel.

Images