3D data encoding device and 3D data decoding device

Abstract

It is not possible to appropriately refine three-dimensional data using a plurality of images. A 3D data decoding device for decoding encoded data and decoding 3D data including attribute information, the 3D data decoding device being provided with: a header decoding unit for decoding refined characteristic information and refined activation information; and a refinement unit that performs a refinement process of the attribute frame or the geometry frame, decodes, from the encoded data of the characteristic information, information to be refined indicating which of the occupancy map, geometry, and attributes is used, and performs refinement using an image specified in accordance with the information to be refined.

Description

Technical Field

[0001] Embodiments of the present invention relate to a 3D data encoding device and a 3D data decoding device. Background Art

[0002] In order to efficiently transmit or record 3D data, there is a 3D data encoding device that converts 3D data into a two-dimensional image and encodes it using a motion image encoding method to generate encoded data; and a 3D data decoding device that decodes the two-dimensional image from the encoded data and reconstructs it to generate 3D data. In addition, there is a technology that uses auxiliary enhancement information of deep learning post-filtering to filter the two-dimensional image.

[0003] As specific 3D data encoding methods, for example, MPEG-I's ISO / IEC23090-5 V3C (Visual Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression) can be listed. V3C can be used for encoding and decoding of point clouds composed of point positions and attribute information. In addition, it can also be used for encoding and decoding of multi-viewpoint images and mesh images through ISO / IEC 23090-12 (MPEG Immersive Video (MPEG Immersive Video: MIV)) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding (V-DMC)) which is being standardized. In addition, in the auxiliary enhancement information of the neural network post-filtering in non-patent document 1, the model information of the neural network is transmitted through the characteristic SEI, and the frame specified by the application SEI is paged, thereby enabling appropriate filtering processing of the point group data. Prior art literature Non-patent literature Non-patent document 1

[0004] K. Takada, Y. Tokumo, T. Chujoh. T. Ikai, "[V-PCC][EE2.8]V3Cneural-networkpost-filter SEI messages," ISO / IEC JTC 1 / SC 29 / WG7, m61805, Jan.2023 Summary of the invention Problem that the invention aims to solve

[0005] In Non-Patent Document 1, filtering is performed only on attributes, so filtering using the relationship between occupancy maps, geometry, and attributes cannot be performed. In other words, in the method of decoding 3D data using multiple images / videos, refinement using information between images cannot be performed.

[0006] An object of the present invention is to solve the above-mentioned problem in encoding / decoding of 3D data using a moving picture encoding / decoding method, further reduce encoding distortion using refined auxiliary information, and encode / decode 3D data with high quality. Technical Solution

[0007] In order to solve the above-mentioned problem, a 3D data decoding device of one embodiment of the present invention decodes encoded data and decodes 3D data including geometry and attributes. The 3D data decoding device is characterized in that it comprises: a refinement information decoding unit, which decodes refined characteristic information and refined activation information from the above-mentioned encoded data; a geometry decoding unit, which decodes a geometry frame from the above-mentioned encoded data; an attribute decoding unit, which decodes an attribute frame from the above-mentioned encoded data; and a refinement unit, which performs refinement processing on the above-mentioned attribute frame or geometry frame, and the above-mentioned refinement information decoding unit decodes refinement object information indicating which one of the occupancy map, geometry, and attribute is used from the encoded data of the above-mentioned characteristic information, and uses an image specified according to the refinement object information for refinement. Beneficial Effects

[0008] According to one aspect of the present invention, distortion caused by encoding of color images can be reduced, and 3D data can be encoded / decoded with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 This is a schematic diagram showing the configuration of a 3D data transmission system according to this embodiment. Figure 2 This is a diagram showing the hierarchical structure of the data of the coded stream. Figure 3 It is a graph used to illustrate 3D data, occupancy map, geometry frame, and attribute frame. Figure 4 It is a diagram showing the relationship between characteristic SEI and application SEI according to the embodiment of the present invention. Figure 5 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. Figure 6 It is a functional block diagram showing the configuration of the refinement unit 306 . Figure 7 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. Figure 8It is a functional block diagram showing the configuration of the refinement unit 306 . Fig. 9 : is a flowchart showing the decoding process of SEI. Fig.10 : is a flowchart showing the operation of the refinement unit 306 . Fig.11 The syntax of the property SEI is shown. Fig.12 is a diagram showing an example of refinement target information. Fig.13 A portion of the syntax of the characteristic SEI is shown. Fig.14 A portion of the syntax of the characteristic SEI is shown. Fig.15 A portion of the syntax of the characteristic SEI is shown. Fig.16 An example of the syntax structure of application SEI is shown. Fig.17 This is a functional block diagram showing a schematic configuration of the 3D data encoding device 11 according to the embodiment of the present invention. Fig.18 This is a diagram showing the relationship between a 3D data encoding device, a 3D data decoding device, and an SEI message according to an embodiment of the present invention. Fig.19 An example of the syntax structure of application SEI is shown. DETAILED DESCRIPTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] Figure 1 It is a schematic diagram showing the configuration of a 3D data transmission system 1 according to the present embodiment.

[0012] The 3D data transmission system 1 is a system that transmits a coded stream obtained by encoding the 3D data to be encoded, decodes the transmitted coded stream and displays the 3D data. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31 and a 3D data display device 41.

[0013] The 3D data T is input to the 3D data encoding device 11 .

[0014] The network 21 transmits the coded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 is the Internet, a wide area network (WAN), a small network (LAN) or a combination thereof. The network 21 is not necessarily limited to a two-way communication network, but may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasting and satellite broadcasting. In addition, the network 21 may also be replaced by a storage medium such as a DVD (Digital Versatile Disc: Digital Versatile Disc, registered trademark) or a BD (Blu-ray Disc: Blu-ray Disc, registered trademark) that records the coded stream Te.

[0015] The 3D data decoding device 31 decodes the encoded stream Te transmitted by the network 21 respectively, and generates one or more decoded 3D data Td.

[0016] The 3D data display device 41 displays all or part of one or more decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 includes, for example, a liquid crystal display, an organic EL (Electro-luminescence) display, and other display devices. Examples of the display type include fixed type, mobile type, and HMD (Head Mounted Display). In addition, when the 3D data decoding device 31 has high processing power, it displays images with high image quality, and when it has only low processing power, it displays images that do not require high processing power and high display power. <operator>

[0017] The operators used in this specification are described below.

[0018] >> is a right shift, << is a left shift, & is a bitwise AND, | is a bitwise OR, |= is an OR substitution operator, and || represents a logical AND.

[0019] x?y:z is a ternary operator that takes y when x is true (other than 0) and takes z when x is false (0).

[0020] y..z represents the sequential set of integers from y to z.

[0021] Floor(a) is a function that returns the largest integer below a. <Structure of coded stream Te>

[0022] The data structure of the encoded stream Te generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31 will be described.

[0023] Figure 2 The diagram shows the hierarchical structure of data in the coded stream Te. The coded stream Te has any data structure of a V3C sample stream or a V3C unit stream. The V3C sample stream includes a sample stream header and a V3C unit, and the V3C unit stream includes a V3C unit.

[0024] The V3C unit includes a V3C unit header and a V3C unit payload. The starting point of the V3C unit (= V3C unit header) is the unit type (Unit Type) which is an ID indicating the type of the V3C unit and takes the value indicated by a label such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, or V3C_OVD.

[0025] In case the unit type is V3C_VPS (Video Parameter Set), the V3C unit payload includes a V3C parameter set.

[0026] When the unit type is V3C_AD (Atlas Data), the V3C unit payload includes a VPS ID, an atlas ID, a sample stream nal header, and a plurality of NAL units. ID is an abbreviation of Idenfication and takes an integer value greater than 0. The atlas ID can be used as an element of the application SEI.

[0027] The NAL unit includes a NAL unit type, a layer ID, a temporary ID, and RBSP (Raw byte sequence payload).

[0028] NAL units are identified by NAL unit types, including ASPS (Atlas Sequence Parameter Set), AAPS (Atlas Adaptation Parameter Set), ATL (Atlas Tilelayer), SEI (Supplemental Enhancement Information), and the like.

[0029] ATL includes an ATL header and an ATL data unit. The ATL data unit includes patch information data (patchinformation data) and other information such as the location and size of the patch.

[0030] SEI includes a payload type (payload Type) indicating the type of SEI, a payload size (payload Size) indicating the size (number of bytes) of SEI, and sei_payload indicating the data of SEI.

[0031] When the unit type is V3C_AVD (Attribute Video Data), it includes VPS ID, atlasID, attrIdx (syntax name vuh_attribute_index) of attribute frame ID, partIdx (vuh_attribute_partition_index) of partition ID, mapIdx (vuh_map_index) of map ID, auxFlag (vuh_auxiliary_video_flag) indicating whether it is auxiliary data, and video stream. Video stream indicates data such as HEVC and VVC.

[0032] When the unit type is V3C_GVD (Geometry Video Data), it includes VPS ID, atlasID, mapIdx, auxFlag, and video stream.

[0033] When the unit type is V3C_OVD (occupancy video data: Occupancy Video Data, occupancy data), it includes VPS ID, atlas ID, and video stream. (Data structure of three-dimensional information)

[0034] The three-dimensional information (3D data) of this specification is a collection of position information (x, y, z) and attribute information in three-dimensional space. For example, 3D data is expressed in the form of a point cloud, which is a point group of position information and attribute information in three-dimensional space, or a mesh composed of vertices and faces of triangles (or polygons).

[0035] Figure 3It is a diagram for explaining 3D data, an occupancy frame (a two-dimensional image indicating occupancy information, hereinafter referred to as an occupancy frame), a geometry frame (a two-dimensional image indicating depth or position information, hereinafter referred to as a geometry frame), and an attribute frame (a two-dimensional image indicating attribute information, hereinafter referred to as an attribute frame). The point cloud and mesh constituting the 3D data are divided into a plurality of parts (regions) by the 3D data encoding device 11, and the point group included in each part is projected onto a 3D bounding box ( Figure 3 (a)). The 3D data encoding device 11 generates multiple patches from the projected point group. Information related to the 3D bounding box (coordinates, size, etc.) and information related to the mapping to the projection surface (projection surface, coordinates, size, rotation, etc. of each patch) is called atlas information. The occupied frame is a two-dimensional image (a) that represents the valid area of ​​each patch (the area where the point cloud or mesh exists) as a 2D binary image (for example, the valid area is represented as 1 and the invalid area is represented as 0). Figure 3 (b)). It should be noted that the values ​​of the valid area and invalid area can also use values ​​other than 0 and 1, such as 255 and 0. The geometric frame is a two-dimensional image that represents the depth value (distance) of each patch relative to the projection surface ( Figure 3 (c-0) and (c-1)). The relationship between the depth value and the pixel value can be linear, or the distance can be derived from the pixel value through a lookup table, a formula, or a relationship based on a combination of value-based branches. The attribute frame is a two-dimensional image that represents the attribute information of the point (such as RGB color). Figure 3 The projection in the same direction may have different depths (distant view and near view) of the object surface, and mapIdx indicates the depth of the projection.

[0036] These occupancy frames, geometric frames, attribute frames, and atlas information may also be images that map (encapsulate) partial images (patches) from different projection surfaces onto a two-dimensional image. The atlas information includes the number of patches and information about the projection surfaces corresponding to the patches. The 3D data decoding device 31 reconstructs the coordinates and attribute information of the point group or mesh based on the atlas information, occupancy frames, geometric frames, and attribute frames. Here, a point refers to each point of a point cloud or a vertex of a mesh. It should be noted that the mesh information (position information) representing the vertices of the mesh may be encoded, decoded, and transmitted instead of the occupancy frame and the geometric frame. In addition, the mesh information may also be divided into a basic mesh and a mesh displacement that constitute a basic mesh as a partial set of meshes for encoding, decoding, and transmission. The mesh displacement is a displacement from the basic mesh used to represent a mesh other than the basic mesh. <Term>

[0037] Atlas: A collection of 2D bounding boxes and their associated information arranged in a rectangular frame, corresponding to a volume in 3D space in which the volumetric data is rendered.

[0038] Attributes: Any scalar or vector property associated with each point in the volume's frame, such as color, reflectivity, surface normal, transparency, material ID, etc.

[0039] Attribute frame: A two-dimensional rectangular array created as a collection of patches containing values ​​of specific attributes.

[0040] Attribute map: An attribute frame that includes attribute patch information projected to a specific depth represented by the corresponding geometry map.

[0041] Bitstream: An ordered series of bits that form a representation that encodes data. Encoded data.

[0042] CAS (Coded Atlas Sequence): A sequence of coded atlas AUs with NoOutputBeforeRecoveryFlag equal to 1 that follows an IRAP coded atlas AU with NoOutputBeforeRecoveryFlag equal to 1, in decoding order. This includes all coded atlas AUs that follow the IRAP coded atlas AU with NoOutputBeforeRecoveryFlag equal to 1, but does not include the coded atlas AUs that follow it.

[0043] The IRAP coding map AU can be any one of the IDR (instantaneous decoding refresh) coding access unit, the BLA (broken link access) coding access unit, and the CRA (clean random access) coding access unit.

[0044] Geometry: A collection of Cartesian coordinates associated with the frame of the volume.

[0045] Geometry Frame: A 2D arrangement created by integrating the geometry patch information associated with each patch.

[0046] Geometry Map: A geometry frame that contains information about geometry patches projected to a specific depth.

[0047] Occupancy map: A value indicating whether a spectrum sample corresponds to an associated sample in 3D space.

[0048] Occupancy frame: A collection of occupancy map values ​​forming a 2D arrangement, representing all occupancy maps for one atlas frame.

[0049] Patch: A rectangular area within an atlas that is associated with volume information. (Overview of characteristic SEI and application SEI)

[0050] Figure 4 (a) is a diagram showing the relationship between the refinement characteristic SEI (hereinafter referred to as characteristic SEI, nn_refinement_characteristics SEI, NNRC_SEI, characteristic information) and the refinement application SEI (hereinafter referred to as application SEI, nn_refinement_activation SEI, NNRA_SEI, activation SEI, activation information) of an embodiment of the present invention. The NNC decoding unit decodes the compressed NN (neural network) model transmitted by the characteristic SEI and derives the NN model. MPEG Neural Network Coding (ISO / IEC 15938-19) can be used in the encoding and decoding of the NN model. The refinement unit 306 has an NN filter unit 611, and the NN filter unit 611 performs filtering processing using the derived NN model. The details of the NN filter unit 611 will be described later. The characteristic SEI used in the refinement is specified by the syntax element of the application SEI, such as nnra_target_id. The application SEI uses syntax elements to transmit persistence information and specifies filtering processing of attribute frames during the duration specified by the persistence information. It should be noted that the NN filter unit 611 may include an NNC decoder. Application SEI (activation information) is not limited to Fig.16 The SEI may also include NAL units and graph data.

[0051] Figure 4 (b) is another diagram showing the relationship between the refinement characteristic SEI and the refinement application SEI according to an embodiment of the present invention. The refinement unit 306 includes a linear filter unit (ALF unit) 610. The filter parameter decoding unit decodes the syntax of the filter parameters transmitted by the characteristic SEI and derives the coefficients of the linear filter. It is not limited to loop filtering and can also be post-filtering. In addition, although it is recorded as linear, it can also include simple clipping, square components and other nonlinear elements. The ALF unit 610 performs filtering processing on the attribute frame or geometric frame using the derived linear model. (Configuration of 3D Data Decoding Device)

[0052] Figure 5 , Figure 7 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention.

[0053] The 3D data decoding device 31 is composed of a V3C unit decoding unit 301, a spectrum decoding unit 302 (refinement information decoding unit), an occupancy map decoding unit 303, a geometry decoding unit 304, an attribute decoding unit 305, a post-decoding transformation unit 308, a pre-reconstruction unit 310, a reconstruction unit 311, and a post-reconstruction unit 312.

[0054] The V3C unit decoding unit 301 receives coded data (bitstream) in the form of byte stream, ISO Base Media File Format (ISOBMFF), etc., and decodes the V3C unit header and V3C VPS. The V3C unit decoding unit 301 selects the spectrum decoding unit 302, the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305 according to the unit type of the V3C unit header. When the unit type is V3C_AD, the spectrum decoding unit 302 is used. Similarly, when the unit type is V3C_OVD, V3C_GVD, or V3C_AVD, the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305 are used to decode the occupancy frame, the geometry frame, and the attribute frame, respectively.

[0055] The spectrum decoding unit 302 receives the spectrum data and decodes the spectrum information.

[0056] The graph decoding unit 302 (refinement information decoding unit) decodes the characteristic SEI indicating the characteristic of refinement processing from the coded data. The refinement information decoding unit decodes the information of the refinement application target (refinement target information). Furthermore, the graph decoding unit 302 decodes the application SEI from the coded data.

[0057] The atlas decoding unit 302 decodes the identifier atlasID indicating the object graph information indicating the refined application object from the V3C unit including the application SEI.

[0058] The occupancy map decoding unit 303 decodes the occupancy data encoded by VVC, HEVC, etc., and outputs the occupancy frame DecOccFrames[frameIdx][compIdx][y][x]. Here, DecOccFrames, frameIdx, compIdx, y, and x represent the decoded occupancy frame, frame ID, component ID, row index, and column index, respectively. In DecOccFrames, compIdx can be 0.

[0059] The geometry decoding unit 304 decodes the geometry data encoded by VVC, HEVC, etc., and outputs the geometry frame DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]. Here, DecGeoFrames, frameIdx, mapIdx, compIdx, y, and x represent the decoded geometry frame, frame ID, map ID, component ID, row index, and column index, respectively. DecGeoBitDepth, DecGeoHeight, DecGeoWidth, and DecGeoChromaFormat are marked as the bit depth of the geometry frame, the height of the geometry frame, the width of the geometry frame, and the chroma format of the geometry frame, respectively. The decoded geometry frame sometimes includes multiple geometry maps (geometry frames with different depth projections), and mapIdx is used to distinguish the maps. In DecGeoFrames, compIdx can be 0.

[0060] The attribute decoding unit 305 decodes the attribute data encoded by VVC, HEVC, etc., and outputs the attribute frame DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]. Here, DecAttrFrames, frameIdx, attrIdx, partIdx, mapIdx, compIdx, y, and x represent the decoded attribute frame, frame ID, attribute ID, partition ID, map ID, component ID, row index, and column index, respectively. DecAttrBitDepth, DecAttrHeight, DecAttrWidth, and DecAttrChromaFormat represent the bit depth of the attribute frame, the height of the attribute frame, the width of the attribute frame, and the chroma format of the attribute frame. The decoded attribute frame sometimes includes multiple attribute maps (attribute frames with different depth projections), and mapIdx is used to distinguish the maps. The decoded attribute frame consists of multiple attributes such as color (R, G, B), reflection, α, and normal direction. Furthermore, multiple attributes can be transmitted through multiple attribute data, and attrIdx is used to distinguish them. For example, {R, G, B} is attribute data 0 (attrIdx=0), {reflection} is attribute data 1 (attrIdx=1), and {α} is attribute data 2 (attrIdx=2). Attributes can be split into multiple bit streams for transmission, and partIdx is used to distinguish them. mapIdx is as described above.

[0061] The post-decoding conversion unit 308 receives the decoded atlas information, the decoded occupancy frames DecOccFrames, the decoded geometry frames DecGeoFrames and the decoded attribute frames DecAttrFrames, and converts them into nominal formats. It outputs the nominal formats of occupancy frames, geometry frames and attribute frames OccFramesNF[frameIdx][CompTimeIdx][y][x], GeoFramesNF[mapIdx][CompTimeIdx][frameIdx][y][x], AttrFramesNF[attrIdx][mapIdx][CompTimeIdx][compIdx][y][x]. Here, frameIdx, CompTimeIdx, y, x, mapIdx, attrIdx and compIdx represent the frame ID, the index of synthesis time, the row index, the column index, the map ID, the attribute ID and the component ID respectively.

[0062] The nominal format is a general term for the index of the nominal bit depth, resolution, chroma format, and composition time of the decoded image to be converted.

[0063] Each image sub-bitstream and each region of the padded image sub-bitstream is associated with a nominal bit depth. This is the target bit depth envisioned in all operations used for reconstruction. The nominal bit depth OccBitDepthNF of the occupancy map bit depth is set to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1. oi_occupancy_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the occupancy frame of the atlas with atlasID=j to be converted. pin_occupancy_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the decoded region of the atlas with atlasID=j that includes occupancy map data. When pin_occupancy_present_flag[j] is equal to 0, it indicates that the padded image frame does not include the atlas with atlasID=j and has an area of ​​occupancy map data. When pin_occupancy_present_flag[j] is equal to 1, it indicates that there is an area of ​​the filled image frame of the atlas with atlasID=j with occupancy map data. When pin_occupancy_present_flag[j] is not present, its value is presumed to be equal to 0. The nominal bit depth GeoBitDepthNF of each geometry frame is set to gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 when pin_geometry_present_flag[ConvAtlasID] is equal to 1. gi_geometry_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of all geometry frames of the atlas with converted atlasID=j. pin_geometry_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the decoded area of ​​the atlas with converted atlasID=j that includes geometry data. pin_geometry_present_flag[j]=0 indicates that the filled image frame of the atlas with atlasID=j does not include an area with geometric data. pin_geometry_present_flag[j]=1 indicates that the filled image frame of the atlas with atlasID=j contains an area with geometric data.In the absence of pin_geometry_present_flag[j], its value is presumed to be equal to 0. Finally, in the case of pin_attribute_present_flag[ConvAtlasID]=1, the nominal bit depth AttrBitDepthNF[attrIdx] for each attribute frame with attrIdx is set to ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1 or pin_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]. Increasing ai_attribute_2d_bit_depth_minus1[j][i] by 1 indicates the nominal 2D bit depth of all attribute frames with attrIdx=i for atlasID=j. Increasing pin_attribute_2d_bit_depth_minus1[j][i] by 1 indicates the nominal 2D bit depth of the region of the atlasID=j attribute that is included in the atlas conversion. pin_attribute_present_flag[j]=0 indicates that the image frame after filling of the atlas with atlasID=j does not include an area of ​​attribute data. pin_attribute_present_flag[j]=1 indicates that the image frame after filling of the atlas with atlasID=j includes an area of ​​attribute data.

[0064] When ConvAtlasID is set equal to vuh_atlas_id or the V3C unit header cannot be used, it is determined by external means. vuh_atlas_id is represented by the V3C unit header such as V3C_AD, V3C_OVD, V3C_GVD, V3C_AVD, etc., and specifies the ID of the atlas corresponding to the current V3C unit.

[0065] For asps_frame_width, the frame width of the atlas is represented by an integer number of samples, corresponding to the luma samples of the image component. For asps_frame_width, it is a necessary condition for V3C bitstream consistency to be equal to the value of vps_frame_width[j] (j is the current atlasID). For asps_frame_height, the frame height of the atlas is represented by an integer number of samples, corresponding to the luma samples of the image component. It is a necessary condition for V3C bitstream consistency to have the value of asps_frame_height equal to the value of vps_frame_height[j], where j is the current atlasID. The nominal frame resolution of the auxiliary image component is defined by the nominal width and height specified by the variables AuxVideoWidthNF and AuxVideoHeightNF, respectively. AuxVideoWidthNF and AuxVideoHeightNF are derived from the auxiliary image sub-bitstream associated with the atlas.

[0066] The nominal chroma format is defined as 4:4:4.

[0067] The functions of the post-decoding conversion unit 308 are composed of bit depth conversion, resolution conversion, output order conversion, spectrum composition alignment, spectrum dimension alignment, chroma upsampling, geometric map synthesis, and attribute map synthesis. The image frames provided by the V3C decoding unit 309 sometimes require additional processing steps before entering the reconstruction process. Such processing steps may include conversion to the nominal format of the decoded frame (e.g., resolution, bit depth, chroma format, etc.). The information of the nominal format is notified by signaling in the V3CVPS.

[0068] The pre-reconstruction unit 310 can receive decoded atlas information, decoded occupancy frames, decoded geometry frames, and decoded attribute frames, and refine / correct them. Specifically, when the occupancy map synthesis flag os_method_type[k] is equal to 1 indicating patch border filtering, OccFramesNF[compTimeIdx][0] and GeoFramesNF[0][compTimeIdx][0] are input, and the corrected arrangement OccFramesNF[compTimeIdx][0] is used as output to start the occupancy map synthesis. OccFramesNF represents an occupancy frame decoded in a nominal format, and GeoFramesNF represents a geometry frame decoded in a nominal format. compTimeIdx is an index of the synthesis time.

[0069] The refinement unit 306 refines the occupancy map, geometry, and attributes in units of frames within the pre-reconstruction unit 310. The refinement may be filtering of any one of the input occupancy map, geometry, and attributes and any one of the output occupancy map, geometry, and attributes. The refinement unit 306 may also use two-dimensional occupancy maps, geometry, and attributes simultaneously for refinement.

[0070] Furthermore, the refinement unit 306 includes an NN filter unit 611 or an ALF unit 610, and can perform filtering processing based on the transmitted neural network parameters or linear parameters.

[0071] The reconstruction unit 311 inputs the atlas information, occupancy map, geometry, and reconstructs the position, attributes, or mesh of the vertices or point groups in the 3D space. The reconstruction unit 311 reconstructs the point cloud data or mesh of the 3D data based on the reconstructed geometry information (e.g., recPcGeo) and attribute information (e.g., recPcAttr). Specifically, the reconstruction unit 311 inputs the nominal image frame, OccFramesNF[compTimeIdx][0][y][x], GeoFramesNF[mapIdx][compTimeIdx][0][y][x], and AttrFramesNF[attrIdx][mapIdx][compTimeIdx][ch][y][x] derived by the pre-reconstruction unit 310 to reconstruct the point cloud data and mesh of the 3D data. AttrFramesNF represents the attribute frame decoded in the nominal format. The reconstruction unit 311 exports the variable pointCnt as the number of points in the reconstructed point group frame, exports the one-dimensional array pointToPatch[pointCnt] as the patch index corresponding to each reconstructed point, and exports the two-dimensional array pointToPixel[pointCnt][dimIdx] as the atlas coordinates corresponding to each reconstructed point. There is a 2D array recPcGeo[pointCnt][dimIdx] as a list of coordinates corresponding to each reconstructed point, and there is a 3D array recPcAttr[pointCnt][attrIdx][compIdx] as an attribute associated with the point in the reconstructed point group frame. Here, pointCnt, attrIdx, and compIdx correspond to the size of the reconstructed point, the ID of the attribute frame, and the component ID, respectively. dimIdx represents the (x, y) component of each reconstructed point.

[0072] Specifically, the reconstruction unit 311 derives recPcGeo and recPcAttr as follows.

[0073] Here, pIdx is the index of the patch. compTimeIdx represents the composition time index, rawPos1D represents the one-dimensional position, gFrame and aFrame represent the geometry frame and attribute frame respectively. TilePatch3dOffsetU is the tile patch parameter associated with the patch. ai_attribute_count[j] represents the number of attributes associated with the atlas with atlasID=j. AtlasPatch3dOffsetU, AtlasPatch3dOffsetV, AtlasPatch3dOffsetD represent Figure 3 (a) Parameters of the location of the 3D bounding box on the 3D model.

[0074] Here, AtlasPatchRawPoints, AtlasPatch2dPosX, AtlasPatch2dPosY, AtlasPatch2dSizeX, and AtlasPatch2dSizeY are patch information derived from the atlas information by the atlas decoding unit 302 .

[0075] The arrangement oFrame[y][x], gFrame[mapIdx][y][x], aFrame[mapIdx][attrIdx][compTimeIdx][y][x] is exported as follows:

[0076] Here, ai_attribute_dimension_minus1+1 is signaled in the V3C VPS and indicates the total dimension (ie, the number of channels) of the attributes decoded by the V3C unit decoder 301. asps_map_count_minus1+1 indicates the number of maps used in encoding the geometry data and attribute data of the current atlas. RecAtlasID indicates the ID of the decoded atlas.

[0077] The post-reconstruction unit 312 changes (updates) the 3D data point group data or mesh processed by the reconstruction unit 311. The post-reconstruction unit 312 receives pointCnt as the number of reconstructed points of the current point group frame associated with the current atlas, a one-dimensional array attrBitDepth[] as the nominal bit depth, oFrame[][], recPcGeo[][], recPcAttr[][][], applies geometric smoothing, and outputs the changed recPcGeo[][], recPcAttr[][][]. (Detailing section 306)

[0078] Figure 6 It is a functional block diagram showing the structure of the refinement unit 306 (306DEC).

[0079] The refinement unit 306 (306DEC) decodes through the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305, and applies refinement to the DecOccFrames, DecGeoFrames, and DecAttrFrames before conversion by the post-decoding conversion unit 308. Then, the refined DecOccFrames, DecGeoFrames, and DecAttrFrames are output to the post-decoding conversion unit 308.

[0080] The refinement unit 306 inputs DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x] of each mapIdx and frameIdx, and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] of each attrIdx, partIdx, mapIdx, frameIdx, and compIdx, and outputs the corrected DecOccFrames, DecGeoFrames, and DecAttrFrames. The refinement unit 306 stores the output in DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x], and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] according to the refinement target information.

[0081] The refinement unit 306 may also refine the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1). In this case, oFrame=DecOccFrames[0][frameIdx], gFrame=DecGeoFrames[mapIdx][frameIdx][0], and aFrame=DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx] may be input. Here, attrIdx and mapIdx may be the values ​​nnrc_map_index and nnrc_attribute_index obtained by decoding the NNRC SEI. attrIdx=nnrc_attribute_index, mapIdx=nnrc_map_index. frameIdx may be compTimeIdx which is an index of composition time.

[0082] Furthermore, attrIdx and mapIdx may be values ​​nnra_map_index and nnra_attribute_index obtained by decoding the NNRA SEI. attrIdx=nnra_attribute_index, mapIdx=nnra_map_index. The same applies to the following.

[0083] According to the above configuration, the network model to be applied (filtering characteristics) is specified by the characteristic SEI, and the refinement object information specified by the characteristic SEI is used to specify any one of the occupancy map, geometry, and attributes, and refinement processing is applied to multiple inputs simultaneously, thereby achieving the effect of improving image quality.

[0084] Figure 7 An example is shown in which the thinning section 306 applies thinning to an image signal (video signal) in a nominal format.

[0085] Figure 8306NF). The refinement unit 306 (306NF) applies refinement to the nominal format OccFramesNF, GeoFramesNF, and AttrFramesNF converted by the post-decoding conversion unit 308, respectively, based on the refinement target information, and outputs the applied OccFramesNF, GeoFramesNF / AttrFramesNF to the pre-reconstruction unit 310. The processing of the refinement unit 306 is described below, but the processing of the refinement unit 306DEC replaces OccFramesNF, GeoFramesNF, and AttrFramesNF with Figure 6 The DecOccFrames, DecGeoFrames, and DecAttrFrames are used for processing.

[0086] The refinement unit 306 may refine the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1). In this case, oFrame=OccFramesNF[compTimeIdx][0], gFrame=GeoFramesNF[mapIdx][compTimeIdx][frameIdx][0], and aFrame=AttrFramesNF[attrIdx][mapIdx][compTimeIdx].

[0087] exist Fig.10 In (a), when the refinement object information indicates an occupancy map (S3062I), the occupancy map is added to the input tensor (S3063I). For example, the following (Equation IN-OCC) may be used.

[0088] When the refinement object information indicates geometry ( S3064I ), the geometry is added to the input tensor ( S3065I ). For example, the following (Equation IN-GEO) may be used.

[0089] When the refinement object information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, the following (Equation IN-ATTR) may be used.

[0090] According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristic SEI. The refinement target information specified by the characteristic SEI is used to specify any one of the occupancy map, geometry, and attribute, and refinement processing is applied to multiple inputs simultaneously, thereby achieving an effect of improving image quality. (NN filter unit 611)

[0091] The NN filter unit 611 performs filtering processing using a neural network. The neural network is expressed by a neural network model and may include convolution (Conv).

[0092] The neural network model (hereinafter referred to as NN model) refers to the elements and combination relationship (topology) of the neural network, and the parameters (weight, bias) of the neural network. It should be noted that the NN filter unit 611 can fix the topology and only switch the parameters according to the filter target image. In addition, the neural network can also have a kernel size, a number of input channels, and a convolution determined by the number of output channels.

[0093] The input to the refinement unit 306 is set to DecFrame. The refinement unit 306 derives the input InputTensor to the NN filter unit 611 from the input image DecFrame. The NN filter unit 611 uses the inputTensor to perform filtering based on the neural network model and derives the outputTensor. The neural network model used is the model corresponding to nnra_target_id. The input image can be an image of each component or an image having multiple components as channels.

[0094] The NN filter unit 611 may repeatedly apply the following processing.

[0095] The NN filter unit 611 performs convolution operation (conv, convolution) on the inputTensor and the kernel k[m][n][yy][xx] according to the number of layers to generate an output image outputTensor with a bias added.

[0096] Here, m represents the number of channels of inputTensor, n represents the number of channels of outputTensor, yy represents the height of kernel k, and xx represents the width of kernel k.

[0097] In each layer, outputTensor is generated from inputTensor. OutputTensor[nn][yy][xx]=∑∑∑(k[mm][nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn])

[0098] Here, nn=0..n-1, mm=0..m-1, yy=0..height-1, xx=0..width-1, i=0..yy-1, j=0..xx-1. width is the width of inputTensor and outputTensor, height is the height of inputTensor and outputTensor. ∑ is the sum of mm=0..m-1, i=0..yy-1, j=0..xx-1 respectively. of is the width or height of the area around inputTensor required to generate outputTensor.

[0099] In the case of 1x1 Conv, ∑ represents the sum of mm=0..m-1, i=0, j=0. In this case, set of=0. In the case of 3x3 Conv, ∑ represents the sum of mm=0..m-1, i=0..2, j=0..2. In this case, set of=1.

[0100] When the value of yy+j-of is less than 0 or greater than height, or when the value of xx+i-of is less than 0 or greater than width, the value of inputTensor[mm][yy+j-of][xx+i-of] can be set to 0. Alternatively, the value of inputTensor[mm][yy+j-of][xx+i-of] can be set to inputTensor[mm][yclip][xclip]. Here, yclip is max(0, min(yy+j-of, height-1)), and xclip is (0, min(xx+i-of, width-1)).

[0101] In the next layer, the obtained outputTensor is used as the new inputTensor, and the same process is repeated according to the number of layers. You can also set an activation layer between layers. In addition, you can also use pooling layers and jump combinations. Finally, derive the FilteredFrame from the obtained outputTensor.

[0102] Alternatively, a process called Depth wise Conv represented by the following equation may be performed using a kernel k[n][yy][xx]: Here, nn=0..n-1, xx=0..width-1, and yy=0..height-1. outputTensor[nn][yy][xx]=∑∑(k[nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn])

[0103] In addition, nonlinear processing called Activate can also be used, such as ReLU. ReLU(x)=x>=0? x: 0

[0104] Alternatively, leakyReLU shown in the following formula may be used. leakyReLU(x)=x>=0? x:a*x

[0105] Here, a is a predetermined value less than 1, such as 0.1 or 0.125. In addition, in order to perform integer operations, all the above-mentioned values ​​of k, bias, and a may be set to integers, and right shifted after conv to generate outputTensor.

[0106] In ReLU, 0 is always output for values ​​less than 0, and the input value is kept unchanged for values ​​above 0. On the other hand, in leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. In ReLU, the gradient for values ​​less than 0 disappears, so sometimes learning is difficult to progress. In leakyReLU, the gradient for values ​​less than 0 is retained, which is less likely to cause the above problem. In addition, PReLU can also be used by parameterizing the value of a in the above leakyReLU(x). (Filtering Processing by NN Filter Unit 611)

[0107] The NN filter unit 611 derives input data inputTensor[][][] to the NN filter unit 611 based on one or more of the occupied frame oFrame, the geometric frame gFrame, and the attribute frame aFrame. The following is an example of a configuration, and the configuration described later may also be used. nnra_strength_idc is a parameter that is fed into the input tensor and indicates the strength of the refinement.

[0108] It should be noted that you can also use It is defined as follows. Instead of Inp(oFrame[yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, oFrame)) may be used with screen boundary processing.

[0109] Instead of Inp(gFrame[yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, gFrame)) in which screen boundary processing is added may be used.

[0110] Instead of Inp(aFrame[k][yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, aFrame[k])) with screen boundary processing added may be used. k=0, 1, 2, ..., etc. The same applies to the rest.

[0111] Here, overlapSize is the size of the overlap, and a value decoded from the encoded data of the characteristic SEI can be used.

[0112] The NN filter unit 611 performs NN filtering processing to derive an outputTensor from the inputTensor. As described below, a refinement process (filtering process) shown by RefineFilter() may be performed in units of blocks (blockWidth x blockHeight).

[0113] Here, DeriveInputTensors() is a function indicating input data setting, and StoreOutputTensors() is a function indicating output data storage. FrameWidth and FrameHeight are the size of the input data. BlockWidth and blockHeight are the width and height of the block. An example of the process of StoreOutputTensors() is shown below. (ALF Department 610)

[0114] It should be noted that the refinement specified by the persistence information of the NNRA SEI and the identifier of the characteristic SEI can be performed using a filter (Wiener Filter) using a linear model. Linear filtering can be performed by the ALF unit 610. Specifically, the filtered object image DecFrame is divided into small areas (x=xSb..xSb+bSW-1, y=ySb..ySb+bSH-1) of a certain size (for example, 4x4, 1x1). (xSb, ySb) is the upper left coordinate of the small area, and bSW and bSH are the width and height of the small area. Then, filtering is performed in units of small areas. The selected filter coefficient coeff[] is used to derive the refined image outFrame from DecFrame. outframe[cIdx][y][x]=∑(oeff[i]*DecFrame[cIdx][y+ofy][x+ofx]+offset)>>shift

[0115] Here, ofx and ofy are offsets of the reference position determined according to the filter position i. offset = 1<<(shift-1) Shift corresponds to the accuracy of the filter coefficient and is a constant such as 6, 7, or 8.

[0116] It should be noted that the image may be classified in small regions, and the filter coefficient coeff[classId][] may be selected based on the derived classId (=classId[y][x]) of the small region to perform filtering processing. outframe[cIdx][y][x]=(∑(coeff[classId][i]*DecFrame[cIdx][y+ofy][x+ofx]+offset)>>shift

[0117] Here, DecFrame[ch]=inputTensor[ch][yP+overlapSize][xP+overlapSize], that is, the following may be used.

[0118] It should be noted that classId can be derived using the activity and directionality of a block (one pixel in the case of a 1x1 block). For example, it can also be derived using the activity Act derived from the following difference absolute value sum. classId=Act

[0119] It should be noted that Act can also be exported as follows. Act=∑|x-xi|

[0120] Here, i=0..7, xi represents the neighboring pixels of the target pixel x. The directions of the neighboring pixels can be eight directions obtained by adding four directions at 45 degrees to the up, down, left, and right directions relative to i=0..7. In addition,

[0121] Act = ∑|-x1+2*x-xr|+∑|-ya+2*x-yb| may also be used. It should be noted that 1, r, a, and b are opposite to left, right, above, and below, respectively, and represent the pixels to the left, right, top, and bottom of x. It should be noted that

[0122] It can be clipped into NumA-1 based on quantization by shift value in the manner of Act=Min(Num-1, Act>>shift).

[0123] Furthermore, the classId can be derived using the following formula using the directivity D. classId=Act+D*NumA

[0124] Here, for example, Act=0..NumA-1, D=0..4, etc.

[0125] The ALF unit 610 may directly output OutFrame as FilteredFrame. (Decoding and filtering application with SEI)

[0126] Fig. 9 3D data decoding device performs the following processing including decoding of an application SEI message.

[0127] S6001: The map decoding unit 302 decodes nra_cancel_flag and nnra_target_id from the application SEI. nnra_cancel_flag may also be nnrc_cancel_flag.

[0128] S6002: When nnra_cancel_flag is 1, the processing of the frame that is the target of nnra_cancel_flag is terminated. When nnra_cance_flag is 0, the process proceeds to S6003.

[0129] S6003: The map decoding unit 302 decodes nnra_persistence_fiag from the application SEI.

[0130] S6005: Determine a characteristic SEI having the same nnrc_id as nnra_target_id, and derive parameters of the NN model from the characteristic SEI.

[0131] S6006: Using the exported NN model parameters, refinement (filtering) processing is performed through the NN filter unit 611 and the ALF unit 610. <Grammar Structure Example> (Neural Network Refinement Characteristics SEI)

[0132] Fig.11 The syntax of nn_refinement_characteristics(payloadSize) (Characteristics SEI) is shown. The argument payload size (payloadSize) indicates the number of bytes of this SEI message.

[0133] The persistence scope of the application characteristic SEI is CAS. That is, SEI is applied in units of CAS.

[0134] The syntax element nnrc_id indicates the ID of the characteristic SEI. In the application SEI, by transmitting the value of nnrc_id of the characteristic SEI indicating the refinement characteristic to be applied as the object ID (nnra_target_id), the refinement processing to be applied is executed.

[0135] When nnrc_mode_idc is 0, the SEI message includes the ISO / IEC15938-17 bitstream, specifies the base NNRE (neural-network refinement) or indicates that it is an update of the base NNRE from the same nnrc_id value.

[0136] The NNRC SEI message has a specific nnrc_id value in the current CAS. In the case of the first NNRCSEI message in the decoding order, if nnrc_mode_idc is equal to 1, it is determined that the basic NNRE associated with the nnrc_id value is in the format identified by nnrc_tag_uri as the tag URI, which is the neural network identified by the URI indicated by nnrc_uri.

[0137] An NNRC SEI message has a specific nnrc_id value within the current CAS, which, when not the first NNRC SEI message in decoding order, nor a duplicate of the first NNRC SEI message in decoding order, and nnrc_mode_idc is equal to 1, indicates an update to the base NNRE with the same nnrc_id value defined by the URI indicated by nnrc_uri. nnrc_uri is in the format identified by nnrc_tag_uri of the tag URI.

[0138] The value of nnrc_mode_idc in a bitstream according to this specification must be in the range 0 to 1 (inclusive).

[0139] This SEI message has a specific nnrc_id value within the current CAS, and in the case of the first NNRCSEI message in decoding order, RefineFilter() is assigned the same value as the base NNRE.

[0140] This SEI message has a specific nnrc_id value within the current CAS. If it is not the first NNRC SEI message in decoding order, nor a repetition of the first NNRC SEI message in decoding order, the updates defined by this SEI message are applied to the base NNRE to obtain RefineFilter().

[0141] Updates are not cumulative, each update is applied to the base NNRE. The base NNRE is the NNRE specified by the first NNRC SEI message (decoding order) with a specific nnrc_id value in the current CAS.

[0142] nnrc_reserved_zero_bit_a MUST be equal to 0 in a bitstream according to this specification. A decoder MUST ignore NNRCSEI messages where nnrc_reserved_zero_bit_a is not 0.

[0143] nnrc_tag_uri includes a tag URI having the construction and semantics specified in IETF RFC4151, which identifies the format and associated information of the base NNRE with the same nnrc_id value specified by nnrc_uri or a neural network used as an update for the base NNRE.

[0144] nnrc_tag_uri equals "tag:iso.org,2023:15938-17" indicating that the neural network data identified by nnrc_uri is based on ISO / IEC15938-17.

[0145] nnrc_uri comprises a URI having the construction and semantics specified in IETF Internet Standard 66 that identifies a neural network used as a base NNRE or an update to a base NNRE having the same nnrc_id value.

[0146] When nnrc_property_present_flag is equal to 1, it indicates that the input format, output format, and context elements associated with complexity exist. When nnrc_property_present_flag is 0, it indicates that the input format, output format, and context elements associated with complexity do not exist.

[0147] When this SEI message has a specific nnrc_id value in the current CAS and is the first NNRC SEI message in decoding order, nnrc_property_present_flag is equal to 1.

[0148] When nnrc_property_present_flag is equal to 0, the values ​​of all context elements that may only exist when nnrc_property_present_flag is equal to 1 are inferred to be the same as their respective corresponding context elements in the NNRC SEI message in which the SEI includes the base NNRE to be updated.

[0149] nnrc_base_flag equal to 1 indicates that the SEI message specifies a base NNRE. nnrc_base_flag equal to 0 indicates that the SEI message specifies updates associated with a base NNRE. In its absence, the value of nnrc_base_flag is inferred to be equal to 0.

[0150] The following restrictions apply to the value of nnrc_base_flag: - In case the NNRC SEI message is the first NNRC SEI message in decoding order with a specific nnrc_id value within the current CAS, the value of nnrc_base_flag is equal to 1. - In case NNRC SEI message nnrcB is not the first NNRC SEI message in decoding order with a specific nnrc_id value in the current CAS, and the value of nnrc_base_flag is equal to 1, the NNRC SEI message is a repetition of the first NNRC SEI message nnrcA with the same nnrc_id in decoding order. That is, the payload content of the next nnrcB must be the same as the payload content of nnrcA.

[0151] When the NNRC SEI message is not the first NNRC SEI message in decoding order with a specific nnrc_id within the current CAS, nor is it a repetition of the first NNRC SEI message with the specific nnrc_id, the following applies: - This SEI message defines updates to existing base NNREs in decoding order with the same nnrc_id value. - This SEI message continues until the end of the current CAS or in the current CAS in the output order of the current decoded picture, until the picture that precedes the current decoded picture in the output order, except for the decoded picture associated with the subsequent NNRC SEI message in the decoding order with the specific nnrc_id value in the current CAS The decoding of a slice and all subsequent layers completes picture association.

[0152] The syntax element nnrc_input_mode indicates the object of refinement processing (refinement object information). The refinement object information includes at least attributes. In addition, it may also include geometry and occupancy maps.

[0153] In addition, when the refinement object information includes attributes, the input channel of the attribute can be specified. Furthermore, attrID can be specified. Furthermore, mapID of the geometry and attribute can also be specified.

[0154] The refined object information may be selected from {occupancy map, attribute}, {geometry, attribute}, or {occupancy map, geometry, attribute}. The refined object information is characterized in that multiple refined objects can be repeatedly selected.

[0155] Fig.12 is a diagram showing an example of refined object information. Fig.12 As shown, nnrc_input_mode can indicate whether to refine the occupancy map, geometry, and attributes. Y indicates refinement, and - indicates no refinement. For nnrc_input_mode, the most significant bit indicates the occupancy map, the second bit indicates the geometry, and the least significant bit indicates the attribute. 0 No refinement 1 Refine the attributes 2 Refine the geometry 3 Refine geometry and attributes 4 Refine the occupancy map 5 Refinement of occupancy maps and attributes 6 Refine the occupancy map and geometry 7 Refine occupancy maps, geometry, and attributes

[0156] Whether to apply to the occupancy map, geometry, and attributes can be determined through the bitwise sum of &4, &2, and &1 respectively. OccupancyUsedFlag=(nnrc_input_mode&4) GeometryUsedFlag=(nnrc_input_mode&2) AttributeUsedFlag=(nnrc_input_mode&1)

[0157] The bit positions are not limited to those described above. OccupancyUsedFlag=(nnrc_input_mode&1) GeometryUsedFlag=(nnrc_input_mode&2) AttributeUsedFlag=(nnrc_input_mode&4)

[0158] It can also be.

[0159] It is also possible to exclude the refinement and define it as follows. 0 Refine the attributes 1 Refine the geometry 2 Refine geometry and attributes 3 Refine the occupancy map 4 Refinement of occupancy maps and attributes 5 Refine the occupancy map and geometry 6 Refine occupancy maps, geometry, and attributes In this case, OccupancyUsedFlag=(nnrc_input_mode*1)&4 GeonetryUsedFlag=(nnrc_input_moce+1)&2 AttributeUsedFlag=(nnrc_input_mode+1)&1

[0160] The syntax element nnrc_inp_out_format_idc (input / output tensor format information) indicates a method of converting the pixel values ​​of the decoded image into input / output values ​​for the refinement process. When the value of nnrc_inp_out_format_idc is 0, the input values ​​for the refinement process (particularly the input tensor) are real numbers (in the form of floating point values) specified in IEEE754, and the function Inp is specified as follows. The range of the value of the input tensor is 0..1. Inp(x)=x÷((1<<BitDepth)-1 InpS(x)=x

[0161] When the value of nnrc_inp_out_format_idc is 1, the input / output value to the thinning process is an unsigned integer, and the function Inp is specified as follows: The range of the value of the input tensor is 0..1<<(inpTensorBitDepth)-1. The value obtained by adding 8 to the value of the syntax element nnrc_inp_tensor_bitdepth_minus8 represents the pixel bit length of the luma pixel value of the input integer tensor. The value of the variable inpTensorBitDepth is derived from the syntax element nnrc_inp_tensor_bitdepth_minus8 as follows. inpTensorBitDepth=nnrc_inp_tenscr_bitdepth_minus8+8

[0162] A block is an arrangement of pixels. Thinning is performed in fixed blocks. A block is also called a patch.

[0163] The syntax element nnrc_block_size_idc indicates the block size. The block size can be 64, 128, 192, or multiples of 64. blockWidth=64*nnrc_block_size_idc(=nnre_block_size_idc<<6) blockHeight=64*nnrc_block_size_idc(=nnrc_block_size_idc<<6)

[0164] Alternatively, the value obtained by removing 0 from nnrc_block_size_idc may be the value below. blockWidth=(nnre_block_size_idc+1)<<6 blockHeight=(nnrc_block_size_idc+1)<<6

[0165] The syntax element nnrc_overlap_size_idc specifies the number of horizontal and vertical pixels by which adjacent input tensors are overlapped. The value of nnrc_overlap_size_idc can be a multiple of 4 of the size of the overlap. overlapSize=nnrc_overlap_size_idc<<2

[0166] The syntax element nnrc_auxiliary_inp_idc indicates whether there is data added to inputTensor. If it is 0, there is no additional data, and if it is greater than 0, additional data is input to inputTensor. The additional data may be a parameter decoded from NNRA SEI (eg, nnra_strength_idc).

[0167] The function Out which converts the brightness pixel value and the color difference pixel value output by post-processing into integer values ​​of the pixel bit length is specified as follows using the pixel bit length BitDepth. Out(k)=Clip3(0,(1<<BitDepth)-1,Round(x*((1<<BitDepth)-1))) The function Out is specified as follows.

[0168] nnrc_out_tensor_bitdepth_minus8+8 specifies the pixel bit depth of the pixel values ​​of the output integer tensor. The value of outTensorBitDepth is derived from the syntax element nnrc_out_tensor_bitdepth_minus8 in the following way. outTensorBitDepth=nnrc_out_tensor_bitdepth_minus8+8

[0169] The syntax element nnrc_reserved_zero_bit_b must be equal to 0.

[0170] The syntax element nnrc_payload_byte[i] contains the i-th byte of the bit stream according to ISO / IEC 15938-17. nnrc_payload_byte[i] must be a bit stream completely according to ISO / IEC 15938-17. <Specify the input tensor and output tensor according to the refined object information>

[0171] Fig.10 306 is a flowchart showing the operation of the refinement unit 306 .

[0172] for Fig.10 The example shown is further described in detail.

[0173] exist Fig.10 In (a), the refinement unit 306 or the atlas decoding unit 302 decodes the refinement target information (S3061).

[0174] The refinement unit 306 can derive inputTensor according to nnrc_input_mode. Hereinafter, ch represents the index (position) of the channel to be set. ch++ is an abbreviation of ch=ch+1, which represents the index +1 of the channel to be set. ch=0

[0175] When the refined object information indicates an occupancy map ( S3062I ), the occupancy map is added to the input tensor ( S3063I ). For example, the following (Equation IN-OCC) may be used.

[0176] When the refinement object information indicates geometry (S3064I), the geometry is added to the input tensor (S3065I). For example, the following (Eouation IN-GEO) may be used.

[0177] When the refinement target information indicates an attribute (S3066I), an attribute is added to the input tensor (S3067I). For example, the following (Equation IN-ATTR) may be used.

[0178] exist Fig.10 In (b), the refinement unit 306 may derive (update) the output frame from the outputTensor according to nnrc_input_mode. ch=0

[0179] When the refined object information represents an occupancy map ( S3062O ), a specific component of the output tensor may be set for the occupancy map ( S3063O ). For example, the following (EquationOUT-OCC) may be used.

[0180] When the refinement object information represents geometry (S3064O), a specific component of the output tensor is set for the geometry (S3065O). For example, the following (Equation OUT-GEO) may be used.

[0181] When the refinement object information indicates an attribute (S3066O), a specific component of the output tensor is set for the attribute (S3067O). For example, the following (EquationOUT-ATTR) may be used.

[0182] In the above, the atlas decoding unit 302 decodes the refinement object information from the encoded data, and the refinement unit 306 uses the decoded refinement object information to set the occupancy map, geometry, and attributes for the input tensor. Moreover, the refinement unit 306 refines the input tensor to obtain an output tensor. Using the decoded refinement object information, the occupancy map, geometry, and attributes are set for the output tensor. According to the above configuration, multiple occupancy maps, geometries, and attributes can be combined to perform refinement processing, thereby improving efficiency. In addition, by decoding the refinement object information from the encoded data, the input / output of the model can be freely set according to the complexity. In addition, the content of the refinement can be determined and processed.

[0183] Fig.13 A portion of the syntax of nn_refinement_characteristics(payloadSize) (Characteristics SEI) is shown. Fig.13 The characteristic SEI shown in (a) further includes a syntax element nnrc_attribute_num_minus1 indicating the number of attributes as refinement target information.

[0184] The refinement unit 306 may derive inputTensor according to nnrc_input_mode. ch=0

[0185] When the refined object information represents an occupancy map, the occupancy map is appended to the input tensor, for example (Equation IN-OCC).

[0186] In the case where the refined object information represents geometry, append the geometry to the input tensor. For example (Equation IN-GEO).

[0187] When the refined object information represents an attribute, an attribute is added to the input tensor, for example, the following (EquationIN-ATTR-NUM).

[0188] It should be noted that nnrc_attribute_num_minus1 is not included as a syntax element of the characteristic SEI, and the refinement unit 306 may use ai_attribute_dimension_minus1[RecAtlasID][attrIdx] obtained by decoding the coded data of the V3C VPS. Alternatively, ai_attribute_dimension_minus1=2 may be used.

[0189] The refinement unit 306 may derive (update) an output frame from the outputTensor according to nnrc_input_mode. ch=0

[0190] In the case where the refined object information represents an occupancy map, a specific component of the output tensor is set for the occupancy map, for example (Equation OUT-OCC).

[0191] In the case where the refined object information represents geometry, append specific components of the output tensor for the geometry. For example (Equation OUT-GEO).

[0192] When the refined object information represents an attribute, a specific component of the output tensor is added to the attribute, for example, the following (Equation OUT-ATTR-NUM).

[0193] In the above, the atlas decoding unit 302 further decodes the number of attributes from the encoded data as refinement object information, and the refinement unit 306 uses the decoded refinement object information to select any one of the occupancy map, geometry, and attribute to set the input tensor. In addition, the refinement unit 306 refines the input tensor to obtain an output tensor. The decoded refinement object information is used to set the occupancy map, geometry, and attribute for the obtained output tensor. Any one of the occupancy map, geometry, and attribute is selected to set the occupancy map, geometry, and attribute from the output tensor. According to the above configuration, the number of attributes to which refinement is applied can be changed, and the input / output of the model can be freely set according to the complexity.

[0194] Fig.13 The characteristic SEI shown in (b) further includes, as refinement target information, a syntax element nnrc_attribute_start_pos indicating the initial position of the attribute and a syntax element nnrc_attribute_num_minus1 indicating the number of attributes.

[0195] The refinement unit 306 may derive inputTensor according to nnrc_input_mode. ch=0

[0196] When the refined object information represents an occupancy map, the occupancy map is appended to the input tensor. For example (Equation IN-OCC)).

[0197] In the case where the refined object information represents geometry, append the geometry to the input tensor. For example (Equation IN-GEO).

[0198] When the refined object information represents an attribute, an attribute is added to the input tensor, for example, the following (Equation IN-ATTR-POS).

[0199] Alternatively, the refinement unit 306 may derive (update) the output frame from the outputTensor according to nnrc_input_mode. ch=0

[0200] In the case where the refined object information represents an occupancy map, a specific component of the output tensor is set for the occupancy map, for example (Equation OUT-OCC).

[0201] In the case where the refined object information represents geometry, append specific components of the output tensor for the geometry. For example (Equation OUT-GEO).

[0202] When the refined object information represents an attribute, a specific component of the output tensor is set for the attribute, for example, as shown in the following (Equation OUT-ATTR-POS).

[0203] In the above, the graph decoding unit 302 decodes the number of attributes from the encoded data as refinement target information, and the refinement unit 306 sets the occupancy map, geometry, and attributes for the input tensor using the refinement target information. Furthermore, the refinement unit 306 refines the input tensor to obtain an output tensor. The output tensor is set for the occupancy map, geometry, and attributes using the refinement target information. According to the above configuration, the number of attributes to which refinement is applied can be changed, and the input / output of the model can be freely set according to the complexity.

[0204] Fig.14 A portion of the syntax of nn_refinement_characteristics(payloadSize) (Characteristics SEI) is shown. Fig.14The characteristic SEI shown in includes syntax elements nnrc_occupancy_flag, nnrc_geometry_flag, and nnrc_attribute_num as refinement target information. nnrc_occupancy_flag is a flag indicating whether to refine the occupancy map, and nnrc_geometry_flag is a flag indicating whether to refine the geometry. nnrc_attribute_num is the number of attributes that become refinement target information, and can indicate whether to refine the attribute.

[0205] The refinement unit 306 is set as follows so as to be able to perform the processing already described. OccupancyUsedFlag=nnrc_occupancy_flag GecmetryUsedFlag=nnrc_geometry_flag AttributeUsedFlag=nnrc_attribute_num

[0206] The characteristic SEI may include nnrc_attribute_flag and nnrc_attribute_num_minus1 as syntax elements instead of nnrc_attribute_num. In this case, nnrc_attribute_flag and nnrc_attribute_num_minus1 may be decoded instead of nnrc_attribute_num. The refinement unit 306 is set as follows to perform the processing described above. OccupancyUsedFlag=nnrc_pocupaney_flag GeometryUsedFlag=nnrc_geometry_flag AttributeUsedFlag=nnrc_attribute_flag

[0207] The refinement unit 306 may derive the inputTensor according to the refinement object information. ch=0

[0208] When nnrc_occupancy_flag of the refinement target information is other than 0, that is, when it indicates an occupancy map, the occupancy map is added to the input tensor. For example (Equation IN-OCC).

[0209] When nnrc_geometry_flag of the refinement target information is other than 0, that is, indicates geometry, geometry is added to the input tensor. For example, (Equation IN-GEO).

[0210] Only the number of nnrc_attribute_num of refined attribute object information is appended to the input tensor. For example, the following (Equation IN-ATTR-NUM2).

[0211] When nnrc_attribute_num=0, the attribute frame is not added to the inputTensor. Therefore, the case where the attribute frame is not refined (nnrc_attribute_num>0) can be controlled by the value of nnrc_attribute_num (nnrc_attribute_num==0).

[0212] The refinement unit 306 may derive (update) an output frame from the outputTensor according to nnrc_input_mode. ch=0

[0213] When nnrc_occupancy_flag of the refinement target information is other than 0, that is, when the occupancy map is indicated, a specific component of the output tensor is set for the occupancy map. For example, (Equation OUT-OCC).

[0214] When the nnrc_geometry_flag of the refinement target information is other than 0, that is, indicates geometry, a specific component of the output tensor is set for the geometry, for example (Equation OUT-GEO).

[0215] The attributes are set to specific components of the output tensor with the number of nnrc_attribute_num of the detailed object information, for example, as shown below (Equation OUT-ATTR-NUM2).

[0216] In the above, the atlas decoding unit 302 uses separate syntax elements to set input tensors for occupancy maps, geometry, and attributes. In addition, the refinement unit 306 refines the input tensor to obtain an output tensor. The refinement object information is also used to set the output tensor for the occupancy map, geometry, and attributes.

[0217] The refinement object information may also include a syntax element nnrc_attribute_start_pos. nnrc_attribute_start_pos indicates the Fig.13 (b) The grammatical element of the starting position of the refined attribute.

[0218] Add attributes to the input tensor with the number of nnrc_attribute_num_minus1+1 of refined object information. For example, the following (Equation IN-ATTR-POS2).

[0219] The attributes are set to specific components of the output tensor with the number of nnrc_attribute_num_minus1+1 of the refined object information, for example, as shown below (Equation OUT-ATTR-POS2).

[0220] Fig.15 A portion of the syntax of nn_refinement_characteristics(payloadSize) (Characteristics SEI) is shown. Fig.15 The characteristic SEI shown in FIG. 1 includes a syntax element nnrc_map_index indicating a map of a refinement target when geometry and attributes are included as refinement target information.

[0221] When the refined object information represents an occupancy map, the occupancy map is appended to the input tensor, for example (Equation IN-OCC).

[0222] When the refinement target information indicates geometry, the geometry indicated by nnrc_map_index is added to the input tensor. For example, mapIdx=nnrc_map_index is set by (Equation IN-GEO).

[0223] When the refinement object information indicates an attribute, the attribute indicated by nnrc_map_index is added to the input tensor. For example, mapIdx = nnrc_map_index is set by (Equation IN-ATTR), (Equation IN-ATTR-NUM), (Equation IN-ATTR-POS), (Equation IN-ATTR-NUM2), (Equation IN-ATTR-POS2).

[0224] In the case where the refined object information represents an occupancy map, specific components of the output tensor may also be set for the occupancy map, for example (Equation OUT-OCC).

[0225] When the refinement object information represents geometry, a specific component of the output tensor is set for the geometry indicated by nnrc_map_index, for example, mapIdx=nnrc_map_index through (Equation OUT-GEO).

[0226] When the refinement object information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by nnrc_map_index. For example, mapIdx = nnrc_map_index is set through (Equation OUT-ATTR), (Equation OUT-ATTR-NUM), (Equation OUT-ATTR-POS), (Equation OUT-ATTR-NUM2), (Equation OUT-ATTR-POS2).

[0227] In the above, the atlas decoding unit 302 decodes the nnrc_map_index of the geometry and attribute to be refined from the characteristic SEI and derives mapIdx. The geometry and attribute specified by mapIdx are selected for refinement, so that the optimal refinement can be applied to a specific mapIdx.

[0228] Furthermore, when the property SEI includes an attribute as refinement target information, it may include a syntax element nnrc_attribute_index indicating attrIdx of the refinement target.

[0229] When the refined object information indicates an attribute, the attributes indicated by nnrc_attribute_index and nnrc_map_index are added to the input tensor. For example, mapIdx = nnrc_map_index and attrIdx = nnrc_attribute_index can be set by (Equation IN-ATTR), (Equation IN-ATTR-NUM), (Equation IN-ATTR-POS), (Equation IN-ATTR-NUM2), and (Equation IN-ATTR-POS2).

[0230] When the refined object information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by nnrc_map_index. For example, mapIdx = nnrc_map_index, attrIdx = nnrc_attribute_index can be set through (Equation OUT-ATTR), (Equation OUT-ATTR-NUM), (Equation OUT-ATTR-POS), (Equation OUT-ATTR-NUM2), (Equation OUT-ATTR-POS2).

[0231] Furthermore, the characteristic SEI may include nnrc_attribute_partition_index and nnrc_auxiliary_video_flag. nnrc_attribute_partition_index is an index of a partition indicating an attribute of a refinement object, and nnrc_auxiliary_video_flag is a flag indicating auxiliary data of a refinement object.

[0232] In the above, the atlas decoding unit 302 decodes the nnrc_attribute_index of the geometry and attribute to be refined from the characteristic SEI, and derives attrIdx. The attribute specified by attrIdx is selected for refinement, so that the optimum refinement is applied to a specific attrIdx. (Neural Network Refinement Application SEI)

[0233] Fig.16 (a) shows an example of a syntax structure to which SEI is applied.

[0234] In response to the application of SEI, the graph decoding unit 302 (detailed information decoding unit) and the graph encoding unit 102 (detailed information encoding unit) decode and encode the following syntax elements.

[0235] nnra_target_id indicates the ID of the characteristic SEI to be applied (an identifier of refined characteristic information, identification information). The refinement process specified by the characteristic SEI having the nnrc_id equal to nnra_target_id is applied to the image.

[0236] nnra_cancel_flag is a cancellation flag. When nnra_cancel_flag is 1, it indicates that the maintenance of the refinement of the picture setting by the decoded NNRASEI is canceled. When nnra_cancel_flag is 0, it indicates that the subsequent syntax element (nnra_persistence_flag) is transmitted / encoded / decoded.

[0237] nnra_persistence_flag indicates persistence information of object refinement. When nnra_persistence_flag is 0, it indicates that the refinement of the object is only applied to the picture indicated by atlasID. When nnra_persistence_flag is 1, it indicates that the refinement of the object indicated by nnra_target_id is applied to the current picture and all subsequent pictures until any of the following conditions is met. Start a new CAS · Bitstream ends The NNRA SEI message with nnra_cancel_flag=1 is output following the current picture in the output order. That is, it is cancelled by nnra_cancel_flag=1.

[0238] In addition, it may be the following. · The NNRA SEI message with the same nnra_target_id as the current SEI message and with nnra_cancel_flag=1 is output following the current picture in the output order. That is, it is cancelled by nnra_cancel_flag=1.

[0239] Fig.16 (b) represents an example of a syntax structure of an application SEI. The atlas decoding unit 302 and the atlas encoding unit 102 decode and encode the syntax element representing the second refinement object information. The second refinement object information may include nnra_map_index representing mapIdx as a refinement application object, nnra_attribute_index representing attrIdx as a refinement application object, nnra_attribute_partition_index representing partIdx as a refinement application object, and nnra_auxiliary_video_flag indicating whether the refinement application object is auxiliary data.

[0240] When the refined object information represents an occupancy map, the occupancy map is appended to the input tensor, for example (Equation IN-OCC).

[0241] When the refinement target information indicates geometry, the geometry indicated by the second refinement target information nnra_map_index is added to the input tensor. For example, mapIdx=nnra_map_index can be set by (Equation IN-GEO).

[0242] When the refinement target information indicates an attribute, the attribute indicated by the second refinement target information nnra_map_index is added to the input tensor. For example, mapIdx=nnra_map_index can be set by (Equation IN-ATTR), (Equation IN-ATTR-NUM), (Equation IN-ATTR-POS), (Equation IN-ATTR-NUM2), (Equation IN-ATTR-POS2).

[0243] In the case where the refined object information represents an occupancy map, specific components of the output tensor may also be set for the occupancy map, for example (Equation OUT-OCC).

[0244] When the refinement target information indicates geometry, a specific component of the output tensor is set for the geometry indicated by the second refinement target information nnra_map_index. For example, it can be set as mapIdx=nnra_map_index through (Equation OUT-GEO).

[0245] When the refinement target information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by the second refinement target information nnra_map_index. For example, mapIdx=nnra_map_index can be set through (Equation OUT-ATTR), (Equation OUT-ATTR-NUM), (Equation OUT-ATTR-POS), (Equation OUT-ATTR-NUM2), (Equation OUT-ATTR-POS2).

[0246] In the above, the atlas decoding unit 302 derives mapIdx from the nnra_map_index of the geometry and attribute to be refined from the application SEI. The geometry and attribute specified by the mapIdx of the application SEI are selected for refinement, thereby achieving the effect of being able to apply the most optimized refinement to a specific mapIdx. Moreover, the refinement specified by the same characteristic SEI can be applied to the geometry and attribute represented by the same value of mapIdx. The refinement specified by different characteristic SEIs can be applied to the geometry and attribute represented by different mapIdx at the same time.

[0247] Furthermore, when an attribute is included as refinement target information, a syntax element nnra_attribute_index indicating attrId of the refinement target may be included.

[0248] When the refinement target information indicates an attribute, the attribute indicated by the second refinement target information nnra_attribute_index, nnra_map_index is added to the input tensor. For example, mapIdx = nnra_map_index, attrIdx = nnra_attribute_index can be set by (Equation IN-ATTR), (Equation IN-ATTR-NUM), (Equation IN-ATTR-POS), (Equation IN-ATTR-NUM2), (Equation IN-ATTR-POS2).

[0249] When the refinement object information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by the second refinement object information nnra_map_index and nnra_attribute_index. For example, mapIdx = nnra_map_index and attrIdx = nnra_attribute_index can be set by (Equation OUT-ATTR), (Equation OUT-ATTR-NUM), (Equation OUT-ATTR-POS), (Equation OUT-ATTR-NUM2), and (Equation OUT-ATTR-POS2).

[0250] Also, nnrc_attribute_partition_index, nnrc_auxiliary_video_flag may be included.

[0251] In the above, the atlas decoding unit 302 derives attrIdx from the nnra_attribute_index of the geometry and attribute to be refined by the application SEI decoding. The attribute specified by attrIdx is selected for refinement. Therefore, the effect of being able to apply the most optimized refinement to a specific attrIdx is achieved. In addition, the refinement specified by the same characteristic SEI can be applied to the attributes represented by the same value of attrIdx. Refinements specified by different characteristic SEIs can be applied to attributes represented by different attrIdx at the same time.

[0252] Fig.19An example of the syntax structure of applying SEI is shown. Here, nnra_attribute_count, the number of attributes, and nnra_map_count_minus1, the number of maps -1, are included as syntax elements. In addition, nnra_target_id[i][j] indicating the id of the characteristic SEI of the NNRE applied to each attribute attrIdx=i and map mapIdx=j is included. In addition, nnra_enabled_flag[i][j], nnra_cancel_flag[i][j], and nnra_persistent_flag[i][j] indicating whether to apply refinement to each attribute attrIdx=i and map mapIdx=j may also be included. In addition, nnra_strength_present_flag[i][j] may also be included.

[0253] When the object information is refined to indicate attributes, the attributes indicated by i and j are added. For example, mapIdx=i and attrIdx=j can be set by (Equation IN-ATTR), (Equation IN-ATTR-NUM), (Equation IN-ATTR-POS), (Equation IN-ATTR-NUM2), and (Equation IN-ATTR-POS2).

[0254] When the refined object information indicates attributes, the attributes indicated by i and j are set. For example, mapIdx=i and attrIdx=j can be set by (Equation OUT-ATTR), (Equation OUT-ATTR-NUM), (Equation OUT-ATTR-POS), (EquationOUT-ATTR-NUM2), and (EquationOUT-ATTR-POS2). (Configuration of 3D Data Encoding Device)

[0255] Fig.17 This is a functional block diagram showing a schematic configuration of the 3D data encoding device 11 according to the embodiment of the present invention.

[0256] The 3D data encoding device 11 is composed of a patch generation unit 101, a graph encoding unit 102, an occupancy map generation unit 103, an occupancy map encoding unit 104, a geometry generation unit 105, a geometry encoding unit 106, an attribute generation unit 108, an attribute encoding unit 109, a refinement parameter derivation unit 110, and a multiplexing unit 111. The 3D data encoding device 11 inputs a point cloud or a mesh as 3D data and outputs encoded data.

[0257] The patch generation unit 101 inputs 3D data, generates a set of patches (here, rectangular images) and outputs them. Specifically, the 3D data is divided into multiple regions, each of which is projected onto any plane in a 3D bounding box set in the 3D space to generate multiple patches. The patch generation unit 101 outputs information related to the 3D bounding box (coordinates, size, etc.) and information related to the mapping to the projection surface (projection surface, coordinates, size, rotation, etc. of each patch) as atlas information.

[0258] The atlas encoding unit 102 encodes the atlas information output from the patch generating unit 101 and outputs atlas data.

[0259] The occupancy map generation unit 103 inputs the set of patches output from the patch generation unit 101, and generates an occupancy map that represents the valid area (area where 3D data exists) of each patch as a 2D binary image (for example, the valid area is represented as 1, and the invalid area is represented as 0). It should be noted that the values ​​of the valid area and the invalid area may also use other values ​​such as 255 and 0.

[0260] The occupancy map encoding unit 104 inputs the occupancy map output from the occupancy map generating unit 103, and outputs the occupancy map and occupancy map data. VVC, HEVC, etc. are used as the encoding method.

[0261] The geometry generation unit 105 generates a geometry frame storing the depth value of each patch relative to the projection surface based on the 3D data, the occupancy map, the occupancy map data, and the atlas information. The geometry generation unit 105 derives the point with the smallest depth relative to the projection surface among the points projected on the pixel g(x,y) as p_min(x,y,z). In addition, the point with the largest depth among the points projected on the pixel g(x,y) and located at a prescribed distance d from p_min(x,y,z) is derived as p_max(x,y,z). The geometry frame in which p_min(x,y,z) is projected to all pixels of the projection surface is set as the geometry frame of the near layer. The geometry frame in which p_max(x,y,z) is projected to all pixels of the projection surface is set as the geometry frame of the far layer.

[0262] The geometry encoding unit 106 inputs a geometry frame and outputs a geometry frame and geometry data, and uses VVC, HEVC, etc. as an encoding method.

[0263] The attribute generation unit 108 generates an attribute frame storing color information (e.g., YUV value, RGB value, etc.) of each patch relative to the projection surface based on the 3D data, the occupancy map, the geometry frame, and the atlas information. The attribute generation unit 108 obtains the value of the attribute corresponding to the point p_min(x, y, z) with the minimum depth calculated by the geometry generation unit 105, and sets the attribute frame projected with the value as the attribute frame of the near layer. For p_max(x, y, z), the obtained attribute frame is also set as the attribute frame of the far layer.

[0264] The attribute encoding unit 109 inputs an attribute frame and outputs an attribute and attribute data. VVC, HEVC, etc. are used as encoding methods.

[0265] The refinement parameter deriving unit 110 inputs the attribute frame and the original attribute frame or the geometry frame and the original geometry frame, selects or derives and outputs the most appropriate filtering parameters in the NN filtering process, and sets the values ​​of nnra_target_id, nnra_cancel_flag, nnra_persistance_flag, etc. for the above SEI.

[0266] The multiplexing unit 111 inputs the filter parameters output from the refinement parameter derivation unit 110, and outputs them in a prescribed format. The prescribed format is, for example, SEI, which is auxiliary enhancement information of image data, ASPS and AFPS, which are designated information of data structure in the V3C standard, and ISOBMFF, which is a media file format standard. The multiplexing unit 111 also multiplexes the atlas data, occupancy map data, geometry data, attribute data, and the above-mentioned filter parameters, and outputs them as coded data. As a multiplexing method, a byte stream format, ISOBMFF, etc. are used.

[0267] Fig.18 This is a block diagram showing the relationship between the 3D data encoding device 11 , the 3D data decoding device 31 , and the SEI message according to the embodiment of the present invention.

[0268] The 3D data encoding device 11 is composed of a video encoding unit and a SEI encoding unit. Fig.17 The SEI coding unit corresponds to the occupancy map coding unit 104, the geometry coding unit 106, and the attribute coding unit 109. Fig.17 The structure corresponds to the atlas encoding unit 102.

[0269] The 3D data decoding device 31 is composed of a video decoding unit, a SEI decoding unit, a switch, and a refinement unit. Figure 5 The SEI decoding unit corresponds to the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305. Figure 5 The refinement unit corresponds to the spectrum decoding unit 302 in the structure of Figure 5 The refinement unit 306 corresponds to the refinement unit 306. The refinement unit 306 may include either the NN filter unit 611 or the ALF unit 610, or both.

[0270] The occupancy map, geometry frame, and attribute frame generated based on the 3D data are encoded by the image encoding unit. The encoded data is decoded by the image decoding unit to reconstruct the decoded image. The SEI encoding unit generates a characteristic SEI and an application SEI based on the 3D data. The SEI decoding unit decodes these SEI messages. The application SEI specifies the image to be refined, so the input switch only inputs the image to be refined to the refinement unit. The characteristic SEI is input to the refinement unit to specify the refinement applied to the decoded image. The refined image or the decoded image is displayed through the 3D data display device 41( Figure 1 )show.

[0271] As mentioned above, one embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to the above embodiment, and various design changes and the like can be made within the scope not departing from the gist of the present invention.

[0272] The embodiments of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical solutions appropriately modified within the scope of the claims are also included in the technical scope of the present invention. Industrial Applicability The embodiments of the present invention can be preferably applied to a 3D data decoding device that decodes coded data obtained by encoding 3D data and a 3D data encoding device that generates coded data obtained by encoding 3D data. In addition, the embodiments of the present invention can be preferably applied to the data structure of the coded data generated by the 3D data encoding device and referred to by the 3D data decoding device. Description of Reference Numerals 11: 3D data encoding device 101: Patch Generation Department 102: Graph Coding Department 103: Occupancy map generation unit 104: Occupancy map encoding unit 105: Geometry Generation Department 106: Geometry Coding Department 108: Attribute Generation Department 109: Attribute Coding Unit 110: Refinement parameter export section 111: Multiplexing Department 21: Network 31: 3D data decoding device 301: Header decoding unit 302: Spectrum information decoding unit 303: Occupancy map decoding unit 304: Geometry Decoding Department 306: Geometry Reconstruction Department 307: Attribute decoding unit 308: Refinement Department 309: V3C decoding unit 310: Pre-reconstruction department 311: Reconstruction Department 41: 3D data display device

Claims

1. A 3D data decoding device, which decodes coded data and decodes 3D data including geometry and attributes, wherein the 3D data decoding device is characterized by: a refinement information decoding unit that decodes the refinement characteristic information and the refinement activation information from the encoded data; A geometry decoding unit, which decodes a geometry frame from the encoded data; an attribute decoding unit, which decodes the attribute frame from the encoded data; as well as A refinement unit performs refinement processing on the attribute frame or the geometric frame. The refinement information decoding unit decodes refinement target information indicating which one of the occupancy map, geometry, and attribute is used from the encoded data of the characteristic information, and performs refinement using an image specified by the refinement target information.

2. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit further decodes the number of attributes used as the refinement target information when the refinement target information indicates the use of attributes, and the refinement unit performs refinement using the designated number of attributes.

3. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit separately decodes a syntax element indicating whether to use an occupancy map, geometry, or attribute as the refinement target information, and when the occupancy map, geometry, or attribute is specified, the refinement unit uses the occupancy map, geometry, or attribute to perform refinement.

4. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit further decodes information indicating a map ID to be applied from the characteristic information, and the refinement unit refines the designated map ID.

5. The 3D data decoding device according to claim 4, It is characterized in that When the refinement information indicates that geometry or attributes are used as the refinement target information, the refinement information decoding unit further decodes information indicating a map ID to be applied.

6. The 3D data decoding device according to claim 4, It is characterized in that When the refinement information indicates that an attribute is used as the refinement target information, the refinement information decoding unit further decodes information indicating an attribute index, a partition index, and an auxiliary flag to be applied.

7. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit decodes an index indicating characteristic information, a cancel flag, and a persistence flag as the activation information.

8. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit decodes information indicating a map ID of an application object as the activation information, and the refinement unit performs refinement using geometry or attributes specified by the map ID.

9. The 3D data decoding device according to claim 1, It is characterized in that The refinement information decoding unit decodes information indicating an attribute of an application object as the activation information, and the refinement unit performs refinement using an attribute specified by the information indicating the attribute.

10. A 3D data encoding device, encoding coded data, encoding 3D data including geometry and attributes, It is characterized in that The 3D data encoding device comprises: A refinement information encoding unit, encoding the refinement feature information and the refinement activation information of the neural network; A geometry encoding unit, encoding the geometry frame; An attribute encoding unit, encoding the attribute frame; as well as A refinement unit performs refinement processing on the attribute frame or the geometric frame. The refinement information encoding unit encodes refinement target information indicating which one of an occupancy map, geometry, and attribute is used as encoded data of the characteristic information, and performs refinement using an image specified by the refinement target information.

Images