3D data encoding device and 3D data decoding device

Abstract

Appropriate refinement cannot be applied to three-dimensional data using a plurality of images. This 3D data decoding device is provided with: a geometry decoding unit that decodes a geometry frame from encoded data; the 3D data decoding device is provided with a refinement information decoding unit for decoding refinement characteristic information and refinement activation information from the encoded data, and is characterized in that the 3D data decoding device is provided with a refinement information decoding unit for decoding a grammar element indicating the number of refinement from the activation information, and for decoding a grammar element indicating the number of refinement from the grammar element. The 3D data decoding device decodes an index of the characteristic information indicating the number of decoded refinement, and is provided with a refinement processing unit that performs refinement processing of the attribute frame or the geometric frame on the basis of the characteristic information.

Description

Technical Field

[0001] Embodiments of the present invention relate to a 3D data encoding device and a 3D data decoding device. Background Art

[0002] In order to efficiently transmit or record 3D data, there is a 3D data encoding device that converts 3D data into a two-dimensional image and encodes it using a motion image encoding method to generate encoded data; and a 3D data decoding device that decodes the two-dimensional image from the encoded data and reconstructs it to generate 3D data. In addition, there is a technology that uses auxiliary enhancement information of deep learning post-filtering to filter the two-dimensional image.

[0003] As specific 3D data encoding methods, for example, MPEG-I's ISO / IEC 23090-5 V3C (Volumetric Video-based Coding) and V-PCC (Video-based Point Cloud Compression) can be listed. V3C can be used for encoding and decoding of point clouds composed of point positions and attribute information. In addition, it can also be used for encoding and decoding of multi-viewpoint images and mesh images through ISO / IEC 23090-12 (MPEG Immersive Video (MPEG Immersive Video: MIV)) and ISO / IEC 23090-29 (Video-based Dynamic Mesh Coding (V-DMC)) which is being standardized. In addition, in the auxiliary enhancement information of the neural network post-filtering in non-patent document 1, the model information of the neural network is transmitted through the characteristic SEI, and the frames specified by the application SEI are paged, thereby enabling appropriate filtering processing of the point group data. Prior art literature Non-patent literature Non-patent document 1

[0004] K. Takada, Y. Tokumo, T. Chujoh. T. Ikai, "[V-PCC][EE2.8]V3C neural-networkpost-filter SEI messages," ISO / IEC JTC 1 / SC 29 / WG7, m61805, Jan.2023 Summary of the invention Problem that the invention aims to solve

[0005] In Non-Patent Document 1, filtering is performed only on attributes, so filtering using the relationship between occupancy maps, geometry, and attributes cannot be performed. In other words, in the method of decoding 3D data using multiple images / videos, refinement using information between images cannot be performed.

[0006] An object of the present invention is to solve the above-mentioned problem in encoding / decoding of 3D data using a moving picture encoding / decoding method, further reduce encoding distortion using refined auxiliary information, and encode / decode 3D data with high quality. Technical Solution

[0007] In order to solve the above-mentioned problem, a 3D data decoding device of one embodiment of the present invention comprises: a geometry decoding unit, which decodes a geometry frame from encoded data; and an attribute decoding unit, which decodes an attribute frame. The 3D data decoding device is characterized in that it comprises a refinement information decoding unit, which decodes refined characteristic information and refined activation information from the above-mentioned encoded data, the 3D data decoding device decodes a syntactic element indicating the number of refinements from the above-mentioned activation information, and decodes an index of the above-mentioned characteristic information indicating the number of decoded refinements, and the 3D data decoding device comprises a refinement processing unit, which performs refinement processing on the above-mentioned attribute frame or geometry frame according to the above-mentioned characteristic information.

[0008] The 3D data encoding device comprises: a geometry encoding unit for encoding a geometry frame; and an attribute encoding unit for encoding an attribute frame. The 3D data encoding device is characterized in that it comprises a refinement information encoding unit for encoding refined characteristic information and refined activation information from the above-mentioned encoded data, the 3D data encoding device encodes a grammatical element indicating the number of refinements from the above-mentioned activation information, and encodes an index of the above-mentioned characteristic information indicating the number of encoded refinements, and the 3D data encoding device comprises a refinement processing unit for performing refinement processing on the above-mentioned attribute frame or the geometry frame according to the above-mentioned characteristic information. Beneficial Effects

[0009] According to one aspect of the present invention, distortion caused by encoding of color images can be reduced, and 3D data can be encoded / decoded with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a schematic diagram showing the configuration of a 3D data transmission system according to this embodiment. Figure 2 This is a diagram showing the hierarchical structure of the data of the coded stream. Figure 3 It is a graph used to illustrate 3D data, occupancy map, geometry frame, and attribute frame. Figure 4 It is a diagram showing the relationship between characteristic SEI and application SEI according to the embodiment of the present invention. Figure 5 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. Figure 6 It is a functional block diagram showing the configuration of the refinement unit 306 . Figure 7 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention. Figure 8 It is a functional block diagram showing the configuration of the refinement unit 306 . Fig. 9 : is a flowchart showing the decoding process of SEI. Fig.10 : is a flowchart showing the operation of the refinement unit 306 . Fig.11 A part (first half) of the syntax of the characteristic SEI is shown. Fig.12 A part (latter half) of the syntax of the characteristic SEI is shown. Fig.13 This shows an example of the syntax structure of the application SEI. Fig.14 This is a functional block diagram showing a schematic configuration of the 3D data encoding device 11 according to the embodiment of the present invention. Fig.15 This is a diagram showing the relationship between a 3D data encoding device, a 3D data decoding device, and an SEI message according to an embodiment of the present invention. DETAILED DESCRIPTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0012] Figure 1 It is a schematic diagram showing the configuration of a 3D data transmission system 1 according to the present embodiment.

[0013] The 3D data transmission system 1 is a system that transmits a coded stream obtained by encoding the 3D data to be encoded, decodes the transmitted coded stream and displays the 3D data. The 3D data transmission system 1 includes a 3D data encoding device 11, a network 21, a 3D data decoding device 31 and a 3D data display device 41.

[0014] The 3D data T is input to the 3D data encoding device 11 .

[0015] The network 21 transmits the encoded stream Te generated by the 3D data encoding device 11 to the 3D data decoding device 31. The network 21 is the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network 21 is not necessarily limited to a two-way communication network and may also be a one-way communication network that transmits broadcast waves such as terrestrial digital broadcasts and satellite broadcasts. In addition, the network 21 may be replaced by a storage medium such as a DVD (Digital Versatile Disc, registered trademark) or a BD (Blu-ray Disc, registered trademark) on which the encoded stream Te is recorded.

[0016] The 3D data decoding device 31 decodes the encoded stream Te transmitted by the network 21 respectively to generate one or more decoded 3D data Td after decoding.

[0017] The 3D data display device 41 displays all or part of the one or more decoded 3D data Td generated by the 3D data decoding device 31. The 3D data display device 41 includes, for example, a display device such as a liquid crystal display or an organic EL (Electro-luminescence) display. As the form of the display, a fixed type, a mobile type, an HMD (Head Mounted Display), etc. can be cited. In addition, when the 3D data decoding device 31 has a high processing capacity, an image with high display quality is displayed, and when it only has a low processing capacity, an image that does not require high processing capacity and high display capacity is displayed. <Operator>

[0018] The following describes the operators used in this specification.

[0019] >> is a right shift, << is a left shift, & is a bitwise AND, | is a bitwise OR, |= is an OR assignment operator, and || represents a logical OR.

[0020] x? y : z is a ternary operator that takes y when x is true (other than 0) and takes z when x is false (0). y..z represents an ordered set of integers from y to z.

[0021] Clip3(a, b, c) is a function that clips c to a value greater than or equal to a and less than or equal to b, and returns a when c < a, returns b when c > b, and returns c in other cases (where a <= b).

[0022] abs(a) is a function that returns the absolute value of a.

[0023] Floor(a) is a function that returns the largest integer below a.

[0024] Round(a) is a function that returns an integer close to a by rounding.

[0025] a / d means a divided by d (discarding decimal places). <Structure of coded stream Te>

[0026] The data structure of the encoded stream Te generated by the 3D data encoding device 11 and decoded by the 3D data decoding device 31 will be described.

[0027] Figure 2 The diagram shows the hierarchical structure of data in the coded stream Te. The coded stream Te has any data structure of a V3C sample stream or a V3C unit stream. The V3C sample stream includes a sample stream header and a V3C unit, and the V3C unit stream includes a V3C unit.

[0028] The V3C unit includes a V3C unit header and a V3C unit payload. The starting point of the V3C unit (= V3C unit header) is the unit type (Unit Type) which is an ID indicating the type of the V3C unit and takes the value indicated by a label such as V3C_VPS, V3C_AD, V3C_AVD, V3C_GVD, or V3C_OVD.

[0029] In case the unit type is V3C_VPS (Video Parameter Set), the V3C unit payload includes a V3C parameter set.

[0030] When the unit type is V3C_AD (Atlas Data), the V3C unit payload includes a VPS ID, an atlas ID, a sample stream nal header, and multiple NAL units. ID is an abbreviation for Idenfication and takes an integer value greater than 0. The atlas ID can be used as an element of the application SEI.

[0031] The NAL unit includes the NAL unit type, layer ID, temporary ID, and RBSP (Raw byte sequence payload).

[0032] NAL units are identified by NAL unit types, including ASPS (Atlas Sequence Parameter Set), AAPS (Atlas Adaptation Parameter Set), ATL (Atlas Tilelayer), SEI (Supplemental Enhancement Information), etc.

[0033] ATL includes an ATL header and an ATL data unit. The ATL data unit includes patch information data (patchinformation data) and other information such as the location and size of the patch.

[0034] SEI includes a payload type (payload Type) indicating the type of SEI, a payload size (payload Size) indicating the size (number of bytes) of SEI, and sei_payload indicating the data of SEI.

[0035] When the unit type is V3C_AVD (Attribute Video Data), it includes VPS ID, atlasID, attrIdx (syntax name vuh_attribute_index) of attribute frame ID, partIdx (vuh_attribute_partition_index) of partition ID, mapIdx (vuh_map_index) of map ID, auxFlag (vuh_auxiliary_video_flag) indicating whether it is auxiliary data, and video stream. Video stream indicates data such as HEVC and VVC.

[0036] When the unit type is V3C_GVD (Geometry Video Data), it includes VPS ID, atlasID, mapIdx, auxFlag, and video stream.

[0037] When the unit type is V3C_OVD (occupancy video data: Occupancy Video Data, occupancy data), it includes VPS ID, atlas ID, and video stream.

[0038] Attribute data and geometry data have a plurality of maps distinguished by mapIdx, and attribute data have a plurality of attributes distinguished by attrIdx. (Data structure of three-dimensional information)

[0039] The three-dimensional information (3D data) of this specification is a collection of position information (x, y, z) and attribute information in three-dimensional space. For example, 3D data is expressed in the form of a point cloud, which is a point group of position information and attribute information in three-dimensional space, or a mesh (or polygon) composed of vertices and faces of triangles.

[0040] Figure 3 It is a diagram for explaining 3D data, an occupancy frame (a two-dimensional image indicating occupancy information, hereinafter referred to as an occupancy frame), a geometry frame (a two-dimensional image indicating depth or position information, hereinafter referred to as a geometry frame), and an attribute frame (a two-dimensional image indicating attribute information, hereinafter referred to as an attribute frame). The point cloud and mesh constituting the 3D data are divided into a plurality of parts (regions) by the 3D data encoding device 11, and the point group included in each part is projected onto a 3D bounding box ( Figure 3 (a)). The 3D data encoding device 11 generates a plurality of patches from the projected point group. Information related to 3D differentiation (coordinates, size, etc.) and information related to mapping to the projection surface (projection surface, coordinates, size, rotation, etc. of each patch) is called atlas information. The occupied frame is a two-dimensional image (a) that represents the valid area of ​​each patch (the area where the point cloud or mesh exists) as a 2D binary image (for example, the valid area is represented as 1 and the invalid area is represented as 0). Figure 3 (b)). It should be noted that the values ​​of the valid area and invalid area can also use values ​​other than 0 and 1, such as 255 and 0. The geometric frame is a two-dimensional image that represents the depth value (distance) of each patch relative to the projection surface ( Figure 3 (c-0) and (c-1)). The relationship between the depth value and the pixel value can be linear, or the distance can be derived from the pixel value through a lookup table, a formula, or a relationship based on a combination of value-based branches. The attribute frame is a two-dimensional image that represents the attribute information of the point (such as RGB color). Figure 3 The projection in the same direction may have different depths (distant view and near view) of the object surface, and mapIdx indicates the depth of the projection.

[0041] These occupancy frames, geometric frames, attribute frames, and atlas information may also be images that map (encapsulate) partial images (patches) from different projection surfaces onto a two-dimensional image. The atlas information includes the number of patches and information about the projection surfaces corresponding to the patches. The 3D data decoding device 31 reconstructs the coordinates and attribute information of the point group or mesh based on the atlas information, occupancy frames, geometric frames, and attribute frames. Here, a point refers to each point of a point cloud or a vertex of a mesh. It should be noted that the mesh information (position information) representing the vertices of the mesh may be encoded, decoded, and transmitted instead of the occupancy frame and the geometric frame. In addition, the mesh information may also be divided into a basic mesh and a mesh displacement that constitute a basic mesh as a partial set of meshes for encoding, decoding, and transmission. The mesh displacement is a displacement from the basic mesh used to represent a mesh other than the basic mesh. <Term>

[0042] Atlas: A collection of 2D bounding boxes and their associated information arranged in a rectangular frame, corresponding to a volume in 3D space in which the volumetric data is rendered. Attributes: Any scalar or vector property associated with each point in the volume's frame, such as color, reflectivity, surface normal, transparency, material ID, etc. Attribute frame: A two-dimensional rectangular array created as a collection of patches containing the values ​​of specific attributes. Attribute map: An attribute frame that includes attribute patch information projected to a specific depth represented by the corresponding geometry map. Bitstream: An ordered series of bits that form a representation that encodes data. Encoded data. CAS (Coded Atlas Sequence): A sequence of coded atlas AUs with NoOutputBeforeRecoveryFlag equal to 1 that follows an IRAP coded atlas AU with NoOutputBeforeRecoveryFlag equal to 1, in decoding order. This includes all coded atlas AUs that follow the IRAP coded atlas AU with NoOutputBeforeRecoveryFlag equal to 1, but does not include the coded atlas AUs that follow it. The IRAP coding map AU can be any one of the IDR (instantaneous decoding refresh) coding access unit, the BLA (broken link access) coding access unit, and the CRA (clean random access) coding access unit. Geometry: A collection of Cartesian coordinates associated with the frame of the volume. Geometry Frame: A 2D array created by integrating the geometry patch information associated with each patch. Geometry Map: A geometry frame that contains information about geometry patches projected to a specific depth. Occupancy map: A value indicating whether a spectrum sample corresponds to an associated sample in 3D space. Occupancy Frame: A collection of occupancy map values ​​forming a 2D array, representing all occupancy maps for one atlas frame. Patch: A rectangular area within an atlas that is associated with volume information. (Overview of characteristic SEI and application SEI)

[0043] Figure 4 (a) is a diagram showing the relationship between the refinement characteristics SEI (hereinafter referred to as characteristics SEI, nn_refinement_characteristics SEI, NNRC_SEI, characteristics information) and the refinement application SEI (hereinafter referred to as application SEI, nn_refinement_activation SEI, NNRA_SEI, activation SEI, activation information) of an embodiment of the present invention. The NNRC_SEI message specifies a neural network that can be used for refinement. The NNRA_SEI message indicates the use of a specified neural network refinement (neural-network refinement) for a specific picture. The NNRA_SEI message activates or deactivates the possible use of the target neural network refinement. The NNC decoding unit decodes the compressed NN (neural network) model transmitted by the characteristics SEI and derives the NN model. MPEG Neural Network Coding (ISO / IEC 15938-19) can be used in the encoding and decoding of the NN model. The refinement unit 306 has an NN filter unit 611, and the NN filter unit 611 performs filtering processing using the derived NN model. The details of the NN filter unit 611 will be described later. The characteristic SEI used in the refinement is specified by the syntax element of the application SEI, such as nnra_target_id. The application SEI uses the syntax element to transmit the persistence information, and specifies the filtering process of the attribute frame during the duration specified by the persistence information. It should be noted that the NN filter unit 611 can have an NNC decoding unit. The application SEI (activation information) is not limited to Fig.13 The SEI may also include NAL units and graph data.

[0044] Figure 4(b) is another diagram showing the relationship between the refined characteristic SEI and the refined application SEI according to an embodiment of the present invention. The refinement unit 306 includes a linear filter unit (ALF unit) 610. The filter parameter decoding unit decodes the syntax of the filter parameters transmitted by the characteristic SEI and derives the coefficients of the linear filter. It is not limited to loop filtering and can also be post-filtering. In addition, although it is recorded as linear, it can also include simple clipping, square components and other nonlinear elements. The ALF unit 610 performs filtering processing on the attribute frame or geometric frame using the derived linear model. (Configuration of 3D Data Decoding Device)

[0045] Figure 5 , Figure 7 This is a functional block diagram showing a schematic configuration of a 3D data decoding device 31 according to an embodiment of the present invention.

[0046] The 3D data decoding device 31 is composed of a V3C unit decoding unit 301, a spectrum decoding unit 302 (refinement information decoding unit), an occupancy map decoding unit 303, a geometry decoding unit 304, an attribute decoding unit 305, a post-decoding transformation unit 308, a pre-reconstruction unit 310, a reconstruction unit 311, and a post-reconstruction unit 312.

[0047] The V3C unit decoding unit 301 receives coded data (bitstream) in the form of byte stream, ISO Base Media File Format (ISOBMFF), etc., and decodes the V3C unit header and V3C VPS. The V3C unit decoding unit 301 selects the spectrum decoding unit 302, the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305 according to the unit type of the V3C unit header. When the unit type is V3C_AD, the spectrum decoding unit 302 is used. Similarly, when the unit type is V3C_OVD, V3C_GVD, or V3C_AVD, the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305 are used to decode the occupancy frame, the geometry frame, and the attribute frame, respectively.

[0048] The spectrum decoding unit 302 receives the spectrum data and decodes the spectrum information.

[0049] The graph decoding unit 302 (refinement information decoding unit) decodes the characteristic SEI indicating the characteristic of refinement processing from the coded data. The refinement information decoding unit decodes the information of the refinement application target (refinement target information). Furthermore, the graph decoding unit 302 decodes the application SEI from the coded data.

[0050] The atlas decoding unit 302 decodes the identifier atlasID indicating the object graph information indicating the refined application object from the V3C unit including the application SEI.

[0051] The occupancy map decoding unit 303 decodes the occupancy data encoded by VVC, HEVC, etc., and outputs the occupancy frame DecOccFrames[frameIdx][compIdx][y][x]. Here, DecOccFrames, frameIdx, compIdx, y, and x represent the decoded occupancy frame, frame ID, component ID, row index, and column index, respectively. In DecOccFrames, compIdx can be 0.

[0052] The geometry decoding unit 304 decodes the geometry data encoded by AVC, VVC, HEVC, etc., and outputs the geometry frame DecGeoFrames[mapIdx][frameIdx][compIdx][y][x]. Here, DecGeoFrames, frameIdx, mapIdx, compIdx, y, and x represent the decoded geometry frame, frame ID, map ID, component ID, row index, and column index, respectively. DecGeoBitDepth, DecGeoHeight, DecGeoWidth, and DecGeoChromaFormat are marked as the bit depth of the geometry frame, the height of the geometry frame, the width of the geometry frame, and the chroma format of the geometry frame, respectively. The decoded geometry frame sometimes includes multiple geometry maps (geometry frames with different depth projections), and mapIdx is used to distinguish the maps. In DecGeoFrames, compIdx can be 0.

[0053] The attribute decoding unit 305 decodes the attribute data encoded by VVC, HEVC, etc., and outputs the attribute frame DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x]. Here, DecAttrFrames, frameIdx, attrIdx, partIdx, mapIdx, compIdx, y, and x represent the decoded attribute frame, frame ID, attribute ID, partition ID, map ID, component ID, row index, and column index, respectively. DecAttrBitDepth, DecAttrHeight, DecAttrWidth, and DecAttrChromaFormat represent the attribute frame, the bit depth of the attribute frame, the height of the attribute frame, the width of the attribute frame, and the chroma format of the attribute frame. The decoded attribute frame sometimes includes multiple attribute maps (attribute frames of different depth projections), and mapIdx is used to distinguish the maps. The decoded attribute frame consists of multiple attributes such as color (R, G, B), reflection, α, and normal direction. Moreover, multiple attributes can be transmitted through multiple attribute data, and attrIdx is used to distinguish them. For example, {R, G, B} is attribute data 0 (attrIdx=0), {reflection} is attribute data 1 (attrIdx=1), and {α} is attribute data 2 (attrIdx=2). Attributes can be split into multiple bit streams for transmission, and partIdx is used to distinguish them. mapIdx is as described above.

[0054] The post-decoding conversion unit 308 receives the decoded atlas information, the decoded occupancy frames DecOccFrames, the decoded geometry frames DecGeoFrames and the decoded attribute frames DecAttrFrames, and converts them into nominal formats. It outputs the nominal formats of occupancy frames, geometry frames and attribute frames OccFramesNF[frameIdx][CompTimeIdx][y][x], GeoFramesNF[mapIdx][CompTimeIdx][frameIdx][y][x], AttrFramesNF[attrIdx][mapIdx][CompTimeIdx][compIdx][y][x]. Here, frameIdx, CompTimeIdx, y, x, mapIdx, attrIdx and compIdx represent the frame ID, the index of synthesis time, the row index, the column index, the map ID, the attribute ID and the component ID respectively.

[0055] The nominal format is a general term for the nominal bit depth, resolution, chroma format, and composition time index of the decoded image to be converted.Each image sub-bitstream and each region of the padded image sub-bitstream is associated with a nominal bit depth. This is the target bit depth envisioned in all operations used for reconstruction. The nominal bit depth OccBitDepthNF of the occupancy map bit depth is set to oi_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_occupancy_2d_bit_depth_minus1[ConvAtlasID]+1. oi_occupancy_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the occupancy frame of the atlas with atlasID=j to be converted. pin_occupancy_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the decoded region of the atlas with atlasID=j that includes occupancy map data. When pin_occupancy_present_flag[j] is equal to 0, it indicates that the padded image frame does not include the atlas with atlasID=j and has an area of ​​occupancy map data. When pin_occupancy_present_flag[j] is equal to 1, it indicates that there is an area of ​​the filled image frame of the atlas with atlasID=j with occupancy map data. When pin_occupancy_present_flag[j] is not present, its value is presumed to be equal to 0. The nominal bit depth GeoBitDepthNF of each geometry frame is set to gi_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 or pin_geometry_2d_bit_depth_minus1[ConvAtlasID]+1 when pin_geometry_present_flag[ConvAtlasID] is equal to 1. gi_geometry_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of all geometry frames of the atlas with converted atlasID=j. pin_geometry_2d_bit_depth_minus1[j]+1 represents the nominal 2D bit depth of the decoded area of ​​the atlas with converted atlasID=j that includes geometry data. pin_geometry_present_flag[j]=0 indicates that the filled image frame of the atlas with atlasID=j does not include an area with geometric data. pin_geometry_present_flag[j]=1 indicates that the filled image frame of the atlas with atlasID=j contains an area with geometric data.In the absence of pin_geometry_present_flag[j], its value is assumed to be 0. Finally, in the case of pin_attribute_present_flag[ConvAtlasID]=1, the nominal bit depth AttrBitDepthNF[attrIdx] of each attribute frame with attrIdx is set to ai_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]+1 or pin_attribute_2d_bit_depth_minus1[ConvAtlasID][attrIdx]. Increasing ai_attribute_2d_bit_depth_minus1[j][i] by 1 indicates that the nominal 2D bit depth of all attribute frames with attrIdx=i for atlasID=j is converted. Pin_attribute_2d_bit_depth_minus1[j][i] plus 1 indicates the nominal 2D bit depth of the region of the atlas conversion including the attribute of attrIdx=i for atlasID=j. pin_attribute_present_flag[j]=0 indicates that the filled image frame of the atlas of atlasID=j does not include the region of attribute data. pin_attribute_present_flag[j]=1 indicates that the filled image frame of the atlas of atlasID=j includes the region of attribute data.

[0056] When ConvAtlasID is set equal to vuh_atlas_id or the V3C unit header cannot be used, it is determined by external means. vuh_atlas_id is represented by the V3C unit header such as V3C_AD, V3C_OVD, V3C_GVD, V3C_AVD, etc., and specifies the ID of the atlas corresponding to the current V3C unit.

[0057] For asps_frame_width, the frame width of the atlas is represented by an integer number of samples, corresponding to the luma samples of the image component. For asps_frame_width, it is a necessary condition for V3C bitstream consistency to be equal to the value of vps_frame_width[j] (j is the current atlasID). For asps_frame_height, the frame height of the atlas is represented by an integer number of samples, corresponding to the luma samples of the image component. It is a necessary condition for V3C bitstream consistency to have the value of asps_frame_height equal to the value of vps_frame_height[j], where j is the current atlasID. The nominal frame resolution of the auxiliary image component is defined by the nominal width and height specified by the variables AuxVideoWidthNF and AuxVideoHeightNF, respectively. AuxVideoWidthNF and AuxVideoHeightNF are derived from the auxiliary image sub-bitstream associated with the atlas.

[0058] The nominal chroma format is defined as 4:4:4.

[0059] The post-decoding conversion unit 308 is composed of bit depth conversion, resolution conversion, output order conversion, atlas composition alignment, atlas dimension alignment, chroma upsampling, geometric map synthesis, and attribute map synthesis. The image frames provided by the V3C decoding unit 309 sometimes require additional processing steps before entering the reconstruction process. Such processing steps may include conversion to the nominal format of the decoded frame (e.g., resolution, bit depth, chroma format, etc.). The information of the nominal format is notified by signaling in the V3C VPS.

[0060] The pre-reconstruction unit 310 can receive decoded atlas information, decoded occupancy frames, decoded geometry frames, and decoded attribute frames, and refine / correct them. Specifically, when the occupancy map synthesis flag os_method_type[k] is equal to 1 indicating patch border filtering, OccFramesNF[compTimeIdx][0] and GeoFramesNF[0][compTimeIdx][0] are input, and the corrected array OccFramesNF[compTimeIdx][0] is used as output to start the occupancy map synthesis. OccFramesNF represents an occupancy frame decoded in a nominal format, and GeoFramesNF represents a geometry frame decoded in a nominal format. compTimeIdx is an index of the synthesis time.

[0061] The refinement unit 306 refines the occupancy map, geometry, and attributes in units of frames within the pre-reconstruction unit 310. The refinement may be filtering of any one of the input occupancy map, geometry, and attributes and any one of the output occupancy map, geometry, and attributes. The refinement unit 306 may also use two-dimensional occupancy maps, geometry, and attributes simultaneously for refinement.

[0062] Furthermore, the refinement unit 306 includes an NN filter unit 611 or an ALF unit 610, and can perform filtering processing based on the transmitted neural network parameters or linear parameters.

[0063] The reconstruction unit 311 inputs the atlas information, occupancy map, geometry, and reconstructs the position, attributes or mesh of the vertices or point groups in the 3D space. The reconstruction unit 311 reconstructs the point cloud data or mesh of the 3D data based on the reconstructed geometry information (e.g., recPcGeo) and attribute information (e.g., recPcAttr). Specifically, the reconstruction unit 311 inputs the nominal image frame derived by the pre-reconstruction unit 310, OccFramesNF[compTimeIdx][0][y][x], GeoFramesNF[mapIdx][compTimeIdx][0][y][x] and AttrFramesNF[attrIdx][mapIdx][compTimeIdx][ch][y][x] to reconstruct the point cloud data and mesh of the 3D data. AttrFramesNF represents the attribute frame decoded by the nominal format. The reconstruction unit 311 derives the variable pointCnt as the number of points in the reconstructed point group frame, derives the one-dimensional array pointToPatch[pointCnt] as the patch index corresponding to each reconstructed point, and derives the two-dimensional array pointToPixel[pointCnt][dimIdx] as the atlas coordinates corresponding to each reconstructed point. There is a 2D array recPcGeo[pointCnt][dimIdx] as a list of coordinates corresponding to each reconstructed point, and there is a 3D array recPcAttr[pointCnt][attrIdx][compIdx] as the attributes associated with the points in the reconstructed point group frame. Here, pointCnt, attrIdx, and compIdx correspond to the size of the reconstructed point, the ID of the attribute frame, and the component ID, respectively. dimIdx represents the (x, y) components of each reconstructed point. Specifically, the reconstruction unit 311 derives recPcGeo and recPcAttr in the following manner. Here, pIdx is the index of the patch. compTimeIdx represents the composition time index, rawPos1D represents the one-dimensional position, gFrame and aFrame represent the geometry frame and attribute frame respectively. TilePatch3dOffsetU is the tile patch parameter associated with the patch. ai_attribute_count[j] represents the number of attributes associated with the atlas with atlasID=j (the number of attributes encoded and decoded by VPS). AtlasPatch3dOffsetU, AtlasPatch3dOffsetV, AtlasPatch3dOffsetD represent Figure 3 (a) Parameters of the location of the 3D bounding box on the 3D model.

[0064] Here, AtlasPatchRawPoints, AtlasPatch2dPosX, AtlasPatch2dPosY, AtlasPatch2dSizeX, and AtlasPatch2dSizeY are patch information derived from the atlas information by the atlas decoding unit 302. The arrays oFrame[y][x], gFrame[mapIdx][y][x], aFrame[mapIdx][attrIdx][compTimeIdx][y][x] are derived as follows: Here, ai_attribute_dimension_minus1 indicates that encoding and decoding are performed in the encoded data of V3C VPS, and ai_attribute_dimension_minus1+1 indicates the total dimension (i.e., the number of channels and components) of the attribute to be decoded in the V3C unit decoding unit 301. asps_map_count_minus1 is encoded and decoded in the encoded data of V3C VPS, and sps_map_count_minus1+1 indicates the number of maps of the geometric data and attribute data of the current atlas. RecAtlasID indicates the ID of the decoded atlas.

[0065] The post-reconstruction unit 312 changes (updates) the 3D data point group data or mesh processed by the reconstruction unit 311. The post-reconstruction unit 312 receives pointCnt as the number of reconstructed points of the current point group frame associated with the current atlas, a one-dimensional array attrBitDepth[] as the nominal bit depth, oFrame[][], recPcGeo[][], recPcAttr[][][], applies geometric smoothing, and outputs the changed recPcGeo[][], recPcAttr[][][]. (Detailing section 306)

[0066] Figure 6 It is a functional block diagram showing the structure of the refinement unit 306 (306DEC).

[0067] The refinement unit 306 (306DEC) decodes through the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305, and applies refinement to the DecOccFrames, DecGeoFrames, and DecAttrFrames that have not been converted by the post-decoding conversion unit 308. Then, the refined DecOccFrames, DecGeoFrames, and DecAttrFrames are output to the post-decoding conversion unit 308.

[0068] The refinement unit 306 inputs DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x] of each mapIdx and frameIdx, and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] of each attrIdx, partIdx, mapIdx, frameIdx, and compIdx, and outputs the corrected DecOccFrames, DecGeoFrames, and DecAttrFrames. The refinement unit 306 stores the output in DecOccFrames[0][frameIdx][y][x], DecGeoFrames[mapIdx][frameIdx][0][y][x], and DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx][compIdx][y][x] according to the refinement target information.

[0069] The refinement unit 306 may also refine the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1)×asps_frame_height×asps_frame_width. In this case, oFrame=DecOccFrames[0][frameIdx], gFrame=DecGeoFrames[mapIdx][frameIdx][0], and aFrame=DecAttrFrames[attrIdx][partIdx][mapIdx][frameIdx] may be input. Here, attrIdx and mapIdx may be the values ​​nnrc_attribute_index and nnrc_map_index obtained by decoding the NNRC SEI. attrIdx=nnrc_attribute_index, mapIdx=nnra_target_map_index. frameIdx may be compTimeIdx which is an index of composition time. Furthermore, attrIdx and mapIdx may be values ​​nnra_target_map_index and nnra_target_attribute_index obtained by decoding the NNRA SEI. attrIdx=nnra_target_attribute_index, mapIdx=nnrc_map_index. The same applies to the following.

[0070] According to the above configuration, the network model to be applied (filtering characteristics) is specified by the characteristic SEI, and the refinement object information specified by the characteristic SEI is used to specify any one of the occupancy map, geometry, and attributes, and refinement processing is applied to multiple inputs simultaneously, thereby achieving the effect of improving image quality.

[0071] Figure 7 An example is shown in which the thinning section 306 applies thinning to an image signal (video signal) in a nominal format.

[0072] Figure 8306NF). The refinement unit 306 (306NF) applies refinement to the nominal formats OccFramesNF, GeoFramesNF, and AttrFramesNF converted by the post-decoding conversion unit 308, respectively, based on the refinement target information, and outputs the GeoFramesNF / AttrFramesNF after the refinement is applied to the pre-reconstruction unit 310. The processing of the refinement unit 306 is described below, but the processing of the refinement unit 306DEC replaces OccFramesNF, GeoFramesNF, and AttrFramesNF with Figure 6 The DecOccFrames, DecGeoFrames, and DecAttrFrames are used for processing.

[0073] The refinement unit 306 may also refine the two-dimensional array oFrame of asps_frame_height×asps_frame_width, the two-dimensional array gFrame of asps_frame_height×asps_frame_width, and the three-dimensional array aFrame of (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1)×asps_frame_height×asps_frame_width. In this case, oFrame=OccFramesNF[compTimeIdx][0], gFrame=GeoFramesNF[mapIdx][compTimeIdx][frameIdx], and aFrame=AttrFramesNF[attrIdx][mapIdx][compTimeIdx].

[0074] exist Fig.10 In (a), when the refinement object information indicates an occupancy map (S3062I), the occupancy map is added to the input tensor (S3063I). For example, the following (Eq. IN-OCC) can be used.

[0075] When the refinement object information indicates geometry ( S3064I ), the geometry is added to the input tensor ( S3065I ). For example, the following (Eq. IN-GEO) can be used.

[0076] When the refinement object information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, the following (Eq. IN-ATTR) can be used.

[0077] According to the above configuration, the network model (filter characteristics) to be applied is specified by the characteristic SEI. The refinement target information specified by the characteristic SEI is used to specify any one of the occupancy map, geometry, and attribute, and refinement processing is applied to multiple inputs simultaneously, thereby achieving an effect of improving image quality. (NN filter unit 611)

[0078] The NN filter unit 611 performs filtering processing using a neural network. The neural network is expressed by a neural network model and may include convolution (Conv).

[0079] The neural network model (hereinafter referred to as NN model) refers to the elements and combination relationship (topology) of the neural network, and the parameters (weight, bias) of the neural network. It should be noted that the NN filter unit 611 can fix the topology and only switch the parameters according to the filter object image. In addition, the neural network can also have a kernel size, a number of input channels, and a convolution determined by the number of output channels.

[0080] The input to the refinement unit 306 is set to DecFrame. The refinement unit 306 derives the input InputTensor to the NN filter unit 611 from the input image DecFrame. The NN filter unit 611 uses the inputTensor to perform filtering based on the neural network model and derives the outputTensor. The neural network model used is the model corresponding to nnra_target_id. The input image can be an image of each component or an image having multiple components as channels.

[0081] The NN filter unit 611 may repeatedly apply the following processing.

[0082] The NN filter unit 611 performs convolution operation (conv, convolution) on the inputTensor and the kernel k[m][n][yy][xx] according to the number of layers to generate an output image outputTensor with a bias added. Here, m represents the number of channels of inputTensor, n represents the number of channels of outputTensor, yy represents the height of kernel k, and xx represents the width of kernel k.

[0083] In each layer, outputTensor is generated from inputTensor. outputTensor[nn][yy][xx]=ΣΣΣ (k[mm][nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn])

[0084] Here, nn=0..n-1,mm=0..m-1,yy=0..height-1,xx=0..width-1,i=0..yy-1,j=0..xx-1。width is the width of inputTensor and outputTensor,height is the height of inputTensor and outputTensor。Σ is the sum of mm=0..m-1,i=0……yy-1,j=0..xx-1。of is the width or height of the area around inputTensor required to generate outputTensor。

[0085] In the case of 1x1 Conv, Σ represents the sum of mm=0..m-1, i=0, j=0. In this case, set of=0. In the case of 3×3 Conv, Σ represents the sum of mm=0...m-1, i=0...2, j=0...2. In this case, set of=1. When the value of yy+j-of is less than 0 or greater than height, or when the value of xx+i-of is less than 0 or greater than width, the value of inputTensor[mm][yy+j-of][xx+i-of] can be set to 0. Alternatively, the value of inputTensor[mm][yy+j-of][xx+i-of] can be set to inputTensor[mm][yclip][xclip]. Here, yclip is max(0, min(yy+j-of, height-1)), and xclip is (0, min(xx+i-of, width-1)).

[0086] In the next layer, the obtained outputTensor is used as the new inputTensor, and the same process is repeated according to the number of layers. You can also set an activation layer between layers. In addition, you can also use pooling layers and jump combinations. Finally, derive the FilteredFrame from the obtained outputTensor.

[0087] Alternatively, a process called Depth wise Conv represented by the following equation may be performed using a kernel k[n][yy][xx]: Here, nn=0..n-1, xx=0..width-1, and yy=0..height-1. outputTensor[nn][yy][xx]=ΣΣ (k[nn][i][j]*inputTensor[mm][yy+j-of][xx+i-of]+bias[nn])

[0088] In addition, nonlinear processing called activation can also be used, such as ReLU. ReLU(x)=x>=0? x:0 Alternatively, leakyReLU shown in the following formula may be used. leakyReLU(x)=x>=0? x:a*x

[0089] Here, a is a predetermined value less than 1, such as 0.1 or 0.125. In addition, in order to perform integer operations, all the above-mentioned values ​​of k, bias, and a may be set to integers, and right shifted after conv to generate outputTensor.

[0090] In ReLU, 0 is always output for values ​​less than 0, and the input value is kept unchanged for values ​​above 0. On the other hand, in leakyReLU, linear processing is performed for values ​​less than 0 with the gradient set by a. In ReLU, the gradient for values ​​less than 0 disappears, so sometimes learning is difficult to progress. In leakyReLU, the gradient for values ​​less than 0 is retained, which is less likely to cause the above problem. In addition, PReLU can also be used by parameterizing the value of a in the above leakyReLU(x). (Filtering Processing by NN Filter Unit 611)

[0091] Set nnrc_tareget_attribute_index to attrIndex, set nnra_tareget_map_index[i] to mapIndex, and apply the following. Call the following process with OccFramesNF[compTimeIdx][0], GeoFramesNF[mapIdx][compTimeIdx][0], AttrFramesNF[attrIdx][mapIdx][compTimeIdx], and nnra_strength[i] as inputs and the modified GeoFramesNF[mapIdx][compTimeIdx][0] and AttrFramesNF[attrIdx][mapIdx][compTimeIdx] as outputs. The inputs of this process include variable nnrc_occupancy_flag, variable nnrc_geometry_flag, variable nnrc_attribute_flag, variable nnrc_strength_num, 2D array oFrame, 2D array gFrame, 3D array aFrame, and 1D array nnraStrength. The variable nnrc_occupancy_flag, defined by the feature SEI, indicates whether occupancy frames are used. The variable nnrc_geometry_flag, specified by the feature SEI, indicates whether the geometry frame is used. The variable nnrc_attribute_flag is defined by the feature SEI and indicates whether to use the attribute frame. The variable nnrc_strength_num, specified by the SEI property, indicates the amount of strength information used. 2D array oFrame, size is asps_frame_height × asps_frame_width. 2D array gFrame, size is asps_frame_height × asps_frame_width. 3D array aFrame, size is (ai_attribute_dimension_minus1[RecAtlasID][attrIdx]+1)× asps_frame_height×asps_frame_width. 1D array nnraStrength, size nnrc_strength_num. nnraStrength is a parameter representing the strength of refinement decoded from the encoded data of the activation information (nnra_Strength[]) and input to the input tensor. The output of this process is the changed arrays gFrame and aFrame.

[0092] Here, ai_attribute_dimension_minus1 indicates the number of elements - 1. This element is an element of an attribute to be encoded or decoded by VPS. In the case of YUV or RGB, it is usually 3.

[0093] In addition, in a configuration that supports simultaneous refinement of a plurality of attributes, nnra_tareget_map_index[i] may be set for mapIndex, and the following content may be applied. Call the following process with OccFramesNF[compTimeIdx][0], GeoFramesNF[mapIdx][compTimeIdx][0], AttrFramesNF[attrIdx][mapIdx][compTimeIdx], and nnra_strength[i] as inputs and the changed GeoFramesNF[mapIdx][compTimeIdx][0] and AttrFramesNF as outputs. The inputs to this process include variables OccupancyUsedFlag, GeometryUsedFlag, AttributeUsedFlag, variable nnrc_strength_num, 2D array gFrame, 5D array aFrame, and 1D array nnraStrength. The variable OccupancyUsedFlag (nnrc_occupancy_flag), defined by the feature SEI, indicates whether the occupancy frame is used. The variable GeometryUsedFlag (nnrc_geometry_flag), specified by the feature SEI, indicates whether the geometry frame is used. The variable AttributeUsedFlag (nnrc_attribute_flag) is specified by the characteristic SEI and indicates whether the attribute frame is used. The variable nnrc_strength_num, specified by the SEI property, indicates the amount of strength information used. 2D array oFrame, size is asps_frame_height × asps_frame_width. 2D array gFrame, size is asps_frame_height × asps_frame_width. 5D array aFrame, the size is ai_attribute_count[RecAtlasID]×ai_attribute_dimension_minus1[RecAtlasID][attrIdx]×asps_frame_height×asps_frame_width. 1D array nnraStrength, size nnrc_strength_num. Specified by the SEI feature, represents the strength information that controls the strength of the refinement process. The output of this process is the changed arrays gFrame and aFrame.

[0094] The NN filter unit 611 performs NN filtering processing to derive an outputTensor from an inputTensor. A refinement process (filtering process) shown by RefineFilter() may be performed in units of blocks (blockWidth x blockHeight) as described below.

[0095] Here, DeriveInputTensors() is a function indicating input data setting, and StoreOutputTensors() is a function indicating output data storage. FrameWidth and FrameHeight are the size of input data, which can be asps_frame_width and asps_frame_height. BlockWidth and blockHeight are the width and height of a block.

[0096] In DeriveInputTensors(), the NN filter unit 611 derives input data inputTensor[][][] to the NN filter unit 611 according to the refined object information and based on one or more of the occupancy frame oFrame, the geometry frame gFrame, and the attribute frame aFrame. The following is an example of a configuration, and the configuration described later may also be used.

[0097] Here, OccupancyUsedFlag, GeometryUsedFlag, and AttributeUsedFlag are variables indicating refinement target information, and syntax elements described later obtained by decoding the attribute SEI can be used as follows. OccupancyUsedFlag=nnrc_occupancy_flag GeometryUsedFlag=nnrc_geometry_flag AttributeUsedFlag=nnrc_attribute_flag <Specification of input tensor and output tensor corresponding to refinement target information>

[0098] The other configurations described above will be further described.

[0099] Fig.10 : is a flowchart showing the operation of the refinement unit 306 .

[0100] for Fig.10The example shown is further described in detail.

[0101] exist Fig.10 In (a), the refinement unit 306 or the atlas decoding unit 302 decodes the refinement target information (S3061).

[0102] The refinement unit 306 can derive the inputTensor according to the refinement target information. Hereinafter, ch represents the index (position) of the channel of the setting target. ch++ is an abbreviation of ch=ch+1, which represents the index of the channel of the setting target +1. ch=0

[0103] When the refinement target information indicates an occupancy map ( S3062I ), the occupancy map is added to the input tensor ( S3063I ). For example, the following (formula IN-OCC) may be used.

[0104] When the refinement target information indicates geometry (S3064I), the geometry is added to the input tensor (S3065I). For example, the following (formula IN-GEO) may be used.

[0105] When the refinement object information indicates an attribute (S3066I), the attribute is added to the input tensor (S3067I). For example, as shown in the following (formula IN-ATTR-DIM), all the channels of the attribute specified by the index of attrIdx are input to inputTensor.

[0106] Note the following. InpSampleVal(y, x, FrameHeight, FrameWidth, targetPic) { As another configuration, when the refinement target information indicates an attribute, all the channels of the attribute specified by the index of attrIdx are input to inputTensor. The attrIdx starts from nnrc_attribute_index, and the number of attributes is nnrc_attribute_num_minus1+1.

[0107] exist Fig.10In (b), the refinement unit 306 may derive (update) the output frame from the outputTensor according to the refinement object information in DeriveInputTensors.

[0108] When the refinement target information indicates geometry (S3064O), a specific component of the output tensor is set for the geometry (S3065O). For example, the following (formula OUT-GEO) may be used.

[0109] When the refined object information represents an attribute (S3066O), a specific component of the output tensor is set for the attribute (S3067O). For example, as shown in the following (formula OUT-ATTR-DIM), the value of outputTensor may be set to the number of channels of the attribute specified by the index of attrIdx.

[0110] As another configuration, when the refined object information indicates an attribute, all the channels of the attribute specified by the index of attrIdx are input to inputTensor. The attrIdx starts from nnrc_attribute_index, and the number of attributes is nnrc_attribute_num_minus1+1.

[0111] The refinement information decoding unit is characterized in that the refinement information decoding unit further decodes the number strengthNum of strength information from the characteristic information. The refinement processing unit inputs at least one of the occupied frame, the geometric frame or the attribute frame to the input tensor of the neural network, and then inputs the strengthNum values ​​from the decoded strength information into the strengthNum channels of the input tensor of the neural network. Instead of Inp(oFrame[yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, oFrame)) in which screen boundary processing is added may be used. Instead of Inp(gFrame[yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, gFrame)) in which screen boundary processing is added may be used. Instead of Inp(aFrame[k][yT][xT]), Inp(InSampleVal(yT, xT, FrameHeight, FrameWidth, aFrame[k])) with screen boundary processing added may be used. k=0, 1, 2, ..., etc. The same applies to the rest. Here, overlapSize is the size of the overlap, and a value decoded from the encoded data of the characteristic SEI can be used.

[0112] Next, in StoreOutputTensors(), the NN filter unit 611 sets the content of outputTensor to the changed occupied frame oFrame, geometric frame gFrame, and attribute frame aFrame according to the refinement target information.

[0113] (ALF Department 610)

[0114] It should be noted that the refinement specified by the persistence information of the NNRA SEI and the identifier of the characteristic SEI can be performed using a filter (Wiener Filter) using a linear model. Linear filtering can be performed by the ALF unit 610. Specifically, the filtered object image DecFrame is divided into small areas (x=xSb..xSb+bSW-1, y=xSb..xSb+bSH-1) of a certain size (for example, 4x4, 1x1). (xSb, ySb) is the upper left coordinate of the small area, and bSW and bSH are the width and height of the small area. Then, filtering is performed in units of small areas. The selected filter coefficient coeff[] is used to derive the refined image outFrame from DecFrame. outframe[cIdx][y][x]=Σ (coeff[i]*DecFrame[cIdx][y+ofy][x+ofx]+offset)>>shift

[0115] Here, ofx and ofy are offsets of the reference position determined according to the filter position i. offset = 1<<(shift-1). shift corresponds to the accuracy of the filter coefficient and is a constant such as 6, 7, or 8.

[0116] It should be noted that the image may also be classified in small regions, and based on the derived classId (=classId[y][x]) of the small region, the filter coefficient coeff[classId][] is selected to perform filtering processing. outframe[cIdx][y][x]=(Σ(coeff[classId][i]*DecFrame[cIdx][y+ofy][x+ofx]+offset)>>shift

[0117] Here, DecFrame[ch]=inputTensor[ch][yP+overlapSize][xP+overlapSize], that is, the following may be used.

[0118] It should be noted that classId can be derived using the activity and directionality of a block (pixel in the case of 1x1). For example, it can also be derived using the activity Act derived from the following difference absolute value sum. classId=Act

[0119] It should be noted that Act can also be derived as follows. Act=Σ|x-xi|

[0120] Here, i=0..7, xi represents the neighboring pixels of the target pixel x. The directions of the neighboring pixels can be eight directions obtained by adding four directions at 45 degrees to the up, down, left, and right directions relative to i=0..7. In addition, You can also use the following formula: Act = Σ|-xl+2*x-xr|+Σ|-ya+2*x-yb. It should be noted that l, r, a, b are opposite to left, right, above, and below, respectively, and represent the pixels to the left, right, top, and bottom of x. It should be noted that You can use Act=Min(NumA-1, Act>>shift) The shift value is quantized based on the clipping into NumA-1, where NumA represents the number of active indicators. Furthermore, the classId can be derived using the following formula using the directionality D. classId=Act+D*NumA

[0121] Here, for example, Act=0..NumA-1, D=0..4, etc.

[0122] The ALF unit 610 may directly output OutFrame as FilteredFrame. (Decoding and filtering application with SEI)

[0123] Fig. 93D data decoding apparatus performs the following processing including decoding of an application SEI message.

[0124] S6001: The map decoding unit 302 decodes nra_cancel_flag and nnra_target_id from the application SEI. nnra_cancel_flag may also be nnrc_cancel_flag.

[0125] S6002: When nnra_cancel_flag is 1, the processing of the frame that is the target of nnra_cancel_flag is terminated. When nnra_cancel_flag is 0, the process proceeds to S6003.

[0126] S6003: The map decoding unit 302 decodes nnra_persistence_flag from the application SEI.

[0127] S6005: Determine a characteristic SEI having the same nnrc_id as nnra_target_id, and derive parameters of the NN model from the characteristic SEI.

[0128] S6006: Using the derived parameters of the NN model, refinement (filtering) processing is performed through the NN filter unit 611 and the ALF unit 610.

[0129] <Grammar Structure Example> (Neural Network Refinement Characteristics SEI)

[0130] Fig.11 , Fig.12 The syntax of nn_refinement_characteristics(payloadSize) (Characteristics SEI) is shown. The argument payload size (payloadSize) indicates the number of bytes of this SEI message.

[0131] The persistence scope of the application characteristic SEI is CAS. That is, SEI is applied in units of CAS.

[0132] The syntax element nnrc_id indicates the ID of the characteristic SEI. In the application SEI, by transmitting the value of nnrc_id of the characteristic SEI indicating the refinement characteristic to be applied as the object ID (nnra_target_id), the refinement processing to be applied is executed. When nnrc_mode_idc is 0, the SEI message includes the ISO / IEC15938-17 bitstream, specifies the base NNRE (neural-network refinement) or indicates that it is an update of the base NNRE from the same nnrc_id value. The NNRC SEI message has a specific nnrc_id value in the current CAS. In the case of the first NNRCSEI message in the decoding order, if nnrc_mode_idc is equal to 1, it is determined that the basic NNRE associated with the nnrc_id value is in the format identified by the nnrc_tag_uri of the tag URI, which is the neural network identified by the URI indicated by nnrc_uri. An NNRC SEI message has a specific nnrc_id value within the current CAS, which, when not the first NNRC SEI message in decoding order, nor a duplicate of the first NNRC SEI message in decoding order, and nnrc_mode_idc is equal to 1, indicates an update to the base NNRE with the same nnrc_id value defined by the URI indicated by nnrc_uri. nnrc_uri is in the format identified by nnrc_tag_uri of the tag URI. The value of nnrc_mode_idc in a bitstream according to this document can be in the range of 0 to 1 (inclusive). This SEI message has a specific nnrc_id value within the current CAS, and in the case of the first NNRCSEI message in decoding order, RefineFilter() is assigned the same value as the base NNRE. This SEI message has a specific nnrc_id value within the current CAS. If it is not the first NNRC SEI message in decoding order, nor a repetition of the first NNRC SEI message in decoding order, the updates defined by this SEI message are applied to the base NNRE to obtain RefineFilter(). Updates are not cumulative, each update is applied to the base NNRE. The base NNRE is the NNRE specified by the first NNRC SEI message (decoding order) with a specific nnrc_id value in the current CAS.

[0133] nnrc_reserved_zero_bit_a may be equal to 0 in a bitstream according to this document. A decoder may ignore NNRC SEI messages for which nnrc_reserved_zero_bit_a is not 0.

[0134] nnrc_tag_uri includes a tag URI having the construction and semantics specified in IETF RFC4151, which identifies the format and associated information of the base NNRE with the same nnrc_id value specified by nnrc_uri or a neural network used as an update for the base NNRE. nnrc_tag_uri equals "tag:iso.org,2023:15938-17" indicating that the neural network data identified by nnrc_uri is based on ISO / IEC15938-17.

[0135] nnrc_uri comprises a URI having the construction and semantics specified in IETF Internet Standard 66 that identifies a neural network used as a base NNRE or an update to a base NNRE having the same nnrc_id value.

[0136] When nnrc_property_present_flag is equal to 1, it indicates that the input format, output format, and context elements associated with complexity exist. When nnrc_property_present_flag is 0, it indicates that the input format, output format, and context elements associated with complexity do not exist. When this SEI message has a specific nnrc_id value in the current CAS and is the first NNRC SEI message in decoding order, nnrc_property_present_flag is equal to 1. When nnrc_property_present_flag is equal to 0, the values ​​of all context elements that may only exist when nnrc_property_present_flag is equal to 1 are inferred to be the same as their respective corresponding context elements in the NNRC SEI message in which the SEI includes the base NNRE to be updated.

[0137] nnrc_base_flag equal to 1 indicates that the SEI message specifies a base NNRE. nnrc_base_flag equal to 0 indicates that the SEI message specifies updates associated with a base NNRE. In its absence, the value of nnrc_base_flag is inferred to be equal to 0. The following restrictions apply to the value of nnrc_base_flag: - The NNRC SEI message is the first NNRC in decoding order with a specific nnrc_id value in the current CAS In the case of SEI message, the value of nnrc_base_flag is equal to 1. - In case that NNRC SEI message nnrcB is not the first NNRC SEI message in decoding order with a specific nnrc_id value within the current CAS, and the value of nnrc_base_flag is equal to 1, the NNRC SEI message is a repetition of the first NNRC SEI message nnrcA with the same nnrc_id in decoding order. That is, the payload content of the next nnrcB may be the same as the payload content of nnrcA. When the NNRC SEI message is not the first NNRC SEI message in decoding order with a specific nnrc_id within the current CAS, nor is it a repetition of the first NNRC SEI message with the specific nnrc_id, the following applies: - This SEI message defines updates to existing base NNREs in decoding order with the same nnrc_id value.

[0138] - This SEI message is associated with the current decoded picture and all decoded pictures of the layers following it in output order, until the picture at the end of the current CAS or the subsequent pictures in the decoding order following the current decoded picture in the current CAS in output order, except for the decoded pictures, and having a specific nnrc_id value in the current CAS The NNRC SEI message establishes the front picture among the associated pictures.

[0139] The syntax elements nnrc_occupancy_flag, nnrc_geometry_flag, and nnrc_attribute_flag indicate the target of refinement processing (refinement target information), and respectively indicate whether any of occupancy, geometry, and attribute is included in the refinement input or output.

[0140] The syntax element nnrc_attribute_index is refinement object information, indicating the index (attrIdx) of the attribute of the refinement object. When nnrc_attribute_flag! = 0, nnrc_attribute_index is encoded and decoded. In addition, although not shown in the figure, the syntax element of nnra_all_attribute_flag can also be decoded and encoded. Here, nnra_all_attribute_flag = 1 means that all attributes are refined. That is, attrIdx indicating the refinement application object includes all attrIdx of 0..ai_attribute_count[RecAtlasID]-1. When nnra_all_attribute_flag = 1, nnrc_attribute_index and nnrc_attribute_num_minus1 are not encoded and decoded, and it is inferred (infer) that nnrc_attribute_index = 0 and nnrc_attribute_num_minus1 = ai_attribute_count[RecAtlasID]-1. In addition, the list of nnrc_attribute_index may be inferred to be 0..ai_attribute_count[RecAtlasID] - 1. When nnra_all_attribute_flag = 0, nnrc_attribute_index and nnrc_attribute_num_minus1 are encoded and decoded.

[0141] The syntax element nnrc_attribute_num_minus1 is refinement target information, and indicates the number of refinement target attributes. When nnrc_attribute_flag!=0, nnrc_attribute_num_minus1 is encoded and decoded.

[0142] The syntax element nnrc_strength_num is refinement input additional information, and indicates the number strengthNum of strength information nnra_strength input to the refinement process (inputTensor).

[0143] The syntax element nnrc_inp_out_format_idc (input / output tensor format information) indicates a method of converting the pixel values ​​of the decoded image into input / output values ​​for the refinement process. When the value of nnrc_inp_out_format_idc is 0, the input values ​​for the refinement process (particularly the input tensor) are real numbers (in the form of floating point values) specified in IEEE754, and the function Inp is specified as follows. The range of the value of the input tensor is 0..1. Inp(x)=x÷((1< <BitDepth)-1) InpS(x)=x÷63

[0144] When the value of nnrc_inp_out_format_idc is 1, the input / output values ​​to the thinning process are unsigned integers, and the function Inp is specified as follows: The range of the input tensor value is 0..1<<(inpTensorBitDepth)-1. shift=BitDepth-inpTensorBitDepth if(inpTensorBitDepth>=BitDepth) Inp(x)=x<<(inpTensorBitDepth-BitDepth) else Inp(x)=Clip3(0,(1< <inpTensorBitDepth)-1,(x+(1<<(shift-1)))> >shift) InpS(x)=x

[0145] The value obtained by adding 8 to the value of the syntax element nnrc_inp_tensor_bitdepth_minus8 represents the pixel bit depth of the luma pixel value of the input integer tensor. The value of the variable inpTensorBitDepth is derived from the syntax element nnrc_inp_tensor_bitdepth_minus8 as follows. inpTensorBitDepth=nnrc_inp_tensor_bitdepth_minus8+8

[0146] A block is an array of images. Thinning is performed in units of fixed blocks. Blocks are also called patches.

[0147] The syntax element nnrc_block_size_idc indicates the block size. The block size can be 64, 128, 192, or multiples of 64. blockWidth=64*nnrc_block_size_idc(=nnrc_block_size_idc<<6) blockHeight=64*nnrc_block_size_idc(=nnrc_block_size_idc<<6)

[0148] Alternatively, the value may be the value obtained by removing 0 from nnrc_block_size_idc. blockWidth=(nnrc_block_size_idc+1)<<6 blockHeight=(nnrc_block_size_idc+1)<<6

[0149] The syntax element nnrc_overlap_size_idc specifies the number of horizontal and vertical pixels by which adjacent input tensors are overlapped. The value of nnrc_overlap_size_idc can be a multiple of 4 of the size of the overlap. overlapSize=nnrc_overlap_size_idc<<2

[0150] The function Out for converting the brightness pixel value and the color difference pixel value output by the post-processing into integer values ​​of the pixel bit length respectively is specified as follows using the pixel bit length BitDepth. Out(x)=Clip3(0,(1< <BitDepth)-1,Round(x*((1<<BitDepth)-1)))

[0151] The function Out is specified as follows. shift=outTensorBitDepth-BitDepth if(outTensorBitDepth>=BitDepth) Out(x)=Clip3(0,(1< <BitDepth)-1,(x+(1<<(shift-1)))> >shift) else Out(x)=x<<(BitDepth-outTensorBitDepth)

[0152] nnrc_out_tensor_bitdepth_minus8+8 specifies the pixel bit depth of the pixel values ​​of the output integer tensor. The value of outTensorBitDepth is derived from the syntax element nnrc_out_tensor_bitdepth_minus8 in the following way. outTensorBitDepth=nnrc_out_tensor_bitdepth_minus8+8

[0153] The syntax element nnrc_reserved_zero_bit_b may be equal to 0.

[0154] The syntax element nnrc_payload_byte[i] includes the i-th byte of the bit stream according to ISO / IEC 15938-17. nnrc_payload_byte[i] may be a bit stream completely according to ISO / IEC 15938-17. (Neural Network Refinement Application SEI)

[0155] Fig.13 An example of the syntax structure of application SEI is shown.

[0156] In response to the application of SEI, the map decoding unit 302 (detailed information decoding unit) and the map encoding unit 102 (detailed information encoding unit) decode and encode the following syntax elements.

[0157] nnra_target_id indicates the ID of the characteristic SEI to be applied (an identifier of refined characteristic information, identification information). The refinement process specified by the characteristic SEI having the nnrc_id equal to nnra_target_id is applied to the image.

[0158] nnra_cancel_flag is a cancellation flag. When nnra_cancel_flag is 1, it indicates that the persistence of the refinement set for the picture by the decoded NNRASEI is canceled. When nnra_cancel_flag is 0, it indicates that the subsequent syntax element (nnra_persistence_flag) is transmitted / encoded / decoded.

[0159] nnra_persistence_flag indicates persistence information of object refinement. When nnra_persistence_flag is 0, it indicates that the refinement of the object is only applied to the picture indicated by atlasID. When nnra_persistence_flag is 1, it indicates that the refinement of the object indicated by nnra_target_id is applied to the current picture and all subsequent pictures until any of the following conditions is met. Start a new CAS · Bitstream ends The NNRA SEI message with nnra_cancel_flag=1 is outputted in the output order following the current picture. That is, the above continuity is cancelled by nnra_cancel_flag=1. In addition, it may be the following. The NNRA SEI message with the same nnra_target_id as the current SEI message and nnra_cancel_flag = 1 is output in the output order following the current picture. That is, the above continuity is canceled by nnra_cancel_flag = 1.

[0160] like Fig.13 As shown, the atlas decoding unit 302 and the atlas encoding unit 102 decode and encode the syntax elements representing the second refined object information. The second refined object information includes nnra_target_map_index representing mapIdx as a refined application object. Moreover, although not shown in the figure, the second refined object information may also include nnra_target_attribute_index representing the index attrIdx of the attribute of the refined application object, nnra_attribute_num representing the number of attributes of the refined application object, nnra_attribute_partition_index representing the partIdx of the refined application object, and nnra_auxiliary_video_flag indicating whether the refined application object is auxiliary data. It should be noted that nnra_target_attribute_index and nnra_attribute_num are the same as nnrc_target_attribute_index and nnrc_attribute_num that have been described.

[0161] exist Fig.13In the syntax structure of , multiple refinements are specified and transmitted by one application SEI. Here, in the refinement number N (>=1), the value of N (actually the value of N-1) is transmitted, and N refinements are specified. nnra_count_minus1 indicating the number of instances of refinement is included as a syntax element. In addition, for each i from i=0 to i=nnra_count_minus1+1, nnra_cancel_flag[i], nnra_persistent_flag[i], and nnra_target_id[i] indicating the id of the characteristic SEI showing the NNRE may also be included. nnra_strength_num[i] may also be included. nnra_strength_num[i] is a parameter indicating the amount of strength information used. nnra_streangth_num[i] must be equal to nnrc_strength_num specified in the NNRC SEI message identified by nnra_target_id[i].

[0162] Furthermore, for each i, nnra_target_map_index[i] is included as the second refinement target information.

[0163] Moreover, for each i, the syntax element indicating nnra_all_map_flag[i] as the second refinement object information may be decoded or encoded. Here, nnra_all_map_flag[i]=1 indicates that all maps are refined. That is, mapIdx indicating the refinement application object includes all mapIdx. When nnra_all_map_flag[i]=1, nnra_target_map_index[i] is not encoded or decoded, and it is inferred that nnra_target_map_index[i]=1, i=0..asps_map_count_minus1. When nnra_all_map_flag[i]=0, nnra_target_map_index[i] is encoded or decoded. It should be noted that nnra_all_map_flag[i] and nnra_target_map_index[i] may be encoded or decoded through characteristic SEI instead of activating SEI. In this case, in the characteristic SEI, nnrc_all_map_flag and nnrc_target_map_index are encoded and decoded as corresponding syntax elements.

[0164] Furthermore, although not shown in the figure, for each i, nnra_all_partition_flag[i] may be included as the second refinement target information. Here, nnra_all_partiotion_flag[i]=1 indicates that all attributes are refined. That is, it indicates that all attrIdx are included in the attrIdx of the refinement application object. When nnra_all_partiotion_flag[i]=1, nnra_target_partiotion_index[i] is not encoded or decoded, and it is inferred that nnra_target_partiotion_index[i]=1, i=0..asps_map_count_minus1. When nnra_all_partiotion_flag[i]=0, nnra_target_partiotion_index[i] is encoded or decoded.

[0165] When nnra_cancel_flag[i] indicates no cancellation, for example, when nnra_cancel_flag[i] == 0 (!nnra_cancel_flag[i]), the spectrum decoding unit 302 decodes the persistence information nnra_persistent_flag[i], and then decodes nnra_strength_num[i] indicating the number of applied strength information and nnra_strength[i][j] indicating the applied strength information.

[0166] The refinement unit 306 sets the input data to mapIdx=nnra_target_map_index[i], attrIdx=nnra_target_attribute_index for each i of i=0..nnra_count_minus1+1. Furthermore, oFrame=OccFramesNF[compTimeIdx][0], gFrame=GeoFramesNF[mapIdx][compTimeIdx][frameIdx][0], aFrame=AttrFramesNF[attrIdx][mapIdx][compTimeIdx]. Based on this, the neural network model of the characteristic SEI (NNRCS SEI having the same value as nnra_target_id[i] as nnrc_id) specified and activated by the activation information is used to perform the following processing.

[0167] When the refinement target information indicates an attribute, the attribute frame of the map indicated by the second refinement target information nnra_target_map_index[i] is added to the input tensor. For example, mapIdx=nnra_target_map_index[i] may be set in (Eq.IN-ATTR-DIM) and (Eq.IN-ATTR-POS).

[0168] When the refined object information represents an occupancy map, a specific component of the output tensor may also be set for the occupancy map, for example (Eq. OUT-OCC).

[0169] When the refinement target information represents geometry, a specific component of the output tensor is set to the geometry frame having mapIdx indicated by the second refinement target information nnra_target_map_index[i]. For example, mapIdx=nnra_map_index can be set in (Eq.OUT-GEO).

[0170] When the refinement target information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by the second refinement target information nnra_target_map_index. For example, mapIdx=nnra_target_map_index[i] may be set in (Eq.OUT-ATTR), (Eq.OUT-ATTR-DIM), and (Eq.OUT-ATTR-POS).

[0171] When an attribute is included as refinement target information, a syntax element nnrc_target_attribute_index[i] indicating attrId of the refinement target may be included.

[0172] When the refinement target information indicates an attribute, the attribute indicated by the second refinement target information nnrc_target_attribute_index and nnra_target_map_index is added to the input tensor. For example, mapIdx=nnra_target_map_index[i](or nnrc_target_map_index) and attrIdx=nnrc_target_attribute_index(or nnra_target_attribute_index[i]) can be set in (Eq.IN-ATTR-POS) and (Eq.IN-ATTR-POS2).

[0173] When the refinement target information indicates an attribute, a specific component of the output tensor is set for the attribute indicated by the second refinement target information nnra_target_map_index[i] and nnra_target_attribute_index[i]. For example, mapIdx=nnra_target_map_index[i] (or nnrc_target_map_index) and attrIdx=nnrc_target_attribute_index(or nnra_target_attribute_index[i]) can be set in (Eq.OUT-ATTR), (Eq.OUT-ATTR-POS), and (Eq.OUT-ATTR-POS2). Furthermore, nnra_attribute_partition_index and nnra_auxiliary_video_flag may also be included.

[0174] In the above description, the atlas decoding unit 302 decodes the syntax element nnra_target_map_index from the application SEI, derives the mapIdx as the refinement object, and derives the attrIdx as the attribute of the refinement object from the application SEI decoding syntax element nnra_target_attribute_index[i]. The attribute specified by attrIdx is selected for refinement. Therefore, the effect of being able to apply the most optimized refinement to specific mapIdx and attrIdx is achieved. In addition, the refinement specified by the same characteristic SEI can be applied to the attributes represented by the same value of attrIdx. The refinements specified by different characteristic SEIs can be applied to the attributes represented by different attrIdx at the same time.

[0175] In summary, the 3D data decoding device includes: a geometry decoding unit, which decodes a geometry frame from encoded data; and an attribute decoding unit, which decodes an attribute frame. The 3D data decoding device is characterized in that it includes a refinement information decoding unit, which decodes refined characteristic information and refined activation information from the above-mentioned encoded data, decodes a grammatical element representing the number of refinements from the above-mentioned activation information, and decodes an index of the above-mentioned characteristic information representing the number of decoded refinements. The 3D data decoding device includes a refinement processing unit, which performs refinement processing on the above-mentioned attribute frame or geometry frame according to the above-mentioned characteristic information.

[0176] The above-mentioned 3D data decoding device is characterized in that the above-mentioned refinement information decoding unit also decodes the number strengthNum of strength information from the above-mentioned characteristic information, decodes the above-mentioned strengthNum strength information, and the refinement processing unit inputs at least one of the occupied frame, the geometric frame or the attribute frame into the input tensor of the neural network, and then inputs the strengthNum values ​​into the strengthNum channels of the input tensor of the neural network.

[0177] The 3D data decoding device is characterized in that the refinement information decoding unit decodes a flag indicating whether to refine all maps, and when the flag indicates that all maps are not to be refined, decodes an index indicating which map is to be refined.

[0178] The above-mentioned 3D data decoding device is characterized in that the above-mentioned refinement information decoding unit decodes the index of the attribute frame and the number of attribute frames as the refinement object from the encoded data of the above-mentioned characteristic information or the above-mentioned activation information, takes the index of the decoded attribute frame as the starting point, and inputs the attribute frames of the number of decoded attribute frames into the input tensor of the above-mentioned neural network for refinement processing.

[0179] The above-mentioned 3D data decoding device is characterized in that the above-mentioned refinement information decoding unit decodes the refined characteristic information and the information of the mapping map of the refinement object from the encoded data of the above-mentioned activation information, decodes the cancellation flag indicating whether to cancel the refinement processing associated with the above-mentioned refined characteristic information and the mapping map of the refinement object, and further decodes the continuity information when the above-mentioned cancellation flag indicates not to cancel.

[0180] The 3D data encoding device comprises: a geometry encoding unit for encoding a geometry frame; and an attribute encoding unit for encoding an attribute frame. The 3D data encoding device is characterized in that it comprises a refinement information encoding unit for encoding refined characteristic information and refined activation information from the above-mentioned encoded data, encoding a grammatical element indicating the number of refinements from the above-mentioned activation information, and encoding an index of the above-mentioned characteristic information indicating the number of encoded refinements. The 3D data encoding device comprises a refinement processing unit for performing refinement processing on the above-mentioned attribute frame or the geometry frame according to the above-mentioned characteristic information.

[0181] The above-mentioned 3D data encoding device is characterized in that the above-mentioned refinement information encoding unit also encodes the number strengthNum of intensity information from the above-mentioned characteristic information, encodes the above-mentioned strengthNum intensity information, and the refinement processing unit inputs at least one of the occupied frame, the geometric frame or the attribute frame into the input tensor of the neural network, and then inputs the strengthNum values ​​derived from the encoded intensity information into the strengthNum channels of the input tensor of the neural network.

[0182] The 3D data encoding device is characterized in that the refinement information encoding unit encodes a flag indicating whether to refine all maps, and when the flag indicates that all maps are not to be refined, encodes an index indicating which map is to be refined.

[0183] The above-mentioned 3D data encoding device is characterized in that the above-mentioned refinement information decoding unit encodes the index of the attribute frame and the number of attribute frames that are the refinement objects from the encoded data of the above-mentioned characteristic information or the above-mentioned activation information, and uses the index of the encoded attribute frame as the starting point, and inputs the attribute frames of the number of the encoded attribute frames into the input tensor of the above-mentioned neural network for refinement processing.

[0184] The above-mentioned 3D data encoding device is characterized in that the above-mentioned refinement information decoding unit encodes the information of the refined characteristic information and the information of the mapping map of the refinement object from the encoded data of the above-mentioned activation information, encodes a cancellation flag indicating whether to cancel the refinement processing associated with the above-mentioned refined characteristic information and the mapping map of the refinement object, and further encodes the continuity information when the above-mentioned cancellation flag indicates not to cancel. (Configuration of 3D Data Encoding Device)

[0185] Fig.14 This is a functional block diagram showing a schematic configuration of the 3D data encoding device 11 according to the embodiment of the present invention.

[0186] The 3D data encoding device 11 is composed of a patch generation unit 101, a map encoding unit 102, an occupancy map generation unit 103, an occupancy map encoding unit 104, a geometry generation unit 105, a geometry encoding unit 106, an attribute generation unit 108, an attribute encoding unit 109, a refinement parameter derivation unit 110, and a multiplexing unit 111. The 3D data encoding device 11 inputs a point cloud or a mesh as 3D data and outputs encoded data.

[0187] The patch generation unit 101 inputs 3D data, generates a set of patches (here, rectangular images) and outputs them. Specifically, the 3D data is divided into multiple regions, each of which is projected onto any plane in a 3D bounding box set in the 3D space to generate multiple patches. The patch generation unit 101 outputs information related to the 3D bounding box (coordinates, size, etc.) and information related to the mapping to the projection surface (projection surface, coordinates, size, rotation, etc. of each patch) as atlas information.

[0188] The atlas encoding unit 102 encodes the atlas information output from the patch generating unit 101 and outputs atlas data.

[0189] The occupancy map generation unit 103 inputs the set of patches output from the patch generation unit 101, and generates an occupancy map that represents the valid area (area where 3D data exists) of each patch as a 2D binary image (for example, the valid area is represented as 1, and the invalid area is represented as 0). It should be noted that the values ​​of the valid area and the invalid area may also use other values ​​such as 255 and 0.

[0190] The occupancy map encoding unit 104 inputs the occupancy map output from the occupancy map generating unit 103, and outputs the occupancy map and occupancy map data. As the encoding method, VVC, HEVC, etc. are used.

[0191] The geometry generation unit 105 generates a geometry frame storing the depth value of each patch relative to the projection surface based on the 3D data, the occupancy map, the occupancy map data and the atlas information. The geometry frame generation unit 105 derives the point with the smallest depth relative to the projection surface from the points projected on the pixel g(x, y) as p_min(x, y, z). In addition, the point with the largest depth from the points projected on the pixel g(x, y) at a prescribed distance d from p_min(x, y, z) is derived as p_max(x, y, z). The geometry frame in which p_min(x, y, z) is projected to all pixels of the projection surface is set as the geometry frame of the near layer. The geometry frame in which p_max(x, y, z) is projected to all pixels of the projection surface is set as the geometry frame of the far layer.

[0192] The geometry encoding unit 106 inputs a geometry frame and outputs a geometry frame and geometry data. As an encoding method, VVC, HEVC, etc. are used.

[0193] The attribute generation unit 108 generates an attribute frame storing color information (e.g., YUV value, RGB value, etc.) of each patch relative to the projection surface based on the 3D data, the occupancy map, the geometry frame, and the atlas information. The attribute generation unit 108 obtains the value of the attribute corresponding to the point p_min(x, y, z) with the minimum depth calculated by the geometry generation unit 105, and sets the attribute frame projected with the value as the attribute frame of the near layer. For p_max(x, y, z), the obtained attribute frame is also set as the attribute frame of the far layer.

[0194] The attribute coding unit 109 inputs an attribute frame and outputs an attribute and attribute data. As a coding method, VVC, HEVC, etc. are used.

[0195] The refinement parameter derivation unit 110 inputs the attribute frame and the original attribute frame or the geometric frame and the original geometric frame, selects or derives and outputs the most appropriate filtering parameters in the NN filtering process, and sets the values ​​of nnra_target_id, nnra_cancel_flag, nnra_persistance_flag, etc. for the above SEI.

[0196] The multiplexing unit 111 inputs the filter parameters output from the refinement parameter derivation unit 110, and outputs them in a prescribed format. The prescribed format refers to, for example, SEI as auxiliary enhancement information of image data, ASPS and AFPS as designated information of data structure in the V3C standard, ISOBMFF as a media file format standard, etc. The multiplexing unit 111 also multiplexes the atlas data, occupancy map data, geometry data, attribute data, and the above-mentioned filter parameters, and outputs them as coded data. As a multiplexing method, a byte stream format, ISOBMFF, etc. are used.

[0197] Fig.15 This is a block diagram showing the relationship between the 3D data encoding device 11 , the 3D data decoding device 31 , and the SEI message according to the embodiment of the present invention.

[0198] The 3D data encoding device 11 is composed of a video encoding unit and a SEI encoding unit. Fig.14 The SEI coding unit corresponds to the occupancy map coding unit 104, the geometry coding unit 106, and the attribute coding unit 109. Fig.14 The structure corresponds to the atlas encoding unit 102.

[0199] The 3D data decoding device 31 is composed of a video decoding unit, a SEI decoding unit, a switch, and a refinement unit. Figure 5 The SEI decoding unit corresponds to the occupancy map decoding unit 303, the geometry decoding unit 304, and the attribute decoding unit 305. Figure 5 The refinement unit corresponds to the spectrum decoding unit 302 in the structure of Figure 5 The refinement unit 306 corresponds to the refinement unit 306. The refinement unit 306 may include either the NN filter unit 611 or the ALF unit 610, or both.

[0200] The occupancy map, geometry frame, and attribute frame generated based on the 3D data are encoded by the image encoding unit. The encoded data is decoded by the image decoding unit to reconstruct the decoded image. The SEI encoding unit generates a characteristic SEI and an application SEI based on the 3D data. The SEI decoding unit decodes these SEI messages. The application SEI specifies the image to be refined, so the input switch only inputs the image to be refined to the refinement unit. The characteristic SEI is input to the refinement unit to specify the refinement applied to the decoded image. The refined image or the decoded image is displayed through the 3D data display unit 41( Figure 1 )show.

[0201] As mentioned above, one embodiment of the present invention has been described in detail with reference to the drawings, but the specific configuration is not limited to the above embodiment, and various design changes and the like can be made within the scope not departing from the gist of the present invention.

[0202] The embodiments of the present invention are not limited to the above-mentioned embodiments, and various modifications can be made within the scope of the claims. That is, embodiments obtained by combining technical solutions appropriately modified within the scope of the claims are also included in the technical scope of the present invention. Industrial Applicability

[0203] The embodiments of the present invention can be preferably applied to a 3D data decoding device that decodes coded data obtained by encoding 3D data and a 3D data encoding device that generates coded data obtained by encoding 3D data. In addition, the embodiments of the present invention can be preferably applied to the data structure of the coded data generated by the 3D data encoding device and referred to by the 3D data decoding device. Description of Reference Numerals 11: 3D data encoding device 101: Patch Generation Department 102: Graph Coding Department 103: Occupancy map generation unit 104: Occupancy map encoding unit 105: Geometry Generation Department 106: Geometry Coding Department 108: Attribute Generation Department 109: Attribute Coding Unit 110: Refinement parameter derivation section 111: Multiplexing Department 21: Network 31: 3D data decoding device 301: Header decoding unit 302: Spectrum information decoding unit 303: Occupancy map decoding unit 304: Geometry Decoding Department 306: Geometry Reconstruction Department 307: Attribute decoding unit 308: Refinement Department 309: V3C decoding unit 310: Pre-reconstruction department 311: Reconstruction Department 41: 3D data display device

Claims

1. A 3D data decoding device, comprising: a geometry decoding unit for decoding a geometry frame from coded data; and an attribute decoding unit for decoding an attribute frame, wherein the 3D data decoding device is characterized in that: A refinement information decoding unit is provided, the refinement information decoding unit decoding the refinement characteristic information and the refinement activation information from the encoded data, The 3D data decoding device decodes a syntax element indicating the number of refinements from the activation information, and decodes an index of the characteristic information indicating the number of decoded refinements, The 3D data decoding device includes a thinning processing unit configured to perform thinning processing on the attribute frame or the geometry frame based on the characteristic information.

2. The 3D data decoding device according to claim 1, characterized in that: The refinement information decoding unit also decodes the number strengthNum of strength information from the characteristic information, decodes the strengthNum strength information, and the refinement processing unit inputs at least one of the occupied frame, the geometric frame or the attribute frame into the input tensor of the neural network, and then inputs the strengthNum values ​​into the strengthNum channels of the input tensor of the neural network.

3. The 3D data decoding device according to claim 1, characterized in that: The refinement information decoding unit decodes a flag indicating whether refinement is to be performed on all maps, and when the flag indicates that refinement is not to be performed on all maps, decodes an index indicating which map is to be refined.

4. The 3D data decoding device according to claim 1, characterized in that: The refinement information decoding unit decodes the index of the attribute frame and the number of attribute frames as refinement objects from the encoded data of the characteristic information or the activation information, takes the index of the decoded attribute frame as a starting point, and inputs the attribute frames of the number of decoded attribute frames into the input tensor of the neural network for refinement processing.

5. The 3D data decoding device according to claim 1, characterized in that: The refinement information decoding unit decodes the refined characteristic information and information on a map of a refinement object from the encoded data of the activation information, decodes a cancellation flag indicating whether to cancel a refinement process associated with the refined characteristic information and the map of the refinement object, and further decodes persistence information when the cancellation flag indicates not to cancel.

6. A 3D data encoding device, comprising: a geometry encoding unit for encoding a geometry frame; and an attribute encoding unit for encoding an attribute frame, wherein the 3D data encoding device generates encoded data, wherein the 3D data encoding device is characterized in that: A refinement information encoding unit is provided, the refinement information encoding unit encodes the refinement characteristic information and the refinement activation information, the 3D data encoding device generates the encoded data, The 3D data encoding device encodes a syntax element indicating the number of refinements from the activation information, and encodes an index of the characteristic information indicating the number of the encoded refinements, The 3D data encoding device includes a thinning processing unit configured to perform thinning processing on the attribute frame or the geometry frame based on the characteristic information.

7. The 3D data encoding device according to claim 6, characterized in that: The refinement information encoding unit also encodes the number strengthNum of strength information from the characteristic information, and encodes the strengthNum strength information. The refinement processing unit inputs at least one of the occupation frame, the geometry frame or the attribute frame into the input tensor of the neural network, and then inputs the strengthNum values ​​derived from the encoded strength information into the strengthNum channels of the input tensor of the neural network.

8. The 3D data encoding device according to claim 6, characterized in that: The refinement information encoding unit encodes a flag indicating whether refinement is to be performed on all maps, and, when the flag indicates that refinement is not to be performed on all maps, encodes an index indicating which map is to be refined.

9. The 3D data encoding device according to claim 6, characterized in that: The refinement information decoding unit encodes the index of the attribute frame and the number of attribute frames as refinement objects from the encoded data of the characteristic information or the activation information, and inputs the attribute frames of the number of the encoded attribute frames into the input tensor of the neural network for refinement processing, taking the index of the encoded attribute frame as the starting point.

10. The 3D data encoding device according to claim 6, characterized in that: The refinement information decoding unit encodes the refined characteristic information and the information of the map of the refinement object from the encoded data of the activation information, encodes a cancellation flag indicating whether to cancel the refinement process associated with the refined characteristic information and the map of the refinement object, and further encodes persistence information when the cancellation flag indicates not to cancel.

Images