Encoding device, decoding device, program, and computer-readable recording medium

Abstract

To provide an encoding device, etc. capable of generating a bit stream suppressing a data amount when the number of types of point group properties (such as reflection coefficients) increases.SOLUTION: In an encoding device, a residual signal generation unit generates a residual signal based on a texture image representing a color of a pixel included in an image and a reflection coefficient image representing a reflection coefficient of the pixel. A reflection coefficient information residual video encoding unit encodes a sequence of images representing the residual signal of the pixel and outputs an encoded reflection coefficient information residual video. A texture video encoding unit encodes the sequence of texture images and outputs an encoded texture video. A bit stream multiplexing unit generates a bit stream including at least the encoded reflection coefficient information residual video output by the reflection coefficient information residual video encoding unit and the encoded texture video output by the texture video encoding unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an encoding device, a decoding device, a program, and a computer-readable recording medium. [Background technology]

[0002] 3D sensors are becoming widespread, and 3D applications such as AR / VR are also becoming more common. AR is an abbreviation for Augmented Reality. VR is an abbreviation for Virtual Reality. 3D point clouds are one of the methods of expressing 3D solid shapes. 3D point clouds have multiple 3D coordinates and are described together with information such as the colors and normals of the vertices associated with those coordinates.

[0003] 3D point clouds are sometimes drawn with shading calculations according to the vectors and intensities of light rays placed in a virtual 3D scene. Various methods have been proposed for drawing points, but one of the classical shading calculation methods is a method called the Phong reflection model. The Phong reflection model is a model of lighting and shading for casting shadows on points on a modeled surface. In the Phong reflection model, the specular reflection component i of a light source in a 3D scene is s , the diffuse reflection component i k , the diffuse reflection component i d We also define the specular reflection coefficient ks, diffuse reflection coefficient kd, and environmental reflection coefficient ka for the material on which the light is applied. Furthermore, if the direction vector from a point on the object surface to each light source is L, the normal at that point on the surface is N, the point on the surface where the light is completely reflected is R, and the direction vector toward the viewpoint is V, then the shading (light intensity I p ) can be calculated using the following formula (1):

[0004]

number

[0005] Point clouds are used in various fields, such as understanding the surroundings in autonomous driving vehicles and visualizing three-dimensional shapes using AR / VR technology. One method of using point clouds is volumetric video, which creates three-dimensional images by playing back continuous point clouds in chronological order. For example, a volumetric video capture studio has been developed that captures texture information and supports relighting by adding lighting information to the scene.

[0006] Because the 3D vertices of a point cloud have no relationship to one another, when a point cloud is treated as raw information, the coordinates, colors, normals, etc. of all the vertices are often described in a list format, which tends to result in a large amount of data.

[0007] Recently, ISO / IEC 23090-5 has been proposed as a method to compress dynamic point clouds into a time series that is treated as a volumetric video. In this method, the point cloud is projected onto an axial bounding box surface that surrounds it in three dimensions and treated as a two-dimensional continuous image. This image can be compressed using existing video codecs such as H.264 / AVC and H.265 / HEVC to significantly reduce the bit rate.

[0008] The configuration (prior art) of an encoding device for compressing and encoding a V3C stream is as follows. That is, a point group input to the encoding device is projected onto each plane of a three-dimensional bounding box that surrounds the point group. The bounding box has, for example, a rectangular parallelepiped shape. The encoding device then handles the point group projected onto each plane as a two-dimensional image, and arranges the point group so that it fits within a rectangular area of ​​a pre-prepared size. Furthermore, the encoding device generates an occupancy image, a geometry image, and a texture image from the rectangular area in which the two-dimensionally projected point group is arranged. The encoding device then encodes each image with an existing video encoding codec (such as H.265 or H.264), and multiplexes it with auxiliary information (auxiliary data) that indicates the effective pixel area in the two-dimensional image, thereby outputting a V3C bit stream.

[0009] By using the above procedure, the encoding device can significantly reduce the bit rate compared to when uncompressed point clouds are recorded or transmitted as is.

[0010] The technical configuration of the decoder is as follows: The decoder demultiplexes the input V3C bitstream. The decoder then decodes the coded occupancy, geometry, and texture images to obtain the respective uncompressed images. The decoder reconstructs and outputs a 3D point cloud based on the three decoded uncompressed images and the auxiliary data mentioned above.

[0011] A V3C stream that encodes a dynamic 3D point cloud is structured so that each unit, called a V3C unit, can be accessed. A V3C stream is composed of five V3C units, for example, V3C_VPS, V3C_AD, V3C_OVD, V3C_GVD, and V3C_AVD. The V3C_VPS unit contains an identifier that indicates that the stream is a V3C stream, information that indicates what information is contained in the following V3C unit, and auxiliary extension information. The V3C_AD unit contains information about the effective pixel area in a rectangular area and information that indicates the rotation of the point cloud placed in the rectangular information. The V3C_OVD, V3C_GVD, and V3C_AVD units that follow the V3C_AD unit store the compressed and encoded video stream. The V3C_OVD unit contains an occupancy map (occupancy information), and the V3C_GVD unit contains geometry (coordinate information). A 3D shape is reconstructed using the occupancy map, geometry, and effective pixel area information contained in the V3C_AD unit. The V3C_AVD unit contains attributes, and in standard reference encoding / decoding software, it is assumed that this will store color information of point clouds expressed in 8-bit RGB (red, green, and blue).

[0012] The V3C_AVD unit can store not only the color information of the point cloud, but also normals and other information. The type of information stored in the V3C_AVD unit can be identified by using an identifier defined in the supplemental extension information included in the V3C_VPS.

[0013] Specifically, the V3C standard defines the values ​​of ai_attribute_type_id (attribute type identifier) ​​as follows: The value "0" is assigned to the identifier "ATTR_TEXTURE" and the attribute type is "Texture". The value "1" is assigned to the identifier "ATTR_MATERIAL_ID" and the attribute type is "Material ID". The value "2" is assigned to the identifier "ATTR_TRANSPARENCY" and the attribute type is "Transparency". The value "3" is assigned to the identifier "ATTR_REFLECTANCE" and the attribute type is "Reflectance". The value "4" is assigned to the identifier "ATTR_NORMAL" and the attribute type is "Normals". Values ​​"5" to "14" are "ATTR_RESERVED", that is, they are reserved as reserved values ​​(undefined values). The value "15" is assigned to the identifier "ATTR_UNSPECIFIED" and the attribute type is "unspecified."

[0014] By using these identifiers, various attribute information can be stored in the V3C_AVD unit. In other words, not only normal information but also reflection coefficient information for expressing the texture of a surface can be stored in the V3C_AVD unit.

[0015] An encoding device for generating a bit stream including reflection coefficient information may include a reflection coefficient information image generation unit for generating an image of reflection coefficient information, and a reflection coefficient information image encoding unit for encoding the reflection coefficient information image.

[0016] A decoding device for decoding a bit stream including reflection coefficient information demultiplexes an input bit stream and decodes the resulting reflection coefficient image bit stream to obtain a reflection coefficient image. Such a decoding device outputs point cloud information to which reflection coefficient information has been added.

[0017] Non-Patent Document 1 describes video coding and point cloud data compression. In particular, "8.4.4.5 Attribute information semantics" (from page 54) of Non-Patent Document 1 describes the semantics of attribute information. Table 3 therein explains the values ​​of V3C attribute type identifiers. [Prior art documents] [Non-patent literature]

[0018] [Non-Patent Document 1] ISO / IEC 23090-5 Information technology - Coded representation of immersive media - Part 5: Visual volumetric video-based coding (V3C) and video-based point cloud compression (V-PCC), 2021 Summary of the Invention [Problem to be solved by the invention]

[0019] As mentioned above, by using attribute type identifiers, it is possible to include reflection coefficient information for expressing texture in a V3C stream. There is also room for compressing and encoding point cloud files of types undefined in the V3C standard as V3C streams. In other words, for example, when properties such as the diffuse reflection coefficient and specular reflection coefficient of a point cloud are obtained, it is possible to use an undefined ID as an attribute type identifier in order to multiplex that information into the V3C properties.

[0020] However, in this case, there is a problem that, as the number of types of properties to be multiplexed increases, the number of compressed video streams also increases, resulting in an increase in the data amount of the compressed bitstream.

[0021] The present invention has been made based on the recognition of the above problems, and aims to provide an encoding device capable of generating a bitstream with a reduced amount of data when the number of types of point cloud properties increases, a decoding device capable of performing decoding based on that bitstream, programs for performing these operations, and a computer-readable recording medium on which the bitstream can be recorded. [Means for solving the problem]

[0022] [1] In order to solve the above problem, an encoding device according to one aspect of the present invention includes a residual signal generation unit that generates a residual signal based on a texture image representing a color of a pixel included in an image and a reflection coefficient image representing a reflection coefficient of the pixel; a reflection coefficient information residual video encoding unit that encodes a sequence of images representing the residual signal of the pixel and outputs an encoded reflection coefficient information residual video; a texture video encoding unit that encodes the sequence of texture images and outputs an encoded texture video; and a bitstream multiplexing unit that generates a bitstream including at least the encoded reflection coefficient information residual video output by the reflection coefficient information residual video encoding unit and the encoded texture video output by the texture video encoding unit.

[0023] [2] Also, one aspect of the present invention is the encoding device of [1] above, further comprising a point cloud input unit that acquires point cloud data including at least vertex color information and reflection coefficient information related to a point cloud, which is a group of points having three-dimensional coordinates; a projection unit that projects the point cloud onto the two-dimensional image, and outputs the texture image based on the vertex color information contained in the point cloud data, outputs the reflection coefficient image based on the reflection coefficient information contained in the point cloud data, and outputs information about pixel regions in the image; and an auxiliary data generation unit that generates auxiliary data based on the information about pixel regions in the image output by the projection unit, and the bit stream multiplexing unit further generates the bit stream including the auxiliary data generated by the auxiliary data generation unit.

[0024] [3] Furthermore, a decoding device according to one aspect of the present invention includes a bitstream demultiplexing unit that demultiplexes an input bitstream and outputs at least an encoded reflection coefficient information residual image and an encoded texture image; a texture image decoding unit that decodes the encoded texture image and outputs a sequence of texture images representing colors of pixels included in an image; a reflection coefficient information residual image decoding unit that decodes the encoded reflection coefficient information residual image and outputs a sequence of reflection coefficient residual images representing residual signals related to the reflection coefficients of the pixels; and a reflection coefficient information restoration unit that restores and outputs a sequence of reflection coefficient images representing the reflection coefficients of the pixels based on the decoded texture image and the decoded reflection coefficient residual image.

[0025] [4] Moreover, according to one aspect of the present invention, in the decoding device of [3] above, the decoding device further includes an auxiliary data extraction unit that extracts auxiliary data, which is information relating to a pixel region within an image, based on information passed from the bit stream demultiplexing unit, and a 3D point cloud reconstruction unit that reconstructs point cloud data relating to a point cloud, which is a group of points having 3D coordinates, based on the auxiliary data extracted by the auxiliary data extraction unit, the texture image output by the texture video decoding unit, and the reflection coefficient image output by the reflection coefficient information restoration unit.

[0026] [5] Also, one aspect of the present invention is a program for causing a computer to function as an encoding device including: a residual signal generation unit that generates a residual signal based on a texture image representing a color of a pixel included in an image and a reflection coefficient image representing a reflection coefficient of the pixel; a reflection coefficient information residual video encoding unit that encodes a sequence of images representing the residual signal of the pixel and outputs an encoded reflection coefficient information residual video; a texture video encoding unit that encodes the sequence of texture images and outputs an encoded texture video; and a bit stream multiplexing unit that generates a bit stream including at least the encoded reflection coefficient information residual video output by the reflection coefficient information residual video encoding unit and the encoded texture video output by the texture video encoding unit.

[0027] [6] Also, one aspect of the present invention is a program for causing a computer to function as a decoding device including: a bitstream demultiplexing unit that demultiplexes an input bitstream and outputs at least an encoded reflection coefficient information residual image and an encoded texture image; a texture image decoding unit that decodes the encoded texture image and outputs a sequence of texture images representing colors of pixels included in an image; a reflection coefficient information residual image decoding unit that decodes the encoded reflection coefficient information residual image and outputs a sequence of reflection coefficient residual images representing residual signals related to the reflection coefficients of the pixels; and a reflection coefficient information restoration unit that restores and outputs a sequence of reflection coefficient images representing the reflection coefficients of the pixels based on the decoded texture image and the decoded reflection coefficient residual image.

[0028] [7] Also, one aspect of the present invention is a computer-readable recording medium comprising: a texture unit memory area for storing, as an attribute, an encoded texture image obtained by encoding a sequence of texture images representing the colors of pixels included in an image; one or more reflection coefficient unit memory areas for storing, as an attribute, an encoded reflection coefficient information residual image obtained by encoding a sequence of reflection coefficient residual images representing a residual signal obtained based on the texture images representing the colors of pixels included in the image and a reflection coefficient image representing the reflection coefficient of the pixels; and a parameter set unit memory area having an attribute type identifier for identifying the type of attribute stored in each of the texture unit and the reflection coefficient unit, wherein each of the one or more reflection coefficient unit memory areas stores the reflection coefficient information residual image for any one of an ambient light coefficient (Ka), a diffuse reflection coefficient (Kd), and a specular reflection coefficient (Ks). Effect of the Invention

[0029] According to the present invention, the residual signal generating unit generates a residual signal having a narrower range of values ​​than the reflection coefficient image itself. Therefore, the reflection coefficient information residual video encoding unit outputs a code having a smaller data size than when the reflection coefficient image is encoded as is. In other words, the encoding device can generate a more compact code. The decoding device can decode the code generated by such an encoding device. [Brief description of the drawings]

[0030] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a schematic functional configuration of an encoding device according to the embodiment. [Diagram 3] FIG. 2 is a block diagram showing a schematic functional configuration of a decoding device according to the embodiment. [Figure 4] 11 is a flowchart showing a processing procedure performed by the encoding device of the embodiment. [Diagram 5] 11 is a flowchart showing a processing procedure performed by the decoding device of the embodiment. [Figure 6] 1 is a schematic diagram showing an example of a V3C bitstream structure (with reflection coefficient residual information added) output by the encoding device according to the embodiment. FIG. [Figure 7] 10 is a schematic diagram showing an example of the syntax of attribute information (attribute_information) used in the embodiment. FIG. [Figure 8] 2 is a block diagram showing an example of the internal configuration of an encoding device or a decoding device according to the embodiment when implemented by a computer. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] Next, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, video is a sequence of images. In other words, an apparatus processing video is equivalent or nearly equivalent to an apparatus processing a sequence of images (or images). Also, an apparatus encoding / decoding video is equivalent or nearly equivalent to an apparatus encoding / decoding a sequence of images (or images). Also, in this embodiment, data in which pixel values ​​for each pixel are arranged is called an image.

[0032] 1 is a block diagram showing a schematic functional configuration of a system according to the present embodiment. As shown in the figure, the system 1 includes an encoding device 10, a decoding device 60, a recording medium 91, and a transmission medium 92.

[0033] The encoding device 10 inputs point cloud data including reflection coefficient information for each point, and outputs a V3C stream including the reflection coefficient information. That is, the encoding device 10 compression-encodes the input point cloud data and outputs it. The V3C stream output from the encoding device 10 is passed to the decoding device 60 by some means. The V3C stream output from the encoding device 10 is passed to the decoding device 60 by, for example, being recorded on a recording medium 91 or being transmitted using a transmission medium 92.

[0034] The decoding device 60 receives the V3C stream encoded by the encoding device 10, performs a decoding process on the V3C stream, and outputs point cloud data. The information decoded by the decoding device 60 includes reflection coefficient information. The decoding device 60 outputs the point cloud data including the reflection coefficient information.

[0035] The recording medium 91 at least temporarily records the data of the V3C stream output from the encoding device 10. The recording medium 91 records and holds information by a recording method using a magnetic code or a recording method using the state of a semiconductor circuit. The data recorded on the recording medium 91 can be read by the decoding device 60.

[0036] The transmission medium 92 transmits the V3C stream output from the encoding device 10. The transmission medium 92 transmits signals such as electrical signals, electromagnetic wave signals, and optical signals. The transmission medium 92 may be an element constituting a communication line such as the Internet. The data of the V3C stream transmitted using the transmission medium 92 can be read by the decoding device 60.

[0037] The system 1 may be configured to include only one of the recording medium 91 and the transmission medium 92. The system 1 may be configured to include only one of the encoding device 10 and the decoding device 60.

[0038] At least a part of the functions of the encoding device 10 and the decoding device 60 can be realized, for example, by a computer and a program. Each function has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. At least a part of the functions may be realized as a dedicated electronic circuit or the like rather than a program.

[0039] 2 is a block diagram showing a schematic functional configuration of the encoding device according to the present embodiment. As shown in the figure, the encoding device 10 includes a point cloud input unit 21, a two-dimensional projection unit 22, an occupancy image generation unit 23, an occupancy video encoding unit 24, a geometry image generation unit 25, a geometry video encoding unit 26, a texture image generation unit 27, a texture video encoding unit 28, an auxiliary data generation unit 29, a residual signal generation unit 31, a reflection coefficient information image generation unit 34, a reflection coefficient information residual video encoding unit 35, and a bit stream multiplexing unit 39. The reflection coefficient information image generation unit 34 and the reflection coefficient information residual video encoding unit 35 may be collectively referred to as a reflection coefficient information processing unit 32. The functions of each unit are as follows.

[0040] The point cloud input unit 21 acquires point cloud data including at least vertex color information and reflection coefficient information regarding a point cloud, which is a group of points having three-dimensional coordinates. The point cloud data acquired by the point cloud input unit 21 may further include occupancy information related to the point cloud. Also, the point cloud data acquired by the point cloud input unit 21 may further include geometry information related to the point cloud.

[0041] The two-dimensional projection unit 22 is also simply called a "projection unit." The two-dimensional projection unit 22 projects the point cloud acquired by the point cloud input unit 21 onto a two-dimensional image, outputs a texture image based on vertex color information contained in the point cloud data, outputs a reflection coefficient image based on reflection coefficient information contained in the point cloud data, and outputs information regarding pixel regions within the image. The image onto which the two-dimensional projection unit 22 projects the point cloud may be an image in which a bounding box, which will be described later, has been appropriately rearranged. The above "information regarding pixel regions within an image" includes information regarding the allocation and rearrangement of the image.

[0042] The occupancy image generating unit 23 generates a two-dimensional occupancy image based on the input point cloud data. A sequence of the occupancy images is an occupancy video. The occupancy image generating unit 23 passes the occupancy video to the occupancy video encoding unit 24.

[0043] The occupancy video encoding unit 24 encodes the occupancy video passed from the occupancy image generating unit 23. The occupancy video encoding unit 24 uses a method based on existing technology (for example, H.265 / HEVC (High. Efficiency Video Coding) or H.264 / AV; the same applies below) as a video encoding method.

[0044] The geometry image generator 25 generates a two-dimensional geometry image based on the input point cloud data. A sequence of the geometry images is a geometry image. The geometry image generator 25 passes the geometry image to the geometry image encoder 26.

[0045] The geometry video encoding unit 26 encodes the geometry video passed from the geometry image generation unit 25. The geometry video encoding unit 26 uses a method according to existing technology as a video encoding method.

[0046] The texture image generator 27 generates a two-dimensional texture image based on the input point cloud data. A texture image is an image having pixel color information as pixel values. A sequence of texture images is a texture video. The texture image generator 27 passes the texture video to the texture video encoder 28 and the residual signal generator 31.

[0047] The texture image encoder 28 encodes the sequence of texture images and outputs an encoded texture image.

[0048] The auxiliary data generating unit 29 generates auxiliary data based on "information about pixel regions in an image" output by the two-dimensional projection unit 22. The auxiliary data is information about the allocation and arrangement of pixels in a two-dimensional image.

[0049] The residual signal generator 31 receives a texture image from the texture image generator 27. The residual signal generator 31 generates a residual signal based on a texture image representing the color of a pixel included in the image and a reflection coefficient image representing the reflection coefficient of the pixel.

[0050] The reflection coefficient information processing unit 32 has a function of performing processing related to encoding of reflection coefficient information (residual signal). The reflection coefficient information processing unit 32 includes a reflection coefficient information image generating unit 34 and a reflection coefficient information residual video encoding unit 35.

[0051] The reflection coefficient information image generator 34 generates an image of a residual signal of the reflection coefficient. The value of each pixel in this image is the residual value for that pixel. The reflection coefficient information image generator 34 passes this image of the residual signal to the reflection coefficient information residual video encoder 35.

[0052] The reflection coefficient information residual video encoding unit 35 encodes a sequence of images (video) representing the residual signals of pixels, and outputs the encoded reflection coefficient information residual video. The reflection coefficient information residual video encoding unit 35 encodes the video using a video encoding method according to existing technology. The encoding method used by the reflection coefficient information residual video encoding unit 35 may be, for example, H.265 / HEVC (High. Efficiency Video Coding) or H.264 / AVC (Advanced Video Coding).

[0053] The bitstream multiplexing unit 39 generates a bitstream (V3C bitstream) including at least the coded reflection coefficient information residual video output by the reflection coefficient information residual video coding unit 35 and the coded texture video output by the texture video coding unit 28. The bitstream multiplexing unit 39 may further multiplex coded occupancy video to generate a bitstream. The bitstream multiplexing unit 39 may further multiplex coded geometry video to generate a bitstream. The bitstream multiplexing unit 39 may further multiplex auxiliary data generated by the auxiliary data generating unit 29 to generate a bitstream.

[0054] 3 is a block diagram showing a schematic functional configuration of a decoding device according to this embodiment. As shown in the figure, the decoding device 60 includes a bitstream demultiplexing unit 71, an occupancy video decoding unit 73, a geometry video decoding unit 75, a texture video decoding unit 77, an auxiliary data extraction unit 79, a reflection coefficient information residual video decoding unit 84, a reflection coefficient information restoration unit 85, and a 3D point cloud reconstruction unit 88. The reflection coefficient information residual video decoding unit 84 and the reflection coefficient information restoration unit 85 may be collectively referred to as a reflection coefficient information processing unit 82. The functions of each unit are as follows.

[0055] The bitstream demultiplexing unit 71 demultiplexes the input bitstream (V3C bitstream) and outputs at least a coded reflection coefficient information residual image and a coded texture image. The bitstream demultiplexing unit 71 may further output a coded occupancy image. The bitstream demultiplexing unit 71 may further output a coded geometry image. The bitstream demultiplexing unit 71 may further output auxiliary data. The bitstream demultiplexing unit 71 passes information on which the auxiliary data is based to the auxiliary data extraction unit 79.

[0056] The occupancy video decoding unit 73 outputs an occupancy video (a sequence of occupancy images) by decoding the code of the occupancy video passed from the bit stream demultiplexing unit 71. The occupancy video decoding unit 73 passes the decoded occupancy video to the 3D point group reconstruction unit 88.

[0057] The geometry video decoding unit 75 outputs a geometry video (a sequence of geometry images) by decoding the code of the geometry video passed from the bitstream demultiplexing unit 71. The geometry video decoding unit 75 passes the decoded geometry video to the 3D point cloud reconstruction unit 88.

[0058] The texture video decoding unit 77 outputs a texture video (a sequence of texture images) by decoding the code of the texture video passed from the bit stream demultiplexing unit 71. The texture image represents the color of the pixels included in the image. The texture video decoding unit 77 passes the decoded texture video to the 3D point cloud reconstruction unit 88 and the reflection coefficient information restoration unit 85.

[0059] The auxiliary data extraction unit 79 extracts auxiliary data, which is information relating to pixel regions in an image, based on the information passed from the bit stream demultiplexing unit 71. The auxiliary data extraction unit 79 passes the extracted auxiliary data to the 3D point cloud reconstruction unit 88.

[0060] The reflection coefficient information processing unit 82 has a function of performing processes relating to decoding and restoration of the reflection coefficients. The reflection coefficient information processing unit 82 includes a reflection coefficient information residual video decoding unit 84 and a reflection coefficient information restoration unit 85.

[0061] The reflection coefficient information residual video decoding unit 84 decodes the encoded reflection coefficient information residual video passed from the bitstream demultiplexing unit 71, and outputs a reflection coefficient information residual video (a sequence of reflection coefficient residual images) representing a residual signal related to the reflection coefficients of pixels.

[0062] The reflection coefficient information restoration unit 85 restores and outputs a sequence of reflection coefficient images representing the reflection coefficients of pixels based on the decoded texture image and the decoded reflection coefficient residual image. Specifically, the reflection coefficient information restoration unit 85 restores the reflection coefficient of a pixel by adding the value of the residual signal corresponding to that pixel to the pixel value of the decoded texture image.

[0063] The three-dimensional point cloud reconstruction unit 88 reconstructs point cloud data relating to a point cloud, which is a group of points having three-dimensional coordinates, based on the auxiliary data extracted by the auxiliary data extraction unit 79, the texture image output by the texture video decoding unit 77, and the reflection coefficient image output by the reflection coefficient information restoration unit 85. The three-dimensional point cloud reconstruction unit 88 may further reconstruct the point cloud data as data including information on an occupancy image. The three-dimensional point cloud reconstruction unit 88 may further reconstruct the point cloud data as data including information on a geometry image.

[0064] Next, the operation procedures of the encoding device 10 and the decoding device 60 will be described with reference to flowcharts.

[0065] 4 is a flowchart showing the procedure of processing by the encoding device 10. The operation procedure of the encoding device 10 will be described below with reference to this flowchart.

[0066] In step S11, the point cloud input unit 21 inputs point cloud data. Here, the data acquired by the point cloud input unit 21 includes, for example, information on three-dimensional coordinates of the point cloud, occupancy information, geometry information, texture information, and reflection coefficient information. The reflection coefficient information may include a diffuse reflection coefficient, a specular reflection coefficient, and an environmental reflection coefficient (ambient light coefficient). The point cloud input unit 21 interprets each property of the input point cloud. In other words, the encoding device 10 grasps what property information the input point cloud data has.

[0067] In step S12, the two-dimensional projection unit 22 assigns the point cloud to a bounding box surface surrounding the point cloud based on the three-dimensional coordinate information of the point cloud. The bounding box surface is a plurality of surfaces surrounding the point cloud. Typically, the bounding box surface is a total of six planes, two perpendicular to each of the X-axis, Y-axis, and Z-axis in an XYZ orthogonal coordinate system in space. These six planes form a rectangular parallelepiped. However, the bounding box surface may be a surface of another shape. The two-dimensional projection unit 22 projects the point cloud onto the bounding box surface.

[0068] In step S13, the two-dimensional projection unit 22 allocates and stores the two-dimensional image resulting from the projection of the point cloud in a two-dimensional rectangular area. That is, the two-dimensional projection unit 22 allocates the point cloud projected as a two-dimensional image in the rectangular area while dividing the bounding box surface according to the allocation. The two-dimensional projection unit 22 passes information on the allocation arrangement as a two-dimensional image to the auxiliary data generation unit 29. The auxiliary data generation unit 29 generates auxiliary data based on the information passed from the two-dimensional projection unit 22.

[0069] In step S14, the occupancy image generating unit 23 generates an occupancy image (also called an "occupancy state image") based on the point cloud data. The occupancy image generating unit 23 passes the generated occupancy image to the occupancy video encoding unit 24. The occupancy video encoding unit 24 also performs compression encoding of the occupancy image generated by the occupancy image generating unit 23. The occupancy video encoding unit 24 passes the occupancy video stream obtained as a result of the compression encoding process to the bit stream multiplexing unit 39.

[0070] In step S15, the geometry image generation unit 25 generates a geometry image (also called a "depth information image") based on the point cloud data. The geometry image generation unit 25 passes the generated geometry image to the geometry video encoding unit 26. The geometry video encoding unit 26 also performs compression encoding of the geometry image generated by the geometry image generation unit 25. The geometry video encoding unit 26 passes the geometry video stream obtained as a result of the compression encoding process to the bit stream multiplexing unit 39.

[0071] In step S16, texture image generation unit 27 generates a texture image (also called a "vertex color image") based on the point cloud data. Texture image generation unit 27 passes the generated texture image to texture video encoding unit 28. Furthermore, texture video encoding unit 28 performs compression encoding of the texture image generated by texture image generation unit 27. Texture video encoding unit 28 passes a texture video stream obtained as a result of the compression encoding process to bit stream multiplexing unit 39.

[0072] In step S17, the residual signal generation unit 31 extracts a residual signal between the image of reflection coefficient information included in the point cloud data and the texture image (vertex color information) generated by the texture image generation unit 27. The residual signal generation unit 31 extracts a signal representing the residual between corresponding pixels in the image of reflection coefficient information and the texture image. Specifically, in order to obtain a residual signal, the residual signal generation unit 31 calculates the difference between these pixel values ​​and generates an image of the residual signal. When there are multiple types of images of reflection coefficient information (for example, a specular reflection coefficient ks, a diffuse reflection coefficient kd, and an environmental reflection coefficient (ambient light coefficient) ka), the residual signal generation unit 31 extracts the residual between the texture image and each reflection coefficient, and generates a residual signal. The residual signal generation unit 31 passes the generated residual signal to the reflection coefficient information image generation unit 34.

[0073] In step S18, the reflection coefficient information image generating unit 34 generates a reflection coefficient residual image based on the residual signal passed from the residual signal generating unit 31. That is, the reflection coefficient residual image is an image in which the pixel value is a residual based on the texture image. When there are multiple types of reflection coefficient information (for example, a specular reflection coefficient ks, a diffuse reflection coefficient kd, and an environmental reflection coefficient (ambient light coefficient) ka), the reflection coefficient information image generating unit 34 generates a reflection coefficient residual image for each type of reflection coefficient. The reflection coefficient information image generating unit 34 passes the generated reflection coefficient residual image to the reflection coefficient information residual video encoding unit 35. The reflection coefficient information residual video encoding unit 35 performs compression encoding of the reflection coefficient residual image (video). The reflection coefficient information residual video encoding unit 35 passes a stream of the reflection coefficient residual image (video) obtained as a result of the compression encoding process to the bit stream multiplexing unit 39.

[0074] In step S19, the bitstream multiplexing unit 39 multiplexes data passed from the upstream processing units to generate a V3C bitstream. Specifically, the bitstream multiplexing unit 39 multiplexes the video streams passed from the occupancy video encoding unit 24, the geometry video encoding unit 26, the texture video encoding unit 28, and the reflection coefficient information residual video encoding unit 35. Furthermore, the bitstream multiplexing unit 39 multiplexes auxiliary data passed from the auxiliary data generating unit 29 to generate a V3C bitstream. The bitstream multiplexing unit 39 assigns an appropriate identifier (the attribute type identifier described above) to the video stream of each property. That is, the bitstream multiplexing unit 39 associates different identifiers with a unit that stores a video stream of a residual signal for the specular reflection coefficient ks, a unit that stores a video stream of a residual signal for the diffuse reflection coefficient kd, and a unit that stores a video stream of a residual signal for the environmental reflection coefficient (ambient light coefficient) ka.

[0075] In step S20, the bit stream multiplexing unit 39 outputs the generated V3C bit stream to the outside. That is, the bit stream multiplexing unit 39 outputs the V3C bit stream multiplexed with the reflection coefficient information. The bit stream multiplexing unit 39 may write the generated V3C bit stream to a recording medium 91, for example, or may transmit the generated V3C bit stream to an external device or the like via a transmission medium 92 (see FIG. 1).

[0076] As described above, the encoding device 10 encodes the generated residual signal of the reflection coefficient in the reflection coefficient information residual video encoding unit 35 to generate a compressed bitstream. The encoding device 10 can multiplex information on the type of reflection coefficient into a V3C bitstream as an attribute type identifier and output the bitstream. The reflection coefficient information assigned to the point group input to the encoding device 10 may include one or more types of coefficients, such as a specular reflection coefficient ks, a diffuse reflection coefficient kd, and an environmental reflection coefficient (ambient light coefficient) ka.

[0077] The encoding device 10 may generate residual signals for all of these reflection coefficients and encode the image of the residual signals. Alternatively, the encoding device 10 may generate residual signals for only some types of reflection coefficients and encode the image of the residual signals. As an example, the processing of the residual signal generating unit 31 may be applied only to the diffuse reflection coefficients, and the specular reflection coefficients may be encoded as they are without applying the processing of the residual signal generating unit 31.

[0078] A texture image and a reflection coefficient image tend to be statistically similar images. Therefore, the range of values ​​that the residual signal takes is narrower than the range of values ​​that the pixel values ​​of the reflection coefficient image take. Therefore, it is expected that the size of the code obtained when compressing and encoding the residual signal will be smaller than the size of the code obtained when compressing and encoding the reflection coefficient image as is. In other words, this embodiment makes it possible to perform encoding with a higher degree of compression.

[0079] 5 is a flowchart showing the procedure of processing by the decoding device 60. The operation procedure of the decoding device 60 will be described below with reference to this flowchart.

[0080] In step S61, the bitstream demultiplexing unit 71 of the decoding device 60 acquires a V3C bitstream and performs a process of demultiplexing the bitstream. The bitstream acquired by the bitstream demultiplexing unit 71 is a bitstream including reflection coefficient information, and is encoded by, for example, the encoding device 10 described above. Through the demultiplexing process, the bitstream demultiplexing unit 71 extracts a plurality of V3C units from the input V3C bitstream. That is, the bitstream demultiplexing unit 71 extracts a unit of occupancy video, a unit of geometry video, a unit of texture video, and a unit of reflection coefficient information residual video from the V3C stream. Note that the bitstream demultiplexing unit 71 may extract a plurality of types of units of reflection coefficient information residual video.

[0081] The bitstream demultiplexing unit 71 passes the occupancy video to an occupancy video decoding unit 73. Furthermore, the bitstream demultiplexing unit 71 passes the geometry video to a geometry video decoding unit 75. Furthermore, the bitstream demultiplexing unit 71 passes the texture video to a texture video decoding unit 77. Furthermore, the bitstream demultiplexing unit 71 passes the reflection coefficient information residual video to a reflection coefficient information residual video decoding unit 84. Furthermore, the bitstream demultiplexing unit 71 extracts auxiliary data and passes it to an auxiliary data extraction unit 79.

[0082] In step S62, the auxiliary data extraction unit 79 extracts information such as an identifier of effective pixel region information within an image from the auxiliary data.

[0083] In step S63, the occupancy video decoding unit 73 decodes the compressed stream of the occupancy video (also called "occupancy map image") obtained by demultiplexing. The occupancy video decoding unit 73 passes the occupancy video obtained as the decoding result to the 3D point group reconstruction unit 88.

[0084] In step S64, the geometry video decoding unit 75 decodes the compressed stream of the geometry video (also called "geometry image") obtained by demultiplexing. The geometry video decoding unit 75 passes the geometry video obtained as the decoding result to the 3D point group reconstruction unit 88.

[0085] In step S65, the texture image decoding unit 77 decodes the compressed stream of the texture image (also called a "texture image") obtained by demultiplexing. The texture image decoding unit 77 passes the texture image obtained as a result of the decoding to the 3D point group reconstruction unit 88.

[0086] In step S66, the reflection coefficient information residual video decoding unit 84 decodes the compressed stream of the reflection coefficient information residual video (also called a "reflection coefficient residual image"). When reflection coefficient information residual videos for a plurality of types of reflection coefficients exist, the reflection coefficient information residual video decoding unit 84 decodes the compressed stream for each of them. As described in the description of the encoding device 10, the reflection coefficient information residual video is composed of an image of the residual between the image of the reflection coefficient information and the texture image. In other words, the pixel value of each pixel of the reflection coefficient information residual video corresponds to a residual signal.

[0087] In step S67, the reflection coefficient information restoration unit 95 restores the reflection coefficient image from the texture image decoded in step S65 and the reflection coefficient residual signal decoded in step S66. Specifically, the reflection coefficient information restoration unit 95 finds the pixel value of the reflection coefficient image by adding, for each pixel, the pixel value of the texture image decoded in step S65 to the value of the reflection coefficient residual decoded in step S66. By performing this calculation for all pixels, the reflection coefficient information restoration unit 95 can restore the reflection coefficient image.

[0088] When there is information on a plurality of types of reflection coefficients, the reflection coefficient information restoration unit 95 restores the reflection coefficient images for each of the types of reflection coefficients.

[0089] In step S68, the 3D point cloud reconstruction unit 88 restores (reconstructs) the 3D point cloud data based on the decoded image group and the auxiliary data extracted by the auxiliary data extraction unit 79. That is, the 3D point cloud reconstruction unit 88 restores the 3D point cloud data from the occupancy image, the geometry image, the texture image, and one or more types of reflection coefficient information.

[0090] In step S69, the 3D point cloud reconstruction unit 88 outputs the reconstructed 3D point cloud data to the outside.

[0091] That is, the decoding device 60 of this embodiment inputs a V3C stream that has been coded to include reflection coefficient information, and reconstructs and outputs 3D point group data that includes the reflection coefficient information.

[0092] As described above, the decoding device 60 decodes the V3C bitstream. In this case, for the reflection coefficient, the image of the residual signal is encoded and stored in the V3C bitstream. The reflection coefficient information may include one or more types of coefficients, such as the specular reflection coefficient ks, the diffuse reflection coefficient kd, and the environmental reflection coefficient (ambient light coefficient) ka. The decoding device 60 first obtains a residual signal related to the reflection coefficient by decoding, and reconstructs an image of the reflection coefficient from the image of the residual signal and a texture image.

[0093] The V3C bitstream may store the coded residual signal image for all types of reflection coefficients. Alternatively, the V3C bitstream may store the coded residual signal image for only some types of reflection coefficients. As an example, the V3C bitstream may store the residual signal image for only the diffuse reflection coefficients, and store the specular reflection coefficient image as it is for the specular reflection coefficients. The decoding device 60 performs decoding and restores the reflection coefficient information in an appropriate manner depending on the coding method (whether or not the residual signal is used).

[0094] Next, the bit stream structure to which the reflection coefficient residual signal is added will be described.

[0095] 6 is a schematic diagram showing an example of a V3C bitstream structure with reflection coefficient residual information added. As shown in the figure, a V3C bitstream 800 includes a V3C_VPS unit 801, a V3C_AD unit 802, a V3C_OVD unit 803, a V3C_GVD unit 804, and four V3C_AVD units 805, 806, 807, and 808. Each unit is individually accessible.

[0096] The V3C_VPS unit 801 stores information indicating that the stream is a V3C stream, supplementary extension information, etc. The V3C_VPS unit 801 also contains an attribute type identifier for each of the following V3C units, which makes it possible to identify what information each of the following V3C units contains.

[0097] The V3C_AD unit 802 stores information about the effective pixel area of ​​a two-dimensional image (within a rectangular area) and information indicating the rotation of the point cloud arranged in the rectangular information. The V3C_OVD unit 803 stores occupancy map information. The V3C_GVD unit 804 stores geometry information. The V3C_AVD units 805 to 808 store various attribute information of the pixels included in the two-dimensional image. Specifically, this is as follows:

[0098] Of the four V3C_AVD units (805 to 808), for example, an encoded texture image is stored in V3C_AVD unit 805. In this case, in V3C_VPS unit 801, "ATTR_TEXTURE" (value "0") is described as an attribute type identifier in correspondence with V3C_AVD unit 805.

[0099] In addition, the V3C_AVD unit 806 includes, for example, K d The V3C_AVD unit 807 stores residual information on the coefficients (diffuse reflection coefficients). a In addition, the V3C_AVD unit 808 stores, for example, K s Residual information about the coefficient (specular reflection coefficient) is stored. d coefficient (diffuse reflection coefficient) and K a Coefficient (ambient light coefficient) and K sEach of the coefficients (specular reflection coefficients) has r·g·b (red·green·blue) components. Each of the V3C_AVD units 806, 807, and 808 can be assigned an attribute type identifier, which is undefined in the standard and has a value from 5 to 14, by redefining it as appropriate. As an example, d For the coefficients, assign the attribute type identifier value 5, and a For coefficients, assign the attribute type identifier value 6, and s The coefficient may be assigned an attribute type identifier value of 7. Note that this is not the only example, and attribute type identifier values ​​may be assigned in other ways. By using undefined IDs in this way, it is possible to indicate what kind of attribute information is stored in each V3C_AVD unit.

[0100] The information of the bitstream may be recorded and at least temporarily stored in a computer-readable recording medium (recording medium 91 in FIG. 1) having the structure of the bitstream shown in FIG. 6. Also, the bitstream shown in FIG. 6 may be transmitted via the transmission medium 92 in FIG. 1.

[0101] The recording medium 91 is a computer-readable recording medium as follows. That is, the recording medium 91 stores, as an attribute, an encoded texture video obtained by encoding a sequence of texture images representing the colors of pixels included in an image in a texture unit storage area (V3C_AVD), a texture image representing the colors of pixels included in the image, a reflection coefficient image representing the reflection coefficients of the pixels, and a sequence of reflection coefficient residual images representing residual signals obtained based on these, in one or more reflection coefficient unit storage areas (V3C_AVD) that store, as an attribute, the encoded reflection coefficient information residual video obtained by encoding the sequence of reflection coefficient residual images. It is a computer-readable recording medium including a parameter set unit storage area (V3C_VPS) having an attribute type identifier for identifying the type of each attribute stored in the texture unit and the reflection coefficient unit. Further, in the recording medium 91, each of the one or more reflection coefficient unit storage areas may store the reflection coefficient information residual video for any one of the ambient light coefficient (Ka), the diffuse reflection coefficient (Kd), and the specular reflection coefficient (Ks).

[0102] Figure 7 is a schematic diagram showing an example of the syntax of attribute information (attribute_information). This syntax is defined in a language having a block structure. For convenience, line numbers are assigned. As shown in the figure, from the first line to the fifteenth line represents the entire attribute information. The second line is numerical information (ai_attribute_count) representing the number of attributes. The number of these attributes is expressed as an unsigned 7-bit numerical value. From the third line to the fourteenth line is a syntax representing a loop (for). That is, the variable i starts from 0, increments by 1, and the block inside the for statement is repeated as long as the condition i < ai_attribute_count[atlasID] is satisfied. Among them, the fourth line is the attribute type identifier (ai_attribute_type_id). As already explained, the attribute type identifier is information for identifying the type of each individual attribute.

[0103] As shown in Fig. 7, the attribute information has information for identifying the type of each attribute, making it possible to identify the type of texture image or reflection coefficient image included in the V3C stream. The number of attributes is specified by the above ai_attribute_count and can be made variable.

[0104] FIG. 8 is a block diagram showing an example of the internal configuration of the encoding device 10 and the decoding device 60. Each device can be realized by using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized by using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0105] At least some of the functions of the encoding device 10 and the decoding device 60 in the embodiment can be realized by a computer and a program. In that case, a program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above-mentioned program may be a program for realizing some of the functions described above, and may further be a program that can realize the functions described above in combination with a program already recorded in the computer system.

[0106] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples. Note that a number of modified examples may be combined to the extent that they are combinable.

[0107] [Variation 1] As a variant, the point cloud data may not have occupancy information. In this case, the encoding device 10 may output a bitstream that does not include occupancy video. Furthermore, when such a bitstream is input, the decoding device 60 does not decode the occupancy video.

[0108] [Variation 2] As a variant, the point cloud data may not have geometry information. In this case, the encoding device 10 may output a bitstream that does not include geometry images. Furthermore, when such a bitstream is input, the decoding device 60 does not decode the geometry images.

[0109] [Variation 3] The encoding device 10 may be configured not to have, for example, the point cloud input unit 21 or the two-dimensional projection unit 22. In this case, the encoding device 10 acquires the two-dimensionally projected image (or a sequence of images) from outside and performs encoding processing.

[0110] [Variation 4] The encoding device 10 may have a configuration that does not include, for example, the bit stream multiplexing unit 39. In this case, the bit stream multiplexing process can be performed outside the encoding device 10.

[0111] [Variation 5] The decoding device 60 may have a configuration that does not include, for example, the bit stream demultiplexing unit 71. In this case, the demultiplexing process of the bit stream can be performed outside the decoding device 60.

[0112] [Variation 6] The decoding device 60 may be configured not to have, for example, the 3D point group reconstruction unit 88. In this case, outside the decoding device 60, a process of reconstructing a 3D point group can be performed based on information output from the decoding device 60.

[0113] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0114] As described above, in the V3C stream defined in ISO / IEC23090-5, a standard for volumetric video, the amount of data can be significantly compressed by encoding a dynamic point group in a time series using an existing encoding device. In the V3C stream, extended properties such as reflection coefficient information can be described in addition to existing vertex color information, but there is a risk that the bit rate of the compressed stream will increase. According to this embodiment (including the modified example), when reflection coefficient information is added as an extension of the V3C stream, encoding is performed based on a residual signal from the original vertex colors. This makes it possible to significantly compress the bit rate of the output stream. [Industrial Applicability]

[0115] The present invention can be used, for example, for recording or transmitting images. In particular, the present invention can be used for recording or transmitting visual volumetric images. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0116] 1 System 10 Encoding device 21 Point Cloud Input Section 22 2D Projection Section (Projection Section) 23 Occupancy Image Generation Unit 24 Occupancy video coding unit 25 Geometry image generator 26 Geometry video coding unit 27 Texture image generation unit 28 Texture image coding unit 29 Auxiliary Data Generation Unit 31 Residual signal generation section 32 Reflection coefficient information processing section 34 Reflection coefficient information image generator 35 Reflection coefficient information residual image coding unit 39 Bitstream multiplexing section 60 Decryption Device 71 Bitstream Demultiplexing Unit 73 Occupancy Video Decoding Unit 75 Geometry video decoding unit 77 Texture video decoding unit 79 Auxiliary Data Extraction Section 82 Reflection coefficient information processing section 84 Reflection coefficient information residual video decoding unit 85 Reflection coefficient information restoration unit 88 3D point cloud reconstruction unit 91 Recording media 92 Transmission Media 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a residual signal generating unit that generates a residual signal based on a texture image that represents a color of a pixel included in an image and a reflection coefficient image that represents a reflection coefficient of the pixel; a reflection coefficient information residual image coding unit for coding a sequence of images representing the residual signal of the pixels and outputting a coded reflection coefficient information residual image; a texture image encoder for encoding the sequence of texture images and outputting an encoded texture image; a bitstream multiplexing unit that generates a bitstream including at least the encoded reflection coefficient information residual image output from the reflection coefficient information residual image encoding unit and the encoded texture image output from the texture image encoding unit; An encoding device comprising:

2. a point cloud input unit that acquires point cloud data including at least vertex color information and reflection coefficient information regarding a point cloud, which is a group of points having three-dimensional coordinates; a projection unit that projects the point cloud onto the two-dimensional image, outputs the texture image based on the vertex color information of the point cloud data, outputs the reflection coefficient image based on the reflection coefficient information of the point cloud data, and outputs information regarding pixel regions in the image; an auxiliary data generating unit that generates auxiliary data based on information about a pixel region in the image output by the projection unit; Further equipped with The bit stream multiplexing unit further generates the bit stream including the auxiliary data generated by the auxiliary data generating unit. The encoding device according to claim 1 .

3. a bitstream demultiplexing unit that demultiplexes an input bitstream and outputs at least a coded reflection coefficient information residual image and a coded texture image; a texture image decoder for decoding the encoded texture image to output a sequence of texture images representing colors of pixels contained in the image; a reflection coefficient information residual video decoder for decoding the encoded reflection coefficient information residual video to output a sequence of reflection coefficient residual images representing residual signals related to the reflection coefficients of the pixels; a reflection coefficient information restoration unit that restores and outputs a sequence of reflection coefficient images representing reflection coefficients of the pixels based on the decoded texture image and the decoded reflection coefficient residual image; A decoding device comprising:

4. an auxiliary data extraction unit that extracts auxiliary data, which is information regarding pixel regions within an image, based on information passed from the bitstream demultiplexing unit; a three-dimensional point cloud reconstruction unit that reconstructs point cloud data relating to a point cloud, which is a group of points having three-dimensional coordinates, based on the auxiliary data extracted by the auxiliary data extraction unit, the texture image output by the texture video decoding unit, and the reflection coefficient image output by the reflection coefficient information restoration unit; The decoding device of claim 3 further comprising:

5. a residual signal generating unit that generates a residual signal based on a texture image that represents a color of a pixel included in an image and a reflection coefficient image that represents a reflection coefficient of the pixel; a reflection coefficient information residual image encoding unit for encoding a sequence of images representing the residual signal of the pixels and outputting an encoded reflection coefficient information residual image; a texture image encoder for encoding the sequence of texture images and outputting an encoded texture image; a bitstream multiplexing unit that generates a bitstream including at least the encoded reflection coefficient information residual image output from the reflection coefficient information residual image encoding unit and the encoded texture image output from the texture image encoding unit; A program for causing a computer to function as an encoding device comprising the above.

6. a bitstream demultiplexing unit that demultiplexes an input bitstream and outputs at least a coded reflection coefficient information residual image and a coded texture image; a texture image decoder for decoding the encoded texture image to output a sequence of texture images representing colors of pixels contained in the image; a reflection coefficient information residual video decoder for decoding the encoded reflection coefficient information residual video to output a sequence of reflection coefficient residual images representing residual signals related to the reflection coefficients of the pixels; a reflection coefficient information restoration unit that restores and outputs a sequence of reflection coefficient images representing reflection coefficients of the pixels based on the decoded texture image and the decoded reflection coefficient residual image; A program for causing a computer to function as a decoding device comprising the above.

7. a texture unit storage area for storing, as an attribute, an encoded texture image obtained by encoding a sequence of texture images representing the colors of pixels included in the image; one or more reflection coefficient unit storage areas for storing, as attributes, coded reflection coefficient information residual images obtained by coding a sequence of reflection coefficient residual images representing a residual signal obtained based on a texture image representing the color of a pixel included in an image and a reflection coefficient image representing the reflection coefficient of the pixel; a parameter set unit storage area having an attribute type identifier for identifying the type of an attribute stored in each of the texture unit storage area and the reflection coefficient unit storage area; Equipped with Each of the one or more reflection coefficient unit storage areas stores the reflection coefficient information residual image for any one of an ambient light coefficient (Ka), a diffuse reflection coefficient (Kd), and a specular reflection coefficient (Ks); A computer-readable recording medium.

Images