Image processing device, decoding device, encoding device, image processing method, decoding method, encoding method, and program

Abstract

This technology provides a method for compressing multiple elements, each having attribute information containing a variable number of parameters, into point cloud data, such as 3DGS represented by Gaussian primitives with adaptive SH order. [Solution] Compressed data of element group data is obtained, wherein the element group data represents the respective positions of multiple elements and the attribute information of each of the multiple elements, the attribute information indicates at least a scale, the attribute information includes a variable number of parameters, and the compressed data is configured to indicate the number of parameters that each of the multiple elements' attribute information has. The compressed data is decompressed by referring to the number of parameters that each of the multiple elements' attribute information has obtained based on the compressed data.

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an encoding apparatus, an image processing method, an encoding method, and a program, and more particularly to a method for encoding and decoding element group data. [Background technology]

[0002] In recent years, technologies for modeling three-dimensional scenes or objects based on multiple images have been actively explored. Using these generated models, it is possible to create virtual viewpoint images from any viewpoint specified by the user. As a new representation model for three-dimensional scenes or objects, 3D Gaussian Splatting (3DGS) has been proposed (Non-Patent Literature 1). In 3DGS, a three-dimensional scene or object is represented using a set of three-dimensional Gaussian distributions with defined opacity and color. A virtual viewpoint image is generated by rasterizing the three-dimensional scene or object represented in this way against a projection plane corresponding to the virtual viewpoint. This method yields better results than conventional methods in terms of learning speed during model construction, image quality, and rendering speed.

[0003] 3DGS primitives (hereinafter referred to as Gaussian primitives, or simply Gauss) are represented by the mean (3 values), covariance, opacity (1 value), and color information for each direction (48 values) of a Gaussian distribution. The covariance includes a quaternion representing the orientation and a scale (magnitude) representing the standard deviation. The color information for each direction is represented by Spherical Harmonics (SH) coefficients. SH is a model that responds with energy according to the coefficients for each direction. By treating SH as the luminance of each RGB channel, color according to direction is represented. The order of SH is a parameter that controls the complexity of the response to direction. By using a higher order, complex changes in the response to direction can be represented. In general, SH coefficients up to order 3 are often used. 3DGS is obtained by repeatedly adding and deleting Gaussian primitives and updating (optimizing) the values ​​of Gaussian primitives. During optimization, the pixel values ​​of the multiple images mentioned above are used as training data. The difference between the training data and the pixel values ​​of the image obtained by rendering 3DGS is used as the loss.

[0004] Using 3DGS makes it possible to generate high-quality virtual viewpoint images. On the other hand, a very large amount of Gaussian encoding is required to maintain the quality of 3DGS. For example, modeling a space such as a park, rather than individual objects, may require 1.4GB of data per frame. To enable faster transmission and data storage, it is necessary to reduce the required storage space. For this reason, the development of compression technologies specifically for 3DGS is underway. MPEG (ISO / IEC JTC1 SC29 WG2-WG8) has established an ad hoc group (GSC) to standardize compression technologies for static or dynamic 3DGS.

[0005] As a means of reducing the memory usage of 3DGS, a method of adaptively reducing the redundant SH order has been reported (Non-Patent Literature 2). As mentioned above, the SH coefficients account for about 80% of the parameters representing the Gaussian primitive. Furthermore, higher-order coefficients are more numerous. Non-Patent Literature 2 reports that about 90% of the colors of a Gaussian primitive can be represented by only the 0th-order component (3 values). Non-Patent Literature 2 discloses that by adaptively reducing the higher-order components of SH in this way, memory usage can be reduced to about one-third. Furthermore, Non-Patent Literature 2 reports that by combining the deletion and quantization of Gaussian primitives that do not contribute to rendering, overall memory usage was reduced to 1 / 27 without significantly degrading image quality (PSNR).

[0006] Data volume is a major challenge in 3DGS rendering. While data compression can reduce the file size, it needs to be decompressed during rendering, returning the data size to its original size. Therefore, it is necessary not only to compress the file size but also to reduce the memory usage after decompression (especially the VRAM memory footprint). [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Patent No. 7466849 [Non-patent literature]

[0008] [Non-Patent Document 1] B. Kerbl et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering", arXiv:2308.04079 (2023). [Non-Patent Document 2] P. Papantonakis et al. "Reducing the Memory Footprint of 3D Gaussian Splatting", arXiv:2406.17074 (2024).

Summary of the Invention

Problems to be Solved by the Invention

[0009] For the compression of 3DGS, it is conceivable to apply Geometry based Point Cloud Compression (G-PCC), which is a point cloud compression method with normalized attribute information. In G-PCC, efficient compression is achieved by converting the point cloud into hierarchical voxels represented by an octree. The details of this technology are disclosed in Patent Document 1. Since a Gaussian has a structure similar to a point cloud, it is possible to compress based on the G-PCC standard by regarding the mean of the Gaussian as the position of the point and other parameters as attribute information.

[0010] However, conventional point cloud compression techniques including G-PCC do not consider the case where the number of parameters of attribute information (e.g., the number of information or the dimension of each information) is different for each point. Therefore, conventional point cloud compression techniques do not handle the case where each of a plurality of points has a variable number of parameters as attribute information. For example, when adaptively reducing the degree of SH and the coefficients of SH have an adaptive degree, 3DGS cannot be compressed using conventional techniques.

[0011] Also, by using a memory-saving representation of 3DGS, such as the method disclosed in Non-Patent Document 2 or other methods, the memory usage during rendering can be suppressed. However, there are cases where a memory-saving representation of 3DGS is not desired. For example, currently, most of the implemented 3DGS renderers only support conventional 3DGS. That is, it is assumed that there are cases where rendering based on extended 3DGS cannot be performed because the renderer does not support the memory-saving representation of 3DGS.

[0012] One embodiment of the present invention provides a technique for compressing a plurality of elements each having attribute information including variable parameters as element group data, such as 3DGS represented by a Gaussian primitive having an appropriate SH degree. Another embodiment of the present invention provides a technique for restoring the extended element group data to a conventional format as needed. **Means for Solving the Problems**

[0013] Decoding according to one embodiment of the present invention has the following configuration. That is, compression data of element group data, wherein the element group data represents the position of each of a plurality of elements and the attribute information of each of the plurality of elements, the attribute information at least indicates the coefficients of a model that responds to colors with respect to directions, the attribute information includes variable parameters, and the compression data is configured to indicate the number of the parameters included in the attribute information of each of the plurality of elements, an acquisition means for acquiring the compression data, and an expansion means for expanding the compression data by referring to the number of the parameters included in the attribute information of each of the plurality of elements acquired based on the compression data. A decoding device characterized by comprising these is provided. **Advantages of the Invention**

[0014] One embodiment of the present invention can compress a plurality of elements each having attribute information including variable parameters as element group data, such as 3DGS represented by a Gaussian primitive having an appropriate SH degree. Another embodiment of the present invention can restore the extended element group data to a conventional format as needed. **Brief Description of the Drawings**

[0015] [Figure 1] A block diagram showing a configuration example of an encoding device according to one embodiment. [Figure 2] A flowchart showing the flow of an encoding method according to one embodiment. [Figure 3] A flowchart showing the flow of an optimization process. [Figure 4] A flowchart illustrating the rasterization process. [Figure 5] A flowchart illustrating the model compression process. [Figure 6] A schematic diagram illustrating an example of a partitioned Gaussian group. [Figure 7] A diagram illustrating an example of metadata. [Figure 8] A diagram showing the changes in data representing the Gaussian group. [Figure 9] A diagram explaining the rasterization process. [Figure 10] This figure shows an example of the seq_parameter_set used by SH when compressing a third-order Gaussian group. [Figure 11] A block diagram showing an example configuration of a decoding device according to one embodiment. [Figure 12] A flowchart illustrating the flow of a decoding method according to one embodiment. [Figure 13] A flowchart illustrating the metadata analysis process. [Figure 14] A flowchart illustrating the process of decompression. [Figure 15] A block diagram showing an example of a computer hardware configuration. [Figure 16] A block diagram showing an example configuration of an encoding device according to the second embodiment. [Figure 17] A flowchart showing the flow of the encoding method according to the second embodiment. [Figure 18] A block diagram showing an example configuration of a decoding device according to the second embodiment. [Figure 19] A flowchart showing the flow of the decoding method according to the second embodiment. [Figure 20] A diagram illustrating an example of a data structure within a file. [Modes for carrying out the invention]

[0016] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0017] [First Embodiment] <Encoding device> First, an image processing device 101, which is an encoding device according to one embodiment of the present invention, will be described. Figure 1 is a block diagram showing the functional parts of the image processing device 101.

[0018] The model compression unit 106 compresses the point cloud data. This point cloud data represents the position of each of the multiple points and the attribute information of each of the multiple points. Here, the attribute information can indicate at least the coefficients of a model that responds to color with respect to direction. The attribute information can also include a variable number of parameters. The attribute information may include a variable number of parameters that indicate the coefficients of a model that responds to color with respect to direction. Such point cloud data can represent, for example, 3DGS data (Gaussian primitive group) when the SH coefficients have an adaptive order. More specifically, the Gaussian mean corresponds to the geometry of the points, and the other components correspond to the attribute information of the points. In this case, the model that responds to color with respect to direction is defined by Spherical Harmonics (SH). The coefficients of the model that responds to color with respect to direction correspond to the SH coefficients. That is, the model compression unit 106 can compress the point cloud data in such a way that it compresses attribute information that includes one or more of the following: opacity of each point, standard deviation (scale) that defines the magnitude of the spread, orientation that defines the direction of the spread, and SH coefficients that define the color. In this embodiment, a group of Gaussian primitives is treated as a point cloud, and point cloud data is generated by compressing the point cloud. However, the method described herein is not limited to this implementation example and can be used to generate element group data by compressing an element group, which is a collection of elements that have positional information and attribute information such as scale. Elements are not limited to points and include forms such as voxels or Gaussian structures. Furthermore, the element group includes point clouds, voxel groups, and Gaussian groups.

[0019] Furthermore, the model compression unit 106 can configure point cloud data to indicate the number of parameters each of the multiple points possesses. This configuration facilitates the decompression of the compressed data.

[0020] Furthermore, the model compression unit 106 can divide the point cloud data into multiple point cloud data sets according to the number of parameters included in the attribute information. The model compression unit 106 can then compress each of the divided point cloud data sets. With this configuration, the compressed point cloud data can be represented in a simpler format. Alternatively, the model compression unit 106 may divide the point cloud data into multiple point cloud data sets according to the positional relationship of multiple points.

[0021] The following describes an embodiment in which point cloud data (element group data) represents a Gaussian primitive group. Hereafter, point cloud data is synonymous with a Gaussian primitive group (sometimes abbreviated as a Gaussian group). In this embodiment, the image acquisition unit 102, the initial point cloud acquisition unit 103, and the model optimization unit 104 acquire point cloud data to be compressed. In this embodiment, these processing units generate 3DGS data based on the captured image. The model compression unit 106 adaptively reduces the SH order for each of the multiple points. The model compression unit 106 then compresses the point cloud data after the SH order has been reduced.

[0022] The functions of each processing unit are described in more detail below. The image acquisition unit 102 acquires one or more captured images obtained by photographing a scene or object. The image acquisition unit 102 outputs the acquired images as training images to the next stage.

[0023] The initial point cloud acquisition unit 103 calculates a point cloud representing a scene or object based on the training image. For example, the initial point cloud acquisition unit 103 can extract two-dimensional feature points from the training image. Then, based on the feature points in each training image, the initial point cloud acquisition unit 103 calculates camera parameters and distortion parameters corresponding to the training image, as well as a three-dimensional colored point cloud. The initial point cloud acquisition unit 103 outputs the obtained data to subsequent stages. For example, the initial point cloud acquisition unit 103 can output the colored point cloud to the model optimization unit 104. The initial point cloud acquisition unit 103 can also perform distortion correction on the training image according to the distortion parameters. Then, the initial point cloud acquisition unit 103 can output the distortion-corrected training image and camera parameters to the model optimization unit 104 and the model compression unit 106.

[0024] The model optimization unit 104 generates multiple Gaussian primitive groups representing a scene or object through optimization. First, the model optimization unit 104 can generate an initial Gaussian group from the colored point cloud acquired from the initial point cloud acquisition unit 103. Then, the model optimization unit 104 optimizes the Gaussian group using the training image as training data. During the optimization process, the model optimization unit 104 outputs the 3DGS data represented by the Gaussian group being optimized, and camera parameters corresponding to the training image, to the drawing unit 105. The model optimization unit 104 also acquires a drawing image from the drawing unit 105 to be used to calculate the loss. Finally, the model optimization unit 104 outputs the 3DGS data represented by the optimized Gaussian group to the model compression unit 106.

[0025] The drawing unit 105 renders an image by rasterizing the 3DGS data according to the camera parameters obtained from the model optimization unit 104. The drawing unit 105 then outputs the generated rendered image to the model optimization unit 104.

[0026] The model compression unit 106 compresses the optimized 3DGS data, represented by the Gaussian group, obtained from the model optimization unit 104. The model compression unit 106 outputs the obtained compressed data (compressed 3DGS data) to the output unit 107.

[0027] The output unit 107 outputs the compressed data acquired from the model compression unit 106 to the outside of the image processing device 101.

[0028] Next, the encoding process performed by the image processing device 101 will be described with reference to Figure 2. Figure 2 is a flowchart of the encoding process according to this embodiment. In this embodiment, moving image data is input in frame units. Alternatively, still image data for one frame may be input. Furthermore, intra predictive coding is used in the processing of this embodiment. On the other hand, this embodiment is also applicable when inter predictive coding is used.

[0029] In S201, the image acquisition unit 102 acquires image data (training images) from outside the image processing device 101. However, the method of acquiring the image data is not particularly limited. For example, the image acquisition unit 102 itself may take the images. Alternatively, the image acquisition unit 102 may generate images using computer graphics (CG) technology.

[0030] In S202, the initial point cloud acquisition unit 103 extracts image feature points from the training images. Then, based on the feature points, the initial point cloud acquisition unit 103 analyzes the correspondence between feature points in the training images. Based on the correspondence between feature points, the initial point cloud acquisition unit 103 calculates camera parameters (position, orientation, focal length, and principal point, etc.) and distortion parameters (coefficients of the lens distortion model) corresponding to each training image. In this specification, the viewpoint of the training image represented by the camera parameters corresponding to the training image may be called the input viewpoint. The initial point cloud acquisition unit 103 also calculates a colored three-dimensional point cloud corresponding to the extracted image feature points. The initial point cloud acquisition unit 103 can perform the processing in S202 using any method for reconstructing a three-dimensional scene or object from images from multiple viewpoints. In this embodiment, the initial point cloud acquisition unit 103 uses COLMAP. On the other hand, the initial point cloud acquisition unit 103 may use other methods such as Visual SFM.

[0031] In step S203, the initial point cloud acquisition unit 103 corrects the training image based on the obtained distortion parameters. In this way, the effect of lens distortion can be removed from the training image. By correcting the lens distortion in advance, processing that takes lens distortion into account in subsequent processing can be omitted. However, this step is not necessary when using a distortion-free training image created with computer graphics.

[0032] In S204, the model optimization unit 104 and the drawing unit 105 perform Gaussian group optimization. For this purpose, the model optimization unit 104 generates an initial Gaussian group from the colored point cloud. The model optimization unit 104 also uses the pixel values ​​of the distortion-corrected training image as training data for optimization. In S204, the usual 3DGS data generation process described in Non-Patent Literature 1 can be used. Details of the processing in this embodiment will be described later.

[0033] In S205, the model compression unit 106 compresses the optimized Gaussian group. Details of the process will be described later.

[0034] In S206, the output unit 107 outputs the compressed data to the outside of the image processing apparatus 101.

[0035] Next, the flow of the optimization process in S204 will be described in detail with reference to FIG. 3. First, the camera parameters and the specific parameters constituting the Gaussian primitive are defined. [ ] represents the number of dimensions. [a] represents an a-dimensional vector. [a×b] represents a matrix of a×b. Also, the absence of [ ] means a scalar.

[0036] (Definition of Camera Parameters) Rotation matrix: R view [3×3] Position: t[3] Focal length: f x , f y Principal point: c x , c y Image size: w, h

[0037] (Definition of Gaussian Primitive) Mean of Gaussian: p[3] Orientation of Gaussian: r[4]=(q r ,q i ,q j ,q k ) Standard deviation of Gaussian: s[3] Opacity: o 0th coefficient of color SH: f dc [3] 1st coefficient of color SH: f rest1 [9] 2nd coefficient of color SH: f rest2

[15] 3rd coefficient of color SH: f rest3

[21]

[0038] Next, the processing flow will be explained. In S301, the model optimization unit 104 generates an initial Gaussian group from the colored point cloud. The model optimization unit 104 can create a Gaussian primitive corresponding to each point included in the colored point cloud, for example, as follows: The mean p is set to the position of the point. The orientation r is set to a fixed value [1,0,0,0]. The standard deviation s is set to the average of the distances to the three nearest points. The opacity o is set to a fixed value of 0.1. The zeroth coefficient f of the color SH dc The value obtained from the zeroth-order inverse transform of SH is set as the value. For example, the zeroth-order coefficient f of the R channel. dc,r It is calculated as follows: f dc,r =2(r-0.5)√π Furthermore, the coefficient f of the higher-order SH rest1 ~f rest3 This value is set to 0.

[0039] In S302, the model optimization unit 104 selects a learning viewpoint. The model optimization unit 104 can select one camera parameter and one corresponding training image. The selection can be done randomly.

[0040] In S303, the drawing unit 105 rasterizes the Gaussian group based on the camera parameters of the selected learning viewpoint. In this way, the drawing unit 105 generates a drawn image. Details of the rasterization will be described later.

[0041] In S304, the model optimization unit 104 calculates the loss based on the difference between the training image and the drawn image. Then, the model optimization unit 104 updates each Gauss in the Gaussian group based on backpropagation. Adam can be used to calculate the gradient. The loss can be defined as follows: L = (1 - λ)L1 + λL D-SSIM L1 is the average of the absolute differences in pixel values, i.e., the L1 loss. D-SSIM This is an SSIM that corresponds to floating-point numbers. λ is assumed to be 0.2 here.

[0042] In S305, the model optimization unit 104 checks whether it is time to update the Gaussian density (i.e., Gaussian primitive density). If it is time to update, the process proceeds to S306. Otherwise, the process proceeds to S307. In this embodiment, the number of loops for the optimization process that updates the Gaussian density is defined in advance. The model optimization unit 104 determines that it is time to update if the number of loops is the number defined in advance.

[0043] In S306, the model optimization unit 104 updates the Gaussian density. The model optimization unit 104 can delete Gaussian primitives with low opacity. This process deletes transparent Gaussian primitives that have little impact on the rendering result. In addition, the model optimization unit 104 can duplicate or split Gaussian primitives in areas with large rendering errors.

[0044] In S307, convergence is determined. Here, the number of optimization iterations is counted, and the value is set to "Yes" when it reaches a predetermined value. However, other methods, such as monitoring the loss trend, can also be used to determine convergence.

[0045] <Processing flow of the drawing unit 105> Next, the flow of the rasterization process in S303 will be explained with reference to Figure 4. In S401, the drawing unit 105 culls Gaussian primitives outside the field of view. An example of the process is shown below.

[0046] Projection matrix M required for projection prj This is calculated in advance as shown in the following formula.

number

[0047] The drawing unit 105 projects the three-dimensional Gaussian average onto the two-dimensional coordinate system on the drawn image (i.e., the camera's image coordinate system). The calculation formula is as follows.

number

[0048] In S402, the drawing unit 105 converts the three-dimensional Gaussian variance into a two-dimensional Gaussian variance by projecting it onto the camera's image coordinate system based on the camera parameters. First, the drawing unit 105 calculates the three-dimensional Gaussian covariance matrix Σ 3D We seek.

number

number

[0049] Then, the drawing unit 105 displays the covariance matrix Σ 3D By projecting it, the two-dimensional covariance matrix Σ on the drawn image is obtained. 2D Obtain it.

number

[0050] The processes S401 and S402 are performed for each Gaussian primitive. Below, each projected Gaussian primitive, represented by the mean and variance of the two-dimensional Gaussian, is referred to as the projected Gaussian.

[0051] In S403, the drawing unit 105 separates the drawing image to be drawn into 16x16 [pix] tiles. Therefore, the number of tiles is [w / 16] × [h / 16]. The size of the tiles is not particularly limited and does not necessarily have to be 16. By setting a value suitable for the hardware performing the calculations, it is possible to expect faster processing.

[0052] In S404, the drawing unit 105 detects the tiles that overlap each projected Gaussian. The drawing unit 105 then creates a list containing pairs of projected Gaussians, the tiles that overlap the projected Gaussians, and the depth of the projected Gaussian relative to the camera. The drawing unit 105 can duplicate projected Gaussians in the list. The drawing unit 105 can create a list of the same number of projected Gaussians as the number of tiles that overlap the projected Gaussian.

[0053] For example, Figure 9(A) shows three projected Gaussian curves projected onto an image plane containing four tiles. The drawing unit 105 first creates a list containing pairs of projected Gaussian curves (PG) and depth values, as shown in Figure 9(B). The drawing unit 105 also creates the list shown in Figure 9(C) by duplicating the projected Gaussian curves.

[0054] In S405, the drawing unit 105 sorts the list created in S404. The drawing unit 105 can sort the depth list with tile IDs shown in Figure 9(C) in ascending order, using the tile ID as the 1st key and the depth as the 2nd key. The sorted list is shown in Figure 9(D). Through the above process, projection information can be aggregated for each tile. In addition, the list shown in Figure 9(D) has the projected Gaussian curves arranged in descending order of depth. Therefore, it becomes easy to perform alpha blending starting from the projected Gaussian curves on the near side of the camera.

[0055] In S406, the drawing unit 105 identifies the reference range in the list for each tile. The drawing unit 105 can refer to the sorted depth list with tile IDs shown in Figure 9(D) and identify the data range on the list for each tile ID.

[0056] In S407, the drawing unit 105 initializes the image buffer. The image buffer is a 4-channel buffer with w x h pixels. The drawing unit 105 can set all values ​​in the image buffer to 0.

[0057] S408 to S412 are tile-by-tile loops. S409 to S411 are loops for each 16x16 pixel within a tile. In S408, the drawing unit 105 selects one tile to be processed (the tile to be processed). In S409, the drawing unit 105 selects one pixel to be processed (the pixel to be processed) from among the tiles to be processed.

[0058] In S410, the drawing unit 105 forms the pixel value of the pixel to be processed. The drawing unit 105 can form the pixel value by α-blending the projected Gaussians that overlap the tile to be processed in order from the front. α-blending can be performed according to the following equation: N represents the number of projected Gaussians that overlap the tile to be processed. i and j represent the numbers of the projected Gaussians after sorting in ascending order based on depth. c represents the response from SH and corresponds to the color obtained according to the Gaussian primitive. α represents opacity.

number

[0059] By repeating the processes from S408 to S412, each tile is rendered and a drawn image is generated.

[0060] <Processing flow of model compression unit 106> Next, the processing flow of the model compression unit 106 will be explained with reference to Figure 5. In S501, the model compression unit 106 removes Gaussian graphs from the input Gaussian graph group that have little impact on the rendering result, such as those with low opacity. As a method of removal, for example, the method described in Non-Patent Literature 2 can be used.

[0061] In this embodiment, the model compression unit 106 calculates the resolution r [mm / pix] of the Gaussian of interest in the training image corresponding to the input viewpoint with the highest resolution among the input viewpoints that include the Gaussian of interest within the field of view. The model compression unit 106 then calculates the number of Gaussians within a radius r centered on the Gaussian of interest as redundancy for the Gaussian of interest. The model compression unit 106 then determines a redundancy threshold based on the mean μ and standard deviation σ of the redundancy calculated for all Gaussians. The model compression unit 106 then deletes Gaussians with redundancy exceeding the threshold.

[0062] On the other hand, the deletion method is not particularly limited. Alternatively, the model compression unit 106 may delete Gaussian blends in the alpha blending performed when generating the rendering image, where the maximum contribution rate to the pixel values ​​of the rendering image corresponding to each input viewpoint is less than a threshold. Furthermore, performing the processing in S501 is not mandatory.

[0063] In S502, the model compression unit 106 obtains encoding parameters from the application. The encoding parameters include a variable-order flag, a Gaussian group partitioning flag, and an uncompressed mode flag.

[0064] In S503, the model compression unit 106 determines whether the variable order flag is ON or OFF. The variable order flag is set as a parameter for the model compression process. If it is determined that the variable order flag is ON, the process proceeds to S504. Otherwise, the process proceeds to S506.

[0065] In S504, the model compression unit 106 performs a process to reduce the order of each Gaussian SH. As a reduction method, for example, the method described in Non-Patent Document 2 can be employed.

[0066] In this embodiment, the model compression unit 106 reduces the order of the SH that defines Gaussian color when the impact of reducing the SH order is small. For example, the model compression unit 106 calculates the variance of Gaussian pixel values ​​in the drawn image corresponding to the input viewpoint where the contribution rate of the Gaussian color of interest is high. The contribution rate is expressed by the following formula.

number

[0067] Alternatively, the model compression unit 106 may reduce the order of SH by one step at a time. In this case, the model compression unit 106 can remove bands that have little impact on the loss to the training image. Alternatively, the model compression unit 106 can calculate the distance d between the color of the Gaussian of interest calculated using SH up to order 3 in the drawing image corresponding to each input viewpoint and the Gaussian of interest calculated using SH up to order q (q=0,1,2). q The model compression unit 106 then calculates the distance d evaluated for each input viewpoint. q The average (or weighted average) of the values ​​is calculated. If the obtained value is below a threshold, the model compression unit 106 can reduce SH up to the qth order. Thus, the method for reducing the order of SH is not particularly limited.

[0068] In this embodiment, the Gaussian group before compression is represented using SH up to the third order. On the other hand, as a result of the processing in S504, the order of SH for each Gaussian becomes 0, 1, 2, or 3. In other words, the order of SH for each Gaussian is variable. Furthermore, each Gaussian has a variable number of SH coefficients. That is, each Gaussian has a variable number of parameters.

[0069] In S505, the model compression unit 106 divides the Gaussian group according to the number of parameters included in the attribute information. In this embodiment, the model compression unit 106 separates the Gaussian group by order. That is, the model compression unit 106 separates the Gaussian group after the order of SH has been reduced into four Gaussian groups corresponding to orders 0, 1, 2, and 3, respectively. Here, the model compression unit 106 divides the Gaussian group for each order i (i=0,1,2,3) to create a Gaussian group i corresponding to order i.

[0070] In S506, the model compression unit 106 determines whether the Gaussian group separation flag is ON or OFF. The Gaussian group separation flag is set as a parameter for the model compression process. If it is determined that the Gaussian group separation flag is ON, the process proceeds to S507. Otherwise, the process proceeds to S508.

[0071] In S507, the model compression unit 106 separates Gaussian groups according to the positional relationships of multiple Gaussians. In this embodiment, the model compression unit 106 separates Gaussian groups based on the distance between Gaussians. For example, the model compression unit 106 can cluster Gaussian groups based on the distance between Gaussians and separate Gaussian groups for each cluster. If the Gaussian groups have already been separated in S505, the processing in S507 can be applied independently to each of the separated Gaussian groups. For example, the processing in S507 can be performed on each of the Gaussian groups i separated in S505. By performing the separation process on Gaussian group i, Gaussian groups i,j (j=0,···,J; J is the number of clusters) are obtained. Alternatively, the processing in S507 may be performed on the Gaussian groups after processing in S501.

[0072] For example, the model compression unit 106 can connect Gaussians whose mean distances are less than or equal to a threshold td using edges. Gaussians connected by edges are treated as a single cluster. Then, the Gaussian group is separated for each cluster. In this way, the model compression unit 106 can extract spatially continuous Gaussian primitives as a single block. This processing suppresses the sparseness of points when compressing the Gaussian group as a point cloud. Therefore, an improvement in compression ratio is expected. However, performing the processing in S507 is not mandatory.

[0073] Figure 6 shows the clustering results when the variable-order flag and the Gaussian group separation flag are ON. Each solid circle represents a single Gaussian group. The order is indicated inside each circle. Furthermore, Gaussian groups that are close to each other are connected by nodes. The final clusters are represented by dashed lines. Each cluster is assigned a Gaussian group name. The first number in the name represents the order, and the second number represents the index within clusters of the same order.

[0074] In S508, the model compression unit 106 compresses the Gaussian groups. The model compression unit 106 can compress one or more Gaussian groups individually. In this embodiment, the model compression unit 106 can compress each of the Gaussian groups separated in S505 and / or S507. As described above, each Gaussian group can be represented as point cloud data. Therefore, the model compression unit 106 can compress each of the separated Gaussian groups using point cloud data compression techniques. The specific compression technique is not particularly limited. In this embodiment, the model compression unit 106 compresses the Gaussian groups according to G-PCC.

[0075] The model compression unit 106 creates one or more independent bitstreams by compressing one or more Gaussian groups. According to the method of this embodiment, the number of parameters (i.e., the coefficients of SH) for each Gaussian in a single Gaussian group is the same. That is, since the number of parameters that have as attribute information is the same in a single Gaussian group, point cloud data can be compressed using conventional compression algorithms.

[0076] Furthermore, the model compression unit 106 may perform only quantization instead of compression using G-PCC for Gaussian groups with fewer elements than the element threshold tn. This is because improvement in compression ratio based on spatial correlation cannot be expected. Such switching of the compression method according to the number of elements may be performed when the uncompressed mode flag, which is set as a parameter of model compression, is ON.

[0077] The following describes a method for compressing Gaussian groups using G-PCC. In G-PCC, the number of attribute information attached to a point cloud is defined by seq_parameter_set (SPS). The SPS records the number of attribute information (num_attributes), the number of dimensions of each attribute information (attr_components_minus1), and signals representing the type (attr_label_known, attr_label, attr_label_oid). Thus, the model compression unit 106 can describe information indicating the number of parameters that the Gaussian group has (for example, the number of dimensions of SH) in the parameter set of the compressed data for each Gaussian group, such as the SPS, or in metadata recorded in the file format.

[0078] When compressing the color SH of the 0th order Gaussian group, the model compression unit 106 sets num_attributes in seq_parameter_set to 4. The model compression unit 106 also sets opacity o, Gaussian standard deviation s, Gaussian orientation rot, and the 0th order coefficient f of the color SH. dcThe attr_components_minus1 for each component is set to 0, 2, 3, and 2. The model compression unit 106 then compresses the Gaussian group as a point cloud with four attribute pieces of information. Since opacity is an attribute piece defined in G-PCC, the model compression unit 106 sets the attr_label_known for opacity to 1 and sets the attr_label to identifier 2, which represents opacity. Since the other attribute pieces are not defined in G-PCC, the model compression unit 106 sets the attr_label_known for the other attribute pieces to 0 and assigns different identifiers to attr_label_oid for each piece.

[0079] When the color SH compresses the first-order Gaussian group, one attribute is added. That is, the model compression unit 106 sets num_attributes to 5. Also, the model compression unit 106 sets the first-order coefficient f of the color SH. rest1 Set attr_components_minus1 to 8.

[0080] When the color SH compresses a second-order Gaussian group, the model compression unit 106 sets num_attributes to 6. The model compression unit 106 also sets the second-order coefficient f of the color SH. rest2 Set attr_components_minus1 to 14. If the color SH compresses a third-order Gaussian group, the model compression unit 106 sets num_attributes to 7. Also, the third-order coefficient f of the color SH rest3 Set attr_components_minus1 to 20. Figure 10 shows an example of specific values ​​for seq_parameter_set when SH compresses a third-order Gaussian group.

[0081] In this embodiment, the number of attribute information (num_attributes) is increased by one for each additional order of SH. However, other configurations can be adopted. For example, each SH coefficient can be considered as attribute information with three dimensions: RGB (or YUV). In this case, if the SH of the color compresses a first-order Gaussian group, three attribute information pieces are added. That is, the model compression unit 106 sets num_attributes to 7. The model compression unit 106 also sets the first-order coefficient f of the SH of the color. rest1 Set attr_components_minus1 to 2. If the color SH compresses a second-order Gaussian group, five more attribute pieces are added. That is, the model compression unit 106 sets num_attributes to 12. In this case as well, the model compression unit 106 sets the second-order coefficient f of the color SH. rest2 Set attr_components_minus1 to 2. If the SH of the color is a third-order Gaussian group, seven more attribute pieces are added. That is, the model compression unit 106 sets num_attributes to 19. In this case as well, the model compression unit 106 sets the third-order coefficient f of the SH of the color. rest3 Set `attr_components_minus1` to 2. In G-PCC, attribute information with more than 3 dimensions is compressed in RAW format. On the other hand, with this method, the number of dimensions for each element becomes 3 or less. Therefore, compression efficiency can be improved by using attribute information compression mechanisms such as RAHT.

[0082] In S509, the model compression unit 106 creates metadata for the compressed Gaussian groups. By using such metadata, multiple Gaussian groups can be managed as a single data at the file format level. This metadata can indicate the number of parameters contained in the attribute information of the Gaussian group. For example, the metadata can indicate the SH order for the Gaussian group. Figure 7 shows an example of such metadata. First, the model compression unit 106 writes the variable order flag, the number of Gaussian groups, and the uncompressed mode flag in order. The number of Gaussian groups indicates the number of Gaussian groups after separation. If the variable order flag and the Gaussian group separation flag are OFF and the Gaussian groups are not separated, the number of Gaussian groups is 0. The variable order flag is flag information that indicates that the number of parameters in the attribute information of each of the multiple points is variable. Next, if the variable order flag is ON, the model compression unit 106 describes the order and the number of elements (number of Gaussian primitives) contained in the Gaussian group for each Gaussian group. Furthermore, if the variable order flag is OFF, the model compression unit 106 first writes the order, and then sequentially writes the number of elements contained in each Gaussian group.

[0083] In S510, the model compression unit 106 serializes the bitstreams of each compressed Gaussian group and stores them in a file. The model compression unit 106 also adds metadata to the file. The model compression unit 106 may also record the metadata in the header of the compressed data. Finally, the model compression unit 106 outputs the resulting file.

[0084] Figure 8 schematically shows how the Gaussian group to be compressed is compressed while changing its representation within the model compression unit 106 and stored in the bitstream.

[0085] According to this embodiment, particularly in steps S503 to S505, Gaussian groups represented using variable-order SHs are separated according to their order. The separated Gaussian groups are then treated as point cloud data with attribute information and compressed. The bitstreams representing the compressed data thus created are stored in a file along with metadata that manages them. According to this embodiment, Gaussian groups in a memory-efficient format obtained by reducing the order of SHs can be efficiently compressed. As a result, memory usage can be reduced. Furthermore, by shortening the data transfer time, it is expected that the operating speed of the device will be improved.

[0086] In the above embodiment, the Gaussian groups were separated outside the G-PCC encoder. Bitstreams were then generated by compressing each of the separated Gaussian groups. Metadata for integrated management of each bitstream was then added. In this configuration, the bitstreams generated for each order are stored in a single file. Furthermore, information indicating the separation method, such as the order corresponding to each bitstream, is described as metadata in the file format. During decoding, the data before separation can be obtained by combining the decoded Gaussian group data using this metadata. The above metadata may also be added to the bitstream as auxiliary information (SEI messages).

[0087] Note that it is not necessary to describe the SH order in the metadata. For example, you can impose a constraint that bitstreams be stored in the file in order of order. For example, bitstreams corresponding to lower orders can be stored at the beginning. In this case, the order of each bitstream can be identified according to the order in which they are stored in the file (or the order in which they are described in the metadata).

[0088] Alternatively, a sequence may be defined for each of one or more Gaussian groups. For example, multiple (e.g., four) seq_parameter_sets may be created internally by the G-PCC encoder for each order. Each Gaussian group may then be compressed into a single bitstream by the G-PCC encoder. With this configuration, a single bitstream can be obtained. In this case, the sequence parameters can include information indicating the number of parameters contained in the attribute information (e.g., the number of dimensions). During decoding, multiple compressed point cloud sequences can be obtained from the compressed data. The point cloud sequences can then be decompressed according to the number or number of dimensions of the attribute information described in the metadata.

[0089] In the embodiment described above, the SPS encoded the number of attribute information (num_attributes), the number of dimensions of each attribute information (attr_components_minus1), and signals representing the type (attr_label_known, attr_label, attr_label_oid). However, other configurations can be adopted. For example, this information may be encoded in other parameter sets. For example, this information may be encoded in tile_inventory, which is separately defined in G-PCC. Also, if the Gaussian group for each degree is divided into slices, similar information may be obtained from slice_header, which records the header information of each slice. Such a configuration makes it possible to encode the Gaussian group as a single G-PCC sequence containing Gaussian groups of different degrees, instead of encoding the Gaussian group as a separate G-PCC sequence for each degree. In addition, a variable-degree flag, variable_attribute_num_flag, may be encoded in the SPS. When this flag is false, each piece of information is encoded in the SPS; only when this flag is true can each piece of information be encoded in a different location, such as the slice_header. This configuration prevents a decrease in compression efficiency due to the encoding of redundant information.

[0090] Furthermore, in the embodiment described above, the Gaussian group was divided, and the divided Gaussian group was compressed individually. However, other methods can be used to compress Gaussian groups that have different numbers of parameters as attribute information. For example, the number or type of parameters (e.g., the number of dimensions of SH) that each Gaussian has may be encoded as another attribute information. During decoding, the information indicating the number or type of parameters that each Gaussian has may be decoded first, and then the attribute information of each Gaussian, including a variable number of parameters, may be decoded by referring to the obtained information.

[0091] <Decryption device> Next, a display device 1101, which is a decoding device according to one embodiment of the present invention, will be described. The input unit 1102 acquires compressed data of point cloud data. This compressed data may be compressed data output by the image processing device 101. That is, the point cloud data represents the position of each of the multiple points and the attribute information of each of the multiple points. Here, the attribute information includes at least a variable number of parameters that indicate the coefficients of a model that responds to color with respect to direction. Furthermore, the compressed data is configured to indicate the number of parameters that each of the multiple points' attribute information has. As described above, such point cloud data can represent 3DGS data (Gaussian primitive group) when the SH coefficients have an adaptive order.

[0092] The model decompression unit 1103 decompresses the compressed data by referring to the number of parameters in the attribute information of each of the multiple points obtained based on the compressed data.

[0093] Compressed data can include compressed point cloud data (compressed element group data), which are compressed data for one or more point cloud data (element group data). The number of parameters included in the attribute information is the same for each compressed point cloud data. The model decompression unit 1103 can decompress each compressed point cloud data by referring to the number of parameters included in the attribute information for each compressed point cloud data. The model decompression unit 1103 can also integrate the point cloud data obtained by decompressing each compressed point cloud data. With this configuration, the original point cloud data can be restored when the point cloud data has been divided and compressed.

[0094] The following describes an embodiment in which point cloud data represents a group of Gaussian primitives in 3DGS. In the following description, the display device 1101 acquires the compressed Gaussian group and displays a scene or object from a viewpoint specified by the user via the display. As described above, the Gaussian group is encoded as a point cloud having attribute information such as Gaussian orientation, Gaussian standard deviation, opacity, and SH coefficient. The following describes the case in which the display device 1101 processes the Gaussian group compressed by the image processing device 101.

[0095] Figure 11 is a block diagram showing the functional components of the display device 1101. The input unit 1102 acquires compressed Gaussian group data from outside the display device 1101. The input unit 1102 then outputs the acquired compressed data to the model decompression unit 1103.

[0096] The model decompression unit 1103 decompresses the compressed data of the Gaussian group. The model decompression unit 1103 then outputs the Gaussian group data obtained by decompression to the drawing unit 1105. In an embodiment in which the compressed data is compressed according to G-PCC, the model decompression unit 1103 can decompress the compressed data according to G-PCC. By decompressing the compressed data, the model decompression unit 1103 can output one or more of the following as attribute information: the opacity of each point, the standard deviation defining the magnitude of the spread, the orientation defining the direction of the spread, and the SH coefficient defining the color.

[0097] The operation unit 1104 receives user input. The operation unit 1104 can receive user input specifying a viewpoint. For example, the operation unit 1104 can receive user input via a user interface such as a mouse or touch panel. Then, the operation unit 1104 generates camera parameters corresponding to the specified viewpoint in response to the input. The operation unit 1104 outputs the generated camera parameters to the drawing unit 1105.

[0098] The drawing unit 1105 generates a drawing image by rasterizing the Gaussian group acquired from the model expansion unit 1103 according to the camera parameters acquired from the operation unit 1104. The drawing unit 1105 can generate a drawing image using the same processing as the drawing unit 105. The drawing unit 1105 then outputs the generated drawing image to the display unit 1106.

[0099] The display unit 1106 displays the acquired drawing image on the display.

[0100] Figure 12 is a flowchart showing the processing performed by the display device 1101 in this embodiment. At S1201, the input unit 1102 acquires metadata and compressed data bitstreams from outside the display device 1101. The input unit 1102 can read metadata and bitstreams from a file. However, the input unit 1102 may acquire these data by other means. For example, the input unit 1102 may acquire metadata and bitstreams as packets from a network.

[0101] In S1202, the model decompression unit 1103 analyzes the header and metadata of the bitstream, such as SPS. Details will be described later.

[0102] In S1203, the model decompression unit 1103 decompresses one or more Gaussian groups according to the bitstream header information. Details will be described later.

[0103] In S1204, the control unit 1104 receives user input as described above and generates camera parameters.

[0104] In S1205, the drawing unit 1105 generates a drawn image by rasterizing the Gaussian group according to the camera parameters.

[0105] In S1206, the display unit 1106 displays the drawn image on the display.

[0106] <Processing flow of model extension unit 1103> Next, the details of the metadata analysis process performed by the model expansion unit 1103 in S1202 will be explained with reference to Figure 13. Here, an example of analyzing the metadata shown in Figure 7 will be described.

[0107] In S1301, the model decompression unit 1103 sequentially reads the variable order flag (uint8_t), the Gaussian group number (uint16_t), and the uncompressed mode flag (uint8_t).

[0108] In S1302, the model decompression unit 1103 determines whether the uncompressed mode flag is ON. The model decompression unit 1103 can determine that the uncompressed mode flag is ON if the uncompressed mode flag is 1 or greater. If it is determined that the uncompressed mode flag is ON, the process proceeds to S1303. Otherwise, the process proceeds to S1304.

[0109] In S1303, the model expansion unit 1103 reads the element count threshold (uint32_t).

[0110] In S1304, the model decompression unit 1103 determines whether the variable order flag is ON. In this embodiment, the model decompression unit 1103 decompresses the compressed data by referring to the variable order flag. The model decompression unit 1103 can determine that the variable order flag is ON if the variable order flag is 1 or greater. If it is determined that the variable order flag is ON, the process proceeds to S1308. Otherwise, the process proceeds to S1305.

[0111] S1305 to S1307 are loops for each Gaussian group (point cloud). S1306 performs processing on the Gaussian group of interest. As already explained, Gaussian groups may be divided. The model expansion unit 1103 controls the loop according to the number of Gaussian groups read in S1301.

[0112] In S1306, the model expansion unit 1103 sequentially reads the order of SH (uint8_t) and the number of elements in the Gaussian group of interest (uint32_t). Then, based on the order of SH F, the model expansion unit 1103 can obtain the number of SH coefficients that the attribute information of the Gaussian group has. Specifically, the model expansion unit 1103 obtains the number of SH coefficients F corresponding to 1 channel of color. 2 The model extension section 1103 has a coefficient F of SH. 2 This can be calculated. On the other hand, the number of coefficients F of SH corresponding to the order F is 2 A table defining this may be provided beforehand.

[0113] In S1308, the model expansion unit 1103 reads the SH order (uint8_t) common to all Gaussian groups.

[0114] S1309 to S1311 are loops for each Gaussian group (point group). In S1310, processing is performed for the Gaussian group of interest. The model expansion unit 1103 controls the loop according to the number of Gaussian groups read in S1301.

[0115] In S1310, the model expansion unit 1103 reads the number of elements (uint32_t) of the Gaussian group of interest.

[0116] Through the above process, the model extension unit 1103 can analyze the metadata.

[0117] Next, the details of the Gaussian group expansion process performed by the model expansion unit 1103 in S1203 will be explained with reference to Figure 14. In S1203, the model expansion unit 1103 performs processing according to the information obtained from the metadata analysis in S1202.

[0118] S1401 to S1407 are loops for each Gaussian group (point group). S1402 to S1406 perform processing on the Gaussian group of interest.

[0119] In S1402, the model decompression unit 1103 determines whether the uncompressed mode flag is ON. The model decompression unit 1103 can perform this determination in the same way as in S1302. If it is determined that the uncompressed mode flag is ON, the process proceeds to S1403. Otherwise, the process proceeds to S1406.

[0120] In S1403, the model extension unit 1103 determines whether the number of elements in the Gaussian group of interest is less than the element threshold tn. If it is determined that the number of elements in the Gaussian group of interest is less than the element threshold tn, the process proceeds to S1405. Otherwise, the process proceeds to S1406.

[0121] S1405 is performed when the uncompressed mode flag is ON and the number of elements in the Gaussian group is less than the element threshold tn. In this embodiment, as already explained, quantization is performed on the Gaussian group in this case. Therefore, in S1405, the model decompression unit 1103 dequantizes the Gaussian group of interest.

[0122] In S1406, the model decompression unit 1103 decompresses the Gaussian group of interest. In this embodiment, the Gaussian group is compressed in G-PCC format. In this case, the model decompression unit 1103 obtains the number of attribute information (num_attributes), the number of dimensions of each attribute information (attr_components_minus1), and signals representing the type (attr_label_known, attr_label, attr_label_oid) from the SPS of the Gaussian group. From this information, the model decompression unit 1103 can read the number and type of attribute information that the Gaussian group possesses. Therefore, the G-PCC decoder can decode the Gaussian groups individually, regardless of the difference in the SH order of each Gaussian group.

[0123] As described above, the compressed data can contain bitstreams corresponding to one or more Gaussian groups. In this case, the model decompression unit 1103 can decompress the bitstream corresponding to the Gaussian group of interest by referring to the number of parameters included in the attribute information of the Gaussian group of interest.

[0124] Furthermore, as described above, the compressed data may include a compressed point cloud sequence (compressed element sequence) corresponding to one or more point cloud data (element group data). In this case, the model decompression unit 1103 can decompress each compressed point cloud sequence by referring to the number of parameters included in the attribute information of each point cloud data.

[0125] In S1408, the model expansion unit 1103 determines whether the dimension unification flag is ON or OFF. The dimension unification flag is set as a parameter for model expansion. The model expansion unit 1103 can determine that the dimension unification flag is ON if the dimension unification flag is 1 or greater. If it is determined that the dimension unification flag is ON, the process proceeds to S1409. Otherwise, the process proceeds to S1410.

[0126] In S1409, the model stretching unit 1103 increases a parameter that indicates the coefficients of the model that respond to color with respect to direction to a predetermined number. For example, the model stretching unit 1103 can formally restore the order of the stretched SH to the order before reduction (three dimensions in this example). The coefficients of the SH corresponding to the reduced dimension are 0. This process makes it possible for applications that do not support dimensionality reduction representations to render the Gaussian group.

[0127] In S1410, the model extension unit 1103 determines whether the integration flag is ON or OFF. The integration flag is set as a parameter for model extension. The model extension unit 1103 can determine that the integration flag is ON if the integration flag is 1 or greater. If it is determined that the integration flag is ON, the process proceeds to 1411. Otherwise, the process in S1203 ends.

[0128] In S1411, the model stretching unit 1103 integrates the multiple stretched Gaussian groups into a single Gaussian group. This process enables applications that do not support simultaneous rendering of multiple Gaussian groups to render the Gaussian groups.

[0129] Through the above process, the model expansion unit 1103 can decode the Gaussian group.

[0130] According to this embodiment, a bitstream obtained by compressing a Gaussian group represented by a variable-order SH, such as a bitstream generated by an image processing device 101, can be decoded. Then, based on the decoded Gaussian group, an image from a viewpoint can be displayed. By handling compressed data according to this embodiment, the amount of memory required for decoding and rendering can be reduced. Furthermore, by shortening the data transfer time, it is expected that the operating speed of the device will be improved.

[0131] It is not necessary to obtain attribute information from SPS. As already explained, attribute information may be encoded in other parameter sets, such as tile_inventory or slice_header. The model decompression unit 1103 can obtain attribute information from this metadata. Also, as already explained, the number or type of parameters (e.g., the number of dimensions of SH) that each Gaussian has may be encoded as attribute information. In this case, the model decompression unit 1103 can first decode such information and then decode the attribute information of each Gaussian containing a variable number of parameters. In this way, the model decompression unit 1103 can use various methods to decode Gaussian groups with different numbers of parameters as attribute information.

[0132] In this embodiment, an example was shown in which the order of SH is variable. However, attribute information with a variable number of elements (e.g., number of parameters or number of dimensions) is not limited to attribute information with a variable number of SH coefficients. For example, the amount of data may be reduced by making the scale of a Gaussian function with a small scale in a particular direction two-dimensional and approximating it as a plane Gaussian function. In this case, the attribute information can include a variable number of parameters representing the standard deviation of Gauss. The same technique can be applied in this case as well.

[0133] (Variation 1) In the above embodiment, sequences were separated for each SH order both inside and outside the G-PCC encoder. That is, the number of attribute information parameters in a single sequence remained unchanged. This configuration allows for efficient compression of Gaussian groups.

[0134] On the other hand, it is not necessary to separate sequences by SH order. For example, the concepts of sequences and tiles can be used. For example, the model compression unit 106 can define a tile for each of one or more point cloud data. Here, a consistent tile can be defined for the sequence. In this case, the tile does not change from frame to frame within the sequence. The model compression unit 106 can then compress each tile. The model compression unit 106 can also describe information in the tile parameters indicating the number of parameters included in the attribute information. In this case, the compressed data can include compressed point cloud sequences that correspond to each of one or more point cloud data and are divided into tiles. In this case, the model decompression unit 1103 can decompress each compressed point cloud sequence by referring to the number of parameters included in the attribute information for each point cloud data.

[0135] Specifically, instead of describing the attributes in seq_parameter_set, the properties related to the attribute information of seq_parameter_set shown in Figure 10 can be described in the tile_inventory data unit syntax, which is defined separately in G-PCC. This syntax associates tiles with related sequences using ti_seq_parameter_set_id. By defining Gaussian groups of different orders to be divided into tiles, a configuration in which attribute information is consistent within tiles rather than within sequences can be achieved. In this case, compression and decompression can be performed independently without referencing between tiles. Furthermore, it is also possible to define slices for each separated Gaussian group and compress the Gaussian groups. Note that a tile is defined as a collection of slices. Also, when using time correlation compression, correlation with neighboring frames within the same tile can be performed. Therefore, it becomes unnecessary to consider the differences in attribute information between the objects being correlated.

[0136] This configuration allows information corresponding to one frame of the original data to be placed in close proximity within the compressed data. Therefore, it becomes possible to efficiently decode a specific frame. Furthermore, there is no need to manage multiple sequences. And, there is no need to decompress multiple bitstreams in parallel during decoding. For these reasons, an improvement in the efficiency of the decoding process is expected.

[0137] (Modification 2) In the above embodiment, the Gaussian group was compressed as a point cloud with attribute information for each SH order. However, the data size of the compressed Gaussian group is still large. In addition, it may be necessary to partially decode the Gaussian group in order to provide a preview to the user. For example, LumaAI employs a method of rendering a high-quality image by first displaying the Gaussian group as a colored point cloud and then applying a Gaussian distribution. In this modified example, in order to efficiently support such rendering processing, the concept of layer coding used in HEVC is introduced for the compression of the Gaussian group.

[0138] The processing for a specific frame is described below. First, the 0th-order component of SH is compressed as attribute information in the base layer. The base layer contains the point cloud and the 0th-order component of SH. Therefore, the base layer can be decompressed on its own. By decompressing the base layer, it becomes possible to display the colored point cloud.

[0139] Next, opacity, orientation, and standard deviation are compressed as attribute information contained in the first-order enhancement layer. When decoding this information, the point cloud obtained by decoding the base layer is referenced. Furthermore, by referencing the first-order enhancement layer, it becomes possible to reproduce the spread and transparency based on a Gaussian distribution.

[0140] Furthermore, the SH order (or order-1) and the coefficients of the 1st to 3rd order of SH for each Gaussian are compressed as attribute information included in the 2nd order enhancement layer. If the order has been reduced, a temporary value (e.g., 0) can be set as the coefficient corresponding to the reduced order. When decoding this information, the point cloud obtained by decoding the base layer is also referenced.

[0141] Thus, the Gaussian group compressed by the model compression unit 106 can have information as attribute information indicating the number of parameters included in the attribute information. The model decompression unit 1103 can decompress information indicating the number of parameters each of the compressed points has. Furthermore, the model decompression unit 1103 can reduce some of the parameters indicating the coefficients of the model that responds to color in a given direction, obtained by decompressing the compressed data, by not outputting them, according to the number of parameters in the attribute information of each of the points. In many point cloud compression techniques such as G-PCC, the order of points is not maintained during compression in order to improve the compression ratio. In this case, it is difficult to infer the Gaussian order based on the order after decompressing the Gaussian group. According to this modified example, the SH order for Gauss is compressed as attribute information. Therefore, during decompression, the SH order of Gauss can be reduced again based on the attribute information.

[0142] This modified approach allows for faster provision of preview images to the user by expanding lower-level layers without expanding all attribute information. Furthermore, it ensures scalability.

[0143] Alternatively, instead of using the methods described above, it may be possible to improve the encoder / decoder for point cloud compression methods such as G-PCC to enable compression / decompression of attribute information that differs for each point.

[0144] [Second Embodiment] In the first embodiment, memory usage is significantly reduced by employing a memory-saving representation in which the order of SH representing color is variable. On the other hand, other data representation methods (memory-saving representations) that enable reductions in memory usage may be used. For example, the SH coefficients can be converted to YUV and the AC components (coefficients of the first order and above) of U and V can be removed. This conversion can reduce the data size of the Gaussian primitive from 236 bytes (59 parameters) to 116 bytes (29 parameters). YUV represents luminance and color difference information, and in this example, color is represented by luminance and color difference. The main component of color variation due to changes in direction, such as specular reflection, is the luminance component Y. Therefore, even if the AC components of U and V, which are color differences, are removed, the impact on the drawn image is small. During drawing, the YUV signal for the viewpoint direction can be calculated while omitting the calculation of the AC component of UV, and the obtained YUV signal can be inversely converted to an RGB signal. In this embodiment, memory usage is reduced by combining such a memory-saving representation with the quantization performed when applying G-PCC. In this specification, information indicating color includes Y, which indicates brightness.

[0145] In the following, the representation form that conforms to the definition of a Gaussian primitive described in the first embodiment (Non-Patent Literature 1) will be referred to as basic 3DGS. This representation form is currently called I-3DGS in the Gaussian Splat Coding group of MPEG. This representation form is supported by many applications and is already being used in services. However, Gaussian primitives that have parameters representing the same information as defined in I-3DGS at different bit depths are included in the category of basic 3DGS. For example, a form obtained by simply converting the elements constituting basic 3DGS to fixed-point numbers is included in basic 3DGS. Furthermore, in this specification, a memory-saving representation obtained by reducing the number of parameters constituting a Gaussian primitive will be referred to as compact 3DGS. Compact 3DGS can be obtained, for example, by processing the SH coefficient in basic 3DGS as described above. In this specification, 3DGS data (Gaussian group) that conforms to the basic 3DGS and 3DGS data (Gaussian group) that conforms to the simplified 3DGS may be simply referred to as basic 3DGS and simplified 3DGS, respectively.

[0146] Figure 16 is a block diagram showing an example configuration of an encoding device according to this embodiment. The encoding device 1601 includes a simplification unit 1602, a compression unit 1603, and a control unit 1605. The encoding device 1601 can acquire either a basic 3DGS or a simplified 3DGS generated outside the encoding device 1601. The encoding device 1601 can then output a compressed 3DGS by encoding the acquired 3DGS.

[0147] The simplification unit 1602 generates a second set of elements, such as a simplified 3DGS, by reducing the amount of attribute information data contained in a first set of elements, such as a basic 3DGS. For example, the simplification unit 1602 can output a simplified 3DGS by simplifying the input 3DGS. To this end, the simplification unit 1602 can perform at least one of the following processes: conversion, quantization, and deletion. These processes will be described later.

[0148] The compression unit 1603 compresses the element group. The compression unit 1603 can compress the element group selected by the control unit 1605, as described later. For example, either basic 3DGS or simplified 3DGS is input to the compression unit 1603. The compression unit 1603 outputs a compressed 3DGS by compressing the input 3DGS. The basic 3DGS input to the compression unit 1603 is the data input to the encoding device 1601. The simplified 3DGS input to the compression unit 1603 is data generated outside the encoding device 1601 and input to the encoding device 1601, or data output by the simplification unit 1602.

[0149] In the example shown in Figure 16, the optimization unit 1604 is located outside the encoding device 1601. The optimization unit 1604 can optimize the simplified 3DGS by referring to the captured image. The optimization unit 1604 thus generates higher quality data and outputs it to the encoding device 1601. The optimization unit 1604 can generate a simplified 3DGS that is optimized in the same way as the model optimization unit 104 described above, except that the amount of attribute information data is reduced. In this example, the compression unit 1603 can acquire the simplified 3DGS generated by the optimization unit 1604 instead of the simplified 3DGS generated by the simplification unit 1602, and compress the acquired simplified 3DGS. The encoding device 1601 may also have the optimization unit 1604 instead of the simplification unit 1602.

[0150] The control unit 1605 controls the processing performed by the encoding device 1601. The control unit 1605 can select either a first element group, such as basic 3DGS, or a second element group, such as simplified 3DGS. In another aspect, the control unit 1605 can select either the first element group input to the encoding device 1601 or the second element group obtained by simplifying the first element group. The selected element group is output to the compression unit 1603 for compression. Based on user instructions, the control unit 1605 can output either the first or second element group to the compression unit 1603. The control unit 1605 can also control the processing of the encoding device 1601 according to control flags (for example, simplification flags described later) obtained from outside the encoding device 1601. The control unit 1605 may also determine whether the amount of attribute information data in the first element group input to the encoding device 1601 has already been reduced. For example, the amount of attribute information data may be reduced by at least one of the following processes: transformation, quantization, and deletion, applied to at least one of the information types: orientation, scale, opacity, and color. If the amount of attribute information data in the first group of elements input to the encoding device 1601 has already been reduced, the control unit 1605 can output the first group of elements input to the encoding device 1601 to the compression unit 1603. In this case, the control unit 1605 may control the simplification unit 1602 to skip the process of simplifying the first group of elements input to the encoding device 1601.

[0151] Next, with reference to Figure 17, the processing flow of the encoding device 1601 in this embodiment will be described. In S1701, the control unit 1605 acquires encoding parameters. The control unit 1605 can acquire encoding parameters from an application that requests encoding processing. The application can change the encoding parameters based on user instructions. These encoding parameters include a simplification flag. For example, if the user wants lossless compression, the application can turn off the simplification flag. In this case, the application inputs basic 3DGS to the encoding device 1601.

[0152] In S1702, the encoding device 1601 acquires a Gaussian group. For example, an application requesting encoding can input a Gaussian group to the encoding device 1601. The encoding device 1601 can acquire either basic 3DGS or simplified 3DGS.

[0153] In S1703, the control unit 1605 determines whether the input Gaussian group is a simplified 3DGS. If it is determined that the Gaussian group is a simplified 3DGS, the process proceeds to S1706. If it is determined that it is not a simplified 3DGS, the process proceeds to S1704. The control unit 1605 can determine whether the Gaussian group is a simplified 3DGS based on the difference in the input interface of the Gaussian group to the encoding device 1601. Alternatively, the control unit 1605 may determine whether the Gaussian group is a simplified 3DGS based on a flag for a common input interface or other methods such as analysis of the input Gaussian group.

[0154] In S1704, it is determined whether the simplification flag is ON or OFF. As mentioned above, the simplification flag is included in the encoding parameters obtained in S1701. If the simplification flag is ON, the process proceeds to S1705. Otherwise, the process proceeds to S1706.

[0155] In S1705, the simplification unit 1602 generates a simplified 3DGS by simplifying the Gaussian primitives that constitute the Gaussian group. At this time, the simplification unit 1602 can reduce the amount of data of the attribute information possessed by the Gaussian group by at least one of the following processes: transformation, quantization, and deletion. This attribute information can represent at least one of the following: orientation, opacity, and color. In this embodiment, the simplification unit 1602 reduces the amount of data of the attribute information possessed by the Gaussian group by an irreversible process. In one embodiment, the simplification unit 1602 performs a process to reduce the amount of data for a specific set of parameters in the attribute information possessed by the Gaussian group, but does not perform a process to reduce the amount of data for other parameters. For example, the simplification unit 1602 can perform a process to reduce the amount of data for a parameter that represents at least one of the following: orientation, opacity, and color. In one embodiment, the number of parameters in the attribute information possessed by the Gaussian group is common to each Gaussian. That is, the simplified attribute information may include a fixed number of parameters. By unifying attribute information within the Gaussian group in this way, the implementation of the encoding and decoding devices can be simplified. Furthermore, the processing time for encoding and decoding can be reduced.

[0156] For example, the simplification unit 1602 can convert the parameters represented by the attribute information. In this embodiment, the simplification unit 1602 converts RGB information to luminance and color difference information. More specifically, the simplification unit 1602 converts the SH coefficients representing RGB to YUV and removes the AC components of the U and V components. In this way, parameter conversion and the removal or quantization of the converted parameters can be used in combination to reduce the amount of attribute information data.

[0157] The simplification method is not limited to this method. For example, the simplification unit 1602 can delete parameters represented by attribute information. For example, by changing the Gaussian primitive from an ellipse to a sphere, the scale can be represented with one-dimensional information. By representing the scale with one-dimensional information, the remaining two-dimensional information can be deleted. Also, for example, by constraining the length of the quaternion representing rotation to 1, rotation can be represented with three values. Furthermore, information regarding opacity may be fixed to a specific numerical value. In this case, it is not necessary to assign a fixed value to each Gaussian primitive. That is, the simplification unit 1602 may delete one or more parameters represented by attribute information and set a common default value for each Gaussian primitive for the deleted parameter. The simplification unit 1602 can encode such uniform fixed values ​​as parameters in header information such as SPS.

[0158] Furthermore, the simplification unit 1602 can quantize the parameters represented by the attribute information. For example, the simplification unit 1602 can use quantization techniques such as vector quantization. In this case, the compression unit 1603 can record the codebook in the simplified or compressed data. Here, the simplified or compressed data may also include codebooks for one-dimensional or two-dimensional or more information.

[0159] In S1706, the compression unit 1603 compresses the 3DGS. First, as a preprocessing step, the compression unit 1603 can integerize the geometry and attribute information. For example, each parameter representing the geometry can be represented by 16 bits, and each parameter representing the attribute information can be represented by 8 bits. If scaling is required for each, the offset and scale factor can be notified to the encoder. If the attribute information is vector quantized, the index representing the codebook reference is encoded losslessly. The codebook is also compressed. Subsequently, the compression unit 1603 can generate a compressed 3DGS by compressing the integerized Gaussian group as a point cloud with attribute information using a G-PCC encoder.

[0160] Additionally, the sequence parameter set (seq_parameter_set) for compressed 3DGS can include a simplification flag (is_compacted) indicating whether the 3DGS to be compressed has been simplified or not. This simplification flag can be set to 1 if the 3DGS has been simplified, and 0 otherwise.

[0161] Figure 20 shows an example of the structure of an output file from the encoding device 1601. The metadata 2001 at the beginning contains metadata for defining the format. HLS2002 contains high-level syntax such as the sequence parameter set mentioned above. Payload 2003 represents the main body of the compressed data. SEI2004 represents an SEI message for the application to efficiently handle the compressed data. Simplification flags can also be included in the output file. As mentioned above, the simplification flags can be included in HLS2002, more specifically in the sequence parameter set. Simplification flags may also be included in other locations. For example, as described in the first embodiment, the simplification flags may be included in the tile_inventory data unit syntax. Alternatively, the simplification flags may be included as auxiliary information in the SEI message or metadata. This concludes the explanation of the processing flow of the encoding device 1601.

[0162] Figure 18 is a block diagram showing an example configuration of a decoding device according to this embodiment. The decoding device 1801 is an image processing device that processes compressed data of an element group (e.g., a Gaussian group). The decoding device 1801 has a decompression unit 1802, a restoration unit 1803, and a control unit 1806. Compressed 3DGS is input to the decoding device 1801 from outside the decoding device 1801. The compressed 3DGS can be compressed basic 3DGS or compressed simplified 3DGS.

[0163] The decompression unit 1802 decompresses the compressed data. In this embodiment, the compressed data is a compressed 3DGS input to the decoding device 1801. The decompression unit 1802 can generate a first set of elements, such as a simplified 3DGS, by decompressing compressed data, such as a compressed simplified 3DGS.

[0164] The reconstruction unit 1803 generates a second set of elements, such as a basic 3DGS, by increasing the amount of attribute information data of a first set of elements, such as a simplified 3DGS. This attribute information can represent at least one of the following: orientation, opacity, and color. To this end, the reconstruction unit 1803 can perform at least one of the following processes: transformation, inverse quantization, and assignment of a specified value. Details of these processes will be described later.

[0165] The control unit 1806 can determine, according to predetermined conditions, whether to output a first element group, such as a simplified 3DGS, or a second element group, such as a basic 3DGS. Then, the control unit 1806 can output either the first or second element group according to the determination result. In this way, the decoding device 1801 can selectively output either a simplified 3DGS or a basic 3DGS. For example, based on the state of the input compressed 3DGS or the control parameters of the decoding device 1801, the control unit 1806 can output a simplified 3DGS or a basic 3DGS. In one embodiment, the compressed data includes information indicating whether or not to output a second element group (for example, a restoration flag described later). If this information indicates that the second element group should not be output, the control unit 1806 can output the first element group. In this case, the control unit 1806 can control the restoration unit 1803 to skip the generation of the second element group from the first element group.

[0166] Furthermore, the decoding device 1801 can select whether to output a first element group or a second element group, such as a basic 3DGS, depending on whether the drawing unit available to the decoding device 1801 can draw a first element group, such as a simplified 3DGS. In one embodiment, the decoding device 1801 has a drawing unit that draws the element group, or is connected to such a drawing unit. The control unit 1806 then determines whether the drawing unit can draw the first element group. If the drawing unit can draw the first element group, the control unit 1806 can output the first element group. If the drawing unit cannot draw the first element group, the control unit 1806 can output the second element group.

[0167] In this embodiment, a simplified drawing unit 1804 and a drawing unit 1805 are provided outside the decoding device 1801. The simplified drawing unit 1804 outputs a drawn image by drawing a simplified 3DGS. The simplified drawing unit 1804 draws a memory-saving simplified 3DGS obtained from the decoding device 1801. In the process of drawing the simplified 3DGS, the simplified drawing unit 1804 partially converts Gaussian primitives to basic 3DGS in batch units (a collection of 16x16 pixel tiles on the drawing screen). The simplified drawing unit 1804 can then draw the basic 3DGS obtained by the conversion in batch units. The decoding device 1801 according to this embodiment can output a simplified 3DGS. Therefore, by adopting a configuration that generates a drawn image based on a simplified 3DGS, the amount of memory used to temporarily hold the 3DGS during drawing can be reduced.

[0168] The drawing unit 1805 outputs a drawn image by drawing the basic 3DGS. The drawing unit 1805 draws the basic 3DGS acquired from the decoding device 1801. The decoding device 1801 according to this embodiment can output the basic 3DGS. Therefore, even if a drawing unit that supports drawing simplified 3DGS is not available, a drawn image can be generated.

[0169] Thus, in this embodiment, two drawing units (simplified drawing unit 1804 and drawing unit 1805) are used. However, other configurations may be used. For example, a drawing unit capable of drawing both basic 3DGS and simplified 3DGS can be used.

[0170] The processing flow of the decoding device 1801 will be explained with reference to Figure 19. In S1901, the control unit 1806 obtains decoding parameters. The control unit 1806 can obtain decoding parameters from the application requesting the decoding process. The application can change the decoding parameters based on user instructions. These decoding parameters include a restore flag. The restore flag is a flag that instructs the system to restore the decompressed simplified 3DGS to the basic 3DGS.

[0171] In S1902, the decoding device 1801 obtains the compressed Gaussian group. The decoding device 1801 can obtain the compressed basic 3DGS or compressed simplified 3DGS from outside the device.

[0172] In S1903, the decompression unit 1802 decompresses the compressed Gaussian group. For example, the decompression unit 1802 can decompress 3DGS using a G-PCC decoder.

[0173] In S1904, the control unit 1806 determines whether the decompressed Gaussian group obtained in S1903 is a simplified 3DGS. For example, the control unit 1806 can determine whether the decompressed Gaussian group is a simplified 3DGS by referring to a simplification flag encoded in the compressed data. If the decompressed Gaussian group is a simplified 3DGS, the process proceeds to S1905. Otherwise, the process proceeds to S1907.

[0174] In S1905, the control unit 1806 determines whether the restoration flag is ON or OFF. As mentioned above, the restoration flag is included in the decoding parameters obtained in S1901. If the decoding flag is ON, the process proceeds to S1906. Otherwise, the process proceeds to S1907.

[0175] In S1906, the restoration unit 1803 restores the basic 3DGS from the simplified 3DGS. Parameters that were reduced by the conversion when creating the simplified 3DGS are restored by the inverse conversion. The parameters are restored by the inverse conversion in such a way that the number of parameters increases or the amount of data for the parameters increases. In one embodiment, the restoration unit 1803 restores specific parameters and does not perform a process to increase the amount of data for other parameters. For example, the simplification unit 1602 can restore a parameter that indicates at least one of the following: orientation, opacity, and color.

[0176] For example, the reconstruction unit 1803 can convert the parameters represented by attribute information. For example, the reconstruction unit 1803 can convert luminance and color difference information to RGB information. In simplification, if the AC components of U and V representing color difference are removed after YUV conversion, the amount of parameter data increases due to the color conversion from YUV to RGB in reconstruction. The reconstruction unit 1803 may also convert the one-dimensional scale parameters representing a sphere to three-dimensional scale parameters by increasing the number of parameters through copying. The reconstruction unit 1803 can also perform inverse quantization on the parameters represented by attribute information. For example, in reconstruction, the amount of data may be increased by vector inverse quantization.

[0177] Furthermore, the restoration unit 1803 can assign default values ​​to parameters. The restoration unit 1803 may set default values ​​for parameters in place of parameters deleted during simplification. These default values ​​may be set by analyzing information in a header such as SPS. For example, if the header contains information that all opacity parameters should be set to 1, the restoration unit 1803 can assign "1" as the opacity parameter to all Gaussian curves. The restoration unit 1803 may also assign predetermined values ​​to all Gaussian curves without referring to the header information. The restoration unit 1803 may also perform similar processing for parameters other than opacity.

[0178] Here, the restoration unit 1803 can represent the values ​​of all parameters as 4-byte floating-point numbers, as described in Non-Patent Document 1. Alternatively, the restoration unit 1803 may output the parameter values ​​as half-precision floating-point numbers. Furthermore, the restoration unit 1803 may output the parameters, which have been integerized by scaling and offsetting, as 2-byte integers. Thus, the restored parameters may have undergone a simple reduction in the number of bytes.

[0179] In S1907, the control unit 1806 outputs the Gaussian group to the outside of the decoder 1801. The control unit 1806 can output simplified 3DGS or basic 3DGS. In one embodiment, if the Gaussian primitive is not simplified (No in S1903) or if the restoration flag is ON (Yes in S1904), the control unit 1806 outputs basic 3DGS. On the other hand, if the Gaussian primitive is simplified and the restoration flag is OFF (No in S1904), the control unit 1806 outputs simplified 3DGS. When outputting simplified 3DGS, the control unit 1806 can further output information to restore the pre-encoded values, such as scale, offset, or codebook.

[0180] In this embodiment, whether or not to restore the basic 3DGS was controlled by a restore flag. However, the control method is not limited to this method. For example, the decision of whether or not to restore may be made by functional limitations based on the MPEG standard profile.

[0181] In this embodiment, in the simplified 3DGS representation, Gaussian primitives are represented by 28 bytes. Internally in the program, values ​​are basically represented by combinations of basic types of 1, 2, 4, or 8 bytes. Taking this into consideration, an effective data structure can be designed by quantitatively evaluating the amount of data usage required for a single primitive. Simplified 3DGS can be represented using various combinations of simplification techniques in addition to the specific example described above.

[0182] According to this embodiment, one of the elements selected from the first and second element groups can be output. For example, since a simplified Gaussian group can be output, the amount of memory required for drawing based on the output Gaussian group can be reduced. On the other hand, the simplified Gaussian group can be restored to the conventional Gaussian group as needed. Since the conventional Gaussian group can be output in this way, drawing based on 3DGS can be performed even if the available 3DGS renderer does not support the simplified Gaussian group. Furthermore, in one embodiment, attribute information is unified within the Gaussian group, which facilitates simplification of implementation and reduction of processing time.

[0183] (Other embodiments) In the embodiments described above, the processing units of the image processing device 101 and display device 1101 shown in Figures 1 and 11 are configured as hardware. On the other hand, the image processing device 101 and display device 1101 can be implemented by a computer equipped with a processor and memory. Similarly, the encoding device 1601 and decoding device 1801 can also be implemented by a computer equipped with a processor and memory. In other words, the processing performed by at least a portion of each processing unit may be implemented by a computer program.

[0184] Figure 15 is a block diagram showing an example of a computer hardware configuration that can be used as an image processing device 101 and a display device 1101 according to each of the above embodiments. The CPU 1501 controls the entire computer using computer programs or data stored in RAM 1502 or ROM 1503. The CPU 1501 also executes the above-mentioned processes performed by the image processing device 101 and the display device 1101. In other words, the CPU 1501 can function as the processing unit shown in Figures 1, 11, 16, or 18.

[0185] RAM 1502 has an area for temporarily storing computer programs or data loaded from the external storage device 1506, or data acquired from an external source via the I / F (interface) 1507. Furthermore, RAM 1502 has a work area used by the CPU 1501 when executing various processes. In other words, RAM 1502 can be provided with frame memory or other memory areas.

[0186] ROM 1503 stores computer configuration data or boot programs. The control unit 1504 can be a keyboard or mouse. The computer user can input various instructions to the CPU 1501 by operating the control unit 1504. The display unit 1505 displays the processing results from the CPU 1501. The display unit 1505 can be, for example, a liquid crystal display.

[0187] The external storage device 1506 is a large-capacity information storage device such as a hard disk drive. The external storage device 1506 stores the OS (operating system) or computer programs that enable the CPU 1501 to implement the functions of each processing unit shown in Figures 1, 11, 16, or 18. The external storage device 1506 may also store image data to be processed.

[0188] In this way, a processor such as the CPU 1501 can realize the functions of each processing unit shown in Figures 1, 11, 16, or 18 by executing a program stored in memory such as RAM 1502, ROM 1503, or external storage device 1506. For example, a computer program or data stored in the external storage device 1506 is loaded into RAM 1502 according to the control of the CPU 1501. The CPU 1501 then performs processing according to the loaded computer program or data. In this way, the CPU 1501 can control the operations described by referring to each flowchart.

[0189] I / F1507 can be connected to other devices such as LANs or the Internet, projection devices, or display devices. The computer can acquire or transmit various types of information via I / F1507. Bus 1508 connects the above-mentioned components.

[0190] The image processing device 101, display device 1101, encoding device 1601, and decoding device 1801 may be composed of multiple information processing devices connected, for example, via a network. For example, the functions of the image processing device 101, display device 1101, encoding device 1601, and decoding device 1801 may be provided as a cloud service.

[0191] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0192] The disclosures herein include the following image processing apparatus, decoding apparatus, encoding apparatus, image processing method, decoding method, encoding method, and program. (Item 1) Compressed data of a group of elements, wherein the group of elements includes the position of each of the multiple elements, An acquisition means for acquiring the compressed data, wherein the attribute information of each of the plurality of elements represents, the attribute information indicates at least a scale, the attribute information includes a variable number of parameters, and the compressed data is configured to indicate the number of parameters that each of the attribute information of the plurality of elements has, A decompression means for decompressing the compressed data by referring to the number of parameters that each of the multiple elements' attribute information has, obtained based on the compressed data, A decoding device characterized by comprising: (Item 2) The compressed data includes compressed element group data, which are compressed data for each of one or more element group data, and in one compressed element group data, the number of parameters included in the attribute information is the same. The decoding device according to item 1, wherein the decompression means decompresses each of the compressed element group data by referring to the number of parameters included in the attribute information in each of the compressed element group data. (Item 3) The decoding device according to item 2, wherein the decompression means integrates the respective element group data obtained by decompressing each of the compressed element group data. (Item 4) The compressed data includes a bitstream corresponding to each of one or more elemental data sets. In the element set data corresponding to a single bitstream, the number of parameters included in the attribute information is the same. The number of parameters included in the attribute information for each element data is indicated by the metadata contained in the file storing the compressed data, or by the storage order of the bitstream in the file. The decoding device according to any one of items 1 to 3, wherein the decompression means decompresses each bitstream by referring to the number of parameters included in the attribute information in each element group data. (Item 5) The compressed data includes a compressed element group sequence corresponding to one or more element group data, and the number of parameters included in the attribute information is the same in the element group data corresponding to one compressed element group sequence. The number of parameters included in the attribute information for each element group data is indicated by the parameters of the compressed element group sequence. The decoding device according to any one of items 1 to 3, wherein the decompression means decompresses each compressed element group sequence by referring to the number of parameters included in the attribute information in each element group data. (Item 6) The compressed data includes a compressed element sequence divided into tiles, each corresponding to one or more element group data, and the number of parameters included in the attribute information is the same in the element group data corresponding to one tile. The number of parameters included in the attribute information for each element data set is indicated by the tile parameters. The decoding device according to any one of items 1 to 3, wherein the decompression means decompresses each compressed element group sequence by referring to the number of parameters included in the attribute information in each element group data. (Item 7) The compressed data includes, as attribute information, information indicating the number of parameters that each of the plurality of elements has, The decoding device according to any one of items 1 to 3, characterized in that the decompression means decompresses information indicating the number of parameters each of the plurality of elements has. (Item 8) The compressed data includes flag information indicating that the number of parameters in the attribute information of each of the plurality of elements is variable. The decoding device according to any one of items 1 to 7, wherein the decompression means decompresses the compressed data by referring to the flag information. (Item 9) The decoding device according to any one of items 1 to 8, characterized in that the attribute information indicates coefficients of a model that responds to color in relation to direction. (Item 10) The decoding device according to item 9, characterized in that the decompression means increases a parameter indicating a coefficient of a model that responds to color in the direction to a predetermined number after the decompression of the compressed data. (Item 11) The decoding device according to any one of items 9 to 10, characterized in that the decompression means reduces some of the parameters that indicate the coefficients of a model that responds to color in direction, obtained by decompressing the compressed data, to not output them, according to the number of parameters that the attribute information of each of the plurality of elements has. (Item 12) A decoding device according to any one of items 9 to 11, characterized in that the model for responding to color in the aforementioned direction is defined by Spherical Harmonics (SH), and the coefficients of the model are the coefficients of SH. (Item 13) The compressed data is configured to show the order of SH for each of the multiple elements, The decoding device according to item 12, characterized in that the decompression means obtains the number of coefficients of SH that each of the multiple elements' attribute information has, based on the order of SH. (Item 14) The decoding device according to any one of items 1 to 13, characterized in that the decompression means decompresses the compressed data and outputs one or more of the following as attribute information: opacity of each element, standard deviation defining the magnitude of the spread, orientation defining the direction of the spread, and coefficient of Spherical Harmonics defining the color. (Item 15) The aforementioned set of elemental data represents the Gaussian primitive group of 3DGS, The aforementioned compressed data is compressed according to G-PCC. The decoding device according to any one of items 1 to 14, wherein the decompression means decompresses the compressed data according to G-PCC. (Item 16) Element group data representing the position of each of the multiple elements and the attribute information of each of the multiple elements, wherein the attribute information at least indicates a scale. The attribute information includes a variable number of parameters, and the acquisition means acquires the element group data, The system includes a compression means that generates compressed data by compressing the aforementioned element group data, The encoding device is characterized in that the compression means configures the compressed data to indicate the number of parameters each of the plurality of elements has. (Item 17) The encoding device according to item 16, characterized in that the compression means compresses one or more of the element group data, and in one of the element group data, the number of parameters included in the attribute information is the same. (Item 18) The encoding device according to any one of items 16 to 17, wherein the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information, and compresses each of the divided element group data. (Item 19) The encoding device according to any one of items 16 to 18, wherein the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information and the positional relationship of the plurality of elements, and compresses each of the divided element group data. (Item 20) The compression means generates one or more bitstreams by compressing one or more of the element data sets, and serializes the one or more bitstreams and stores them in the file such that the number of parameters included in the attribute information in the element data sets corresponding to each bitstream is indicated by the metadata of the file or the storage order of the bitstreams in the file. An encoding device according to any one of items 16 to 19, characterized in that the number of parameters included in the attribute information is the same in one of the element group data. (Item 21) A sequence is defined for each of the one or more element data sets, the compression means generates a single bitstream by compressing each element data set, and the parameters of the sequence describe information indicating the number of parameters included in the attribute information. An encoding device according to any one of items 16 to 19, characterized in that the number of parameters included in the attribute information is the same in one of the element group data. (Item 22) A tile is defined for each of the one or more element data sets, the compression means compresses each of the tiles, and describes in the tile's parameters information the number of parameters included in the attribute information, An encoding device according to any one of items 16 to 19, characterized in that the number of parameters included in the attribute information is the same in one of the element group data. (Item 23) The encoding device according to any one of items 16 to 19, characterized in that the element group data compressed by the compression means includes, as attribute information, information indicating the number of parameters each of the plurality of elements has. (Item 24) The encoding device according to any one of items 16 to 23, characterized in that the compression means records flag information in the metadata of the compressed data indicating that the number of parameters in the attribute information of each of the plurality of elements is variable. (Item 25) The encoding device according to any one of items 16 to 24, characterized in that the attribute information indicates coefficients of a model that responds to color with respect to direction. (Item 26) The encoding device according to item 25, characterized in that the model for responding to color in the aforementioned direction is defined in Spherical Harmonics (SH), and the coefficients of the model are the coefficients of SH. (Item 27) The encoding device according to item 26, characterized in that the compression means generates the compressed data such that it shows the order of SH for each of the plurality of elements. (Item 28) The encoding device according to any one of items 26 to 27, characterized in that the compression means adaptively reduces the order of the SH for each of the plurality of elements, and compresses the element group data after the order of the SH has been reduced. (Item 29) The encoding device according to any one of items 16 to 28, characterized in that the compression means compresses the element group data so as to compress the attribute information which includes one or more of the opacity of each element, the standard deviation defining the magnitude of the spread, the orientation defining the direction of the spread, and the coefficient of Spherical Harmonics defining the color. (Item 30) The aforementioned set of elemental data represents the Gaussian primitive group of 3DGS, The encoding device according to any one of items 16 to 29, characterized in that the compression means compresses the element group data according to G-PCC. (Item 31) An acquisition means for acquiring element group data, which represents the position of each of a plurality of elements and the attribute information of each of the plurality of elements, wherein the attribute information includes a variable number of parameters. The system includes a compression means that generates compressed data by compressing the aforementioned element group data, The encoding device is characterized in that the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information, and compresses each of the divided element group data. (Item 32) A decoding method performed by a decoding device, A step of obtaining compressed data of a group of element data, wherein the group of element data represents the respective positions of a plurality of elements and attribute information of each of the plurality of elements, the attribute information indicates at least a scale, the attribute information includes a variable number of parameters, and the compressed data is configured to show the number of parameters that each of the plurality of elements' attribute information has; A step of decompressing the compressed data by referring to the number of parameters that each of the multiple elements' attribute information has, obtained based on the compressed data, A decoding method characterized by comprising: (Item 33) An encoding method performed by an encoding device, A step of acquiring element group data that represents the position of each of a plurality of elements and attribute information of each of the plurality of elements, wherein the attribute information indicates at least a scale and the attribute information includes a variable number of parameters; The process includes generating compressed data by compressing the aforementioned element group data, An encoding method characterized in that, in the step of generating the compressed data, the element group data is configured such that each of the plurality of elements has the number of parameters. (Item 34) An image processing device for processing compressed data of a set of elements, Decompression means for decompressing the compressed data and generating a first group of elements having at least attribute information indicating scale, A restoration means for generating a second set of elements by increasing the amount of data of the attribute information of the first set of elements through at least one of the following processes: transformation, inverse quantization, and assignment of a specified value; A control means that determines, according to predetermined conditions, whether to output the first element group or the second element group, and outputs either the first element group or the second element group according to the determination result, An image processing apparatus characterized by comprising: (Item 35) The image processing apparatus according to item 34, characterized in that the attribute information indicates at least one of posture, opacity, and color. (Item 36) The image processing apparatus according to any one of items 34 to 35, characterized in that the conversion is a conversion from luminance and color difference information to RGB information. (Item 37) The image processing apparatus according to any one of items 34 to 35, characterized in that the inverse quantization is vector inverse quantization. (Item 38) The image processing apparatus according to any one of items 34 to 37, wherein the compressed data includes information indicating whether or not to output the second set of elements, and if the information indicates that the second set of elements will not be output, the generation of the second set of elements by the restoration means is skipped, and the control means outputs the first set of elements. (Item 39) The image processing apparatus further comprises drawing means for drawing the group of elements, The image processing apparatus according to any one of items 34 to 38, characterized in that the control means determines whether the drawing means can draw a first group of elements, outputs the first group of elements if the drawing means can draw the first group of elements, and outputs the second group of elements if the drawing means cannot draw the first group of elements. (Item 40) A simplification means for generating a second set of elements by reducing the amount of data of at least scale-indicating attribute information possessed by the first set of elements through at least one of the following processes: transformation, quantization, and deletion. A control means for selecting either the first element group or the second element group, A compression means for compressing the group of elements selected by the control means, An encoding device characterized by comprising: (Item 41) The encoding device according to item 40, characterized in that the attribute information indicates at least one of posture, opacity, and color. (Item 42) The encoding device according to any one of items 40 to 41, characterized in that the conversion is a conversion from RGB information to luminance and color difference information. (Item 43) The encoding device according to any one of items 40 to 41, characterized in that the quantization is vector quantization. (Item 44) The encoding device according to any one of items 40 to 43, characterized in that the control means determines whether the amount of attribute information data possessed by the first group of elements has already been reduced by at least one of the following processes: transformation, quantization, and deletion, for at least one of the information possessed by the first group of elements, and if the amount of attribute information data possessed by the first group of elements has already been reduced, outputs the first group of elements to the compression means. (Item 45) The encoding device according to any one of items 40 to 44, characterized in that the control means outputs either the first group of elements or the second group of elements to the compression means based on the user's instructions. (Item 46) An image processing method performed by an image processing device that processes compressed data of a set of elements, The steps include decompressing the compressed data to generate a first set of elements having at least attribute information indicating scale, A step of generating a second set of elements by increasing the amount of data of the attribute information of the first set of elements through at least one of the following processes: transformation, inverse quantization, and assignment of a specified value; A step of determining whether to output the first group of elements or the second group of elements according to predetermined conditions, and outputting either the first group of elements or the second group of elements according to the determination result, An image processing method characterized by comprising: (Item 47) An encoding method performed by an encoding device, A step of generating a second set of elements by reducing the amount of data of attribute information that at least indicates scale, which is present in the first set of elements, through at least one of the following processes: transformation, quantization, and deletion. A step of selecting either the first group of elements or the second group of elements, A step of compressing the group of elements selected in the above selection step, An encoding method characterized by comprising: (Item 48) A program for causing a computer to function as a decoding device according to any one of Items 1 to 15. (Item 49) A program for causing a computer to function as an encoding device according to any one of Items 16 to 31. (Item 50) A program for causing a computer to function as an image processing device according to any one of Items 34 to 39. (Item 51) A program for causing a computer to function as an encoding device according to any one of Items 40 to 45.

[0193] The invention is not limited to the above embodiments, and various changes and modifications are possible without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Description of Signs

[0194] 101: Image processing device, 102: Image acquisition unit, 103: Initial point cloud acquisition unit, 104: Model optimization unit, 105: Drawing unit, 106: Model compression unit, 107: Output unit, 1101: Display device, 1102: Input unit, 1103: Model expansion unit, 1104: Operation unit, 1105: Drawing unit, 1106: Display unit, 1601: Encoding device, 1602: Simplification unit, 1603: Compression unit, 1801: Decoding device, 1802: Expansion unit, 1803: Restoration unit, 1806: Control unit

Claims

1. Compressed data of a group of elements, wherein the group of elements includes the position of each of the multiple elements, An acquisition means for acquiring the compressed data, wherein the attribute information of each of the plurality of elements represents, the attribute information indicates at least a scale, the attribute information includes a variable number of parameters, and the compressed data is configured to indicate the number of parameters that each of the attribute information of the plurality of elements has, A decompression means for decompressing the compressed data by referring to the number of parameters that each of the multiple elements' attribute information has obtained based on the compressed data, A decoding device characterized by comprising:

2. The compressed data includes compressed element group data, which are compressed data for each of one or more element group data, and in one compressed element group data, the number of parameters included in the attribute information is the same. The decoding device according to claim 1, wherein the decompression means decompresses each of the compressed element group data by referring to the number of parameters included in the attribute information in each of the compressed element group data.

3. The decoding device according to claim 2, wherein the decompression means integrates the respective element group data obtained by decompressing each of the compressed element group data.

4. The compressed data includes a bitstream corresponding to each of one or more elemental data sets. In the element set data corresponding to a single bitstream, the number of parameters included in the attribute information is the same. The number of parameters included in the attribute information for each element data is indicated by the metadata contained in the file storing the compressed data, or by the storage order of the bitstream in the file. The decoding device according to claim 1, wherein the decompression means decompresses each bitstream by referring to the number of parameters included in the attribute information in each element group data.

5. The compressed data includes a compressed element group sequence corresponding to one or more element group data, and the number of parameters included in the attribute information is the same in the element group data corresponding to one compressed element group sequence. The number of parameters included in the attribute information for each element group data is indicated by the parameters of the compressed element group sequence. The decoding device according to claim 1, wherein the decompression means decompresses each compressed element group sequence by referring to the number of parameters included in the attribute information in each element group data.

6. The compressed data includes a compressed element sequence divided into tiles, each corresponding to one or more element group data, and the number of parameters included in the attribute information is the same in the element group data corresponding to one tile. The number of parameters included in the attribute information for each element data set is indicated by the tile parameters. The decoding device according to claim 1, wherein the decompression means decompresses each compressed element group sequence by referring to the number of parameters included in the attribute information in each element group data.

7. The compressed data includes, as attribute information, information indicating the number of parameters that each of the plurality of elements has, The decoding device according to claim 1, characterized in that the decompression means decompresses information indicating the number of parameters each of the plurality of elements has.

8. The compressed data includes flag information indicating that the number of parameters in the attribute information of each of the plurality of elements is variable. The decoding device according to claim 1, wherein the decompression means decompresses the compressed data by referring to the flag information.

9. The decoding device according to claim 1, characterized in that the attribute information indicates coefficients of a model that responds to color with respect to direction.

10. The decoding device according to claim 9, characterized in that the decompression means increases a parameter indicating a coefficient of a model that responds to color in the direction to a predetermined number after the decompression of the compressed data.

11. The decoding device according to claim 9, characterized in that the decompression means reduces some of the parameters that indicate the coefficients of a model that responds to color in direction, obtained by decompressing the compressed data, to not output them, according to the number of parameters that the attribute information of each of the plurality of elements has.

12. The decoding device according to claim 9, characterized in that the model for responding to color in the aforementioned direction is defined in Spherical Harmonics (SH), and the coefficients of the model are the coefficients of SH.

13. The compressed data is configured to show the order of SH for each of the multiple elements, The decoding device according to claim 12, characterized in that the expansion means obtains the number of coefficients of SH that each of the attribute information of the plurality of elements has, based on the order of SH.

14. The decoding device according to claim 1, characterized in that the decompression means decompresses the compressed data and outputs one or more of the following as attribute information: opacity of each element, standard deviation defining the magnitude of the spread, orientation defining the direction of the spread, and coefficient of Spherical Harmonics defining the color.

15. The aforementioned set of elemental data represents the Gaussian primitive group of 3DGS, The aforementioned compressed data is compressed according to G-PCC. The decoding device according to claim 1, wherein the decompression means decompresses the compressed data according to G-PCC.

16. Element group data representing the position of each of the multiple elements and the attribute information of each of the multiple elements, wherein the attribute information at least indicates a scale. The attribute information includes a variable number of parameters, and the acquisition means acquires the element group data, The system includes a compression means that generates compressed data by compressing the aforementioned element group data, The encoding device is characterized in that the compression means configures the compressed data to indicate the number of parameters each of the plurality of elements has.

17. The encoding device according to claim 16, characterized in that the compression means compresses one or more of the element group data, and in one of the element group data, the number of parameters included in the attribute information is the same.

18. The encoding device according to claim 16, wherein the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information, and compresses each of the divided element group data.

19. The encoding device according to claim 16, wherein the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information and the positional relationship of the plurality of elements, and compresses each of the divided element group data.

20. The compression means generates one or more bitstreams by compressing one or more of the element data sets, and serializes the one or more bitstreams and stores them in the file such that the number of parameters included in the attribute information in the element data sets corresponding to each bitstream is indicated by the metadata in the file or the storage order of the bitstreams in the file. The encoding device according to claim 16, characterized in that the number of parameters included in the attribute information is the same in one of the element group data.

21. A sequence is defined for each of the one or more element data sets, the compression means generates a single bitstream by compressing each element data set, and the parameters of the sequence include information indicating the number of parameters included in the attribute information. The encoding device according to claim 16, characterized in that the number of parameters included in the attribute information is the same in one of the element group data.

22. A tile is defined for each of the one or more element data sets, the compression means compresses each of the tiles, and describes in the tile's parameters information the number of parameters included in the attribute information, The encoding device according to claim 16, characterized in that the number of parameters included in the attribute information is the same in one of the element group data.

23. The encoding device according to claim 16, characterized in that the element group data compressed by the compression means includes, as attribute information, information indicating the number of parameters each of the plurality of elements has.

24. The encoding device according to claim 16, characterized in that the compression means records flag information in the metadata of the compressed data indicating that the number of parameters in the attribute information of each of the plurality of elements is variable.

25. The encoding device according to claim 16, characterized in that the attribute information indicates coefficients of a model that responds to color with respect to direction.

26. The encoding device according to claim 25, characterized in that the model for responding to color in the aforementioned direction is defined by Spherical Harmonics (SH), and the coefficients of the model are the coefficients of SH.

27. The encoding device according to claim 26, characterized in that the compression means generates the compressed data such that it indicates the order of SH for each of the plurality of elements.

28. The encoding apparatus according to claim 26, characterized in that the compression means adaptively reduces the order of the SH for each of the plurality of elements, and compresses the element group data after the order of the SH has been reduced.

29. The encoding device according to claim 16, characterized in that the compression means compresses the element group data so as to compress the attribute information which includes one or more of the opacity of each element, the standard deviation defining the magnitude of the spread, the orientation defining the direction of the spread, and the coefficient of Spherical Harmonics defining the color.

30. The aforementioned set of elemental data represents the Gaussian primitive group of 3DGS, The encoding device according to claim 16, wherein the compression means compresses the element group data according to G-PCC.

31. An acquisition means for acquiring element group data, which represents the position of each of a plurality of elements and the attribute information of each of the plurality of elements, wherein the attribute information includes a variable number of parameters. The system includes a compression means that generates compressed data by compressing the aforementioned element group data, The encoding device is characterized in that the compression means divides the element group data into a plurality of element group data according to the number of parameters included in the attribute information, and compresses each of the divided element group data.

32. A decoding method performed by a decoding device, A step of obtaining compressed data of a group of element data, wherein the group of element data represents the respective positions of a plurality of elements and attribute information of each of the plurality of elements, the attribute information indicates at least a scale, the attribute information includes a variable number of parameters, and the compressed data is configured to show the number of parameters that each of the plurality of elements' attribute information has; A step of decompressing the compressed data by referring to the number of parameters that each of the multiple elements' attribute information has, obtained based on the compressed data, A decoding method characterized by comprising the following:

33. An encoding method performed by an encoding device, A step of acquiring element group data that represents the position of each of a plurality of elements and attribute information of each of the plurality of elements, wherein the attribute information indicates at least a scale and the attribute information includes a variable number of parameters; The process includes generating compressed data by compressing the aforementioned element group data, An encoding method characterized in that, in the step of generating the compressed data, the element group data is configured such that each of the plurality of elements has the number of parameters.

34. An image processing device for processing compressed data of a set of elements, Decompression means for decompressing the compressed data and generating a first group of elements having at least attribute information indicating scale, A restoration means for generating a second set of elements by increasing the amount of data of the attribute information of the first set of elements through at least one of the following processes: transformation, inverse quantization, and assignment of a specified value; A control means that determines whether to output the first element group or the second element group according to predetermined conditions, and outputs either the first element group or the second element group according to the determination result, An image processing apparatus characterized by comprising:

35. The image processing apparatus according to claim 34, characterized in that the attribute information indicates at least one of posture, opacity, and color.

36. The image processing apparatus according to claim 34, characterized in that the conversion is a conversion from luminance and color difference information to RGB information.

37. The image processing apparatus according to claim 34, characterized in that the inverse quantization is vector inverse quantization.

38. The image processing apparatus according to claim 34, wherein the compressed data includes information indicating whether or not to output the second set of elements, and if the information indicates that the second set of elements will not be output, the generation of the second set of elements by the restoration means is skipped, and the control means outputs the first set of elements.

39. The image processing apparatus further comprises drawing means for drawing the group of elements, The image processing apparatus according to claim 34, characterized in that the control means determines whether the drawing means can draw a first group of elements, outputs the first group of elements if the drawing means can draw the first group of elements, and outputs the second group of elements if the drawing means cannot draw the first group of elements.

40. A simplification means for generating a second set of elements by reducing the amount of data of at least scale-indicating attribute information possessed by the first set of elements through at least one of the following processes: transformation, quantization, and deletion. A control means for selecting either the first element group or the second element group, A compression means for compressing the group of elements selected by the control means, An encoding device characterized by comprising:

41. The encoding device according to claim 40, characterized in that the attribute information indicates at least one of posture, opacity, and color.

42. The encoding device according to claim 40, characterized in that the conversion is a conversion from RGB information to luminance and color difference information.

43. The encoding device according to claim 40, characterized in that the quantization is vector quantization.

44. The encoding device according to claim 40, characterized in that the control means determines whether the amount of attribute information data possessed by the first group of elements has already been reduced by at least one of the following processes: transformation, quantization, and deletion, for at least one of the information possessed by the orientation, scale, opacity, and color, and if the amount of attribute information data possessed by the first group of elements has already been reduced, outputs the first group of elements to the compression means.

45. The encoding apparatus according to claim 40, characterized in that the control means outputs either the first element group or the second element group to the compression means based on the user's instructions.

46. An image processing method performed by an image processing device that processes compressed data of a set of elements, The steps include decompressing the compressed data to generate a first set of elements having at least attribute information indicating scale, A step of generating a second set of elements by increasing the amount of data of the attribute information of the first set of elements through at least one of the following processes: transformation, inverse quantization, and assignment of a specified value; A step of determining whether to output the first group of elements or the second group of elements according to predetermined conditions, and outputting either the first group of elements or the second group of elements according to the determination result, An image processing method characterized by comprising:

47. An encoding method performed by an encoding device, A step of generating a second set of elements by reducing the amount of data of at least scale-indicating attribute information possessed by the first set of elements through at least one of the following processes: transformation, quantization, and deletion. A step of selecting either the first group of elements or the second group of elements, A step of compressing the group of elements selected in the above selection step, An encoding method characterized by comprising:

48. A program for causing a computer to function as a decoding device according to any one of claims 1 to 15.

49. A program for causing a computer to function as an encoding device according to any one of claims 16 to 31.

50. A program for causing a computer to function as an image processing device according to any one of claims 34 to 39.

51. A program for causing a computer to function as an encoding device according to any one of claims 40 to 45.

Images