Information processing apparatus and method

By rendering based on voxel shapes and generating bitstreams containing rendering information, the problem of inconsistent rendering quality is solved, and the rendering effect is improved.

CN118941688APending Publication Date: 2024-11-12SONY GROUP CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411049930.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-01-16
Filing Date
2019-01-07
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In existing technologies, the constant rendering method leads to inconsistent voxel shapes, resulting in reduced rendering quality and gap or overlap issues.

Method used

The voxel data is rendered according to the shape of each voxel, and a bitstream containing rendering information is generated to ensure that the rendering result matches the voxel shape.

Benefits of technology

By using a rendering method that matches voxel shapes, gaps and overlaps between rendering results are suppressed, thus improving rendering quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941688A_ABST
    Figure CN118941688A_ABST
Patent Text Reader

Abstract

The present disclosure relates to a decoding device comprising circuitry configured to: acquire a bitstream with respect to a point cloud standardized using a bounding box; and decoding the bit stream to obtain quantization information of position information of points of the point cloud and metadata representing a form of the points, where the form of the points is represented as a regular cube; and rendering the points to have a regular cubic shape.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the PCT application with international application number PCT / JP2019 / 000050 filed on January 7, 2019 and invention name “Information Processing Device and Method”. The date on which the PCT application entered the Chinese national phase is July 9, 2020, and the national application number is 201980007871.7. Technical Field

[0002] The present disclosure relates to an information processing apparatus and an information processing method, and particularly to an information processing apparatus and an information processing method capable of suppressing a reduction in the quality of a rendering result. Background Art

[0003] Generally, there is known a method of encoding using voxels (e.g., octree) for compressing a point cloud representing a three-dimensional structure by position information, attribute information, etc. about the point group, or compressing vertex data about a mesh configured with vertices, edges, and faces and using a polygonal representation to define a three-dimensional shape (e.g., refer to non-patent document 1).

[0004] In this encoding, a bounding box is set to normalize the position information about the object to be encoded. Currently, the main voxelization method used for normalization so far is a method for determining voxels on each of the x-axis, y-axis, and z-axis by the same number of divisions. Therefore, the shape of each voxel always depends on the setting of the bounding box and is not necessarily constant.

[0005] Reference List

[0006] Non-patent literature

[0007] Non-Patent Literature 1 (NPL 1)

[0008] R.Mekuria, IEEE Student Member, K.Blom, P.Cesar., IEEE Member, "Design, Implementation and Evaluation of a Point Cloud Codec for Tele-Immersive Video", tcsvt_paper_submitted_february.pdf Summary of the invention

[0009] Technical issues

[0010] However, the rendering method is constant regardless of the shape of the voxel. Therefore, the shape of the rendering result is inconsistent with the shape of the voxel, which may lead to a reduction in the quality (appearance quality) of the rendering result, such as gaps or overlaps between the rendering results of each voxel.

[0011] The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to suppress degradation in the quality of rendering results.

[0012] Solution to the problem

[0013] An information processing device according to an aspect of the present technology is an information processing device including a rendering section that performs rendering on voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure, according to the shape of each voxel.

[0014] An information processing method according to an aspect of the present technology is an information processing method including performing rendering on voxel data including 3D data obtained by using voxel-quantized position information and representing a three-dimensional structure according to the shape of each voxel.

[0015] An information processing device according to another aspect of the present technology is an information processing device including a bitstream generating unit, which generates a bitstream containing encoded data about voxel data and rendering information, wherein the voxel data includes 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and the rendering information includes information related to the rendering of the voxel data.

[0016] An information processing method according to another aspect of the present technology is an information processing method for generating a bit stream containing encoded data about voxel data and rendering information, wherein the voxel data includes 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and the rendering information includes information related to the rendering of the voxel data.

[0017] An information processing device according to another aspect of the present technology is an information processing device including a rendering unit that performs rendering on voxel data based on rendering information, the rendering information including information related to the rendering of voxel data, the voxel data including 3D data obtained by using voxel quantization position information and representing a three-dimensional structure.

[0018] An information processing method according to still another aspect of the present technology is an information processing method including performing rendering on voxel data based on rendering information, the rendering information including information related to rendering of voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure.

[0019] In an information processing device and an information processing method according to an aspect of the present technology, rendering is performed on voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure according to the shape of each voxel.

[0020] In an information processing device and an information processing method according to another aspect of the present technology, a bit stream containing encoded data about voxel data and rendering information is generated, wherein the voxel data includes 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and the rendering information includes information related to the rendering of the voxel data.

[0021] In an information processing device and an information processing method according to another aspect of the present technology, rendering is performed on voxel data based on rendering information, which includes information related to rendering of voxel data, and the voxel data includes 3D data obtained by using voxel quantization position information and representing a three-dimensional structure.

[0022] According to one aspect of the present technology, a decoding device includes a circuit system configured to: obtain a bit stream about a point cloud normalized using a bounding box; and decode the bit stream to obtain quantized information of position information of points of the point cloud and metadata representing the form of the points, wherein the form of the points is represented as a regular cube; and render the points to have a regular cube shape.

[0023] Beneficial effects of the present invention

[0024] According to the present disclosure, information can be processed, and in particular, the degradation of the quality of the rendering result can be suppressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] [ Figure 1 ] Figure 1 Illustration depicting an example of a point cloud.

[0026] [ Figure 2 ] Figure 2 An illustration depicting an example of a rendering state.

[0027] [ Figure 3 ] Figure 3 An explanatory diagram depicting an example of the state of a rendering result.

[0028] [ Figure 4 ] Figure 4 is a block diagram depicting an example of a main configuration of an encoding device.

[0029] [ Figure 5 ] Figure 5 An explanatory diagram depicting an example of an overview of encoding.

[0030] [ Figure 6 ] Figure 6 is a diagram depicting an example of nonlinear region segmentation.

[0031] [ Figure 7 ] Figure 7 is a flowchart showing an example of the flow of encoding processing.

[0032] [ Figure 8 ] Figure 8 is a block diagram depicting an example of a main configuration of a decoding device.

[0033] [ Fig. 9 ] Fig. 9 An illustration depicting an example of a rendering state.

[0034] [ Fig.10 ] Fig.10 is a flowchart showing an example of the flow of a decoding process.

[0035] [ Fig.11 ] Fig.11 is a block diagram depicting an example of a main configuration of an encoding device.

[0036] [ Fig.12 ] Fig.12 : is a block diagram depicting an example of a main configuration of a voxelization section.

[0037] [ Fig.13 ] Fig.13 An illustration depicting an example of a rendering state.

[0038] [ Fig.14 ] Fig.14 This is an explanatory diagram of an example of the rendering state.

[0039] [ Fig.15 ] Fig.15 An illustrative diagram depicting an example of information sent using a signal.

[0040] [ Fig.16 ] Fig.16 is a flowchart showing an example of the flow of encoding processing.

[0041] [ Fig.17 ] Fig.17 : is a flowchart showing an example of the flow of voxelization processing.

[0042] [ Fig.18 ] Fig.18 is a block diagram depicting an example of a main configuration of a decoding device.

[0043] [ Fig.19 ] Fig.19 is a flowchart showing an example of the flow of a decoding process.

[0044] [ Fig. 20 ] Fig. 20 is a block diagram depicting an example of a main configuration of a computer. DETAILED DESCRIPTION

[0045] Hereinafter, a mode for carrying out the present disclosure (hereinafter, referred to as “embodiment”) will be described. Note that the description will be given in the following order.

[0046] 1. Encoding and decoding of 3D data

[0047] 2. First Implementation (Rendering Based on Voxel Shape)

[0048] 3. Second Implementation (Rendering Based on Metadata)

[0049] 4. Note

[0050] <1.3D data encoding and decoding>

[0051] <Point Cloud>

[0052] Generally, there are data such as a point cloud representing a three-dimensional structure by position information, attribute information, etc. on a point group, and a mesh configured with vertices, edges, faces and defining a three-dimensional shape using polygonal representation.

[0053] For example, in the case of point clouds, Figure 1 The three-dimensional structure depicted in A is represented as Figure 1 A collection of multiple points (point groups) depicted in B. In other words, point cloud data is configured with position information and attribute information (e.g., color) about each point in the point group. Therefore, the data structure is relatively simple, and any three-dimensional structure can be represented with sufficiently high accuracy by using enough points.

[0054] <Quantization and rendering using voxel position information>

[0055] However, since the data volume of data such as point clouds and meshes is relatively large, it is necessary to compress the data volume by encoding, etc. Encoding methods using voxels, such as octrees and KDTrees, have been designed. Voxels are data structures for quantizing position information about an object to be encoded.

[0056] In the case of voxel application, such as Figure 2 As depicted in A, the position information about the point is quantized as the center of the voxel. Therefore, when rendering, the point is rendered as a graphic centered at the center position of the voxel.

[0057] If the rendering result (rendering graphics) of each voxel is smaller than the voxel, the overall rendering result becomes a sparse point group, which may reduce the quality of the point group (image quality). On the contrary, if the rendering result of each voxel is larger than the voxel (the rendering result overflows from the voxel), the rendering results of each voxel overlap, which may reduce the quality of the overall rendering result (image quality).

[0058] To solve this problem, for example, Figure 2 As depicted in B, each voxel is rendered so that the rendering result (rendering pattern) of the voxel is the maximum value within the voxel. Thus, the gap between the rendering results of each voxel can be reduced and the generation of overlap can be prevented.

[0059] exist Figure 2 In the case of example B, the voxel is rendered as a sphere with a radius of r or as a regular cube with a side of I. Each voxel is rendered so that any such figure (sphere or cube) is as large as possible within the voxel, so that the gap can be narrowed and the overlap between voxels can be reduced. Therefore, the reduction in the quality (image quality) of the overall rendering result can be suppressed.

[0060] However, it is necessary to use such voxels for encoding in order to standardize the position information about the object to be encoded as a prerequisite, and therefore, in order to satisfy this prerequisite, a job for setting a solid body called a "bounding box" and containing the object to be encoded is performed. Currently, the main voxelization method used for standardization to date is a method for determining voxels on the x-axis, y-axis, and z-axis by the same number of divisions. Therefore, the shape of each voxel always depends on the setting of the bounding box and is not necessarily constant.

[0061] like Figure 3 As depicted in A, for example, the aspect ratio of the cube of each voxel depends on the aspect ratio of the bounding box, and this aspect ratio varies between voxels. Therefore, if a constant rendering method is used regardless of the shape of each voxel, the rendering result may overflow from the voxel and may produce gaps within the voxel, such as Figure 3 In other words, the quality (appearance quality) of the entire rendering result may be reduced.

[0062] Additionally, for example, the bounding box can be set as Figure 3 The cylindrical shape depicted in C. In this case, the shape of each voxel is, for example Figure 3 The shape depicted in D, which further differs from Figure 3 Therefore, if rendering is performed on voxels of such a shape by a method similar to the method for rendering rectangular voxels, it may be more difficult to suppress gaps and overlaps between rendering results. In other words, the quality of the entire rendering result may be further reduced.

[0063] <2. First Embodiment>

[0064] <Rendering based on the shape of each voxel>

[0065] In order to solve this problem, rendering is performed on voxel data according to the shape of each voxel, the voxel data being 3D data obtained by using voxel quantization position information and representing a three-dimensional structure. For example, an information processing device is configured with a rendering unit that performs rendering on voxel data according to the shape of each voxel, the voxel data being 3D data obtained by using voxel quantization position information and representing a three-dimensional structure.

[0066] By doing so, regardless of the shape of each voxel, gaps and overlaps between rendering results can be suppressed, and therefore, degradation in the quality of the rendering results can be suppressed.

[0067] <Encoding device>

[0068] A more specific example will be described below. First, an encoding device that encodes a point cloud using voxels will be described. Figure 4 is a block diagram depicting an example of the main configuration of such a decoding device. Figure 4 The encoding device 100 depicted in encodes data on a point cloud input as an object to be encoded using voxels, and outputs the resulting encoded data and the like.

[0069] like Figure 4 As depicted in , the encoding device 100 has a control section 101 , a preprocessing section 111 , a BB setting section 112 , a voxelization section 113 , an octree encoding section 114 , and a signal sequence encoding section 115 .

[0070] The control unit 101 performs processing related to the control of each processing unit in the encoding device 100. For example, the control unit 101 controls the execution or skipping (omission) of the processing performed by each processing unit. For example, the control unit 101 performs such control based on predetermined control information. By doing so, the control unit 101 can suppress the execution of unnecessary processing and suppress the increase of load.

[0071] Although the control unit 101 can adopt any configuration, the control unit 101 can have, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and the CPU can perform processing by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0072] The preprocessing section 111 performs predetermined processing as preprocessing on the point cloud data (object to be encoded) input to the encoding device 100 under the control of the control section 101 , and supplies the processed data to the BB setting section 112 .

[0073] Note that the point cloud used as an object to be encoded may be a moving image or a still image. In addition, the preprocessing may have any content. For example, the preprocessing unit 111 may perform processing for reducing noise or processing for changing the resolution (the number of points) as preprocessing. In addition, for example, the preprocessing unit 111 may update the arrangement of each point so that the density of the point group is uniform or has a desired deviation. In addition, for example, data other than the point cloud (such as image information with depth information) may be input to the encoding device 100, and the preprocessing unit 111 may convert the input data into data about the point cloud as preprocessing.

[0074] Although the preprocessing section 111 may adopt any configuration, the preprocessing section 111 may have, for example, a CPU, a ROM, a RAM, etc., and the CPU may perform preprocessing by loading a program and data stored in the ROM, etc. to the RAM to execute the program.

[0075] The BB setting section 112 performs processing related to setting of a bounding box for normalizing position information about an object to be encoded under the control of the control section 101. For example, the BB setting section 112 sets a bounding box for each object as an object to be encoded. Figure 5 In the case where the point cloud data depicted in A represents objects 131 and 132, the BB setting unit 112 is as follows Figure 5 As depicted in FIG. 1B , bounding boxes 141 and 142 are set to contain objects 131 and 132 , respectively. Return to reference Figure 4 When setting the bounding box, the BB setting unit 112 provides information related to the bounding box to the voxelization unit 113.

[0076] Note that although the BB setting section 112 may adopt any configuration, the BB setting section 112 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to setting of a bounding box by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0077] The voxelization section 113 performs processing related to quantization of position information on an object to be encoded under the control of the control section 101 .

[0078] For example, the voxelization unit 113 sets the voxels within the bounding box set by the BB setting unit 112. Figure 5 As depicted in FIG. 1C, the voxelization unit 113 divides the bounding box 141 and sets voxels 151. Then, the voxelization unit 113 quantizes (that is, voxelizes) the point cloud data within the bounding box by voxels. By doing so, the point cloud data is converted into voxel data including position information and attribute information (color information, etc.) about each voxel, as shown in FIG. Figure 5Depicted in D.

[0079] For example, the voxelization unit 113 equally divides the bounding box in the directions of the three axes of the x-axis, the y-axis, and the z-axis by a common number of divisions in the directions of the three axes of the x-axis, the y-axis, and the z-axis (three axes perpendicular to each other), and sets each divided region as a voxel. The common number of divisions in the three axes is included in a bit stream to be described later as metadata and is transmitted to the decoding side.

[0080] Alternatively, for example, the voxelization unit 113 equally divides the bounding box in the directions of the three axes (three axes perpendicular to each other) of the x-axis, y-axis, and z-axis by the number of divisions per axis (the number of divisions corresponding to each axis independent of each other), and sets each divided region as a voxel. The number of divisions corresponding to each axis is included in a bit stream to be described later as metadata, and is transmitted to the decoding side.

[0081] In another alternative embodiment, for example, the voxelization unit 113 divides the bounding box unequally in the directions of the three axes of the x-axis, the y-axis, and the z-axis (three axes perpendicular to each other), and sets each divided area as a voxel (also called a "non-linear voxel").

[0082] For example, Figure 6 is a schematic diagram representing a two-dimensional voxel. In this case, Figure 6 As depicted in , voxels different in size and shape from each other can be set in the same layer for the bounding box 161. In other words, small voxels can be increased by nonlinear voxels without increasing the number of layers, thereby making it possible to improve the quantization accuracy of more important regions.

[0083] In other words, in the case of nonlinear voxels, the quantization accuracy can be improved without increasing the number of voxel layers. In other words, the quantization accuracy can be improved while suppressing a decrease in encoding efficiency and an increase in processing time.

[0084] In this case, the voxelization section 113 sets identification information indicating a nonlinear voxel, and sets mapping information on the set voxel. The identification information and the mapping information are contained in a bit stream to be described later as metadata, and are transmitted to the decoding side.

[0085] It goes without saying that the voxel setting method may be any method and is not limited to these examples.

[0086] Note that, in the case where there are multiple bounding boxes, the voxelization unit 113 voxelizes the point cloud data of each bounding box. Figure 5 In the case of the example of B, the voxelization unit 113 performs similar processing on the bounding box 142. Figure 4The voxelization unit 113 provides the voxelized point cloud data (also referred to as “voxel data”) (information related to the data structure used for quantization of position information), attribute information, etc. to the octree encoding unit 114.

[0087] Note that although the voxelization unit 113 can adopt any configuration, the voxelization unit 113 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to the setting of voxels by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0088] The octree encoding unit 114 performs processing related to octree encoding under the control of the control unit 101. For example, the octree encoding unit 114 encodes the voxel data provided from the voxelization unit 113 based on its data structure (for example, performs octree encoding on it), and generates a signal sequence. The octree encoding unit 114 provides the generated signal sequence to the signal sequence encoding unit 115.

[0089] Note that although the octree encoding unit 114 can adopt any configuration, the octree encoding unit 114 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to the generation of the signal sequence by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0090] The signal sequence encoding section 115 performs processing related to the encoding of the provided signal sequence under the control of the control section 101. For example, the signal sequence encoding section 115 encodes the signal sequence provided by the octree encoding section 114 and generates encoded data (bit stream). The encoding method may be any method. The signal sequence encoding section 115 outputs the encoded data (bit stream) thus obtained to the outside of the encoding device 100. The data (encoded data and control information) output by the encoding device 100 may be decoded by, for example, a subsequent processing section not depicted to reconstruct data about a point cloud, may be transmitted by a communication section not depicted and transmitted via a predetermined transmission line to another device such as a decoding device (for example, the decoding device 200 described later), or may be recorded in a recording medium not depicted.

[0091] Note that although the signal sequence encoding unit 115 can adopt any configuration, the signal sequence encoding unit 115 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to encoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0092] <Encoding Process Flow>

[0093] Will refer to Figure 7The flowchart describes an example of the flow of the encoding process performed by the encoding device 100 configured as described above.

[0094] When the encoding process starts, in step S101 , the preprocessing section 111 performs preprocessing on input 3D data.

[0095] In step S102 , the BB setting unit 112 sets a bounding box for the pre-processed 3D data.

[0096] In step S103 , the voxelization unit 113 voxelizes the bounding box.

[0097] In step S104 , the octree encoding unit 114 encodes the voxel data based on the data structure and generates a signal sequence.

[0098] In step S105 , the signal sequence encoding unit 115 encodes the signal sequence.

[0099] In step S106 , the signal sequence encoding section 115 outputs a bit stream obtained by encoding (a bit stream including encoded data on 3D data and metadata such as control information) to the outside of the encoding device 100 .

[0100] When the process in step S106 ends, the encoding process ends. In the case where the object to be encoded is, for example, a moving image, a series of processes are performed for each frame.

[0101] <Decoding device>

[0102] Figure 8 : is a block diagram showing an example of a main configuration of a decoding device as one embodiment of an image processing device to which the present technology is applied. Figure 8 The decoding device 200 depicted in Figure 4 The decoding device 200 corresponds to the encoding device 100, which decodes the bit stream (a bit stream containing encoded data about 3D data, etc.) output by the encoding device 100, and reconstructs the 3D data. At this time, the decoding device 200 performs decoding (reconstruction) by applying the method of the present technology as described below.

[0103] like Figure 8 As depicted in , the decoding device 200 has a control section 201 , a signal sequence decoding section 211 , an octree decoding section 212 , a voxel shape determination section 213 , a rendering section 214 , a display control section 215 , and a display section 216 .

[0104] The control unit 201 performs processing related to the control of each processing unit in the decoding device 200. For example, the control unit 201 controls the execution or skipping (omission) of the processing performed by each processing unit. For example, the control unit 201 performs such control based on predetermined control information. By doing so, the control unit 201 can, for example, suppress the execution of unnecessary processing and suppress the increase in load.

[0105] Although the control section 201 may adopt any configuration, the control section 201 may have, for example, a CPU, a ROM, a RAM, and the like, and the CPU may perform processing by loading a program and data stored in the ROM and the like to the RAM to execute the program.

[0106] The signal sequence decoding unit 211 performs processing related to decoding of a bit stream including encoded data on 3D data representing a three-dimensional structure (encoded data of voxel data obtained by voxelizing the 3D data), etc., under the control of the control unit 201. The 3D data may be, for example, a point cloud.

[0107] For example, the signal sequence decoding unit 211 acquires a bit stream provided from the encoding device 100. In addition, for example, the signal sequence decoding unit 211 decodes the acquired bit stream containing the encoded data by a decoding method corresponding to the encoding performed by the encoding device 100 (signal sequence encoding unit 115). The signal sequence decoding unit 211 supplies a signal sequence obtained by decoding the encoded data to the octree decoding unit 212 together with metadata such as control information.

[0108] Note that although the signal sequence decoding unit 211 can adopt any configuration, the signal sequence decoding unit 211 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to decoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0109] The octree decoding section 212 performs processing related to octree decoding under the control of the control section 201. For example, the octree decoding section 212 decodes (for example, performs octree decoding on) the signal sequence provided from the signal sequence decoding section 211 based on its data structure. The octree decoding section 212 provides the voxel data obtained by decoding to the voxel shape determination section 213 together with metadata such as control information.

[0110] Note that although the octree decoding unit 212 can adopt any configuration, the octree decoding unit 212 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to decoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0111] The voxel shape determination section 213 performs processing related to determination of the shape of each voxel under the control of the control section 201. For example, the voxel shape determination section 213 determines the shape (form, size, etc.) of each voxel based on the voxel data, metadata, etc. supplied from the octree decoding section 212.

[0112] As described above, voxel data is obtained by quantifying position information about 3D data such as a point cloud representing a three-dimensional structure using voxels. In addition, voxels are obtained by segmenting the area of ​​a bounding box that normalizes the 3D data. The metadata contains information such as the size of the bounding box and the number of segmentations when setting voxels based on the bounding box. The voxel shape determination unit 213 determines the shape (form, size, etc.) of each voxel based on such pieces of information.

[0113] For example, in the case where the metadata contains information indicating the number of divisions common to three axes (three axes perpendicular to each other) among the x-axis, y-axis, and z-axis, the voxel shape determination unit 213 obtains (determines) the shape of each voxel by equally dividing the size of the bounding box by the common number of divisions in each of the x-axis, y-axis, and z-axis directions. In other words, the voxel shape determination unit 213 determines the shape of each voxel based on the number of divisions common to the three axes of the bounding box that normalizes the 3D data.

[0114] Alternatively, for example, in the case where the metadata contains information indicating the number of divisions of each of the three axes of the x-axis, y-axis, and z-axis (three axes perpendicular to each other) (the number of divisions corresponding to each axis), the voxel shape determination unit 213 obtains (determines) the shape of each voxel by equally dividing the size of the bounding box by the corresponding number of divisions in each of the x-axis, y-axis, and z-axis directions. In other words, the voxel shape determination unit 213 determines the shape of each voxel based on the number of divisions corresponding to each axis of the bounding box that normalizes the 3D data.

[0115] In another alternative embodiment, for example, in the case where the metadata contains mapping information about the voxels, the voxel shape determination unit 213 obtains (determines) the shape of each voxel from the mapping information. In other words, the voxel shape determination unit 213 determines the shape of each voxel based on information indicating the nonlinear segmentation of the bounding box that normalizes the 3D data. Note that the voxel shape determination unit 213 can use the size information about the bounding box as needed.

[0116] The voxel shape determination section 213 supplies the determination result of the shape of each voxel obtained as described above (information indicating the shape of each voxel) to the rendering section 214 together with the voxel data and the metadata.

[0117] Note that although the voxel shape determination unit 213 can adopt any configuration, the voxel shape determination unit 213 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to decoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0118] The rendering unit 214 performs processing related to image rendering under the control of the control unit 201. For example, the rendering unit 214 performs rendering according to the shape of each voxel based on the information indicating the voxel data, metadata, and the shape of each voxel, etc., which is provided from the voxel shape determination unit 213. In other words, the rendering unit 214 performs rendering according to the shape of each voxel determined by the voxel shape determination unit 213. Specifically, the rendering unit 214 renders the voxel data obtained by decoding the encoded data by the signal sequence decoding unit 211 according to the shape of each voxel.

[0119] For example, in a voxel the shape is like Fig. 9 In the case of a regular cube such as the voxel 231 depicted in A, the rendering unit 214 may render the voxel as Fig. 9 The voxel may be rendered as a sphere like the rendering results 232 and 233 depicted in B, or the voxel may be rendered as Fig. 9 The rendering results 234 and 235 depicted in B are regular cube shapes.

[0120] Furthermore, at this time, since the rendering section 214 grasps the shape of the voxel 231 based on the information indicating the shape of the voxel or the like, the rendering section 214 can perform rendering according to the shape of the voxel 231 subjected to rendering.

[0121] For example, the rendering results 232 and 234 are the results of rendering performed by the rendering section 214 so that the rendering results 232 and 234 are each inscribed in the voxel 231. By performing rendering in this manner, the rendering section 214 can enhance the rendering result as much as possible within a range where the rendering result does not exceed the voxel 231. In other words, by performing rendering in this manner, the rendering section 214 can reduce the gap as much as possible while suppressing the generation of overlaps between the rendering results of the respective voxels. Therefore, it is possible to suppress the quality reduction of the rendering result.

[0122] Note that if there is an error in the rendering result or in the determination result of the voxel shape, the error may cause the generation of overlaps between the rendering results when each rendering result is inscribed in the voxel. To solve this problem, a margin (space) may be given between each rendering result and the voxel without inscribing the rendering result in the voxel, so as to allow the generation of overlaps between the rendering results to be more suppressed. For example, in Fig. 9In the case of the rendering results 233 and 235 depicted in B, rendering is performed so that Fig. 9 The rendering results 233 and 235 depicted in B are smaller than the rendering results 232 and 234. Therefore, the gap between each rendering result and the voxel becomes larger, and the generation of overlap between the rendering results is suppressed as much as the larger gap.

[0123] Alternatively, for example, the shape of the voxel is like Fig. 9 In the case of a cube (rectangular parallelepiped) such as the voxel 241 depicted in C, the rendering unit 214 can render the voxel into an ellipsoidal shape such as the rendering results 242 and 243, or as Fig. 9 As depicted in D of , the voxel can be rendered in a cubic shape (rectangular parallelepiped shape) like the rendering results 244 and 245 .

[0124] Furthermore, at this time, since the rendering section 214 grasps the shape of the voxel 241 based on the information indicating the voxel shape or the like, the rendering section 214 can perform rendering according to the shape of the voxel 241 subjected to rendering. Fig. 9 The rendering results 242 and 244 depicted in D are the results of rendering performed by the rendering unit 214, so that the rendering results 242 and 244 are each inscribed in the voxel 241. By performing rendering in this way, the rendering unit 214 can expand the rendering result as much as possible within the range where the rendering result does not exceed the voxel 241. In other words, the rendering unit 214 can reduce the gap as much as possible while suppressing the generation of overlap between the rendering results of each voxel by performing rendering in this way. Therefore, it is possible to suppress the reduction in the quality of the rendering result.

[0125] Note that in this case, similarly to the above case, a margin (space) may be given between each rendering result and the voxel without inscribing the rendering result in the voxel, to allow more suppression of the generation of overlap between the rendering results. Fig. 9 In the case of the rendering results 243 and 245 depicted by D, rendering is performed so that the rendering results 243 and 245 are smaller than Fig. 9 Rendering results 242 and 244 are depicted by D. Therefore, the gap between each rendering result and the voxel becomes larger, and the generation of overlap between the rendering results is suppressed as much as the larger gap.

[0126] The rendering section 214 supplies the rendering result (that is, 3D data) obtained as described above to the display control section 215 .

[0127] Note that although the rendering section 214 may adopt any configuration, the rendering section 214 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to decoding by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0128] The display control section 215 performs processing related to display control under the control of the control section 201. For example, the display control section 215 acquires 3D data provided from the rendering section 214 and indicating a rendering result. In addition, for example, the display control section 215 generates a 2D display image based on the 3D data. Then, the display control section 215 provides the display image to the display section 216.

[0129] Note that although the display control unit 215 may adopt any configuration, the display control unit 215 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to decoding by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0130] The display section 216 performs processing related to image projection under the control of the control section 201. For example, the display section 216 acquires a display image provided from the display control section 215. In addition, the display section 216 displays the acquired display image on a display device owned by the display section 216. In other words, the display section 216 displays an image generated by performing rendering according to the shape of each voxel of the rendering section 214.

[0131] By performing the above-described processing, regardless of the shape of each voxel, gaps and overlaps between rendering results can be suppressed, and thus degradation in the quality of the rendering results can be suppressed.

[0132] <Decoding Process Flow>

[0133] Will refer to Fig.10 The flowchart describes an example of the flow of a decoding process performed by the decoding device 200 configured as described above.

[0134] When the decoding process starts, in step S201 , the signal sequence decoding unit 211 decodes an input bit stream and obtains a signal sequence.

[0135] In step S202 , the octree decoding unit 212 decodes the signal sequence based on its data structure and obtains voxel data.

[0136] In step S203 , the voxel shape determination unit 213 determines the shape of each voxel of the voxel data.

[0137] In step S204 , the rendering section 214 performs rendering on the voxel data based on the shape of each voxel determined in step S203 .

[0138] In step S205 , the display control unit 215 generates a 2D display image using the rendering result (3D data).

[0139] In step S206 , the display unit 216 displays the display image obtained in step S205 .

[0140] When the process in step S206 ends, the decoding process ends. For example, in the case where the object to be decoded is a moving image, a series of processes are performed for each frame.

[0141] By performing the above-described decoding process, the decoding device 200 can suppress gaps and overlaps between rendering results regardless of the shape of each voxel, and thus suppress degradation in the quality of the rendering results.

[0142] <3. Second Embodiment>

[0143] <Signaling of rendering information>

[0144] Note that when encoding 3D data as described above, the encoding side can set the rendering method of the decoding side and signal the method to the decoding side (provide the method as metadata, etc.). The decoding side can then perform rendering based on this information (that is, perform rendering by the method specified by the encoding side).

[0145] In other words, a bitstream containing coded data on voxel data, which is 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and rendering information, which is information related to rendering of the voxel data, is generated. For example, the information processing device may be configured with a bitstream generation unit that generates a bitstream containing coded data on voxel data, which is 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and rendering information, which is information related to rendering of the voxel data.

[0146] By doing so, regardless of the shape of each voxel, gaps and overlaps between rendering results can be suppressed, and thus, degradation in the quality of the rendering results can be suppressed.

[0147] <Encoding device>

[0148] A more specific example will be described below. First, an encoding apparatus for encoding a point cloud using voxels will be described. Fig.11 1 is a block diagram depicting an example of the main configuration of such a decoding device. Fig.11The encoding device 300 depicted in encodes data on a point cloud input as an object to be encoded using voxels, and outputs the resulting encoded data, etc.

[0149] like Fig.11 As shown, the encoding device 300 includes a control unit 301 , a preprocessing unit 311 , a BB setting unit 312 , a voxelization unit 313 , an information extraction unit 314 , an octree encoding unit 315 , a signal sequence encoding unit 316 , and an association unit 317 .

[0150] The control section 301 is a processing section similar to the control section 101, and performs processing related to the control of each processing section within the encoding device 300. For example, the control section 301 controls the execution or skipping (omission) of the processing performed by each processing section. For example, the control section 301 performs such control based on predetermined control information. By doing so, the control section 301 can suppress the execution of unnecessary processing and suppress the increase in load.

[0151] Although the control section 301 may adopt any configuration, the control section 301 may have, for example, a CPU, a ROM, a RAM, and the like, and the CPU may perform processing by loading a program and data stored in the ROM and the like to the RAM to execute the program.

[0152] The preprocessing section 311 performs processing related to preprocessing under the control of the control section 301. For example, the preprocessing section 311 acquires 3D data (e.g., point cloud data) input to the encoding device 300 as an object to be encoded (arrow 321). In addition, the preprocessing section 311 performs predetermined processing on the acquired 3D data as preprocessing. In addition, the preprocessing section 311 provides the appropriately processed data to the BB setting section 312 (arrow 322).

[0153] Note that the point cloud used as an object to be encoded may be a moving image or a still image. In addition, the preprocessing may have any content. For example, the preprocessing unit 311 may perform processing for reducing noise or processing for changing the resolution (the number of points) as preprocessing. In addition, for example, the preprocessing unit 311 may update the arrangement of each point so that the density of the point group is uniform or has a desired deviation. In addition, for example, data other than the point cloud (such as image information with depth information) may be input to the encoding device 300, and the preprocessing unit 311 may convert the input data into data about the point cloud as preprocessing.

[0154] Although the preprocessing section 311 may adopt any configuration, the preprocessing section 311 may have, for example, a CPU, a ROM, a RAM, etc., and the CPU may perform preprocessing by loading a program and data stored in the ROM, etc. to the RAM to execute the program.

[0155] Similar to the BB setting section 112, the BB setting section 312 performs processing related to the setting of a bounding box for standardizing position information about an object to be encoded under the control of the control section 301. For example, the BB setting section 312 acquires 3D data provided by the preprocessing section 311 and having been subjected to appropriate preprocessing (arrow 322). In addition, the BB setting section 312 sets a bounding box for each object of the 3D data that is an object to be encoded. This processing is similar in content to the processing performed by the BB setting section 112. In addition, the BB setting section 312 provides information related to the set bounding box together with the 3D data to the voxelization section 313 (arrow 323).

[0156] Note that although the BB setting section 312 may adopt any configuration, the BB setting section 312 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to setting of a bounding box by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0157] Similar to the voxelization section 113, the voxelization section 313 performs processing related to quantization of position information about the object to be encoded under the control of the control section 301. For example, the voxelization section 313 acquires data supplied from the BB setting section 312 (arrow 323). In addition, the voxelization section 313 voxelizes the 3D data using the data, and generates voxel data. Furthermore, the voxelization section 313 supplies the generated voxel data to the information extraction section 314 together with other data (arrow 324).

[0158] The process for voxelization may have any content. For example, the voxelization unit 313 may set voxels within the bounding box set by the BB setting unit 312. In addition, the voxelization unit 313 may generate distribution information indicating the distribution state of the 3D data. In addition, the voxelization unit 313 may quantize the position information about the 3D data. In addition, the voxelization unit 313 may set a method of rendering voxel data obtained by quantizing the position information about the 3D data.

[0159] Note that although the voxelization unit 313 can adopt any configuration, the voxelization unit 313 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to the setting of voxels by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0160] The information extraction section 314 performs processing related to the extraction of information under the control of the control section 301. For example, the information extraction section 314 acquires data provided from the voxelization section 313 (arrow 324). In addition, the information extraction section 314 extracts desired information from the acquired data, and generates information related to rendering (rendering information). Furthermore, the information extraction section 314 supplies the rendering information as metadata to the association section 317 (arrow 326). Furthermore, the information extraction section 314 supplies the voxel data to the octree encoding section 315 (arrow 325).

[0161] The information extracted by the information extraction section 314 (that is, rendering information) may have any content. For example, the information may contain information about the bounding box set by the BB setting section 312. In addition, the information may contain information about the voxel set by the voxelization section 313. Furthermore, the information may contain distribution information generated by the voxelization section 313. Furthermore, the information may contain information about the rendering method set by the voxelization section 313. The detailed content of the rendering information will be described later.

[0162] Note that although the information extraction unit 314 can adopt any configuration, the information extraction unit 314 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to the setting of voxels by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0163] Similar to the octree encoding section 114, the octree encoding section 315 performs processing related to octree encoding under the control of the control section 301. For example, the octree encoding section 315 acquires the voxel data provided from the information extraction section 314 (arrow 325). In addition, the octree encoding section 315 encodes the acquired voxel data based on its data structure (for example, performs octree encoding on it), and generates a signal sequence. In addition, the octree encoding section 315 provides the generated signal sequence to the signal sequence encoding section 316 (arrow 327).

[0164] Note that although the octree encoding unit 315 can adopt any configuration, the octree encoding unit 315 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to the generation of the signal sequence by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0165] Similar to the signal sequence encoding unit 115, the signal sequence encoding unit 316 performs processing related to the encoding of the provided signal sequence under the control of the control unit 301. For example, the signal sequence encoding unit 316 obtains the signal sequence provided by the octree encoding unit 315 (arrow 327). In addition, the signal sequence encoding unit 316 encodes the obtained signal sequence and generates encoded data (bit stream). The encoding method may be any method. The signal sequence encoding unit 316 provides the encoded data obtained in this way to the association unit 317.

[0166] Note that although the signal sequence encoding unit 316 can adopt any configuration, the signal sequence encoding unit 316 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to encoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0167] The association section 317 performs processing related to association of data under the control of the control section 301. For example, the association section 317 acquires metadata (rendering information) provided from the information extraction section 314 (arrow 326). In addition, the association section 317 acquires the encoded data provided from the signal sequence encoding section 316 (arrow 327). Furthermore, the association section 317 associates the acquired metadata with the acquired encoded data. For example, the association section 317 adds the metadata to a bit stream containing the encoded data.

[0168] As described above, the encoded data provided from the signal sequence encoding unit 316 is obtained by encoding voxel data. In addition, the voxel data is obtained by quantizing the position information about 3D data (such as point cloud) representing a three-dimensional structure using voxels. In addition, the voxels are obtained by segmenting the area of ​​the bounding box that normalizes the 3D data.

[0169] Furthermore, the metadata includes not only information such as the size of the bounding box and the number of divisions when setting voxels from the bounding box, but also rendering information and the like.

[0170] In other words, the associating section 317 generates a bit stream including encoded data on voxel data which is 3D data obtained by using voxel quantization position information and indicating a three-dimensional structure, and rendering information which is information related to rendering of the voxel data.

[0171] Furthermore, the associating section 317 outputs the bit stream to the outside of the encoding device 300 (arrow 329). The data (encoded data and control information) output by the encoding device 300 may be decoded by, for example, a subsequent processing section not depicted to reconstruct data on a point cloud, may be transmitted by a communication section not depicted and transmitted to another device such as a decoding device (for example, a decoding device 400 described later) via a predetermined transmission line, or may be recorded in a recording medium not depicted.

[0172] Although the associating section 317 may adopt any configuration, the associating section 317 may have, for example, a CPU, a ROM, a RAM, etc., and the CPU may load a program and data stored in the ROM, etc. into the RAM to execute the program to perform processing related to encoding.

[0173] <Configuration of Voxelization Unit>

[0174] Fig.12 31 is a block diagram depicting an example of a main configuration of the voxelization section 313 . Fig.12 The voxelization section 313 in the example depicted in has a voxel setting section 341 , a distribution information generating section 342 , a rendering method setting section 343 , and a position quantization section 344 .

[0175] The voxel setting unit 341 performs processing related to the setting of voxels for quantizing position information. For example, the voxel setting unit 341 obtains the BB setting unit 312 ( Fig.11 ) provided by the BB setting section 312 (arrow 351). In addition, the voxel setting section 341 sets voxels with reference to each bounding box set by the BB setting section 312. Furthermore, the voxel setting section 341 provides information related to the set voxels together with other data to the distribution information generating section 342 (arrow 352).

[0176] The voxel setting method is similar to the case of the voxelization unit 113, and may be, for example, a method using a division number common to the three axes, a method using a division number corresponding to each axis, a method using information indicating nonlinear division (nonlinear voxel mapping information, etc.), or other methods.

[0177] Although the voxel setting section 341 may adopt any configuration, the voxel setting section 341 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to encoding by loading a program and data stored in the ROM, etc. to the RAM to execute the program.

[0178] The distribution information generation section 342 performs processing related to the generation of distribution information. For example, the distribution information generation section 342 acquires data supplied from the voxel setting section 341 (arrow 352). In addition, the subdivision information generation section 342 generates distribution information indicating the distribution state of points in the 3D data before quantization using the acquired data. The distribution information generation section 342 supplies the generated distribution information to the rendering method setting section 343 together with other data (arrow 353).

[0179] The distribution information is information indicating how points before quantization are distributed within a voxel. The distribution information is signaled (provided to the decoding side as metadata) and is used for rendering performed by the decoding side.

[0180] For example, Fig.13 As depicted in A of FIG. 1 , when a voxel 371 including information about a plurality of points is quantized, position information about those points is quantized to the center position of a voxel such as voxel 372. Therefore, it is difficult to perform rendering based only on information about voxel 372 after quantization so as to reproduce (closer to) the point group in voxel 371 before quantization.

[0181] In order to solve this problem, the distribution information generating unit 342 generates a distribution state (in Fig.13 In the example case of A, distribution information of the distribution status of points in voxel 371 is obtained, and the distribution information is notified by a signal in the bit stream.

[0182] By doing so, the decoding side can perform rendering so that the distribution of points before quantization is reproduced (more approximated) while referring to the distribution information. In other words, the decoding side can obtain a rendering result that is closer to the 3D data before quantization. Therefore, the quality reduction of the rendering result can be suppressed.

[0183] The distribution information may be configured with any information. In other words, the method of expressing the distribution state of the points before quantization may be any method. For example, the distribution state of the points may be expressed by shape, size, aspect ratio, etc.

[0184] For example, Fig.13 The outline of the distribution state of the points in voxel 373 depicted in B can be expressed as a cuboid as depicted in voxel 374. In other words, in this case, the distribution state of the points can be expressed by a cuboid indicating the shape of the distribution (general shape), the size of the cuboid, and the aspect ratio of the cuboid.

[0185] In addition, for example, Fig.13The outline of the distribution state of the points in voxel 375 depicted in C can be represented as a small ellipsoid as depicted in voxel 376. In other words, in this case, the distribution state of the points can be represented by an ellipsoid indicating the shape of the distribution and the size of the ellipsoid. Note that in the case where the size of the ellipsoid is sufficiently small in this way, the aspect ratio can be omitted.

[0186] In addition, for example, Fig.13 The outline of the distribution state of the points in voxel 377 depicted in D can be expressed as a large ellipsoid as depicted in voxel 378. In other words, in this case, the distribution state of the points can be expressed by an ellipsoid indicating the shape of the distribution, the size of the ellipsoid, and the aspect ratio of the ellipsoid.

[0187] In this way, the distribution state of points before quantization can be simply represented by information such as shape, size, and aspect ratio.

[0188] Note that although the distribution information generating unit 342 may adopt any configuration, the distribution information generating unit 342 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform encoding-related processing by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0189] The rendering method setting section 343 performs processing related to setting of the rendering method. For example, the rendering method setting section 343 acquires data provided from the distribution information generating section 342 (arrow 353). In addition, the rendering method setting section 343 sets the rendering method using the acquired data. Furthermore, the rendering method setting section 343 supplies information related to the set rendering method together with other data to the position quantization section 344 (arrow 354).

[0190] The rendering method defines how to render the quantized voxels. For example, the quantized voxels are rendered based on the distribution information about the voxels 382 (e.g., shape, size, and aspect ratio). Fig.14 The voxel 381 is obtained by the voxel 382. At this time, the voxel 382 can be rendered like the voxel 383 so that the distribution information (the state of distribution indicated by the distribution information) is represented as an object, or can be rendered like the voxel 384 so that the distribution method is represented as a point group.

[0191] The rendering method setting unit 343 performs the setting of such a rendering method. The setting method may be any method. For example, the rendering method setting unit 343 may perform the setting based on a user's instruction, a preset setting, etc. In addition, the rendering method setting unit 343 may adaptively set the rendering method according to the situation.

[0192] Note that when rendering voxels as Fig.14In the case of a point group such as voxel 384 of FIG. 1 , the number of points can be set according to the color information amount ratio (the number of color information fragments) of each voxel contained in the voxel data, or the number of points can be set to the number of points specified in the syntax. For example, color information about the point to be rendered can be sent as the number of points.

[0193] Although the rendering method setting section 343 may adopt any configuration, the rendering method setting section 343 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to encoding by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0194] The position quantization section 344 performs processing related to quantization of position information. For example, the position quantization section 344 acquires data provided from the rendering method setting section 343 (arrow 354). In addition, the position quantization section 344 uses the acquired data to quantize the position information about the 3D data, and generates voxel data. In other words, the position quantization section 344 uses the voxels set by the voxel setting section 341 to quantize the position information about the 3D data. In addition, the position quantization section 344 provides the generated voxel data together with other data to the information extraction section 314 ( Fig.11 )(arrow 355).

[0195] Note that although the position quantization unit 344 can adopt any configuration, the position quantization unit 344 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to encoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0196] <rendering information>

[0197] Next, signaling of rendering information in a bitstream will be described. As described above, the information extraction section 314 extracts rendering information, and the association section 317 includes the rendering information in a bitstream as metadata.

[0198] The rendering information may be any information as long as the information can be used when rendering is performed by the decoding side. For example, the rendering information may contain information on the shape of each voxel set by the voxel setting section 341 (voxel shape information).

[0199] The voxel shape information may have any content as long as the content is related to the shape of each voxel set by the voxel setting section 341. For example, the voxel shape information may contain information indicating the aspect ratio of the voxel (voxel aspect ratio). Fig.15 As depicted in A, voxel shape information (that is, rendering information) may include an index (Index, index) indicating an aspect ratio of each voxel.

[0200] In addition, the voxel shape information may include, for example, information indicating the size of each voxel. The size may be a size common to the three-axis directions, a size corresponding to each of the three-axis directions, or mapping information about nonlinear voxels.

[0201] Furthermore, the rendering information may contain, for example, information indicating an aspect ratio of rendering (rendering aspect ratio), that is, an aspect ratio of a rendering range at the time of rendering.

[0202] The aspect ratio of rendering is information set based on, for example, the aspect ratio of the distribution of points before quantization contained in the distribution information when the information extraction section 314 extracts information. For example, the information extraction section 314 sets the aspect ratio of the distribution of points before quantization contained in the distribution information as the aspect ratio of rendering as it is. Alternatively, the information extraction section 314 may correct the aspect ratio of the distribution of points before quantization contained in the distribution information based on, for example, other information such as the rendering method set by the rendering method setting section 343, and may set the corrected aspect ratio as the aspect ratio of rendering.

[0203] For example Fig.15 As depicted in FIG. 8B , the rendering information may include an index (Index, index) indicating an aspect ratio of the rendering.

[0204] In addition, the rendering information may include, for example, information indicating the shape of the rendering, that is, the shape of the rendering range (rendering) when rendering. The rendering information may include, for example, information indicating whether the shape of the rendering range is inscribed in the voxel or whether the shape of the rendering range is a cuboid, an ellipsoid, a point group, etc.

[0205] The shape of rendering is information set based on, for example, the shape of the distribution of points before quantization contained in the distribution information when the information extraction section 314 extracts information. For example, the information extraction section 314 sets the shape of the distribution of points before quantization contained in the distribution information as the shape of rendering as it is. Alternatively, the information extraction section 314 may change the shape of the distribution of points before quantization contained in the distribution information based on other information such as the rendering method set by the rendering method setting section 343, and set the changed shape as the shape of rendering.

[0206] For example, Fig.15 As depicted in C, the rendering information may include an index (Index, index) indicating a rendered shape.

[0207] In addition, the rendering information may include, for example, information indicating rendering density, that is, the number of points to be rendered at the time of rendering. The rendering information may include, for example, information indicating whether to render the same number of points as the number of color information fragments corresponding to voxels contained in the voxel data, whether to render a predetermined number of points, whether to render the number of points indicated by the syntax, etc.

[0208] The density of rendering (the number of dots) is, for example, information set based on the rendering method set by the rendering method setting section 343 or the like when the information extraction section 314 extracts information.

[0209] like Fig.15 As depicted in D of , for example, the rendering information may include an index (Index, index) indicating a rendering density (the number of points).

[0210] Furthermore, the rendering information may include, for example, information indicating the size of the rendering, that is, the size of the rendering range at the time of rendering.

[0211] The size of the rendering is information set based on, for example, the size of the distribution of points before quantization contained in the distribution information when the information extraction section 314 extracts information. For example, the information extraction section 314 sets the size of the distribution of points before quantization contained in the distribution information as the size of the rendering as it is. Alternatively, the information extraction section 314 may change the size of the distribution of points before quantization contained in the distribution information based on other information of the rendering method set by, for example, the rendering method setting section 343, and set the changed size as the size of the rendering. The size may be a size common to the three axis directions or a size corresponding to each of the three axis directions.

[0212] like Fig.15 As depicted in E, for example, the rendering information may include an index (Index, index) indicating a rendering size.

[0213] By signaling such rendering information (providing such rendering information to the decoding side), the decoding side can render in a manner that more correctly reproduces the distribution before quantization. Therefore, it is possible to suppress a decrease in the quality of the rendering result.

[0214] <Encoding Process Flow>

[0215] Will refer to Fig.16 The flowchart describes an example of the flow of the encoding process in this case.

[0216] When the encoding process is started, in step S301 , the preprocessing section 311 performs preprocessing on input 3D data.

[0217] In step S302 , the BB setting unit 312 sets a bounding box for the pre-processed 3D data.

[0218] In step S303 , the voxelization unit 313 voxelizes the bounding box.

[0219] In step S304 , the information extraction section 314 extracts information related to rendering (rendering information) from the data acquired in the processing up to step S303 .

[0220] In step S305 , the octree encoding unit 315 encodes the voxel data obtained in step S303 based on the data structure, and generates a signal sequence.

[0221] In step S306 , the signal sequence encoding unit 316 encodes the signal sequence.

[0222] In step S307 , the associating section 317 associates the rendering information extracted in step S304 with the encoded data obtained in step S306 as metadata, and generates a bit stream.

[0223] In step S308 , the associating section 317 outputs the obtained bit stream (a bit stream containing the encoded data on the 3D data and metadata such as control information) to the outside of the encoding device 300 .

[0224] When the process in step S308 ends, the encoding process ends. In the case where the object to be encoded is, for example, a moving image, a series of processes are performed for each frame.

[0225] <Flow of Voxelization Processing>

[0226] Next, we will refer to Fig.17 The flowchart is described in Fig.16 An example of the flow of the voxelization process performed in step S303 of FIG.

[0227] When the voxelization process is started, in step S321 , the voxel setting section 341 sets voxels as 3D data with the set bounding box as a reference.

[0228] In step S322 , the distribution information generation unit 342 generates distribution information.

[0229] In step S323 , the rendering method setting section 343 sets a rendering method.

[0230] In step S324 , the position quantization section 344 quantizes the position information on the 3D data, and generates voxel data.

[0231] When the processing in step S324 ends, the voxelization processing ends, and the processing returns to Fig.16.

[0232] By performing the above series of processing, the encoding device 300 can notify the rendering information with a signal. Therefore, the decoding side can render in a manner that reproduces the distribution before quantization more correctly. Therefore, the reduction in the quality of the rendering result can be suppressed.

[0233] <Use of Signaled Rendering Information>

[0234] The decoding side may perform rendering based on the rendering information signaled by the encoding side as described above.

[0235] In other words, the decoding side may render the voxel data based on the rendering information, which is information related to the rendering of the voxel data, which is 3D data obtained by using the voxel quantization position information and represents the three-dimensional structure. For example, the information processing device may be configured with a rendering unit that renders the voxel data based on the rendering information, which is information related to the rendering of the voxel data, which is 3D data obtained by using the voxel quantization position information and represents the three-dimensional structure.

[0236] By doing so, it is possible to perform rendering in such a manner that the distribution state of the point group before quantization is more correctly reproduced, and thus suppress a decrease in the quality of the rendering result.

[0237] <Decoding device>

[0238] Fig.18 : is a block diagram showing an example of a main configuration of a decoding device as one embodiment of an information processing device to which the present technology is applied in this case. Fig.18 The decoding device 400 shown is basically implemented as Figure 8 The decoding device 400 is a device that performs processing similar to the processing performed by the decoding device 200 of Fig.11 The decoding device 400 decodes a bit stream (a bit stream containing encoded data about 3D data, etc.) output by the encoding device 300, for example, and reconstructs the 3D data. At this time, the decoding device 400 performs such decoding (reconstruction) by applying the method of the present technology as described below.

[0239] like Fig.18 As depicted, the decoding device 400 has a control unit 401 , a signal sequence decoding unit 411 , an octree decoding unit 412 , a voxel shape determination unit 413 , a rendering unit 414 , a display control unit 415 , and a display unit 416 .

[0240] Similar to the control section 201, the control section 401 performs processing related to the control of each processing section in the decoding device 400. For example, the control section 401 controls the execution or skipping (omission) of processing performed by each processing section. For example, the control section 401 performs such control based on predetermined control information. By doing so, the control section 201 can, for example, suppress the execution of unnecessary processing and suppress the increase in load.

[0241] Although the control section 401 may adopt any configuration, the control section 401 may have, for example, a CPU, a ROM, a RAM, and the like, and the CPU may perform processing by loading a program and data stored in the ROM and the like to the RAM to execute the program.

[0242] Similar to the signal sequence decoding section 411, the signal sequence decoding section 411 performs processing related to decoding of a bit stream containing encoded data on 3D data representing a three-dimensional structure (encoded data on voxel data obtained by voxelizing the 3D data), etc., under the control of the control section 401. The 3D data may be, for example, a point cloud.

[0243] For example, the signal sequence decoding unit 411 acquires a bit stream provided from the encoding device 300 (arrow 421). In addition, for example, the signal sequence decoding unit 411 decodes the encoded data included in the acquired bit stream by a decoding method corresponding to the encoding performed by the encoding device 300 (signal sequence encoding unit 316). The signal sequence decoding unit 411 provides a signal sequence obtained by decoding the encoded data to the octree decoding unit 412 (arrow 422).

[0244] In addition, the signal sequence decoding unit 411 extracts metadata such as control information from the bit stream. The metadata includes information about the bounding box, information about the number of divisions, the above-mentioned rendering information, etc. The signal sequence decoding unit 411 provides the metadata (rendering information, etc.) to the voxel shape determination unit 413 and the rendering unit 414 (arrow 423).

[0245] Note that although the signal sequence decoding unit 411 can adopt any configuration, the signal sequence decoding unit 411 can have, for example, a CPU, ROM, RAM, etc., and the CPU loads the program and data stored in the ROM, etc. into the RAM to execute the program to perform processing related to decoding.

[0246] Similar to the octree decoding section 212, the octree decoding section 412 performs processing related to octree decoding under the control of the control section 401. For example, the octree decoding section 412 acquires the signal sequence provided from the signal sequence decoding section 411 (arrow 422). In addition, the octree decoding section 412 decodes the acquired signal sequence based on its data structure (for example, performs octree decoding on it). The octree decoding section 412 provides the voxel data obtained by decoding to the voxel shape determination section 413 (arrow 424).

[0247] Note that although the octree decoding unit 412 can adopt any configuration, the octree decoding unit 412 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to decoding by loading the program and data stored in the ROM, etc. into the RAM to execute the program.

[0248] Similar to the voxel shape determination section 213, the voxel shape determination section 413 performs processing related to determination of the shape of each voxel under the control of the control section 401. For example, the voxel shape determination section 413 acquires voxel data supplied from the octree decoding section 412. In addition, the voxel shape determination section 413 acquires metadata supplied from the signal sequence decoding section 411.

[0249] The voxel shape determination section 413 determines the shape (form, size, etc.) of each voxel based on metadata or the like.

[0250] For example, in the case where the metadata contains information about the size of the bounding box and information indicating the number of divisions common to the three axes of the x-axis, y-axis, and z-axis (three axes perpendicular to each other), the voxel shape determination unit 413 obtains (determines) the shape of each voxel by equally dividing the size of the bounding box by the common number of divisions in each direction of the x-axis, y-axis, and z-axis. In other words, the voxel shape determination unit 213 determines the shape of each voxel based on the number of divisions common to the three axes of the bounding box of the normalized 3D data.

[0251] Alternatively, for example, in the case where the metadata contains information about the size of the bounding box and information indicating the number of divisions of each of the three axes of the x-axis, y-axis, and z-axis (three axes perpendicular to each other) (the number of divisions corresponding to each axis), the voxel shape determination unit 413 obtains (determines) the shape of each voxel by equally dividing the size of the bounding box by the corresponding number of divisions in each of the x-axis, y-axis, and z-axis directions. In other words, the voxel shape determination unit 413 determines the shape of each voxel based on the number of divisions corresponding to each axis of the bounding box of the normalized 3D data.

[0252] In another alternative embodiment, for example, in the case where the metadata contains information about the size of the bounding box and map information about the voxels, the voxel shape determination unit 413 obtains (determines) the shape of each voxel from the map information. In other words, the voxel shape determination unit 413 determines the shape of each voxel based on information indicating nonlinear segmentation of the bounding box that normalizes the 3D data. Note that the voxel shape determination unit 413 may use the size information about the bounding box as needed.

[0253] The voxel shape determination section 413 supplies the determination result of the shape of each voxel obtained as described above (information indicating the shape of each voxel) to the rendering section 414 together with other data (arrow 425 ).

[0254] Note that although the voxel shape determination unit 413 can adopt any configuration, the voxel shape determination unit 413 can have, for example, a CPU, ROM, RAM, etc., and the CPU loads the program and data stored in the ROM, etc. into the RAM to execute the program to perform processing related to decoding.

[0255] Similar to the rendering section 214, the rendering section 414 performs processing related to rendering under the control of the control section 401. For example, the rendering section 414 acquires voxel data and information indicating the shape of each voxel (arrow 425) supplied from the voxel shape determination section 413. In addition, the rendering section 414 acquires metadata supplied from the signal sequence decoding section 411 (arrow 423).

[0256] The rendering unit 414 performs rendering based on such information fragments according to the rendering information and the shape of each voxel. In other words, the rendering unit 414 renders the voxel data provided by the voxel shape determination unit 413 according to the shape of each voxel determined by the voxel shape determination unit 413 and the rendering information included in the metadata provided from the signal sequence decoding unit 411.

[0257] Rendering information contains references Fig.15 The rendering section 414 sets how to render data on each voxel by directly using such pieces of information or by appropriately combining information to obtain new parameters, and performs rendering according to the settings.

[0258] For example, in Fig.15 In the case where the aspect ratio of each voxel (Vx, Vy, Vz) is signaled as in example A, the shape of each voxel (the dimensions in the three axis directions (Vx×size, Vy×size, Vz×size)) can be obtained by multiplying the voxel aspect ratio (Vx, Vy, Vz) by the common size of the three axes.

[0259] Similarly, by multiplying the voxel aspect ratio (Vx, Vy, Vz) by the size (Sx, Sy, Sz) corresponding to each axis, the shape of each voxel (Vx×Sx, Vy×Sy, Vz×Sz) can also be obtained.

[0260] In addition, for example, in Fig.15 In the case where the rendering aspect ratio (Rx, Ry, Rz) is signaled as in example B, the rendered shape (rendering range) (sizes in the three axis directions (Rx×size, Ry×size, Rz×size)) can be obtained by multiplying the rendering aspect ratio (Rx, Ry, Rz) by the common size of the three axes.

[0261] Furthermore, the product of the voxel aspect ratio (Vx, Vy, Vz), the rendering aspect ratio (Rx, Ry, Rz), and the common size of the three axes (Vx×Rx×Size, Vy×Ry×Size, Vz×Rz×Size) may be used.

[0262] As described above, the rendering unit 414 performs rendering based on the shape of each voxel and the rendering information, and thus can perform rendering by the rendering method specified by the encoding side. Fig.13 As shown in A to D of FIG. 1 , the rendering unit 414 can perform rendering so as to reproduce the distribution state of the point group before quantization more accurately (that is, to be consistent with or closer to the distribution state of the point group before quantization). Therefore, it is possible to suppress the degradation of the quality of the rendering result.

[0263] Note that, similarly to the case of the first embodiment, the rendering section 414 may perform rendering such that, at the time of such rendering, each rendering result is inscribed in a voxel.

[0264] The rendering section 414 supplies the rendering result (that is, 3D data) obtained as described above to the display control section 215 (arrow 426 ).

[0265] Note that although the rendering section 414 may adopt any configuration, the rendering section 414 may have, for example, a CPU, ROM, RAM, etc., and the CPU may perform processing related to decoding by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0266] Similar to the display control section 215, the display control section 415 performs processing related to display control under the control of the control section 401. For example, the display control section 415 acquires 3D data (arrow 426) provided from the rendering section 414 and indicating the rendering result. In addition, for example, the display control section 415 generates a 2D display image based on the 3D data. Then, the display control section 415 provides the display image to the display section 416 (arrow 427).

[0267] Note that although the display control unit 415 can adopt any configuration, the display control unit 415 can have, for example, a CPU, ROM, RAM, etc., and the CPU can perform processing related to decoding by loading a program and data stored in the ROM, etc. into the RAM to execute the program.

[0268] Similar to the display section 216, the display section 416 performs processing related to display under the control of the control section 401. For example, the display section 416 acquires a display image (arrow 427) provided from the display control section 415. In addition, the display section 416 displays the acquired display image on a display device owned by the display section 416. That is, the display section 416 displays an image generated by rendering by the rendering section 414 based on the shape of each voxel and the rendering information.

[0269] By performing the above-described processing, it is possible to perform rendering so as to more correctly reproduce the distribution state of the point group before quantization, and thus suppress a decrease in the quality of the rendering result.

[0270] <Decoding Process Flow>

[0271] Will refer to Fig.19 The flowchart describes an example of the flow of a decoding process performed by the decoding device 400 configured as described above.

[0272] When the decoding process starts, in step S401 , the signal sequence decoding unit 411 decodes an input bit stream and obtains a signal sequence.

[0273] In step S402 , the octree decoding unit 412 decodes the signal sequence based on its data structure and obtains voxel data.

[0274] In step S403 , the voxel shape determination section 413 determines the shape of each voxel based on the metadata.

[0275] In step S404 , the rendering section 414 performs rendering based on the shape of each voxel determined in step S403 and the rendering information included in the metadata.

[0276] In step S405 , the display control unit 415 generates a 2D display image using the rendering result (3D data).

[0277] In step S406 , the display unit 416 displays the display image obtained in step S405 .

[0278] When the process in step S406 ends, the decoding process ends. For example, in the case where the object to be decoded is a moving image, a series of processes are performed for each frame.

[0279] By performing the above-described decoding process, the decoding device 400 can perform rendering so as to more correctly reproduce the state of distribution of the point group before quantization, and thus suppress a decrease in the quality of the rendering result.

[0280] <4. Note>

[0281] <Control Information>

[0282] Control information related to the present technology described in the embodiments so far can be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) for controlling whether to allow (or prohibit) the application of the above-mentioned present technology can be transmitted. Alternatively, for example, control information for specifying a range (e.g., one or both of the upper and lower limits of a block size, slice, picture, sequence, component, view, and layer) for allowing (or prohibiting) the application of the above-mentioned present technology can be transmitted.

[0283] <Computer>

[0284] The above series of processing can be performed by hardware or by software. In the case of performing a series of processing by software, the program constituting the software is installed in the computer. Here, the type of computer includes a computer incorporated into special hardware, and a computer such as a general-purpose personal computer that can perform various functions by installing various programs in the computer.

[0285] Fig. 20 : is a block diagram depicting an example of the configuration of hardware of a computer that executes the series of processes described above by a program.

[0286] exist Fig. 20 In a computer 900 depicted in , a CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 are connected to one another via a bus 904 .

[0287] An input / output interface 910 is also connected to the bus 904. An input section 911, an output section 912, a storage section 913, a communication section 914, and a drive 915 are connected to the input / output interface 910.

[0288] The input unit 911 is configured by, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 912 is configured by, for example, a display, a speaker, and an output terminal. The storage unit 913 is configured by, for example, a hard disk, a RAM disk, and a nonvolatile memory. The communication unit 914 is configured by, for example, a network interface. The drive 915 drives a removable medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0289] In the computer 200 configured as described above, the CPU 901 loads a program stored in, for example, the storage unit 913 to the RAM 903 via the input / output interface 910 and the bus 904, and executes the program, thereby executing the above-described series of processes. Data and the like necessary for the CPU 901 to execute various processes are also appropriately stored in the RAM 903.

[0290] The program executed by the computer (CPU 901 ) can be applied by, for example, recording the program in the removable medium 921 as a package medium, etc. In this case, the program can be installed in the storage section 913 via the input / output interface 910 by attaching the removable medium 921 to the drive 915 .

[0291] In addition, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or a digital satellite service. In this case, the program may be received by the communication section 914 and installed in the storage section 913.

[0292] In another alternative embodiment, the program may be installed in the ROM 902 or the storage section 913 in advance.

[0293] <Objects to which the prior art is applied>

[0294] Although the case where the present technology is applied to voxelization of point cloud data has been described above, the application of the present technology is not limited to this case, and the present technology is applicable to voxelization of 3D data under any standard. In other words, various types of processing in encoding schemes, decoding schemes, etc. and various types of data such as 3D and metadata may have any specifications without contradicting the above-mentioned present technology. In addition, part of the above-mentioned processing and specifications may be omitted without contradicting the present technology.

[0295] Furthermore, the present technology is applicable to any configuration, and the encoding device 100 , the decoding device 200 , the encoding device 300 , and the decoding device 400 have been described above as examples to which the present technology is applied.

[0296] For example, the present technology is applicable to various electronic devices such as transmitters and receivers (e.g., television receivers and cellular phones) distributed over satellite broadcasting, cable broadcasting, etc., and the Internet, and distributed to terminals via cellular communications, as well as devices for recording images in media such as optical disks, magnetic disks, and flash memories and reproducing images from such storage media (e.g., hard disk recorders and video cameras).

[0297] In addition, the present technology is performed as part of the configuration of an apparatus such as a processor used as a system LSI (large scale integration) or the like (e.g., a video processor), a module using multiple processors or the like (e.g., a video module), a unit using multiple modules or the like (e.g., a video unit), or a collection obtained by further adding other functions to the unit (e.g., a video collection).

[0298] For example, the present technology is also applicable to a network system configured with a plurality of devices. For example, the present technology can be implemented as cloud computing for enabling a plurality of devices to perform processing via a network in a shared or collaborative manner. For example, the present technology can be implemented in a cloud service for providing services associated with images (moving images) to any terminal such as a computer, an AV (audio-visual) device, a mobile information processing terminal, or an IoT (Internet of Things) device.

[0299] Note that in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all components are provided in the same housing. Therefore, multiple devices housed in separate housings and connected to each other via a network and a device in which multiple modules are housed in one housing can each be referred to as a "system".

[0300] <Application fields and applications of this technology>

[0301] Systems, devices, processing units, etc. using the present technology can be used in any field, for example, in the fields of transportation, medical treatment, crime prevention, agriculture, livestock, mining, beauty, factories, consumer electronics, weather and natural monitoring. In addition, such systems, devices, processing units, etc. can be used in any application.

[0302] For example, the present technology is applicable to a system or device for providing listening and viewing content, etc. In addition, the present technology is applicable to a system or device for transportation, such as monitoring of traffic conditions and autonomous driving control, for example. In addition, the present technology is applicable to a system or device for safety, for example. In addition, the present technology is applicable to a system or device for automatic control of machines, etc., for example. In addition, the present technology is applicable to a system or device for agriculture and animal husbandry, for example. In addition, the present technology is applicable to a system or device for monitoring natural conditions such as volcanoes, forests and oceans, wild animals, etc. In addition, the present technology is applicable to a system or device for sports, for example.

[0303] <Other aspects>

[0304] Note that in this specification, a "flag" is information used to identify multiple states, and includes not only information used when identifying the two states of true (1) and false (0), but also information capable of identifying three or more states. Therefore, the value that a "flag" may take may be binary, such as 1 or 0, or may be three or more values. In other words, the number of bits that make up a "flag" may be any number, and may be one or may be two or more. In addition, not only a form in which identification information (including a flag) is included in a bit stream is assumed, but also a form in which difference information about identification information for information forming a specific basis in a bit stream is assumed; therefore, in this specification, a "flag" or "identification information" includes not only information, but also difference information for information forming the basis.

[0305] In addition, various information (such as metadata) related to the coded data (bitstream) can be sent or recorded in any form, as long as the various information is associated with the coded data. The term "association" here means, for example, allowing other data to be used (linked) when processing one data. In other words, data associated with each other can be compiled into one data or a separate data fragment. For example, information associated with the coded data (image) can be sent on a transmission line different from the transmission line used to send the coded data (image). In addition, for example, the information associated with the coded data (image) can be recorded in a recording medium different from the recording medium recorded with the coded data (image) (or in different recording areas in the same recording medium). Note that "association" may not be an association of the entire data, but may be an association of partial data. For example, an image and information corresponding to the image can be associated with each other in any unit such as multiple frames, one frame, or a portion of a frame.

[0306] Note that in this specification, terms such as "combine", "multiplex", "add", "integrate", "include / comprise", "store", "merge", "insert" and "insert" mean compiling multiple things into one, for example, compiling encoded data and metadata into one data, and mean a method for the above-mentioned "association".

[0307] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various changes may be made without departing from the spirit of the present technology.

[0308] For example, a configuration described as one device (or one processing unit) may be divided and configured as a plurality of devices (or processing units). Conversely, a configuration described above as a plurality of devices (or processing units) may be integrated and configured as one device (or one processing unit). Furthermore, needless to say, a configuration other than each device (or each processing unit) described above may be added to the configuration of each device (or each processing unit). Furthermore, a portion of the configuration of a device (or a processing unit) may be included in the configuration of other devices (or other processing units) if the configuration or operation is substantially the same as the entire system.

[0309] In addition, the above-mentioned program can be executed by any device, for example. In this case, the device can be configured with necessary functions capable of obtaining necessary information.

[0310] In addition, each step in a flowchart can be performed by one device or by multiple devices in a shared manner, for example. In addition, in the case where a step includes multiple types of processing, the multiple types of processing can be performed by one device or by multiple devices in a shared manner. In other words, multiple processing included in one step can be performed as processing of multiple steps. Conversely, processing described as multiple steps can be integrated into one step and performed together.

[0311] In addition, the program executed by the computer may be configured so that, for example, a series of processes in the steps describing the program are executed in time series in the order described in this specification, or are executed separately in parallel or at necessary timings such as the timing of calling. In other words, unless a contradiction occurs, a series of processes in the steps may be executed in an order different from the above order. In addition, the processing of the steps describing the program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

[0312] In addition, unless a contradiction occurs, multiple present technologies related to the present technology can be performed independently and individually. Needless to say, multiple prior arts can be combined to perform. For example, part or all of the present technology described in any embodiment can be combined with part or all of the present technology described in another embodiment, and the combination can be performed. In addition, part or all of any of the above-mentioned prior arts can be combined with other technologies not described above, and the combination of these technologies can be performed.

[0313] Note that the present technology can be configured as follows.

[0314] (1) An information processing device comprising:

[0315] A rendering section performs rendering on voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure, according to a shape of each voxel.

[0316] (2) The information processing device according to (1), wherein:

[0317] The rendering section performs rendering such that a rendered image is inscribed in each voxel.

[0318] (3) The information processing device according to (1) or (2), further comprising:

[0319] A shape determination unit determines the shape of each voxel, wherein

[0320] The rendering section performs rendering according to the shape of each voxel determined by the shape determination section.

[0321] (4) The information processing device according to (3), wherein:

[0322] The shape determination unit determines the shape of each voxel based on a number of divisions common to three axes of a bounding box that normalizes the 3D data.

[0323] (5) The information processing device according to (3) or (4), wherein:

[0324] The shape determination section determines the shape of each voxel based on the number of divisions corresponding to each axis of a bounding box that normalizes the 3D data.

[0325] (6) The information processing device according to (3) or (4), wherein:

[0326] The shape determination section determines the shape of each voxel based on information indicating nonlinear segmentation of a bounding box that normalizes the 3D data.

[0327] (7) The information processing device according to any one of (1) to (6), further comprising:

[0328] A decoding unit is configured to decode the encoded data regarding the voxel data, wherein:

[0329] The rendering section performs rendering on voxel data obtained by decoding the encoded data by the decoding section according to a shape of each voxel.

[0330] (8) The information processing device according to any one of (1) to (7), further comprising:

[0331] A display unit displays an image generated by rendering performed by the rendering unit.

[0332] (9) The information processing device according to any one of (1) to (8), wherein:

[0333] The 3D data includes a point cloud.

[0334] (10) An information processing method comprising:

[0335] Rendering is performed on voxel data according to the shape of each voxel, the voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure.

[0336] (11) An information processing device comprising:

[0337] A bitstream generation unit generates a bitstream including coded data on voxel data including 3D data obtained by using voxel quantization position information and representing a three-dimensional structure, and rendering information including information related to rendering of the voxel data.

[0338] (12) The information processing device according to (11), wherein:

[0339] The rendering information includes voxel shape information including information related to the shape of each voxel.

[0340] (13) The information processing device according to (12), wherein:

[0341] The voxel shape information includes information indicating an aspect ratio of each voxel.

[0342] (14) The information processing device according to (12) or (13), wherein:

[0343] The voxel shape information contains information indicating the size of each voxel.

[0344] (15) The information processing device according to any one of (11) to (14), wherein:

[0345] The rendering information includes information indicating an aspect ratio of a rendering range.

[0346] (16) The information processing device according to any one of (11) to (15), wherein:

[0347] The rendering information includes information indicating the shape of a rendering range.

[0348] (17) The information processing device according to any one of (11) to (16), wherein:

[0349] The rendering information includes information indicating the number of points to be rendered.

[0350] (18) The information processing device according to any one of (11) to (17), further comprising:

[0351] A rendering information generating unit generates the rendering information, wherein:

[0352] The bitstream generation unit is configured to generate a bitstream including the rendering information generated by the rendering information generation unit.

[0353] (19) The information processing device according to (18), further comprising:

[0354] A rendering method setting unit is used to set a rendering method, wherein:

[0355] The rendering information generation section is configured to generate rendering information corresponding to the rendering method set by the rendering method setting section.

[0356] (20) The information processing device according to (18) or (19), further comprising:

[0357] a voxel setting unit for setting voxels for quantifying position information about the 3D data, wherein:

[0358] The rendering information generation section is configured to generate rendering information corresponding to the shape of each voxel set by the voxel setting section.

[0359] (21) The information processing device according to (20), further comprising:

[0360] a position quantization section that generates the voxel data by quantizing the position information on the 3D data using the voxels set by the voxel setting section, wherein

[0361] The bitstream generation section is configured to generate a bitstream including encoded data on the voxel data generated by the position quantization section.

[0362] (22) The information processing device according to (20) or (21), further comprising:

[0363] A bounding box setting unit is configured to set a bounding box for normalizing position information about the 3D data, wherein:

[0364] A voxel setting section is configured to set voxels for quantizing the position information with reference to the bounding box set by the bounding box setting section.

[0365] (23) The information processing device according to any one of (11) to (22), further comprising:

[0366] an encoding unit, which encodes the voxel data to generate encoded data, wherein

[0367] The bitstream generation section is configured to generate a bitstream including the encoded data generated by the encoding section.

[0368] (24) The information processing device according to any one of (11) to (23), wherein:

[0369] The 3D data includes a point cloud.

[0370] (25) An information processing method comprising:

[0371] A bitstream including encoded data on voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure and rendering information including information related to rendering of the voxel data is generated.

[0372] (31) An information processing device comprising:

[0373] A rendering section performs rendering on voxel data based on rendering information including information related to rendering of voxel data, the voxel data including 3D data obtained by using voxel quantized position information and representing a three-dimensional structure.

[0374] (32) The information processing device according to (31), wherein:

[0375] The rendering information includes voxel shape information, the voxel shape information including information related to the shape of each voxel, and

[0376] The rendering section is configured to perform rendering based on the voxel shape information.

[0377] (33) The information processing device according to (32), wherein:

[0378] The voxel shape information includes information indicating an aspect ratio of each voxel, and

[0379] The rendering section performs rendering based on the aspect ratio of each voxel indicated by the information.

[0380] (34) The information processing device according to (32) or (33), wherein:

[0381] The voxel shape information includes information indicating the size of each voxel, and

[0382] The rendering section performs rendering based on the size of each voxel indicated by the information.

[0383] (35) The information processing device according to any one of (31) to (34), wherein:

[0384] The rendering information includes information indicating an aspect ratio of the rendering range, and

[0385] The rendering section performs rendering according to the aspect ratio of the rendering range indicated by the information.

[0386] (36) The information processing device according to any one of (31) to (35), wherein:

[0387] The rendering information includes information indicating a shape of a rendering range, and

[0388] The rendering section performs rendering according to a shape of a rendering range indicated by the information.

[0389] (37) The information processing device according to any one of (31) to (36), wherein:

[0390] The rendering information includes information indicating the number of points to be rendered.

[0391] The rendering section performs rendering by as many points as indicated by the information.

[0392] (38) The information processing device according to any one of (31) to (37), further comprising:

[0393] A decoding unit is configured to decode the encoded data regarding the voxel data, wherein:

[0394] The rendering section performs rendering on voxel data obtained by decoding the encoded data by the decoding section.

[0395] (39) The information processing device according to (38), wherein:

[0396] The decoding unit decodes the encoded data included in the bit stream, and

[0397] The rendering section performs rendering on voxel data obtained by decoding the encoded data by the decoding section based on rendering information included in the bit stream.

[0398] (40) The information processing device according to any one of (31) to (39), further comprising:

[0399] A shape determination unit is configured to determine the shape of each voxel based on the rendering information, wherein:

[0400] The rendering section performs rendering according to the rendering information and the shape of each voxel determined by the shape determination section.

[0401] (41) The information processing device according to any one of (31) to (40), further comprising:

[0402] A display section displays an image generated by the rendering performed by the rendering section.

[0403] (42) The information processing device according to any one of (31) to (41), wherein:

[0404] The 3D data includes a point cloud.

[0405] (43) An information processing method comprising:

[0406] Rendering is performed on the voxel data based on the rendering information including information related to the rendering of the voxel data including 3D data obtained by using the voxel quantized position information and representing a three-dimensional structure.

[0407] [reference numerals list]

[0408] 100 encoding device, 101 control unit, 111 preprocessing unit, 112 BB setting unit, 113 voxelization unit, 114 octree encoding unit, 115 signal sequence encoding unit, 200 decoding device, 201 control unit, 211 signal sequence decoding unit, 212 octree decoding unit, 213 voxel shape determination unit, 214 rendering unit, 215 display control unit, 216 display unit, 300 encoding device, 311 preprocessing unit, 312 BB setting unit B setting unit, 313 voxelization unit, 314 information extraction unit, 315 octree encoding unit, 316 encoding signal sequence, 317 association unit, 341 voxel setting unit, 342 distribution information generation unit, 343 rendering method setting unit, 344 position quantization unit, 400 decoding device, 411 signal sequence decoding unit, 412 octree decoding unit, 413 voxel shape determination unit, 414 rendering unit, 415 display control unit, 416 display unit

Claims

1. A decoding device, comprising: A circuit system, the circuit system being configured to: Get a bitstream of the point cloud normalized using a bounding box; as well as Decoding the bit stream to obtain quantized information of position information of points of the point cloud and metadata representing the form of the points, wherein the form of the points is represented as a regular cube; as well as The points are rendered to have a regular cube shape.