Information Processing Apparatus and Information Processing Method

Abstract

The present invention relates to an information processing apparatus and an information processing method capable of achieving higher image quality while suppressing an increase in the processing amount of a client. By performing projection in a plurality of projection directions, 3D data is converted into two dimensions to generate image data in a plurality of projection directions, and projection direction information indicating the projection direction of the image data is generated as projection metadata. Further, the projection direction information includes additional projection direction identification information indicating that the image data is generated by performing projection in an additional projection direction. For example, the present invention can be applied to a data generation apparatus that generates data for distributing point clouds.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus and an information processing method, and more particularly, to an information processing apparatus and an information processing method that can enhance image quality while reducing an increase in client throughput. Background Art

[0002] Conventionally, as disclosed in Non-Patent Document 1, a method for compressing a point cloud, which is a set of points having position information and attribute information (particularly color information) in a three-dimensional space, has been defined.

[0003] In addition, as one of the point cloud compression methods, Non-Patent Document 2 discloses a method that divides point cloud data into a plurality of regions (hereinafter referred to as segmentation), generates a texture image, a geometry image, an occupancy map image, and auxiliary information by projecting each region onto a plane, and encodes these three images using a video codec. Here, the geometry image includes depth information of a set of points included in the point cloud. Similarly, this method is called video-based point cloud coding (V-PCC).

[0004] Incidentally, six projection planes (four planes in the horizontal direction and two planes in the vertical direction) have been used to generate a texture image, a geometry image, and an occupancy map image by projecting each region onto a plane after segmentation of the point cloud data. As described above, in the case of using six projection planes, many points disappear during the conversion from 3D data to 2D data, resulting in holes being generated on the surface of the reconstructed point cloud object in some cases.

[0005] In view of the above situation, a technique disclosed in Non-Patent Document 3 has been developed, which increases the number of projection planes to 10 (eight planes in the horizontal direction and two planes in the vertical direction) by adding four new planes, and this technique is also discussed in MPEG (Moving Picture Experts Group).

[0006] In addition, Non-Patent Document 4 discloses a technique that allows partial access to each block into which an object box is divided.

[0007] [Citation List]

[0008] [Non-Patent Document]

[0009] [Non-Patent Document 1]

[0010] MPEG-I Part5 Point Cloud Compression(ISO / IEC 23090-5)

[0011] [Non - Patent Document 2]

[0012] w17534, Working draft of Point Cloud Coding for Category 2(Draft 1), April 2018, San Diego, USA

[0013] [Non - Patent Document 3]

[0014] w17871, PCC Core Experiment on Patch packing, July 2018, Ljubljana, Slovenia

[0015] [Non - Patent Document 4]

[0016] ISO / IEC JTC 1 / SC 29 / WG 11 Coding of moving pictures and audio Convenorship: UNI(Italy) “Technologies Under Consideration for Carriage of Point Cloud Data” Summary of the Invention

[0017] [Technical Problem]

[0018] Incidentally, it is assumed that adding a new projection plane as described above to enhance the image quality increases the throughput of the client. Moreover, a conventional client cannot pre - identify whether the transport stream is generated from the normal six projection planes or from the images generated from four new additional projection planes. For this reason, the client is required to acquire the entire stream, which may cause reproduction interruption or image quality degradation and increased processing overhead on the client side in an environment with limited network bandwidth.

[0019] In view of such a situation, the present disclosure is designed, and the object of the present disclosure is to reduce the increase in the throughput of the client while enhancing the image quality.

[0020] [Solution to the Problem]

[0021] An information processing apparatus according to an aspect of the present disclosure includes a pre - processing unit adapted to generate image data in a plurality of projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two - dimensional data, and generate projection direction information indicating the projection direction of the image data as projection metadata.

[0022] The information processing method according to one aspect of the present disclosure includes: generating image data in a plurality of projection directions by projecting 3D data in the plurality of projection directions and converting the data into two-dimensional data, and generating projection direction information indicating the projection direction of the image data as projection metadata.

[0023] In one aspect of the present disclosure, image data in a plurality of projection directions is generated by projecting 3D data in the plurality of projection directions and converting the 3D data into two-dimensional data, and projection direction information indicating the projection direction of each image data is generated as projection metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1

[0025] Figure 1 is a diagram illustrating a point cloud compression method.

[0026] Figure 2

[0027] Figure 2 is a diagram illustrating an image generated in the case of using six projection planes.

[0028] Figure 3

[0029] Figure 3 is a diagram showing an example of establishing eight projection planes in the horizontal direction.

[0030] Figure 4

[0031] Figure 4 is a diagram illustrating the improvement of image quality achieved by adding projection planes.

[0032] Figure 5

[0033] Figure 5 is a diagram illustrating an image generated in the case of using ten projection planes.

[0034] Figure 6

[0035] Figure 6 is a diagram illustrating the generation of a file in the case of using ten projection planes.

[0036] Figure 7

[0037] Figure 7 depicts a diagram illustrating an example of obtaining a file according to a viewing direction.

[0038] Figure 8

[0039] ​​​​​​​​​​​​​​​​Figure 8 It is a diagram showing MPD samples in the first signaling method of the first extension method.

[0040] Figure 9

[0041] Figure 9 It is a diagram describing an example of obtaining a stream.

[0042] Figure 10

[0043] Figure 10 It is a diagram showing MPD samples in the second signaling method of the first extension method.

[0044] Figure 11

[0045] Figure 11 It is a diagram showing MPD samples signaling additional projection direction identification information.

[0046] Figure 12

[0047] Figure 12 It depicts a diagram describing object boxes and blocks.

[0048] Figure 13

[0049] Figure 13 It is a diagram showing MPD samples signaling additional projection direction identification information set for each block.

[0050] Figure 14

[0051] Figure 14 It is a diagram describing each attribute of the gpcc:blockInfo element.

[0052] Figure 15

[0053] Figure 15 It is a block diagram showing a configuration example of a data generation device.

[0054] Figure 16

[0055] Figure 16 It is a block diagram showing a configuration example of a data reproduction device.

[0056] Figure 17

[0057] Figure 17 It is a flowchart describing file generation processing.

[0058] Figure 18 ​​​​​​​​​​​​​​​​​​​​

[0059] Figure 18 It is a flowchart describing point cloud reproduction processing.

[0060] Figure 19

[0061] Figure 19 It is a diagram showing an MPD sample to which image quality metadata has been added.

[0062] Figure 20

[0063] Figure 20 It is a diagram describing a modification example of using an extractor track.

[0064] Figure 21

[0065] Figure 21 It is a diagram showing an MPD sample using an extractor track.

[0066] Figure 22

[0067] Figure 22 It is a diagram showing an example of a newly defined VPCCGroupBox.

[0068] Figure 23

[0069] Figure 23 It is a diagram describing the definitions of projection_direction, present_alone, and point_cloud_resolution.

[0070] Figure 24

[0071] Figure 24 It is a diagram showing a sample of a VPCCGroupBox that signals additional projection direction identification information.

[0072] Figure 25

[0073] Figure 25 It is a diagram describing the definitions of 45degree_projection, projection_direction, present_alone, and point_cloud_resolution.

[0074] Figure 26

[0075] Figure 26 ​​​​​​​​​​​​​​​​It is a diagram showing a sample of a VPCCGroupBox that signals the additional projection direction identification information set for each block.

[0076] Figure 27

[0077] Figure 27 It is a diagram describing the extension of auxiliary information.

[0078] Figure 28

[0079] Figure 28 It is a diagram describing the definitions of projection_direction, present_alone, and point_cloud_resolution.

[0080] Figure 29

[0081] Figure 29 It is a diagram showing a part of the auxiliary information to which 45degree_projection has been added.

[0082] Figure 30

[0083] Figure 30 It is a diagram describing the definitions of 45degree_projection, projection_direction, present_alone, and point_cloud_resolution.

[0084] Figure 31

[0085] Figure 31 It is a block diagram showing a configuration example of a computer to which the present technology is applied. Detailed Embodiments

[0086] Hereinafter, the detailed embodiments to which the present technology is applied will be described in detail with reference to the accompanying drawings.

[0087] <Conventional Signaling Notification>

[0088] Before describing the signaling notification to which the present technology is applied, conventional signaling notification will be described with reference to Figures 1 to 4 Describe conventional signaling notification.

[0089] Figure 1 It is a diagram briefly describing the V-PCC disclosed in Non-Patent Document 2 above.

[0090] As Figure 1 shown, first, point cloud data (3D data) representing a three-dimensional structure is input, and then the point cloud content is divided into multiple regions. In​​​​​​​​​​Figure 1 In the example shown, the input represents point cloud data of a three-dimensional structure in which a hemispherical shape and a conical shape are combined, and then the point cloud content is segmented into three regions, where the hemispherical shape is segmented into one region and the conical shape is segmented into two regions.

[0091] Next, a projection onto a plane is performed for each region, thereby generating a texture image, a geometry image, and an occupancy map image. The texture image includes color information representing the appearance of the surface of each region. The geometry image includes position information representing the depth to the surface of each region. The occupancy map image includes tile position information within the component picture. Then, the texture image, the geometry image, and the occupancy map image are encoded using a video codec such as AVC (Advanced Video Coding) or HEVC (High Efficiency Video Coding).

[0092] Conventionally, as Figure 2 shown, a texture image with tiles can be generated, where the tiles are projected onto six projection planes (X+ projection plane, X− projection plane, Y+ projection plane, Y− projection plane, Z+ projection plane, and Z− projection plane) and arranged together for each projection direction. Similarly, a geometry image and an occupancy map image with tiles can be generated in a similar manner, where the tiles are projected onto six projection planes and arranged together for each projection direction.

[0093] In contrast, as disclosed in Non-Patent Document 3 above, a technique for generating a texture image, a geometry image, and an occupancy map image with 10 projection planes has been developed. That is, as Figure 3 shown, although four projection planes (solid lines) spaced 90 degrees apart from each other have been used, another four projection planes (dashed lines) spaced 45 degrees from these projection planes are added, that is, a total of 10 projection planes including two vertical planes are set.

[0094] For example, in the case of six projection planes, holes may be created on the surface of the reconstructed point cloud object, as shown in the upper part of Figure 4 . In contrast, by increasing the number of projection planes to 10, the holes created in the case of six projection planes shown in the lower part of Figure 4 can be filled, and the point cloud object can be reconstructed in such a way as to prevent the creation of such holes on the surface, thereby providing improved subjective image quality.

[0095] Then, as Figure 5As shown, a texture image with tiles can be generated, and the tiles are projected onto ten projection planes (X+ projection plane, X− projection plane, Y+ projection plane, Y− projection plane, Z+ projection plane, Z− projection plane, X+Y+ projection plane, X−Y− projection plane, X−Y+ projection plane, and X+Y− projection plane) and arranged together for each projection direction. Moreover, a geometric image and an occupancy map image with tiles can be similarly generated, and these tiles are projected onto ten projection planes and arranged together for each projection direction.

[0096] Here, when projecting a point cloud onto a projection plane, the points in the point cloud that are projected onto four conventional projection planes (X+ projection plane, X− projection plane, Y+ projection plane, and Y− projection plane) in the horizontal direction are also projected onto four additional projection planes (X+Y+ projection plane, X−Y− projection plane, X−Y+ projection plane, and X+Y− projection plane) in the horizontal direction in an overlapping manner.

[0097] Then, regarding the projection onto the four additional projection planes, it is determined whether to encode all the overlapping portions (including all the overlapping portions in the tiles of the four additional projection planes) between the four additional projection planes in the horizontal direction and the four existing projection planes as overlapping information, or only encode the difference in the horizontal direction from the four existing projection planes after removing the overlapping portions from the tiles of the four additional projection planes as overlapping information, depending on the implementation of the encoder. For example, in the case where the overlapping information is encoded to include all the above-mentioned overlapping portions in the tiles of the four additional projection planes, all the tiles projected onto the X+Y+ projection plane have all the projection direction information of the X+Y+ projection plane. At the same time, in the case where the overlapping information is encoded by removing the overlapping portions from the tiles of the four additional projection planes, some of the tiles projected onto the X+Y+ projection plane only include the differential information lost in the tiles of the X+ projection plane and the Y+ projection plane.

[0098] Therefore, in the case where the overlapping information including all the above-mentioned overlapping portions in the tiles of the four additional projection planes is encoded, the segmentation streams of the individual projection planes can be displayed separately. At the same time, in the case where the overlapping information is encoded by removing the overlapping portions from the tiles of the four additional projection planes, it is impossible to display the segmentation streams of the individual projection planes separately.

[0099] That is, although the subjective image quality can be enhanced by increasing the number of projection planes to ten, the increased number of projection planes results in an increase in the number of tiles in the image, thereby increasing the amount of information in the texture image, geometric image, and occupancy map image.

[0100] Moreover, conventionally, it is premised that the entire streams of each of the texture image, the geometry image, and the occupancy map image are acquired. Therefore, in response to an increase in the number of projection planes to 10, when transmitting the V-PCC stream, the bitrate inevitably increases compared to the stream of six projection planes.

[0101] As a result, for example, reproduction interruptions may occur in an environment with limited network bandwidth. If an attempt is made to reduce the bitrate and perform reproduction without interruption in the case of limited network bandwidth to solve this problem, although the number of projection planes is increased to enhance the image quality, the image quality may deteriorate due to encoding at a reduced bitrate.

[0102] Therefore, an effective method for reducing the transmitted bitrate is to transmit the V-PCC streams (texture stream, geometry stream, and occupancy map stream) in a segmented manner for each projection plane, and select and acquire the segmented V-PCC streams of the necessary projection planes according to the user's viewing direction.

[0103] However, even in this case, it is impossible to identify whether the tiles of the four additional projection planes (X+Y+ projection plane, X-Y- projection plane, X-Y+ projection plane, and X+Y- projection plane) have all the information of each projection plane. This makes it impossible to determine whether the image can be displayed separately, and the stream is continuously acquired on the premise that not all the information in the projection direction can be obtained.

[0104] For example, in the case of the segmented stream where the user's viewing direction corresponds to the projection direction X+Y+, and even if the segmented stream has all the information in the projection direction and can configure the point cloud content with only the projection direction of that stream, the segmented streams in the projection directions X+ and Y+ which are adjacent directions to the projection direction X+Y+ are acquired. As described above, acquiring the segmented streams that are originally unnecessary for display may cause reproduction interruptions or a reduction in image quality, and increase the processing overhead of the client.

[0105] Therefore, it is necessary to transmit the V-PCC stream with enhanced image quality by adding projection planes, while effectively using the available network bandwidth by segmenting the V-PCC and transmitting only the streams required for display, thereby reducing the increase in client throughput. This requires a device that provides information about the additional planes (i.e., the projection direction information of the blocks in the segmented V-PCC stream and the yes / no separate display information indicating whether the segmented stream can be displayed separately) to the client.

[0106] The present disclosure proposes a device that generates a file including information about the additional plane as described above and presents the file to a client. As a specific implementation thereof, a description of an extension method (DASH MPD extension, ISOBMFF extension, or basic code stream extension) will be given below. In addition, the present disclosure proposes a device that provides effective pixel information of a V-PCC stream, which is information effective for enhancing image quality, as image quality metadata to a client to enhance the image quality of the V-PCC stream in the viewing direction.

[0107] Here, before describing the extension method, the following will refer to Figure 6 and Figure 7 The beneficial effects of providing projection direction information and whether or not to display the image separately to a client are described.

[0108] like Figure 6 As shown in the middle, for example, a texture image, a geometric image, and an occupancy map image are generated by putting together tiles projected onto 10 projection planes (X+projection plane, X-projection plane, Y+projection plane, Y-projection plane, Z+projection plane, Z-projection plane, X+Y+projection plane, XY-projection plane, X-Y+projection plane, and X+Y-projection plane) in the same arrangement for each projection direction, and auxiliary information including projection metadata is generated, which includes projection direction information and whether to display separately / not.

[0109] Afterwards, if Figure 6 As shown on the right, each generated image is divided and encoded for each projection direction, thereby generating a texture stream, a geometry stream, and an occupancy map stream for each projection direction. Then, a projection direction file (ISOBMFF: ISO Base Media File Format) storing these streams and auxiliary information is generated.

[0110] For example, the projection direction file may include a single track storing a single stream generated by putting together not only the texture stream, geometry stream, and occupancy map stream of each projection direction but also the auxiliary information. In addition, the projection direction file may include four tracks each storing the texture stream, geometry stream, and occupancy map stream of each projection direction and the auxiliary information. As described above, 10 projection direction files corresponding to the respective projection directions are generated by generating a projection direction file for each projection direction.

[0111] As described above, the projection direction file includes auxiliary information, and projection direction information indicating the projection direction of the file and yes / no separate display information indicating whether display can be achieved using only the file are added to the auxiliary information. Therefore, the client can select and obtain the necessary files for reproduction according to the viewing direction based on the above information, create a point cloud, and display the point cloud.

[0112] Meanwhile, Figure 7 the arrow “View” shown in A of Figure 7 indicates the viewing direction, and a description will be given of an example of obtaining a file when the user's viewing direction is near the projection direction X+Y+.

[0113] For example, in the case where the yes / no individual display information indicates that the stream can be individually displayed, the client obtains the projection direction files for six projection directions, namely, Figure 7 the four projection directions shown in B of Figure 7 plus the two projection directions in the Z direction (projection direction X+Y+, projection direction X-Y-, projection direction X-Y+, projection direction X+Y-, projection direction Z+ and projection direction Z-). In contrast, in the case where the yes / no individual display information indicates that the stream cannot be individually displayed, the client obtains the projection direction files for seven projection directions, namely, Figure 7 the five projection directions shown in C of Figure 7 plus the two projection directions in the Z direction (projection direction X+Y+, projection direction Y+, projection direction X-, projection direction Y-, projection direction X+, projection direction Z+ and projection direction Z-).

[0114] As described above, in the absence of projection metadata, the number of files obtained by the client is less than 10 files, and regardless of whether the yes / no individual display information indicates that the stream can be individually displayed, the subjective image quality in the viewing direction is appreciable. That is, it is possible to transmit a V-PCC stream with enhanced image quality by adding a projection plane, and at the same time, reduce the processing overhead of the client by effectively utilizing the network bandwidth.

[0115] <First Extension Method>

[0116] A description will be given of a first extension method for notifying projection direction information, yes / no individual display information, and image quality metadata for each file reference by DASH MPD (Dynamic Adaptive Streaming over HTTP Media Presentation Description). Figures 8 to 21

[0117] For example, the projection metadata includes projection direction information and yes / no individual display information. The yes / no individual display information indicates whether the texture stream, geometry stream, and occupancy map stream included in the file can be individually displayed, and there are two methods for signaling this information, namely, the first signaling method and the second signaling method described below.

[0118] <First Signaling Method in the First Extension Method>

[0119] The first signaling method signals with a flag that indicates whether the texture stream, geometry stream, and occupancy map stream included in the file can be individually displayed.

[0120] AsFigure 8 As shown in the MPD sample, the projection direction descriptor (SupplementalProperty schemeIdUri = "urn:mpeg:mpegI:pc_proj_direction:2018") is newly defined and used to signal the projection direction information of the segmented V-PCC stream. Here, the @value signals the projection direction information of the tiles within the stream.

[0121] For example, the projection direction with respect to the local coordinate system of the point cloud is signaled by @value (0 to 9). That is, in the case of @value = 0, the projection direction X+ is signaled, in the case of @value = 1, the projection direction Y+ is signaled, in the case of @value = 2, the projection direction X- is signaled, in the case of @value = 3, the projection direction Y- is signaled, in the case of @value = 4, the projection direction Z+ is signaled, in the case of @value = 5, the projection direction Z- is signaled, in the case of @value = 6, the projection direction X+Y+ is signaled, in the case of @value = 7, the projection direction X-Y+ is signaled, in the case of @value = 8, the projection direction X-Y- is signaled, and in the case of @value = 9, the projection direction X+Y- is signaled.

[0122] It should be noted that a single segmented V-PCC stream can include tiles with multiple projection directions. In this case, the multiple projection directions are signaled, for example, by commas separated in @value.

[0123] As Figure 8 shown in the MPD sample, the present alone descriptor (SupplementalProperty schemeIdUri = "urn:mpeg:mpegI:pc_present_alone:2018") is newly defined and used to signal the is / are not present alone display information of the segmented V-PCC stream. Here, 0 or 1 is specified as @value, and in the case of @value = 0, this indicates that the segmented stream cannot be displayed alone, and in the case of @value = 1, this indicates that the segmented stream can be displayed alone.

[0124] For example, Figure 8 the example shown depicts that the segmented stream including tiles projected in the projection directions X+ and Y+ can be displayed alone, and the segmented stream including tiles projected in the projection direction X+Y+ cannot be displayed alone.

[0125] Note that a single segmented V-PCC stream may include a tile that can be displayed separately and another tile that cannot be displayed separately. In this case, the @value of the separate presentation descriptor is 0.

[0126] In addition, a set of segmented V-PCC streams included in a single point cloud is newly defined and signaled with a point cloud group descriptor (SupplementalProperty schemeIdUri = "urn:mpeg:mpegI:pc_group:2018"). Here, the @value signals the group identifier, and the segmented V-PCC streams with the same value are included in a single point cloud.

[0127] For example, when the user's viewing direction is near the projection direction X+Y+ and only the point cloud viewed from this direction is reconstructed, the client selects AdaptationSet@id = vpcc45 by referring to the projection direction descriptor. Then, the client can identify the streams that can be displayed separately from the separate presentation descriptor, so it selects AdaptationSet@id = vpcc0 and AdaptationSet@id = vpcc90 as adjacent directions. As described above, the client can display the point cloud covering the viewing direction by obtaining three AdaptationSets.

[0128] In addition, for example, when the user's viewing direction is near the projection direction X+, the client can identify that he or she only needs to select and obtain AdaptationSet@id = vpcc0, thus eliminating the need to obtain other redundant segmented streams to cover the viewing direction.

[0129] Note that the segmented V-PCC streams included in the point cloud on the back surface that is not visible from the viewing direction can be further obtained. At this time, holes on the back surface of the point cloud are allowed. Therefore, the transmission bitrate can be reduced by obtaining only the minimum required number of segmented streams. For example, when the stream in the projection direction X+Y+ cannot be displayed separately when the user's line of sight direction is near the projection direction X+Y+, it is only necessary to obtain the streams in 5 projection directions (X+Y+ projection plane, X+ projection plane, Y+ projection plane, X projection plane, and Y- projection plane) in the horizontal direction, as Figure 9 shown.

[0130] As described above, the projection direction descriptor and the separate presentation descriptor allow for efficient transmission according to the viewing direction of the V-PCC stream.

[0131] It should be noted that the newly defined descriptor can be signaled by using EssentialProperty.

[0132] At this time, DASH clients that do not support the schemeIdUri of EssentialProperty must ignore the Preselection and AdaptationSet (and Representations in some cases) written in this property. Additionally, DASH clients that do not support the schemeIdUri of SupplementalProperty may ignore the value of this Property and use its Preselection and AdaptationSet (or, in some cases, Representations, etc.).

[0133] <The second signaling notification method in the first extension method>

[0134] In the case where the texture stream, geometry stream, and occupancy map stream included in the file cannot be displayed separately, the second signaling notification method signals the association with the file required to implement the display.

[0135] First, the association between the Representation of the split stream that cannot be displayed separately and the Representation of the split stream with the information required to implement the display is through

[0136] Representation@dependencyId. That is, Representation@dependencyId is used as the yes / no separate display information for the split V-PCC stream. For example, in the presence of Representation@dependencyId, this indicates that the split stream cannot be displayed separately, while in the absence of Representation@dependencyId, this indicates that the split stream can be displayed separately.

[0137] Figure 10 It is a diagram showing an MPD sample in the second signaling notification method. The second signaling notification method provides beneficial effects similar to those of the first signaling notification method described above.

[0138] Additionally, groups that can be displayed separately can be signaled by defining a new descriptor. For example, the projection direction X+Y+ that cannot be displayed separately can be grouped with the projection direction X+ and projection direction Y+ required to implement the display to indicate that the streams can be displayed separately group by group.

[0139] <Further extension of the first extension method>

[0140] A description of the further extension of the first extension method will be given here.

[0141] As described above, the first extension method allows the conventional DASH MPD to be extended in a manner that the projection direction information can be obtained. Further, the DASH MPD can be further extended such that four additional projection planes (X+Y+ projection plane, X-Y- projection plane, X-Y+ projection plane, and X+Y- projection plane) have additional projection direction identification information indicating that the image is generated from tiles in four additional projection directions.

[0142] For example, as Figure 11 shown in the MPD sample of

[0143] , a 45-degree projection descriptor (SupplementalPropertyschemeIdUri = "urn:mpeg:mpegI:pc_45degree_projection:2018") is newly defined and used to signal the additional projection direction identification information of the projection direction file. Here, @value = 0 in the 45-degree projection descriptor indicates that the stream is generated from tiles in six projection directions, and @value = 1 indicates that the stream is generated from tiles in four additional projection directions.

[0144] It should be noted that, as Figure 11 shown in the MPD sample of Figure 8 , signaling the 45-degree projection descriptor and the projection direction descriptor in each AdaptationSet results in an MPD equivalent to the MPD sample shown in

[0145] That is, when @value = 1 of the 45-degree projection descriptor indicates that the image has been generated from tiles in four additional projection directions, it indicates that the projection direction is the direction obtained by rotating 45 degrees counterclockwise around the Z axis in the direction indicated by @value of the projection direction descriptor. For example, as shown in the AdaptationSet with @id = "vpcc45", @value = 0 of the projection direction descriptor signaled together with the 45-degree projection descriptor with @value = 1 indicates that the projection direction is X+Y+.

[0146] Alternatively, in the case where the 45-degree projection descriptor is not signaled, this may indicate that a stream has been generated from tiles in six projection directions. Meanwhile, in the case where the 45-degree projection descriptor is signaled, this may indicate that a stream has been generated from tiles in four projection directions.

[0147] In addition, it is possible to signal only the 45-degree projection descriptor without signaling the projection direction descriptor. In this case, if @value = 0 in the 45-degree projection descriptor, this indicates that a stream has been generated from tiles in all six projection directions, while if @value = 1, this indicates that a stream has been generated from all four additional projection directions.

[0148] Here, as disclosed in Non-Patent Document 4 described above, in the case where each block into which the object box is divided can be partially accessed, a projection direction file can be generated based on each block. For example, the object box has a rectangular parallelepiped shape including the entire point cloud object and can be divided into blocks having a rectangular parallelepiped shape smaller than the object box.

[0149] Figure 12 An example is shown in which the object box is divided into four blocks. For example, the object box has X, Y, and Z axes arranged as shown in A of Figure 12 and can be divided into four blocks such that these blocks are arranged in pairs along the X-axis direction and the Y-axis direction as viewed from the Z-axis direction shown in B of Figure 12 Then, as shown in the lower right of B of Figure 12 four projection planes (dashed lines) can be added, and the projection directions of these four projection planes are offset by 45 degrees with respect to one of the four projection planes (solid lines) of the block.

[0150] Alternatively, in the case where a projection direction file is generated for each block, the 45-degree projection descriptor signaled for each block can be used together with the block position information for identifying the three-dimensional space information of each block.

[0151] That is, as shown in Figure 13As shown in the MPD sample, the projection direction descriptor, 45-degree projection descriptor, and block information descriptor are signaled in each AdaptationSet. Here, the block information descriptor (SupplementalPropertyschemeIdUri = "urn:mpeg:mepgI:gpcc:block_information:2018") signals the three-dimensional spatial information of the block. For example, the three-dimensional spatial information of the block can be indicated by block_offset_x, block_offset_y, block_offset_z, block_size_x, block_size_y, and block_size_z, which are attributes of the gpcc:blockInfo element. In addition, blocks with the same object_id attribute indicate that these blocks are included in the same object box.

[0152] Here, each attribute of the gpcc:blockInfo element is signaled by a relative value obtained by assuming that each side along the X, Y, and Z axes of the object box is 1, as Figure 14 shown.

[0153] It should be noted that the 45-degree projection descriptor and block information descriptor can be signaled only without signaling the projection direction descriptor. In this case, if @value = 0 in the 45-degree projection descriptor, it indicates that the projection direction file of each block is a stream generated from tiles in all six projection directions, and if @value = 1, it indicates that the projection direction file of each block is a stream generated from tiles in all four additional projection directions.

[0154] As described above, the first extension method allows the MPD to be extended in a way that additional projection direction identification information is signaled in addition to the projection direction information.

[0155] <Configuration Example of Information Processing Device>

[0156] Figure 15 is a block diagram showing a configuration example of a data generation device, which is an information processing device that generates a PC stream from point cloud data on the server side that provides content and performs file generation processing to generate a file for storing the PC stream in ISOBMFF.

[0157] Figure 15 The data generation device 51 shown includes a control unit 61 and a file generation unit 62. The data generation device 51 generates segmented files and MPD files of a V-PCC stream transmitted through MPEG-DASH or the like and uploads these files to the server.

[0158] The control unit 61 controls all the operations of the data generation device 51. For example, the control unit 61 controls the file generation unit 62 to generate a segmented file storing the V-PCC stream and an MPD file including metadata, and uploads the segmented file and the MPD file.

[0159] The file generation unit 62 not only generates a segmented file and an MPD file under the control of the control unit 61, but also uploads (sends) the segmented file and the MPD file to the server via the network.

[0160] The file generation unit 62 includes a data input unit 71, a data encoding / generation unit 72, an MPD file generation unit 73, a recording unit 74, and an upload unit 75.

[0161] The data input unit 71 not only acquires point cloud data and provides the data to the data encoding / generation unit 72, but also acquires the metadata required to generate the MPD file and provides the metadata to the MPD file generation unit 73.

[0162] The data encoding / generation unit 72 not only generates a texture image, a geometric image, and an occupancy map image based on the point cloud data provided from the data input unit 71, but also generates a segmented file storing its V-PCC stream and provides the segmented file to the recording unit 74.

[0163] The data encoding / generation unit 72 includes a preprocessing unit 76, an encoding unit 77, and a file generation unit 78.

[0164] The preprocessing unit 76 not only generates a texture image, a geometric image, and an occupancy map image based on the point cloud data provided from the data input unit 71, but also generates projection metadata (projection direction information and yes / no separate display information) and image quality metadata, and provides the above images and data to the encoding unit 77. In addition, the preprocessing unit 76 divides the texture image, the geometric image, and the occupancy map image for each projection direction, as described in Figure 6 described.

[0165] The encoding unit 77 encodes the texture image, the geometric image, and the occupancy map image provided from the preprocessing unit 76, generates segmented V-PCC streams (i.e., texture streams, geometric streams, and occupancy map streams segmented for each projection direction and auxiliary information including projection metadata (projection direction information and yes / no separate display information)), and provides this information to the file generation unit 78.

[0166] The file generation unit 78 converts the segmented V-PCC stream provided from the encoding unit 77 into a file based on the metadata provided from the data input unit 71 etc., and provides the resulting segmented file to the recording unit 74. It should be noted that the file generation unit 78 may obtain the metadata etc. required for generating the segmented file from the encoding unit 77, or extract such metadata from the segmented V-PCC stream.

[0167] The MPD file generation unit 73 generates an MPD file including information on the point cloud data and the V-PCC stream including the point cloud data based on the metadata provided from the data input unit 71 etc., and provides the file to the recording unit 74. That is, the MPD file generation unit 73 generates an MPD file that signals the projection metadata (projection direction information and yes / no separate display information). It should be noted that the MPD file generation unit 73 may obtain the metadata required for generating the MPD file etc. from the file generation unit 78, or extract such metadata from the segmented file.

[0168] The recording unit 74 records the MPD file provided from the MPD file generation unit 73 and the segmented file provided from the file generation unit 78.

[0169] The upload unit 75 reads out the MPD file and the segmented file of the point cloud data from the recording unit 74, and uploads these files to the server. That is, the upload unit 75 serves as a communication unit for sending the MPD file and the segmented file to the server.

[0170] It should be noted that although an example in which the data generation device 51 serves as a device for uploading the MPD file and the segmented file to the server will be described here, the data generation device 51 may serve as a server. In this case, the upload unit 75 of the data generation device 51 sends the MPD file and the segmented file to the client device via the network.

[0171] Figure 16 is a block diagram showing a configuration example of a data reproduction device, which is an information processing device that performs point cloud reproduction processing of generating a display image according to a file and reproducing point cloud data on the client side of the reproduced content.

[0172] Figure 16 The shown data reproduction device 52 is a DASH client, and includes a control unit 81 and a reproduction processing unit 82.

[0173] The control unit 81 controls all operations of the data reproduction device 52. For example, the control unit 81 controls the reproduction processing unit 82 to obtain the MPD file and the segmented file from the server, and reproduce the point cloud data based on the segmented file.

[0174] The reproduction processing unit 82 reproduces the point cloud data under the control of the control unit 81. The reproduction processing unit 82 includes a file acquisition unit 91, a file processing unit 92, a display control unit 94, a data analysis / decoding unit 95, and a display unit 96.

[0175] Based on the MPD file provided by the file acquisition unit 91, the file processing unit 92 selects the V-PCC stream to be acquired and feeds back its selection result to the file acquisition unit 91. It should be noted that the viewing direction of the user (e.g., viewpoint position, line-of-sight direction, field of view angle) provided by the display control unit 94, etc. is appropriately used for selecting the V-PCC stream to be acquired.

[0176] Based on the selection result provided by the file processing unit 92, the file acquisition unit 91 acquires the segmented file storing the V-PCC stream required for reproducing the point cloud data from the server and provides the segmented file to the file processing unit 97 of the data analysis / decoding unit 95.

[0177] The display control unit 94 controls the reproduction (display) of the point cloud data. For example, the display control unit 94 acquires the detection result of the viewing direction of the user viewing the point cloud and provides the detection result to the file processing unit 92 and the data analysis / decoding unit 95.

[0178] Based on the segmented file provided by the file acquisition unit 91, the data analysis / decoding unit 95 generates an image of the 3D model as the point cloud data and provides the image to the display unit 96. The data analysis / decoding unit 95 includes a file processing unit 97, a decoding unit 98, and a display information generation unit 99.

[0179] The file processing unit 97 extracts the encoded data of the V-PCC stream from the segmented file provided by the file acquisition unit 91 and provides the encoded data to the decoding unit 98.

[0180] The decoding unit 98 decodes the encoded data provided by the file processing unit 97 and provides the obtained V-PCC stream as a result to the display information generation unit 99.

[0181] The display information generation unit 99 reconfigures the point cloud data according to the V-PCC stream provided by the decoding unit 98, generates point cloud image data according to the direction of the user's viewing field based on the detection result of the user's viewing direction provided by the display control unit 94, and provides the point cloud image data to the display unit 96.

[0182] The display unit 96 includes, for example, a liquid crystal display panel and displays (reproduces) the point cloud image based on the data provided by the display information generation unit 99.

[0183] <Examples of processing for file generation processing and point cloud reproduction processing>

[0184] Figure 17is a flowchart for describing a description file generation process, in which Figure 15 the data generation device 51 shown generates a file based on point cloud data.

[0185] For example, when point cloud data is input to the data generation device 51, the process starts. In step S11, the preprocessing unit 76 arranges tiles for each projection direction of the point cloud, and generates a texture image, a geometric image, an occupancy map image, and auxiliary information. Additionally, at this time, the preprocessing unit 76 generates projection metadata (projection direction information and whether to display individually information) for each projection direction.

[0186] In step S12, the preprocessing unit 76 divides the texture image, the geometric image, and the occupancy map image for each projection direction, and provides the divided images to the encoding unit 77.

[0187] In step S13, the encoding unit 77 encodes the texture image, the geometric image, and the occupancy map image and adds auxiliary information, thereby generating a stream, and provides the stream to the file generation unit 78.

[0188] In step S14, the file generation unit 78 stores the divided streams of each projection direction in separate files, and provides the segmented files obtained as a result to the recording unit 74.

[0189] In step S15, the MPD file generation unit 73 generates an MPD to which projection metadata (projection direction information and whether to display individually information) associated with each file generated by the file generation unit 78 has been added, and provides the MPD to the recording unit 74, and then terminates the file generation process. Thereafter, the upload unit 75 reads out the MPD file and the segmented files from the recording unit 74 at a desired timing, and uploads the files to the server.

[0190] Figure 18 is a flowchart for describing a point cloud reproduction process, in which Figure 16 the data reproduction device 52 shown generates and reproduces a display image based on the file.

[0191] For example, when a file is provided to the data reproduction device 52 from the beginning of the file, the process starts, and in step S21, the file acquisition unit 91 selects an AdaptationSet that is closest to the user's viewing direction based on the projection direction information signaled by the projection metadata (projection direction information and whether to display individually information) of the MPD.

[0192] In step S22, the file processing unit 92 identifies the whether to display individually information of the AdaptationSet selected by the file acquisition unit 91 in step S21.

[0193] In step S23, the file processing unit 92 determines whether the AdaptationSet can be displayed alone based on the recognition result in step S23.

[0194] When the file processing unit 92 determines in step S23 that the AdaptationSet cannot be displayed alone, the process proceeds to step S24. In step S24, the file acquisition unit 91 additionally selects the AdaptationSet required for the direction of the user's viewing direction, and then the process enters step S25.

[0195] Meanwhile, when the file processing unit 92 determines in step S23 that the AdaptationSet can be displayed alone, the process enters step S25.

[0196] In step S25, the file acquisition unit 91 additionally selects the AdaptationSet corresponding to the projection direction for directions other than the user's viewing direction.

[0197] In step S26, the file acquisition unit 91 acquires the file referred to by all the selected AdaptationSets, and the data analysis / decoding unit 95 reproduces the point cloud.

[0198] In step S27, the display control unit 94 determines whether the user's viewing direction has changed.

[0199] When the display control unit 94 determines in step S27 that the user's viewing direction has changed, the process returns to step S21 to repeat similar processing.

[0200] Meanwhile, when the display control unit 94 determines in step S27 that the user's viewing direction has not changed, the process proceeds to step S28.

[0201] In step S28, the data analysis / decoding unit 95 determines whether the end of the PC stream has been reached. If the end of the PC stream has not been reached, the process returns to step S26, and if the end of the PC stream has been reached, the process terminates.

[0202] As described above, by sending the MPD with the projection metadata (projection direction information and yes / no separate display information) added by the data generation device 51, the data reproduction device 52 can identify whether the AdaptationSet closest to the user's viewing direction can be displayed alone. This enables the data reproduction device 52 to appropriately acquire the AdaptationSet required for displaying the user's viewing direction, thereby allowing the reproduction of a point cloud with enhanced image quality while reducing the increase in throughput required for point cloud reproduction processing.

[0203] It should be noted that different colors can be assigned to the points of the point cloud, with one color for each projection direction. For example, in the case where the projection direction X+Y+ has information overlapping with the projection direction X+, the colors of the overlapping projection points can be different between one projection direction and another. This enables the display of effects such as glints when the color changes according to the viewing direction, thus allowing for the configuration of a more realistic point cloud.

[0204] <Image quality metadata>

[0205] In addition, image quality metadata can be defined to enhance the image quality of the segmentation stream corresponding to the viewing direction.

[0206] For example, the image quality of a point cloud depends on the sparsity or density of the points included in the point cloud. If the points included in the point cloud are sparse, the image quality is low. If the points included in the point cloud are dense, the image quality is high. Then, in V-PCC, the sparsity or density of the points in the reconfigured point cloud depends on the resolutions of the texture image and the geometry image.

[0207] Here, the term "resolution" refers to the pixel count of the effective tile area, rather than the pixel count of the entire texture image or the entire geometry image. That is, the larger this effective pixel count, the higher the image quality of the reconfigured point cloud. Therefore, the effective pixel information of the V-PCC stream provided as image quality metadata is an effective indicator for enhancing the image quality of the point cloud region corresponding to the user's viewing direction.

[0208] Therefore, by adding image quality metadata to each projection direction file, the client can select and obtain the required files for reproduction based on this information according to the viewing direction, configure a point cloud that provides high image quality in the viewing direction, and display the point cloud.

[0209] A description of the technique for extending the DASH MPD and adding image quality metadata to each referenced file will be given below.

[0210] The effective pixel information of the V-PCC stream is signaled by newly defining a point cloud resolution descriptor (SupplementalPropertyschemeIdUri = "urn:mpeg:mpegI:pc_resolution:2018"). Here, @value indicates the frame average of the total pixel count of the tiles of each frame in the V-PCC stream. For example, the frame average of the total pixel count of the tile area (e.g., the white pixels of the occupancy map shown in Figure 1 ) that can be calculated from the occupancy map stream is set.

[0211] In addition, a frame average value of an approximate tile size calculated based on delta_size_u0 and delta_size_v0 of auxiliary information that is a component of a V-PCC stream may be set.

[0212] Figure 19 An MPD sample to which image quality metadata has been added is shown.

[0213] An example of a streaming acquisition method based on this MPD sample will be described in the case where the viewing direction of a user corresponds to a projection plane of a projection direction X+. For example, according to the first signaling notification method described above, a segmented stream of AdaptationSet@id = vpcc0 is acquired to construct a point cloud in the viewing direction, and further other AdaptationSets (omitted in the Figure 19 MPD sample) are acquired to construct point clouds viewed from other directions.

[0214] At this time, based on the point cloud resolution descriptor, a representation for constructing a point cloud with high image quality is selected for AdaptationSet@id = vpcc0 corresponding to the viewing direction, and a representation for constructing a point cloud with low image quality is selected for other AdaptationSets. As described above, the image quality of the point cloud in the viewing direction can be further enhanced by effectively using network bandwidth by using image quality metadata.

[0215] It should be noted that the point cloud resolution descriptor is applicable to an unsegmented V-PCC stream. Similarly, the point cloud resolution descriptor is effective for selecting a V-PCC stream according to the processing ability of a client. For example, a client with low processing ability may select a V-PCC stream configured with a small amount of point clouds.

[0216] In addition, a segmented V-PCC stream may be selected by using only image quality metadata.

[0217] <Using an extractor>

[0218] A description of how to use an extractor will be given with reference to Figure 20 and 21 .

[0219] As described above, the client has room by separately selecting and acquiring segmented streams. At the same time, the segmented streams must be decoded separately, resulting in an increase in the number of decoder instances required by the client. For this reason, as Figure 20 shown, an extractor track that is an ISOBMFF tool is used, allowing decoding to be performed using the same number of decoder instances as the number of streams before segmentation.

[0220] It should be noted that an extractor track refers to a track that uses the extractor function specified in ISO / IEC 14496-15, and allows the extraction of bitstream components by referring to other tracks and reconfiguring a bitstream. Here, the term "bitstream component" refers to one or more NAL units that can be decoded independently in the case of AVC or HEVC.

[0221] For example, as Figure 21 shown in the MPD sample of, a high-quality direction descriptor (SupplementalProperty schemeIdUri = "urn:mpeg:mpegI:pc_hq_direction:2018") is newly defined, and the projection direction information that enhances the image quality in the V-PCC stream including the extractor track is signaled. Here, @value signals the projection direction information that enhances the image quality in the V-PCC stream including the extract track.

[0222] For example, the projection direction relative to the point cloud local coordinate system is signaled by @value (0 to 9). That is, in the case of @value = 0, the projection direction X+ is signaled, in the case of @value = 1, the projection direction Y+ is signaled, in the case of @value = 2, the projection direction X- is signaled, in the case of @value = 3, the projection direction Y- is signaled, in the case of @value = 4, the projection direction Z+ is signaled, in the case of @value = 5, the projection direction Z- is signaled, in the case of @value = 6, the projection direction X+Y+ is signaled, in the case of @value = 7, the projection direction X-Y+ is signaled, in the case of @value = 8, the projection direction X-Y- is signaled, and in the case of @value = 9, the projection direction X+Y- is signaled.

[0223] In addition, in the case where the image quality is enhanced in multiple directions, multiple directions can be signaled, for example, by being separated by commas in @value. Furthermore, the relative image quality levels of the point cloud between the projection directions can be signaled.

[0224] In this case, the client can reconfigure the V-PCC stream with enhanced image quality in the viewing direction by first selecting and obtaining an appropriate extractor track according to the viewing direction and further obtaining the file referred to by the extractor track. This V-PCC stream can be decoded by the same number of decoders as the number of the V-PCC stream before segmentation.

[0225] In addition, as a modified example of the first signaling notification method, the subjective image quality can be enhanced by additionally transmitting a stream of a point cloud object projected after rotating the point cloud object 45 degrees around the Z axis while keeping the number of projection planes at six, as was done when adding projection planes. At this time, in addition to the above metadata, the rotation information of the point cloud object at the time of projection can also be signaled in the DASH MPD. It should be noted that this modified example applies to the ISOBMFF described later.

[0226] <Second Extension Method>

[0227] The second extension method will be described with reference to Figure 22 and Figure 26 The second extension method signals the projection direction information, the yes / no separate display information, and the image quality metadata for each track through the ISOBMFF extension.

[0228] For example, the projection metadata (projection direction information and yes / no separate display information) and the image quality metadata can be added for each track by extending the ISOBMFF.

[0229] That is, the segmented V-PCC streams are each stored in a track, and then these tracks are stored in a multi-track file defined in the ISOBMFF. Then, as Figure 22 shown, the VPCCGroupBox is redefined as a new track group that groups together the tracks of the segmented V-PCC streams included in one point cloud content.

[0230] For example, the VPCCGroupBox extends the TrackGroupTypeBox and signals the projection metadata (projection direction information and yes / no separate display information) and the image quality metadata. Here, the TrackGroupTypeBox is a tool for grouping together multiple tracks with the same characteristics and specified in the ISOBMFF.

[0231] In addition, as Figure 23As shown, the projection_direction indicates the projection direction of each tile. That is, when projection_direction = 0, the signaling notifies the projection direction X+, when projection_direction = 1, the signaling notifies the projection direction Y+, when projection_direction = 2, the signaling notifies the projection direction X-, when projection_direction = 3, the signaling notifies the projection direction Y-, when projection_direction = 4, the signaling notifies the projection direction Z+, when projection_direction = 5, the signaling notifies the projection direction Z-, when projection_direction = 6, the signaling notifies the projection direction X+Y+, when projection_direction = 7, the signaling notifies the projection direction X-Y+, when projection_direction = 8, the signaling notifies the projection direction X-Y-, and when projection_direction = 9, the signaling notifies the projection direction X+Y-.

[0232] In addition, present_alone indicates whether the trajectory can be used alone to construct and display the point cloud. For example, when present_alone = 0, this indicates that the trajectory cannot be used alone to display the point cloud, while when present_alone = 1, this indicates that the trajectory can be used alone to display the point cloud.

[0233] In addition, point_cloud_resolution indicates the frame average of the total pixel count of the tiles in each frame in the V-PCC stream, and indicates the frame average count of the constructed point cloud.

[0234] This ISOBMFF extension allows selecting tracks according to the viewing direction, decoding only the segmented V-PCC streams required for display, reconstructing the point cloud and preferentially displaying it, rather than decoding all segmented V-PCC stream tracks.

[0235] Here, tiles with multiple projection directions can be included in a single segmented V-PCC stream. In this case, we assume, for example, that the projection_direction is 10 bits long, and the projection directions X+ etc. are assigned to these bits, up to the projection direction X+Y-, starting from the most significant bit, and if 1 is set in each bit field, that projection direction is included. For example, projection_direction = 1100000000 indicates that the projection directions X+ and Y+ are included.

[0236] In addition, a single split V-PCC stream may include tiles that can be displayed separately and another tile that cannot be displayed separately. In this case, present_alone = 0.

[0237] It should be noted that the projection metadata and the image quality metadata may also be signaled by, for example, a Sample Entry other than the VPCCGroupBox. Additionally, the yes / no separately display information may not be indicated as a field of present_alone, but rather by means of the track reference (reference_type = vpcc) of the V-PCC stream track required for display to indicate the track that cannot be displayed separately.

[0238] Furthermore, a group that can be displayed separately can be defined by extending the TrackGroupTypeBox to a new track group. For example, it is indicated that the projection directions X+Y+ that can be displayed separately and the projection directions X+ and Y+ required for its display are grouped together, and these projection directions can be displayed separately group by group.

[0239] <Further Extension of the Second Extension Method>

[0240] A description of the further extension of the second extension method will be given here.

[0241] As described above, the second extension method extends the conventional ISOBMFF in such a way that projection direction information and the like can be obtained. Additionally, the ISOBMFF can be further extended in such a way that additional projection direction identification information can be obtained to indicate that the four additional projection planes (X+Y+ projection plane, X-Y- projection plane, X-Y+ projection plane, and X+Y- projection plane) are images generated from tiles of four additional projection directions.

[0242] For example, by defining a VPCCGroupBox as shown in Figure 24 , the additional projection direction identification information is signaled by 45degree_projection. Then, in the case where 45degree_projection = 0 as shown in Figure 25 , this indicates that the stream is generated from tiles of six projection directions, and in the case where 45degree_projection = 1, this indicates that the stream is generated from tiles of four additional projection directions. Additionally, projection_direction indicates the projection direction information (0: X+, 1: Y+, 2: X-, 3: Y-, 4: Z+, 5: Z-).

[0243] For example, in the case as shown in Figure 24When the indication is signaled using 45degree_projection and projection_direction signaling and it has been indicated that an image is generated from tiles in four additional projection directions by 45degree_projection = 1, this indicates that the projection direction is the direction rotated counterclockwise by 45 degrees about the Z-axis in the direction indicated by projection_direction. Further, projection_direction = 0 signaled together with 45degree_projection = 1 indicates that the projection direction is X+Y+.

[0244] It should be noted that it is possible to signal only 45degree_projection without signaling projection_direction. In this case, if 45degree_projection = 0, this indicates that the stream is generated from tiles in all six projection directions, and if 45degree_projection = 1, this indicates that the stream is generated from tiles in all four additional projection directions.

[0245] Here, in the case of generating a projection direction file for each block for a further extension of the above first extension method, the 45degree_projection signaled for each block can be used together with the block position information for identifying the three-dimensional space information of each block.

[0246] For example, by defining a VPCCGroupBox as shown in Figure 26 the block position information is signaled. For example, the block position information can be indicated by respective fields (i.e., block_offset_x, block_offset_y, block_offset_z, block_size_x, block_size_y, and block_size_z). All of these fields are signaled by relative values obtained by assuming that each side along the X-axis, Y-axis, and Z-axis of the object box is 1, as shown in Figure 14 Moreover, the blocks included in the same object box can be signaled by using the function of the TrackGroupTypeBox extended from it.

[0247] It should be noted that only the 45degree_projection and block position information can be signaled, and the projection_direction is not signaled. In this case, if 45degree_projection = 0, this indicates that the projection direction file for each block is a stream generated from tiles of six projection directions, and if 45degree_projection = 1, this indicates that the projection direction file for each block is a stream generated from tiles of four additional projection directions.

[0248] As described above, the second extension method allows the ISOBMFF to be extended in the following way: in addition to the projection direction information, additional projection direction identification information is also signaled.

[0249] <The Third Extension Method>

[0250] Reference will be made to Figures 27 to 30 A description of the third extension method for signaling projection direction information, yes / no separate display information, and image quality metadata for each tile through Elementary Stream extension will be given.

[0251] For example, projection metadata and image quality metadata can be added for each tile by extending the high-level syntax of the Elementary Stream. That is, the projection metadata and image quality metadata are signaled by extending the high-level syntax of the V-PCC stream.

[0252] Figure 27 An example of extending the auxiliary information that is a component of the V-PCC stream is shown.

[0253] For example, as Figure 28As shown, the projection_direction indicates the projection direction of each tile. That is, when projection_direction = 0, the projection direction X+ is signaled; when projection_direction = 1, the projection direction Y+ is signaled; when projection_direction = 2, the projection direction X- is signaled; when projection_direction = 3, the projection direction Y- is signaled; when projection_direction = 4, the projection direction Z+ is signaled; when projection_direction = 5, the projection direction Z- is signaled; when projection_direction = 6, the projection direction X+Y+ is signaled; when projection_direction = 7, the projection direction X-Y+ is signaled; when projection_direction = 8, the projection direction X-Y- is signaled; when projection_direction = 9, the projection direction X+Y- is signaled.

[0254] Similarly, present_alone indicates whether a tile can be used alone to construct and display a point cloud. For example, when present_alone = 0, it indicates that a tile cannot be used alone to display a point cloud, while when present_alone = 1, it indicates that a tile can be used alone to display a point cloud.

[0255] In addition, point_cloud_resolution indicates the tile pixel count.

[0256] It should be noted that when all tiles included in a single V-PCC stream have the same projection_direction value or the same present_alone value, each tile can be signaled with a single field (default_projection_direction, default_present_alone). Moreover, a field indicating the sum of the point_cloud_resolution values of each tile can be signaled. Then, each field is signaled immediately after the occupancy_aux_stream_size field.

[0257] Furthermore, when a tile cannot be displayed alone, the identifier of the additional tile required to display the tile can be signaled.

[0258] For example, in the case where a split V-PCC stream includes tiles of multiple projection directions, where these tiles have been grouped together in regions for each projection direction, and where each region has been encoded by an encoding method that allows independent decoding of each region, only the required regions can be selected and decoded from the split V-PCC stream according to the viewing direction, the point cloud can be reconstructed, and the point cloud can be preferentially displayed by referring to the projection metadata and image quality metadata at the Elementary Stream level. For example, HEVC is used as the encoding codec, and in this case, the regions encoded with HEVC slices can be decoded independently.

[0259] At this time, the duplicated_patch field can be added by extending the VPCCGroupBox of the ISOBMFF track storing the V-PCC stream. If the V-PCC stream stored in the track includes tiles with overlapping information, it is signaled with 1 in this field, otherwise with 0. Additionally, in the case of duplicated_patch = 1, preparation can be made for post-processing in the case where overlapping information exists before the decoding process.

[0260] <Further Extension of the Third Extension Method>

[0261] A description of the further extension of the third extension method will be given here.

[0262] As described above, the third extension method extends the conventional ElementaryStream in such a way that projection direction information and the like can be obtained. Additionally, the Elementary Stream can be further extended in the following way: additional projection direction identification information can be obtained to indicate that the four additional projection planes (X+Y+ projection plane, X-Y- projection plane, X-Y+ projection plane, and X+Y- projection plane) are images generated from tiles of four additional projection directions.

[0263] For example, as Figure 29 shown, the additional projection direction identification information is signaled with 45degree_projection. It should be noted that Figure 29 the auxiliary information shown has been extracted from Figure 27 a part of the auxiliary information shown.

[0264] For example, in the case where, as Figure 30When 45degree_projection = 0 as shown, this indicates that the tile is for six projection directions, while when 45degree_projection = 1, this indicates that the tile is for four additional projection directions. Additionally, projection_direction indicates the projection direction information (0: X+, 1: Y+, 2: X-, 3: Y-, 4: Z+, 5: Z-).

[0265] In the case as Figure 29 shown, where it is signaled using 45degree_projection and projection_direction, and an image has been generated from tiles of four additional projection directions with 45deg_projection = 1, this indicates that the projection direction is the direction rotated counterclockwise by 45 degrees around the Z axis in the direction indicated by projection_direction. In addition, projection_direction = 0 signaled together with 45degree_projection = 1 indicates the projection direction as X+Y+.

[0266] As described above, the third extension method can extend the Elementary Stream such that additional projection direction identification information is signaled in addition to the projection direction information.

[0267] As described above, according to the present technology, DASH MPD metadata is defined, and this DASH MPD metadata signals the projection direction information and the yes / no separate display information of the tiles in the segmented stream when transmitted via DASH after segmenting the V-PCC stream. In addition, DASH MPD metadata is defined, and this DASH MPD metadata signals the additional projection direction identification information. This enables the transmission of a V-PCC stream with enhanced image quality by adding projection planes, while reducing the increase in client processing overhead by effectively utilizing the network bandwidth.

[0268] For example, the conventional technology is based on the fact that obtaining the entire stream of the texture image, geometric image, and occupancy map image inevitably results in an increase in the bit rate as much as the increase in the number of projection planes compared to a stream with six projection planes when transmitting the V-PCC stream. To solve this problem, a possible solution to this problem is to segment the V-PCC stream (texture stream, geometric stream, and occupancy map stream) for each projection plane and select and transmit the segmented V-PCC stream of the required projection plane according to the user's viewing direction in the case of limited network bandwidth. However, in this case, the client cannot identify whether the tiles of the additional projection plane have all the projection direction information, thus making it impossible to determine whether the image can be correctly displayed separately.

[0269] Contrary to such conventional techniques, according to the present technique, in an environment with limited network bandwidth, the projection direction information and the yes / no separate display information can be signaled to selectively select and transmit only the necessary segmented V-PCC streams according to the viewing direction of the customer. That is, on the premise that not all projection direction information is always available, it is not necessary to obtain all V-PCC streams, thus avoiding the acquisition of unnecessary segmented streams.

[0270] In addition, the image quality of the V-PCC stream in the viewing direction can be enhanced by extending the DASH MPD and signaling the image quality metadata.

[0271] Moreover, according to the present technique, the track can be selected according to the viewing direction, only the necessary segmented V-PCC streams for display are decoded, the point cloud is reconstructed, and the point cloud is preferentially displayed by extending the ISOBMFF to segment and store the V-PCC stream instead of decoding all segmented V-PCC stream tracks.

[0272] In addition, in the case where the segmented V-PCC stream includes tiles of multiple projection directions, according to the present technique, only the required area can be selected and decoded according to the viewing direction, the point cloud is reconstructed, and the point cloud is preferentially displayed by extending the high-level syntax of the V-PCC stream and signaling the projection metadata and the image quality metadata.

[0273] <Configuration example of a computer>

[0274] Next, the above series of processes (information processing methods) can be executed by hardware or software. In the case of executing the series of processes by software, the program included in the software is installed in a general-purpose computer or the like.

[0275] Figure 31 It is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above series of processes is installed.

[0276] The program can be pre-recorded in the hard disk 105 or the ROM 103 which is a recording medium built into the computer.

[0277] Alternatively, the program can be stored (recorded) in the removable recording medium 111 driven by the drive 109. The above removable recording medium 111 can be provided as software generally called package software. Here, as examples of the removable device, a floppy disk, a CD-ROM (Compact Disc Read-Only Memory), an MO (Magneto-Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory can be cited.

[0278] It should be noted that, in addition to being installed in the computer from the removable recording medium 111 as described above, the program can also be downloaded to the computer via a communication network or a broadcast network and installed in the built-in hard disk 105. That is, the program can be wirelessly transmitted from a download site to the computer via an artificial satellite for digital satellite broadcasting, or the program can be transmitted to the computer in a wired manner via a network such as a LAN (local area network) or the Internet.

[0279] The computer includes a CPU (central processing unit) 102, and an input / output interface 110 is connected to the CPU 102 via a bus 101.

[0280] When an instruction is input, for example, due to an operation by the user on the input unit 107, the CPU 102 executes a program stored in the ROM (read-only memory) 103 via the input / output interface 110. Alternatively, the CPU 102 loads the program stored in the hard disk 105 into the RAM (random access memory) 104 for execution.

[0281] This allows the execution of the processing according to the above flowchart or the processing performed by the components in the above block diagram. Then, the CPU 102 outputs the processing result from the output unit 106 via the input / output interface 110 or transmits it from the communication unit 108, and further records the processing result, for example, in the hard disk 105.

[0282] It should be noted that the input unit 107 includes a keyboard, a mouse, a microphone, etc. In addition, the output unit 106 includes an LCD (liquid crystal display), a speaker, etc.

[0283] Here, in this specification, the processing performed by the computer according to the program does not necessarily have to be executed in chronological order as described in the flowchart. That is, the processing performed by the computer according to the program includes processing that is executed in parallel or separately (for example, parallel processing or object-based processing).

[0284] Moreover, the program can be processed by a single computer (processor) or by multiple computers in a distributed manner. In addition, the program can be transmitted to a remote computer for execution.

[0285] The term "system" in this specification refers to a group of multiple components (for example, devices, modules (components)), and it does not matter whether all the components are housed in the same housing. Therefore, multiple devices housed in different housings and connected via a network, as well as a single device in which multiple modules are housed in a single housing, are both systems.

[0286] In addition, for example, components described as a single device (or processing unit) can be divided and configured as multiple devices (or processing units). Conversely, components described above as multiple devices (or processing units) can be combined and configured as a single device (or processing unit). Additionally, components other than those described above can be naturally added to the components of each device (or processing unit). Furthermore, as long as the components and operations of the system generally remain substantially the same, a part of the components of a device (or processing unit) can be included in the components of another device (or processing unit).

[0287] In addition, for example, the present technology can have a cloud computing configuration in which multiple devices process functions in a shared and collaborative manner via a network.

[0288] Moreover, for example, the above program can be executed by a desired device. In that case, it is sufficient for the device to have the necessary functions (e.g., functional blocks) to obtain the necessary information.

[0289] Furthermore, for example, each step described in the above flowchart can be executed not only by a single device but also by multiple devices in a shared manner. Additionally, in the case where a single step includes multiple processes, the multiple processes included in that single step can be executed by only a single device or by multiple devices in a shared manner. In other words, the multiple processes included in a single step can be executed as processes in multiple steps. Conversely, processes described as multiple steps can be executed together as a single step.

[0290] It should be noted that the processes in the steps of the program described as being executed by a computer can be executed chronologically in the order described in this specification, or can be executed in parallel or separately, for example, at necessary timings when the program is called. That is, unless there is an inconsistency, the processes of each step can be executed in an order different from the above order. Additionally, the processes of the steps of the program can be executed in parallel with the processes of another program, or can be combined with the processes of another program and executed together.

[0291] Unless there is an inconsistency, each of the various current technologies described in this specification can be executed independently and separately. It goes without saying that multiple desired present technologies can be executed in combination. For example, some or all of the present technologies described in any one embodiment can be executed together with some or all of the present technologies described in other embodiments. Moreover, some or all of the above-mentioned desired present technologies can be executed in combination with other technologies not described above.

[0292] <Examples of combinations of components>

[0293] It should be noted that the present technology can also have the following configuration: (1)

[0295] An information processing apparatus, comprising:

[0296] A preprocessing unit adapted to generate image data in a plurality of projection directions by projecting 3D data in the plurality of projection directions and converting the 3D data into two-dimensional data, and to generate projection direction information indicating the projection directions of the image data as projection metadata. (2)

[0298] The information processing apparatus according to (1), wherein,

[0299] The projection metadata includes additional projection direction identification information indicating that the image data has been generated by projecting in an additional projection direction. (3)

[0301] The information processing apparatus according to (2), wherein,

[0302] The image data has a texture image, a geometry image, and an occupancy map image for each of the projection directions. (4)

[0304] The information processing apparatus according to (3), wherein,

[0305] The preprocessing unit further generates yes / no individual display information as the projection metadata, the yes / no individual display information indicating whether the image data in the projection direction indicated by the projection direction information or the additional projection direction identification information can be individually displayed. (5)

[0307] The information processing apparatus according to any one of (1) to (4), wherein,

[0308] The preprocessing unit further generates an effective pixel count of the image data as the image quality metadata. (6)

[0310] The information processing apparatus according to any one of (1) to (5), further comprising:

[0311] An MPD file generation unit adapted to generate an MPD (Media Presentation Description), the MPD signaling coupled to the generated projection metadata for each of the files, each of the files being a data unit referred to when displaying the 3D data. (7)

[0313] The information processing apparatus according to (6), wherein,

[0314] In a case where one of the files cannot be used to separately display the 3D data, the MPD file generation unit signals the association with the other file among the files, where the other file is required to display the 3D data. (8)

[0316] The information processing apparatus according to (6), wherein

[0317] In a case where one of the files cannot be used to separately display the 3D data, the MPD file generation unit groups the one file with the other file among the files to signal, where the other file is required to display the 3D data. (9)

[0319] The information processing apparatus according to any one of (1) to (5) further includes:

[0320] A file generation unit configured to generate an ISOBMFF (ISO Base Media File Format) file, where the ISOBMFF file signals projection metadata for each track, and each of the tracks is a data unit in a plurality of the projection directions. (10)

[0322] The information processing apparatus according to (9), wherein

[0323] In a case where one of the tracks cannot be used to separately display the 3D data, the file generation unit groups the one track with another track among the tracks to signal, where the other track is required to display the 3D data. (11)

[0325] The information processing apparatus according to any one of (1) to (5) further includes:

[0326] An encoding unit configured to encode an Elementary Stream, where the Elementary Stream signals projection metadata for each tile, and each of the tiles is a data unit in a plurality of the projection directions. (12)

[0328] The information processing apparatus according to (11), wherein

[0329] In a case where one of the tiles cannot be used to separately display the 3D data, the encoding unit groups the one tile with identifiers of additional tiles required to display the 3D data to signal. (13)

[0331] The information processing apparatus according to (4), wherein,

[0332] In a case where image data in a projection direction has been generated based on a block that is a part of an object frame having a rectangular parallelepiped shape including the 3D data, the preprocessing unit further generates block position information as the projection metadata, and the block position information is used to identify the position of each of the blocks in a three-dimensional space relative to the object frame. (14)

[0334] An information processing method, comprising:

[0335] Performing the following operations by an information processing apparatus,

[0336] Generating image data in a plurality of projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two-dimensional data; and

[0337] Generating projection direction information indicating the projection direction of the image data as projection metadata.

[0338] It should be noted that the present embodiment is not limited to the above embodiment, and can be modified in various ways without departing from the gist of the present disclosure. In addition, the beneficial effects described in this specification are merely exemplary and not restrictive, and there may be other beneficial effects.

[0339] [List of reference numerals]

[0340] 51: Data generation device

[0341] 52: Data reproduction device

[0342] 61: Control unit

[0343] 62: File generation unit

[0344] 71: Data input unit

[0345] 72: Data encoding / generation unit

[0346] 73: MPD file generation unit

[0347] 74: Recording unit

[0348] 75: Upload unit

[0349] 76: Preprocessing unit

[0350] 77: Encoding unit

[0351] 78: File generation unit

[0352] 81: Control unit

[0353] 82: Reproduction processing unit

[0354] 91: File acquisition unit

[0355] 92: File processing unit

[0356] 94: Display control unit

[0357] 95: Data analysis / decoding unit

[0358] 96: Display unit

[0359] 97: File processing unit

[0360] 98: Decoding unit

[0361] 99: Display information generation unit

Claims

1. An information processing apparatus, comprising: A preprocessing unit, which is adapted to generate image data of a plurality of the projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two-dimensional data, and generate projection direction information indicating the projection direction of the image data and yes / no separate display information indicating whether the image data in the projection direction indicated by the projection direction information can be separately displayed as projection metadata; And A file generation unit, which is adapted to generate an ISO Base Media File Format (ISOBMFF) file, and the ISOBMFF file signals the projection metadata of each track, and each of the tracks is a data unit of a plurality of the projection directions.

2. The information processing apparatus according to claim 1, wherein The projection metadata includes additional projection direction identification information, and the additional projection direction identification information indicates that the image data has been generated by projecting in an additional projection direction.

3. The information processing apparatus according to claim 2, wherein The image data has a texture image, a geometry image, and an occupancy map image for each of the projection directions.

4. The information processing apparatus according to claim 3, wherein The yes / no separate display information further indicates whether the image data in the projection direction indicated by the additional projection direction identification information can be separately displayed.

5. The information processing apparatus according to claim 1, wherein The preprocessing unit further generates an effective pixel count of the image data as image quality metadata.

6. The information processing apparatus according to claim 1, wherein In a case where one of the tracks cannot be used to separately display the 3D data, the file generation unit groups the one track with another one of the tracks for signaling, and the another one of the tracks is required to display the 3D data.

7. The information processing apparatus according to claim 4, wherein In a case where the image data of the projection direction has been generated based on a block that is a part of an object frame in a cuboid shape including the 3D data, the preprocessing unit further generates block position information as the projection metadata, and the block position information is used to identify the position of each of the blocks in three-dimensional space relative to the object frame.

8. An information processing method, comprising: Performing the following operations by an information processing apparatus, Generating image data of a plurality of the projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two-dimensional data; Generating projection direction information indicating the projection direction of the image data and yes / no separate display information indicating whether the image data in the projection direction indicated by the projection direction information can be separately displayed as projection metadata; And Generating an ISO Base Media File Format (ISOBMFF) file, and the ISOBMFF file signals the projection metadata of each track, and each of the tracks is a data unit of a plurality of the projection directions.

9. A non-transitory computer-readable medium having a program implemented thereon, the program causing the computer to execute an information processing method when executed by the computer, the information processing method comprising: Generating image data of a plurality of the projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two-dimensional data; Generating projection direction information indicating the projection directions of the image data and yes / no displayable information indicating whether the image data in the projection directions indicated by the projection direction information can be separately displayed as projection metadata; And Generating an ISO Base Media File Format (ISOBMFF) file that signals the projection metadata for each track, each track being a data unit of a plurality of the projection directions.

10. A computer program product comprising a computer program that causes a computer to execute an information processing method when executed by the computer, the information processing method comprising: Generating image data of a plurality of the projection directions by projecting 3D data in a plurality of projection directions and converting the 3D data into two-dimensional data; Generating projection direction information indicating the projection directions of the image data and yes / no displayable information indicating whether the image data in the projection directions indicated by the projection direction information can be separately displayed as projection metadata; And Generating an ISO Base Media File Format (ISOBMFF) file that signals the projection metadata for each track, each track being a data unit of a plurality of the projection directions.

Images