Point cloud data processing method, device, storage medium and electronic device

Abstract

An embodiment of the present invention provides a point cloud data processing method, device, storage medium and electronic device, wherein the method includes: determining static geometrically encoded point cloud data containing spatial area information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data; decoding partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data; rendering the partial spatial area of ​​the 3D scene according to the decoded data, which can solve the problem in the related art that it is usually necessary to traverse the complete point cloud data before identifying the partial point cloud data required by the user, resulting in low parsing efficiency and long time consumption. The required partial project data is determined according to user needs, so that the required partial project data can be obtained without traversing the complete point cloud data, thereby improving decoding efficiency, shortening decoding time, and shortening the user's rendering waiting time.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of communications, and in particular, to a point cloud data processing method, device, storage medium and electronic device. Background Art

[0002] There are many ways to describe the three-dimensional world in immersive media technology. In addition to VR panoramic videos, a three-dimensional object can be directly digitized by a laser radar or camera matrix, and the three-dimensional object can be represented by independent points and surfaces, that is, a three-dimensional point cloud. A point cloud is a set of irregularly distributed discrete points in space that express the spatial structure and surface properties of a three-dimensional object or scene. Each point in a point cloud has at least three-dimensional position information, and may also have color, material or other information depending on the application scenario. Point clouds can be captured by multiple cameras and depth sensors, and the number of points can reach thousands to billions. There is no connection or order between points, and they can be sorted arbitrarily. Therefore, point clouds have attracted widespread attention in the industry with their flexible and convenient expression form and high data accuracy.

[0003] Compared with other volumetric media formats, point clouds are relatively easy to capture and render. Application scenarios include: autonomous driving, real-time inspection, cultural heritage, 6DoF immersive real-time communication, etc. According to the time information representation of point clouds, point clouds can be divided into two categories: static point clouds and dynamic point clouds. For the application scenarios of static point clouds, such as large static point clouds such as cities and streets, users usually only need to focus on part of the point cloud data when applying point cloud data, without obtaining the complete point cloud object. For example:

[0004] 1) Large and complex scene point cloud data such as geographic information systems, select scenes according to geographical areas, and view randomly; digital cultural heritage supports selection based on viewpoint perspective and virtual tours;

[0005] 2) Large and complex scenes and maps support viewing from far to near and from coarse to fine; digital cultural heritage supports progressive rendering, screen zooming, and detail viewing;

[0006] 3) For scenarios where multiple point cloud objects are associated with each other, select part of the 3D point cloud for decoding and rendering based on the object.

[0007] For the above application scenarios, users only pay attention to the point cloud data of some areas or the point cloud data containing some details. In the existing technical solutions for static point clouds, the decoder usually needs to traverse the complete point cloud code stream before it can identify the partial point cloud data required by the user. For large point cloud objects, such as urban scenes, map navigation, etc., this solution has low parsing efficiency and takes a long time. In addition, in the existing static point cloud storage solutions, the entire object is usually used as the storage unit, which is not conducive to parallel decoding and has low flexibility.

[0008] There is no solution yet to the problem that in related technologies, it is usually necessary to traverse the entire point cloud data before identifying the partial point cloud data required by the user, resulting in low parsing efficiency and long time consumption. Summary of the invention

[0009] The embodiments of the present invention provide a point cloud data processing method, device, storage medium and electronic device to at least solve the problem in the related art that it is usually necessary to traverse the complete point cloud data before identifying the part of the point cloud data required by the user, resulting in low parsing efficiency and long time consumption.

[0010] According to one embodiment of the present invention, a point cloud data processing method is provided, the method comprising:

[0011] Determining static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud item data;

[0012] Decoding partial item data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data;

[0013] A partial spatial area of ​​the 3D scene is rendered according to the decoded data.

[0014] In one embodiment, the point cloud project data is determined by the project type in the metadata data box, wherein the project type includes at least: point cloud project, point cloud block project, point cloud space area project.

[0015] In one embodiment, the decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes:

[0016] In the case where the project type is the point cloud project, the partial project data corresponding to one or more blocks are determined according to the subsample project characteristics associated with the point cloud project in the meta information data box, and the partial project data are decoded.

[0017] In one embodiment, determining the partial project data corresponding to one or more blocks according to the subsample project characteristic data box associated with the point cloud project in the meta information data box comprises:

[0018] Partial project data corresponding to one or more chunks are determined according to the first sub-sample data type and the chunk identifier in the sub-sample project characteristics.

[0019] In one embodiment, the decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes:

[0020] In the case where the project type is the point cloud project, the partial project data corresponding to one or more spatial areas is determined based on the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial area project characteristics, and the partial project data is decoded.

[0021] In one embodiment, determining the portion of project data corresponding to one or more spatial regions according to the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial region project characteristics comprises:

[0022] The portion of the project data corresponding to one or more spatial regions is determined based on the second sub-sample data type and the spatial region identifier in the sub-sample project characteristics.

[0023] In one embodiment, the 3D space region project characteristics include:

[0024] The number of spatial regions, description information of the spatial regions, the number of blocks contained in the spatial regions, and the corresponding identifiers of the blocks contained in the spatial regions.

[0025] In one embodiment, the spatial region description information includes: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0026] In one embodiment, the decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes:

[0027] In the case where the project type is a point cloud segmentation project, identifying point cloud data containing the 3D static scene according to the point cloud full cluster group;

[0028] According to the point cloud block set in the point cloud configuration item characteristics and one or more point cloud block items in the meta-information data box, the partial project data corresponding to one or more blocks are determined, and the partial project data are decoded.

[0029] In one embodiment, the point cloud tile project includes a single tile described in a point cloud tile set, wherein each of the one or more tiles represents an independently decodable subset of data in a point cloud spatial region.

[0030] In one embodiment, determining the portion of static point cloud data based on spatial region division includes:

[0031] In the case where the project type is a point cloud spatial region project, identifying point cloud data containing the 3D static scene according to the point cloud full cluster group;

[0032] According to one or more point cloud spatial area items in the metadata data box, spatial area description item characteristics associated with the point cloud spatial items are determined to determine partial item data corresponding to the one or more point cloud spatial areas.

[0033] In an embodiment, each of the one or more point cloud spatial regions corresponds to a single spatial region of the spatial region description item property description.

[0034] In one embodiment, the spatial region description item characteristics include: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0035] In one embodiment, the point cloud full cluster group includes:

[0036] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes: a point cloud subset divided according to blocks, and a point cloud subset divided according to spatial regions.

[0037] In one embodiment, the decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes:

[0038] In the case where the project type is a point cloud space area project, partial project data for indicating a marked scene is determined according to the area mark group, and the partial project data is decoded.

[0039] In one embodiment, the area marking group includes at least a marking type and a marking text, and the partial point cloud items indicating the marking scene are described by the marking type and the marking text.

[0040] In one embodiment, the method further comprises:

[0041] Determining and obtaining one or more media resources associated with the portion of project data;

[0042] While decoding the portion of the project data, decoding the one or more media resources;

[0043] The one or more media resources are played while rendering a partial spatial area of ​​the 3D scene according to the decoded data.

[0044] In one embodiment, determining and obtaining one or more media resources associated with the portion of project data includes:

[0045] One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

[0046] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0047] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data.

[0048] According to another embodiment of the present invention, a point cloud data processing method is also provided, the method comprising:

[0049] Determining static geometrically encoded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by cloud project data;

[0050] Decoding partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data;

[0051] Rendering partial level details of the 3D scene according to the decoded data.

[0052] In one embodiment, the point cloud project data is determined by a project type in a metadata data box, and the project type includes at least: a point cloud project and a point cloud level project.

[0053] In one embodiment, decoding partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data comprises:

[0054] In the case where the project type is a point cloud project, partial project data corresponding to one or more levels are determined according to subsample project characteristics associated with the point cloud project in the meta information data box, and the partial project data are decoded.

[0055] In one embodiment, determining the partial project data corresponding to one or more levels according to the subsample project characteristic data box associated with the point cloud project in the meta information data box includes:

[0056] Partial item data corresponding to one or more levels are determined according to the third sub-sample data type and the level value in the sub-sample item characteristic.

[0057] In one embodiment, the decoding of partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data further includes:

[0058] In the case where the project type is a point cloud level project, identifying point cloud data containing all details of the 3D static scene according to the point cloud full cluster group;

[0059] According to one or more point cloud level items in the meta information data box and level detail item characteristics associated with the point cloud level items, partial item data corresponding to one or more levels are determined, and the partial item data are decoded.

[0060] In an embodiment, each of the one or more levels corresponds to a single level of level-detail item property description.

[0061] In one embodiment, the level detail item characteristics include at least: point cloud level value, level point number.

[0062] In one embodiment, the point cloud full cluster group includes:

[0063] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of the point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes a point cloud subset divided according to the level of detail.

[0064] In one embodiment, the method further comprises:

[0065] Determining and obtaining one or more media resources associated with the portion of project data;

[0066] While decoding the portion of project data, decoding the one or more media resources;

[0067] The one or more media resources are played while rendering some level details of the 3D scene according to the decoded data.

[0068] In one embodiment, determining and obtaining one or more media resources associated with the portion of project data includes:

[0069] One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

[0070] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0071] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and attribute project data are divided into multiple levels, and each level is composed of a group of evenly distributed point sets.

[0072] According to another embodiment of the present invention, there is further provided a point cloud data processing device, the device comprising:

[0073] A first determination module is used to determine static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data;

[0074] A first decoding module, used for decoding partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data;

[0075] The first rendering module is used to render a partial spatial area of ​​the 3D scene according to the decoded data.

[0076] In one embodiment, the point cloud project data is obtained through a project type in a metadata data box, wherein the project type includes at least: a point cloud project, a point cloud block project, and a point cloud space area project.

[0077] In one embodiment, the first determining module includes:

[0078] The first determination submodule is used to determine the partial project data corresponding to one or more blocks according to the subsample project characteristics associated with the point cloud project in the metadata data box, and decode the partial project data when the project type is the point cloud project.

[0079] In one embodiment, the first determining submodule is further configured to:

[0080] Partial project data corresponding to one or more chunks are determined according to the first sub-sample data type and the chunk identifier in the sub-sample project characteristics.

[0081] In one embodiment, the first determining module includes:

[0082] The second determination submodule is used to determine the partial project data corresponding to one or more spatial areas according to the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial area project characteristics, and decode the partial project data when the project type is the point cloud project.

[0083] In one embodiment, the second determining submodule is further used to

[0084] The portion of the project data corresponding to one or more spatial regions is determined based on the second sub-sample data type and the spatial region identifier in the sub-sample project characteristics.

[0085] In one embodiment, the 3D space region project characteristics include:

[0086] The number of spatial regions, description information of the spatial regions, the number of blocks contained in the spatial regions, and the corresponding identifiers of the blocks contained in the spatial regions.

[0087] In one embodiment, the spatial region description information includes: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0088] In one embodiment, the first determining module includes:

[0089] A first identification submodule is used to identify point cloud data containing the 3D static scene according to the point cloud full cluster group when the project type is a point cloud block project;

[0090] The third determination submodule is used to determine the partial project data corresponding to one or more blocks according to the point cloud block set in the point cloud configuration project characteristics and one or more point cloud block items in the meta-information data box, and decode the partial project data.

[0091] In one embodiment, the point cloud tile project includes a single tile described in a point cloud tile set, wherein each of the one or more tiles represents an independently decodable subset of data in a point cloud spatial region.

[0092] In one embodiment, the first determining module includes:

[0093] A second identification submodule is used to identify point cloud data containing the 3D static scene according to the point cloud full cluster group when the project type is a point cloud space area project;

[0094] The fourth determination submodule is used to determine partial project data corresponding to one or more point cloud space areas according to one or more point cloud space area items in the metadata data box and the spatial area description project characteristics associated with the point cloud space items, and decode the partial project data.

[0095] In an embodiment, each of the one or more point cloud spatial regions corresponds to a single spatial region of the spatial region description item property description.

[0096] In one embodiment, the spatial region description item characteristics include: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0097] In one embodiment, the point cloud full cluster group includes:

[0098] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes: a point cloud subset divided according to blocks, and a point cloud subset divided according to spatial regions.

[0099] In one embodiment, the first determining module includes:

[0100] The fifth determination submodule is used to determine partial project data for indicating a marked scene according to the area marking group when the project type is a point cloud space area project, and decode the partial project data.

[0101] In one embodiment, the area marking group includes at least a marking type and a marking text, and the partial point cloud items indicating the marking scene are described by the marking type and the marking text.

[0102] In one embodiment, the device further comprises:

[0103] A first resource determination module, used to determine and obtain one or more media resources associated with the portion of project data;

[0104] A first decoding resource module, configured to decode the one or more media resources while decoding the portion of project data;

[0105] The first playback module is configured to play the one or more media resources while rendering a partial spatial area of ​​the 3D scene according to the decoded data.

[0106] In one embodiment, the first resource determination module is further configured to:

[0107] One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

[0108] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0109] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data.

[0110] According to another embodiment of the present invention, there is further provided a point cloud data processing device, the device comprising:

[0111] A second determination module is used to determine static geometrically encoded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by cloud project data;

[0112] A second decoding module, used for decoding partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data;

[0113] The second rendering module is used to render some level details of the 3D scene according to the decoded data.

[0114] In one embodiment, the point cloud project data is determined by a project type in a metadata data box, and the project type includes at least: a point cloud project and a point cloud level project.

[0115] In one embodiment, the second determining module includes:

[0116] The sixth determination submodule is used to determine partial project data corresponding to one or more levels according to subsample project characteristics associated with the point cloud project in the meta-information data box when the project type is a point cloud project, and decode the partial project data.

[0117] In one embodiment, the sixth determining submodule is further used to

[0118] Partial item data corresponding to one or more levels are determined according to the third sub-sample data type and the level value in the sub-sample item characteristic.

[0119] In one embodiment, the second determining module includes:

[0120] A second identification submodule is used for identifying point cloud data containing all details of the 3D static scene according to the point cloud full cluster group when the project type is a point cloud level project;

[0121] The seventh determination submodule is used to determine partial project data corresponding to one or more levels based on one or more point cloud level items in the meta-information data box and level detail project characteristics associated with the point cloud level items, and decode the partial project data.

[0122] In an embodiment, each of the one or more levels corresponds to a single level of level-detail item property description.

[0123] In one embodiment, the level detail item characteristics include at least: point cloud level value, level point number.

[0124] In one embodiment, the point cloud full cluster group includes:

[0125] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of the point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes a point cloud subset divided according to the level of detail.

[0126] In one embodiment, the device further comprises:

[0127] A second resource determination module, used to determine and obtain one or more media resources associated with the portion of project data;

[0128] A second decoding resource module, configured to decode the one or more media resources while decoding the portion of project data;

[0129] The second playback module is configured to play the one or more media resources while rendering some level details of the 3D scene according to the decoded data.

[0130] In one embodiment, the second resource determination module includes:

[0131] An eighth determination submodule is used to determine and obtain one or more media resources associated with the partial project data according to the multi-entity playback group.

[0132] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0133] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and attribute project data are divided into multiple levels, and each level is composed of a group of evenly distributed point sets.

[0134] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0135] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0136] Through the present invention, the problem that in the related technology, it is usually necessary to traverse the complete point cloud data before identifying the partial point cloud data required by the user, resulting in low parsing efficiency and long time consumption can be solved. The required partial project data is determined according to user needs, so that the required partial project data can be obtained without traversing the complete point cloud data, thereby improving the decoding efficiency, shortening the decoding time, and shortening the user's rendering waiting time. BRIEF DESCRIPTION OF THE DRAWINGS

[0137] Figure 1 is a hardware structure block diagram of a mobile terminal of a point cloud data processing method according to an embodiment of the present invention;

[0138] Figure 2 The process of the point cloud data processing method according to an embodiment of the present invention is Figure 1 ;

[0139] Figure 3 The process of the point cloud data processing method according to an embodiment of the present invention is Figure 2 ;

[0140] Figure 4 is a schematic diagram of a single-item storage structure of a static point cloud according to this embodiment;

[0141] Figure 5 is a schematic diagram of a static point cloud multi-project storage structure according to this embodiment;

[0142] Figure 6 is a schematic diagram of a static point cloud storage based on Subsample according to this embodiment;

[0143] Figure 7 is a schematic diagram of Subsample division based on spatial regions according to this embodiment;

[0144] Figure 8 is a schematic diagram of static point cloud storage based on Tile Item according to this embodiment;

[0145] Fig. 9 is a schematic diagram of static point cloud storage based on 3D region property and Subsample according to this embodiment;

[0146] Fig.10 is a schematic diagram of static point cloud storage based on Region Item according to this embodiment;

[0147] Fig.11 is a schematic diagram of subsample division based on hierarchical information according to this embodiment;

[0148] Fig.12 is a schematic diagram of static point cloud storage based on Layer Item according to this embodiment;

[0149] Fig.13 is a schematic diagram of static point cloud storage based on spatial regions and hierarchical details according to the present embodiment;

[0150] Fig.14 is a schematic diagram of a static point cloud spatial region marking according to this embodiment;

[0151] Fig.15 is a schematic diagram of associating and storing a static point cloud and a dynamic point cloud according to this embodiment;

[0152] Fig.16 is a frame of the point cloud data processing device according to this embodiment Figure 1 ;

[0153] Fig.17 is a frame of the point cloud data processing device according to this embodiment Figure 2 . DETAILED DESCRIPTION

[0154] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0155] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0156] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is a hardware structure block diagram of a mobile terminal of a point cloud data processing method according to an embodiment of the present invention, such as Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0157] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data processing method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0159] In this embodiment, a point cloud data processing method running on the above mobile terminal or network architecture is provided. Figure 2 The process of the point cloud data processing method according to an embodiment of the present invention is Figure 1 ,like Figure 2 As shown, the process includes the following steps:

[0160] Step S202, determining static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data;

[0161] In this embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and attribute project data are divided into multiple levels, and the point cloud project data of each level is composed of a group of evenly distributed point sets.

[0162] Step S204, decoding partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data;

[0163] Step S206: Render a partial spatial area of ​​the 3D scene according to the decoded data.

[0164] Through the above steps S202 to S206, the problem that in the related technology, it is usually necessary to traverse the complete point cloud data before identifying the partial point cloud data required by the user, resulting in low parsing efficiency and long time consumption can be solved. The required partial project data is determined according to user needs, so that the required partial project data can be obtained without traversing the complete point cloud data, thereby improving the decoding efficiency, shortening the decoding time, and shortening the user's rendering waiting time.

[0165] In this embodiment, the point cloud project data is determined by the project type in the metadata data box, wherein the project type includes at least: point cloud project, point cloud block project, point cloud space area project.

[0166] In one embodiment, the above-mentioned step S204 may specifically include: when the project type is the point cloud project, determining the partial project data corresponding to one or more blocks according to the sub-sample project characteristics associated with the point cloud project in the metadata data box, further determining the partial project data corresponding to one or more blocks according to the first sub-sample data type and the block identifier in the sub-sample project characteristics, and decoding the partial project data.

[0167] In one embodiment, the above-mentioned step S204 may specifically include: when the project type is the point cloud project, determining the partial project data corresponding to one or more spatial areas according to the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial area project characteristics; further, determining the partial project data corresponding to one or more spatial areas according to the second subsample data type and the spatial area identifier in the subsample project characteristics, wherein the 3D spatial area project characteristics include: the number of spatial areas, spatial area description information, the number of blocks contained in the spatial area, and the corresponding identifiers of the blocks contained in the spatial area, and the spatial area description information includes: the spatial area identifier, anchor point coordinates, length, width and height of the spatial area, and decoding the partial project data.

[0168] In one embodiment, the above step S204 may specifically include: when the project type is a point cloud block project, identifying the point cloud data containing the 3D static scene according to the full point cloud cluster group; determining the partial project data corresponding to one or more blocks according to the point cloud block set in the point cloud configuration project characteristics and one or more point cloud block items in the meta information data box, and decoding the partial project data. In one embodiment, the point cloud block project includes a single block described in the point cloud block set, wherein each of the one or more blocks represents a data subset that can be independently decoded in the point cloud spatial area.

[0169] In one embodiment, the above step S204 may specifically include: when the project type is a point cloud spatial region project, identifying the point cloud data containing the 3D static scene according to the full point cloud cluster group; determining partial project data corresponding to one or more point cloud spatial regions according to the one or more point cloud spatial region projects in the metadata data box and the spatial region description project characteristics associated with the point cloud spatial project, and decoding the partial project data, wherein each point cloud spatial region in the one or more point cloud spatial regions corresponds to a single spatial region described by the spatial region description project characteristics. In one embodiment, the spatial region description project characteristics include: a spatial region identifier, anchor point coordinates, and the length, width, and height of the spatial region.

[0170] In one embodiment, the point cloud full cluster group includes: the number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes: point cloud subsets divided according to blocks and point cloud subsets divided according to spatial areas.

[0171] In one embodiment, the above step S202 may specifically include: when the project type is a point cloud space region project, determining partial project data for indicating a marked scene according to a region tag group, and decoding the partial project data. The region tag group includes at least a tag type and a tag text, and the partial point cloud project indicating the marked scene is described by the tag type and the tag text.

[0172] In one embodiment, one or more media resources associated with the partial project data are determined and obtained. Specifically, one or more media resources associated with the partial project data can be determined and obtained according to a multi-entity playback group; the one or more media resources are decoded while decoding the partial project data; and the one or more media resources are played while rendering a partial spatial area of ​​the 3D scene according to the decoded data. In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0173] According to another embodiment of the present invention, a point cloud data processing method is also provided. Figure 3 The process of the point cloud data processing method according to an embodiment of the present invention is Figure 2 ,like Figure 3 As shown, the process includes the following steps:

[0174] Step S302, determining static geometrically coded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically coded point cloud data is represented by cloud project data;

[0175] In this embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and attribute project data are divided into multiple levels, and each level is composed of a group of evenly distributed point sets.

[0176] Step S304, decoding partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data;

[0177] Step S306: Rendering some level details of the 3D scene according to the decoded data.

[0178] Through the above steps S302 to S306, the problem that in the related technology, it is usually necessary to traverse the complete point cloud data before identifying the partial point cloud data required by the user, resulting in low parsing efficiency and long time consumption can be solved. The required partial project data is determined according to user needs, so that the required partial project data can be obtained without traversing the complete point cloud data, thereby improving the decoding efficiency, shortening the decoding time, and shortening the user's rendering waiting time.

[0179] In this embodiment, the point cloud project data is determined by the project type in the meta information data box, and the project type at least includes: point cloud project and point cloud level project.

[0180] In one embodiment, the above-mentioned step S304 may specifically include: when the project type is a point cloud project, determining partial project data corresponding to one or more levels according to sub-sample project characteristics associated with the point cloud project in the metadata data box; further, determining partial project data corresponding to one or more levels according to a third sub-sample data type and level value in the sub-sample project characteristics, and decoding the partial project data.

[0181] In one embodiment, the above-mentioned step S304 may specifically include: when the project type is a point cloud level project, identifying the point cloud data containing all the details of the 3D static scene according to the full point cloud cluster group; determining partial project data corresponding to one or more levels according to one or more point cloud level projects in the metadata data box, and level detail project characteristics associated with the point cloud level projects, and decoding the partial project data, wherein each level of the one or more levels corresponds to a single level described by the level detail project characteristics; the level detail project characteristics include at least: point cloud level value, level point number.

[0182] In one embodiment, the point cloud full cluster group includes: the number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes a point cloud subset divided according to hierarchical details.

[0183] In one embodiment, one or more media resources associated with the partial project data are determined and obtained. Specifically, one or more media resources associated with the partial project data can be determined and obtained based on a multi-entity playback group; the one or more media resources are decoded while decoding the partial project data; and the one or more media resources are played while rendering partial hierarchical details of the 3D scene based on the decoded data.

[0184] In this embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0185] In this embodiment, the ISO International Organization for Standardization (ISO) basic media file format can be used to describe 3D point cloud compressed data that does not contain time information, support the description of 3D point cloud spatial area information and hierarchical information, and further provide a partial access mechanism for point cloud data, so that the decoder can select part of the 3D point cloud data for parsing, decoding, and rendering based on the user's spatial position, viewing direction and other information, thereby improving the efficiency of static point cloud processing. Partial access to 3D point clouds includes at least the following two methods: partial access based on spatial regions; progressive access based on hierarchical details. In this embodiment, the items (Item) in the ISOBMFF media file format and the related data boxes are used to store static 3D point cloud data and metadata information required for partial access, which can meet the needs of partial access to 3D point clouds in various scenarios. The specific semantics are further described in the embodiments.

[0186] The implementation method of this embodiment is to store the spatial position information, hierarchical information, and association relationship information of the static 3D point cloud in the media file based on the ISO (basic media file format. The basic media file format can be operated with reference to the MPEG-4 Part 12 ISO Base Media File Format formulated by the ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG for short). Among them, the static point cloud compression data format can be operated with reference to the MPEG-I Part 9: Geometry-based Point Cloud Compression (G-PCC) based on the point cloud compression technology based on geometry coding formulated by the ISO / IEC JTC1 / SC29 / WG11 Moving Picture Experts Group (MPEG).

[0187] All data in ISOBMFF media files are contained in data boxes, and their types and sizes can be described in the data box header; if a box supports nesting (a box contains another sub-box), the data box is called a container box that supports nesting. Static data (such as images) in ISOBMFF media files are usually stored in the "item" format. Meta information data boxes can store project description information and metadata, and can also contain specific payloads. Meta information data boxes support the integrated storage of multiple static media objects, such as image collections.

[0188] Static point cloud data supports two basic packaging methods:

[0189] 1) Single-item: Figure 4 is a schematic diagram of a single-item storage structure of a static point cloud according to this embodiment. Figure 4 As shown in the figure, all the geometric data and attribute data of the G-PCC point cloud are encapsulated in a single project. Parameter information such as the sequence parameter set (SPS), attribute parameter set (APS), geometry parameter set (GPS) and other general metadata information such as the tile set (TileInventory) are described in the item property (Item Property), the type is "gpcc", and it is associated with the point cloud project (G-PCC Item).

[0190] 2) Multiple-item: Figure 5 is a schematic diagram of a static point cloud multi-project storage structure according to this embodiment. Figure 5 As shown in Figure 1, the geometric data and attribute data of the G-PCC point cloud are encapsulated in multiple items, that is, the complete point cloud contains a point cloud geometry item (G-PCC Geometry Item) and one or more point cloud attribute items (G-PCCAttribute Item), such as Figure 2 Point cloud parameter information such as SPS, APS, GPS, and general metadata information such as Tile Inventory are described in Item Property, and are associated with multiple items such as multiple geometric data and attribute data. The geometric items of the point cloud reference one or more attribute items through the Item Reference Box (iref) of type 'gpcg'.

[0191] The present embodiment supports two basic static point cloud encapsulation formats, namely, Single-item and Multiple-item. The Single-item encapsulation format is used as an example for extended definition in the further detailed description of the embodiment, which is only used to explain the present invention and is not used to limit the present invention.

[0192] Example 1: Partial access to static point cloud based on spatial region

[0193] This embodiment describes a method for partially accessing a static 3D point cloud based on spatial regions, and a method for describing point cloud spatial region information in a media file. For static G-PCC point cloud compressed data that supports partial access based on spatial regions, the point cloud data corresponding to each spatial region supports independent decoding and rendering.

[0194] In this embodiment scenario, four implementation methods are supported:

[0195] The complete point cloud data is stored in a single item. The tile division of the point cloud data is described by the bounding box information and sub-sample item property provided by the tile set. The terminal can flexibly select one or more tile sub-samples according to user needs to complete operations such as decoding and rendering.

[0196] The point cloud data is stored in multiple point cloud tile items (G-PCC Tile Items) according to different spatial regions, corresponding one to one with the regional divisions in the tile set. The terminal can select one or more tile items to complete decoding, rendering, etc.

[0197] The complete point cloud data is stored in a single item. The 3D region item property and sub-sample item property are used to describe the 3D region division of the point cloud data. The terminal can select one or more region sub-samples to complete decoding, rendering and other operations.

[0198] The point cloud data is stored in multiple point cloud spatial region items (G-PCC RegionItem) according to different spatial regions. Each region item describes the spatial region range through the spatial region item property (Region Description item property). The terminal can select one or more region items to complete decoding, rendering, etc.

[0199] The terminal parsing process includes:

[0200] 1) The terminal identifies the item containing static point cloud data in the meta information data box according to one of the following item types:

[0201] Point cloud projects: gpcc, gpcg, gpca;

[0202] Point cloud segmentation projects: gpct, gptg, gpta;

[0203] Point cloud spatial area projects: GPCR, GPRG, GPRA;

[0204] 2) The terminal reads the point cloud configuration item property (G-PCC Configuration ItemProperty) associated with the point cloud project, reads parameter information and configuration information such as SPS, APS, GPS, Tile Inventory, and completes decoder initialization;

[0205] 3) The terminal determines the spatial area information and data range of the static point cloud data in one of the following ways:

[0206] Method 1: Read the subsample item property. If flags is 1 and data_type is 0, read the G-PCC Tile data range and tile_id indicated by each sub-sample.

[0207] Method 2: Determine the G-PCC Tile data range and its tile_id based on the point cloud segmentation project;

[0208] Method 3: Read the 3D Region Item Property, parse the coordinates, range and other information of each 3D region, read the Subsample Item Property, if flags is 1 and data_type is 1, read the G-PCC 3D region data range and region_id indicated by each Sub-sample;

[0209] Method 4: Determine the G-PCC 3D region data range and its region_id based on the point cloud spatial region project;

[0210] 4) The terminal calculates according to the user demand and the spatial region information to determine one or more tiles or 3D regions to be decoded;

[0211] 5) The terminal reads part of the G-PCC data and inputs it into the decoder to complete decoding;

[0212] 6) The renderer renders a partial spatial area of ​​the 3D scene.

[0213] Method 1: Subsample describes the spatial region division

[0214] Figure 6 is a schematic diagram of the static point cloud storage based on Subsample according to this embodiment, such as Figure 6As shown in the figure, the complete point cloud data is stored in a single item, and the spatial area division of the point cloud data is described by the Sub-sample item property attribute information. The terminal obtains the partial G-PCC point cloud data according to the spatial area as a minimum of one G-PCC Tile. The tile spatial position, range and other information are described in the G-PCC Configuration Item Property. At the same time, the Subsample Item Property is used to describe the spatial area division of the point cloud data in the G-PCC Item. Taking the Single-item basic encapsulation format as an example, the storage structure based on Subsample is as follows: Figure 6 shown.

[0215] For the Multiple-item basic encapsulation format, the storage structure based on Subsample is Figure 6 Similar, the difference is that the static point cloud access entry is the G-PCC Geometry Item of type 'gpcg', and is associated with two project properties, G-PCCGeometry Item Property and Subsample Item Property. The G-PCCGeometry Item references one or more G-PCC Attribute Item of type 'gpca' through an Item Reference Box (iref) of type 'gpca', and each attribute item is associated with a Subsample Item Property.

[0216] Figure 7 is a schematic diagram of the subsample division based on the spatial region according to this embodiment, such as Figure 7 As shown, the G-PCC Subsample Item Property is as follows:

[0217]

[0218]

[0219] When Subsample Item Property indicates G-PCC Item spatial region division, flags is 1 and the codec_specific_parameters extended syntax is as follows:

[0220]

[0221] Semantics:

[0222] data_type, indicating the type of spatial region corresponding to a single sub-sample. A data_type of 0 indicates that the sub-sample corresponds to a G-PCC tile. A data_type of 0 indicates that the sub-sample corresponds to a 3D region, i.e., multiple G-PCC tiles.

[0223] tile_id, indicating the G-PCC tile identifier corresponding to the sub-sample, which is consistent with the tile identifier in the Tile Inventory;

[0224] 3D_region_id, indicating the G-PCC 3D region identifier corresponding to the sub-sample, which is consistent with the identifier in the 3D RegionItem Property;

[0225] Method 2: Tile Item describes the spatial area division

[0226] Figure 8 is a schematic diagram of static point cloud storage based on Tile Item according to this embodiment, such as Figure 8 As shown, this embodiment describes a method for accessing a static 3D point cloud part based on multiple items. The point cloud data of different spatial regions are stored in independent G-PCC Tile Items. Multiple Region Items are described as complete static 3D point clouds through the point cloud full cluster group data box (CompletePointCloudGroupBox). The group type is 'cppc'. At the same time, the group describes the mapping relationship between the Tile Item containing geometric data and the Tile_id in the Tile Inventory. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure based on Tile item is as follows: Figure 8 shown.

[0227] For the Multiple-item basic encapsulation format, the storage structure based on Tile Item is Figure 8 Similar, the difference is that the static point cloud access entry is a G-PCC Geometry Tile Item of type 'gptg', and is associated with the G-PCCConfiguration Item Property item property. The G-PCC Geometry Tile Item references one or more property items of type 'gpta' through an Item Reference Box (iref) of type 'gpca'.

[0228] G-PCC Tile Item

[0229] The 'gpct' type represents a G-PCC Tile Item, which consists of some G-PCC project data. Each Tile Item represents an independently decodable point cloud subset belonging to a certain block, and corresponds one-to-one with the Tile description in the Tile Inventory.

[0230] The 'gptg' type represents G-PCC Geometry Tile Item, which consists of G-PCC partial project data. Each item represents the point cloud geometry data belonging to a certain tile.

[0231] The 'gpta' type represents a G-PCC Attribute Tile Item, which consists of some G-PCC item data. Each item represents a type of point cloud attribute data belonging to a certain tile.

[0232]

[0233]

[0234] partial_gpcc_num, which indicates the maximum number of partial access subsets that can be divided in the complete point cloud object;

[0235] tile_enable, the value is 1, indicating that this group describes the combination relationship of Tile Item, otherwise the value is 0;

[0236] region_enable, a value of 1 indicates that this group describes the combination relationship of Region Item, otherwise the value is 0;

[0237] scalable_enable, a value of 1 indicates that this group describes the combination relationship of Layer Item, otherwise the value is 0;

[0238] entry_id, indicating the index of a subset G-PCC Item (Tile Item, Region Item or Layer Item) in EntityToGroupBox;

[0239] tile_id, describes the tile_id corresponding to the Tile Item;

[0240] 3DSpatialRegionStruct(), describes the spatial region information corresponding to the Region Item;

[0241] lod_value indicates the level of detail of the Layer Item data. A higher value means a higher level of the corresponding sub-sample data and richer details.

[0242] This implementation method is based on Tile Item to describe the spatial area division of the point cloud. In CompletePointCloudGroupBox, tile_enable is 1, and region_enable and scalable_enable are 0.

[0243] Method 3: 3D region property and Subsample to describe spatial region division

[0244] Fig. 9 is a schematic diagram of static point cloud storage based on 3D region property and Subsample according to this embodiment, such as Fig. 9 As shown in the figure, the terminal obtains some G-PCC point cloud data based on the spatial region, which is at least a G-PCC 3DRegion, that is, a spatial region. All 3D Region information is described by 3D Region Item Property, including the number of spatial regions, coordinates, range, etc. Tile spatial position, range and other information are described in G-PCC Configuration ItemProperty, and Subsample Item Property is used to describe the spatial region division of point cloud data in G-PCC Item. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure based on 3D Region and Subsample is as follows: Fig. 9 shown.

[0245] For the Multiple-item basic encapsulation format, the storage structure based on Subsample is Fig. 9 Similar, the difference is that the static point cloud access entry is the G-PCC Geometry Item of type 'gpcg', and is associated with three item properties: G-PCCConfiguration Item Property, Subsample Item Property, and 3D Region Item Property. The G-PCC Geometry Item references one or more attribute items of type 'gpca' through an item reference data box (ItemReference Box, iref) of type 'gpca'.

[0246]

[0247]

[0248] Semantics:

[0249] 3d_region_id, indicating the 3D spatial region identifier;

[0250] anchor_x, anchor_y, anchor_z, indicating the coordinates of the origin of the 3D space area;

[0251] region_dx, region_dy, region_dz, indicating the x, y, z direction range of the 3D space region;

[0252] num_regions, indicating the number of 3D spatial regions described in the Item Property;

[0253] num_tiles, indicating the number of G-PCC tiles contained in each 3D space;

[0254] tile_id, indicating the G-PCC tile identifier, which is consistent with the tile identifier in the Tile Inventory;

[0255] Method 4: Region Item describes spatial region division

[0256] Fig.10 is a schematic diagram of static point cloud storage based on Region Item according to this embodiment, such as Fig.10 As shown in the figure, the static 3D point cloud partial access method based on multiple items stores the point cloud data of different spatial regions in independent G-PCCRegion Items, and describes the entire spatial region division of the point cloud data through the 3D region item property information. Multiple Region Items containing geometric data are described as complete static 3D point clouds through CompletePointCloudGroupBox, and the group type is 'cppc'. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure based on Region Item is as follows: Fig.10 shown.

[0257] For the Multiple-item basic encapsulation format, the storage structure based on Region Item is Fig.10Similar, the difference is that the static point cloud access entry is a G-PCC Geometry Region Item of type 'gprg', and is associated with the G-PCCConfiguration Item Property item property. The G-PCC Geometry Region Item references one or more property items of type 'gpra' through an Item Reference Box (iref) of type 'gpca'.

[0258] G-PCC Region Item

[0259] The 'gpcr' type represents a G-PCC Region Item, which consists of some G-PCC project data. Each RegionItem represents a subset of independently decodable point clouds belonging to a certain spatial region. Each 'gpcr' type Item should be associated with a Regin Description Item to provide spatial region description information.

[0260] The 'gprg' type represents a G-PCC Geometry Region Item, which consists of some G-PCC item data. Each item represents the point cloud geometry data belonging to a certain spatial region.

[0261] The 'gpra' type represents a G-PCC Attribute Region Item, which consists of some G-PCC item data. Each item represents a type of point cloud attribute data belonging to a certain spatial region.

[0262]

[0263] The definition of CompletePointCloudGroupBox is the same as that of implementation mode 2. This implementation mode describes the spatial region division of the point cloud based on Region Item. In CompletePointCloudGroupBox, region_enable is 1, tile_enable and scalable_enable are 0.

[0264] Example 2: Partial access to static point cloud based on LoD (level of detail)

[0265] This embodiment describes a method for progressive access to static 3D point clouds based on level of details (LoD), as well as a method for describing point cloud level information in a media file. For static G-PCC point cloud compression data that supports progressive access, its geometric data and attribute data can be divided into multiple levels according to the geometric positions of the points, and each level is composed of a set of evenly distributed point sets. The higher the level, the denser the point set, and the richer the observable details of the point cloud objects; the lower the level, the sparser the point set, and the more blurred the observable details of the point cloud objects. After the terminal obtains the point cloud compressed data, it can select the appropriate level for decoding and rendering according to user needs, thereby ensuring the user experience while avoiding the waste of decoder resources caused by rendering unnecessary details.

[0266] In this embodiment, two implementation modes are supported:

[0267] 1) The complete point cloud data is stored in a single item. The hierarchical division of the point cloud data is described by the LoD-related parameters described in the SPS and APS parameter sets and the Sub-sample item property attribute information. The terminal can flexibly select one or more LoD Sub-samples according to user needs to complete decoding, rendering and other operations;

[0268] 2) The point cloud data is stored in multiple point cloud layer items (G-PCC Layer Items) according to different LoD levels. Each Layer Item describes the layer value and number of points through the layer detail item property (LoD Description item property). The terminal can select one or more Layer Items to complete decoding, rendering, etc.

[0269] The terminal parsing process includes:

[0270] 1) The terminal identifies the project containing static point cloud data in the metadata data box according to one of the following project types:

[0271] Point cloud projects: gpcc, gpcg, gpca;

[0272] Point cloud level projects: gpcl, gplg, gpla;

[0273] 2) The terminal reads the point cloud configuration item property (G-PCC Configuration ItemProperty) associated with the point cloud project, reads the parameter information and configuration information such as SPS, APS, GPS, etc., identifies the relevant parameter information such as the LoD division level, and completes the decoder initialization;

[0274] 3) The terminal determines the level detail information of the static point cloud data and the range of each level of point cloud data according to one of the following methods:

[0275] Method 1: Read Subsample information (Subsample Item Property), where flags is 2, and read the level value, number of points, and data range of the LoD single-level point cloud data indicated by each Sub-sample;

[0276] Method 2: Determine the point cloud data range, LoD level value, and number of points for each level based on the point cloud level item and LoD Description Item Property;

[0277] 4) The terminal performs calculations based on user requirements, LoD level values ​​and point information to determine one or more LoD levels to be decoded;

[0278] 5) The terminal reads the G-PCC data of some layers and inputs it into the decoder to complete decoding;

[0279] 6) The renderer renders a 3D static scene containing some details.

[0280] Method 1: Subsample describes the LoD level division

[0281] This embodiment describes a method for progressively accessing a static 3D point cloud based on hierarchical details, and a method for describing hierarchical information of a point cloud in a media file. For static G-PCC point cloud compression data that supports progressive access, the complete point cloud data is stored in a single item, and the hierarchical division of the point cloud data is described by the Sub-sample item property information. Taking the Single-item basic encapsulation format as an example, the multi-level static point cloud storage structure based on Subsample is as follows: Figure 6 shown.

[0282] For the Multiple-item basic encapsulation format, the storage structure based on Subsample is Figure 6 Similar, the difference is that the static point cloud access entry is the G-PCC Geometry Item of type 'gpcg', and is associated with two project properties, G-PCCConfiguration Item Property and Subsample Item Property. The G-PCCGeometry Item references one or more attribute items of type 'gpca' through an Item Reference Box (iref) of type 'gpca'.

[0283] Fig.11 is a schematic diagram of the subsample division based on the hierarchical information according to this embodiment, such as Fig.11 As shown, the G-PCC Subsample Item Property is as follows:

[0284] When Subsample Item Property indicates G-PCC Item spatial region division, flags is 2 and the codec_specific_parameters extended syntax is as follows:

[0285]

[0286] Semantics:

[0287] lod_value indicates the level of detail of the G-PCC sub-sample data. A higher value means a higher level of sub-sample data and richer details.

[0288] point_count, indicating the number of points contained in the G-PCC sub-sample;

[0289] Method 2: Layer Item describes the LoD level division

[0290] Fig.12 is a schematic diagram of static point cloud storage based on Layer Item according to this embodiment, such as Fig.12 As shown in the figure, this embodiment describes another method for accessing part of a static 3D point cloud based on hierarchical details. The point cloud data of different spatial regions are stored in independent G-PCC Layer Items. The layer value, number of points, and other information of the Layer Item are described by the LoD Description Item Property attribute information. Multiple Layer Items containing geometric data are combined into a complete static 3D point cloud through the type 'gpcc'. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure based on Layer Item is as follows: Fig.12 shown.

[0291] For the Multiple-item basic encapsulation format, the storage structure based on Layer Item is Fig.12Similar, the difference is that the static point cloud access entry is a G-PCC Geometry Layer Item of type 'gplg', and is associated with the G-PCCConfiguration Item Property project property. The G-PCC Geometry Layer Item references one or more property items of type 'gpla' through an Item Reference Box (iref) of type 'gpca'.

[0292] G-PCC Layer Item

[0293] The 'gpcl' type represents a G-PCC Layer Item, which consists of some G-PCC project data. Each Layer Item represents a subset of independently decodable point clouds belonging to a certain layer. Each 'gpcl' type Item should be associated with the LoDDescription Item property to provide description information such as the layer value and number of points of the layer.

[0294] The 'gplg' type represents a G-PCC Geometry Layer Item, which consists of some G-PCC project data. Each Item represents point cloud geometry data belonging to a certain layer.

[0295] The 'gpla' type represents the G-PCC Attribute Layer Item, which consists of some G-PCC project data. Each Item represents a type of point cloud attribute data belonging to a certain level.

[0296]

[0297] Semantics:

[0298] lod_value indicates the level of detail of the G-PCC sub-sample data. The higher the value, the higher the level of the corresponding sub-sample data and the richer the details.

[0299] point_count, indicating the number of points contained in the G-PCC sub-sample;

[0300] CompletePointCloudGroupBox

[0301] The definition of CompletePointCloudGroupBox is the same as that of Example 1. This implementation method is based on LayerItem to describe the hierarchical division of the point cloud. In CompletePointCloudGroupBox, scalable_enable is 1, and tile_enable and region_enable are 0.

[0302] Example 3: Partial access combining spatial regions and LoD

[0303] This embodiment describes a method for partially accessing a static 3D point cloud based on spatial regions and level details, wherein different spatial regions can be divided into different numbers of detail levels according to different point cloud densities. Fig.13 is a schematic diagram of static point cloud storage based on spatial regions and hierarchical details according to this embodiment. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure based on spatial regions and hierarchical details is as follows: Fig.13 As shown in the figure, the point cloud data of different spatial regions are stored in independent G-PCC Region Items. The hierarchical information of the point cloud data of each region is described by the Subsample Item Property attribute information. The three G-PCC Region Items correspond to 3-layer, 8-layer, and 12-layer hierarchical divisions respectively. Multiple Region Items are combined into a complete static 3D point cloud through the type 'gpcc'.

[0304] Example 4: Partial access to a static point cloud based on the author’s intent

[0305] This embodiment also provides a method for accessing part of a static 3D point cloud based on the creator's intention. For static 3D point clouds such as cities, streets, and large buildings, content creators can mark the point cloud data therein during the 3D point cloud acquisition, encoding, and packaging stages, such as the floor division and room division of large buildings, the block division of street scenes, etc. In addition, static 3D point cloud scenes can be displayed in association with other media objects, such as providing different audio explanations for different areas of street scenes or digital museums, and different characters corresponding to different floors in large buildings, and other dynamic point cloud objects.

[0306] For the above two scenarios, this implementation can be divided into two implementation methods:

[0307] Area tag, that is, a combination of multiple items;

[0308] Static point clouds are displayed in association with dynamic media content, i.e., a combination of Item and Track.

[0309] Method 1: Area Marking

[0310] Fig.14 is a schematic diagram of a static point cloud spatial region marking according to this embodiment. In this embodiment, point cloud data subsets of multiple different regions are combined through a region marking group data box (RegionMarkingGroupBox), and content description information is provided. Taking the Single-item basic encapsulation format as an example, the static point cloud storage structure supporting spatial region marking is as follows: Fig.14 As shown, the complete point cloud object can be divided into 5 Region Items according to different spatial regions, corresponding to two room scenes respectively, marked by an entity group data box (EntitytoGroupBox) of type 'pcrm', and describing the two room scene information respectively.

[0311] RegionMarkingGroupBox

[0312] Box Types:'pcrm'

[0313] Container:GroupsListBox

[0314] Mandatory:No

[0315] Quantity:Zero or more

[0316] aligned(8)class RegionMarkingGroupBox extends EntityToGroupBox('pcrm'){

[0317] unsigned int(8)region_type;

[0318] string region_description;

[0319] }

[0320] Semantics:

[0321] region_type, provides the partial project data tag type corresponding to the EntitytoGroup group;

[0322] region_description, a null-terminated UTF-8 string providing the textual description of the point cloud marker;

[0323] Method 2: Static point cloud and dynamic point cloud co-rendering

[0324] Fig.15is a schematic diagram of the associated storage of static point clouds and dynamic point clouds according to this embodiment. Multiple point cloud data subsets in different regions are combined through a multiple entity playout group data box (MultipleEntityPlayoutGroupBox), and content description information is provided. Taking the Single-item basic encapsulation format as an example, the associated storage structure of static point clouds and dynamic point clouds is as follows: Fig.15 shown.

[0325] The static point cloud objects contained in the Meta box can be divided into G-PCC RegionItem according to different spatial regions: Region 1 and Region 2. G-PCC Region 1 is associated with the dynamic point cloud objects in the movie box (Movie box), and the association information is described through MultipleEntityPlayoutGroupBox, indicating that G-PCC Region 1 needs to be displayed together with the point cloud data contained in track 1. G-PCC Region 2 is associated with the dynamic point cloud objects and audio in the Movie box, and the association information is described through MultipleEntityPlayoutGroupBox, indicating that G-PCC Region 2 needs to be displayed together with the point cloud data contained in track 2 and the audio data contained in track 3.

[0326]

[0327] According to another embodiment of the present invention, a point cloud data processing device is provided. Fig.16 is a frame of the point cloud data processing device according to this embodiment Figure 1 ,like Fig.16 As shown, the device comprises:

[0328] A first determination module 162 is used to determine static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data;

[0329] A first decoding module 164 is used to decode partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data;

[0330] The first rendering module 166 is configured to render a partial spatial area of ​​the 3D scene according to the decoded data.

[0331] In one embodiment, the point cloud project data is obtained through a project type in a metadata data box, wherein the project type includes at least: a point cloud project, a point cloud block project, and a point cloud space area project.

[0332] In one embodiment, the first determining module includes:

[0333] The first determination submodule is used to determine the partial project data corresponding to one or more blocks according to the subsample project characteristics associated with the point cloud project in the metadata data box, and decode the partial project data when the project type is the point cloud project.

[0334] In one embodiment, the first determining submodule is further configured to:

[0335] Partial project data corresponding to one or more chunks are determined according to the first sub-sample data type and the chunk identifier in the sub-sample project characteristics.

[0336] In one embodiment, the first determining module includes:

[0337] The second determination submodule is used to determine the partial project data corresponding to one or more spatial areas according to the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial area project characteristics, and decode the partial project data when the project type is the point cloud project.

[0338] In one embodiment, the second determining submodule is further used to

[0339] The portion of the project data corresponding to one or more spatial regions is determined based on the second sub-sample data type and the spatial region identifier in the sub-sample project characteristics.

[0340] In one embodiment, the 3D space region project characteristics include:

[0341] The number of spatial regions, description information of the spatial regions, the number of blocks contained in the spatial regions, and the corresponding identifiers of the blocks contained in the spatial regions.

[0342] In one embodiment, the spatial region description information includes: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0343] In one embodiment, the first determining module includes:

[0344] A first identification submodule is used to identify point cloud data containing the 3D static scene according to the point cloud full cluster group when the project type is a point cloud block project;

[0345] The third determination submodule is used to determine the partial project data corresponding to one or more blocks according to the point cloud block set in the point cloud configuration project characteristics and one or more point cloud block items in the meta-information data box, and decode the partial project data.

[0346] In one embodiment, the point cloud tile project includes a single tile described in a point cloud tile set, wherein each of the one or more tiles represents an independently decodable subset of data in a point cloud spatial region.

[0347] In one embodiment, the first determining module includes:

[0348] A second identification submodule is used to identify point cloud data containing the 3D static scene according to the point cloud full cluster group when the project type is a point cloud space area project;

[0349] The fourth determination submodule is used to determine partial project data corresponding to one or more point cloud space areas according to one or more point cloud space area items in the metadata data box and the spatial area description project characteristics associated with the point cloud space items, and decode the partial project data.

[0350] In an embodiment, each of the one or more point cloud spatial regions corresponds to a single spatial region of the spatial region description item property description.

[0351] In one embodiment, the spatial region description item characteristics include: a spatial region identifier, anchor point coordinates, and length, width, and height of the spatial region.

[0352] In one embodiment, the point cloud full cluster group includes:

[0353] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes: a point cloud subset divided according to blocks, and a point cloud subset divided according to spatial regions.

[0354] In one embodiment, the first determining module includes:

[0355] The fifth determination submodule is used to determine partial project data for indicating a marked scene according to the area marking group when the project type is a point cloud space area project, and decode the partial project data.

[0356] In one embodiment, the area marking group includes at least a marking type and a marking text, and the partial point cloud items indicating the marking scene are described by the marking type and the marking text.

[0357] In one embodiment, the device further comprises:

[0358] A first resource determination module, used to determine and obtain one or more media resources associated with the portion of project data;

[0359] A first decoding resource module, configured to decode the one or more media resources while decoding the portion of project data;

[0360] The first playback module is configured to play the one or more media resources while rendering a partial spatial area of ​​the 3D scene according to the decoded data.

[0361] In one embodiment, the first resource determination module is further configured to:

[0362] One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

[0363] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0364] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data.

[0365] According to another embodiment of the present invention, a point cloud data processing device is provided. Fig.17 is a frame of the point cloud data processing device according to this embodiment Figure 2 ,like Fig.17 As shown, the device comprises:

[0366] A second determination module 172 is used to determine static geometrically encoded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by cloud project data;

[0367] A second decoding module 174 is used to decode partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data;

[0368] The second rendering module 176 is used to render some level details of the 3D scene according to the decoded data.

[0369] In one embodiment, the point cloud project data is determined by a project type in a metadata data box, and the project type includes at least: a point cloud project and a point cloud level project.

[0370] In one embodiment, the second determining module includes:

[0371] The sixth determination submodule is used to determine partial project data corresponding to one or more levels according to subsample project characteristics associated with the point cloud project in the meta-information data box when the project type is a point cloud project, and decode the partial project data.

[0372] In one embodiment, the sixth determining submodule is further used to

[0373] Partial item data corresponding to one or more levels are determined according to the third sub-sample data type and the level value in the sub-sample item characteristic.

[0374] In one embodiment, the second determining module includes:

[0375] A second identification submodule is used for identifying point cloud data containing all details of the 3D static scene according to the point cloud full cluster group when the project type is a point cloud level project;

[0376] The seventh determination submodule is used to determine partial project data corresponding to one or more levels based on one or more point cloud level items in the meta-information data box and level detail project characteristics associated with the point cloud level items, and decode the partial project data.

[0377] In an embodiment, each of the one or more levels corresponds to a single level of level-detail item property description.

[0378] In one embodiment, the level detail item characteristics include at least: point cloud level value, level point number.

[0379] In one embodiment, the point cloud full cluster group includes:

[0380] The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of the point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes a point cloud subset divided according to the level of detail.

[0381] In one embodiment, the device further comprises:

[0382] A second resource determination module, used to determine and obtain one or more media resources associated with the portion of project data;

[0383] A second decoding resource module, configured to decode the one or more media resources while decoding the portion of project data;

[0384] The second playback module is configured to play the one or more media resources while rendering some level details of the 3D scene according to the decoded data.

[0385] In one embodiment, the second resource determination module includes:

[0386] An eighth determination submodule is used to determine and obtain one or more media resources associated with the partial project data according to the multi-entity playback group.

[0387] In one embodiment, the media resources include at least: video, audio, and dynamic point cloud.

[0388] In one embodiment, the point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and attribute project data are divided into multiple levels, and each level is composed of a group of evenly distributed point sets.

[0389] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0390] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0391] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0392] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0393] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0394] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0395] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0396] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A point cloud data processing method, characterized in that: The method comprises: Determine static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data, and the point cloud project data is determined by a project type in a meta-information data box, wherein the project type includes at least: a point cloud project, a point cloud block project, and a point cloud spatial region project; Decoding partial item data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data; A partial spatial area of ​​the 3D scene is rendered according to the decoded data.

2. The method according to claim 1, characterized in that The decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud project, the partial project data corresponding to one or more blocks are determined according to the subsample project characteristics associated with the point cloud project in the meta information data box, and the partial project data are decoded.

3. The method according to claim 2, characterized in that Determining the partial project data corresponding to one or more blocks according to the subsample project characteristic data box associated with the point cloud project in the meta information data box includes: Partial project data corresponding to one or more chunks are determined according to the first sub-sample data type and the chunk identifier in the sub-sample project characteristics.

4. The method according to claim 1, characterized in that: The decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud project, the partial project data corresponding to one or more spatial areas is determined based on the subsample project characteristics associated with the point cloud project in the metadata data box and the 3D spatial area project characteristics, and the partial project data is decoded.

5. The method according to claim 4, characterized in that Determining the portion of project data corresponding to one or more spatial regions according to the subsample project characteristics associated with the point cloud project in the meta information data box and the 3D spatial region project characteristics comprises: The portion of the project data corresponding to one or more spatial regions is determined based on the second sub-sample data type and the spatial region identifier in the sub-sample project characteristics.

6. The method according to claim 4, characterized in that The 3D spatial area project features include: The number of spatial regions, description information of the spatial regions, the number of blocks contained in the spatial regions, and the corresponding identifiers of the blocks contained in the spatial regions.

7. The method according to claim 6, characterized in that The spatial region description information includes: a spatial region identifier, anchor point coordinates, and the length, width, and height of the spatial region.

8. The method according to claim 1, characterized in that The decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud segmentation project, identifying point cloud project data containing the 3D scene according to the point cloud full cluster group; According to the point cloud block set in the point cloud configuration item characteristics and one or more point cloud block items in the meta-information data box, the partial project data corresponding to one or more blocks are determined, and the partial project data are decoded.

9. The method according to claim 8, characterized in that The point cloud tile project contains a single tile described in a point cloud tile set, wherein each of the one or more tiles represents an independently decodable subset of data in a point cloud spatial region.

10. The method according to claim 1, characterized in that The decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud spatial area project, identifying point cloud data containing a 3D static scene according to the point cloud full cluster group; According to one or more point cloud spatial area items in the metadata data box, the spatial area description item characteristics associated with the point cloud spatial items are determined, and the partial project data corresponding to the one or more point cloud spatial areas are decoded.

11. The method according to claim 10, characterized in that Each of the one or more point cloud spatial regions corresponds to a single spatial region of the spatial region description item property description.

12. The method according to claim 10, characterized in that The spatial region description item characteristics include: a spatial region identifier, anchor point coordinates, and the length, width, and height of the spatial region.

13. The method according to any one of claims 8 to 12, characterized in that The point cloud full cluster group includes: The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes: a point cloud subset divided according to blocks, and a point cloud subset divided according to spatial regions.

14. The method according to claim 1, characterized in that The decoding of partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data includes: In case that the project type is the point cloud space area project, the partial project data for indicating the marking scene is determined according to the area marking group, and the partial project data is decoded.

15. The method according to claim 14, characterized in that The area marking group includes at least a marking type and a marking text, and the partial point cloud items indicating the marking scene are described by the marking type and the marking text.

16. The method according to any one of claims 1 to 12, 14 to 15, characterized in that The method further comprises: Determining and obtaining one or more media resources associated with the portion of project data; While decoding the portion of the project data, decoding the one or more media resources; The one or more media resources are played while rendering a partial spatial area of ​​the 3D scene according to the decoded data.

17. The method according to claim 16, characterized in that The determining and obtaining one or more media resources associated with the portion of project data includes: One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

18. The method according to claim 16, characterized in that The media resources include at least: video, audio, and dynamic point cloud.

19. The method according to any one of claims 1 to 12, 14, 15, 17 and 18, characterized in that The point cloud project data is composed of geometric project data and one or more attribute project data.

20. A point cloud data processing method, characterized in that: The method comprises: Determine static geometrically encoded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data, and the point cloud project data is determined by a project type in a meta information data box, and the project type includes at least: a point cloud project and a point cloud hierarchical project; Decoding partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data; Rendering partial level details of the 3D scene according to the decoded data.

21. The method according to claim 20, characterized in that The decoding of partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud project, the partial project data corresponding to one or more levels are determined according to the subsample project characteristics associated with the point cloud project in the meta information data box, and the partial project data are decoded.

22. The method according to claim 21, characterized in that Determining the partial project data corresponding to one or more levels according to the subsample project characteristic data box associated with the point cloud project in the meta information data box includes: The portion of the item data corresponding to one or more levels is determined according to the third sub-sample data type and the level value in the sub-sample item characteristic.

23. The method according to claim 20, characterized in that The decoding of partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data includes: In the case where the project type is the point cloud level project, identifying point cloud project data containing all details of the 3D static scene according to the point cloud full cluster group; Determine the partial project data corresponding to one or more levels according to one or more point cloud level items in the meta information data box and level detail project characteristics associated with the point cloud level items, and decode the partial project data.

24. The method according to claim 23, characterized in that Each of the one or more levels corresponds to a single level of level-detail item property description.

25. The method according to claim 23, characterized in that The hierarchical detail item characteristics include at least: point cloud hierarchical value and hierarchical point number.

26. The method according to any one of claims 23 to 25, characterized in that The point cloud full cluster group includes: The number of point cloud subsets accessed by the most divided part of the complete point cloud object, the type of the point cloud subset, and the identifier of the point cloud subset, wherein the point cloud subset includes a point cloud subset divided according to the level of detail.

27. The method according to any one of claims 20 to 25, characterized in that The method further comprises: Determining and obtaining one or more media resources associated with the portion of project data; While decoding the portion of the project data, decoding the one or more media resources; The one or more media resources are played while rendering some level details of the 3D scene according to the decoded data.

28. The method according to claim 27, characterized in that The determining and obtaining one or more media resources associated with the portion of project data includes: One or more media resources associated with the portion of project data are determined and acquired according to the multi-entity playback group.

29. The method according to claim 28, characterized in that The media resources include at least: video, audio, and dynamic point cloud.

30. The method according to any one of claims 20 to 25, 28 to 29, characterized in that The point cloud project data is composed of geometric project data and one or more attribute project data. The geometric project data and the attribute project data are divided into multiple levels, and each level is composed of a group of evenly distributed point sets.

31. A point cloud data processing device, characterized in that: The device comprises: A first determination module is used to determine static geometrically encoded point cloud data containing spatial region information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data, and the point cloud project data is determined by a project type in a meta information data box, wherein the project type includes at least: a point cloud project, a point cloud block project, and a point cloud spatial region project; A first decoding module, used for decoding partial project data corresponding to a partial spatial area of ​​the 3D scene in the static geometrically encoded point cloud data; The first rendering module is used to render a partial spatial area of ​​the 3D scene according to the decoded data.

32. A point cloud data processing device, characterized in that: The device comprises: A second determination module is used to determine static geometrically encoded point cloud data containing hierarchical detail information of a 3D scene, wherein the static geometrically encoded point cloud data is represented by point cloud project data, and the point cloud project data is determined by a project type in a meta information data box, and the project type at least includes: point cloud project, point cloud hierarchical project; A second decoding module is used to decode partial item data corresponding to partial level details of the 3D scene in the static geometrically encoded point cloud data; The second rendering module is used to render some level details of the 3D scene according to the decoded data.

33. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 19 and 20 to 30 when executed.

34. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method described in any one of claims 1 to 19 and 20 to 30.

Images