Method for encapsulating and transmitting point cloud data

The method of encapsulating point cloud data by data type classification addresses inefficiencies in transmission, enhancing efficiency and flexibility, and enabling selective data transmission over heterogeneous networks.

JP7577357B2Active Publication Date: 2024-11-05SHANGHAI JIAOTONG UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022542466
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-01-10
Filing Date
2021-01-08
Publication Date
2024-11-05
Estimated Expiration
2041-01-08

AI Technical Summary

Technical Problem

Existing multimedia transmission technologies face inefficiencies in transmitting point cloud data over heterogeneous networks, leading to low media content transmission efficiency and wasted storage space, particularly when handling complex and diverse point cloud data types.

Method used

A method for encapsulating point cloud media files by setting classification records based on data types, allowing for flexible transmission of partial data and enabling efficient bandwidth usage by identifying and transmitting only the necessary data segments.

Benefits of technology

Enhances transmission efficiency and flexibility over heterogeneous networks, reduces bandwidth requirements, and allows users to selectively acquire media content, improving user experience and network service management.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577357000006
    Figure 0007577357000006
  • Figure 0007577357000007
    Figure 0007577357000007
  • Figure 0007577357000008
    Figure 0007577357000008
Patent Text Reader

Abstract

The present invention provides a point cloud data encapsulation method and transmission method, which compresses a point cloud media file, encapsulates the compressed point cloud media file, and sets a classification record according to the data type, where the classification record is used to represent the data type of each part in the point cloud media file. The present invention solves the problems of low media content transmission efficiency and wasted storage space in point cloud data transmission, increases the flexibility of media content combination, strengthens management and control over multimedia services, gives users the right to independently select and obtain more media content, provides diverse application scenarios, and also provides network operators with a better multimedia content delivery and transmission scheme.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of multimedia transmission technology, and more particularly to a method for encapsulating and transmitting point cloud data. [Background technology]

[0002] In recent years, media transmission technology has been rapidly developing and improving, the cost of transmitting image and video information has decreased while the accuracy has improved, and the maturity of related application fields has been further promoted. 3D point cloud data is a new image media data, which can provide users with more accurate and vivid image information, and bring them a more realistic and interactive communication experience.

[0003] Point cloud data is the surface information of an object obtained by scanning, including three-dimensional coordinate data, depth information, color information, etc. With the maturity of 3D scanning technology and the improvement of system architecture, the amount of point cloud data has increased significantly while improving the accuracy of point cloud data. Among them, there are different point cloud data compression algorithms for different types of point cloud data, such as static and dynamic point cloud data, as well as machine sensing and human eye sensing. For example, for dynamic human eye sensing point cloud data, a typical point cloud compression algorithm converts 3D point cloud data into 2D image data and then processes the data, one of which is the video-based point cloud compression (VPCC) algorithm. In this compression method, the 3D point cloud is first projected onto a 2D plane to obtain occupancy map information, geometric information, attribute information and auxiliary information. The attribute information usually includes texture information and color information, so the compressed information is also usually divided into four types of data for transmission.

[0004] It can be seen that when transmitting VPCC data, the information to be processed mainly includes four kinds of data: geometric information, attribute information, occupancy map information and auxiliary information. The decoding of geometric information depends on the occupancy map information and auxiliary information, and the decoding of attribute information depends on the geometric information, occupancy map information and auxiliary information. Compared with traditional media data, point cloud media data is more complex and diverse, and different types of data need to be simultaneously processed and matched before the media with rich space and texture characteristics is presented to the user. At present, there has been some progress in the research of encapsulating and transmitting 3D point cloud data by existing multimedia transmission protocols, but with the development of network transmission technology, the heterogeneous network environment is becoming increasingly complex, and the traditional technology has poor transmission effect of point cloud data in heterogeneous network environment. Summary of the Invention [Problem to be solved by the invention]

[0005] In view of the shortcomings of the prior art, it is an object of the present invention to provide a method for encapsulating and transmitting point cloud data. [Means for solving the problem]

[0006] The method for encapsulating point cloud data according to the present invention includes the steps of: Compressing the point cloud media files; encapsulating the compressed point cloud media file and setting classification records according to data types, the classification records being used to represent the data type of each portion in the point cloud media file.

[0007] Preferably, the point cloud media file includes multiple point cloud contents, and a waiting flag is set in the encapsulated point cloud media file, and the waiting flag is used to indicate whether the media resources of all data types of each point cloud content have already been transmitted.

[0008] Preferably, the classification record is a classification list representing a correspondence between media segments and media versions and data types; encapsulating the compressed point cloud media file and setting classification records according to data type, encapsulating the compressed point cloud media file into a media resource, the media resource including a plurality of media segments; and associating a plurality of the media segments with a media version and a data type based on the classification list.

[0009] Preferably, the multiple media segments have different media segment numbers, and the classification list includes a type list, a content list and a segment number list, the type list corresponding to a data type of the point cloud media file, the content list being a collection of multiple media versions, and the segment number list being a collection of all media segment numbers, each data type corresponding to at least one media version, and each media version corresponding to at least one media segment number.

[0010] Preferably, the classification record is a classification flag representing a correspondence between a media segment and a media version and a data type; encapsulating the compressed point cloud media file and setting classification records according to data type, encapsulating the compressed point cloud media file into a media resource, the media resource including a plurality of media segments; and adding one of the classification flags to each media segment.

[0011] Preferably, if different media segments have the same media version and the same data type, the classification flags added are the same.

[0012] Preferably, the classification flag comprises two flag bits, one flag bit is used to define a data type of the media segment and another flag bit is used to define a version of the media segment.

[0013] Preferably, the classification record is a classification list representing the correspondence between media resources and data types; encapsulating the compressed point cloud media file and setting classification records according to data type, encapsulating the compressed point cloud media file to obtain a plurality of media resources; and associating a plurality of the media resources with data types based on the classification list.

[0014] Preferably, the multiple media resources have different media resource numbers, the classification list includes a media resource list and a resource number list, the media resource list corresponds to a data type of a point cloud media file, the resource number list is a collection of all media resource numbers, each media resource list corresponds to at least one media resource number, and the number of media resource numbers corresponding to each media resource list is consistent with the media version.

[0015] A method for transmitting point cloud data according to the present invention includes the steps of: Adding the relevant content of a classification record to the signaling information, the classification record being used to represent the data type of each part in the point cloud media file during encapsulation; Obtaining the classification record according to signaling information; and transmitting partial data that needs to be updated of the current point cloud media file according to the classification record; The method further includes transmitting all data of the current point cloud media file before transmitting the partial data of the current point cloud media file that needs to be updated.

[0016] Preferably, the point cloud media file includes multiple point cloud contents, and a waiting flag is set in the encapsulated point cloud media file, and the waiting flag is used to indicate whether the media resources of all data types of each point cloud content have already been transmitted.

[0017] Preferably, the compressed point cloud media file is encapsulated, and a classification record is set according to the data type, and the classification record is a classification list representing the correspondence between the media resource and the data type.

[0018] Preferably, the step of encapsulating the compressed point cloud media file and setting classification records according to data type comprises: Encapsulating the compressed point cloud media files as a set of media resources to obtain the multiple media resources; and and associating a plurality of the media resources with data types based on the classification list.

[0019] Preferably, the multiple media resources have different media resource numbers, the classification list includes a media resource list and a resource number list, the media resource list corresponds to a data type of a point cloud media file, the resource number list is a collection of all media resource numbers, each media resource list corresponds to at least one media resource number, and the number of media resource numbers corresponding to each media resource list is consistent with the media version.

[0020] Preferably, the classification record is a classification list representing a correspondence between media segments and media versions and data types; The step of adding related content of a classification record to the signaling information includes: Defining the classification list as a DTG information file, and writing the DTG information file into signaling information.

[0021] Preferably, the DTG information file is written to an MPI table, or one descriptive descriptor is added to describe the DTG information file, or one signaling table is added to describe the DTG information file.

[0022] Preferably, the classification record is a classification list representing the correspondence between media resources and data types; The step of adding related content of a classification record to the signaling information includes: The classification list is defined as an ACR information file, and the ACR information file is written into signaling information.

[0023] Preferably, the ACR information file is written to an MPI table, or a descriptive descriptor is added to describe the ACR information file, or ACR Add one signaling table to describe the information file.

[0024] Preferably, the classification record is a classification flag indicating a correspondence between a media segment and a media version and a data type; The step of adding related content of a classification record to the signaling information includes: A reserved field is used to define one classification flag for each media segment.

[0025] Preferably, the classification flag includes two flag bits, one flag bit is used to define a data type of the media segment and another flag bit is used to define a version of the media segment, and if the version of the media segment is 1, the media segment only has a flag bit for defining the data type of the media segment.

[0026] Preferably, one flag bit is defined using 2 bits of a reserved field in the MPU Box to indicate the data type of the media segment, and another flag bit is defined using 3 bits of a reserved field in the MPU Box to indicate the version of the media segment. Effect of the Invention

[0027] Compared with the prior art, the present invention has the following beneficial effects: Based on the conventional point cloud data processing method, the present invention sets classification records in point cloud media files according to data types, making it easy to identify different data of point cloud data according to the classification records, thereby making it easy to flexibly transmit partial data of point cloud data according to application needs, and does not need to transmit all point cloud data, so as to meet application needs while saving bandwidth, making it easy to transmit point cloud data more flexibly and efficiently, and also enabling flexible transmission of point cloud data over heterogeneous networks. Therefore, the present invention can solve the problems of low transmission efficiency of media content and wasted storage space in point cloud data transmission, increase the flexibility of media content combination, strengthen management and control of multimedia services, give users the right to independently select and obtain more media content, provide diverse application scenarios, and provide network operators with a better multimedia content provision and transmission scheme. [Brief description of the drawings]

[0028] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of the non-limiting embodiments with reference to the following figures.

[0029] [Figure 1] 1 is a flowchart illustrating encapsulation of point cloud data according to an embodiment of the present invention. [Diagram 2] 1 is a flowchart illustrating encapsulation of point cloud data according to the first embodiment of the present invention. [Diagram 3] 1 shows definition contents of a DTG information file according to a first embodiment of the present invention. [Figure 4] 11 is a flowchart illustrating encapsulation of point cloud data according to the second embodiment of the present invention. [Diagram 5] FIG. 11 is a schematic diagram illustrating adding a flag bit to a media segment according to Example 2 of the present invention. [Figure 6] 13 is a flowchart illustrating encapsulation of point cloud data according to the third embodiment of the present invention. [Figure 7] 13 is a definition content of an ACR information file according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] The present invention will be described in detail below with reference to specific embodiments. The following examples are useful for those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make some modifications and improvements without departing from the spirit of the present invention. All of these fall within the scope of protection of the present invention.

[0031] The inventors of the present application have found that since there may be interactions with users during the presentation of point cloud media content, it is necessary to modify some information of the point cloud. For example, the operation of changing the clothes of a character based on point cloud media usually requires modifying attribute information such as texture and color of the point cloud. In this case, the conventional technology usually transmits all the point cloud data to the client, which not only increases the bandwidth but also reduces the transmission efficiency, affecting the user experience.

[0032] Regarding the above problem, the present invention puts forward a point cloud data encapsulation method, which, as shown in FIG. 1, includes the following steps: Step S01: Compress the point cloud media file. Step S02: Encapsulate the compressed point cloud media file and set a classification record according to the data type, where the classification record is used to represent the data type of each part in the point cloud media file.

[0033] By providing a classification record, the present invention makes it easy to distinguish between different data of the point cloud data according to the classification record when presenting the point cloud content later, thereby enabling adaptive transmission and presentation of the point cloud data.

[0034] At the same time, the present invention further provides a point cloud data transmission method based on the point cloud data encapsulation method of the present invention, which includes the following steps: The relevant contents of the classification records mentioned above are added to the signaling information. Obtain a classification record according to the signaling information, and transmit the part data of the current point cloud media file that needs to be updated according to the classification record, where the whole data of the current point cloud media file is further included before transmitting the part data of the current point cloud media file that needs to be updated.

[0035] Based on the above encapsulation method and transmission method, when the point cloud content is presented for the first time, the complete point cloud data needs to be obtained in order to present the point cloud content. After the point cloud content is presented, by simply performing an operation such as changing clothes, the data that needs to be updated in the current point cloud media file can be selectively transmitted according to the classification record, without the need to transmit all the information.

[0036] In order to make the above objectives, features and advantages of the present invention more clearly understandable, specific embodiments of the present invention are described in detail below with reference to the drawings.

[0037] Example 1 In this embodiment, a classification record is created by a classification list, which represents the correspondence between media segments in a point cloud media file and media versions and data types. More specifically, the compressed point cloud media file is encapsulated as one media asset and divided into multiple media segments, such as the 16 media segments shown in FIG. 3. To Each media segment has a media segment number, which is numbered sequentially, for example, from 1 to 16. It can be seen that each media segment number is different, that is, each media segment is represented by its media segment number.

[0038] 2 is a flowchart of encapsulating point cloud data according to the present embodiment. As shown in FIG. 2, after obtaining the compressed point cloud media file, the method further includes the following steps: Step S11: Encapsulate the compressed point cloud media file as one media resource to obtain multiple media segments. Step S12: According to the classification list, associate multiple media segments with versions and data types. The term "association" refers to establishing a correspondence between data types and media versions and media segment numbers, and the same definition is used for the associations that appear below.

[0039] FIG. 3 is the definition content of a classification list according to this embodiment. As shown in FIG. 3, the classification list includes a type list, a content list and a segment number list. The type list corresponds to the data type of the point cloud media file. The content list is a collection of multiple media versions. The segment number list is a collection of all media segment numbers. Furthermore, each data type corresponds to at least one media version, and each media version corresponds to at least one media segment number. In this embodiment, the number of type lists corresponds to the data type of the point cloud media file, and one media version is one application scene.

[0040] Continuing to refer to FIG. 3, take the video-based point cloud compression algorithm (VPCC) as an example, after a point cloud media file is processed, four kinds of data are obtained, which are occupancy map information, geometry information, attribute information and auxiliary information. At this time, when encapsulating, a media resource is divided to obtain multiple media segments, and these media segments also contain four kinds of data, namely, occupancy map, geometry, attribute and auxiliary information. Assuming that a media resource is divided to obtain 16 media segments, the numbers of which are 1 to 16 in sequence, the data type of each part of a point cloud media file is a collection of media segments.

[0041] In one exemplary embodiment, the above-mentioned four kinds of data are represented by type lists 1, 2, 3, and 4, respectively. Here, type list 1 represents occupancy map information, type list 2 represents geometry information, type list 3 represents attribute information, and type list 4 represents auxiliary information. It is further assumed that the occupancy map information includes two versions, namely content list 1 and content list 2, and content list 1 is composed of media segments with media segment numbers 1 and 2, and content list 2 is composed of media segments with media segment numbers 3, 4, and 5, so that the occupancy map information includes media segments with media segment numbers 1, 2, 3, 4, and 5. It is assumed that the geometry information includes only one version, namely content list 3, and content list 3 is composed of media segments with media segment numbers 6 and 7, so that the geometry information includes media segments with media segment numbers 6 and 7. It is further assumed that the attribute information includes two versions, which are content lists 4 and 5, respectively, where content list 4 is made up of media segments with media segment numbers 8, 9, and 10, and content list 5 is made up of media segments with media segment numbers 11 and 12, i.e., the attribute information includes media segments with segment numbers 8, 9, 10, 11, and 12. It is also assumed that the auxiliary information includes two versions, which are content lists 6 and 7, respectively, where content list 6 is made up of media segments with media segment numbers 13 and 14, and content list 7 is made up of media segments with media segment numbers 15 and 16, i.e., the auxiliary information includes media segments with media segment numbers 13, 14, 15, and 16. When initially presenting one point cloud content, it is necessary to select four type lists, and further select one content list from each type list, and one point cloud content is completely presented according to the media segments corresponding to the content lists.When it is necessary to perform a costume change operation for a character or the like after the point cloud content is initially presented, a media segment corresponding to a certain data type can be selectively transmitted according to a classification list, and there is no need to transmit all the media segments.

[0042] In this embodiment, the point cloud compression algorithm of VPCC is used for the description, but the present invention is not limited to the point cloud compression algorithm, and the present invention does not limit the encapsulation method of the point cloud data. Therefore, the data type of the point cloud media file includes but is not limited to the four types listed here, and other numbers and / or types of data can be obtained by using other compression algorithms.

[0043] Furthermore, in order to transmit and present the point cloud content, the classification list is defined as a DTG (Data Type Group) information file, and the DTG information file is written into the signaling information for transmission. It should be understood that the content defined by the DTG information file is the content defined by the above-mentioned type list, and thus the DTG information file describes the correspondence between the point cloud media file and the media segment number, the content list, and the type list. In this way, the receiving side can obtain the data classification status of the point cloud media file just by reading the DTG information file, and thus can selectively send the data that needs to be updated for the current point cloud media file according to the user's needs.

[0044] Next, based on the above example, the method for transmitting the DTG information file according to this embodiment and how to change the character's clothes based on the point cloud media file will be described in detail.

[0045] Assume that the point cloud content (i.e., point cloud media file) includes occupancy map information, geometry information, attribute information, and auxiliary information. If the occupancy map, geometry, attribute, and auxiliary information each have only one version, then the number of type lists and content lists are the same. However, in other situations, if the occupancy map, geometry, attribute, and auxiliary information each have only one version, then a content list may not be provided, but this increases complexity. If the occupancy map, geometry, attribute, and auxiliary information each have multiple versions, then a content list must be provided.

[0046] For example, if it is necessary to transmit the occupancy map, geometry, attributes and auxiliary information of a point cloud, it is necessary to first find the corresponding type list. Taking Fig. 3 as an example, assuming that type lists 1 to 4 respectively correspond to the occupancy map, geometry, attributes and auxiliary information, and the occupancy map information includes two versions, type list 1 has two corresponding content lists, and when a user selects one of the content lists, the corresponding media segment can be obtained according to the media segment number listed in the content list and used to present the point cloud content.

[0047] Therefore, the DTG information file contains information on the correspondence between point cloud contents and media segment numbers, and this information can be transmitted in multiple ways, such as by defining new signaling units, signaling files, signaling messages, adding new descriptions, or adding header information to transmission packets. Depending on the implementation method, the information described in the DTG information file can be transmitted and used in different ways.

[0048] In this embodiment, three transmission methods are presented as examples of the transmission method of the DTG information file, but the methods of adding the DTG information file to the signaling information include, but are not limited to, the following methods:

[0049] Method 1: Add one DTG information file that describes the DTG information to the signaling information.

[0050] In method 1, the DTG information file is written to the MPI table as shown in Table 1 and transferred as a part of the signaling message. [Table 1]

[0051] The DTG information file is added to the MPI table, and the part that needs to be changed is in the description of PI_content, and the value of PI_content_count is increased by 1. Here, the PI_content_type field describes the type of the DTG information file, and it is possible to select a suitable file format for describing DTG information according to the actual situation. The PI_content_type_length value is the length of the file type, the PI_content_type_length_byte value is the type of the DTG information file, the PI_content_name_length value is the length of the DTG information file name, and the PI_content_name_byte value is the value of the DTG information file name. PI_content_descriptores_length and PI_content_descriptors_byte are respectively the length and content of the description information of the DTG information file, and may be empty currently and left for future expansion. PI_content_length and PI_content_byte are respectively the length and content of the DTG information file.

[0052] According to the above explanation, the DTG information file can be written to the MPI table, and then the DTG information can be output by signaling information.

[0053] Method 2: Add one descriptor that describes the DTG information.

[0054] For example, in the signaling message defined by the existing MMT protocol, several descriptive descriptors are defined, and a descriptor is descriptive information for the signaling part in MMT to define several fields or functions. For example, a dependency descriptor and an MPU_timestamp_descriptor. Therefore, in method 2, one new descriptor that describes DTG information can be defined, and the type information of the media segment can be described in the descriptor. In the MP table, there is an asset_descriptors field, and this can be achieved by adding an MPUtype_descriptor as shown in Table 2 to the asset_descriptors as necessary. [Table 2]

[0055] Here, descriptor_tag is used to indicate the type of the descriptor, descriptor_length is used to indicate the length of the descriptor, num_mpu is used to indicate the number of MPUs, and mpu_type() is used to indicate the data type of the MPU.

[0056] Method 3: Add one signaling table that describes DTG information.

[0057] In addition to the above method, it is possible to add a new signaling table that is used specifically to describe DTG file information to the signaling information. By generating a table that describes DTG file information and then adding a specific header section, it is possible to generate signaling information that describes DTG information, and this signaling information is then combined with PA messages, MPI messages, MP messages, CRI messages, etc. to form new signaling information and transmit it. The receiving side: new Signaling informationBy receiving and analyzing the received content list, content list information can be obtained.

[0058] In one exemplary operation, when a user requests point cloud occupancy map information, the receiving side checks the DTG information file to find the content list 1 corresponding to the occupancy map information, and finds the media segment numbers 1 and 2 corresponding to the content list 1, and then requests the client (point cloud media player) to send these media segments to store locally and update the local database. When the user subsequently requests point cloud geometric information, the receiving side also checks the DTG information file to find the content list 3 corresponding to the geometric information, and finds the media segment numbers 6 and 7 corresponding to the content list 3, and then requests the client to send these media segments to store locally and update the local database. When the user subsequently requests point cloud attribute information, the receiving side checks the DTG information file to find the content list 4 corresponding to the attribute information, and finds the media segment numbers 8, 9, and 10 corresponding to the content list 4, and then requests the client to send these media segments to store locally and update the local database. If the user subsequently requests auxiliary information of the point cloud, the receiving side will find the content list 6 corresponding to the auxiliary information by inspecting the DTG information file, and find the media segment numbers 13 and 14 corresponding to the content list 6, at this time the client will request these media segments to be sent and stored locally, and update the local database. After the user obtains all the information, the information in the four content lists (content lists 1, 3, 4, and 6) can be integrated to completely present one point cloud media. Later, if the user needs to change clothes, he or she can first find the corresponding type list according to the type of data that needs to be changed, then select a content list from the type list, and then extract the required media segment according to the media segment number in the content list.

[0059] In this embodiment, a method for transmitting point cloud data has been described using MMT (Multimedia Transmission Protocol) as an example. However, it is to be further explained that the point cloud data in this embodiment can also be transmitted using other files such as DASH transmission.

[0060] Example 2 In this embodiment, a classification record is formed by a classification flag, which represents the correspondence between a media segment and a media version and a data type.

[0061] 4 is a flowchart of the encapsulation of point cloud data according to the present embodiment. As shown in FIG. 4, after obtaining the compressed point cloud media file, the method further includes the following steps: Step S21: The compressed point cloud media file is encapsulated as one media resource and divided into 16 media segments as shown in Figure 5. To Multiple media segments To It is possible to obtain it. Step S22: When encapsulating, add one classification flag to each media segment, and record the data type and version information of the media segment according to the classification flag.

[0062] When different media segments have the same media version and the same data type, the classification flags added are usually the same. This embodiment does not limit the manner of adding the classification flag, for example, when encapsulating a point cloud media file, a reserved field can be used to add a classification flag to the MPU, and subsequent users can request a certain type of point cloud data according to the classification flag.

[0063] The classification flag usually includes two flag bits, such as type_id and type_sequence_number described below. type_id is used to define the data type of the media segment, and type_sequence_number is used to define the version of the media segment. Taking the example shown in FIG. 5, the data types of the four MPUs in the first row are all defined as type_id=00, the data types of the four MPUs in the second row are defined as type_id=01, the data types of the four MPUs in the third row are defined as type_id=02, and the data types of the four MPUs in the last row are defined as type_id=03, and the four values ​​of type_id correspond to the four types of data of occupation diagram, geometry, attribute, and auxiliary information, respectively. This embodiment does not limit the specific manner of adding type_id and type_sequence_number.

[0064] In an exemplary embodiment, the reserved fields in the MPUBox can be used to define type_id and type_sequence_number. For example, 2 bits in the reserved field can be defined as type_id, 3 bits in the reserved field can be defined as type_sequence_number, and the remaining 1 bit can remain as a reserved field (if more or fewer types of type_id need to be defined in the future as needed, the length of the type_id field can be continuously increased or decreased). This allows all MPUs belonging to one media resource (defined as asset_id) to be divided into different types using type_id, as shown in Table 3. [Table 3]

[0065] As shown in Table 3, the left column is the existing MMT definition method, and the right column is the MMT definition of the present invention. It can be seen that in the present invention, 2 bits are assigned to type_id, and 3 bits are assigned to type_sequence_number.

[0066] For example, when changing a character's clothes, the occupancy map information, the geometric information, and the auxiliary information may each have only one version, but the corresponding attribute information may have multiple versions, so one type_sequence_number can be set for each type_id, and the number of type_sequence_number is the number of versions. For example, if one point cloud character has three outfits, the type_id can be 10, and the type_sequence_number can be 000, 001, and 010, and MPUs with the same type_sequence_number correspond to the same outfit.

[0067] 5, if the value of type_id is 01, the corresponding data has two versions and the type_sequence_number has two different values, e.g., 000 and 001. If the value of type_id is 03, the corresponding data has four versions and the type_sequence_number has four different values, e.g., 000, 001, 002, and 003. It should be understood that an MPU with type_id=00 does not need to define type_sequence_number for an MPU if the data has only one version.

[0068] Example 3 In this embodiment, the classification record is also a classification list. The difference from the first embodiment is that the classification list represents the correspondence between the media resources and the data types in the point cloud media file, more specifically, the correspondence between the media resource numbers and the media resource lists.

[0069] 6 is a flowchart of the encapsulation of point cloud data according to the present embodiment. As shown in FIG. 6, after obtaining the compressed point cloud media file, the method further includes the following steps: Step S31: The compressed point cloud media file can be encapsulated into a set of media resources (asset_list), and multiple media resources can be obtained, such as the 16 media resources shown in Figure 7. Each media resource has a media resource number, which is numbered sequentially, for example, from 1 to 16. It can be seen that each media resource number is not the same, that is, the media resource number represents the media resource. Step S32: Associate a plurality of media resources with a data type based on the classification list.

[0070] As shown in Figure 7, the classification list includes a media resource list and a resource number list. The media resource list corresponds to the data type of the point cloud media file. The resource number list is a collection of all media resource numbers. Each media resource list corresponds to at least one media resource number, and the number of media resource numbers corresponding to each media resource list is consistent with the media version. Each media resource includes at least one media segment, and the time duration of the media segment may be several seconds.

[0071] Assume that a point cloud content consists of 16 media resources, and the media resource numbers are 1 to 16 in order. Assume that there are four media resource lists, which correspond to occupation map information, geometric information, attribute information and auxiliary information respectively. As shown in FIG. 7, the occupation map information is media resource list 1, which includes five versions correspondingly and can correspond to the media resources whose media resource numbers are 1 to 5. The geometric information is media resource list2, which includes two versions correspondingly, and corresponds to the media resources whose media resource numbers are 6 and 7, respectively. The attribute information is a media resource list 3, which includes five versions correspondingly, and corresponds to the media resources whose media resource numbers are 8 to 12, respectively. The auxiliary information is a media resource list 4, which includes four versions correspondingly, and corresponds to the media resources whose media resource numbers are 13 to 16, respectively. The media resource number is a flag field in the media resource. The point cloud content corresponds to the media resources whose media resource numbers are 1 to 16, respectively, and different point cloud contents correspond to different combinations of media resources, for example, the point cloud contents whose media resource numbers are 1, 6, 8, and 13 and the point cloud contents whose media resource numbers are 1, 6, 9, and 13 may correspond to the appearance of the same point cloud character in different clothes. The point cloud media file includes multiple point cloud contents, and a pending flag (pending_flag) is set in the encapsulated point cloud media file, and the pending flag is used to indicate whether the media resources of all data types of each point cloud content are ready. If the media resources of all data types of each point cloud content are ready, the value of pending_flag is 1, otherwise the value is 0.

[0072] For example, when a point cloud content with media resource numbers 1, 6, 8, and 13 and a point cloud content with media resource numbers 1, 6, 9, and 13 are transmitted in succession, and only one media resource is different between the two point cloud contents, the latter point cloud content only needs to transmit the media resource with media resource number 9. After the transmission of the media resources with media resource numbers 1, 6, 8, and 13 included in the former point cloud content is completed, the value of pending_flag becomes 1. When the transmission of the former point cloud is completed and the latter point cloud content (only the media resource with media resource number 9 needs to be transmitted) is being transmitted, the value of pending_flag becomes 0. When the transmission of the latter point cloud content (only the media resource with media resource number 9 needs to be transmitted) is completed, the value of pending_flag becomes 1.

[0073] Each point cloud content can have a corresponding pending flag (pending_flag) that can be set, and the pending flag is used to indicate whether the media resources of all data types of the point cloud content are ready or not.

[0074] Furthermore, in order to transmit and present the point cloud content, in this embodiment, the classification list is defined as an ACR (Asset Content Relationship) information file, and the ACR information file is written into the signaling information. Similarly, the content defined in the ACR information file is the content corresponding to the classification table in this embodiment, and is used to describe the correspondence between the data type and the media resource number of the point cloud media file. One presentable point cloud content corresponds to one media resource list, and the media resource list is a collection of media resource numbers, and the media resource number is the flag field of the media resource. In this way, the classification status of the point cloud media file can be obtained by simply reading the ACR information file, which facilitates selectively sending the data that needs to be updated for the current point cloud media file according to the user's needs.

[0075] Next, based on the above example, the transmission method of the ACR information file according to this embodiment and how to change the costume of the character based on the point cloud media will be described in detail.

[0076] Suppose that to compose a complete point cloud content, we need to find the media resource numbers from the media resource lists corresponding to the occupancy diagram, geometry, attributes, and auxiliary information, and then extract the corresponding media resources. Suppose that the media resource lists corresponding to the occupancy diagram, geometry, attributes, and auxiliary information are asset_list[0], asset_list[1], asset_list[2], and asset_list[3] respectively, and obtain the media resource numbers from them one by one. The media resources are distinguished by the media resource numbers, and the setting rules are as shown in Table 4. [Table 4]

[0077] Here, the media resources with media resource numbers (asset_id) 0 and 1 correspond to the occupancy map and geometric information of the point cloud, respectively, and the media resources with media resource numbers 2, 3 and 4, 5 correspond to two sets of attributes and auxiliary information. Therefore, the media resources with media resource numbers 0, 1, 2, 3 and 0, 1, 4, 5 can respectively compose different point cloud contents, which are different clothes of the same character.

[0078] The ACR information file contains important information of different data types of point clouds, and can transmit these important information in different ways, such as by defining new signaling units, signaling files, signaling messages, adding new descriptors, or adding header information of transmission packets, etc. Depending on the different implementation methods, the information described in the ACR information file can be transmitted and used in different ways.

[0079] In this embodiment, two methods are presented as examples for the transmission method of the ACR information file, but the methods for adding the ACR information file to the signaling information include, but are not limited to, the following methods:

[0080] Method 1: One ACR file that describes ACR information is added to the signaling information.

[0081] In method 1, the ACR information file is entered into the MPI table as shown in Table 5 and transferred as a part of the signaling message. [Table 5]

[0082] More specifically, the ACR information file is added to the MPI table, the part that needs to be changed is in the description of PI_content, and the value of PI_content_count is increased by 1. Here, the PI_content_type field describes the type of the ACR information file, and it is possible to select a file format suitable for describing the ACR information according to the actual situation. The PI_content_type_length value is the length of the file type, the PI_content_type_length_byte value is the type of the ACR information file, the PI_content_name_length value is the length of the ACR information file name, and the PI_content_name_byte value is the value of the ACR information file name. PI_content_descriptores_length and PI_content_descriptors_byte are the length and content of the description information of the ACR information file, respectively, and may be empty at present and left for future expansion. PI_content_length and PI_content_byte are the length and content of the ACR information file, respectively.

[0083] After writing the ACR information file to the MPI table according to the above description, the ACR information can be transferred by signaling information.

[0084] Method 2: Add one signaling table that describes ACR information.

[0085] In addition to the above method, it is possible to add one more signaling table that is specially used to describe ACR file information in the signaling information. After generating a table describing the ACR information, a specific header section is added to generate signaling information describing the ACR information, and the signaling information is transferred together with the PA message, MPI message, MP message, CRI message, etc. to form new signaling information. The receiving side: new Signaling informationAfter receiving and analyzing the received message, related media resource list information can be obtained.

[0086] In order to more clearly explain the implementation method proposed by the present invention, a specific implementation flow is described by taking an example according to Scheme 1.

[0087] When the receiving side logs in to the service, the sending side sends signaling information, and the user obtains the media resource list (asset_list) in the ACR information file, after which the media resource number (asset_id) corresponding to the occupancy map, geometry, attributes and auxiliary information can be known from asset_list[0], asset_list[1], asset_list[2] and asset_list[3]. When requesting the occupancy map of the point cloud from the sending side, the receiving side: DTG By checking the information file, it is found that the asset_id corresponding to asset_list[0] of the occupancy diagram is 0. If the user continues to request geometry, the corresponding media resources (asset) are obtained from the sender by checking the DTG information file and the local database, and these media resources are the media resources corresponding to a certain asset_id contained in asset_list[1]. If the user requests attributes, DTG By checking the information file and the local database, the corresponding media resources are obtained from the sending side, and these media resources are the media resources corresponding to the asset_id contained in asset_list[2]. If the user continues to request auxiliary information, the corresponding media resources are obtained from the sending side by checking the DTG information file and the local database, and these media resources are the media resources corresponding to the asset_id contained in asset_list[3]. After the user obtains all the information, the information of the obtained four media resources can be integrated to present the point cloud media. If the user needs to change clothes, he or she only needs to request the corresponding media resources in asset_list[2] and asset_list[3] again.

[0088] In this embodiment, a method for transmitting point cloud data has been described using MMT (Multimedia Transmission Protocol) as an example, but it will be further explained that the point cloud data in this embodiment can also be transmitted using other files such as DASH transmission.

[0089] The above describes the specific embodiment of the present invention. It should be understood that the present invention is not limited to the above specific embodiment, and those skilled in the art can make various changes or modifications within the scope of the claims of the present invention, which do not affect the essential content of the present invention. In the absence of conflict, the features in the examples and embodiments of the present application can be arbitrarily combined with each other.

Claims

1. A method for encapsulating a point cloud media resource, comprising: Compressing a point cloud media file (S01); Encapsulating the compressed point cloud media file as a set of media resources (S02), and dividing the set of media resources to obtain a plurality of media resources; and associating the plurality of media resources with data types based on a classification list, the classification list being used to represent a correspondence between the media resources and the data types; The compressed point cloud media file includes a plurality of point cloud contents, each of which corresponds to a different combination of media resources; The method further includes setting a wait flag for each point cloud content in the encapsulated point cloud media file, the wait flag being used to indicate whether all media resources of each point cloud content are ready to be rendered.

13. A method for encapsulating a point cloud media resource, comprising:

2. Each media resource has a corresponding media resource number, and the classification list includes a plurality of media resource lists and a resource number list, each media resource list corresponding to at least one data type of a media resource, and the resource number list is a collection of media resource numbers of all media resources; The method for encapsulating point cloud media resources according to claim 1 .

3. Each media resource list corresponds to at least one media resource number, and the number of the at least one media resource number corresponding to each media resource list is equal to the number of media versions, and each media version is an application scene; The method for encapsulating point cloud media resources according to claim 2 .

4. The data type includes occupancy map information, geometric information, attribute information, or auxiliary information; Each data type corresponds to at least one of the plurality of media resources; The classification list includes correspondences between a plurality of data types and a plurality of media resources; The method for encapsulating point cloud media resources according to claim 1 .

5. When a point cloud content is ready to be rendered, set the pending flag for that point cloud content to 1, or If a point cloud content is not ready to be rendered, set the pending flag for that point cloud content to 0. The method for encapsulating point cloud media resources according to claim 1 .

6. Adding related content of a classification list to signaling information, the classification list is used to represent a correspondence relationship between a plurality of data types in a compressed point cloud media file and a plurality of media resources in the compressed point cloud media file, and the compressed point cloud media file includes a plurality of point cloud contents; and transmitting partial data that needs to be updated of a first point cloud content in the compressed point cloud media file based on the classification list; transmitting all data of a second point cloud content in the compressed point cloud media file before transmitting partial data that needs to be updated of a first point cloud content in the compressed point cloud media file; The partial data of the first point cloud content that needs to be updated is different from the entire data of the second point cloud content; 12. A method for transmitting point cloud media resources, comprising:

7. Different point cloud content corresponds to different combinations of media resources, The method comprises: setting a wait flag for each point cloud content in the encapsulated point cloud media file, the wait flag being used to indicate whether all media resources of each point cloud content are ready to be rendered. The method for transmitting point cloud media resources according to claim 6 .

8. compressing the point cloud media file to obtain a compressed point cloud media file; encapsulating a compressed point cloud media file and setting the classification list according to the plurality of data types. The method for transmitting point cloud media resources according to claim 6 .

9. encapsulating the compressed point cloud media file and setting the classification list according to the plurality of data types, Encapsulating the compressed point cloud media file as a set of media resources, and dividing the set of media resources to obtain the plurality of media resources; and associating the plurality of media resources with a plurality of data types of the compressed point cloud media file based on the classification list. The method for transmitting point cloud media resources according to claim 8 .

10. The plurality of media resources have different media resource numbers, and the classification list includes a plurality of media resource lists and a resource number list, the plurality of media resource lists corresponding to a plurality of data types of the set of media resources, and the resource number list is a collection of media resource numbers of all media resources; The method for transmitting point cloud media resources according to claim 9 .

11. Each media resource list corresponds to at least one media resource number, and the number of the at least one media resource number corresponding to each media resource list is equal to the number of media versions; The method for transmitting point cloud media resources according to claim 10 .

12. When a point cloud content is ready to be rendered, set the pending flag for that point cloud content to 1, or If a point cloud content is not ready to be rendered, set the pending flag for that point cloud content to 0. The method for transmitting point cloud media resources according to claim 7 .

Citation Information

Patent Citations

  • Three-dimensional point cloud compression storage method and system based on spatial-temporal data fusion

    CN108009979A

  • Point cloud data compression method, point cloud data compression device and terminal

    CN108510558A

  • 3D point cloud compression method

    CN110363822A

  • Method for encoding a point cloud representing a scene, encoder system, and non-transitory computer-readable recording medium having a program stored thereon

    JP2019521417A