Point cloud file encapsulation and decapsulation method and apparatus, device, and storage medium

Abstract

The present application provides a point cloud file encapsulation and decapsulation method and apparatus, a device, and a storage medium. The decapsulation method comprises: determining first attribute data to be decoded in a point cloud file (S101); determining dependency indication information of the first attribute data (S102), the dependency indication information being used for indicating whether there is a first relationship between the first attribute data and second attribute data, and the first relationship comprising at least one of a decoding dependency relationship and a presentation association relationship; and decoding the first attribute data on the basis of the dependency indication information (S103). That is, according to the present application, a coding / decoding dependency relationship or a presentation association relationship between different point cloud attribute data is indicated by means of the dependency indication information, so as to support partial access, correct decoding, and personalized presentation of a point cloud bitstream.

Description

Point cloud file encapsulation and decapsulation method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 21, 2023, with application number 202311055895.4 and application name “Point cloud file encapsulation and decapsulation method, device, equipment and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of video processing technology, and in particular to a point cloud file encapsulation and decapsulation method, apparatus, device, and storage medium. Background Art

[0003] Immersive media refers to media content that can provide consumers with an immersive experience. Immersive media can be divided into 3DoF media, 3DoF+ media, and 6DoF media according to the user's degree of freedom when consuming media content.

[0004] Immersive media includes point cloud media. Currently, point cloud media can achieve partial access to the point cloud bitstream by defining subsamples or multi-track packaging. However, it is not possible to achieve partial access to the attribute information in the point cloud bitstream.

[0005] Summary of the Invention

[0006] The present application provides a point cloud file encapsulation and decapsulation method, apparatus, device and storage medium, which indicates the association relationship between attribute data through dependency indication information, thereby achieving access, correct decoding and personalized presentation of part of the attribute data in the point cloud bit stream.

[0007] In a first aspect, the present application provides a file decapsulation method, comprising:

[0008] Determine first attribute data to be decoded in a point cloud file, where the point cloud file is a file obtained by encapsulating a point cloud bit stream;

[0009] Determining dependency indication information of the first attribute data, where the dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, where the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship;

[0010] The first attribute data is decoded based on the dependency indication information.

[0011] In a second aspect, the present application provides a file encapsulation method, comprising:

[0012] Obtaining a point cloud bitstream, wherein the point cloud bitstream includes N sets of attribute data, where N is a positive integer;

[0013] For the first attribute data to be encapsulated in the N groups of attribute data, the first attribute data is encapsulated based on the first relationship between the first attribute data and the second attribute data, and dependency indication information of the first attribute data is determined to obtain a point cloud file, where the dependency indication information is used to indicate whether there is a first relationship between the first attribute data and the second attribute data, and the first relationship includes at least one of an encoding dependency relationship and a presentation association relationship.

[0014] In a third aspect, the present application provides a file decapsulation device, comprising:

[0015] a data determining unit, configured to determine first attribute data to be decoded in a point cloud file, wherein the point cloud file is a file obtained by encapsulating a point cloud bit stream;

[0016] an information determining unit, configured to determine dependency indication information of the first attribute data, the dependency indication information being used to indicate whether the first attribute data depends on second attribute data when being decoded or presented;

[0017] A decoding unit is configured to decode the first attribute data based on the dependency indication information.

[0018] In a fourth aspect, the present application provides a file encapsulation device, comprising:

[0019] An acquisition unit, configured to acquire a point cloud bit stream, wherein the point cloud bit stream includes N sets of attribute data, where N is a positive integer;

[0020] an encapsulation unit, configured to encapsulate first attribute data to be encapsulated in the N sets of attribute data based on a first relationship between the first attribute data and second attribute data, and determine dependency indication information of the first attribute data to obtain a point cloud file;

[0021] The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, and the first relationship includes at least one of an encoding dependency relationship and a presentation association relationship.

[0022] In a fourth aspect, an electronic device is provided, comprising a processor and a memory, wherein the memory is configured to store a computer program, and the processor is configured to call and execute the computer program stored in the memory to perform the method of the first or second aspect.

[0023] In a fifth aspect, a chip is provided for implementing the method of any aspect or its implementations described in the first aspect. Specifically, the chip includes a processor configured to call and execute a computer program from a memory, causing a device equipped with the chip to perform the method described in the first or second aspect.

[0024] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method of the first aspect or the second aspect.

[0025] In a seventh aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method of the first or second aspect.

[0026] In an eighth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method of the first or second aspect.

[0027] In summary, in this application, by determining the first attribute data to be decoded in the point cloud file, the dependency indication information of the first attribute data is determined. The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, wherein the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship. Based on the dependency indication information, the first attribute data is decoded. That is, this application indicates the encoding and decoding dependency relationship or presentation association relationship between different point cloud attribute data through the dependency indication information to support partial access, correct decoding, and personalized presentation of the point cloud bitstream. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] FIG1 schematically shows a schematic diagram of three degrees of freedom;

[0030] FIG2 schematically shows a schematic diagram of three degrees of freedom +;

[0031] FIG3 schematically shows a schematic diagram of six degrees of freedom;

[0032] FIG4A is an architecture diagram of an immersive media system provided by an embodiment of the present application;

[0033] FIG4B is a schematic diagram of a point cloud system framework provided by an embodiment of the present application;

[0034] FIG5A is a schematic diagram of a sample structure of a geometric point cloud stored in a single track;

[0035] FIG5B is a schematic diagram of a component-based multi-track packaging structure;

[0036] FIG5C is a schematic diagram of a multi-track packaging structure based on slicing;

[0037] FIG5D is a schematic diagram of a multi-track packaging structure based on a point cloud slice;

[0038] FIG6 is a flowchart of a point cloud file decapsulation method provided in an embodiment of the present application;

[0039] FIG7A is a schematic diagram of a multi-track package according to an embodiment of the present application;

[0040] FIG7B is a schematic diagram of a single-track package according to an embodiment of the present application;

[0041] FIG8 is a flowchart of a point cloud file packaging method provided in an embodiment of the present application;

[0042] FIG9 is a schematic structural diagram of a file decapsulation device provided in one embodiment of the present application;

[0043] FIG10 is a schematic structural diagram of a file packaging device provided in one embodiment of the present application;

[0044] FIG11 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] The embodiments of the present application relate to data processing technology for immersive media.

[0048] Before introducing the technical solution of this application, the following is an introduction to the relevant knowledge of this application:

[0049] Multi-view / multi-viewpoint video: This refers to video captured from multiple angles using a multi-camera array and containing depth information. Multi-view / multi-viewpoint video, also known as free-viewpoint / free-viewpoint video, is an immersive media that provides a six-degree-of-freedom experience.

[0050] Point cloud: A point cloud is a set of randomly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. Each point in a point cloud has at least 3D position information and, depending on the application scenario, may also have color, material, or other information. Typically, each point in a point cloud has the same number of additional attributes.

[0051] PCC: Point Cloud Compression, point cloud compression.

[0052] G-PCC: Geometry-based Point Cloud Compression, point cloud compression based on geometric model.

[0053] V-PCC: Video-based Point Cloud Compression, point cloud compression based on traditional video coding.

[0054] Slice: A point cloud slice / point cloud strip, representing a collection of syntax elements (such as geometric slices, attribute slices) of part or all of the encoded point cloud frame data.

[0055] Sequence Header: Point cloud sequence header parameter set, the parameter set required for point cloud sequence decoding. (AVS definition)

[0056] Geometry Header: Point cloud frame geometry header parameter set, the parameter set required for decoding point cloud geometry data.

[0057] Attribute Header: Point cloud frame attribute header parameter set, the parameter set required for decoding point cloud attribute data.

[0058] SPS: Sequence Parameter Set, a set of parameters required for point cloud sequence decoding. (MPEG definition)

[0059] GPS: Geometry Parameter Set, a set of parameters required for decoding point cloud geometry data.

[0060] APS: Attribute Parameter Set, attribute parameter set, the parameter set required for decoding point cloud attribute data.

[0061] Atlas: Indicates the area information on the 2D plane frame, the area information of the 3D presentation space, the mapping relationship between the two, and the necessary parameter information required for the mapping.

[0062] Track: Track is a collection of media data in the process of media file encapsulation, consisting of multiple time-series samples. A media file can be composed of one or more tracks. For example, a media file can contain a video media track, an audio media track, and a subtitle media track. In particular, metadata information can also be included in the file as a media type in the form of metadata media tracks.

[0063] Sample: A sample is a unit of media file encapsulation. A track consists of many samples, each of which corresponds to a specific timestamp. For example, a video media track can consist of many samples, each of which is typically a video frame. In this embodiment of the present application, a sample in a point cloud media track can be a point cloud frame.

[0064] Sample Number: The number of a specific sample. The first sample in a track has a number of 1.

[0065] Sample Entry: A sample entry that indicates metadata related to all samples in a track. For example, a sample entry for a video track typically contains metadata related to decoder initialization.

[0066] Sample Group: Sample group, used to group some samples in the track according to specific rules.

[0067] Item: A data item is a collection of media data in the process of encapsulating a static media file. For example, a static image is encapsulated as an item.

[0068] Slice: A point cloud slice or strip, representing a collection of syntax elements (such as geometry slices and attribute slices) that represent part or all of the encoded point cloud frame data. A point cloud slice corresponds to a point within a certain spatial area.

[0069] DASH: dynamic adaptive streaming over HTTP, dynamic adaptive streaming over HTTP is an adaptive bitrate streaming technology that enables high-quality streaming media to be delivered over the Internet through traditional HTTP network servers.

[0070] MPD: media presentation description, media presentation description signaling in DASH, used to describe media segment information.

[0071] Representation: In DASH, a combination of one or more media components, such as a video file of a certain resolution, can be considered a Representation.

[0072] Adaptation Sets: In DASH, a collection of one or more video streams. An Adaptation Set can contain multiple Representations.

[0073] Media Segment: A playable segment that conforms to a specific media format. Playback may require the cooperation of zero or more preceding segments and the initialization segment.

[0074] DoF: Degree of Freedom. In a mechanical system, this refers to the number of independent coordinates. In addition to translational degrees of freedom, this also includes rotational and vibrational degrees of freedom. In this embodiment, it refers to the degrees of freedom that allow users to move and interact with content while viewing immersive media.

[0075] 3DoF: Three degrees of freedom, referring to the three degrees of freedom of the user's head around the X, Y, and Z axes. Figure 1 schematically illustrates three degrees of freedom. As shown in Figure 1, any point in a certain location can rotate along three axes: turning the head, tilting it up and down, and tilting it. Through the three-degree-of-freedom experience, users can immerse themselves in a scene 360 ​​degrees. If it is static, it can be understood as a panoramic image. If the panoramic image is dynamic, it is a panoramic video, or VR video. However, VR videos have certain limitations: users cannot move or choose a specific location to view.

[0076] 3DoF+: In addition to the three degrees of freedom, the user also has limited freedom of movement along the X, Y, and Z axes. This can also be called limited six degrees of freedom, and the corresponding media stream can be called a limited six-degree-of-freedom media stream. Figure 2 schematically shows a diagram of 3DoF+.

[0077] 6DoF: In addition to the three degrees of freedom (3DFO), users also have the freedom of movement along the X, Y, and Z axes. The corresponding media stream is called a 6DoF media stream. Figure 3 schematically illustrates 6DoF. 6DoF media refers to 6-degree-of-freedom video, which provides users with a high-degree-of-freedom viewing experience, allowing them to freely move their viewpoint along the X, Y, and Z axes in three-dimensional space, as well as freely rotate their viewpoint around the X, Y, and X axes. 6DoF media is a combination of videos captured from different spatial perspectives by a camera array. To facilitate the expression, storage, compression, and processing of 6DoF media, 6DoF media data is represented as a combination of the following information: texture maps captured by multiple cameras, depth maps corresponding to these texture maps, and corresponding 6DoF media content description metadata. This metadata includes multi-camera parameters and information describing the 6DoF media's stitching layout and edge protection. At the encoder end, the multi-camera texture map information and corresponding depth map information are spliced ​​together, and the description of the splicing method is written into the metadata according to defined syntax and semantics. The stitched multi-camera depth map and texture map information is encoded using a planar video compression method and transmitted to the terminal for decoding. The 6DoF virtual viewpoint requested by the user is then synthesized, thereby providing the user with a 6DoF media viewing experience.

[0078] AVS: Audio Video Coding Standard, China's national video coding standard AVS.

[0079] MPEG: Moving Picture Experts Group, an organization established by ISO (International Standardization Organization) and IEC (International Electrotechnical Commission) to develop international standards for moving picture and speech compression.

[0080] ISOBMFF: ISO Based Media File Format, a media file format based on ISO standards. ISOBMFF is a media file encapsulation standard, and the most typical ISOBMFF file is the MP4 file.

[0081] SMT (Smart Media Transport): A smart media transport standard that specifies smart media transport technologies covering encapsulation formats, transport protocols, and signaling messages for use in the transmission and delivery of multimedia data.

[0082] Asset: Media resource, any multimedia data entity associated with a unique identifier used to construct a multimedia presentation.

[0083] Immersive media refers to media content that provides consumers with an immersive experience. Based on the degree of freedom users have when consuming media content, immersive media can be categorized into 3DoF media, 3DoF+ media, and 6DoF media. Common 6DoF media include multi-view video and point cloud media.

[0084] A point cloud is a collection of randomly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. Each point in a point cloud contains at least 3D position information and, depending on the application scenario, may also contain color, material, or other information. Typically, each point in a point cloud has the same number of additional attributes.

[0085] Point clouds can flexibly and conveniently express the spatial structure and surface properties of three-dimensional objects or scenes, and therefore have a wide range of applications, including virtual reality (VR) games, computer-aided design (CAD), geographic information systems (GIS), autonomous navigation systems (ANS), digital cultural heritage, free viewpoint broadcasting, three-dimensional immersive telepresence, and three-dimensional reconstruction of biological tissues and organs.

[0086] Point clouds are primarily acquired through computer generation, 3D laser scanning, and 3D photogrammetry. Computers can generate point clouds of virtual three-dimensional objects and scenes. 3D scanning can obtain point clouds of static, real-world three-dimensional objects or scenes, generating millions of point clouds per second. 3D imaging can obtain point clouds of dynamic, real-world three-dimensional objects or scenes, generating tens of millions of point clouds per second. Furthermore, in the medical field, MRI, CT, and electromagnetic positioning information can be used to obtain point clouds of biological tissues and organs. These technologies reduce the cost and time required to acquire point cloud data while improving data accuracy. Changes in point cloud data acquisition methods have made it possible to acquire large amounts of point cloud data. With the continuous accumulation of large-scale point cloud data, efficient storage, transmission, publication, sharing, and standardization of point cloud data have become key to point cloud applications.

[0087] After encoding the point cloud content, the encoded data stream needs to be encapsulated and transmitted to the user. Correspondingly, the point cloud media player must first decapsulate the point cloud file, then decode it, and finally present the decoded data stream. Therefore, obtaining specific information during the decapsulation process can improve the efficiency of the decoding process to a certain extent, thereby providing a better experience for the presentation of point cloud media.

[0088] Figure 4A is an architectural diagram of an immersive media system provided in one embodiment of the present application. As shown in Figure 4A, the immersive media system includes an encoding device and a decoding device. The encoding device may refer to a computer device used by the provider of the immersive media, and the computer device may be a terminal (such as a PC (Personal Computer), a smart mobile device (such as a smart phone), etc.) or a server. The decoding device may refer to a computer device used by the user of the immersive media, and the computer device may be a terminal (such as a PC (Personal Computer), a smart mobile device (such as a smart phone), a VR device (such as a VR helmet, VR glasses, etc.)). The data processing process of the immersive media includes a data processing process on the encoding device side and a data processing process on the decoding device side.

[0089] The data processing process on the encoding device side mainly includes:

[0090] (1) The acquisition and production process of immersive media content;

[0091] (2) The process of encoding and file packaging of immersive media. The data processing process on the decoding device side mainly includes:

[0092] (3) The process of decapsulating and decoding immersive media files;

[0093] (4) Rendering process of immersive media.

[0094] In addition, the transmission process of immersive media between the encoding device and the decoding device can be carried out based on various transmission protocols. The transmission protocols here may include but are not limited to: DASH (Dynamic Adaptive Streaming over HTTP, dynamic adaptive streaming media transmission) protocol, HLS (HTTP Live Streaming, dynamic bit rate adaptive transmission) protocol, SMTP (Smart Media Transport Protocol, smart media transmission protocol), TCP (Transmission Control Protocol, transmission control protocol), etc.

[0095] The following will provide a detailed introduction to each process involved in the data processing of immersive media with reference to FIG4A .

[0096] 1. Data processing process on the encoding device side:

[0097] (1) The process of acquiring and producing media content for immersive media.

[0098] 1) The process of acquiring media content of immersive media.

[0099] In one implementation, the capture device may refer to a hardware component provided in the encoding device, for example, the capture device may refer to a microphone, camera, sensor, etc. of a terminal. In another implementation, the capture device may also be a hardware device connected to the encoding device, for example, a camera connected to a server.

[0100] The capture device may include, but is not limited to, an audio device, a camera device, and a sensor device. The audio device may include an audio sensor, a microphone, etc. The camera device may include a standard camera, a stereo camera, a light field camera, etc. The sensor device may include a laser device, a radar device, etc.

[0101] Multiple capture devices can be deployed at specific locations in real space to simultaneously capture audio and video content from different angles within that space. The captured audio and video content is synchronized in both time and space. The media content collected by the capture devices is called immersive media raw data.

[0102] 2) The production process of media content for immersive media.

[0103] The captured audio content itself is suitable for audio encoding for immersive media. The captured video content undergoes a series of production processes before it becomes suitable for video encoding for immersive media. The production process includes:

[0104] ① Stitching. Since the captured video content is shot by the capture device at different angles, stitching refers to stitching the video content shot at various angles into a complete video that can reflect the 360-degree visual panorama of the real space. In other words, the stitched video is a panoramic video (or spherical video) represented in three-dimensional space.

[0105] ② Projection. Projection refers to the process of mapping a spliced ​​3D video onto a 2D image. The resulting 2D image is called a projected image. Projection methods include, but are not limited to, latitude and longitude projection and regular hexahedron projection.

[0106] ③Region encapsulation. The projected image can be encoded directly, or the projected image can be region encapsulated before encoding. In practice, it is found that in the data processing process of immersive media, encoding the two-dimensional projection image after region encapsulation can greatly improve the video coding efficiency of the immersive media. Therefore, the region encapsulation technology is widely used in the video processing process of immersive media. The so-called region encapsulation refers to the process of performing conversion processing on the projection image by region. The region encapsulation process converts the projection image into an encapsulated image. The region encapsulation process specifically includes: dividing the projection image into multiple mapping regions, and then performing conversion processing on the multiple mapping regions to obtain multiple encapsulated regions, and mapping the multiple encapsulated regions to a 2D image to obtain an encapsulated image. Among them, the mapping region refers to the region obtained by division in the projection image before performing region encapsulation; the encapsulation region refers to the region located in the encapsulated image after performing region encapsulation.

[0107] The conversion process may include, but is not limited to, mirroring, rotating, rearranging, upsampling, downsampling, changing the resolution of the region, and moving the region.

[0108] It should be noted that since capture devices can only capture panoramic video, after such video is processed by the encoding device and transmitted to the decoding device for appropriate data processing, users on the decoding device can only view 360-degree video information by performing certain specific actions (such as head rotation). Non-specific actions (such as moving the head) do not produce corresponding video changes, resulting in a poor VR experience. Therefore, it is necessary to provide additional depth information that matches the panoramic video to achieve a better immersion and a more optimal VR experience. This involves 6DoF (six degrees of freedom) production technology. When users can move relatively freely in a simulated scene, it is called 6DoF. When using 6DoF production technology to produce immersive media video content, the capture device generally uses a light field camera, laser equipment, radar equipment, etc. to capture point cloud data or light field data in space. During the above production processes ①-③, some specific processing is also required, such as cutting and mapping the point cloud data and calculating depth information.

[0109] (2) The process of encoding and file packaging of immersive media.

[0110] The captured audio content can be directly audio-encoded to form an audio code stream for immersive media. After the above-mentioned production process ①-② or ①-③, the projected image or encapsulated image is video-encoded to obtain a video code stream for immersive media. For example, the packaged picture (D) is encoded into an encoded image (Ei) or an encoded video bit stream (Ev). The captured audio (Ba) is encoded into an audio bit stream (Ea). Then, according to a specific media container file format, the encoded images, videos and / or audio are combined into a media file (F) for file playback or a sequence of initialization segments and media segments for streaming (Fs). The encoding device also includes metadata, such as projection and region information, into the file or fragment to facilitate the presentation of the decoded packaged picture.

[0111] It should be noted here that if 6DoF production technology is adopted, a specific encoding method (such as point cloud encoding) needs to be used for encoding during the video encoding process. The audio stream and the video stream are encapsulated in a file container according to the file format of immersive media (such as ISOBMFF (ISO Base Media File Format, ISO base media file format)) to form a media file resource of the immersive media. The media file resource can be a media file or a media fragment to form a media file of the immersive media; and according to the file format requirements of the immersive media, the media presentation description information (MPD) is used to record the metadata of the media file resource of the immersive media. The metadata here is a general term for information related to the presentation of immersive media. The metadata may include description information of the media content, description information of the window, and signaling information related to the presentation of the media content, etc. As shown in Figure 4A, the encoding device will store the media presentation description information and media file resources formed after the data processing process.

[0112] The immersive media system supports data boxes, which are data blocks or objects containing metadata. Specifically, a data box contains metadata about the corresponding media content. Immersive media can include multiple data boxes, such as a Sphere Region Zooming Box, which contains metadata describing sphere region zooming information; a 2D Region Zooming Box, which contains metadata describing 2D region zooming information; a Region Wise Packing Box, which contains metadata describing the corresponding information during the region packing process, and so on.

[0113] 2. Data processing on the decoding device side:

[0114] (3) The process of decapsulating and decoding immersive media files;

[0115] The decoding device can obtain the media file resources and corresponding media presentation description information of the immersive media from the encoding device through the recommendation of the encoding device or adaptively and dynamically according to the user needs of the decoding device. For example, the decoding device can determine the user's orientation and position based on the user's head / eye / body movement information, and then dynamically request the encoding device to obtain the corresponding media file resources based on the determined orientation and position. The media file resources and media presentation description information are transmitted from the encoding device to the decoding device through a transmission mechanism (such as DASH, SMT). The file decapsulation process on the decoding device side is the opposite of the file encapsulation process on the encoding device side. The decoding device decapsulates the media file resources according to the file format requirements of the immersive media to obtain audio streams and video streams. The decoding process on the decoding device side is the opposite of the encoding process on the encoding device side. The decoding device performs audio decoding on the audio stream to restore the audio content.

[0116] In addition, the decoding process of the video stream by the decoding device includes the following:

[0117] ① Decoding the video stream to obtain a planar image; based on the metadata provided by the media presentation description information, if the metadata indicates that the immersive media has performed a region encapsulation process, the planar image is a packaged image; if the metadata indicates that the immersive media has not performed a region encapsulation process, the planar image is a projected image;

[0118] ② If the metadata indicates that the immersive media has performed a regional encapsulation process, the decoding device will perform regional decapsulation on the encapsulated image to obtain a projected image. Here, regional decapsulation is the opposite of regional encapsulation. Regional decapsulation refers to the process of performing inverse conversion processing on the encapsulated image according to the region. Regional decapsulation converts the encapsulated image into a projected image. The process of regional decapsulation specifically includes: performing inverse conversion processing on multiple encapsulated regions in the encapsulated image according to the instructions of the metadata to obtain multiple mapping regions, and mapping the multiple mapping regions to a 2D image to obtain a projected image. Inverse conversion processing refers to processing that is opposite to the conversion processing. For example, if the conversion processing refers to a counterclockwise rotation of 90 degrees, then the inverse conversion processing refers to a clockwise rotation of 90 degrees.

[0119] ③ Reconstruct the projected image according to the media presentation description information to convert it into a 3D image. The reconstruction process here refers to the process of re-projecting the two-dimensional projected image into a 3D space.

[0120] (4) Rendering process of immersive media.

[0121] The decoding device renders the audio content obtained by audio decoding and the 3D image obtained by video decoding according to the metadata related to rendering and viewport in the media presentation description information. Once the rendering is completed, the playback output of the 3D image is realized. In particular, if 3DoF and 3DoF+ production technologies are adopted, the decoding device mainly renders the 3D image based on the current viewpoint, parallax, depth information, etc. If 6DoF production technology is adopted, the decoding device mainly renders the 3D image in the viewport based on the current viewpoint. Among them, the viewpoint refers to the user's viewing position, the parallax refers to the difference in line of sight between the user's two eyes or the difference in line of sight caused by movement, and the viewport refers to the viewing area.

[0122] The immersive media system supports data boxes, which are data blocks or objects containing metadata. Specifically, a data box contains metadata about the corresponding media content. Immersive media can include multiple data boxes, such as a Sphere Region Zooming Box, which contains metadata describing sphere region zooming information; a 2D Region Zooming Box, which contains metadata describing 2D region zooming information; and a Region Wise Packing Box, which contains metadata describing the corresponding information during the region packing process.

[0123] FIG4B is a schematic diagram of a point cloud system framework according to an embodiment of the present application. As shown in FIG4B , the point cloud system includes a file encapsulation device and a file decapsulation device. In some embodiments, the file encapsulation device can be understood as the encoding device described above, and the file decapsulation device can be understood as the decoding device described above.

[0124] The real-world visual scene A is captured by a set of cameras or a camera device with multiple lenses and sensors. The acquisition result is point cloud source data B, which is a frame sequence consisting of a large number of point cloud frames. One or more point cloud frames are encoded into an encoded point cloud bit stream E, including an encoded geometry bit stream and an attribute bit stream. After encapsulation according to a specific media container file format (such as ISOBMFF), one or more encoded bit streams are combined into a sequence of initialization segments and media segments (Fs) for streaming transmission or a media file (F) for file playback. At the same time, file encapsulation also includes metadata in the file F or media segment Fs, and uses a transmission mechanism to transmit the segment Fs to the player.

[0125] The file decapsulation process receives the file F' or fragment Fs', extracts the encoded bitstream E' and parses the metadata, and then decodes it to generate the point cloud data D'. During the media processing, the point cloud data is rendered and displayed on the screen of the head-mounted display or any other display device according to the current viewing position, viewing direction, or the window determined by various types of sensors (such as head, position, or eye movement sensors). The point cloud data that is partially accessed and decoded through the current viewing position or viewing direction can be used to optimize the media processing process. In the window-based transmission process, the current viewing position and viewing direction are also passed to the strategy module to determine the track to be received.

[0126] The above process works for both real-time and on-demand use cases.

[0127] The parameters in FIG4B are defined as follows: E / E': the encoded G-PCC bitstream; F / F': the media file including the track format specification, which may include constraints on the elementary streams included in the track samples.

[0128] There are three packaging modes for point cloud code streams: single-track packaging, component-based multi-track packaging, and slice-based multi-track packaging.

[0129] FIG5A is a schematic diagram of a sample structure of a geometric point cloud stored in a single track. As shown in FIG5A , the attribute data and geometric data of the entire point cloud bit stream are encapsulated in one track, which includes one or more samples.

[0130] Figure 5B is a schematic diagram of a component-based multi-track packaging structure. As shown in Figure 5B , the geometric data of the point cloud bitstream is packaged in several component tracks, the attribute 1 data of the point cloud bitstream is packaged in attribute component track 1, and the attribute 2 data of the point cloud bitstream is packaged in attribute component track 2. Each component track includes multiple samples.

[0131] Figure 5C is a schematic diagram of a multi-track packaging structure based on slicing. As shown in Figure 5C, when packaging based on point cloud slices, it includes a point cloud slice base track and one or more point cloud slice tracks. For example, the point cloud bitstreams corresponding to point cloud slices 1 and 2 are packaged in point cloud slice track 1, and the point cloud bitstream corresponding to point cloud slice 3 is packaged in point cloud slice track 2.

[0132] Figure 5D is a schematic diagram of a multi-track packaging structure based on a point cloud slice. When a point cloud slice track contains partial component data, a sample point cloud slice track structure is shown in Figure 5D, including a point cloud slice base track, geometry component tracks for point cloud slices 1 and 2, attribute component tracks for point cloud slices 1 and 2, geometry component track for point cloud slice 3, and attribute component track for point cloud slice 3.

[0133] Each point cloud sample can be divided into one or more point cloud subsamples. SubSampleInformationBox is used in the encapsulation of point cloud data to define subsamples according to the value of the flags field of the subsample information data. The flag field specifies the type of subsample information in this data box, as shown below:

[0134] 0: Subsample based on point cloud data type. A subsample contains only one data type defined by AVSPCCPayloadType.

[0135] 1: Subsample based on a point cloud patch. A subsample contains only information about a point cloud patch. When the corresponding track contains a component information data box, the subsample of the corresponding track only contains the component data corresponding to the component information data box. When the corresponding track does not contain a component information data box, the subsample of the corresponding track contains all component data.

[0136] Other flag values ​​are reserved.

[0137] The codec_specific_parameters field of SubsampleInformationBox is defined as follows:

[0138] AVSPCCPayloadType indicates the data type of the point cloud contained in the subsample. The meaning of the value is shown in Table 1 below:

[0139] Table 1 Point cloud data types

[0140] attr_type indicates the type of attribute data contained in the subsample. A value of 0 indicates color attributes; a value of 1 indicates reflectance attributes.

[0141] slice_data indicates whether the subsample contains point cloud slice data. When the value is 1, it means that the point cloud slice geometry and / or attribute type data is included; when the value is 0, it means that the point cloud parameter information is included.

[0142] slice_id indicates the ID of the point cloud slice corresponding to the data contained in the subsample.

[0143] As can be seen from the above, although it is currently possible to access partial data units of a point cloud frame by defining subsamples, or to access data of a specific type or specific point cloud slice through multi-track encapsulation, when the point cloud bitstream contains multiple sets of attribute data of the same type, or when there is a coding, decoding, or presentation relationship between the attribute data, existing technologies cannot accurately indicate the relationship between these attribute data, and thus cannot correctly decode and personalize the presentation of some attribute data.

[0144] In order to solve the above technical problems, an embodiment of the present application proposes a point cloud file encapsulation and decapsulation method. For the point cloud bit stream, attribute data with encoding or presentation association relationships are encapsulated in the same track or the same attribute group, and the encoding and decoding dependency or association relationship between different point cloud attribute data is indicated through dependency indication information to support partial access, correct decoding and personalized presentation of the point cloud bit stream.

[0145] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0146] Figure 6 is a flow chart of a point cloud file decapsulation method provided by an embodiment of the present application. The present application embodiment can be implemented by the above-mentioned file decapsulation device or decoder, and the following description is made using the file decapsulation device as an example.

[0147] As shown in FIG6 , the method includes the following steps:

[0148] S101: Determine first attribute data to be decoded in a point cloud file.

[0149] The point cloud file is a file obtained by encapsulating the point cloud bit stream.

[0150] In an embodiment of the present application, point cloud data is encoded to obtain a point cloud bitstream. In some embodiments, the point cloud bitstream is also referred to as a point cloud codestream. The point cloud includes geometric information and attribute information. The geometric information is encoded to obtain a geometric codestream (or geometric bitstream), and the attribute information is encoded to obtain an attribute codestream (or attribute bitstream). The point cloud bitstream of an embodiment of the present application includes at least an attribute bitstream. For example, the point cloud bitstream of an embodiment of the present application includes both a geometric bitstream and an attribute bitstream, or the point cloud bitstream of an embodiment of the present application includes only an attribute bitstream.

[0151] In some embodiments, the point cloud bitstream of the present application includes geometric data and N sets of attribute data, where N is a positive integer. The N sets of attribute data can also be understood as N attribute components. For example, each point in the point cloud includes N attribute data, so the entire point cloud includes N sets of attribute data or N attribute components. One set of attribute data or one attribute component includes the attribute data for all points in the point cloud. The geometric data can be understood as the aforementioned geometric bitstream, and the attribute data can be understood as the attribute bitstream.

[0152] In an embodiment of the present application, after the point cloud encapsulation device (such as a server) obtains the point cloud bit stream, it encapsulates the point cloud bit stream to obtain a point cloud file.

[0153] From the above, it can be seen that the way in which the point cloud packaging device packages the point cloud bit stream includes at least single-track packaging, component-based multi-track packaging, and point cloud slice-based multi-track packaging.

[0154] In some embodiments, if the point cloud packaging device uses single-track packaging, the point cloud attribute data can be packaged, that is, point cloud attribute data with a first relationship can be divided into the same attribute group. In this way, when decoding, the decoding end can only decode the attribute data in the same attribute group, achieving partial access to the point cloud attribute data and correct decoding, thereby realizing personalized presentation of the point cloud attribute data.

[0155] In some embodiments, if the point cloud encapsulation device adopts component-based multi-track encapsulation, the point cloud attribute data with a first relationship in the point cloud bitstream can be encapsulated in the same component track. In this way, the decoding end can only decode the attribute data in the same component track during decoding, thereby achieving partial access to the point cloud attribute data and correct decoding, thereby achieving personalized presentation of the point cloud attribute data. For example, the point cloud bitstream includes geometric data and N groups of attribute data. The point cloud encapsulation device can encapsulate the geometric data into a separate component track, encapsulate the M groups of attribute data that have a first relationship with each other in the N groups of attribute data into the same attribute component track, and encapsulate the 1 group of attribute data that does not have a first relationship with other attribute data in the N groups of attribute data into a separate attribute component track.

[0156] In some embodiments, if the point cloud encapsulation device adopts multi-track encapsulation based on point cloud slices, then for a specific point cloud slice, its corresponding geometric and attribute data can be encapsulated in the same point cloud slice track. Alternatively, the point cloud attribute data with a first relationship in the point cloud bitstream can be encapsulated in the same point cloud slice track. In this way, the decoding end can only decode the attribute data in the same point cloud slice track during decoding, realize partial access to the point cloud attribute data, and correct decoding, thereby realizing personalized presentation of the point cloud attribute data. For example, for a specific point cloud slice, its corresponding geometric and attribute data are located in multiple different point cloud slice tracks, such as encapsulating the geometric data into a separate point cloud slice track, encapsulating M groups of attribute data that have a first relationship with each other in N groups of attribute data into a point cloud slice track, and encapsulating 1 group of attribute data that does not have a first relationship with other attribute data in N groups of attribute data into a separate point cloud slice track.

[0157] The embodiment of the present application does not limit the specific type of the first relationship, for example, it can be any association relationship. In some embodiments, the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship.

[0158] Furthermore, the file encapsulation device adds corresponding metadata information based on the specific encapsulation method used to indicate the necessary information for decoding the file track. For example, it indicates the first relationship between different point cloud attribute data, such as codec dependency and presentation association, and indicates the attribute information contained in the file track, namely the number of attribute types and the number of attributes.

[0159] Next, the file encapsulation device transmits the point cloud file F directly to the client based on the transmission method between the file encapsulation device (e.g., server) and the file decapsulation device (e.g., client). Alternatively, the point cloud file F is sliced ​​to obtain a set Fs, and based on the user's needs, the file track data required by the user in the corresponding slice is transmitted to the user.

[0160] The file decapsulation device decapsulates, decodes and presents the received files.

[0161] In the embodiment of the present application, the file decapsulation device determines the first attribute data to be decoded in the point cloud file in at least the following ways:

[0162] Mode 1: The file decoding and packaging device receives the complete point cloud file F, and determines the first attribute data to be decoded based on the relevant metadata and indication information of the point cloud file.

[0163] Mode 2: The file decoding and encapsulation device receives the slice file Fs, and determines the first attribute data to be decoded based on the relevant metadata and indication information of the received file slice.

[0164] Method 3: The file decapsulation device obtains the point cloud component description information of the point cloud file, where the point cloud component description information is used to describe at least one of the type of point cloud components included in each track of the point cloud file, a type list of attribute components and a quantity list of different types of attribute components, an attribute group identifier to which the attribute component belongs, and description information of the attribute group; the file decapsulation device determines the first attribute data to be decoded in the point cloud based on the point cloud component description information.

[0165] In this approach, the point cloud component description, also known as the AVS PCC component descriptor, identifies the type of point cloud component in the Component Adaptation Set. The point cloud component descriptor is an EssentialProperty element with its @schemeIdUri attribute set to "urn:avs:pccs:2022:component".

[0166] At the Adaptation Set level, each point cloud component present in the Representation of a Component Adaptation Set shall be described by a point cloud component descriptor.

[0167] Exemplarily, the point cloud component descriptor should include the elements and attributes defined in Table 2.

[0168] Table 2 Point cloud component description sub-attributes

[0169] As can be seen from Table 2 above, in the embodiment of the present application, the DASH signaling is extended, specifically the point cloud component descriptor is extended. When the type of the point cloud component is an attribute component, a list indicating the number of attribute components of different types, the attribute group identifier to which the attribute component belongs, and the description information of the attribute group are added. That is, in the embodiment of the present application, if the value of component@type is 'attr', the point cloud component descriptor includes not only component@attr_type, which is used to indicate the type list of attribute components, but also component@attr_num, which indicates the number list of attribute components of the corresponding type, component@attr_group_id, which indicates the attribute group identifier to which the point cloud attribute component belongs, and component@attr_group_label, which indicates the description information of the attribute group to which the point cloud attribute component belongs.

[0170] Based on this, the file decapsulation device determines the first attribute data to be decoded in the point cloud based on the received point cloud component description information (i.e., the point cloud component descriptor). For example, based on the point cloud component description information, the type of component included in each track in the point cloud file is determined. If the component included in the track is an attribute component, the first attribute data to be decoded is further determined based on the type list of attribute components included in the track and the number list of different types of attribute components, as well as the attribute group identifier to which the attribute component belongs and the description information of the attribute group.

[0171] In some embodiments, if the point cloud data includes N groups of attribute data, the first attribute data may include at least one group of attribute data in the N groups of attribute data, or part of the attribute data in one group of attribute data.

[0172] Based on the above steps, the point cloud decapsulation device determines the first attribute data to be decoded in the point cloud file and then performs the following step S102.

[0173] S102: Determine dependency indication information of the first attribute data.

[0174] The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, and the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship.

[0175] As can be seen from the above, in the embodiment of the present application, some attribute data in the point cloud data depends on other attribute data when encoding, decoding or presenting. Based on this, before decoding the first attribute data, the file decapsulation device first needs to determine whether the first attribute data has a first relationship with the other attribute data. If it is determined that the first attribute data has a first relationship with the other attribute data, for example, there is a decoding dependency relationship between the first attribute data and the second attribute data, then it is also necessary to decapsulate the second attribute data from the point cloud file, and then decode the first attribute data based on the second attribute data, thereby achieving correct decoding and personalized presentation of the first attribute data while reducing the amount of decapsulation and decoding data, thereby improving the decoding efficiency of the point cloud.

[0176] Therefore, after determining the first attribute data to be decoded in the point cloud file, the file decapsulation device also needs to determine the dependency indication information of the first attribute data, which is used to indicate whether there is a first relationship between the first attribute data and the second attribute data.

[0177] The embodiment of the present application does not limit the specific method of determining the dependency indication information of the first attribute data.

[0178] In some embodiments, if the point cloud bitstream adopts component-based multi-track packaging, the dependency indication information includes at least one of the first information and the second information. In this case, the above S102 includes the following steps S102-A1 to S102-A2:

[0179] S102-A1, determining a first track where first attribute data is located;

[0180] S102-A2: Determine at least one of first information and second information corresponding to the first track, where the first information is used to indicate the number of attribute components included in the first track, and the second information is used to indicate whether the first track includes an attribute group.

[0181] As can be seen from the above, in component-based multi-track packaging, the file packaging device encapsulates attribute data with a first relationship into a single track. Based on this, in this embodiment, if the point cloud bitstream adopts multi-track packaging, the track where the first attribute data is located is first determined. For ease of description, the track where the first attribute data is located is denoted as the first track. For example, if the point cloud bitstream adopts component-based multi-track packaging, the component track where the first attribute data is located is determined as the first track. For another example, if the point cloud bitstream adopts point cloud slice-based multi-track packaging, the point cloud slice track where the first attribute data is located is determined as the first track.

[0182] Next, at least one of the first information and the second information corresponding to the first track is determined.

[0183] In an embodiment of the present application, at least one of the first information and the second information corresponding to the above-mentioned first track is included in the point cloud file. By parsing the point cloud file, at least one of the first information and the second information corresponding to the first track can be obtained.

[0184] In this way, the file decapsulation device can determine whether there is a first relationship between the first attribute data and the second attribute data based on at least one of the first information and the second information corresponding to the first track.

[0185] For example, if the first information indicates that the number of attribute components included in the first track is greater than 1, it indicates that there is a decoding dependency relationship between the first attribute data and the second attribute data;

[0186] If the first information indicates that the number of attribute components included in the first track is equal to 1, it means that there is no decoding dependency relationship between the first attribute data and the second attribute data;

[0187] If the second information indicates that the first track includes an attribute group, it indicates that there is a presentation association relationship between the first attribute data and the second attribute data;

[0188] If the second information indicates that the first track does not include an attribute group, it means that there is no presentation association relationship between the first attribute data and the second attribute data.

[0189] As can be seen from the above, determining the first relationship between the first attribute data and the second attribute data based on at least one of the first information and the second information specifically includes the following:

[0190] Example 1: If the dependency indication information of the first attribute data includes only the first information and does not include the second information, the file decapsulation device can determine whether there is a decoding dependency relationship between the first attribute data and the second attribute data based on the first information. For example, if the first information indicates that the number of attribute components included in the first track is greater than 1, then it indicates that there is a decoding dependency relationship between the first attribute data and the second attribute data; if the first information indicates that the number of attribute components included in the first track is equal to 1, then it indicates that there is no decoding dependency relationship between the first attribute data and the second attribute data.

[0191] Example 2: If the dependency indication information for the first attribute data includes only the second information and does not include the first information, the file decapsulation device may determine whether a presentation association relationship exists between the first attribute data and the second attribute data based on the second information. For example, if the second information indicates that the first track includes an attribute group, then a presentation association relationship exists between the first attribute data and the second attribute data; if the second information indicates that the first track does not include an attribute group, then no presentation association relationship exists between the first attribute data and the second attribute data.

[0192] Example 3: If the dependency indication information of the first attribute data includes the first information and the second information, the file decapsulation device can determine whether there is a decoding dependency relationship between the first attribute data and the second attribute data based on the first information, and determine whether there is a presentation association relationship between the first attribute data and the second attribute data based on the second information. For example, if the first information indicates that the number of attribute components included in the first track is greater than 1, and the second information indicates that the first track includes an attribute group, it means that there is a decoding dependency relationship and a presentation relationship relationship between the first attribute data and the second attribute data. If the first information indicates that the number of attribute components included in the first track is greater than 1, and the second information indicates that the first track does not include an attribute group, it means that there is a decoding dependency relationship and no presentation relationship relationship between the first attribute data and the second attribute data.

[0193] That is, in this embodiment, when decoding the first attribute data, if it is determined that the point cloud bitstream adopts component-based multi-track encapsulation, the file decapsulation device determines the first track where the first attribute data is located, and determines at least one of the first information and the second information corresponding to the first track, and then determines whether a first relationship exists between the first attribute data and the second attribute data based on the at least one of the first information and the second information. For example, if the first information indicates that the number of attribute components included in the first track is greater than 1, it is determined that a decoding dependency relationship exists between the first attribute data and the second attribute data in the first track; if the first information indicates that the number of attribute components included in the first track is equal to 1, it is determined that no decoding dependency relationship exists between the first attribute data and the second attribute data.

[0194] The embodiments of the present application do not limit the specific forms of expression of the first information and the second information.

[0195] In some embodiments, the first information includes first attribute component number indication information attr_num, which is used to indicate the number of attribute components included in the first track.

[0196] In some embodiments, the first information includes attribute component type quantity indication information, second attribute component type indication information attr_type_num and second attribute component number indication information attr_num, wherein the attribute component type quantity indication information attr_type_num is used to indicate the number of different types of attribute components included in the first track, and the second attribute component number indication information attr_num is used to indicate the number of attribute components of each type.

[0197] In some embodiments, the second information includes an attribute group information flag indicating whether the attribute group information is indicated.

[0198] The embodiment of the present application does not limit the specific locations of the first information and the second information in the point cloud file.

[0199] In some embodiments, at least one of the first information and the second information is located in a first track.

[0200] In some embodiments, the first track includes a component information data box, and at least one of the first information and the second information may be included in the component information data box. That is, when decoding the first attribute data, the first track where the first attribute data is located is determined, and at least one of the first information and the second information corresponding to the first track is determined from the component information data box of the first track, thereby determining whether a first relationship exists between the first attribute data and the second attribute data.

[0201] In the embodiment of the present application, the component information data box is expanded to support the implementation steps of the present application, as follows.

[0202] Extension component information data box

[0203] Data box type: 'acif'

[0204] Included in: Sample Port

[0205] Is it mandatory: No

[0206] Quantity: 0 or 1

[0207] The Component Information data box indicates the data type of the point cloud component, i.e., geometry, attributes, etc. When included in a track's sample entry, this data box indicates the type of point cloud component carried in that track. This data box also provides information related to the attribute data in the attribute component track. If the point cloud bitstream is stored as a single track, the Component Information data box should not be included in its sample entry.

[0208] In one example, the expansion of the component information data box is as follows:

[0209] Among them, the above avs_pcc_type indicates the type of component in the track, and its value is shown in Table 3 below.

[0210] Table 3 Component types

[0211] attr_num indicates the number of attribute components contained in the track.

[0212] When multi_attr_type is set to 0, it indicates that the attribute components contained in the component track are all of the same attribute type. When it is set to 1, it indicates that the attribute components contained in the component track are of different attribute types.

[0213] attr_type indicates the type of attribute component contained in the track. A value of 0 indicates a color attribute; a value of 1 indicates a reflectance attribute.

[0214] When the attr_group_info_flag value is 0, it indicates that the attribute group information is not indicated. When the value is 1, it indicates that the attribute group information is indicated, and the attribute components contained in the current component track belong to the corresponding attribute group.

[0215] attr_group_id indicates the identifier of the attribute group. Different attribute data belonging to the same attribute group have dependencies in encoding, decoding or presentation.

[0216] The above-mentioned attr_group_info_flag can be understood as the attribute group information flag, and attr_num can be understood as the first attribute component number indication information.

[0217] In another example, the expansion of the component information data box is as follows:

[0218] Here, attr_type_num indicates the number of attribute component types included in the track.

[0219] attr_type[i] indicates the type of attribute component contained in the track. A value of 0 indicates a color attribute; a value of 1 indicates a reflectance attribute.

[0220] attr_num[i] indicates the number of attribute components of the corresponding type contained in the track.

[0221] The above-mentioned attr_type_num can be understood as the attribute component type quantity indication information, and attr_num[i] can be understood as the second attribute component quantity indication information.

[0222] In some embodiments, if the attribute group information flag indicates attribute group information, and the attribute components included in the first track belong to the corresponding second attribute group, the file decapsulation device determines the identifier of the second attribute group, and based on the identifier of the second attribute group, obtains the attribute data in the second attribute group, the second attribute group including the first attribute data and the second attribute data; and decodes and presents the attribute data in the second attribute group together.

[0223] That is, in this embodiment, if attr_group_info_flag = 1, it indicates that the first track includes an attribute group, and the attribute components included in the first track all belong to the second attribute group. This indicates that the attribute data in the second attribute group has a presentation association relationship. The file decapsulation device continues to parse the above-mentioned component information data box to obtain the identifier of the second attribute group, attr_group_id. Based on the identifier of the second attribute group, attr_group_id, the attribute data included in the second attribute group can be obtained. The attribute data includes the first attribute data and the second attribute data. The attribute data included in the second attribute group is then decoded and presented together.

[0224] The expanded component information data box in the embodiment of the present application does not indicate the type of each attribute component included in the track one by one when indicating the type of each attribute component included in the track. Instead, it indicates the number of attribute component types included in the track and the number of attribute components included in each type, thereby simplifying the indication information.

[0225] In some embodiments, if the attribute component in the point cloud bitstream is encapsulated in a second track, the dependency indication information includes at least one of the third information and the fourth information. In this case, S102 includes the following steps S102-B1 to S102-B2:

[0226] S102-B1. Determine the first attribute group to which the first attribute data belongs, wherein the second track is a track for encapsulating the point cloud bit stream in a single-track encapsulation manner, or a point cloud slice track for encapsulating all components of a specific point cloud slice of the point cloud bit stream;

[0227] S102-B2. Determine at least one of third information and fourth information corresponding to the first attribute group, where the third information is used to indicate whether a decoding dependency relationship exists between attribute components in the first attribute group, and the fourth information is used to indicate whether a presentation association relationship exists between attribute components in the first attribute group.

[0228] In this embodiment, if the point cloud bitstream adopts single-track encapsulation, that is, the geometric data and attribute data of the point cloud bitstream are encapsulated in one track, or the point cloud bitstream adopts multi-track encapsulation based on point cloud slices, and all components of a specific point cloud slice in the point cloud bitstream are encapsulated in one point cloud slice track, for the convenience of description, this track is referred to as the second track. At this time, it can be seen from the above that when the single-track encapsulation or the attribute data of the point cloud bitstream is encapsulated in one track, the attribute data with the first relationship in the point cloud bitstream are divided into the same attribute group. Based on this, if the attribute components in the point cloud bitstream are encapsulated in a second track, the first attribute group to which the first attribute data belongs is first determined.

[0229] The embodiment of the present application does not limit the specific method of determining the first attribute group to which the first attribute data belongs.

[0230] In some embodiments, the second track includes description information of the attribute group, including information such as an identifier and data type of each attribute group included in the second track, and then based on the description information of the attribute group, the attribute group where the first attribute data is located is determined.

[0231] In some embodiments, the above S102-B1 includes the following steps S102-B11 to S102-B13:

[0232] S102-B11. Determine an attribute group flag corresponding to a first subsample containing first attribute data, where the attribute group flag is used to indicate whether the first subsample belongs to an attribute group.

[0233] S102-B12: If the attribute group flag indicates that the first subsample belongs to an attribute group, determine an identifier of the first attribute group to which the first subsample belongs;

[0234] S102-B13: Determine the first attribute group based on the identifier of the first attribute group.

[0235] In this implementation, the attribute group flag corresponding to the first subsample containing the first attribute data is determined. The attribute group flag is used to indicate whether the first subsample belongs to the attribute group. If the first subsample belongs to the attribute group, the identifier of the first attribute group to which the first subsample belongs is determined. Then, based on the identifier of the first attribute group, the first attribute group is determined.

[0236] Optionally, the attribute group flag can be represented by the field attr_group_flag.

[0237] Optionally, the attribute group identifier may be represented by the field attr_group_id.

[0238] The embodiment of the present application does not limit the specific location of the attribute group flag and the attribute group identifier in the point cloud file. In some embodiments, at least one of the attribute group flag and the first attribute group identifier is included in the subsample information data box of the first subsample.

[0239] In the embodiment of the present application, the subsample definition is extended. Each point cloud sample can be divided into one or more point cloud subsamples. The SubSampleInformationBox (subsample information data box) is used in the encapsulation of the point cloud data. The subsample is defined according to the value of the flag field (flags) of the subsample information data. The flag field specifies the type of subsample information in this data box, as shown below:

[0240] 0: Subsample based on point cloud data type. A subsample contains only one data type defined by AVSPCCPayloadType.

[0241] 1: Subsample based on a point cloud patch. A subsample contains only information about a point cloud patch. When the corresponding track contains a component information data box, the subsample of the corresponding track only contains the component data corresponding to the component information data box. When the corresponding track does not contain a component information data box, the subsample of the corresponding track contains all component data.

[0242] Other flag values ​​are reserved.

[0243] The extended subsample definition is as follows:

[0244] The codec_specific_parameters field of SubsampleInformationBox is defined as follows:

[0245] Among them, AVSPCPayloadType indicates the data type of the point cloud contained in the subsample. The meaning of the value is shown in Table 4 below:

[0246] Table 4 Point cloud data types

[0247] attr_type indicates the type of attribute data contained in the subsample. A value of 0 indicates color attributes; a value of 1 indicates reflectance attributes.

[0248] slice_data indicates whether the subsample contains point cloud slice data. When the value is 1, it means that the point cloud slice geometry and / or attribute type data is included; when the value is 0, it means that the point cloud parameter information is included.

[0249] slice_id indicates the ID of the point cloud slice corresponding to the data contained in the subsample.

[0250] When the attr_group_flag value is 0, it indicates that the subsample does not belong to the attribute group. When the value is 1, it indicates that the subsample belongs to the attribute group.

[0251] attr_group_id indicates the identifier of the attribute group. Different attribute data belonging to the same attribute group have dependencies in encoding, decoding or presentation.

[0252] That is to say, when encapsulating a single track or encapsulating point cloud attribute data in one track, the file decapsulation device parses the subsample information data box of the first subsample where the first attribute data is located in the second track to obtain the attribute group flag corresponding to the first subsample. If the attribute group flag indicates that the first subsample belongs to an attribute group, the subsample information data box is continued to be parsed to obtain the identifier of the first attribute group to which the first subsample belongs, and then the first attribute group is determined based on the identifier.

[0253] Next, at least one of the third information and the fourth information corresponding to the first attribute group is determined.

[0254] The embodiments of the present application do not limit the specific forms of expression of the third information and the fourth information.

[0255] In some embodiments, the third information includes an attribute dependency group flag attr_dependency_group_flag, and the fourth information includes a default attribute flag default_attr_flag. The attribute dependency group flag attr_dependency_group_flag is used to indicate whether there is a decoding dependency relationship between attribute components corresponding to the first attribute group, and the default attribute flag default_attr_flag is used to indicate whether the attribute component corresponding to the first attribute group is a default presented attribute component.

[0256] The embodiment of the present application does not limit the specific locations of the third information and the fourth information in the point cloud file.

[0257] In some embodiments, at least one of the third information and the fourth information may be included in a subsample information data box of the first subsample.

[0258] In some embodiments, the present invention adds an attribute group information data box in the second track. The attribute group information data box is used to indicate information related to the attribute group.

[0259] For example, the definition of the attribute group information data box is as follows:

[0260] Data box type: 'agin'

[0261] Contained in:SampleEntry

[0262] Is it mandatory: No

[0263] Quantity: 0 or 1

[0264] attr_group_num indicates the number of attribute groups contained in the point cloud stream corresponding to the current track.

[0265] attr_group_id indicates the identifier of the corresponding attribute group.

[0266] When the value of the default_attr_flag field is 1, it indicates that the attribute component corresponding to the current attribute group is the default attribute component among the attribute components of the corresponding type; when the value of this field is 0, it indicates that the attribute component corresponding to the current attribute group is not the default attribute component among multiple attribute components of the same type.

[0267] When the attr_dependency_group_flag value is 1, it indicates that there is a codec dependency between the attribute components corresponding to the current attribute group; when the value is 0, it indicates that there is no codec dependency between the attribute components corresponding to the current attribute group.

[0268] When the value of attr_group_label_flag is 1, it indicates the descriptive label corresponding to the current attribute group; when the value is 0, it does not indicate the descriptive label corresponding to the current attribute group.

[0269] attr_group_label indicates the description information of the current attribute group.

[0270] Among them, default_attr_flag can be understood as the above-mentioned presentation dependency flag, and attr_dependency_group_flag can be understood as the above-mentioned decoding dependency flag.

[0271] In some embodiments, for scenarios where attribute data is located in different tracks, the attribute group information data box may not be used, and the fields in the attribute group information data box may be directly included in the component information data box, as expanded as follows.

[0272] In an embodiment of the present application, the file decapsulation device determines at least one of the third information and the fourth information, and can then determine whether a first relationship exists between the first attribute data and the second attribute data. For example, if the third information indicates that there is a decoding dependency relationship between the attribute components in the first attribute group, such as attr_dependency_group_flag=1, then it is determined that the first attribute data depends on the second attribute data during decoding. For another example, if the third information indicates that there is no decoding dependency relationship between the attribute components in the first attribute group, such as attr_dependency_group_flag=0, then it is determined that the first attribute data does not depend on the second attribute data during decoding.

[0273] Based on the above steps, the file decapsulation device determines the dependency indication information of the first attribute data and then performs the following step S103.

[0274] S103: Decode the first attribute data based on the dependency indication information.

[0275] In the embodiment of the present application, the file encapsulation device encapsulates attribute data having a first relationship into a track or divides them into an attribute group. Thus, when the file decapsulation device decodes the first attribute data, it first determines the dependency indication information of the first attribute data based on the above steps to determine whether the first attribute data depends on the second attribute data during decoding.

[0276] In some embodiments, if the dependency indication information indicates that there is a decoding dependency relationship between the first attribute data and the second data, the file decapsulation device determines the second attribute data and decodes the first attribute data based on the second attribute data.

[0277] The method of determining the second attribute data includes at least the following situations:

[0278] In case 1, if the point cloud bitstream uses multi-track packaging, the first track containing the first attribute data is determined, and the attribute data in the first track other than the first attribute data is determined as the second attribute data. For example, if the point cloud bitstream uses component-based multi-track packaging, the attribute data in the component track containing the first attribute data other than the first attribute data is determined as the second attribute data. For another example, if the point cloud bitstream uses point cloud slice-based multi-track packaging, the attribute data in the point cloud slice track containing the first attribute data other than the first attribute data is determined as the second attribute data.

[0279] For example, as shown in FIG7A , if component-based multi-track packaging is used, the point cloud bitstream can be packaged into multiple component tracks, for example, into a geometry component track and N attribute component tracks. Each component track includes a component information data box. The file decapsulation device first determines the first track where the first attribute data is located, for example, attribute component track 1, and then determines at least one of the first information from the component information data box of attribute component track 1, and then determines the number of attribute components included in attribute component track 1 based on the first information. If the number of attribute components included in attribute component track 1 is greater than 1, it is determined that the first attribute data depends on the second attribute data during decoding, and the entire attribute component track 1 is decoded to obtain the decoded first attribute data. If the number of attribute components included in attribute component track 1 is equal to 1, it is determined that the first attribute data does not depend on the second attribute data during decoding, and the first attribute data can be decoded separately to obtain the decoded first attribute data.

[0280] Case 2: If the attribute components in the point cloud bitstream are encapsulated in a second track, the first attribute group to which the first attribute data belongs is determined, and the other attribute data in the first attribute group except the first attribute data are determined as the second attribute data. For example, if the point cloud bitstream adopts a single-track encapsulation, the first attribute group to which the first attribute data belongs is determined in the single track, and the other attribute data in the first attribute group except the first attribute data are determined as the second attribute data. For another example, if the point cloud bitstream adopts a multi-track encapsulation based on point cloud slices, and all components of a specific point cloud slice of the point cloud bitstream are encapsulated in a point cloud slice track, the first attribute group to which the first attribute data belongs is determined in the point cloud slice track, and the other attribute data in the first attribute group except the first attribute data are determined as the second attribute data.

[0281] Exemplarily, as shown in FIG7B , if a component-based single-track encapsulation is adopted, the geometric data and attribute data of the point cloud bitstream can be encapsulated in one track. Specifically, different areas of the point cloud bitstream are encapsulated in different subsamples. For example, the geometric component data, attribute component data 1, attribute component data 2, attribute component data 4, and attribute component data 5 of the point cloud bitstream are encapsulated in one sample 1, and there is a first relationship between attribute component data 2 and attribute component data 3 in sample 1, and they are then divided into the same attribute group. For example, attribute component data 2 and attribute component data 3 are divided into attribute group 1, and attribute component data 2 and attribute component data 3 are identified as belonging to attribute group 1, that is, identified as attr_group_id=1. When decoding the first attribute data, the file decapsulation device first determines the first subsample where the first attribute data is located, and then determines whether the first subsample belongs to the attribute group. If it belongs to the attribute group, the identifier of the first attribute group included in the first subsample is determined. It is assumed that the identifier of the first attribute group is determined to be attr_group_id=1. Next, at least one of the third information and the fourth information corresponding to the first attribute group is determined from the descriptive information of each attribute group included in the attribute group information data box, and then, based on at least one of the third information and the fourth information corresponding to the first attribute group, it is determined whether a first relationship exists between the first attribute data and the second attribute data. For example, if the third information includes attr_dependency_group_flag, and if attr_dependency_group_flag = 1, it is determined that the first attribute data depends on the second attribute data during decoding, and then the second attribute data is determined from the first attribute group, and the first attribute data is decoded based on the second attribute data. For another example, if the second information indicates that there is no decoding dependency between the attribute components in the first attribute group, such as attr_dependency_group_flag = 0, it is determined that the first attribute data does not depend on the second attribute data during decoding, and then the first attribute data is decoded separately.

[0282] In some embodiments, if the dependency indication information indicates that there is no decoding dependency between the first attribute data and the second attribute data, the first attribute data information is decoded separately. For example, as can be seen from the above, if there is no encoding dependency between the first attribute data and the second attribute data, in the case of multi-track encapsulation, the file encapsulation device encapsulates the first attribute data separately in a track, and in the case of single-track encapsulation, the file encapsulation device treats the first attribute data as a separate attribute component and does not divide it into an attribute group. In this way, when decapsulating, the file decapsulation device decapsulates and decodes the track where the first attribute data is located in the case of multi-track encapsulation to obtain the decoded first attribute data, and in the case of single-track encapsulation, the first attribute data is decoded separately to obtain the decoded first attribute data.

[0283] In some embodiments, if the dependency indication information indicates that there is a presentation association relationship between the first attribute data and the second attribute data, the second attribute data is determined, and the first attribute data and the second attribute data are decoded and presented together.

[0284] In some embodiments, if the dependency indication information indicates that there is no presentation association relationship between the first attribute data and the second attribute data, the first attribute data is presented separately.

[0285] In the embodiment of the present application, by determining the dependency indication information of the first attribute data, the first attribute data is correctly decoded and the decoded first attribute data is presented, thereby achieving accurate decoding and personalized presentation of part of the attribute data in the point cloud bitstream.

[0286] The point cloud file decapsulation method provided in an embodiment of the present application determines first attribute data to be decoded in a point cloud file, and determines dependency indication information for the first attribute data, wherein the dependency indication information is used to indicate whether a first relationship exists between the first attribute data and second attribute data, wherein the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship. Based on the dependency indication information, the first attribute data is decoded. That is, the present application indicates the encoding and decoding dependency relationship or presentation association relationship between different point cloud attribute data through the dependency indication information, thereby supporting partial access, correct decoding, and personalized presentation of the point cloud bitstream.

[0287] The above describes the decapsulation method of point cloud files. The following describes the encapsulation method of point cloud files.

[0288] FIG8 is a flow chart of a point cloud file packaging method provided in an embodiment of the present application. As shown in FIG8 , the method includes the following steps:

[0289] S201: Obtain point cloud bit stream.

[0290] The point cloud bitstream includes N sets of attribute data, where N is a positive integer.

[0291] In an embodiment of the present application, point cloud data is encoded to obtain a point cloud bitstream. In some embodiments, the point cloud bitstream is also referred to as a point cloud codestream. The point cloud includes geometric information and attribute information. The geometric information is encoded to obtain a geometric codestream (or geometric bitstream), and the attribute information is encoded to obtain an attribute codestream (or attribute bitstream). The point cloud bitstream of an embodiment of the present application includes at least an attribute bitstream. For example, the point cloud bitstream of an embodiment of the present application includes both a geometric bitstream and an attribute bitstream, or the point cloud bitstream of an embodiment of the present application includes only an attribute bitstream.

[0292] In some embodiments, the point cloud bitstream of the present application includes geometric data and N sets of attribute data, where N is a positive integer. The geometric data can be understood as the above-mentioned geometric bitstream, and the attribute data can be understood as the attribute bitstream.

[0293] S202 . For first attribute data to be encapsulated in the N groups of attribute data, encapsulate the first attribute data based on a first relationship between the first attribute data and the second attribute data, and determine dependency indication information of the first attribute data to obtain a point cloud file.

[0294] The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, and the first relationship includes at least one of an encoding dependency relationship and a presentation association relationship.

[0295] In an embodiment of the present application, after the point cloud encapsulation device (such as a server) obtains the point cloud bit stream, it encapsulates the point cloud bit stream to obtain a point cloud file.

[0296] In this embodiment of the present application, when encapsulating first attribute data to be encapsulated from N sets of attribute data, the first attribute data is encapsulated based on whether a first relationship exists between the first attribute data and second attribute data. The first attribute data may be at least one set of attribute data to be encapsulated from any of the N sets of attribute data, or a portion of attribute data from a set of attribute data.

[0297] In the embodiment of the present application, the specific method of encapsulating the first attribute data is not limited based on whether a first relationship exists between the first attribute data and the second attribute data.

[0298] In some embodiments, if the first attribute data and the second attribute data have a presentation dependency relationship, the first attribute data and the second attribute data are encapsulated in the same attribute track, or divided into the same attribute group.

[0299] In some embodiments, if the first attribute data and the second attribute data do not have a presentation dependency relationship, the first attribute data and the second attribute data can be encapsulated in different attribute tracks, for example, the first attribute data can be encapsulated in an attribute track alone, or the first attribute data can be used as an attribute component alone without identifying which attribute group it belongs to.

[0300] In some embodiments, if the dependency indication information indicates that there is no coding dependency relationship between the first attribute data and the second attribute data, the first attribute data is packaged separately. For example, in the case of multi-track packaging, the first attribute data is packaged separately in a track, and in the case of single-track packaging, the first attribute data is treated as a separate attribute component and is not divided into an attribute group.

[0301] In some embodiments, if there is a coding dependency relationship between the first attribute data and the second attribute data, the above S202 includes the following steps S202-A1 to S202-A2:

[0302] S202-A1, obtaining second attribute data;

[0303] S202-A2: Encapsulate the first attribute data based on the second attribute data.

[0304] As can be seen from the above, the point cloud packaging device can encapsulate the point cloud bitstream in at least one of the following ways: single-track encapsulation, component-based multi-track encapsulation, and point cloud slice-based multi-track encapsulation. In this case, the above S202-A2 includes the following ways:

[0305] In some embodiments, if the point cloud bitstream adopts component-based multi-track packaging, the second attribute data and the first attribute data are packaged in the first track.

[0306] That is to say, in this embodiment, if the point cloud encapsulation device adopts component-based multi-track encapsulation, the point cloud attribute data with a first relationship in the point cloud bit stream can be encapsulated in the same component track. In this way, the decoding end can only decode the attribute data in the same component track during decoding, thereby achieving partial access to the point cloud attribute data and correct decoding, thereby achieving personalized presentation of the point cloud attribute data. For example, the point cloud bit stream includes geometric data and N groups of attribute data. The point cloud encapsulation device can encapsulate the geometric data into a separate component track, encapsulate the M groups of attribute data that have a first relationship with each other in the N groups of attribute data into the same attribute component track, and encapsulate the 1 group of attribute data that does not have a first relationship with other attribute data in the N groups of attribute data into a separate attribute component track.

[0307] Alternatively, if the point cloud encapsulation device adopts multi-track encapsulation based on point cloud slices, then for a specific point cloud slice, the point cloud attribute data with a first relationship in the point cloud bit stream can be encapsulated in the same point cloud slice track. In this way, the decoding end can only decode the attribute data in the same point cloud slice track during decoding, thereby achieving partial access to the point cloud attribute data and correct decoding, thereby achieving personalized presentation of the point cloud attribute data. For example, for a specific point cloud slice, its corresponding geometric and attribute data are located in multiple different point cloud slice tracks, such as encapsulating the geometric data into a separate point cloud slice track, encapsulating M groups of attribute data that have a first relationship with each other in N groups of attribute data into a point cloud slice track, and encapsulating 1 group of attribute data that does not have a first relationship with other attribute data in N groups of attribute data into a separate point cloud slice track.

[0308] In multi-track packaging, attribute data with a first relationship is packaged in one track. For example, if the point cloud bitstream uses component-based multi-track packaging, the second attribute data and the first attribute data are packaged in the first component track. For another example, if the point cloud bitstream uses point cloud slice-based multi-track packaging, the second attribute data and the first attribute data are packaged in the first point cloud slice track.

[0309] In the embodiment of the present application, after the first attribute data and the second attribute data are encapsulated in the first track, it is also necessary to determine the dependency indication information of the first attribute data.

[0310] The embodiment of the present application does not limit the specific method of determining the dependency indication information of the first attribute data.

[0311] In some embodiments, a separate flag is set to indicate whether the first attribute data depends on the second attribute data when encoding or presenting.

[0312] In some embodiments, if the point cloud bitstream adopts component-based multi-track encapsulation, the dependency indication information includes at least one of the first information and the second information. In this case, determining the dependency indication information of the first attribute data in S202 includes the following step S202-B1:

[0313] S202-B1. Determine at least one of first information and second information corresponding to a first track, where the first information is used to indicate the number of attribute components included in the first track, and the second information is used to indicate whether the first track includes an attribute group.

[0314] For example, if there is a coding dependency relationship between the first attribute data and the second attribute data, it is determined that the first information indicates that the number of attribute components included in the first track is greater than 1;

[0315] If there is no coding dependency relationship between the first attribute data and the second attribute data, determining that the number of attribute components included in the first track indicated by the first information is equal to 1;

[0316] If a presentation association relationship exists between the first attribute data and the second attribute data, determining that the second information indicates that the first track includes an attribute group;

[0317] If there is no presentation association relationship between the first attribute data and the second attribute data, it is determined that the second information indicates that the first track does not include the attribute group.

[0318] The embodiment of the present application does not limit the specific form of the first information.

[0319] In some embodiments, the first information includes first attribute component number indication information, wherein the first attribute component number indication information is used to indicate the number of attribute components included in the first track.

[0320] In some embodiments, the first information includes attribute component type quantity indication information and second attribute component quantity indication information, wherein the attribute component type quantity indication information is used to indicate the number of different types of attribute components included in the first track, and the second attribute component quantity indication information is used to indicate the number of attribute components of each type.

[0321] In some embodiments, the second information includes an attribute group information flag, and the attribute group information flag indicates whether the attribute group information is indicated.

[0322] In some embodiments, if there is a presentation association relationship between the first attribute data and the second attribute data, and the first attribute data and the second attribute data are divided into a second attribute group, the file encapsulation device also determines the identifier of the second attribute group; and then adds the identifier of the second attribute group to the point cloud file.

[0323] The embodiment of the present application does not limit the specific locations of the first information and the second information in the point cloud file.

[0324] In some embodiments, the first information and the second information are included in a component information data box in the first track.

[0325] In the embodiment of the present application, the component information data box is expanded to support the implementation steps of the present application, as follows.

[0326] Extension component information data box

[0327] Data box type: 'acif'

[0328] Included in: Sample Port

[0329] Is it mandatory: No

[0330] Quantity: 0 or 1

[0331] The Component Information data box indicates the data type of the point cloud component, i.e., geometry, attributes, etc. When included in a track's sample entry, this data box indicates the type of point cloud component carried in that track. This data box also provides information related to the attribute data in the attribute component track. If the point cloud bitstream is stored as a single track, the Component Information data box should not be included in its sample entry.

[0332] In one example, the expansion of the component information data box is as follows:

[0333] Among them, the above avs_pcc_type indicates the type of component in the track, and its value is shown in Table 3 below.

[0334] attr_num indicates the number of attribute components contained in the track.

[0335] When multi_attr_type is set to 0, it indicates that the attribute components contained in the component track are all of the same attribute type. When it is set to 1, it indicates that the attribute components contained in the component track are of different attribute types.

[0336] attr_type indicates the type of attribute component contained in the track. A value of 0 indicates a color attribute; a value of 1 indicates a reflectance attribute.

[0337] When the attr_group_info_flag value is 0, it indicates that the attribute group information is not indicated. When the value is 1, it indicates that the attribute group information is indicated, and the attribute components contained in the current component track belong to the corresponding attribute group.

[0338] attr_group_id indicates the identifier of the attribute group. Different attribute data belonging to the same attribute group have dependencies in encoding, decoding or presentation.

[0339] The above-mentioned attr_num can be understood as the first attribute component number indication information, and attr_group_info_flag can be understood as the attribute group information flag.

[0340] In another example, the expansion of the component information data box is as follows:

[0341] Here, attr_type_num indicates the number of attribute component types included in the track.

[0342] attr_type[i] indicates the type of attribute component contained in the track. A value of 0 indicates a color attribute; a value of 1 indicates a reflectance attribute.

[0343] attr_num[i] indicates the number of attribute components of the corresponding type contained in the track.

[0344] The above-mentioned attr_type_num can be understood as the attribute component type quantity indication information, and attr_num[i] can be understood as the second attribute component quantity indication information.

[0345] In this embodiment of the present application, when a point cloud bitstream is packaged in a multi-track manner, different components in the point cloud bitstream are packaged into multiple file tracks, namely component tracks. Component tracks include a point cloud geometry track and a point cloud attribute track. Each sample in a component track contains one or more point cloud slices of the same type.

[0346] In multi-track packaging mode, component tracks must meet the following constraints:

[0347] a) It must contain a geometry component track and the geometry component track serves as the access point.

[0348] b) It may contain 0 or more attribute component tracks, and the track_in_movie field in the component track should be set to 0.

[0349] c) The sample entry of each component track must contain a component information data box AVSPCCComponentInfoBox to indicate the point cloud component data type contained in the component track.

[0350] d) Geometry component tracks are linked to corresponding attribute component tracks via track indices.

[0351] e) Multiple attribute components with codec dependencies must be included in the same attribute component track.

[0352] Different component tracks of the same point cloud sequence are time-aligned. Samples of different component tracks corresponding to the same point cloud frame should have the same presentation time. When a parameter set exists in a track sample, the decoding time of the parameter set should be equal to or earlier than the decoding time of the corresponding point cloud component data. When all parameter sets of all component tracks are included in a track sample, the sample containing the sequence header parameter set should be equal to or earlier than the sample containing the geometry header parameter set or the attribute header parameter set. In addition, all component tracks of the same point cloud sequence should have the same implicit or explicit edit list.

[0353] When the point cloud bitstream contains multiple point cloud slices, the point cloud bitstream can be encapsulated into multiple tracks based on the point cloud slices, including a point cloud slice base track and a point cloud slice track.

[0354] In the multi-track packaging mode based on point cloud slices, the point cloud slice base track and point cloud slice track must meet the following constraints:

[0355] a) A point cloud slice base track must be included, which contains all the geometry header and attribute header parameter sets in the point cloud bitstream and serves as the access point.

[0356] b) Contains one or more point cloud slice tracks.

[0357] c) The point cloud slice base track is associated to the corresponding point cloud slice track through the track reference.

[0358] When different components of a point cloud slice are encapsulated using different point cloud slice tracks, there must be one or more point cloud slice tracks containing geometry component data, and there may be zero, one, or more point cloud slice tracks containing attribute component data. The track_in_movie field of a point cloud slice track containing attribute component data should be 0. Furthermore, multiple attribute components with codec dependencies must be contained in the same point cloud slice track containing attribute component data.

[0359] For example, as shown in Figure 7A, if component-based multi-track packaging is used, the point cloud bitstream can be packaged into multiple component tracks, such as a geometry component track and N attribute component tracks. Each component track includes a component information data box that indicates the point cloud component type included in the current track and, if it is an attribute component, the number and type of the attribute components.

[0360] In multi-track packaging, each attribute component track is independent of other attribute component tracks during encoding and decoding. The information in the component information data box can be used to obtain label information for different attribute component tracks. Optionally, the attribute group information data box is included in the track sample entry to further obtain attribute group label information, that is, the relationship between different attribute components in presentation. This metadata allows the decoder to select the required attribute component track for presentation.

[0361] In some embodiments, if the attribute components in the point cloud bitstream are encapsulated in a second track, the second attribute data and the first attribute data are encapsulated and identified as belonging to the first attribute group, and the second track is a track that encapsulates the point cloud bitstream using a single-track encapsulation method, or a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream.

[0362] That is, in the embodiment of the present application, if the point cloud encapsulation device adopts single-track encapsulation, or encapsulates all components of a specific point cloud slice of the point cloud bitstream into a single point cloud slice track, then the point cloud attribute data having the first relationship can be divided into the same attribute group. In this way, when decoding, the decoding end can only decode the attribute data belonging to the same attribute group, thereby achieving partial access to the point cloud attribute data and correct decoding, thereby realizing personalized presentation of the point cloud attribute data.

[0363] In some embodiments, the embodiments of the present application further include: determining an attribute group flag corresponding to the first subsample where the first attribute data is located, the attribute group flag being used to indicate whether the first subsample belongs to an attribute group; if the attribute group flag indicates that the first subsample belongs to an attribute group, determining an identifier of the first attribute group to which the first subsample belongs; and encapsulating the attribute group flag and the identifier of the first attribute group in a second track.

[0364] Optionally, the attribute group flag can be represented by the field attr_group_flag.

[0365] Optionally, the attribute group identifier may be represented by the field attr_group_id.

[0366] The embodiment of the present application does not limit the specific location of the attribute group flag and the attribute group identifier in the point cloud file. In some embodiments, at least one of the attribute group flag and the first attribute group identifier is included in the subsample information data box of the first subsample.

[0367] In the embodiment of the present application, the subsample definition is extended. Each point cloud sample can be divided into one or more point cloud subsamples. The SubSampleInformationBox (subsample information data box) is used in the encapsulation of the point cloud data. The subsample is defined according to the value of the flag field (flags) of the subsample information data. The flag field specifies the type of subsample information in this data box, as shown below:

[0368] 0: Subsample based on point cloud data type. A subsample contains only one data type defined by AVSPCCPayloadType.

[0369] 1: Subsample based on a point cloud patch. A subsample contains only information about a point cloud patch. When the corresponding track contains a component information data box, the subsample of the corresponding track only contains the component data corresponding to the component information data box. When the corresponding track does not contain a component information data box, the subsample of the corresponding track contains all component data.

[0370] Other flag values ​​are reserved.

[0371] The extended subsample definition is as follows:

[0372] The codec_specific_parameters field of SubsampleInformationBox is defined as follows:

[0373] Among them, AVSPCPayloadType indicates the data type of the point cloud contained in the subsample, and the meaning of the value is shown in Table 4 below.

[0374] attr_type indicates the type of attribute data contained in the subsample. A value of 0 indicates color attributes; a value of 1 indicates reflectance attributes.

[0375] slice_data indicates whether the subsample contains point cloud slice data. When the value is 1, it means that the point cloud slice geometry and / or attribute type data is included; when the value is 0, it means that the point cloud parameter information is included.

[0376] slice_id indicates the ID of the point cloud slice corresponding to the data contained in the subsample.

[0377] When the attr_group_flag value is 0, it indicates that the subsample does not belong to the attribute group. When the value is 1, it indicates that the subsample belongs to the attribute group.

[0378] attr_group_id indicates the identifier of the attribute group. Different attribute data belonging to the same attribute group have dependencies in encoding, decoding or presentation.

[0379] That is to say, when encapsulating a single track or encapsulating point cloud attribute data in one track, the file encapsulation device expands the subsample information data box of the first subsample where the first attribute data is located in the second track, specifically, adds the attribute group flag corresponding to the first subsample. If the attribute group flag indicates that the first subsample belongs to an attribute group, the identifier of the first attribute group to which the first subsample belongs will continue to be added.

[0380] The file encapsulation device encapsulates the first attribute data and the second attribute data in the second track, and after the first attribute data and the second attribute data are identified as belonging to the first attribute group, it is also necessary to determine dependency indication information of the first attribute data.

[0381] In some embodiments, a flag bit is used to indicate whether a first relationship exists between the first attribute data and the second attribute data.

[0382] In some embodiments, if the attribute component in the point cloud bitstream is encapsulated in a second track, the dependency indication information includes at least one of the third information and the fourth information. In this case, determining the dependency indication information of the first attribute data in S202 includes the following steps S202-C1:

[0383] S202-C1. Determine at least one of third information and fourth information corresponding to the first attribute group, where the third information is used to indicate whether there is a coding dependency relationship between attribute components in the first attribute group, and the fourth information is used to indicate whether there is a presentation association relationship between attribute components in the first attribute group.

[0384] For example, if the first attribute data depends on the second attribute data during encoding, it is determined that the second information indicates that the attribute components in the first attribute group have a dependency relationship during encoding; if the first attribute data does not depend on the second attribute data during encoding, it is determined that the second information indicates that the attribute components in the first attribute group do not have a dependency relationship during encoding.

[0385] The embodiments of the present application do not limit the specific forms of expression of the third information and the fourth information.

[0386] In some embodiments, the third information includes an attribute dependency group flag, and the fourth information includes a default attribute flag. The attribute dependency group flag is used to indicate whether there is a coding dependency relationship between the attribute components corresponding to the first attribute group, and the default attribute flag is used to indicate whether the attribute component corresponding to the first attribute group is the default presented attribute component.

[0387] The embodiment of the present application does not limit the specific locations of the third information and the fourth information in the point cloud file.

[0388] In some embodiments, the present invention adds an attribute group information data box in the second track, the attribute group information data box being used to indicate information related to the attribute group. At least one of the third information and the fourth information is included in the attribute component information data box in the second track.

[0389] For example, the definition of the attribute group information data box is as follows:

[0390] Data box type: 'agin'

[0391] Contained in:SampleEntry

[0392] Is it mandatory: No

[0393] Quantity: 0 or 1

[0394] attr_group_num indicates the number of attribute groups contained in the point cloud stream corresponding to the current track.

[0395] attr_group_id indicates the identifier of the corresponding attribute group.

[0396] When the value of the default_attr_flag field is 1, it indicates that the attribute component corresponding to the current attribute group is the default attribute component among the attribute components of the corresponding type; when the value of this field is 0, it indicates that the attribute component corresponding to the current attribute group is not the default attribute component among multiple attribute components of the same type.

[0397] When the attr_dependency_group_flag value is 1, it indicates that there is a codec dependency between the attribute components corresponding to the current attribute group; when the value is 0, it indicates that there is no codec dependency between the attribute components corresponding to the current attribute group.

[0398] When the value of attr_group_label_flag is 1, it indicates the descriptive label corresponding to the current attribute group; when the value is 0, it does not indicate the descriptive label corresponding to the current attribute group.

[0399] attr_group_label indicates the description information of the current attribute group.

[0400] Among them, default_attr_flag can be understood as the above-mentioned presentation dependency flag, and attr_dependency_group_flag can be understood as the above-mentioned decoding dependency flag.

[0401] That is, the embodiment of the present application introduces an attribute component information data box for indicating attribute component information.

[0402] In some embodiments, for scenarios where attribute data is located in different tracks, the attribute group information data box may not be used, and the fields in the attribute group information data box may be directly included in the component information data box, as expanded as follows.

[0403] In an embodiment of the present application, if the attribute component in the point cloud bitstream is encapsulated in a second track, the other attribute data in the first attribute group where the first attribute data is located, except for the first attribute data, is determined as the second attribute data. For example, if the point cloud bitstream adopts single-track encapsulation, the first attribute group where the first attribute data is located is determined in the single track, and the other attribute data in the first attribute group, except for the first attribute data, is determined as the second attribute data. For another example, if the point cloud bitstream adopts multi-track encapsulation based on point cloud slices, and the attribute data of the point cloud bitstream is encapsulated in a point cloud slice track, the first attribute group where the first attribute data is located is determined in the point cloud slice track, and the other attribute data in the first attribute group, except for the first attribute data, is determined as the second attribute data. Furthermore, the first attribute data and the second attribute data are divided into the same attribute group.

[0404] Exemplarily, as shown in FIG7B , if a component-based single-track encapsulation is adopted, the geometric data and attribute data of the point cloud bitstream can be encapsulated in one track. Specifically, different areas of the point cloud bitstream can be encapsulated in different subsamples. For example, the geometric component data, attribute component data 1, attribute component data 2, attribute component data 4, and attribute component data 5 of the point cloud bitstream are encapsulated in one sample 1, and there is a first relationship between attribute component data 2 and attribute component data 3 in sample 1. They can be divided into the same attribute group, for example, divided into attribute group 1, and the identifier attr_group_id=1 of attribute group 1 is added to both attribute component data 2 and attribute component data 3.

[0405] In other words, when encapsulating attribute data within a single track or within a point cloud slice track, different attribute components can be grouped using the aforementioned subsample definition. For example, the track sample entry contains an attribute group information data box, which can reveal information such as the attribute group's codec dependencies and attribute labels. This allows the relationship between different attribute components in codecs and presentation to be understood. Based on this metadata, the decoder can select the required subsample for decoding.

[0406] The file encapsulation device adds corresponding metadata information according to the specific encapsulation method adopted to indicate the necessary information required for decoding the file track.

[0407] Next, the file encapsulation device transmits the point cloud file F directly to the client based on the transmission method between the file encapsulation device (e.g., server) and the file decapsulation device (e.g., client). Alternatively, the point cloud file F is sliced ​​to obtain a set Fs, and based on the user's needs, the file track data required by the user in the corresponding slice is transmitted to the user.

[0408] In some embodiments, the present application extends DASH signaling, specifically extending the point cloud component descriptor. Specifically, the present application further includes determining point cloud component description information for a point cloud file, where the point cloud component description information is used to describe at least one of the type of point cloud components included in each track of the point cloud file, a list of attribute component types and a list of the number of attribute components of different types, an identifier of an attribute group to which the attribute component belongs, and description information of the attribute group.

[0409] The point cloud component description, also known as the AVS PCC component descriptor, identifies the type of point cloud component in the Component Adaptation Set. The point cloud component descriptor is an EssentialProperty element with its @schemeIdUri attribute set to "urn:avs:pccs:2022:component".

[0410] At the Adaptation Set level, each point cloud component present in the Representation of a Component Adaptation Set shall be described by a point cloud component descriptor.

[0411] Exemplarily, the point cloud component descriptor should include the elements and attributes defined in Table 2.

[0412] As can be seen from Table 2 above, in the embodiment of the present application, the DASH signaling is extended, specifically the point cloud component descriptor is extended. When the type of the point cloud component is an attribute component, a list indicating the number of attribute components of different types, the attribute group identifier to which the attribute component belongs, and the description information of the attribute group are added. That is, in the embodiment of the present application, if the value of component@type is 'attr', the point cloud component descriptor includes not only component@attr_type, which is used to indicate the type list of attribute components, but also component@attr_num, which is used to indicate the number list of attribute components of the corresponding type, component@attr_group_id, which indicates the attribute group identifier to which the point cloud attribute component belongs, and component@attr_group_label, which indicates at least one of the description information of the attribute group to which the point cloud attribute component belongs.

[0413] Based on this, the file decapsulation device determines the first attribute data to be decoded in the point cloud based on the received point cloud component description information (ie, point cloud component descriptor).

[0414] The point cloud file encapsulation method provided in the embodiment of the present application supports partial access, correct decoding and personalized presentation of attribute data in the point cloud bitstream by encapsulating attribute data with a first relationship in the point cloud bitstream in the same track or dividing it into the same attribute group, and by determining whether dependency indication information indicates whether the first relationship exists between the attribute data.

[0415] It should be understood that Figures 6 to 8 are merely examples of the present application and should not be construed as limiting the present application.

[0416] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0417] The above describes in detail the method embodiment of the present application in conjunction with Figures 6 to 8 , and the following describes in detail the device embodiment of the present application in conjunction with Figures 9 to 10 .

[0418] FIG9 is a schematic diagram of the structure of a file decapsulation device provided in an embodiment of the present application. The file decapsulation device 10 includes:

[0419] A data determining unit 11 is configured to determine first attribute data to be decoded in a point cloud file, where the point cloud file is a file obtained by encapsulating a point cloud bit stream;

[0420] an information determining unit 12, configured to determine dependency indication information of the first attribute data, the dependency indication information being used to indicate whether a first relationship exists between the first attribute data and the second attribute data, the first relationship including at least one of a decoding dependency relationship and a presentation association relationship;

[0421] The decoding unit 13 is configured to decode the first attribute data based on the dependency indication information.

[0422] In some embodiments, the decoding unit 13 is specifically used to determine the second attribute data if the dependency indication information indicates that there is a decoding dependency relationship between the first attribute data and the second attribute data; decode the first attribute data based on the second attribute data; if the dependency indication information indicates that there is no decoding dependency relationship between the first attribute data and the second attribute data, decode the first attribute data separately.

[0423] In some embodiments, the decoding unit 13 is specifically used to determine the first track where the first attribute data is located if the point cloud bitstream adopts component-based multi-track encapsulation, and determine the other attribute data in the first track except the first attribute data as the second attribute data; if the attribute component in the point cloud bitstream is encapsulated in a second track, determine the first attribute group to which the first attribute data belongs, and determine the other attribute data in the first attribute group except the first attribute data as the second attribute data, wherein the second track is a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or is a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream.

[0424] In some embodiments, if the point cloud bitstream adopts component-based multi-track encapsulation, the dependency indication information includes at least one of the first information and the second information, and the information determination unit 12 is specifically used to determine the first track where the first attribute data is located; determine at least one of the first information and the second information corresponding to the first track, the first information is used to indicate the number of attribute components included in the first track, and the second information is used to indicate whether the first track includes an attribute group.

[0425] In some embodiments, if the first information indicates that the number of attribute components included in the first track is greater than 1, it indicates that there is a decoding dependency relationship between the first attribute data and the second attribute data;

[0426] If the first information indicates that the number of attribute components included in the first track is equal to 1, it means that there is no decoding dependency relationship between the first attribute data and the second attribute data;

[0427] If the second information indicates that the first track includes the attribute group, it means that there is a presentation association relationship between the first attribute data and the second attribute data;

[0428] If the second information indicates that the first track does not include the attribute group, it means that there is no presentation association relationship between the first attribute data and the second attribute data.

[0429] In some embodiments, the first information includes first attribute component number indication information, wherein the first attribute component number indication information is used to indicate the number of attribute components included in the first track; or, the first information includes attribute component type number indication information and second attribute component number indication information, wherein the attribute component type number indication information is used to indicate the number of different types of attribute components included in the first track, and the second attribute component number indication information is used to indicate the number of attribute components of each type; the second information includes an attribute group information flag, and the attribute group information flag indicates whether attribute group information is indicated.

[0430] In some embodiments, if the attribute group information flag indicates attribute group information, and the attribute components included in the first track belong to the corresponding second attribute group, the decoding unit 13 is further configured to determine an identifier of the second attribute group; obtain attribute data in the second attribute group based on the identifier of the second attribute group, the second attribute group including the first attribute data and the second attribute data; and decode and present the attribute data in the second attribute group together.

[0431] In some embodiments, if the attribute components in the point cloud bitstream are encapsulated in a second track, the dependency indication information includes at least one of the third information and the fourth information, then the information determination unit 12 is specifically used to determine the first attribute group to which the first attribute data belongs, and the second track is a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream; determine at least one of the third information and the fourth information corresponding to the first attribute group, the third information is used to indicate whether there is a decoding dependency relationship between the attribute components in the first attribute group, and the fourth information is used to indicate whether there is a presentation association relationship between the attribute components in the first attribute group.

[0432] In some embodiments, the information determination unit 12 is specifically used to determine an attribute group flag corresponding to a first subsample in which the first attribute data is located, where the attribute group flag is used to indicate whether the first subsample belongs to an attribute group; if the attribute group flag indicates that the first subsample belongs to an attribute group, then determine an identifier of the first attribute group to which the first subsample belongs; and determine the first attribute group based on the identifier of the first attribute group.

[0433] In some embodiments, the third information includes an attribute dependency group flag, and the fourth information includes a default attribute flag, wherein the attribute dependency group flag is used to indicate whether there is a decoding dependency relationship between the attribute components corresponding to the first attribute group, and the default attribute flag is used to indicate whether the attribute component corresponding to the first attribute group is a default presented attribute component.

[0434] In some embodiments, the data determination unit 11 is specifically used to obtain point cloud component description information of the point cloud file, the point cloud component description information is used to describe at least one of the type of point cloud components included in each track of the point cloud file, a type list of attribute components and a quantity list of different types of attribute components, an attribute group identifier to which the attribute component belongs, and description information of the attribute group; based on the point cloud component description information, determine the first attribute data to be decoded in the point cloud.

[0435] It should be understood that the device embodiments and method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be described in detail here. Specifically, the device 10 shown in FIG9 can execute the method embodiment corresponding to the file decapsulation device, and the aforementioned and other operations and / or functions of each module in the device 9 are respectively for implementing the method embodiment corresponding to the file decapsulation device. For the sake of brevity, they will not be described in detail here.

[0436] FIG10 is a schematic diagram of the structure of a file encapsulation device provided in an embodiment of the present application. The file encapsulation device 20 includes:

[0437] An acquiring unit 21 is configured to acquire a point cloud bitstream, wherein the point cloud bitstream includes N sets of attribute data, where N is a positive integer;

[0438] The encapsulation unit 22 is configured to encapsulate the first attribute data to be encapsulated in the N sets of attribute data based on a first relationship between the first attribute data and the second attribute data, and determine dependency indication information of the first attribute data to obtain a point cloud file;

[0439] The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, and the first relationship includes at least one of an encoding dependency relationship and a presentation association relationship.

[0440] In some embodiments, the encapsulation unit 22 is specifically used to obtain the second attribute data if there is a coding dependency relationship between the first attribute data and the second attribute data; encapsulate the first attribute data based on the second attribute data; if there is no coding dependency relationship between the first attribute data and the second attribute data, encapsulate the first attribute data separately.

[0441] In some embodiments, the encapsulation unit 22 is specifically used to encapsulate the first attribute data and the second attribute data in the first track if the point cloud bitstream adopts component-based multi-track encapsulation; if the attribute component in the point cloud bitstream is encapsulated in a second track, the first attribute data and the second attribute data in the second track are identified as belonging to the first attribute group, wherein the second track is a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or is a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream.

[0442] In some embodiments, if the point cloud bitstream adopts component-based multi-track encapsulation, the dependency indication information includes at least one of the first information and the second information, and the encapsulation unit 22 is specifically used to determine at least one of the first information and the second information corresponding to the first track, the first information is used to indicate the number of attribute components included in the first track, and the second information is used to indicate whether the first track includes an attribute group, and the first track is used to encapsulate the first attribute data and the second attribute data.

[0443] In some embodiments, if there is a coding dependency relationship between the first attribute data and the second attribute data, determining that the first information indicates that the number of attribute components included in the first track is greater than 1;

[0444] If there is no coding dependency relationship between the first attribute data and the second attribute data, determining that the first information indicates that the number of attribute components included in the first track is equal to 1;

[0445] If a presentation association relationship exists between the first attribute data and the second attribute data, determining that the second information indicates that the first track includes the attribute group;

[0446] If there is no presentation association relationship between the first attribute data and the second attribute data, it is determined that the second information indicates that the first track does not include the attribute group.

[0447] In some embodiments, the first information includes first attribute component number indication information, wherein the first attribute component number indication information is used to indicate the number of attribute components included in the first track; or

[0448] The first information includes attribute component type quantity indication information and second attribute component quantity indication information, wherein the attribute component type quantity indication information is used to indicate the number of different types of attribute components included in the first track, and the second attribute component quantity indication information is used to indicate the number of attribute components of each type;

[0449] The second information includes an attribute group information flag, and the attribute group information flag indicates whether attribute group information is indicated.

[0450] In some embodiments, if there is a presentation association relationship between the first attribute data and the second attribute data, and the first attribute data and the second attribute data are divided into a second attribute group, the encapsulation unit 22 is further used to determine the identifier of the second attribute group; and add the identifier of the second attribute group to the point cloud file.

[0451] In some embodiments, if the first attribute data and the second attribute data are identified as belonging to the first attribute group, the encapsulation unit 22 is further used to determine the attribute group flag corresponding to the first subsample in which the first attribute data is located, where the attribute group flag is used to indicate whether the first subsample belongs to the attribute group; if the attribute group flag indicates that the first subsample belongs to the attribute group, then determine the identifier of the first attribute group to which the first subsample belongs; and encapsulate the attribute group flag and the identifier of the first attribute group in the second track.

[0452] In some embodiments, if the attribute components in the point cloud bitstream are encapsulated in a second track, the dependency indication information includes at least one of third information and fourth information, and the encapsulation unit 22 is specifically used to determine at least one of the third information and the fourth information corresponding to the first attribute group, the third information is used to indicate whether there is a coding dependency relationship between the attribute components in the first attribute group, and the fourth information is used to indicate whether there is a presentation association relationship between the attribute components in the first attribute group.

[0453] In some embodiments, the third information includes an attribute dependency group flag, and the fourth information includes a default attribute flag. The attribute dependency group flag is used to indicate whether there is a coding dependency relationship between the attribute components corresponding to the first attribute group, and the default attribute flag is used to indicate whether the attribute components corresponding to the first attribute group are default presented attribute components.

[0454] In some embodiments, the encapsulation unit 22 is further used to determine the point cloud component description information of the point cloud file, and the point cloud component description information is used to describe at least one of the type of point cloud components included in each track of the point cloud file, a type list of attribute components and a quantity list of different types of attribute components, an attribute group identifier to which the attribute component belongs, and description information of the attribute group.

[0455] It should be understood that the device embodiments and method embodiments may correspond to each other, and similar descriptions may refer to the method embodiments. To avoid repetition, they will not be described in detail here. Specifically, the device 20 shown in Figure 10 can execute the method embodiment corresponding to the file encapsulation device, and the aforementioned and other operations and / or functions of each module in the device 20 are respectively for implementing the method embodiment corresponding to the file encapsulation device. For the sake of brevity, they will not be described in detail here.

[0456] The apparatus of the embodiment of the present application is described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0457] FIG11 is a schematic block diagram of an electronic device provided in an embodiment of the present application, which may be the above-mentioned file decapsulation device or file encapsulation device.

[0458] As shown in FIG11 , the electronic device 30 may include:

[0459] The memory 31 and the processor 32 are configured to store a computer program 33 and transmit the program code 33 to the processor 32. In other words, the processor 32 can call and run the computer program 33 from the memory 31 to implement the method in the embodiment of the present application.

[0460] For example, the processor 32 may be configured to execute the steps of the above method according to the instructions in the computer program 33 .

[0461] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0462] General-purpose processor, Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0463] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0464] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0465] In some embodiments of the present application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to implement the method for recording a page provided by the present application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 33 in the electronic device.

[0466] As shown in FIG11 , the electronic device 30 may further include:

[0467] The transceiver 34 may be connected to the processor 32 or the memory 31 .

[0468] The processor 32 may control the transceiver 34 to communicate with other devices. Specifically, the processor 32 may send information or data to other devices or receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include one or more antennas.

[0469] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0470] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiment. Alternatively, the present application also provides a computer program product containing instructions. When the computer is executed by the instructions, the computer is enabled to perform the method of the above-mentioned method embodiment.

[0471] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method of the above-described method embodiment.

[0472] The above content is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A point cloud file decapsulation method, characterized in that: include: Determine first attribute data to be decoded in a point cloud file, wherein the point cloud file is a file obtained by encapsulating a point cloud bit stream; Determine dependency indication information of the first attribute data, where the dependency indication information is used to indicate whether there is a first relationship between the first attribute data and the second attribute data, where the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship; Based on the dependency indication information, the first attribute data is decoded.

2. The method according to claim 1, characterized in that The decoding of the first attribute data based on the dependency indication information includes: If the dependency indication information indicates that there is a decoding dependency relationship between the first attribute data and the second attribute data, determining the second attribute data, and decoding the first attribute data based on the second attribute data; If the dependency indication information indicates that there is no decoding dependency relationship between the first attribute data and the second attribute data, the first attribute data is decoded separately.

3. The method according to claim 2, characterized in that The determining the second attribute data comprises: If the point cloud bitstream adopts component-based multi-track encapsulation, determining a first track where the first attribute data is located, and determining other attribute data in the first track except the first attribute data as the second attribute data; If the attribute components in the point cloud bitstream are encapsulated in a second track, the first attribute group to which the first attribute data belongs is determined, and the other attribute data in the first attribute group except the first attribute data are determined as the second attribute data, wherein the second track is a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or is a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream.

4. The method according to any one of claims 1 to 3, characterized in that: If the point cloud bitstream adopts component-based multi-track encapsulation, the dependency indication information includes at least one of the first information and the second information, and the determining of the dependency indication information of the first attribute data includes: Determine a first track where the first attribute data is located; At least one of first information and second information corresponding to the first track is determined, the first information is used to indicate the number of attribute components included in the first track, and the second information is used to indicate whether the first track includes an attribute group.

5. The method according to claim 4, characterized in that If the first information indicates that the number of attribute components included in the first track is greater than 1, then there is a decoding dependency relationship between the first attribute data and the second attribute data; If the first information indicates that the number of attribute components included in the first track is equal to 1, then there is no decoding dependency relationship between the first attribute data and the second attribute data; If the second information indicates that the first track includes the attribute group, then there is a presentation association relationship between the first attribute data and the second attribute data; If the second information indicates that the first track does not include the attribute group, there is no presentation association relationship between the first attribute data and the second attribute data.

6. The method according to claim 4 or 5, characterized in that: The first information includes first attribute component number indication information, wherein the first attribute component number indication information is used to indicate the number of attribute components included in the first track; or, The first information includes attribute component type quantity indication information and second attribute component quantity indication information, wherein the attribute component type quantity indication information is used to indicate the quantity of different types of attribute components included in the first track, and the second attribute component quantity indication information is used to indicate the quantity of attribute components of each type; The second information includes an attribute group information flag, and the attribute group information flag indicates whether to indicate attribute group information.

7. The method according to claim 6, characterized in that If the attribute group information flag indicates attribute group information, and the attribute components included in the first track belong to the corresponding second attribute group, the method further includes: determining an identifier of the second attribute group; Based on the identifier of the second attribute group, obtaining attribute data in the second attribute group, where the second attribute group includes the first attribute data and the second attribute data; The attribute data in the second attribute group are decoded and presented together.

8. The method according to any one of claims 1 to 3, characterized in that: If the attribute component in the point cloud bitstream is encapsulated in a second track, the dependency indication information includes at least one of the third information and the fourth information, and the determining the dependency indication information of the first attribute data includes: Determine a first attribute group to which the first attribute data belongs, the second track being a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream; Determine at least one of third information and fourth information corresponding to the first attribute group, the third information is used to indicate whether there is a decoding dependency relationship between attribute components in the first attribute group, and the fourth information is used to indicate whether there is a presentation association relationship between attribute components in the first attribute group.

9. The method according to claim 8, characterized in that The determining the first attribute group to which the first attribute data belongs includes: determining an attribute group flag corresponding to a first subsample where the first attribute data is located, wherein the attribute group flag is used to indicate whether the first subsample belongs to an attribute group; If the attribute group flag indicates that the first subsample belongs to an attribute group, determining an identifier of a first attribute group to which the first subsample belongs; Based on the identifier of the first attribute group, the first attribute group is determined.

10. The method according to claim 9, characterized in that The third information includes an attribute dependency group flag, and the fourth information includes a default attribute flag, wherein the attribute dependency group flag is used to indicate whether there is a decoding dependency relationship between attribute components corresponding to the first attribute group, and the default attribute flag is used to indicate whether the attribute component corresponding to the first attribute group is a default presented attribute component.

11. The method according to any one of claims 1 to 10, characterized in that: The step of determining first attribute data to be decoded in the point cloud file includes: Acquire point cloud component description information of the point cloud file, the point cloud component description information is used to describe at least one of the type of point cloud components included in each track of the point cloud file, a type list of attribute components and a quantity list of different types of attribute components, an attribute group identifier to which the attribute component belongs, and description information of the attribute group; Based on the point cloud component description information, first attribute data to be decoded in the point cloud is determined.

12. A point cloud file packaging method, characterized in that: include: Obtaining a point cloud bitstream, wherein the point cloud bitstream includes N groups of attribute data, where N is a positive integer; For first attribute data to be encapsulated in the N groups of attribute data, based on a first relationship between the first attribute data and second attribute data, encapsulate the first attribute data, determine dependency indication information of the first attribute data, and obtain a point cloud file; The dependency indication information is used to indicate whether there is a first relationship between the first attribute data and the second attribute data, and the first relationship includes at least one of an encoding dependency relationship and a presentation association relationship.

13. The method according to claim 12, characterized in that The encapsulating the first attribute data based on the first relationship between the first attribute data and the second attribute data includes: If there is a coding dependency relationship between the first attribute data and the second attribute data, acquiring the second attribute data, and encapsulating the first attribute data based on the second attribute data; If there is no coding dependency relationship between the first attribute data and the second attribute data, the first attribute data is encapsulated separately.

14. The method according to claim 13, characterized in that The encapsulating the first attribute data based on the second attribute data includes: If the point cloud bitstream adopts component-based multi-track encapsulation, encapsulating the first attribute data and the second attribute data in a first track; If the attribute components in the point cloud bitstream are encapsulated in a second track, the first attribute data and the second attribute data in the second track are identified as belonging to a first attribute group, wherein the second track is a track that encapsulates the point cloud bitstream in a single-track encapsulation manner, or is a point cloud slice track that encapsulates all components of a specific point cloud slice of the point cloud bitstream.

15. The method according to claim 14, characterized in that If there is a presentation association relationship between the first attribute data and the second attribute data, and the first attribute data and the second attribute data are classified into a second attribute group, the method further includes: determining an identifier of the second attribute group; An identifier of the second attribute group is added to the point cloud file.

16. The method according to claim 14, characterized in that If the first attribute data and the second attribute data are identified as belonging to the first attribute group, the method further includes: determining an attribute group flag corresponding to a first subsample where the first attribute data is located, wherein the attribute group flag is used to indicate whether the first subsample belongs to an attribute group; If the attribute group flag indicates that the first subsample belongs to an attribute group, determining an identifier of a first attribute group to which the first subsample belongs; The attribute group flag and the identifier of the first attribute group are encapsulated in the second track.

17. A point cloud file decapsulation device, characterized in that: include: A data determination unit, configured to determine first attribute data to be decoded in a point cloud file, wherein the point cloud file is a file obtained by encapsulating a point cloud bit stream; an information determining unit, configured to determine dependency indication information of the first attribute data, wherein the dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, wherein the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship; A decoding unit is used to decode the first attribute data based on the dependency indication information.

18. A point cloud file packaging device, characterized in that: include: An acquisition unit, used for acquiring a point cloud bit stream, wherein the point cloud bit stream includes N groups of attribute data, where N is a positive integer; an encapsulation unit, configured to encapsulate the first attribute data to be encapsulated in the N groups of attribute data based on a first relationship between the first attribute data and the second attribute data, and determine dependency indication information of the first attribute data to obtain a point cloud file; The dependency indication information is used to indicate whether a first relationship exists between the first attribute data and the second attribute data, and the first relationship includes at least one of a decoding dependency relationship and a presentation association relationship.

19. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 11 or 12 to 16.

20. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program causing a computer to execute the method as claimed in any one of claims 1 to 11 or 12 to 16.

21. A computer program product comprising a computer program / instructions, characterized in that The computer program / instructions, when executed by a processor, implement a method according to any one of claims 1 to 11 or 12 to 16.

Images