Point cloud data processing method and device and storage medium

Abstract

Embodiments of the present disclosure provide a point cloud data processing method and device and a storage medium. The point cloud data processing method comprises: receiving a point cloud data media file (S101), wherein the point cloud data media file comprises a timed metadata track associated with an object and / or a data box associated with the object, the timed metadata track is used for indicating object metadata in media, and the data box is used for indicating the object metadata in the media; on the basis of the point cloud data media file indicating the object metadata, acquiring object information in the point cloud data media file (S102); and performing media processing on point cloud data in the point cloud data media file on the basis of the object information (S103).

Description

Point cloud data processing method and device, and storage medium

[0001] This disclosure claims priority to Chinese patent application No. 202410216859.X, filed on February 27, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the technical field of media data processing, and in particular to a method and device for processing point cloud data and a storage medium. Background Art

[0003] Immersive media uses technology to allow viewers to fully immerse themselves in media content, creating a truly immersive experience. This media format typically utilizes multi-sensory stimulation, such as vision, hearing, and touch, to create a virtual environment similar to the real world, allowing viewers to feel as if they are immersed in the virtual world. Types of immersive media include virtual reality (VR), augmented reality (AR), and holographic projection. These technologies simulate human sensory organs, such as vision and hearing, to create a new audiovisual experience, allowing viewers to experience more realistic, vivid, and three-dimensional scenes and effects.

[0004] Immersive media can be divided into three degrees of freedom (3DoF) media, enhanced three degrees of freedom (3DoF+) media, and six degrees of freedom (6DoF) media according to the user's degree of freedom when consuming media content. Summary of the Invention

[0005] In one aspect, the present disclosure provides a method for processing point cloud data. The method for processing point cloud data includes:

[0006] Receiving a point cloud data media file; the point cloud data media file includes a timed metadata track associated with the object and / or a data box associated with the object, the timed metadata track is used to indicate object metadata in the media, and the data box is used to indicate object metadata in the media;

[0007] Based on the point cloud data media file indicating the object metadata, obtaining object information in the point cloud data media file;

[0008] Perform media processing on the point cloud data in the point cloud data media file according to the object information.

[0009] In another aspect, the present disclosure provides a point cloud data processing device. The point cloud data processing device includes:

[0010] a receiving module configured to receive a point cloud data media file; the point cloud data media file includes a timed metadata track associated with the object and / or a data box associated with the object, the timed metadata track being used to indicate object metadata in the media, and the data box being used to indicate object metadata in the media;

[0011] a processing module for obtaining object information in the point cloud data media file based on the point cloud data media file indicating the object metadata;

[0012] The processing module is further used to perform media processing on the point cloud data in the point cloud data media file according to the object information.

[0013] In yet another aspect, the present disclosure further provides a point cloud data processing device. The point cloud data processing device comprises: a memory and a processor; the memory and the processor are coupled; the memory is configured to store instructions executable by the processor; and when the processor executes the instructions, it performs the point cloud data processing method provided in the above aspect.

[0014] In another aspect, the present disclosure provides a computer program product comprising computer instructions. When the computer instructions are executed on a computer, the computer is caused to execute the point cloud data processing method provided in the above aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure.

[0016] FIG1 is a schematic diagram of the architecture of an immersive media system according to an embodiment of the present disclosure.

[0017] FIG2 is a flow chart of a method for processing point cloud data according to an embodiment of the present disclosure.

[0018] FIG3 is a flowchart of another method for processing point cloud data according to an embodiment of the present disclosure.

[0019] FIG4 is a schematic diagram of a scenario of yet another method for processing point cloud data according to an embodiment of the present disclosure.

[0020] FIG5 is a flowchart of another method for processing point cloud data according to an embodiment of the present disclosure.

[0021] FIG6 is a schematic diagram of an ISO media file according to an embodiment of the present disclosure.

[0022] FIG7 is a schematic diagram of another ISO media file according to an embodiment of the present disclosure.

[0023] FIG8 is a schematic diagram of another ISO media file according to an embodiment of the present disclosure.

[0024] FIG9 is a flowchart of another method for processing point cloud data according to an embodiment of the present disclosure.

[0025] FIG10 is a schematic diagram showing the composition of a point cloud data processing device according to an embodiment of the present disclosure.

[0026] FIG11 is a schematic structural diagram of a point cloud data processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] To enable those skilled in the art to better understand the technical solutions of the embodiments of the present disclosure, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0028] In the description of the present disclosure, unless otherwise specified, the symbol " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: only A, only B, and A and B. In addition, "at least one" means one or more, and "a plurality" means two or more. Words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0029] It should be noted that in this disclosure, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this disclosure as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0030] In order to better present the viewing content of users, it is necessary to perform differentiated processing on different parts of the content. For example, high-quality media content is used for the content in key areas, and more resources are allocated for media processing. For the content in non-key areas, media content sources of general quality are used. This can not only improve the user's viewing experience, but also reduce the processing pressure on the network and terminals. At present, the immersive media system can indicate object information at the file encapsulation layer so that different contents of the immersive media can be processed differently based on the object information. However, since the application scenarios of the immersive media system are diverse, for example, the objects in the video are almost static and media processing needs to be performed according to the user's position, the current differentiated processing of content cannot meet the transmission and media processing requirements in different scenarios.

[0031] In view of this, the present disclosure provides a method for processing point cloud data, comprising: receiving a point cloud data media file; the point cloud data media file including a timed metadata track associated with an object and / or a data box associated with the object. The timed metadata track is used to indicate object metadata in the media, and the data box is used to indicate object metadata in the media; based on the point cloud data media file indicating the object metadata, obtaining object information in the point cloud data media file; and performing media processing on the point cloud data in the point cloud data media file based on the object information.

[0032] In this way, for various scenarios of immersive media, such as user windows, media processing based on user location, and static objects in the window, the technical solutions provided by the present disclosure can provide various corresponding media processing methods for various scenarios, so as to perform differentiated transmission and media processing based on the object information of the partial scene and one or more point cloud data objects in the scene within the window range of each scenario, thereby meeting the various scenario requirements of immersive media and improving the flexibility of differentiated media processing. In addition, differentiated media processing, for example, uses higher-quality media content for the point cloud data corresponding to the point cloud data object and allocates more resources for media processing to provide a better immersive experience and optimize the media presentation quality. Other areas can also use lower-quality media content and allocate fewer resources for media processing. This can also reduce the processing pressure on the network and terminal, thereby improving the transmission efficiency and processing capacity of the media content.

[0033] Immersive media uses technology to fully immerse viewers in the content, creating a truly immersive experience. This media format typically utilizes multiple sensory stimulations, such as vision, hearing, and touch, to create a virtual environment similar to the real world, allowing viewers to experience a sense of being immersed in the virtual world.

[0034] Point cloud data, a key data source for immersive media, is a collection of randomly distributed discrete points in space that represent the spatial structure and surface properties of a three-dimensional object or scene. Each point in this data contains at least three-dimensional position information and can also include color, material, or other information. Consequently, point cloud data can be used to generate three-dimensional models, scene reconstruction, and virtual reality. In immersive media, point cloud data can be used to construct virtual scenes, virtual characters, virtual objects, and more, creating a more realistic and vivid virtual environment.

[0035] In addition, point cloud data can also be acquired through the following methods: computer generation, three-dimensional (3D) laser scanning, 3D photogrammetry, etc. Computers can generate point clouds of virtual three-dimensional objects and scenes. 3D scanning can obtain point cloud data of static real-world three-dimensional objects or scenes, and can obtain millions of point cloud data per second. 3D photography can obtain point cloud data of dynamic real-world three-dimensional objects or scenes, and can obtain tens of millions of point cloud data per second. This can reduce the cost and time cycle of point cloud data acquisition and improve data accuracy. The change in the way point cloud data is acquired makes it possible to acquire large amounts of point cloud data. With the continuous accumulation of large-scale point cloud data, the efficient storage, transmission, publication, sharing and standardization of point cloud data have become the key to point cloud applications.

[0036] As shown in Figure 1, the present disclosure provides an immersive media system comprising a content production terminal 10 and a client 20. The content production terminal 10 is used to acquire and produce immersive media content, and to perform encoding and file encapsulation to obtain encapsulated media files, such as point cloud data media files, for carrying the immersive media content. Therefore, the content production terminal 10 may be referred to as a content production device, an encoding device, or the like. The client 20 is capable of performing decapsulation, decoding, media rendering, and other processing on the point cloud data media files generated by the content production terminal 10.

[0037] The content production terminal 10 is used to produce media content offline or online, and can also be used to load third-party media content.

[0038] The client 20 includes at least a transmission module, a decoding and decapsulation module, a media processing module, and a display module. The transmission module is used to receive point cloud data media files, or send a request message to request point cloud data media files. The decoding and decapsulation module is used to decode and decapsulate the point cloud data media files received through the transmission module. The media processing module is used to reconstruct, render, and perform other processing on the media data output by the decoding and decapsulation module based on information such as the user's current viewing position, viewing window, or the director's recommended viewing window. The display module is used to present the visual content of the user's current window to the user. The display module may include a variety of display components, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, and the like.

[0039] In some embodiments, the content production end 10 and the client 20 can be in the form of various devices, for example, a server. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data servers. For another example, the content production end 10 and the client 20 can be various terminal devices, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) and virtual reality (VR) devices.

[0040] In some embodiments, the content production terminal 10 and the client 20 in Figure 1 may be two separate devices. Alternatively, the content production terminal 10 and the client 20 may be the same device, that is, the content production terminal 10 or the corresponding functions and the client 20 or the corresponding functions may be integrated into the same device.

[0041] The content production terminal 10 and the client 20 can communicate with each other. For example, the client 20 can directly receive the point cloud data media file from the content production terminal 10. For another example, as shown in FIG1 , a transmission device 30 can be further provided between the content production terminal 10 and the client 20, and the client 20 can receive the point cloud data media file from the content production terminal 10 via the transmission device 30.

[0042] The transmission device 30 includes at least a storage module and a transmission module. The storage module is used to store point cloud data media files created or loaded by the content production end, and may include any of a variety of distributed or locally accessible data storage media, such as a hard disk drive, Blu-ray disc, digital versatile disc (DVD), compact disc read-only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. The transmission module is used to receive request messages from the client or transmit stored point cloud data media files to the client. This transmission process can be achieved through a wireless network provided by a communication provider, a locally established wireless local area network, or a wired method.

[0043] In addition, the transmission device 30 may also be in the form of various devices such as the server, terminal device, etc. described above, which are not listed here one by one.

[0044] It should be noted that FIG1 is only an exemplary framework diagram. The number of devices or nodes included in FIG1 and the names of the devices are not limited. In addition to the functional nodes shown in FIG1 , the system may also include other nodes or devices.

[0045] The system architecture and business scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems.

[0046] The following describes some embodiments provided by the present disclosure in conjunction with the accompanying drawings. For example, this embodiment can be applied to various computing devices capable of performing decapsulation, decoding, media rendering, and other processing on point cloud data media files, such as the client 20 in FIG1 . Specifically, the client 20 can obtain a point cloud data media file and perform media processing on the point cloud data in the point cloud data media file according to this technical solution.

[0047] As shown in FIG2 , the present disclosure provides a method for processing point cloud data, which includes the following S101 - S103 .

[0048] S101: Receive a point cloud data media file.

[0049] Point cloud data media file is a file format used to store and exchange 3D point cloud data.

[0050] In some embodiments, the file format of the point cloud data media file may be an International Organization for Standardization (ISO) base media file format. Since each point in the point cloud data has at least three-dimensional position information, the omnidirectional video data may be stored in the point cloud data media file based on the ISO base media file format. The ISO base media file format may include, but is not limited to, a restricted scheme information box, a track reference box, and a track group box. The ISO base media file format, such as the restricted scheme information box, the track reference box, and the track group box, may be operated with reference to the universal digital media file format MPEG-4 Part 12 ISO Base Media File Format developed by the Moving Picture Experts Group (MPEG) of ISO / IEC JTC1 / SC29 / WG11 (International Organization for Standardization / International Electrotechnical Commission Joint Technical Committee 1 / Subcommittee 29 / Working Group 11).

[0051] Furthermore, the file structure defined by the ISO basic file format can consist of a series of boxes. The ISO basic file format, exemplified by MP4 files, can include several boxes, each with a type and length, and can be considered a data object. A box can contain another box, known as a container box. Furthermore, each box has a specific type and length, and together they constitute the logical structure of the file.

[0052] An MP4 file first has one and only one file type box (ftyp). The "ftyp" box can be the beginning of the file, serving as a marker for the file format and storing basic information about the file, such as the file format version and compatible container formats. Afterward, there is one and only one "MOOV" box (Movie Box), a container box. The "MOOV" box's subboxes contain metadata about the point cloud media file, such as timestamps and encoding information. In some examples, the "MOOV" box can be located at the end of the file, allowing for quick access to all metadata when reading the file. The media data of an MP4 file is contained in an "mdat" box (Media Data Box). This "mdat" box is also a container box, and there can be one, multiple, or no "mdat" boxes in the file. For example, if all media data references other files, the structure of the media data is described by metadata. The "mdat" box contains the actual media data, such as audio samples and video frames. In the case of multiple "mdat" boxes, each box contains a portion of the media data. To further support metadata description of the media, a "meta" type box (Meta box) can be used. It is an optional container box. The "meta" box can be used to describe some general or additional non-scheduled metadata, such as codec information, subtitle data, etc.

[0053] In some embodiments, the point cloud data media file may be obtained based on the user's viewing position or recommended viewing window.

[0054] In one example, as shown in FIG3 , SA1 may be executed to obtain a point cloud data media file based on the user's viewing position.

[0055] For example, the client can analyze and determine the viewing window corresponding to the user's viewing position based on information such as the user's viewing location, and obtain the point cloud data media file corresponding to the viewing window. The point cloud data media file corresponding to the viewing window carries the media content that the client can display to the user through the viewing window. As shown in Figure 4, the current overall scene includes objects 1, 2, and 3. Among them, objects 2 and 3 are within the area that can be viewed by the user at the current position, that is, within the viewing window corresponding to the position. Therefore, the point cloud data media file corresponding to the viewing window obtained contains the relevant point cloud data of objects 2 and 3.

[0056] In another example, as shown in FIG5 , SB1 may be executed to obtain a point cloud data media file according to the recommended window.

[0057] For example, the client can obtain a point cloud data media file in conjunction with a recommendation window. The point cloud data media file obtained in conjunction with the recommendation window carries media content that the client can display to the user through the recommendation window. Furthermore, the recommendation window can be a window corresponding to a preset appropriate display perspective based on the content of the point cloud data media file. The recommendation window can be a dynamic window. For example, in a game or virtual reality scene, the transformation of the dynamic window allows the audience to experience the plot and actions in the scene as if they were there. This can provide a more realistic and three-dimensional visual effect, enhancing the sense of immersion. In some embodiments, the recommendation window can include a recommendation window generated based on the director's or content provider's intentions, a recommendation window based on the current user's viewing data statistics, a recommendation window generated based on windows selected by users other than the current user, a recommended initial window, a recommendation window associated with a specific spatial area, or other possible recommendation windows customized by the user.

[0058] S102 : Based on the point cloud data media file indicating the object metadata, obtain object information in the point cloud data media file.

[0059] The objects in the aforementioned object information refer to the portion of the scene that appears within the user's viewing window, as well as one or more point cloud data objects within the scene. As shown in Figure 4, the scene includes objects 1, 2, and 3. Objects 2 and 3 are within the user's viewing window, thus identifying objects 2 and 3 as the aforementioned objects. Alternatively, the objects can be characters, objects, animals, scene interaction points, and so on within the scene. It should be noted that in immersive media scenes, because user attention is often focused on certain areas of the viewing window, such as characters and important objects within the scene, and because human visual perception is limited, with varying sensitivity and resolution for different areas of the screen, as well as the need for real-time interactive network resources and adaptability to different devices and network conditions, the image within an immersive media scene can be divided into key and non-key areas. High-quality processing of key areas can better match the human eye's visual perception characteristics, more efficiently allocate computing and network resources, and enable more flexible adaptation of media content to different devices and network conditions, providing a consistent and high-quality user experience.

[0060] An immersive scene can contain one or more objects. The area where these objects are located is the area in the scene where the user's attention is focused. The point cloud data corresponding to these objects in the point cloud data object file requires high-quality processing to better match the visual perception characteristics of the human eye. Therefore, the area where these objects are located can be understood as the key area in the immersive scene. Furthermore, all or part of the objects in the immersive scene can be included within the window range. This all or part of the objects can be understood as the current key area in the window, and their content requires the use of high-quality media content and the allocation of more resources for media processing.

[0061] In addition, the above-mentioned object can be a dynamic object or a static object.

[0062] In some embodiments, the object information may include one or more of the following: an identifier of the object information entry, object description information corresponding to the object information entry, anchor point coordinates of the spatial area where the object is located, size information of the spatial area where the object is located, and point cloud information containing the object.

[0063] An object information entry refers to a single data element or record that makes up "object information". Each entry may contain information about one aspect of an object, such as the object's identification, description, location, size, and relationship to point cloud data. The identifier of an object information entry can be a symbol or code that uniquely identifies an object and can be used to distinguish different objects. The object description information corresponding to the object information entry may include a detailed description of the object itself, such as a description of the object's name, type, function, material, etc. The anchor point coordinates of the spatial area where the object is located can be used to indicate the coordinates of a reference point (anchor point) of the object in space, so as to determine the position of the object in space based on the anchor point coordinates. The size information of the spatial area where the object is located is used to describe the size or range of the object in space, and may include length, width, height, etc. A point cloud slice refers to a set of discrete point data, typically used to represent the surface shape of an object. The point cloud slice information containing an object is used to represent part of the point cloud data associated with the object.

[0064] In addition, object metadata can be regarded as a part of object information. Object metadata provides descriptive information about the object itself, which is used to describe various aspects of object information, such as creation time, modification time, file size, data type, etc., which is conducive to better organization, search and management of object information.

[0065] In some embodiments, the point cloud data media file may include object information to describe information related to objects in the scene. For example, the structure of the object information ObjectInfoStruct is as follows:

[0066] ref_object_info_id is used to indicate the identifier of the corresponding object information entry.

[0067] object_oid_flag is used to indicate the object description information format. When the value of object_oid_flag is 0, it can indicate that the object description information is indicated in the form of a human-readable string. When the value of object_oid_flag is 1, it can indicate that the object description information is indicated in the form of an OID (Object Identifier).

[0068] object_info_id is used to indicate the identifier of the corresponding object information item.

[0069] object_oid is used to indicate the object description information corresponding to the corresponding object information entry, which is indicated in the form of OID.

[0070] object_label is used to indicate the object description information corresponding to the corresponding object information entry, which is indicated in the form of a string readable by the human eye.

[0071] When the value of object_spatial_info_flag is 0, it indicates that the specific spatial information corresponding to the object is not indicated; when the value is 1, it indicates the specific spatial information corresponding to the object.

[0072] When the value of object_slice_info_flag is 0, it indicates that the point cloud slice information corresponding to the object is not indicated; when the value is 1, it indicates the point cloud slice information corresponding to the object.

[0073] 3DPoint indicates the coordinates of the anchor point in the spatial region where the object is located;

[0074] CuboidRegionStruct indicates the size information of the spatial region where the object is located, that is, the extension of a 3D spatial region in a Cartesian coordinate system relative to the anchor point on the x, y, and z axes.

[0075] SliceMapping indicates point cloud slice information, which indicates the number of slices associated with the 3D spatial region and identifies the point cloud slice associated with the spatial region.

[0076] In some embodiments, the point cloud data media file includes a timed metadata track associated with the object and / or a data box associated with the object, the timed metadata track is used to indicate object metadata in the media, and the data box is used to indicate object metadata in the media.

[0077] It should be understood that the timed metadata track is a mechanism in the ISO Base Media File Format (ISOBMFF) for establishing timed metadata associated with a specific sample. Timed metadata is less coupled with media data and is usually "descriptive".

[0078] Exemplarily, in the case where the point cloud data media file includes a timed metadata track associated with the object, the timed metadata track associated with the object may include: a first timed metadata track, and / or a second timed metadata track.

[0079] In some embodiments, when a point cloud data media file is obtained according to a user's viewing position, the timed metadata track associated with an object contained in the point cloud data media file may include a first timed metadata track.

[0080] Therefore, the point cloud data in the point cloud data media file can be identified according to the object metadata sample entry in the first timed metadata track to obtain the object information.

[0081] Exemplarily, as shown in FIG3 , SA2 can be executed to decapsulate and decode the acquired point cloud data media file based on information such as the user's viewing position, and perform differentiated media processing on the point cloud data media file based on object information corresponding to the object in the user's viewing window.

[0082] The first timed metadata track contains dynamic window information and object information corresponding to objects in the window.

[0083] Exemplarily, the point cloud data media file may include a first timed metadata track, which may also be referred to as an object information timed metadata track. The object information contained in the first timed metadata track may be used to describe the objects present in each frame of the image / scene at different times. This allows for differentiated media processing of the object information corresponding to the current user's viewing position or viewing window. In addition, the first sample entry is the sample entry of the first timed metadata track (i.e., the object information timed metadata track). The first sample entry may be the sample entry ObjectInfoSampleEntry that extends MetaDataSampleEntry. Furthermore, the first timed metadata track may be identified by the type identifier of the first sample entry, for example, the first timed metadata track, i.e., the object information timed metadata track, may be identified based on the type identifier "obdi" of the first sample entry. And the first sample entry also includes initial object information.

[0084] Exemplarily, the first sample entry ObjectInfoSampleEntry may include:

[0085] Among them, num_objects is used to indicate the number of object information entries contained in the current sample. When the value of this field is 0, it means that the current sample does not contain any object information entries.

[0086] ObjectInfoStruct is used to describe the specific information of each object.

[0087] In some embodiments, each sample in the first timed metadata track, i.e., the object information timed metadata track, has a fixed format. Each sample contains object change information within the user's viewing window. For example, the sample format ObjectInfoSample may include:

[0088] Among them, num_object_update is used to indicate the number of information items of object information included in the current sample that have changed compared to the previous sample. In addition, when the value of the num_object_update field is 0, it means that the object information included in the current sample is the same as that of the previous synchronization sample.

[0089] When the value of object_canceled_flag is 1, it indicates that the corresponding object is no longer included in the current sample. When the value of object_canceled_flag is 0, it indicates that the corresponding object is included in the current sample, but an update has occurred.

[0090] ObjectInfoStruct is used to indicate specific object information.

[0091] ref_object_info_id is used to indicate the object ID in the current sample.

[0092] For example, as shown in Figure 6, the ISO media file may include a "moov" type box, track 1 and track 2 in the moov box, an "mdat" type box, and samples 1, 2, 3, etc. corresponding to each track in the mdat box, and the first sample entry ObjectInfoSampleEntry in the "meta" type box, as well as the corresponding object information ObjectInfoStruct.

[0093] In some embodiments, the point cloud media file can be packaged in a single track mode, that is, all point cloud data can be directly packaged in the media track without being split. In this case, the aforementioned ObjectInfoStruct structure can be added to the description of the subsample of the media track to describe the object information. In this way, the point cloud data contained in the subsample corresponds to one or more objects.

[0094] In some embodiments, when the point cloud data media file is acquired according to the recommended viewport, the timed metadata track associated with the object contained in the point cloud data media file may include a second timed metadata track.

[0095] Therefore, the point cloud data in the point cloud data media file can be identified according to the object metadata sample entry in the second timed metadata track to obtain the object information.

[0096] For example, as shown in Figure 5, SB2 can be executed to decapsulate and decode the acquired point cloud data media file based on the recommended viewport information. Media processing can then be performed on the point cloud data media file based on the object information corresponding to the object in the recommended viewport. The recommended dynamic viewport information includes object information, allowing the client to perform differentiated media processing based on presentation needs.

[0097] The second timed metadata track contains object information corresponding to objects in the recommendation window.

[0098] For example, the point cloud data media file may include a second timed metadata track, which may also include a recommended viewport constant metadata track. The object information contained in the second timed metadata track can be used to describe the director's recommended viewing windows (i.e., the aforementioned recommended windows) at different times. Based on the director's recommended viewing windows, the user terminal can pre-acquire the point cloud data media file corresponding to the viewing window and perform differentiated media processing on the media content (e.g., objects) within the viewing window based on the object information in the point cloud data media file.

[0099] In some embodiments, the second timed metadata track sample entry includes a first identifier, where the first identifier is used to indicate a view type containing object information.

[0100] In addition, the second sample entry is the sample entry of the second timed metadata track, ie, the dynamic view window timed metadata track. The second sample entry may be a sample entry ViewportInfoSampleEntry that extends MetaDataSampleEntry.

[0101] Furthermore, the second timed metadata track can be identified by a type identifier of the second sample entry. For example, the second timed metadata track can be identified based on the type identifier "avpt" of the second sample entry, i.e., a dynamic viewport timed metadata track. The second sample entry also includes initial object information.

[0102] Exemplarily, the second sample entry ViewportInfoSampleEntry may include:

[0103] Among them, object_flag is used to indicate whether the dynamic window timed metadata track contains object information.

[0104] num_objects is used to indicate the number of objects contained in the initial viewport.

[0105] ObjectInfoStruct is used to indicate the specific information of an object.

[0106] In some embodiments, the ObjectInfoStruct() contained in the ViewportInfoSampleEntry() structure can be replaced with an ObjectBox structure. An ObjectBox structure can contain information about one or more objects. That is, a point cloud data media file includes an ObjectBox structure associated with an object. This ObjectBox structure indicates metadata about the object in the media, and can contain one or more objects.

[0107] For example, when obtaining a point cloud data media file based on a recommended viewport, the data box associated with the object may be a viewport information configuration data box ViewportInfoConfigurationBox, which may be used to describe information such as the viewport type, the camera intrinsic and extrinsic parameters corresponding to the viewport, etc. The viewport information configuration data box ViewportInfoConfigurationBox may include:

[0108] Wherein, viewport_type (ie, the first identifier mentioned above) is used to indicate the viewport type of all samples corresponding to the current sample entry. For example, the meaning of the value of viewport_type is shown in Table 1 below.

[0109] Table 1

[0110] viewport_description is a null-terminated string that provides a textual description of the recommended viewport.

[0111] dynamic_int_camera_flag is used to indicate the changes in the camera intrinsic parameters of all samples corresponding to the current sample entry. When dynamic_int_camera_flag is 0, it means that the camera intrinsic parameters of all samples corresponding to the current sample entry are fixed. When dynamic_int_camera_flag is 1, it means that the camera intrinsic parameters of all samples corresponding to the current sample entry are dynamically changing.

[0112] dynamic_ext_camera_flag is used to indicate the changes in the camera extrinsic parameters of all samples corresponding to the current sample entry. When dynamic_ext_camera_flag is 0, it means that the camera extrinsic parameters of all samples corresponding to the current sample entry are fixed. When dynamic_ext_camera_flag is 1, it means that the camera extrinsic parameters of all samples corresponding to the current sample entry are dynamically changing.

[0113] When the viewport_type field is set to 3, the viewport timing metadata indicates the recommended initial viewport information, including the initial viewport position and rotation information. When point cloud media needs to start playback with a specific initial viewport (not the default initial viewport), the viewport timing metadata track can index into the AVS point cloud track. In the absence of this type of metadata, the initial viewport's cam_pos_x, cam_pos_y, cam_pos_z, cam_quat_x, cam_quat_y, and cam_quat_z default values ​​are all 0.

[0114] IntCameraInfoStruct is used to indicate the camera intrinsic parameter information corresponding to the window.

[0115] ExtCameraInfoStruct is used to indicate the camera extrinsic parameter information corresponding to the window.

[0116] In some embodiments, each sample in the second timed metadata track, i.e., the dynamic viewport timed metadata track, has a fixed format. Each sample contains dynamic viewport information and information about changes to objects within the viewport. For example, the sample format ViewportInfoSample may include:

[0117] In some embodiments, the second timed metadata track, ie, the dynamic view window timed metadata track, needs to include camera extrinsic parameter information ExtCameraInfoStruct(), which may be included in the sample entry or in the sample.

[0118] When dynamic_ext_camera_flag is set to 1, at least one of the values ​​of camera_extrinsic_flag[i] in all samples is set to 1.

[0119] num_viewports is used to indicate the number of viewports indicated in the sample.

[0120] viewport_id[i] is used to indicate the identifier of the corresponding viewport.

[0121] When viewport_cancel_flag[i] is 1, it indicates that the viewport with the viewport identifier viewport_id[i] is canceled. When it is 0, it indicates that the information related to the viewport with the viewport identifier viewport_id[i] is indicated in the sample.

[0122] camera_intrinsic_flag[i] is used to indicate whether the i-th window has camera intrinsic parameters. When camera_intrinsic_flag[i] is 1, it means that the i-th window in the current sample has camera intrinsic parameters. When camera_intrinsic_flag[i] is 0, it means that the i-th window in the current sample does not indicate camera intrinsic parameters. In addition, when dynamic_int_camera_flag is 0, the field camera_intrinsic_flag[i] is also 0.

[0123] camera_extrinsic_flag[i] is used to indicate whether the i-th window has camera extrinsic parameters. If camera_extrinsic_flag[i] is 1, it means that the i-th window in the current sample has camera extrinsic parameters. If camera_extrinsic_flag[i] is 0, it means that the i-th window in the current sample does not indicate camera extrinsic parameters. In addition, if dynamic_ext_camera_flag is 0, the camera_extrinsic_flag[i] field is also 0.

[0124] slice_info_flag[i] is used to indicate whether the i-th window in the current sample is associated with the corresponding slice information. When slice_info_flag[i] is 1, it means that the i-th window in the current sample is associated with the corresponding slice information. When slice_info_flag[i] is 0, it means that the i-th window in the current sample is not associated with the corresponding slice information.

[0125] num_slices is used to indicate the number of slices associated with the current window.

[0126] slice_id is used to indicate the identifier of the slice.

[0127] For example, as shown in Figure 7, the ISO media file may include a "moov" type box, track 1 and track 2 in the moov box, an "mdat" type box, and sample 1, sample 2, sample 3, etc. corresponding to each track in the mdat box, a second sample entry ViewportInfoSampleEntry in a "meta" type box, and corresponding object information ObjectInfoStruct.

[0128] In some embodiments, the point cloud data media file includes a first data box, where the first data box includes information about a static object.

[0129] Thus, object information can be obtained according to the first data box in the point cloud data media track sample entry.

[0130] The static object is used to indicate an object that is relatively fixed in the scene, for example, a person, object, animal, scene interaction point, etc. whose position in the scene remains relatively fixed for a certain period of time.

[0131] In some embodiments, the first data box may be identified according to the first data box identifier, and then the object information may be obtained from the first data box in the point cloud data media track sample entry.

[0132] In some embodiments, the first data box may be a geometry component track in a point cloud data media file, and the sample entry AVSPCCSampleEntry of the point cloud track may contain a static object data box objectBox, which is used to describe object information corresponding to the point cloud track and the point cloud geometry component track (and the associated attribute component track), including:

[0133] Among them, num_objects is used to indicate the number of objects.

[0134] ObjectInfoStruct is used to indicate the specific information of an object.

[0135] For example, as shown in Figure 8 , the ISO media file may include a "moov" type box, track 1 and track 2 in the moov box, a sample entry and an ObjectBox data box carrying object information corresponding to track 1, and a sample entry and an ObjectBox data box carrying object information corresponding to track 2. An "mdat" type box and sample 1, sample 2, sample 3, etc. corresponding to each track in the mdat box.

[0136] In some embodiments, when an object is closely associated with a spatial region in a spatial region information data box, the object information may be described in a spatial data box within the point cloud data media file. A spatial data box may contain one or more subspace data boxes, and the object information may be contained within one of the subspace data boxes. Furthermore, an indicator may be added to the point cloud data media file to indicate whether such object information is contained.

[0137] In some embodiments, when the position of an object in a scene / space or the corresponding slice changes relatively infrequently, the object information corresponding to each sample group may be described by grouping samples.

[0138] In some embodiments, num_objects and ObjectInfoStruct in the first sample entry ObjectInfoSampleEntry or the second sample entry ViewportInfoSampleEntry may also use the ObjectBox provided in the present disclosure to represent the number of objects contained and the object structure, and describe the object information.

[0139] In some embodiments, the dynamic window timed metadata track and the object information timed metadata track or the first data box may exist at the same time, wherein the dynamic window timed metadata track only contains the object identifier defined in the object information timed metadata track or the first data box, and the specific object information defined in the object information timed metadata track or the first data box can be associated through the object identifier.

[0140] S103: Perform media processing on the point cloud data in the point cloud data media file according to the object information.

[0141] For example, the content production end can encode the point cloud media, encapsulate the encoded data stream, and transmit it to the client. The client can then decapsulate, decode, and perform media processing on the point cloud file to present the media content recorded in the point cloud data media file to the user. The client can then render the resulting 3D image based on the rendering and viewport-related metadata in the media presentation description. Once the rendering is complete, the 3D image is played and output.

[0142] In some embodiments, the above-mentioned media processing includes differential rendering and / or differential data transmission.

[0143] In some embodiments, the above media processing may be implemented as follows:

[0144] The first point cloud data is processed based on a first processing method, and the second point cloud data is processed based on a second processing method.

[0145] The first point cloud data includes point cloud data corresponding to at least one object, and the point cloud data in the point cloud data media file includes first point cloud data and second point cloud data. The processing resources required by the first processing method are different from the processing resources required by the second processing method.

[0146] It should be noted that at least one object is a key area that needs to be presented to the user in high quality. Therefore, the first processing method can be used to process the first point cloud data, that is, high-quality media content needs to be used for its content, and more resources are allocated for media processing. The second point cloud data is a non-key area in the current scene that needs to be presented, so the second processing method can be used. This can both improve the user viewing experience and reduce the processing pressure on the network and terminal.

[0147] In some embodiments, the media presentation description (MPD) file corresponding to the point cloud data media file includes object information.

[0148] An MPD file is an extensible markup language (XML) document that describes dynamic adaptive streaming over HTTP (DASH) media content, codecs, bitrate, resolution, and other information, as well as how to dynamically select the most appropriate media stream under different network conditions. This allows DASH to adaptively adjust the transmission of media streams based on varying network bandwidth and device capabilities, providing a better user experience.

[0149] In some embodiments, in the media presentation description of HTTP dynamic adaptive streaming media, the scheme identifier attribute (@schemeIdUri) can be used to identify the object information descriptor (OBJI), and the corresponding media data can be requested and downloaded based on the descriptor. The descriptor is a SupplementalProperty element.

[0150] Object information descriptors can be included in the Adaptation Set, Representation level, or Preselection level, but the descriptor cannot appear in two levels at the same time. If the representation is in the lower level set, it is reused in the higher level.

[0151] When object_oid_flag is set to 0, the object description information is indicated in the form of a human-readable string. When it is set to 1, the object description information is indicated in the form of an OID, and the definition of OID follows the definition of ITU-T X.660 | ISO / IEC 9834-1.

[0152] object_info_id indicates the identifier of the corresponding object information item.

[0153] object_oid indicates object description information corresponding to the corresponding object information entry, which is indicated in the form of OID.

[0154] object_label indicates the object description information corresponding to the corresponding object information entry, which is indicated in the form of a string readable by the human eye.

[0155] When the value of object_spatial_info_flag is 0, it indicates that the specific spatial information corresponding to the object is not indicated; when the value is 1, it indicates the specific spatial information corresponding to the object.

[0156] When the value of object_slice_info_flag is 0, it indicates that the point cloud slice information corresponding to the object is not indicated; when the value is 1, it indicates the point cloud slice information corresponding to the object.

[0157] 3DPoint indicates the coordinates of the anchor point in the spatial region where the object is located;

[0158] CuboidRegionStruct indicates the size information of the spatial region where the object is located, that is, the extension of a 3D spatial region in a Cartesian coordinate system relative to the anchor point on the x, y, and z axes.

[0159] SliceMapping indicates point cloud slice information, which indicates the number of slices associated with the 3D spatial region and identifies the AVS PCC slice associated with the spatial region.

[0160] In some embodiments, as shown in Table 2, the present disclosure further provides semantics for describing the attributes of the viewing space description sub-element.

[0161] Table 2

[0162] In some embodiments, the point cloud data media file may also be obtained based on the object information contained in the MPD file.

[0163] Exemplarily, as shown in FIG9 , the method may further include the following S1-S4.

[0164] S1. When a change in the viewing position or direction of the user is detected, determine whether there is an object in the current user's viewing window.

[0165] For example, when the user's viewing position, viewing direction, etc. change, the client can obtain information such as the user's current viewing posture, determine the user's changed viewing position, viewing direction, etc., and then determine whether there is an object in the viewing content (window) of the user's changed viewing position and viewing direction, and whether there is corresponding object information in the corresponding point cloud data media file.

[0166] Furthermore, the client can determine whether the object information is included in the MPD file.

[0167] When the object information is included in the MPD file, the following S2 and S4 are performed.

[0168] Otherwise, if the object information is not included in the MPD file, the following S3 and S4 are executed.

[0169] S2. The MPD file contains object information, and the point cloud data media file is obtained according to the object information contained in the MPD file.

[0170] For example, there is object information in the MPD file, and the client can request the server to perform differentiated processing on the media data of the object, such as sending the viewing content to the client after media processing (such as rendering) of the object, the client requests high-quality media data of the object, or the client requests priority transmission of the media data of the object, etc.

[0171] S3 and MPD files do not contain object information. The client can directly request the point cloud data media file corresponding to the user's current viewing posture.

[0172] S4. The client performs media processing, content presentation, and other processing based on the acquired point cloud data media files.

[0173] Based on the technical solutions provided by the present disclosure, for various scenarios of immersive media such as user windows, media processing based on user position, and static objects in the window, the technical solutions provided by the present disclosure can provide various corresponding media processing methods for various scenarios, so as to perform differentiated transmission and media processing based on the object information of the partial scene and one or more point cloud data objects in the scene within the window range of each scenario, thereby meeting the various scenario requirements of immersive media and improving the flexibility of differentiated media processing. In addition, differentiated media processing, for example, uses higher-quality media content for the point cloud data corresponding to the point cloud data object and allocates more resources for media processing to provide a better immersive experience and optimize the presentation quality of the media. Other areas can also use lower-quality media content and allocate fewer resources for media processing. This can also reduce the processing pressure on the network and terminal, thereby improving the transmission efficiency and processing capacity of the media content.

[0174] The above mainly introduces the solution provided by the present disclosure from the perspective of the interaction between various communication nodes. It is understandable that each node, for example, a device or equipment, includes a hardware structure and / or software module corresponding to the execution of each function in order to realize the above functions. It should be easy for those skilled in the art to realize that, in combination with the algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present disclosure.

[0175] The embodiments of the present disclosure can divide the functional modules of the communication device according to the above-mentioned method embodiments. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one functional module. The above-mentioned integrated modules can be implemented in the form of hardware or software. It should be noted that the division of modules in the embodiments of the present disclosure is schematic and is only a logical functional division. In actual implementation, there may be other division methods. The following is an example of dividing each functional module corresponding to each function.

[0176] FIG10 is a schematic diagram showing the composition of a point cloud data processing device provided by an embodiment of the present disclosure. As shown in FIG10 , the point cloud data processing device 100 includes a receiving module 1001 and a processing module 1002 .

[0177] Receiving module 1001 is configured to receive a point cloud data media file; the point cloud data media file includes a timed metadata track associated with an object and / or a data box associated with the object, the timed metadata track is used to indicate object metadata in the media, and the data box is used to indicate object metadata in the media;

[0178] The processing module 1002 is configured to obtain object information in the point cloud data media file based on the point cloud data media file indicating the object metadata;

[0179] The processing module 1002 is further configured to perform media processing on the point cloud data in the point cloud data media file according to the object information.

[0180] In some embodiments, the receiving module 1001 is used to obtain a point cloud data media file based on the user's viewing position or recommended window; wherein the point cloud data media file includes object information and / or window information associated with the object.

[0181] In some embodiments, the point cloud data media file includes a timed metadata track associated with the object, including: a first timed metadata track, containing dynamic object information contained in the media, the first timed metadata track is identified by a type identifier of a first sample entry; and / or, a second timed metadata track, containing dynamic window information and object information in the window, the second timed metadata track is identified by a type identifier of a second sample entry.

[0182] In some embodiments, the second timed metadata track sample entry includes a first identifier, where the first identifier is used to indicate a view type containing object information.

[0183] In some embodiments, the processing module 1002 is configured to identify point cloud data in the point cloud data media file according to metadata sample entries in the first timed metadata track or the second timed metadata track to obtain object information.

[0184] In some embodiments, the point cloud data media file includes a first data box, where the first data box includes information about a static object.

[0185] In some embodiments, the processing module 1002 is configured to obtain object information according to a first data box in a point cloud data media track sample entry.

[0186] In some embodiments, the processing module 1002 is further configured to identify the first data box according to the first data box identifier.

[0187] In some embodiments, the media presentation description MPD file corresponding to the point cloud data media file includes object information.

[0188] In some embodiments, the point cloud data media file is obtained according to the object information contained in the MPD file.

[0189] In some embodiments, the object information includes: an identifier of the object information entry, object description information corresponding to the object information entry, anchor point coordinates of the spatial area where the object is located, size information of the spatial area where the object is located, and one or more of the point cloud information containing the object.

[0190] In some embodiments, the processing module 1002 is used to process the first point cloud data based on a first processing method; wherein the first point cloud data includes point cloud data corresponding to at least one object, and the point cloud data in the point cloud data media file includes first point cloud data and second point cloud data; the second point cloud data is processed based on a second processing method; the processing resources required for the first processing method are different from the processing resources required for the second processing method.

[0191] In some embodiments, media processing includes differential rendering and / or differential data transfer.

[0192] For a more detailed description of the above-mentioned receiving module 1001, processing module 1002, a more detailed description of each technical feature therein, and a description of the beneficial effects, etc., please refer to the above-mentioned corresponding method embodiment part, which will not be repeated here.

[0193] It should be noted that the modules in Figure 10 may also be referred to as units, for example, the processing module may be referred to as a processing unit. In addition, in the embodiment shown in Figure 10, the names of the modules may not be the names shown in the figure, for example, the receiving module may also be referred to as a communication module.

[0194] If the various units in Figure 10 are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present disclosure. The storage medium for storing computer software products includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the present disclosure provides a schematic diagram of the structure of a point cloud data processing device. As shown in Figure 11, the processing device 110 includes: a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104.

[0196] The memory 1101 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store dynamic information and instructions, an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0197] The processor 1102 may be a logic block, module, and circuit that implements or executes the various exemplary methods described in conjunction with the present disclosure. The processor 1102 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor 1102 may also implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure. The processor 1102 may also be a combination that implements computing functions, for example, a combination of one or more microprocessors, a combination of a DSP (digital signal processor) and a microprocessor, etc.

[0198] The communication interface 1103 is used to connect to other devices via a communication network, such as Ethernet, wireless access network, or wireless local area network (WLAN).

[0199] In some embodiments, the memory 1101 may exist independently of the processor 1102 and may be connected to the processor 1102 via a bus 1104 for storing instructions or program codes. When the processor 1102 calls and executes the instructions or program codes stored in the memory 1101, the methods provided in the embodiments of the present disclosure can be implemented.

[0200] In some embodiments, the memory 1101 may also be integrated with the processor 1102 .

[0201] Bus 1104 can be an Extended Industry Standard Architecture (EISA) bus, for example. Bus 1104 can be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, FIG11 shows bus 1104 with only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0202] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be allocated as needed and completed by different functional modules, that is, the internal structure of the equipment or device is divided into different functional modules to complete all or part of the functions described above.

[0203] The present disclosure also provides a computer-readable storage medium. All or part of the processes in the above-mentioned method embodiments can be completed by instructing the relevant hardware by a computer program, and the program can be stored in the above-mentioned computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The computer-readable storage medium can be an internal storage unit of the device or apparatus of any of the above-mentioned embodiments, for example, a hard disk or memory of a computer device. The above-mentioned computer-readable storage medium can also be an external storage device of the above-mentioned device or apparatus, for example, a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the above-mentioned device or apparatus. Further, the above-mentioned computer-readable storage medium can also include both an internal storage unit of the above-mentioned device or apparatus and an external storage device. The above-mentioned computer-readable storage medium is used to store the above-mentioned computer program and other programs and data required by the above-mentioned device or apparatus. The above-mentioned computer-readable storage medium can also be used to temporarily store data that has been output or is to be output. For example, the above-mentioned computer-readable storage medium includes a non-transitory computer-readable storage medium.

[0204] An embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program product is run on a computer, the computer is enabled to execute any one of the methods provided in the above embodiments.

[0205] Although the present disclosure is described herein in conjunction with various embodiments, in the process of implementing the disclosure for which protection is sought, those skilled in the art may understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple components. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0206] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely illustrative of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various changes and modifications to the present disclosure without departing from the scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these changes and variations. The above is only a specific embodiment of the present disclosure, but the scope of protection of the present disclosure is not limited thereto, and any changes or substitutions within the technical scope disclosed in the present disclosure should be covered within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for processing point cloud data, comprising: Receive point cloud data media files; The point cloud data media file includes a timed metadata track associated with the object and / or a data box associated with the object, the timed metadata track is used to indicate object metadata in the media, and the data box is used to indicate object metadata in the media; obtaining object information in the point cloud data media file based on the point cloud data media file indicating the object metadata; Perform media processing on the point cloud data in the point cloud data media file according to the object information.

2. The method according to claim 1, wherein The receiving of the point cloud data media file includes: The point cloud data media file is obtained according to the user's viewing position or recommended window; wherein the point cloud data media file includes object information and / or window information associated with the object.

3. The method according to claim 1 or 2, wherein: The point cloud data media file includes a timed metadata track associated with the object, including: A first timed metadata track comprising dynamic object information contained in the media, the first timed metadata track being identified by a type identifier of a first sample entry; and / or, The second timed metadata track includes dynamic window information and object information in the window, and the second timed metadata track is identified by a type identifier of the second sample entry.

4. The method according to claim 3, wherein: The second timed metadata track sample entry includes a first identifier, where the first identifier is used to indicate the type of the window containing the object information.

5. The method according to claim 3, wherein The acquiring, based on the point cloud data media file indicating the object metadata, object information in the point cloud data media file includes: Point cloud data in the point cloud data media file is identified according to metadata sample entries in the first timed metadata track or the second timed metadata track to obtain the object information.

6. The method according to claim 1 or 2, wherein: The point cloud data media file includes a first data box, which includes information about a static object.

7. The method according to claim 6, wherein: Obtaining object information in the point cloud data media file based on the point cloud data media file indicating the object metadata includes: Obtain object information according to the first data box in the point cloud data media track sample entry.

8. The method according to claim 7, further comprising: The first data box is identified according to the first data box identifier.

9. The method according to claim 1, wherein The media presentation description MPD file corresponding to the point cloud data media file includes the object information.

10. The method according to claim 9, wherein: The point cloud data media file is obtained according to the object information contained in the MPD file.

11. The method according to claim 1, wherein The object information includes: One or more of the following: the identifier of the object information entry, the object description information corresponding to the object information entry, the anchor point coordinates of the spatial area where the object is located, the size information of the spatial area where the object is located, and the point cloud information containing the object.

12. The method according to claim 1, wherein The performing media processing on the point cloud data in the point cloud data media file according to the object information includes: Processing the first point cloud data based on a first processing method; wherein the first point cloud data includes point cloud data corresponding to at least one of the objects, and the point cloud data in the point cloud data media file includes the first point cloud data and the second point cloud data; The second point cloud data is processed based on a second processing method; the processing resources required by the first processing method are different from the processing resources required by the second processing method.

13. The method according to claim 1, wherein The media processing includes differential rendering and / or differential data transmission.

14. A point cloud data processing device, comprising: A memory and a processor; wherein the memory and the processor are coupled; the memory is used to store instructions executable by the processor; when the processor executes the instructions, it performs the method according to any one of claims 1 to 13.

15. A computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 13.

Images